在 Kubernetes 中要实现容器之间网络的隔离,是通过一个专门的 API 对象 NetworkPolicy
(网络策略)来实现的,要让网络策略生效,就需要特定的网络插件支持,目前已经实现了 NetworkPolicy
的网络插件包括 Calico、Weave 和 kube-router 等项目,但是并不包括 Flannel 项目。所以说,如果想要在使用 Flannel 的同时还使用 NetworkPolicy 的话,你就需要再额外安装一个网络插件,比如 Calico 项目,来负责执行 NetworkPolicy。本测试环境使用的是 Calico 网络插件,可以直接使用
默认情况下 Pod 是可以接收来自任何发送方的请求,也可以向任何接收方发送请求。如果要对这个情况作出限制,就必须通过 NetworkPolicy
对象来指定。
以下定义了一个网络策略资源清单文件,内容如下:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:name: network-policynamespace: default
spec:podSelector: matchLabels:app: nginxpolicyTypes:- Ingress- Egressingress:- from:- ipBlock:cidr: 10.233.0.0/16except:- 10.233.93.0/24- namespaceSelector:matchLabels:project: test- podSelector:matchLabels:app: busybox- ports:- protocol: TCPport: 80egress:- to:- ipBlock:cidr: 10.233.93.0/24ports:- protocol: TCPport: 5978
spec:podSelector: matchLabels:app: nginx ###定义了pod对象,表示当前ns中标签为app: nginx的pod
如果想要NetworkPolicy作用于当前ns中的所有pod,则用如下形式:
spec:podSelector: {}
spec:policyTypes: ###网络策略的类型- Ingress- Egress
每个 NetworkPolicy 包含一个 policyTypes 列表,可以是一个 Ingress、Egress 或者都包含,该字段表示给当前策略是否应用于所匹配的 Pod 的入口流量、出口流量或者二者都包含,如果没有指定 policyTypes,则默认情况下表示 Ingress 入口流量,如果配置了任何出口流量规则,则将指定为 Egress。
ingress: ###配置pod的ingress策略- from:- ipBlock: ###配置允许的网络访问cidr: 10.233.0.0/16except: ###不允许某个网络访问- 10.233.93.0/24- namespaceSelector: ###允许标签为project: test的ns下的所有的pod访问matchLabels:project: test- podSelector: ###此处代表当前ns下的pod,默认是允许同一ns下的pod互通,添加了此处选项代表只允许当前ns下label为app: busybox的pod访问label为app: nginx的pod应用matchLabels:app: busybox- ports: ####允许被访问的端口- protocol: TCPport: 80一旦 Pod 被 NetworkPolicy 选中,那么这个 Pod 就会进入“拒绝所有”(Deny All)的状态,即这个 Pod 既不允许被外界访问,也不允许对外界发起访问,所以 NetworkPolicy 定义的规则,其实就是“白名单”了。
在default的ns下启动两个web服务的pod,如下:
[root@master ~]# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx 1/1 Running 0 9d 10.233.96.3 node2
web 1/1 Running 0 9d 10.233.96.4 node2
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:name: denynetwork
spec:podSelector:matchLabels:app: nginx ####被访问pod的label标签policyTypes:- Egress- Ingress ###添加规则类型ingress:- from:- podSelector: ###此处代表当前ns下的pod,默认是允许同一ns下的pod互通,添加了此处选项代表只允许当前ns下label为app: web的pod访问label为app: nginx的pod应用matchLabels:app: web创建如下:
[root@master ~]# kubectl get networkpolicy
NAME POD-SELECTOR AGE
denynetwork app=nginx 9d ###可以看到被访问的pod标签
[root@master ~]#
##############
如上,web pod位于node2上,进入pod的网络命令空间[root@node2 ~]# crictl ps | grep web
dcf892535ed93 3f8a00f137a0d 9 days ago Running count 0 205811ac7b0f0 web
[root@node2 ~]#
[root@node2 ~]# crictl inspect dcf892535ed93 | grep -i pid"pid": 16814,"pid": 1"type": "pid"
[root@node2 ~]#
[root@node2 ~]# nsenter -t 16814 -n bash ####进入pod的网络命令空间
[root@node2 ~]# ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: tunl0@NONE: mtu 1480 qdisc noop state DOWN group default qlen 1000link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if8: mtu 1480 qdisc noqueue state UP group default link/ether 9a:6b:07:ab:e4:0d brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 10.233.96.4/32 scope global eth0valid_lft forever preferred_lft foreverinet6 fe80::986b:7ff:feab:e40d/64 scope link valid_lft forever preferred_lft forever
[root@node2 ~]# ping 10.233.96.3 #####此处可以看到无放访问nginx pod
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
^C
--- 10.233.96.3 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 5999ms[root@node2 ~]# #######################
######################
为web pod添加app: web的标签,如下:
root@master ~]# kubectl label po web app=web
pod/web labeled
[root@master ~]# #####################
再次测试访问nginx,如下:
root@master ~]# ssh node2
Last login: Sat Mar 4 14:26:28 2023 from 192.168.5.240[root@node2 ~]# nsenter -t 16814 -n bash
[root@node2 ~]# ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: tunl0@NONE: mtu 1480 qdisc noop state DOWN group default qlen 1000link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if8: mtu 1480 qdisc noqueue state UP group default link/ether 9a:6b:07:ab:e4:0d brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 10.233.96.4/32 scope global eth0valid_lft forever preferred_lft foreverinet6 fe80::986b:7ff:feab:e40d/64 scope link valid_lft forever preferred_lft forever
[root@node2 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
64 bytes from 10.233.96.3: icmp_seq=1 ttl=63 time=0.333 ms
64 bytes from 10.233.96.3: icmp_seq=2 ttl=63 time=0.127 ms
^C
--- 10.233.96.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.127/0.230/0.333/0.103 ms[root@node2 ~]# curl 10.233.96.3 ####可以正常访问服务
Welcome to nginx!
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
[root@node2 ~]#
编辑networkpolicy.yaml文件
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:name: denynetwork
spec:podSelector:matchLabels:app: nginx ####被访问pod的label标签policyTypes:- Egress- Ingress ###添加规则类型ingress:- from:- namespaceSelector: ###添加了此处选项代表只允许label为prject:test的ns下的pod访问label为app: nginx的pod应用matchLabels:project: test创建如下:
[root@master ~]# kubectl get networkpolicy
NAME POD-SELECTOR AGE
denynetwork app=nginx 9d ###可以看到被访问的pod标签
创建ns以及测试pod
#######创建ns
[root@master ~]# kubectl create ns test#####创建测试pod
apiVersion: v1
kind: Pod
metadata:name: busyboxnamespace: test #####选择test的ns
spec:containers:- name: busyboximage: docker.io/library/busybox:latestimagePullPolicy: IfNotPresentcommand: ["init"] 启动pod如下:
[root@master ~]# kubectl get po -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 9d 10.233.90.4 node1
[root@master ~]# ###################
进入busybox的网络命令,如下:
[root@node1 ~]# crictl ps | grep busybox
755e001322ef6 2fb6fc2d97e10 9 days ago Running busybox 0 b5e1dfe3d3746 busybox
[root@node1 ~]#
[root@node1 ~]#
[root@node1 ~]# crictl inspect 755e001322ef6| grep -i pid"pid": 44236,"pid": 1"type": "pid"[root@node1 ~]# nsenter -t 44236 -n bash
[root@node1 ~]# ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: tunl0@NONE: mtu 1480 qdisc noop state DOWN group default qlen 1000link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if10: mtu 1480 qdisc noqueue state UP group default link/ether 42:2b:78:3a:6a:c1 brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 10.233.90.4/32 scope global eth0valid_lft forever preferred_lft foreverinet6 fe80::402b:78ff:fe3a:6ac1/64 scope link valid_lft forever preferred_lft forever
[root@node1 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
^C
--- 10.233.96.3 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms[root@node1 ~]# ##############为test ns添加标签,如下:
[root@master ~]# kubectl label ns test project=test
namespace/test labeled
[root@master ~]# 再次测试如下:[root@node1 ~]#
[root@node1 ~]# nsenter -t 44236 -n bash
[root@node1 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
64 bytes from 10.233.96.3: icmp_seq=1 ttl=62 time=2.35 ms
64 bytes from 10.233.96.3: icmp_seq=2 ttl=62 time=1.46 ms
^C
--- 10.233.96.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.468/1.910/2.352/0.442 ms
[root@node1 ~]# curl 10.233.96.3
Welcome to nginx!
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
[root@node1 ~]#
编辑networkpolicy.yaml文件:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:name: denynetworknamespace: default
spec:podSelector: matchLabels:app: nginxpolicyTypes:- Ingress- Egressingress:- from:- ipBlock:cidr: 10.233.0.0/16 #####允许访问的网段except:- 10.233.93.0/24 ####排除此网段(此网段位于node1,所以测试pod也位于node1)
创建测试pod
###########创建pod,如下:
[root@master ~]# kubectl get po -o wide -n test
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 9d 10.233.90.4 node1 ##########
进入pod的网络命名空间
[root@master ~]# ssh node1
Last login: Sat Mar 4 14:55:41 2023 from 192.168.5.240
[root@node1 ~]# crictl ps | grep busybox
755e001322ef6 2fb6fc2d97e10 9 days ago Running busybox 0 b5e1dfe3d3746 busybox
[root@node1 ~]#
[root@node1 ~]# crictl inspect 755e001322ef6 | grep -i pid"pid": 44236,"pid": 1"type": "pid"
[root@node1 ~]#
[root@node1 ~]#
[root@node1 ~]# nsenter -t 44236 -n bash
[root@node1 ~]#
[root@node1 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
^C
--- 10.233.96.3 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5000ms[root@node1 ~]# #############
修改networkpolicy,取消expect参数
[root@master ~]# kubectl edit networkpolicy denynetwork
networkpolicy.networking.k8s.io/denynetwork edited
[root@master ~]# kubectl describe networkpolicy denynetwork
Name: denynetwork
Namespace: default
Created on: 2023-02-22 22:39:54 +0800 CST
Labels:
Annotations:
Spec:PodSelector: app=nginxAllowing ingress traffic:To Port: (traffic allowed to all ports)From:IPBlock:CIDR: 10.233.0.0/16Except: ####已经去掉了10.233.93.0/24的网段的限制Allowing egress traffic: (Selected pods are isolated for egress connectivity)Policy Types: Egress, Ingress
[root@master ~]# #######################
再此测试如下:[root@node1 ~]#
[root@node1 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
64 bytes from 10.233.96.3: icmp_seq=1 ttl=62 time=2.76 ms
64 bytes from 10.233.96.3: icmp_seq=2 ttl=62 time=1.55 ms
^C
--- 10.233.96.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.556/2.158/2.761/0.604 ms
[root@node1 ~]# curl 10.233.96.3
Welcome to nginx!
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
[root@node1 ~]#
每个 NetworkPolicy 包含一个 egress 规则的白名单列表。每个规则都允许匹配 to 和 port 部分的流量。比如我们这里示例规则的配置:
egress:- to:- ipBlock:cidr: 10.0.0.0/24 ##允许访问10.0.0.0段的80端口ports:- protocol: TCPport: 80表示 Kubernetes 会拒绝被隔离 Pod 对外发起任何请求,除非请求的目的地址属于 10.0.0.0/24 网段,并且访问的是该网段地址的 80 端口。
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:name: denynetworknamespace: default
spec:podSelector: matchLabels:app: nginxpolicyTypes:- Ingress- Egressegress:- to:- ipBlock:cidr: 10.233.90.0/24 #####允许访问的网段ports:- protocol: TCPport: 80 ###允许被访问网络的端口#######创建策略查看
[root@master ~]# kubectl describe networkpolicy denynetwork
Name: denynetwork
Namespace: default
Created on: 2023-02-22 22:39:54 +0800 CST
Labels:
Annotations:
Spec:PodSelector: app=nginxAllowing ingress traffic: (Selected pods are isolated for ingress connectivity)Allowing egress traffic:To Port: 80/TCPTo:IPBlock:CIDR: 10.233.90.0/24Except: Policy Types: Ingress, Egress
[root@master ~]# ###########找到app=nginx的pod并访问
[root@master ~]# kubectl get po -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
egress-test 1/1 Running 0 70s 10.233.90.6 node1 [root@master ~]# kubectl get po -o wide | grep nginx
nginx 1/1 Running 0 9d 10.233.96.3 node2 ###############进入app=nginx的pod的命令空间访问
[root@node2 ~]# crictl ps | grep nginx
27fd22bdf596d 3f8a00f137a0d 9 days ago Running count 0 cb2afd0d86bcf nginx
[root@node2 ~]# crictl inspect 27fd22bdf596d| grep -i pid"pid": 11243,"pid": 1"type": "pid"
[root@node2 ~]# nsenter -t 11243 -n bash
[root@node2 ~]#
[root@node2 ~]# ping 10.233.90.6 ###因为只允许了tcp协议,所以ping不通
PING 10.233.90.6 (10.233.90.6) 56(84) bytes of data.
^C
--- 10.233.90.6 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 2999ms[root@node2 ~]# curl 10.233.90.6 ####如下可正常访问80端口
Welcome to nginx!
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
[root@node2 ~]# ###########编辑networkpolicy ,将80端口改为81
[root@master ~]# kubectl describe networkpolicy denynetwork
Name: denynetwork
Namespace: default
Created on: 2023-02-22 22:39:54 +0800 CST
Labels:
Annotations:
Spec:PodSelector: app=nginxNot affecting ingress trafficAllowing egress traffic:To Port: 81/TCPTo:IPBlock:CIDR: 10.233.90.0/24Except: Policy Types: Egress
[root@master ~]# #############再次访问10.233.90.6的80端口如下:
[root@node2 ~]# curl 10.233.90.6 ###无法访问
^C
[root@node2 ~]# ###########编辑networkpolicy,放通整个网段测试
[root@master ~]# kubectl edit networkpolicy denynetwork
networkpolicy.networking.k8s.io/denynetwork edited
[root@master ~]# kubectl describe networkpolicy denynetwork
Name: denynetwork
Namespace: default
Created on: 2023-02-22 22:39:54 +0800 CST
Labels:
Annotations:
Spec:PodSelector: app=nginxNot affecting ingress trafficAllowing egress traffic:To Port: (traffic allowed to all ports)To:IPBlock:CIDR: 10.233.90.0/24Except: Policy Types: Egress
[root@master ~]# ########再次访问10.233.90.6,如下:
[root@node2 ~]# ping 10.233.90.6
PING 10.233.90.6 (10.233.90.6) 56(84) bytes of data.
64 bytes from 10.233.90.6: icmp_seq=1 ttl=62 time=1.56 ms
^C
--- 10.233.90.6 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.560/1.560/1.560/0.000 ms
[root@node2 ~]#
[root@node2 ~]# curl 10.233.90.6
Welcome to nginx!
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
[root@node2 ~]# ########################
但是无法访问其他节点的pod,即使是同网段和节点,如下:
[root@master ~]# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx 1/1 Running 0 9d 10.233.96.3 node2
web 1/1 Running 0 9d 10.233.96.4 node2
[root@node2 ~]# ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: tunl0@NONE: mtu 1480 qdisc noop state DOWN group default qlen 1000link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if7: mtu 1480 qdisc noqueue state UP group default link/ether 56:52:52:7c:82:81 brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 10.233.96.3/32 scope global eth0valid_lft forever preferred_lft foreverinet6 fe80::5452:52ff:fe7c:8281/64 scope link valid_lft forever preferred_lft forever
[root@node2 ~]# ping 10.233.96.4
PING 10.233.96.4 (10.233.96.4) 56(84) bytes of data.
^C
--- 10.233.96.4 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms[root@node2 ~]# curl 10.233.96.4
^C
[root@node2 ~]#