Consul实现Docker Vxlan
首先,我们将构建一个Overlay网络在三个Docker hosts之间,二个节点运行Docker,另一个节点运行Consul,Docker将会使用Consul来存储Overlay网络metadadata
启动Consul服务
Consul节点启动服务
# 启动Consul服务
[root@consul ~]# nohup consul agent -server -dev -ui -client 0.0.0.0 &
[1] 14532
[root@consul ~]# nohup: ignoring input and appending output to ‘nohup.out’
# agent: 启动一个agent
# dev: 创建一个开发者模式的consul
# server: 以server模式启动consul
# ui: 提供一个可视化的UI界面
# client: 设置bind地址
登录Consul
启动自定义dockerd进程
--cluster-store=
参数指向docker daemon所使用key value service的地址(本例中即consul的服务地址)--cluster-advertise=
参数决定了所使用网卡以及docker daemon端口信息-H参数
分别指定了docker demon服务的地址和协议
# 关闭Docker
systemctl disable docker --now
# 二个节点启动自定义Docker
[root@docker1 ~]# nohup dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --cluster-store consul://192.168.0.13:8500 --cluster-advertise ens33:2376 &
[1] 15048
[root@docker1 ~]# nohup: ignoring input and appending output to ‘nohup.out’
[root@docker2 ~]# nohup dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --cluster-store consul://192.168.0.13:8500 --cluster-advertise ens33:2376 &
[1] 2620
[root@docker2 ~]# nohup: ignoring input and appending output to ‘nohup.out’
待二个dockerd进程启动后,可以发现consul已经拿到相关数据了
流量转发图
由于后续分析十分枯燥,这里放上流量转发图,供大家参考
创建Overlay网络
- 在Docker1创建一个Overlay网络,
通过下面可见Docker1的网络信息通过Consul传递给Docker2了
# 创建一个Overlay网络
[root@docker1 ~]# docker network create --driver overlay --subnet 10.70.2.0/24 demonset
743aa849b76afd5a33298083cd4c54548207ddd4cb95ae276c00009936bdadd8
# 查看Docker1网络
[root@docker1 ~]# docker network ls
NETWORK ID NAME DRIVER SCOPE
59bdb4c8ea2b bridge bridge local
1be3297604d1 demonset overlay global
254821e742a1 docker_gwbridge bridge local
969166ca1bf5 host host local
e7736b1206fe none null local
# 查看Docker2网络
[root@docker2 ~]# docker network ls
NETWORK ID NAME DRIVER SCOPE
186f412ffc45 bridge bridge local
1be3297604d1 demonset overlay global
4afd26fa8e1b docker_gwbridge bridge local
e0e57d329138 host host local
02ebc9feeb3b none null local
启动二个容器验证网络
可见通过Consul共享二个Docker之间的数据后,这二个容器实现了夸主机ping,且是在他们自己的一个网段10.70.2.0/24内通信
此时流量转发大致如下图所示
# Docker1启动一个容器
[root@docker1 ~]# docker run -d --ip 10.70.2.111 --net demonset --name busybox1 busybox:1.28 sleep 100000
a00e2a1828c5147091f52e1cbc5ff87d9148369b5ecb202c5209244774fd3e43
# Docker2启动一个容器
[root@docker2 ~]# docker run -d --ip 10.70.2.112 --net demonset --name busybox2 busybox:1.28 sleep 100000
1c6902b826a7da0d22c59b877294d72d696b02df96fccff0ec78a0de08a06f37
# Docker2 ping Docker1测试
[root@docker2 ~]# docker exec busybox2 ping -c 2 10.70.2.111
PING 10.70.2.111 (10.70.2.111): 56 data bytes
64 bytes from 10.70.2.111: seq=0 ttl=64 time=0.548 ms
64 bytes from 10.70.2.111: seq=1 ttl=64 time=0.256 ms
--- 10.70.2.111 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.256/0.402/0.548 ms
网络容器网络浅析
由于实验做了很久,当时没有完整记录分析过程,这里只记录了一部分,只提供给大家一个分析思路
docker1节点
# 查看busybox1网卡信息
[root@docker1 ~]# docker exec busybox1 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
7: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue
link/ether 02:42:0a:46:02:6f brd ff:ff:ff:ff:ff:ff
inet 10.70.2.111/24 brd 10.70.2.255 scope global eth0
valid_lft forever preferred_lft forever
9: eth1@if10: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1
valid_lft forever preferred_lft forever
# 查看busybox1路由信息
[root@docker1 ~]# docker exec busybox1 ip route show
default via 172.18.0.1 dev eth1
10.70.2.0/24 dev eth0 scope link src 10.70.2.111
172.18.0.0/16 dev eth1 scope link src 172.18.0.2
# 查看docker1宿主机网络信息
[root@docker1 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:4c:30:4e brd ff:ff:ff:ff:ff:ff
inet 192.168.0.11/24 brd 192.168.0.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::624c:c1db:e3b4:9165/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::2e8f:238e:c202:4027/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:22:fd:10:94 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge
valid_lft forever preferred_lft forever
inet6 fe80::42:22ff:fefd:1094/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:e6:92:c4:d8 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
10: veth0475f3f@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default
link/ether 2a:8e:ee:f7:af:f3 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::288e:eeff:fef7:aff3/64 scope link
valid_lft forever preferred_lft forever
# 查看docker1 路由信息
[root@docker1 ~]# ip route show
default via 192.168.0.2 dev ens33 proto static metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1
192.168.0.0/24 dev ens33 proto kernel scope link src 192.168.0.11 metric 100
# 确定docker_gwbridge 下行设备
[root@docker1 ~]# brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.0242e692c4d8 no
docker_gwbridge 8000.024222fd1094 no veth0475f3f
# 查看docker_gwbridge设备信息
[root@docker1 ~]# docker inspect docker_gwbridge
[
{
"Name": "docker_gwbridge",
..................
"Driver": "bridge",
"IPAM": {
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
}
]
},
"Options": {
"com.dockerwork.bridge.enable_icc": "false",
"com.dockerwork.bridge.enable_ip_masquerade": "true",
"com.dockerwork.bridge.name": "docker_gwbridge"
},
}
]
# 此网络使用驱动桥(与标准 docker 桥 docker0 使用的相同)
# 它使用子网 172.18.0.0/16,与 eth1 一致
# enable_icc 设置为 false,这意味着我们不能使用这个桥接器进行容器间通信
# enable_ip_masquerade 设置为 true,这意味着来自容器的流量将被 NAT 以访问外部网络(我们之前在成功 ping 8.8.8.8 时看到过)
docker2节点
# 查看busybox2网卡信息
[root@docker2 ~]# docker exec busybox2 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
7: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue
link/ether 02:42:0a:46:02:70 brd ff:ff:ff:ff:ff:ff
inet 10.70.2.112/24 brd 10.70.2.255 scope global eth0
valid_lft forever preferred_lft forever
9: eth1@if10: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1
valid_lft forever preferred_lft forever
# 查看busybox2路由信息
[root@docker2 ~]# docker exec busybox2 ip route show
default via 172.18.0.1 dev eth1
10.70.2.0/24 dev eth0 scope link src 10.70.2.112
172.18.0.0/16 dev eth1 scope link src 172.18.0.2
# 查看docker2宿主机网络信息
[root@docker2 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:19:a1:51 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.12/24 brd 192.168.0.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::624c:c1db:e3b4:9165/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::2e8f:238e:c202:4027/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::919c:e504:f9af:83db/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:1b:fe:6a:6d brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
4: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:97:2e:c3:e1 brd ff:ff:ff:ff:ff:ff
10: veth1b1e9f6@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default
link/ether 4e:5b:1e:f6:37:62 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::4c5b:1eff:fef6:3762/64 scope link
valid_lft forever preferred_lft forever
# 查看docker2路由信息
[root@docker2 ~]# ip route show
default via 192.168.0.2 dev ens33 proto static metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1
192.168.0.0/24 dev ens33 proto kernel scope link src 192.168.0.12 metric 100
# 确定docker_gwbridge 下行设备
[root@docker2 ~]# brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.02421bfe6a6d no
docker_gwbridge 8000.0242972ec3e1 no veth1b1e9f6
查找eth0@if7
- 基于上面的分析我们可以发现一个问题:
veth设备都是成对出现,但是busybox容器内出现了一个eth0@if7,我们在宿主机内找不到index=7的网卡,那么这张网卡必定存在其他的命名空间内
- 基于下面信息,我们可以得到一个信息:
docker1,docker2之间共享了一个network namespace 1-c8669d70d4
,br0设备下面连接了二个设备:vxlan0@if5和veth0@if6
# 查看docker1 容器使用了哪些命令空间
[root@docker1 ~]# ls /var/run/docker/netns
1-743aa849b7 9c002f20bb33
[root@docker1 ~]# nsenter --net=/var/run/docker/netns/1-743aa849b7 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 1e:08:06:d4:40:6b brd ff:ff:ff:ff:ff:ff
inet 10.70.2.1/24 brd 10.70.2.255 scope global br0
valid_lft forever preferred_lft forever
5: vxlan0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UNKNOWN group default
link/ether 26:de:18:1e:16:c2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: veth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UP group default
link/ether 1e:08:06:d4:40:6b brd ff:ff:ff:ff:ff:ff link-netnsid 1
# 查看路由信息
[root@docker1 ~]# nsenter --net=/var/run/docker/netns/1-c8669d70d4 ip route show
10.70.2.0/24 dev br0 proto kernel scope link src 10.70.2.1
# 查看br0设备连接情况
[root@docker1 ~]# nsenter --net=/var/run/docker/netns/1-c8669d70d4 brctl show br0
bridge name bridge id STP enabled interfaces
br0 8000.1e0806d4406b no vxlan0
veth0
[root@docker1 ~]# nsenter --net=/var/run/docker/netns/3fe21b67885e ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 02:42:0a:46:02:6f brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.70.2.111/24 brd 10.70.2.255 scope global eth0
valid_lft forever preferred_lft forever
9: eth1@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1
valid_lft forever preferred_lft forever
# 查看docker2 容器使用了哪些命令空间
[root@docker2 ~]# ls /var/run/docker/netns
1-c8669d70d4 349cb1aac200
[root@docker2 ~]# nsenter --net=/var/run/docker/netns/1-c8669d70d4 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 0e:ad:33:09:ad:c6 brd ff:ff:ff:ff:ff:ff
inet 10.70.2.1/24 brd 10.70.2.255 scope global br0
valid_lft forever preferred_lft forever
5: vxlan0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UNKNOWN group default
link/ether 0e:ad:33:09:ad:c6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: veth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UP group default
link/ether 82:4d:95:fe:51:6b brd ff:ff:ff:ff:ff:ff link-netnsid 1
# 查看路由信息
[root@docker2 ~]# nsenter --net=/var/run/docker/netns/1-c8669d70d4 ip route show
10.70.2.0/24 dev br0 proto kernel scope link src 10.70.2.1
# 查看br0设备连接情况
[root@docker2 ~]# nsenter --net=/var/run/docker/netns/1-c8669d70d4 brctl show br0
bridge name bridge id STP enabled interfaces
br0 8000.0ead3309adc6 no vxlan0
veth0
[root@docker2 ~]# nsenter --net=/var/run/docker/netns/349cb1aac200 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 02:42:0a:46:02:70 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.70.2.112/24 brd 10.70.2.255 scope global eth0
valid_lft forever preferred_lft forever
9: eth1@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1
valid_lft forever preferred_lft forever
参考链接:https://cizixs/2017/09/28/linux-vxlan/
参考链接:https://chengqian90/Linux%E5%86%85%E6%A0%B8/Linux%E5%AF%B9vxlan%E7%9A%84%E6%94%AF%E6%8C%81.html
参考链接:https://blog.revolve.team/2017/08/20/deep-dive-3-into-docker-overlay-networks-part-3/
更多推荐
基于Consul实现Docker Vxlan网络浅析
发布评论