tungsten fabirc vgw报文流向分析:1.浮动ip 2.非浮动ip
环境: ctrl---cmp1---cmp2
vhost(10.x; eth2(public):20.x)
创建vgw:
docker exec -it vrouter_vrouter-agent_1 python /opt/contrail/utils/provision_vgw_interface.py --oper create --interface vgw1 --subnets 192.168.20.128/25 --routes 0.0.0.0/0 --vrf default-domain:admin:public:public
虚机关联浮动ip(1.1.11.5在cmp1,浮动ip:192.168.20.131)
从外部ping虚机浮动ip,在源主机看arp条目如下：
192.168.20.131 ether 52:54:00:f0:de:da C eth2=>该网卡在cmp1(eth2)
即报文会发到eth2（host OS 网卡），查看host路由，目的为同一网段:
192.168.20.0/24 dev eth2 proto kernel scope link src 192.168.20.2
并且已经有arp条目:
192.168.20.131 ether 00:00:5e:00:01:00 C vgw1
送到vgw1:
#vif --list|grep vgw1 -C 3
TX packets:343 bytes:14742 errors:0
Drops:0
vif0/27 OS: vgw1
Type:Gateway HWaddr:00:00:5e:00:01:00 IPaddr:0.0.0.0
Vrf:6 Mcast Vrf:65535 Flags:L3L2Er QOS:-1 Ref:2
RX packets:15 bytes:1238 errors:7
#rt --dump 6 |grep 131
192.168.20.131/32 32 P - 38 2:cf:55:fa:d7:ad(9716)
#nh --get 38
Id:38 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:7
Flags:Valid, Policy, Etree Root,
EncapFmly:0806 Oif:26 Len:14
Encap Data: 02 cf 55 fa d7 ad 00 00 5e 00 01 00 08 00
#vif --get 26
Vrouter Interface Table
Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
Proxy=MAC Requests Proxied Always, Er=Etree Root, Mn=Mirror without Vlan Tag, Ig=Igmp Trap Enabled
vif0/26 OS: tapcf55fad7-ad
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:1.1.11.5
送到主机完成。
vgw和虚机不在同一节点的情况:
IPaddr:211.0.0.6 浮动IP:192.168.20.132
从ping的源主机看：
arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.20.132 ether 52:54:00:f0:de:da C eth2
arp代理 由cmp1所在的eth2(public网卡)完成:
eth2:
link/ether 52:54:00:f0:de:da
即报文会发到eth2（host OS 网卡），查看host路由，目的为同一网段:
192.168.20.0/24 dev eth2 proto kernel scope link src 192.168.20.2
并且已经有arp条目:
192.168.20.132 ether 00:00:5e:00:01:00 C vgw1
直接发到vgw1
#vif --list|grep vgw1 -C 3
TX packets:405 bytes:17346 errors:0
Drops:0
vif0/27 OS: vgw1
Type:Gateway HWaddr:00:00:5e:00:01:00 IPaddr:0.0.0.0
Vrf:6 Mcast Vrf:65535 Flags:L3L2Er QOS:-1 Ref:2
RX packets:20 bytes:1616 errors:7
#rt --dump 6 |grep 132
192.168.20.132/32 32 LP 60 21 2:fb:36:7c:87:53(74884)
#nh --get 21
Id:21 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:20 Vrf:0
Flags:Valid, MPLSoUDP, Etree Root,
Oif:0 Len:14 Data:52 54 00 65 b5 1e 52 54 00 54 da 66 08 00
Sip:192.168.10.2 Dip:192.168.10.3
发往cmp2:(tcpdump vhost0用的网卡)（从上表可以看到,label为60，报文分析如下:）
21:36:00.603484 IP 192.168.10.2.57317 > cmpt2.6635: UDP, length 88
0x0000: 5254 0065 b51e 5254 0054 da66(源mac/目的mac) 0800 4500
0x0010: 0074 b735 0000 4011 2dee {c0a8 0a02}(192.168.10.2为隧道源地址) c0a8
0x0020: 0a03{目的地址) dfe5 19eb(源端口目的端口) 0060(报文长度) 0000(checksum) {一下开始mpls:}0003 c13e(前20位为mpls label,此处为60) 4500
0x0030: 0054(报文长) b735(标识) 4000 3e(TTL)01(协议ICMP) db94(checksum) {源ip/目的ip:c0a8 140a c0a8
0x0040: 1484} 0800 bc72 41a8 0061 002a 2d5d 0000
0x0050: 0000 042a 0900 0000 0000 1011 1213 1415
0x0060: 1617 1819 1a1b 1c1d 1e1f 2021 2223 2425
0x0070: 2627 2829 2a2b 2c2d 2e2f 3031 3233 3435
0x0080: 3637
#从报文可知送过来封装label是60: mpls --get 60
MPLS Input Label Map
Label NextHop
-------------------
60 75
#nh --get 75
Id:75 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:10
Flags:Valid, Policy, Etree Root,
EncapFmly:0806 Oif:30 Len:14
Encap Data: 02 fb 36 7c 87 53 00 00 5e 00 01 00 08 00
#vif --get 30
Vrouter Interface Table
vif0/30 OS: tapfb367c87-53
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:211.0.0.6
非浮动ip的情况:给211.0.0.0/24建立vgw在cmp1:
docker exec -it vrouter_vrouter-agent_1 python /opt/contrail/utils/provision_vgw_interface.py --oper create --interface vgw11 --subnets 211.0.0.0/24 --routes 0.0.0.0/0 --vrf default-domain:admin:left-VN-fw:left-VN-fw
在非计算节点添加到211.0.0.0的路由:
route add -net 211.0.0.7 netmask 255.255.255.255 gw 192.168.20.2
(route add -net 211.0.0.0 netmask 255.255.255.0 gw 192.168.20.2)
从cmp1开始分析:
报文从eth2上收到,走OS协议栈:
211.0.0.0/24 dev vgw11 scope link
送给vgw11:
#vif --list |grep vgw11 -C 3
vif0/28 OS: vgw11
Type:Gateway HWaddr:00:00:5e:00:01:00 IPaddr:0.0.0.0
Vrf:16 Mcast Vrf:65535 Flags:L3L2Er QOS:-1 Ref:2
#(vrouter-agent)[root@cmpt1 /]$ rt --dump 16|grep 211.0.0.
211.0.0.7/32 32 LP 50 21 -
#nh --get 21
Id:21 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:20 Vrf:0
Flags:Valid, MPLSoUDP, Etree Root,
Oif:0 Len:14 Data:52 54 00 65 b5 1e 52 54 00 54 da 66 08 00
Sip:192.168.10.2 Dip:192.168.10.3
发送cmp2，label为50
#nh --get 55
Id:55 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:4 Vrf:10
Flags:Valid, Policy, Etree Root,
EncapFmly:0806 Oif:33 Len:14
Encap Data: 02 48 e1 d6 da 86 00 00 5e 00 01 00 08 00
#vif --get 33
Vrouter Interface Table
vif0/33 OS: tap48e1d6da-86
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:211.0.0.7
如果虚机和vgw在同一节点:(211.0.0.8在cmp1节点)
211.0.0.7/32 32 LP 50 21 2:48:e1:d6:da:86(232268)
211.0.0.8/32 32 P - 91 2:f0:34:dd:34:3(167608)
#nh --get 91
Id:91 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:4 Vrf:16
Flags:Valid, Policy, Etree Root,
EncapFmly:0806 Oif:22 Len:14
Encap Data: 02 f0 34 dd 34 03 00 00 5e 00 01 00 08 00
(vrouter-agent)[root@cmpt1 /]$ vif --get 22
Vrouter Interface Table
vif0/22 OS: tapf034dd34-03
Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:211.0.0.8
直接送到虚机。