ljzsdut
GitHubToggle Dark/Light/Auto modeToggle Dark/Light/Auto modeToggle Dark/Light/Auto modeBack to homepage
  1. Welcome to the ljzsdut's notes
  2. /
  3. 虚拟化
  4. /
  5. Openstack
  6. /
  7. 02 网络
  8. /
  9. Dvr分析 抓包分析
  10. /
  11. 01 Floating Ip与外网互访抓包分析
Edit page

01 Floating Ip与外网互访抓包分析

zcAL8a-1629254806562

过程简述：

首先要知道，fip是配置在qrouter上的iptables规则，并不存在任一个网卡上。

访问外网：报文从VM(二层转发)发送到网关qrouter上，qrouter进行snat转换成fip，并通过路由表到达fip名称空间中，查询fip中的路由表，发送至floating-gw。

返程：floating-gw上的报文通过arp fip，fg开启arp代理后代为应答，报文到达fip上，通过路由表达到qrouter上，进行snat地址还原后，通过路由表发送至qr口，而后二层转发达到VM.

设备名称：

  • tap设备=tap${VM_PORT_ID:0:11}

  • qrouter名称空间：qrouter-${ROUTER_ID}

  • snat名称空间：snat-${ROUTER_ID}

  • NIC_qrouter_rfp: rfp-${ROUTER_ID:0:11}，所以qrouter下只有一个rfp口，但其上有多个IP。rfp口用于连接fip名称空间。（qr口有多个，每个子网一个qr口）

  • NIC_fip_fpr: fpr-${ROUTER_ID:0:11}，因为每个compute节点上只有1个fip名称空间，用于连接该节点上所有的qrouter，所以fip名称空间下有多个fpr-${ROUTER_ID:0:11}接口，用于连接不同的qrouter。通过fpr口的id就能知道其连接哪个qrouter

  • qrouter里的qr口：qr-${子网默认网关对应port_ID:0:11}，port的device_owner为“network:router_interface_distributed”，该port与子网是1:1。

  • fip名称空间： fip-${floating网络networkID}，id为floating网络的ID，所以不同计算节点上都只有1个fip名称空间，而且其ID是相同的。

  • NIC_fip_fg: fg-${PORT_ID:0:11} port的device_owner为“network:floatingip_agent_gateway”，每个计算节点占用一个floating网的IP。

  • NIC_snat_sg: sg-${PORT_ID:0:11} port的device_owner为“network:router_centralized_snat”，该port与子网是1:1。

  • NIC_snat_qg: qg-${PORT_ID:0:11} port的device_owner为“network:router_gateway”，该port与network/router是1:1或n:1。n:1是指为router指定多个外部网关。

  • qdhcp-${networkID}

路由器上的3类port：

  • router_interface_distributed：分布式路由器的接口，路由器上普通的接口即子网的网关接口
  • router_centralized_snat：路由器上集中snat的接口，为没有fip的虚机提供snat转换，与子网是1:1的关系
  • router_gateway：路由器上连接外网网络的接口，即路由器上的网关接口，一般一个路由器只有1个。（注意区分“路由器网关”与“路由器接口(子网网关)”）
network:router_interface_distributed    ：表示路由器上的普通接口（子网的默认网关所在的接口）
network:dhcp                            ：表示
network:router_centralized_snat         ：表示路由器上连接snat名称空间的port，每个子网占用一个port，其IP配置在snat名称空间的sg口
network:floatingip                      ：floatingip所在的port，其IP在qrouter的iptables上
network:floatingip_agent_gateway        ：fip的节点代理端口，fip名称空间下fg口的ip所在的port，每个compute节点占用一个port
network:router_gateway                  ：路由器上的外部网关port。IP在snat名称空间的qg口

image-20220305225104169

snat里的sg口和qg口通过如下sql获取：

SELECT
	id
FROM
	`neutron`.`ports` 
WHERE
	`device_id` = '3682faea-db63-44ee-8638-4fbc53c8956f'       -- ROUTER_ID
-- 	AND `device_owner` = 'network:router_gateway';           -- NIC_snat_qg  snat名称空间下的qg口
-- 	AND `device_owner` = 'network:router_centralized_snat';  -- NIC_snat_sg

通过FloatingIP访问外网 

#虚拟机内ping (192.168.3.69（） --> 100.121.0.75(sdata))

root@mgt01:~# openstack server list --all --long |grep 192.168.3.69
| ce0692bb-93d5-457b-b0d3-f940d6c70d0d | de0b0af2-0d1d-4e33-b297-ff2352b3f402-master-2            | ACTIVE  | None       | Running     | az2=192.168.3.69, 100.122.0.145; service_mgt=100.123.2.16                                                                                                                  | N/A (booted from volume)                       | N/A (booted from volume)             | cks_4C8G200G_general              | e83dd349-6123-4efd-941b-7d426620288c | az-native-test-2  | compute03 | this='that'

#获取到的信息：
server_id：ce0692bb-93d5-457b-b0d3-f940d6c70d0d
宿主机：compute03
routerid：b6a86b9f-6cf4-4ec3-a57a-00ce3e49142d

1、VM虚机内

ssh 100.123.2.16 -p 6233 -i cks/keyposs

root@de0b0af2-0d1d-4e33-b297-ff2352b3f402-master-2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:a4:58:8d brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.69/16 brd 192.168.255.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fea4:588d/64 scope link
       valid_lft forever preferred_lft forever
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:b1:f2:a5 brd ff:ff:ff:ff:ff:ff
    inet 100.123.2.16/16 brd 100.123.255.255 scope global ens4
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feb1:f2a5/64 scope link
       valid_lft forever preferred_lft forever

#查看路由
root@de0b0af2-0d1d-4e33-b297-ff2352b3f402-master-2:~# ip r
default via 192.168.0.1 dev ens3 proto dhcp src 192.168.3.69 metric 100  ##匹配到默认路由，将数据包通过ens3网卡发往网关192.168.0.1（该网关是qrouter中的qr口）
10.151.32.0/24 via 192.168.3.32 dev ens3 proto bird
10.151.49.0/24 via 192.168.1.226 dev tunl0 proto bird onlink
10.151.51.0/24 via 192.168.2.48 dev tunl0 proto bird onlink
10.151.161.0/24 via 192.168.2.161 dev tunl0 proto bird onlink
10.151.194.0/24 via 192.168.2.50 dev tunl0 proto bird onlink
blackhole 10.151.208.0/24 proto bird
10.151.208.1 dev cali442ca741514 scope link
10.151.208.2 dev cali6aaed11e2bf scope link
10.151.208.3 dev cali72f4f2267aa scope link
10.151.208.4 dev cali049041c9a8a scope link
10.200.8.0/22 via 100.123.255.254 dev ens4 proto dhcp src 100.123.2.16 metric 100
100.123.0.0/16 dev ens4 proto kernel scope link src 100.123.2.16
169.254.169.254 via 192.168.0.1 dev ens3 proto dhcp src 192.168.3.69 metric 100
169.254.169.254 via 100.123.0.3 dev ens4 proto dhcp src 100.123.2.16 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.0.0/16 dev ens3 proto kernel scope link src 192.168.3.69


#在en3网卡上抓包
root@de0b0af2-0d1d-4e33-b297-ff2352b3f402-master-2:~# tcpdump -env -i ens3 icmp
tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
15:32:49.080815 fa:16:3e:a4:58:8d > fa:16:3e:ff:d5:8b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 34720, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.3.69 > 100.121.0.75: ICMP echo request, id 22344, seq 16434, length 64
15:32:49.081310 fa:16:3e:ff:d5:8b > fa:16:3e:a4:58:8d, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 61, id 57245, offset 0, flags [none], proto ICMP (1), length 84)
    100.121.0.75 > 192.168.3.69: ICMP echo reply, id 22344, seq 16434, length 64
15:32:50.104820 fa:16:3e:a4:58:8d > fa:16:3e:ff:d5:8b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 34968, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.3.69 > 100.121.0.75: ICMP echo request, id 22344, seq 16435, length 64
15:32:50.105369 fa:16:3e:ff:d5:8b > fa:16:3e:a4:58:8d, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 61, id 57298, offset 0, flags [none], proto ICMP (1), length 84)
    100.121.0.75 > 192.168.3.69: ICMP echo reply, id 22344, seq 16435, length 64
15:32:51.128825 fa:16:3e:a4:58:8d > fa:16:3e:ff:d5:8b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 35122, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.3.69 > 100.121.0.75: ICMP echo request, id 22344, seq 16436, length 64
15:32:51.129304 fa:16:3e:ff:d5:8b > fa:16:3e:a4:58:8d, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 61, id 57367, offset 0, flags [none], proto ICMP (1), length 84)
    100.121.0.75 > 192.168.3.69: ICMP echo reply, id 22344, seq 16436, length 64

br-int

实现安全组和二层转发。

2、qrouter(192.168.0.1)

查询到routerid为b6a86b9f-6cf4-4ec3-a57a-00ce3e49142d，位于compute03节点上。

image-20210805104742093

root@mgt01:~# ssh compute03
root@compute03:~# ip netns exec qrouter-b6a86b9f-6cf4-4ec3-a57a-00ce3e49142d bash
root@compute03:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: rfp-b6a86b9f-6@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 56:fe:b9:c3:13:1c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.74.216/31 scope global rfp-b6a86b9f-6
       valid_lft forever preferred_lft forever
    inet6 fe80::54fe:b9ff:fec3:131c/64 scope link
       valid_lft forever preferred_lft forever
2030: qr-1f6b0aa3-3d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:ff:d5:8b brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/16 brd 192.168.255.255 scope global qr-1f6b0aa3-3d  #VM的网关IP
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feff:d58b/64 scope link
       valid_lft forever preferred_lft forever

#在qr-1f6b0aa3-3d上抓包
root@compute03:~# tcpdump -env -i qr-1f6b0aa3-3d icmp
tcpdump: listening on qr-1f6b0aa3-3d, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
15:36:45.491466 fa:16:3e:a4:58:8d > fa:16:3e:ff:d5:8b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 150, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.3.69 > 100.121.0.75: ICMP echo request, id 22344, seq 16662, length 64
15:36:45.492604 fa:16:3e:ff:d5:8b > fa:16:3e:a4:58:8d, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 61, id 20170, offset 0, flags [none], proto ICMP (1), length 84)
    100.121.0.75 > 192.168.3.69: ICMP echo reply, id 22344, seq 16662, length 64
15:36:46.492906 fa:16:3e:a4:58:8d > fa:16:3e:ff:d5:8b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 201, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.3.69 > 100.121.0.75: ICMP echo request, id 22344, seq 16663, length 64
15:36:46.493312 fa:16:3e:ff:d5:8b > fa:16:3e:a4:58:8d, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 61, id 20377, offset 0, flags [none], proto ICMP (1), length 84)
    100.121.0.75 > 192.168.3.69: ICMP echo reply, id 22344, seq 16663, length 64

#报文进入网卡(内核)后处理过程：PREROUTING --> 路由表 (--> FORWORD) --> POSTROUTING 

#查看iptables规则中的PREROUTING链（没有匹配到任何规则）
root@compute03:~# iptables -S -t nat
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING  #①
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom  
-A neutron-l3-agent-POSTROUTING -d 192.168.0.0/16 -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697#②，不匹配，继续
-A neutron-l3-agent-PREROUTING -d 100.122.0.145/32 -i rfp-b6a86b9f-6 -j DNAT [unsupported revision] #③，不匹配，此时整个PREROUTING链都不匹配，完成PREROUTING链的规则检查，进入的路由检查（dnat转fip）
-A neutron-l3-agent-float-snat -s 192.168.3.69/32 -j SNAT --to-source 100.122.0.145 --random-fully 
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat 
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat 


#PREROUTING上没有匹配的规则，忽略，直接查看策略路由
root@compute03:~# ip rule
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default
51869:	from 192.168.3.69 lookup 16 #① 备注：此处的策略路由是根据源IP过滤的。所以在qrouter名称空间中，直接访问flaoting网络等外部网络，无法匹配到策略路由，所以无法ping通。
3232235521:	from 192.168.0.1/16 lookup 3232235521   #说明：该条目用于没有fip的虚机转发报文到snat名称空间下（匹配整个子网）

root@compute03:~# ip route show table 16
default via 169.254.74.217 dev rfp-b6a86b9f-6 proto static  #②，源IP为192.168.3.69的报文，从rfp口送出，下一跳(169.254.74.217)为fip名称空间下的frp-b6a86b9f-6口的IP。
#说明：路由后，先做snat，再从rfp口送出

##在经过路由后从rfp口发出前，会先通过POSTROUTING链上的snat规则将源地址换成floatingIP的地址。然后从rfp口发出，发送至fip名称空间下的fpr口
#查看iptables规则中的POSTROUTING链
root@compute03:~# iptables -S -t nat
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom  #①
-A neutron-l3-agent-POSTROUTING -d 192.168.0.0/16 -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 100.122.0.145/32 -i rfp-b6a86b9f-6 -j DNAT [unsupported revision]
-A neutron-l3-agent-float-snat -s 192.168.3.69/32 -j SNAT --to-source 100.122.0.145 --random-fully  #④匹配到此条目，修改源ip为100.122.0.145（floatingIP），然后从rfp口发出。即报文流向：100.122.0.145 --> 100.121.0.75
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat #③
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat #②


#在rfp-b6a86b9f-6上抓包
root@compute03:~# tcpdump -nn -v -i rfp-b6a86b9f-6 icmp
tcpdump: listening on rfp-b6a86b9f-6, link-type EN10MB (Ethernet), capture size 262144 bytes
^C  #此处执行了ctrl+c，才看到后面抓取的报文
09:28:32.050201 IP (tos 0x0, ttl 63, id 45790, offset 0, flags [DF], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.75: ICMP echo request, id 62761, seq 1083, length 64
09:28:32.050538 IP (tos 0x0, ttl 62, id 35972, offset 0, flags [none], proto ICMP (1), length 84)
    100.121.0.75 > 100.122.0.145: ICMP echo reply, id 62761, seq 1083, length 64
09:28:33.056629 IP (tos 0x0, ttl 63, id 45977, offset 0, flags [DF], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.75: ICMP echo request, id 62761, seq 1084, length 64
09:28:33.056940 IP (tos 0x0, ttl 62, id 36108, offset 0, flags [none], proto ICMP (1), length 84)
    100.121.0.75 > 100.122.0.145: ICMP echo reply, id 62761, seq 1084, length 64
09:28:34.080524 IP (tos 0x0, ttl 63, id 46147, offset 0, flags [DF], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.75: ICMP echo request, id 62761, seq 1085, length 64
09:28:34.080948 IP (tos 0x0, ttl 62, id 36116, offset 0, flags [none], proto ICMP (1), length 84)
    100.121.0.75 > 100.122.0.145: ICMP echo reply, id 62761, seq 1085, length 64

3、fip

rfp-b6a86b9f-6和fip下的fpr-b6a86b9f-6是一对veth pair设备，所以发送到rfp-b6a86b9f-6的报文会直接到达fip下的fpr-b6a86b9f-6上。

root@compute03:~# ip netns exec fip-e7cb8424-51e4-429a-8020-76e651877db3 bash
root@compute03:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: fpr-b9b7d98f-6@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ae:72:73:6f:fd:98 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.121.215/31 scope global fpr-b9b7d98f-6
       valid_lft forever preferred_lft forever
    inet6 fe80::ac72:73ff:fe6f:fd98/64 scope link
       valid_lft forever preferred_lft forever
......
省略其他的fpr设备
......
33: fpr-b6a86b9f-6@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 1a:cb:46:80:fd:cd brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet 169.254.74.217/31 scope global fpr-b6a86b9f-6
       valid_lft forever preferred_lft forever
    inet6 fe80::18cb:46ff:fe80:fdcd/64 scope link
       valid_lft forever preferred_lft forever
83: fg-2417e6fb-ee: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:23:e6:31 brd ff:ff:ff:ff:ff:ff
    inet 100.122.0.53/16 brd 100.122.255.255 scope global fg-2417e6fb-ee  #fg口上有floating网的IP
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe23:e631/64 scope link
       valid_lft forever preferred_lft forever

#在fpr-b6a86b9f-6上抓包
root@compute03:~# tcpdump -env -i fpr-b6a86b9f-6 icmp
tcpdump: listening on fpr-b6a86b9f-6, link-type EN10MB (Ethernet), capture size 262144 bytes
^C15:42:46.291965 56:fe:b9:c3:13:1c > 1a:cb:46:80:fd:cd, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 43633, offset 0, flags [DF], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.75: ICMP echo request, id 36215, seq 17015, length 64
15:42:46.292578 1a:cb:46:80:fd:cd > 56:fe:b9:c3:13:1c, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 62, id 64185, offset 0, flags [none], proto ICMP (1), length 84)
    100.121.0.75 > 100.122.0.145: ICMP echo reply, id 36215, seq 17015, length 64
15:42:47.315942 56:fe:b9:c3:13:1c > 1a:cb:46:80:fd:cd, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 43817, offset 0, flags [DF], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.75: ICMP echo request, id 36215, seq 17016, length 64
15:42:47.316264 1a:cb:46:80:fd:cd > 56:fe:b9:c3:13:1c, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 62, id 64301, offset 0, flags [none], proto ICMP (1), length 84)
    100.121.0.75 > 100.122.0.145: ICMP echo reply, id 36215, seq 17016, length 64

#iptables、路由表相关的处理同qrouter
root@compute03:~# iptables -S -t nat
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
#说明：此处的iptables规则没有实质性的报文修改，忽略，直接分析路由规则

#查看策略路由
root@compute03:~# ip rule
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default
2852012365:	from all iif fpr-12223d90-d lookup 2852012365  #iif=income-interface
2852012701:	from all iif fpr-36667042-b lookup 2852012701
2852013163:	from all iif fpr-8050ef95-3 lookup 2852013163
2852014809:	from all iif fpr-b6a86b9f-6 lookup 2852014809  #匹配：此处的策略路由通过匹配报文的流入网卡，即源网卡。
2852015145:	from all iif fpr-351b7ee3-0 lookup 2852015145
2852015607:	from all iif fpr-25a5169e-2 lookup 2852015607
2852017051:	from all iif fpr-0dc7fcdb-8 lookup 2852017051
2852017849:	from all iif fpr-19ec0194-3 [detached] lookup 2852017849
2852018185:	from all iif fpr-9a42e244-e lookup 2852018185
2852018311:	from all iif fpr-5183bf82-c lookup 2852018311
2852021685:	from all iif fpr-e11e7c06-4 lookup 2852021685
2852022021:	from all iif fpr-eb781195-6 [detached] lookup 2852022021
2852024465:	from all iif fpr-3087f7a8-7 lookup 2852024465
2852025111:	from all iif fpr-dc7fe650-3 lookup 2852025111
2852025573:	from all iif fpr-4895b2a0-a lookup 2852025573
2852025909:	from all iif fpr-8890c23c-3 lookup 2852025909
2852026371:	from all iif fpr-6a3ceffb-0 lookup 2852026371
2852026839:	from all iif fpr-b9b7d98f-6 lookup 2852026839
2852028353:	from all iif fpr-27d39dd0-0 lookup 2852028353
root@compute03:~# ip route show table 2852014809
default via 100.122.255.254 dev fg-2417e6fb-ee proto static  #匹配默认路由，从fg口发出，发送到网关100.122.255.254(floating网关)
说明：fip名称空间开启了内核的自动转发功能(ip_forward=1)，所以到达fpr口的报文可以自动转发到fg口上。


#在fg口上抓包
root@compute03:~# tcpdump -nn -v -i fg-2417e6fb-ee icmp
tcpdump: listening on fg-2417e6fb-ee, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
10:24:02.438412 IP (tos 0x0, ttl 62, id 6906, offset 0, flags [DF], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.75: ICMP echo request, id 62761, seq 4363, length 64
10:24:02.438741 IP (tos 0x0, ttl 63, id 64920, offset 0, flags [none], proto ICMP (1), length 84)
    100.121.0.75 > 100.122.0.145: ICMP echo reply, id 62761, seq 4363, length 64

br-int

实现二层转发。

br-ex

实现二层转发。

4、enp175s0f1（物理网卡） 

root@compute03:~# tcpdump -env -i enp175s0f1 icmp
tcpdump: listening on enp175s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes
15:46:03.572239 fa:16:3e:23:e6:31 > 00:1e:08:0e:bc:45, ethertype 802.1Q (0x8100), length 102: vlan 20, p 0, ethertype IPv4, (tos 0x0, ttl 62, id 5027, offset 0, flags [DF], proto ICMP (1), length 84)
    b: ICMP echo request, id 36215, seq 17208, length 64
15:46:03.572559 00:1e:08:0e:bc:45 > fa:16:3e:23:e6:31, ethertype 802.1Q (0x8100), length 102: vlan 20, p 0, ethertype IPv4, (tos 0x0, ttl 63, id 23460, offset 0, flags [none], proto ICMP (1), length 84)
    100.121.0.75 > 100.122.0.145: ICMP echo reply, id 36215, seq 17208, length 64
15:46:04.596202 fa:16:3e:23:e6:31 > 00:1e:08:0e:bc:45, ethertype 802.1Q (0x8100), length 102: vlan 20, p 0, ethertype IPv4, (tos 0x0, ttl 62, id 5066, offset 0, flags [DF], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.75: ICMP echo request, id 36215, seq 17209, length 64
15:46:04.596501 00:1e:08:0e:bc:45 > fa:16:3e:23:e6:31, ethertype 802.1Q (0x8100), length 102: vlan 20, p 0, ethertype IPv4, (tos 0x0, ttl 63, id 23481, offset 0, flags [none], proto ICMP (1), length 84)
    100.121.0.75 > 100.122.0.145: ICMP echo reply, id 36215, seq 17209, length 64

返程包

类似，不再分析。

外部访问FloatingIP

在serviceproxy01上ping FloatingIp（100.121.0.57 sdata–> 100.122.0.145 fip）

1、外部服务器 

root@serviceproxy01:~# ip r
100.122.0.0/16 via 100.121.255.254 dev enp4s0.1299 proto static  #100.121.255.254是sdata网的网关，配置在3层交换机上，在交换机上打通了sdata和floating网络
......

root@serviceproxy01:~# ip a show enp4s0.1299
7: enp4s0.1299@enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:00:00:c7:9f:c5 brd ff:ff:ff:ff:ff:ff
    inet 100.121.0.57/16 brd 100.121.255.255 scope global enp4s0.1299  #service data网的ip
       valid_lft forever preferred_lft forever
    inet6 fe80::ff:fec7:9fc5/64 scope link
       valid_lft forever preferred_lft forever
       
#enp4s0.1299抓包
root@serviceproxy01:~# tcpdump -env -i enp4s0.1299 icmp
tcpdump: listening on enp4s0.1299, link-type EN10MB (Ethernet), capture size 262144 bytes
15:52:14.177235 02:00:00:8c:bf:b6 > 00:1e:08:0e:bc:45, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 29906, offset 0, flags [DF], proto ICMP (1), length 84)
    100.121.0.57 > 100.122.0.145: ICMP echo request, id 62038, seq 33, length 64
15:52:14.177765 00:1e:08:0e:bc:45 > 02:00:00:8c:bf:b6, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 61, id 6489, offset 0, flags [none], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.57: ICMP echo reply, id 62038, seq 33, length 64
15:52:15.177002 02:00:00:8c:bf:b6 > 00:1e:08:0e:bc:45, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 30044, offset 0, flags [DF], proto ICMP (1), length 84)
    100.121.0.57 > 100.122.0.145: ICMP echo request, id 62038, seq 34, length 64
15:52:15.177429 00:1e:08:0e:bc:45 > 02:00:00:8c:bf:b6, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 61, id 6721, offset 0, flags [none], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.57: ICMP echo reply, id 62038, seq 34, length 64

br-ex

实现二层转发。

br-int

实现二层转发。

2、proxy_arp

外网中的机器首先要通过 ARP 获取虚机浮动 IP（100.122.0.145）对应的 MAC 地址。浮动 IP 并没有配置在 fip 的端口上，因此 fip 无法直接响应 ARP 请求，那怎么办呢？

DVR模式下，fip不存在任何设备上，只存在于qrouter中的iptables规则中。

Neutron 在fip命名空间的fg口设置了arp proxy，这样，fg口既可以响应它自己的 interface 上的 IP 地址的 ARP 请求，也可以响应能通过它路由到的 IP 地址的 ARP 请求（这里的它指代：fip名称空间下的所有网卡，即fip名称空间下路由表中可以到达的IP都可以被arp代理响应）。当fg口收到fip的arp请求后会代答arp，直接返回fg口的mac地址

# fip netns上配置的proxy_arp
root@compute03:~# ip netns exec fip-e7cb8424-51e4-429a-8020-76e651877db3 sysctl net.ipv4.conf.fg-2417e6fb-ee.proxy_arp
net.ipv4.conf.fg-2417e6fb-ee.proxy_arp = 1

外网中的机器获取到虚机浮动 IP 的 MAC 地址后，发出 ICMP 网络包到达fg口。

arp代答返回哪些fip请求的应答呢?

fg口收到arp请求后，会查看是否存在到请求目标 (100.122.0.145)的路由条目，如果存在则返回本接口的mac地址。

root@compute03:~# ip netns exec fip-e7cb8424-51e4-429a-8020-76e651877db3 ip route 100.122.0.0/16 dev fg-2417e6fb-ee proto kernel scope link src 100.122.0.53 100.122.0.145 via 169.254.74.216 dev fpr-b6a86b9f-6 proto static #此条目匹配，fg口响应100.122.0.145的arp请求

可以通过ip r get 100.122.0.145来测试arp代理是否生效。

验证：

我们可以在其他compute节点的fip名称空间中进行arp验证：

root@compute03:~# ip net exec fip-e7cb8424-51e4-429a-8020-76e651877db3 ip a
....
83: fg-2417e6fb-ee: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:23:e6:31 brd ff:ff:ff:ff:ff:ff
    inet 100.122.0.53/16 brd 100.122.255.255 scope global fg-2417e6fb-ee
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe23:e631/64 scope link
       valid_lft forever preferred_lft forever
       
#我们在compute04的fip名称空间上查看：
root@compute04:~# ip net exec fip-e7cb8424-51e4-429a-8020-76e651877db3 arp -a |grep 100.122.0.145   #100.122.0.145为fip
? (100.122.0.145) at fa:16:3e:23:e6:31 [ether] on fg-2417e6fb-ee

3、fip

通过fg的mac地址，报文送至fg-2417e6fb-ee口。

root@compute03:~# ip netns exec fip-e7cb8424-51e4-429a-8020-76e651877db3 bash
root@compute03:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: fpr-b9b7d98f-6@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ae:72:73:6f:fd:98 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.121.215/31 scope global fpr-b9b7d98f-6
       valid_lft forever preferred_lft forever
    inet6 fe80::ac72:73ff:fe6f:fd98/64 scope link
       valid_lft forever preferred_lft forever
......
省略其他的fpr设备
......
33: fpr-b6a86b9f-6@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 1a:cb:46:80:fd:cd brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet 169.254.74.217/31 scope global fpr-b6a86b9f-6  #该网卡
       valid_lft forever preferred_lft forever
    inet6 fe80::18cb:46ff:fe80:fdcd/64 scope link
       valid_lft forever preferred_lft forever
83: fg-2417e6fb-ee: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:23:e6:31 brd ff:ff:ff:ff:ff:ff
    inet 100.122.0.53/16 brd 100.122.255.255 scope global fg-2417e6fb-ee
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe23:e631/64 scope link
       valid_lft forever preferred_lft forever
       
#在fg-2417e6fb-ee抓包
root@compute03:~# tcpdump -env -i fg-2417e6fb-ee icmp
tcpdump: listening on fg-2417e6fb-ee, link-type EN10MB (Ethernet), capture size 262144 bytes
^C15:59:22.793199 00:1e:08:0e:bc:45 > fa:16:3e:23:e6:31, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 19654, offset 0, flags [DF], proto ICMP (1), length 84)
    100.121.0.57 > 100.122.0.145: ICMP echo request, id 62038, seq 458, length 64
15:59:22.793485 fa:16:3e:23:e6:31 > 00:1e:08:0e:bc:45, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 62, id 60545, offset 0, flags [none], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.57: ICMP echo reply, id 62038, seq 458, length 64
15:59:23.793447 00:1e:08:0e:bc:45 > fa:16:3e:23:e6:31, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 19767, offset 0, flags [DF], proto ICMP (1), length 84)
    100.121.0.57 > 100.122.0.145: ICMP echo request, id 62038, seq 459, length 64
15:59:23.793795 fa:16:3e:23:e6:31 > 00:1e:08:0e:bc:45, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 62, id 60622, offset 0, flags [none], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.57: ICMP echo reply, id 62038, seq 459, length 64

#流量到达fg口后，通过路由表得知下一跳通过fpr口送往169.254.74.216     
root@compute03:~# ip r
100.122.0.0/16 dev fg-2417e6fb-ee proto kernel scope link src 100.122.0.53
100.122.0.145 via 169.254.74.216 dev fpr-b6a86b9f-6 proto static  #匹配此条目（普通路由匹配），从fpr-b6a86b9f-6口发出，发往169.254.74.216
100.122.0.178 via 169.254.118.52 dev fpr-8890c23c-3 proto static
100.122.4.9 via 169.254.118.52 dev fpr-8890c23c-3 proto static
100.122.4.207 via 169.254.66.156 dev fpr-36667042-b proto static
100.122.5.70 via 169.254.112.144 dev fpr-3087f7a8-7 proto static
169.254.65.76/31 dev fpr-12223d90-d proto kernel scope link src 169.254.65.77
169.254.66.156/31 dev fpr-36667042-b proto kernel scope link src 169.254.66.157
169.254.68.106/31 dev fpr-8050ef95-3 proto kernel scope link src 169.254.68.107
169.254.74.216/31 dev fpr-b6a86b9f-6 proto kernel scope link src 169.254.74.217 #veth pair设备的路由（二层的路由表）
169.254.76.40/31 dev fpr-351b7ee3-0 proto kernel scope link src 169.254.76.41
169.254.77.246/31 dev fpr-25a5169e-2 proto kernel scope link src 169.254.77.247
169.254.83.154/31 dev fpr-0dc7fcdb-8 proto kernel scope link src 169.254.83.155
169.254.88.8/31 dev fpr-9a42e244-e proto kernel scope link src 169.254.88.9
169.254.88.134/31 dev fpr-5183bf82-c proto kernel scope link src 169.254.88.135
169.254.101.180/31 dev fpr-e11e7c06-4 proto kernel scope link src 169.254.101.181
169.254.112.144/31 dev fpr-3087f7a8-7 proto kernel scope link src 169.254.112.145
169.254.115.22/31 dev fpr-dc7fe650-3 proto kernel scope link src 169.254.115.23
169.254.116.228/31 dev fpr-4895b2a0-a proto kernel scope link src 169.254.116.229
169.254.118.52/31 dev fpr-8890c23c-3 proto kernel scope link src 169.254.118.53
169.254.120.2/31 dev fpr-6a3ceffb-0 proto kernel scope link src 169.254.120.3
169.254.121.214/31 dev fpr-b9b7d98f-6 proto kernel scope link src 169.254.121.215  
169.254.127.192/31 dev fpr-27d39dd0-0 proto kernel scope link src 169.254.127.193



#在fpr-b6a86b9f-6抓包
root@compute03:~# tcpdump -nn -v -i fpr-b6a86b9f-6 icmp
tcpdump: listening on fpr-b6a86b9f-6, link-type EN10MB (Ethernet), capture size 262144 bytes
^C16:00:25.445270 IP (tos 0x0, ttl 62, id 60674, offset 0, flags [DF], proto ICMP (1), length 84)
    100.121.0.57 > 100.122.0.145: ICMP echo request, id 35921, seq 367, length 64
16:00:25.445623 IP (tos 0x0, ttl 63, id 19655, offset 0, flags [none], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.57: ICMP echo reply, id 35921, seq 367, length 64
16:00:26.454383 IP (tos 0x0, ttl 62, id 60702, offset 0, flags [DF], proto ICMP (1), length 84)
    100.121.0.57 > 100.122.0.145: ICMP echo request, id 35921, seq 368, length 64
16:00:26.454804 IP (tos 0x0, ttl 63, id 19729, offset 0, flags [none], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.57: ICMP echo reply, id 35921, seq 368, length 64
16:00:27.455550 IP (tos 0x0, ttl 62, id 60916, offset 0, flags [DF], proto ICMP (1), length 84)
    100.121.0.57 > 100.122.0.145: ICMP echo request, id 35921, seq 369, length 64
16:00:27.455926 IP (tos 0x0, ttl 63, id 19916, offset 0, flags [none], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.57: ICMP echo reply, id 35921, seq 369, length 64

4、qrouter

fpr-b6a86b9f-6和qrouter下的rfp-b6a86b9f-6是一对veth pair设备，所以发送到fpr-b6a86b9f-6的报文会直接到达qrouter下的rfp-b6a86b9f-6上。

qrouter里的rfp口收到请求后会先做dnat转换(PREROUTING)，将目标地址换成192.168.3.69。然后根据路由从qr口发出送往主机。

root@compute03:~# ip netns exec qrouter-b6a86b9f-6cf4-4ec3-a57a-00ce3e49142d bash
       
root@compute03:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: rfp-b6a86b9f-6@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 56:fe:b9:c3:13:1c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.74.216/31 scope global rfp-b6a86b9f-6
       valid_lft forever preferred_lft forever
    inet6 fe80::54fe:b9ff:fec3:131c/64 scope link
       valid_lft forever preferred_lft forever
2030: qr-1f6b0aa3-3d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:ff:d5:8b brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/16 brd 192.168.255.255 scope global qr-1f6b0aa3-3d
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feff:d58b/64 scope link
       valid_lft forever preferred_lft forever


root@compute03:~# tcpdump -env -i rfp-b6a86b9f-6 icmp
tcpdump: listening on rfp-b6a86b9f-6, link-type EN10MB (Ethernet), capture size 262144 bytes
^C16:03:02.721277 1a:cb:46:80:fd:cd > 56:fe:b9:c3:13:1c, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 62, id 47840, offset 0, flags [DF], proto ICMP (1), length 84)
    100.121.0.57 > 100.122.0.145: ICMP echo request, id 62038, seq 676, length 64
16:03:02.721554 56:fe:b9:c3:13:1c > 1a:cb:46:80:fd:cd, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 22541, offset 0, flags [none], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.57: ICMP echo reply, id 62038, seq 676, length 64
16:03:03.745299 1a:cb:46:80:fd:cd > 56:fe:b9:c3:13:1c, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 62, id 47999, offset 0, flags [DF], proto ICMP (1), length 84)
    100.121.0.57 > 100.122.0.145: ICMP echo request, id 62038, seq 677, length 64
16:03:03.745543 56:fe:b9:c3:13:1c > 1a:cb:46:80:fd:cd, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 22586, offset 0, flags [none], proto ICMP (1), length 84)
    100.122.0.145 > 100.121.0.57: ICMP echo reply, id 62038, seq 677, length 64
       
#到达rfp口的报文，先要经过PREROUTING链   
root@compute03:~# iptables -S -t nat
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING  #①
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-POSTROUTING -d 192.168.0.0/16 -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 100.122.0.145/32 -i rfp-b6a86b9f-6 -j DNAT [unsupported revision] #②，进行dnat，进行目标地址转换：100.122.0.145替换为192.168.3.69
-A neutron-l3-agent-float-snat -s 192.168.3.69/32 -j SNAT --to-source 100.122.0.145 --random-fully
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat


 #进行完DNAT后，再进行路由
 root@compute03:~# ip r
169.254.74.216/31 dev rfp-b6a86b9f-6 proto kernel scope link src 169.254.74.216
192.168.0.0/16 dev qr-1f6b0aa3-3d proto kernel scope link src 192.168.0.1  #匹配此条目（二层路由），从qr-1f6b0aa3-3d发出，发往192.168.3.69


root@compute03:~# tcpdump -nn -v -i qr-1f6b0aa3-3d icmp
tcpdump: listening on qr-1f6b0aa3-3d, link-type EN10MB (Ethernet), capture size 262144 bytes
^C16:12:39.346400 IP (tos 0x0, ttl 61, id 21801, offset 0, flags [DF], proto ICMP (1), length 84)
    100.121.0.57 > 192.168.3.69: ICMP echo request, id 35921, seq 1095, length 64
16:12:39.346537 IP (tos 0x0, ttl 64, id 42372, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.3.69 > 100.121.0.57: ICMP echo reply, id 35921, seq 1095, length 64
16:12:40.347265 IP (tos 0x0, ttl 61, id 21973, offset 0, flags [DF], proto ICMP (1), length 84)
    100.121.0.57 > 192.168.3.69: ICMP echo request, id 35921, seq 1096, length 64
16:12:40.347513 IP (tos 0x0, ttl 64, id 42422, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.3.69 > 100.121.0.57: ICMP echo reply, id 35921, seq 1096, length 64
16:12:41.362365 IP (tos 0x0, ttl 61, id 21993, offset 0, flags [DF], proto ICMP (1), length 84)
    100.121.0.57 > 192.168.3.69: ICMP echo request, id 35921, seq 1097, length 64
16:12:41.362479 IP (tos 0x0, ttl 64, id 42588, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.3.69 > 100.121.0.57: ICMP echo reply, id 35921, seq 1097, length 64

br-int

实现二层转发。实现安全组策略。

5、VM内部 

root@de0b0af2-0d1d-4e33-b297-ff2352b3f402-master-2:~# tcpdump -env -i ens3 icmp
tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
16:08:25.197778 fa:16:3e:ff:d5:8b > fa:16:3e:a4:58:8d, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 61, id 23499, offset 0, flags [DF], proto ICMP (1), length 84)
    100.121.0.57 > 192.168.3.69: ICMP echo request, id 62038, seq 999, length 64
16:08:25.197865 fa:16:3e:a4:58:8d > fa:16:3e:ff:d5:8b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 62059, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.3.69 > 100.121.0.57: ICMP echo reply, id 62038, seq 999, length 64
16:08:25.564161 fa:16:3e:30:a4:a9 > fa:16:3e:a4:58:8d, ethertype IPv4 (0x0800), length 43: (tos 0x0, ttl 64, id 51264, offset 0, flags [DF], proto ICMP (1), length 29)
    192.168.3.32 > 192.168.3.69: ICMP echo request, id 59373, seq 0, length 9
16:08:25.564253 fa:16:3e:a4:58:8d > fa:16:3e:30:a4:a9, ethertype IPv4 (0x0800), length 43: (tos 0x0, ttl 64, id 27771, offset 0, flags [none], proto ICMP (1), length 29)
    192.168.3.69 > 192.168.3.32: ICMP echo reply, id 59373, seq 0, length 9

返程包

类似，不再分析。

官方文档：

https://docs.openstack.org/neutron/latest/admin/deploy-ovs.html

https://docs.openstack.org/neutron/latest/admin/deploy-ovs-ha-dvr.html#deploy-ovs-ha-dvr