當搭建完openstack之后,在創建instance之前,第一件事情就是創建network,一個經典的流程如下:
TENANT_NAME="openstack"
TENANT_NETWORK_NAME="openstack-net"
TENANT_SUBNET_NAME="${TENANT_NETWORK_NAME}-subnet"
TENANT_ROUTER_NAME="openstack-router"
FIXED_RANGE="NEUTRON_FIXED_RANGE"
NETWORK_GATEWAY="NEUTRON_NETWORK_GATEWAY"
PUBLIC_GATEWAY="NEUTRON_PUBLIC_GATEWAY"
PUBLIC_RANGE="NEUTRON_PUBLIC_RANGE"
PUBLIC_START="NEUTRON_PUBLIC_START"
PUBLIC_END="NEUTRON_PUBLIC_END"
(1) 創建private network和subnet
TENANT_ID=$(keystone tenant-list | grep " $TENANT_NAME " | awk '{print $2}')
TENANT_NET_ID=$(neutron net-create --tenant_id $TENANT_ID $TENANT_NETWORK_NAME --provider:network_type gre --provider:segmentation_id 1 | grep " id " | awk '{print $4}')
TENANT_SUBNET_ID=$(neutron subnet-create --tenant_id $TENANT_ID --ip_version 4 --name $TENANT_SUBNET_NAME $TENANT_NET_ID $FIXED_RANGE --gateway $NETWORK_GATEWAY --dns_nameservers list=true 8.8.8.8 | grep " id " | awk '{print $4}')
當僅有private network的時候,會對這個private network創建一個DHCP Server
所以DHCP Agent會執行下面的命令:
ip netns exec qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0 ip -o link show tap452bdfab-31
這個命令試圖從dhcp的namespace里面查找dhcp的網卡,但是很可惜找不到,返回error
Cannot open network namespace "qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0": No such file or directory
于是試圖創建dhcp server的網卡,這個網卡會attach到br-int上,所以先查看br-int
ip -o link show br-int
如果br-int沒有問題,于是創建dhcp server的網卡,并且attach到br-int上
ovs-vsctl -- --if-exists del-port tap452bdfab-31 -- add-port br-int tap452bdfab-31 -- set Interface tap452bdfab-31 type=internal -- set Interface tap452bdfab-31 external-ids:iface-id=452bdfab-3152-44d0-bd9c-40c94a6f8640 -- set Interface tap452bdfab-31 external-ids:iface-status=active -- set Interface tap452bdfab-31 external-ids:attached-mac=fa:16:3e:d7:08:67
為網卡設置mac
ip link set tap452bdfab-31 address fa:16:3e:d7:08:67
查看當前存在的namespace
ip -o netns list
返回
qrouter-26a45e0e-a58a-443b-a972-d62c0c5a1323
qdhcp-760d2c5e-4938-49b0-bffe-c77c5b141d18
發現沒有這個dhcp所對應的namespace,需要創建一個
ip netns add qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0
將io網卡設置為up
ip netns exec qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0 ip link set lo up
將新建的dhcp server的網卡放在這個namespace里面
ip link set tap452bdfab-31 netns qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0
將DHCP server的網卡設置為up
ip netns exec qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0 ip link set tap452bdfab-31 up
查看這個網卡的ip地址
ip netns exec qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0 ip addr show tap452bdfab-31 permanent scope global
為這個網卡配置ip地址
ip netns exec qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0 ip -4 addr add 192.168.10.3/24 brd 192.168.10.255 scope global dev tap452bdfab-31
ip netns exec qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0 ip -4 addr add 169.254.169.254/16 brd 169.254.255.255 scope global dev tap452bdfab-31
第一個地址是dhcp server的地址,第二個地址是metadata server的地址
查看路由表
ip netns exec qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0 ip route list dev tap452bdfab-31
169.254.0.0/16? proto kernel? scope link? src 169.254.169.254
192.168.10.0/24? proto kernel? scope link? src 192.168.10.3
添加路由表
ip netns exec qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0 ip route replace default via 192.168.10.1 dev tap452bdfab-31
查看網卡的配置
ip netns exec qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0 ip addr show tap452bdfab-31
232: tap452bdfab-31: mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether fa:16:3e:d7:08:67 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.3/24 brd 192.168.10.255 scope global tap452bdfab-31
valid_lft forever preferred_lft forever
inet 169.254.169.254/16 brd 169.254.255.255 scope global tap452bdfab-31
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fed7:867/64 scope link tentative
valid_lft forever preferred_lft forever
啟動dhcp server
ip netns exec qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0 env NEUTRON_NETWORK_ID=66b9930b-2871-414c-8c6f-991a6a8cffe0 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap452bdfab-31 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/66b9930b-2871-414c-8c6f-991a6a8cffe0/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/66b9930b-2871-414c-8c6f-991a6a8cffe0/host --addn-hosts=/var/lib/neutron/dhcp/66b9930b-2871-414c-8c6f-991a6a8cffe0/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/66b9930b-2871-414c-8c6f-991a6a8cffe0/opts --leasefile-ro --dhcp-range=set:tag0,192.168.10.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal
啟動metadata proxy
ip netns exec qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0 neutron-ns-metadata-proxy --pid_file=/var/lib/neutron/external/pids/66b9930b-2871-414c-8c6f-991a6a8cffe0.pid --metadata_proxy_socket=/var/lib/neutron/metadata_proxy --network_id=66b9930b-2871-414c-8c6f-991a6a8cffe0 --state_path=/var/lib/neutron --metadata_port=80 --debug --verbose --log-file=neutron-ns-metadata-proxy-66b9930b-2871-414c-8c6f-991a6a8cffe0.log --log-dir=/var/log/neutron
最后查看一下網卡配置
ip netns exec qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0 ip addr show tap452bdfab-31
kill -HUP 17666
這個PID是什么呢?
# ps aux | grep 17666
nobody?? 17666? 0.0? 0.0? 28204? 1112 ???????? S??? Jul14?? 0:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap452bdfab-31 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/66b9930b-2871-414c-8c6f-991a6a8cffe0/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/66b9930b-2871-414c-8c6f-991a6a8cffe0/host --addn-hosts=/var/lib/neutron/dhcp/66b9930b-2871-414c-8c6f-991a6a8cffe0/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/66b9930b-2871-414c-8c6f-991a6a8cffe0/opts --leasefile-ro --dhcp-range=set:tag0,192.168.10.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal
原來是我們的dhcp server
這個命令的作用是:如果想要更改配置而不需停止并重新啟動服務,請使用該命令。在對配置文件作必要的更改后,發出該命令以動態更新服務配置。
最后查看一下路由配置
ip netns exec qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0 ip route list dev tap452bdfab-31
(2) 創建一個router,并且和private network相連
ROUTER_ID=$(neutron router-create --tenant_id $TENANT_ID $TENANT_ROUTER_NAME | grep " id " | awk '{print $4}')
neutron router-interface-add $ROUTER_ID $TENANT_SUBNET_ID
查看br-ex
ip -o link show br-ex
59: br-ex: mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether a0:48:1c:ab:df:b5 brd ff:ff:ff:ff:ff:ff
查看所有的namespace
ip -o netns list
qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0
qrouter-26a45e0e-a58a-443b-a972-d62c0c5a1323
qdhcp-760d2c5e-4938-49b0-bffe-c77c5b141d18
發現沒有這個router的namespace,創建一個
ip netns add qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82
將io網卡設為up
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 ip link set lo up
這是一個router,所以enable ip forward
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 sysctl -w net.ipv4.ip_forward=1
初始化iptables
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 iptables-save –c
# Generated by iptables-save v1.4.21 on Thu Jul 17 01:37:57 2014
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Thu Jul 17 01:37:57 2014
# Generated by iptables-save v1.4.21 on Thu Jul 17 01:37:57 2014
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Thu Jul 17 01:37:57 2014
# Generated by iptables-save v1.4.21 on Thu Jul 17 01:37:57 2014
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Thu Jul 17 01:37:57 2014
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 iptables-restore –c
啟動metadata proxy
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 neutron-ns-metadata-proxy --pid_file=/var/lib/neutron/external/pids/d62d417d-2005-46d7-a83b-b1e5c0a36d82.pid --metadata_proxy_socket=/var/lib/neutron/metadata_proxy --router_id=d62d417d-2005-46d7-a83b-b1e5c0a36d82 --state_path=/var/lib/neutron --metadata_port=9697 --debug --verbose --log-file=neutron-ns-metadata-proxy-d62d417d-2005-46d7-a83b-b1e5c0a36d82.log --log-dir=/var/log/neutron
查看router的網卡
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 ip -o link show qr-29003a09-e7
但是網卡不存在
Device "qr-29003a09-e7" does not exist.
查看br-int,router的網卡會attach到這個網卡上
ip -o link show br-int
58: br-int: mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether 0a:9b:c6:54:ef:46 brd ff:ff:ff:ff:ff:ff
創建router的網卡,并且attach到br-int
ovs-vsctl -- --if-exists del-port qr-29003a09-e7 -- add-port br-int qr-29003a09-e7 -- set Interface qr-29003a09-e7 type=internal -
- set Interface qr-29003a09-e7 external-ids:iface-id=29003a09-e787-49dd-b5f4-11ad107159c7 -- set Interface qr-29003a09-e7 external-ids:iface-status=active -- set Interface qr-29003a09-e7 external-ids:attached-mac=fa:16:3e:84:6e:cc
設置router網卡的mac
ip link set qr-29003a09-e7 address fa:16:3e:84:6e:cc
查看所有的namespace
ip -o netns list
qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82
qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0
qrouter-26a45e0e-a58a-443b-a972-d62c0c5a1323
qdhcp-760d2c5e-4938-49b0-bffe-c77c5b141d18
有這個router的namespace
將這個網卡放在這個namespace里面
ip link set qr-29003a09-e7 netns qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82
將router的網卡設為up
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 ip link set qr-29003a09-e7 up
查看網卡的地址
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 ip addr show qr-29003a09-e7 permanent scope global
設置網卡的地址
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 ip -4 addr add 192.168.10.1/24 brd 192.168.10.255 scope global dev qr-2
9003a09-e7
查看所有的網卡
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 ip -o -d link list
1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0
241: qr-29003a09-e7: mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether fa:16:3e:84:6e:cc brd ff:ff:ff:ff:ff:ff promiscuity 1
(3) 創建外網,并且連接到router
neutron net-create public --router:external=True
neutron subnet-create --ip_version 4 --gateway $PUBLIC_GATEWAY public $PUBLIC_RANGE --allocation-pool start=$PUBLIC_START,end=$PUBLIC_END --disable-dhcp --name public-subnet
neutron router-gateway-set ${TENANT_ROUTER_NAME} public
查看br-ex
ip -o link show br-ex
59: br-ex: mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether a0:48:1c:ab:df:b5 brd ff:ff:ff:ff:ff:ff
列出所有的網卡
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 ip -o -d link list
1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0
241: qr-29003a09-e7: mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether fa:16:3e:84:6e:cc brd ff:ff:ff:ff:ff:ff promiscuity 1
查看qg網卡
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 ip -o link show qg-556ca938-e1
但是網卡不存在
Device "qg-556ca938-e1" does not exist.
查看br-ex
ip -o link show br-ex
創建新的網卡qg,attach到br-ex
ovs-vsctl -- --if-exists del-port qg-556ca938-e1 -- add-port br-ex qg-556ca938-e1 -- set Interface qg-556ca938-e1 type=internal -- set Interface qg-556ca938-e1 external-ids:iface-id=556ca938-e11b-4246-bdc1-ef25c91b7593 -- set Interface qg-556ca938-e1 external-ids:iface-status=active -- set Interface qg-556ca938-e1 external-ids:attached-mac=fa:16:3e:68:12:c0
設置網卡mac
ip link set qg-556ca938-e1 address fa:16:3e:68:12:c0
查看所有的namespace
ip -o netns list
qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82
qdhcp-66b9930b-2871-414c-8c6f-991a6a8cffe0
qrouter-26a45e0e-a58a-443b-a972-d62c0c5a1323
qdhcp-760d2c5e-4938-49b0-bffe-c77c5b141d18
將qg網卡設置到namespace中
ip link set qg-556ca938-e1 netns qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82
將網卡設置為up
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 ip link set qg-556ca938-e1 up
查看網卡地址
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 ip addr show qg-556ca938-e1 permanent scope global
設置網卡地址
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 ip -4 addr add 16.158.165.105/22 brd 16.158.167.255 scope global dev qg
-556ca938-e1
添加router表
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 route add default gw 16.158.164.1
設置iptables
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 iptables-save –c
# Generated by iptables-save v1.4.21 on Thu Jul 17 01:58:30 2014
*nat
:PREROUTING ACCEPT [4:425]
:INPUT ACCEPT [1:229]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
[4:425] -A PREROUTING -j neutron-l3-agent-PREROUTING
[0:0] -A OUTPUT -j neutron-l3-agent-OUTPUT
[0:0] -A POSTROUTING -j neutron-l3-agent-POSTROUTING
[0:0] -A POSTROUTING -j neutron-postrouting-bottom
[0:0] -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
[0:0] -A neutron-l3-agent-snat -jneutron-l3-agent-float-snat
[0:0] -A neutron-postrouting-bottom -j neutron-l3-agent-snat
COMMIT
# Completed on Thu Jul 17 01:58:30 2014
# Generated by iptables-save v1.4.21 on Thu Jul 17 01:58:30 2014
*mangle
:PREROUTING ACCEPT [4:425]
:INPUT ACCEPT [1:229]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Thu Jul 17 01:58:30 2014
# Generated by iptables-save v1.4.21 on Thu Jul 17 01:58:30 2014
*filter
:INPUT ACCEPT [1:229]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
[1:229] -A INPUT -j neutron-l3-agent-INPUT
[0:0] -A FORWARD -j neutron-filter-top
[0:0] -A FORWARD -j neutron-l3-agent-FORWARD
[0:0] -A OUTPUT -j neutron-filter-top
[0:0] -A OUTPUT -j neutron-l3-agent-OUTPUT
[0:0] -A neutron-filter-top -j neutron-l3-agent-local
[0:0] -A neutron-l3-agent-INPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 9697 -j ACCEPT
COMMIT
# Completed on Thu Jul 17 01:58:30 2014
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 iptables-restore –c
顯示網卡信息
ip netns exec qrouter-d62d417d-2005-46d7-a83b-b1e5c0a36d82 ip addr show qg-556ca938-e1
242: qg-556ca938-e1: mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether fa:16:3e:68:12:c0 brd ff:ff:ff:ff:ff:ff
inet 16.158.165.105/22 brd 16.158.167.255 scope global qg-556ca938-e1
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe68:12c0/64 scope link tentative
valid_lft forever preferred_lft forever