CloudStack_usersgroup_20_nakaya_20140801_pub

Apache CloudStack 4.3Virtual Router Deep Drive

Version 2.0

2014/8/1第20回 CloudStackユーザー会 in 大阪

Satoru Nakaya(@giraffeforestg)____http://giraffeforestg.blog.fc2.com/

自己紹介× 中谷悟 / Satoru Nakaya× 岐阜県在住× 大学情報基盤やオープンソースクラウドを担当× ⾃宅SAN友の会× VMware Certified Advanced Professional× CCA for Citrix XenServer

本日はよろしくお願いします。

2

お約束× 本資料の情報を利⽤することによって⽣じるいかなる損害についても責

任を負うものではありません。× 発⾔は個⼈の⾒解であり所属する組織の公式⾒解ではありません。

3

注意(1)今回，発表時間２５分に対してスライド１２4枚となっております。途中で時間切れ可能性大です。または超早口になります。どうぞご了承ください。

※１スライド 10秒程度ならば最後までいける…

4

注意(2)Apache CloudStack 4.4がリリースされましたので先月実施したユーザ会 in 名古屋で発表した「仮想ルータカスタマイズ」の章はばっさりと削除して新機能ネタに差し替えました！

5

6

Virtual RouterVR

仮想ルーターソフトウェアルーター

7

1.概要2.内部構造3.性能4.新機能

8


CloudStack UI/Virtual Router

9

Virtual Router/Virtual Machine

10

Virtual Router/Virtual Machine

11

Hypervisor HypervisorCloudStack

ManagementServer

Storage

Virtual Router

User VM Instance

Guest Network

Public NetworkVirtual Router

Advanced Network

12

User VM Instance

Guest Network

Virtual Router

Basic Network

13

Network Service

14

Network Offering

15

Network Offering

16

Network Offering

17

User VM Instance

Guest Network

Public Network Virtual RouterDHCP / DNS /

Firewall / NAT / Load Balancer …

Network Service

18

External devices as network service providers

19

User VM Instance


NetScaler/F5Load Balancer

JuniperSRXFirewall/NAT

Virtual RouterDHCP/DNS

20


21

System Offering

22

System Offering

23

Virtual RouterScale up

(CPU:500Mhz→2000MHz,Mem:128MB→2048MB,

Net:100Mbps→10000Mbps)

Scale up

24

Virtual RouterHigh Availability

(VRRP)MasterBackup

High Availability

25

VR

VPC(Virtual Private Cloud)

26

Network1(Web)

Network2(AP)

Network3(DB)

外部ネットワーク

他データセンター

Site to Site VPN(IPSEC)

VLAN Routing

Static Route

27


Virtual Router/SSH Login

28

Hypervisor(XenServer) から# ssh -i /root/.ssh/id_rsa.cloud リンクローカルアドレス -p 3922

Linux r-45-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2 x86_64

The programs included with the Debian GNU/Linux system are free software;the exact distribution terms for each program are described in theindividual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extentpermitted by applicable law.Last login: Wed Jan 15 00:27:48 2014 from 10.0.2.2

The programs included with the Debian GNU/Linux system are free software;the exact distribution terms for each program are described in theindividual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extentpermitted by applicable law.root@r-45-VM:~#

29

Network Interface# ip addr show1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host loinet6 ::1/128 scope host

valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 02:00:40:e3:00:02 brd ff:ff:ff:ff:ff:ffinet 10.1.1.1/24 brd 10.1.1.255 scope global eth0inet6 fe80::40ff:fee3:2/64 scope link


link/ether 0e:00:a9:fe:03:18 brd ff:ff:ff:ff:ff:ffinet 169.254.3.24/16 brd 169.254.255.255 scope global eth1inet6 fe80::c00:a9ff:fefe:318/64 scope link


link/ether 06:66:44:00:00:18 brd ff:ff:ff:ff:ff:ffinet 192.168.11.130/24 brd 192.168.11.255 scope global eth2inet6 fe80::466:44ff:fe00:18/64 scope link

valid_lft forever preferred_lft forever

Guest Network

Link Local

Public Network

30

Routing Table# ip route showdefault via 192.168.11.254 dev eth210.1.1.0/24 dev eth0 proto kernel scope link src 10.1.1.1169.254.0.0/16 dev eth1 proto kernel scope link src 169.254.3.24192.168.11.0/24 dev eth2 proto kernel scope link src 192.168.11.130

31

Firewall

32

Firewall# iptables -nL vChain INPUT (policy DROP 443 packets, 29549 bytes)pkts bytes target prot opt in out source destination1880 159K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0

0 0 ACCEPT all -- * * 0.0.0.0/0 224.0.0.180 0 ACCEPT all -- * * 0.0.0.0/0 225.0.0.500 0 ACCEPT all -- eth0 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED

584 68556 ACCEPT all -- eth1 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED692 52648 ACCEPT all -- eth2 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED16 1344 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/01 576 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/00 0 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67

216 14234 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:530 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53

19 1140 ACCEPT tcp -- eth1 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:39220 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:800 0 ACCEPT tcp -- eth0 * 10.1.1.0/24 0.0.0.0/0 state NEW tcp dpt:8080

Chain FORWARD (policy DROP 276 packets, 16560 bytes)pkts bytes target prot opt in out source destination276 16560 NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0

0 0 ACCEPT all -- eth0 eth1 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 state NEW0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED0 0 ACCEPT all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED

276 16560 FW_OUTBOUND all -- eth0 eth2 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT 1338 packets, 176K bytes)pkts bytes target prot opt in out source destination1379 183K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0

Chain FW_OUTBOUND (1 references)pkts bytes target prot opt in out source destination

0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED

Chain NETWORK_STATS (3 references)pkts bytes target prot opt in out source destination276 16560 all -- eth0 eth2 0.0.0.0/0 0.0.0.0/0

0 0 all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0224 35166 tcp -- !eth0 eth2 0.0.0.0/0 0.0.0.0/0224 11648 tcp -- eth2 !eth0 0.0.0.0/0 0.0.0.0/0

33

Firewall

iptables について少しお勉強テーブル

・filterパケットの通過/遮断

・natアドレス変換

・mangleパケットのフィールドを変換(TOS等)

・row

34

User VM Instance

Guest Network

Public Network

SourceNAT

35

SourceNAT

# iptables -nL -v -t natChain PREROUTING (policy ACCEPT 1847 packets, 148K bytes)pkts bytes target prot opt in out source destination

Chain INPUT (policy ACCEPT 400 packets, 26564 bytes)pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 78 packets, 5577 bytes)pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 1 packets, 576 bytes)pkts bytes target prot opt in out source destination77 5001 SNAT all -- * eth2 0.0.0.0/0 0.0.0.0/0 to:192.168.11.130

36

User VM Instance

Guest Network

Public Network

StaticNAT

37

StaticNAT

38

StaticNAT# iptables –nL -v -t natChain PREROUTING (policy ACCEPT 1 packets, 60 bytes)pkts bytes target prot opt in out source destination

0 0 DNAT all -- eth2 * 0.0.0.0/0 192.168.11.131 to:10.1.1.2360 0 DNAT all -- eth0 * 0.0.0.0/0 192.168.11.131 to:10.1.1.236



Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)pkts bytes target prot opt in out source destination

0 0 SNAT all -- * eth2 10.1.1.236 0.0.0.0/0 to:192.168.11.1310 0 SNAT all -- * eth2 0.0.0.0/0 0.0.0.0/0 to:192.168.11.1300 0 SNAT all -- * eth0 10.1.1.0/24 10.1.1.236 to:10.1.1.1

39

User VM Instance

Guest Network

Public Network

Firewall

40

User VM Instance

Guest Network

Public Network

Firewall(Ingress rules)

41

Firewall(Ingress rules)

42

Firewall(Ingress rules) ルール追加前# iptables -nL -v -t mangleChain PREROUTING (policy ACCEPT 164 packets, 19188 bytes)pkts bytes target prot opt in out source destination

87 6751 VPN_192.168.11.130 all -- * * 0.0.0.0/0 192.168.11.1300 0 FIREWALL_192.168.11.130 all -- * * 0.0.0.0/0 192.168.11.130

289 33156 CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED CONNMARK restore


Chain FORWARD (policy ACCEPT 18 packets, 1080 bytes)pkts bytes target prot opt in out source destination



0 0 CHECKSUM udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:68 CHECKSUM fill

Chain FIREWALL_192.168.11.130 (1 references)pkts bytes target prot opt in out source destination

0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0

Chain VPN_192.168.11.130 (1 references)pkts bytes target prot opt in out source destination

87 6751 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0

43

Firewall(Ingress rules) ルール追加後# iptables -nL -v -t mangleChain PREROUTING (policy ACCEPT 12 packets, 856 bytes)pkts bytes target prot opt in out source destination251 17836 VPN_192.168.11.130 all -- * * 0.0.0.0/0 192.168.11.130

0 0 FIREWALL_192.168.11.130 all -- * * 0.0.0.0/0 192.168.11.130402 46776 CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED CONNMARK restore


Chain FORWARD (policy ACCEPT 1 packets, 60 bytes)pkts bytes target prot opt in out source destination



0 0 CHECKSUM udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:68 CHECKSUM fill

Chain FIREWALL_192.168.11.130 (1 references)pkts bytes target prot opt in out source destination

0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED0 0 RETURN tcp -- * * 172.20.0.0/16 0.0.0.0/0 tcp dpt:200000 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0

Chain VPN_192.168.11.130 (1 references)pkts bytes target prot opt in out source destination251 17836 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED

0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 44

User VM Instance

Guest Network

Public Network

Firewall(Egress rules)

45

Firewall(Egress rules) ルール追加前# iptables -nL -v -t filterChain INPUT (policy DROP 443 packets, 29549 bytes)pkts bytes target prot opt in out source destination1880 159K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0








46

Firewall(Egress rules) ルール追加前# iptables -nL -v -t filterChain OUTPUT (policy ACCEPT 1338 packets, 176K bytes)pkts bytes target prot opt in out source destination1379 183K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0


0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED



47

Firewall(Egress rules) ルール追加後# iptables -nL -v -t filterChain INPUT (policy DROP 6 packets, 280 bytes)pkts bytes target prot opt in out source destination1496 113K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0








48

Firewall(Egress rules) ルール追加後# iptables -nL -v -t filterChain OUTPUT (policy ACCEPT 432 packets, 365K bytes)pkts bytes target prot opt in out source destination1587 831K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0

Chain FW_EGRESS_RULES (1 references)pkts bytes target prot opt in out source destination

0 0 ACCEPT tcp -- * * 10.1.1.100 0.0.0.0/0 tcp dpt:11111


0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED5 300 FW_EGRESS_RULES all -- * * 0.0.0.0/0 0.0.0.0/0



49

DNS/DHCPdnsmasq軽量なDNSサーバ(フォワーダ/キャッシュ)とDHCPサーバhttp://www.thekelleys.org.uk/dnsmasq/doc.html

50

DNS/DHCP# grep -v -e '#' -e '^$' /etc/dnsmasq.confdomain-neededbogus-privresolv-file=/etc/dnsmasq-resolv.conflocal=/cs2cloud.internal/interface=eth0except-interface=eth1except-interface=eth2except-interface=lolisten-address=10.1.1.1no-dhcp-interface=eth1no-dhcp-interface=eth2bind-interfacesexpand-hostsdomain=cs2cloud.internaldomain=cs2cloud.internaldomain=cs2cloud.internaldhcp-range=10.1.1.1,staticdhcp-hostsfile=/etc/dhcphosts.txtdhcp-option=15,"cs2cloud.internal"dhcp-option=vendor:MSFT,2,1idhcp-lease-max=2100domain=cs2cloud.internallog-facility=/var/log/dnsmasq.logconf-dir=/etc/dnsmasq.ddhcp-option=option:router,10.1.1.1dhcp-option=6,10.1.1.1,172.16.0.38dhcp-client-updatedhcp-optsfile=/etc/dhcpopts.txt

51

DNS/DHCP# cat /etc/dhcphosts.txt02:00:7b:ed:00:01,set:10_1_1_236,10.1.1.236,vm1,infinite02:00:62:62:00:03,set:10_1_1_162,10.1.1.162,vm2,infinite

# cat /etc/dnsmasq-resolv.confnameserver 172.16.0.38

52

Load balancerHAProxyhttp://haproxy.1wt.eu/

53

Load balancer# cat /etc/haproxy/haproxy.cfgglobal

log 127.0.0.1:3914 local0 infochroot /var/lib/haproxyuser haproxygroup haproxydaemon

defaultslog globalmode tcpoption dontlognullretries 3option redispatchoption forwardforstats enablestats uri /admin?statsstats realm Haproxy¥ Statisticsstats auth admin1:AdMiN123option forceclosetimeout connect 5000timeout client 50000timeout server 50000

listen cloud-default 0.0.0.0:35999option transparent/ 54

Load balancer

55

Load balancer# cat /etc/haproxy/haproxy.cfgglobal

log 127.0.0.1:3914 local0 infochroot /var/lib/haproxyuser haproxygroup haproxydaemon

:

listen cloud-default 0.0.0.0:35999option transparent/

listen 192_168_11_132-80 192.168.11.132:80balance roundrobinserver 192_168_11_132-80_0 10.1.1.236:80 checkserver 192_168_11_132-80_1 10.1.1.162:80 check

56

Correspondence table of network services and applicationsNetwork Services Applications description

Firewall iptables administration tools for packet

filtering and NATSource NAT

Static NAT

Port Forwording

DHCP dnsmasq Small caching DNS proxy and

DHCP/TFTP serverDNS

User Data apache Apache HTTP Server

Load Balancer haproxy fast and reliable load balancing

reverse proxy

VPN xl2tpd layer 2 tunneling protocol

implementation

openswan Internet Key Exchange daemon

Redundant Router conntrackd Connection tracking daemon

keepalived Failover and monitoring daemon

for LVS clusters

57

設定反映スクリプト

CloudStack Database格納情報(ファイアウォールルールや払い出したIP等)を元にSSHで(CloudStack Management Server → XenServer → VR)という経路でスクリプトがキックされ仮想ルータの各種設定を実⾏。

ls /rootbumpup_priority.sh firewallRule_egress.sh monitorServices.pyclearUsageRules.sh firewall_rule.sh reconfigLB.shcreateIpAlias.sh firewall.sh redundant_routerdeleteIpAlias.sh func.sh userdata.pydnsmasq.sh hv-kvp-daemon_3.1_amd64.deb userdata.shedithosts.sh loadbalancer.sh

58

59


60

<注意!>ざっくり計測しているので参考程度にみてね。私の環境では，こうなったというぐらい。

<caution!>I have measured roughly.Performance will vary depending on the environment.

Network Performance

61

HypervisorXenServer 6.2 SP1

CentOS6.464bit

L2スイッチ(1G) Catalyst 3550-12G

CPU:AMD Opteron 3250 HE 2.5GHzMemory:6GBHDD:7200rpm SATANIC:Broadcom BCM57780 1G

CPU:Intel Xeon X5260 3.33GHzMemory:8GBHDD:15000rpm SASNIC:Broadcom BCM5708 1G

Virtual Router System Offerings [CPU:500MHz,Memory:128MB,Nwtork Limit:10Gbps]Network performance measurement tool [nuttcp-6.1.2]

Network Performance(P→P)

62


CentOS6.464bit

# ./nuttcp 192.168.11.2261124.2943 MB / 10.03 sec = 940.0370 Mbps 4 %TX 19 %RX 0 retrans 0.28 msRTT

940 Mbps

Dom0

Network Performance(P→VM)

63


CentOS6.464bit


935 Mbps

CentOS6.464bit

Network Performance(P→VR→VM)

64


CentOS6.464bit

# ./nuttcp 192.168.11.174

CentOS6.464bit

StaticNAT

CloudStack VRSystem OfferingsCPU:500 MHzMemory:128 MBNetwork Limit: 10000Mbps

この部分はユーザ会でのみ公開


65


CentOS6.464bit

# ./nuttcp 192.168.11.175

CentOS6.464bit

Port Transfer




66


CentOS6.464bit

# ./nuttcp 192.168.11.174

CentOS6.464bit

StaticNAT




67


CentOS6.464bit

# ./nuttcp 192.168.11.175

CentOS6.464bit


Port Transfer


Network Performance(VM→VR→P)

68


CentOS6.464bit


940 Mbps

CentOS6.464bit

SouraceNAT


Network Performance(P→VR)

69


CentOS6.464bit


939 Mbps

CentOS6.464bit


Network Performance(VM→VM)

70


CentOS6.464bit

# ./nuttcp 10.1.1.250

CentOS6.464bit


Network Performance(VM→VR→VM)

71


CentOS6.464bit

# ./nuttcp 192.168.11.174

CentOS6.464bit

StaticNAT

SourceNAT


72

Virtual RouterにXenServer Toolsをインストールしてみる

73

結論：効果なし

74

Virtual Routerのカーネルパラメータを変更してみる

75

結論：以下変更では効果なし# sysctl -q net.ipv4.tcp_window_scalingnet.ipv4.tcp_window_scaling = 1# sysctl -q net.ipv4.tcp_syncookiesnet.ipv4.tcp_syncookies = 1# sysctl -q net.core.rmem_maxnet.core.rmem_max = 131071# sysctl -q net.core.wmem_maxnet.core.wmem_max = 131071# sysctl -q net.ipv4.tcp_rmemnet.ipv4.tcp_rmem = 4096 87380 2064032# sysctl -q net.ipv4.tcp_wmemnet.ipv4.tcp_wmem = 4096 16384 2064032# vi /etc/sysctl.conf:

net.core.rmem_max = 16777216net.core.wmem_max = 16777216net.ipv4.tcp_rmem = 4096 87380 16777216net.ipv4.tcp_wmem = 4096 65536 16777216# sysctl -p /etc/sysctl.conf

76

Virtual Router のNICオフロ－ドを無効化してみる

disable the NIC offload of Virtual Router

77

root@r-96-VM:~# ethtool -k eth0

Features for eth0:rx-checksumming: on [fixed]tx-checksumming: on

tx-checksum-ipv4: ontx-checksum-unneeded: off [fixed]tx-checksum-ip-generic: off [fixed]tx-checksum-ipv6: off [fixed]tx-checksum-fcoe-crc: off [fixed]tx-checksum-sctp: off [fixed]

scatter-gather: ontx-scatter-gather: ontx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: ontx-tcp-segmentation: ontx-tcp-ecn-segmentation: off [fixed]tx-tcp6-segmentation: off [fixed]

udp-fragmentation-offload: off [fixed]generic-segmentation-offload: ongeneric-receive-offload: onlarge-receive-offload: off [fixed]

rx-vlan-offload: off [fixed]tx-vlan-offload: off [fixed]ntuple-filters: off [fixed]receive-hashing: off [fixed]highdma: off [fixed]rx-vlan-filter: off [fixed]vlan-challenged: off [fixed]tx-lockless: off [fixed]netns-local: off [fixed]tx-gso-robust: on [fixed]tx-fcoe-segmentation: off [fixed]fcoe-mtu: off [fixed]tx-nocache-copy: onloopback: off [fixed]


78

root@r-96-VM:~# ethtool -K eth0 tx offroot@r-96-VM:~# ethtool -K eth0 gro offroot@r-96-VM:~# ethtool -K eth2 tx offroot@r-96-VM:~# ethtool -K eth2 gro off


79

root@r-96-VM:~# ethtool -k eth0

Features for eth0:rx-checksumming: on [fixed]tx-checksumming: off

tx-checksum-ipv4: offtx-checksum-unneeded: off [fixed]tx-checksum-ip-generic: off [fixed]tx-checksum-ipv6: off [fixed]tx-checksum-fcoe-crc: off [fixed]tx-checksum-sctp: off [fixed]

scatter-gather: offtx-scatter-gather: off [requested on]tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: offtx-tcp-segmentation: off [requested on]tx-tcp-ecn-segmentation: off [fixed]tx-tcp6-segmentation: off [fixed]

udp-fragmentation-offload: off [fixed]generic-segmentation-offload: off [requested on]generic-receive-offload: offlarge-receive-offload: off [fixed]

rx-vlan-offload: off [fixed]tx-vlan-offload: off [fixed]ntuple-filters: off [fixed]receive-hashing: off [fixed]highdma: off [fixed]rx-vlan-filter: off [fixed]vlan-challenged: off [fixed]tx-lockless: off [fixed]netns-local: off [fixed]tx-gso-robust: on [fixed]tx-fcoe-segmentation: off [fixed]fcoe-mtu: off [fixed]tx-nocache-copy: onloopback: off [fixed]


80


CentOS6.464bit

# ./nuttcp 192.168.11.174

CentOS6.464bit


StaticNAT

disable the NIC offload


81

ハードウェアを強化してみるCPU : Intel Xeon X5260 3.33GHz -> AMD FX6300 3.5GHzHDD : SAS 15,000rpm -> Intel SSD 510


82


CentOS6.464bit


934 Mbps

CentOS6.464bit


StaticNAT

Network Performance(VM→VM)

83


CentOS6.464bit

# ./nuttcp 10.1.1.21710880.9491 MB / 10.00 sec = 9125.6862 Mbps 48 %TX 64 %RX 0 retrans 0.79 msRT

9125 Mbps

CentOS6.464bit

84

934 Mbps

9125 Mbps

85

待て! (Wait !)

86

CPUのみ高速にしてHDDを遅くしてみた。

CPU : AMD FX6300 3.5GHzHDD : SATA 7,200rpm


87


CentOS6.464bit

./nuttcp 192.168.11.1751118.8065 MB / 10.06 sec = 933.1727 Mbps 3 %TX 47 %RX 0 retrans 1.51 msRTT

933 Mbps

CentOS6.464bit


StaticNAT

88

933 Mbps

速いCPUだけでもいいね!

Network Performance まとめ

89

・仮想ルーターが動作する物理サーバに高速CPUを搭載するDNAT処理(StaticNATやPortTransfer)のパフォーマンスが良くなる。スペックが⾼い物理サーバを⽤意すれば仮想ルータ(ソフト処理)でも1Gbpsの理論値に近いスループットを出せる。(10Gではどうか？)

・同一サーバの仮想ネットワーク内通信は⾼速。・NICオフロ－ドを無効化すると性能が向上する場合もある。

CloudStackの仮想ルータは優秀!

90


91

Apache CloudStack

おめでとうございます！

Apache Cloudstack 4.4 Design DocumentsBaremetal Advanced Networking SupportCloudstack network-element plugin to orchestrate Juniper's switches (for L2 services)Cloudstack WindowsficationConfiguring load balancing rules for VM nic secondary ipsGPU and vGPU support for CloudStack Guest VMsGranular SCSI Controller support in CloudStack over VMware deploymentsHyper-V support features in 4.4In-memory event busIPv6 in VPC RouterKVM Support For Multiple Template FormatsLXC 2.0

OVS distributed routing and network ACLProposal - Ability to add new guest OS mappingsPVLAN support for CloudStack deployment over Nexus 1000v in VMware environmentRegion level VPC and guest network spanning multiple zonesRoot Resize SupportStorage OverProvisioning as Per Primary BasisSupport OVA files containing multiple disksVirtual Router aggregated command executionVirtual Router Service Failure Alerting

92

93

・OVS→ Open vSwitch (仮想スイッチ)

・distributed routing→ 分散ルーティング

OVS distributed routing

94


95


Open vSwitch

Open vSwitch Open vSwitch

VPC-VR

96

Network1(Web) Network2(DB)

VPC-VR

Open vSwitch

Open vSwitch

HOST-A

HOST-B

通常のルーティング

ルーティングテーブル

97

Network1(Web) Network2(DB)

VPC-VR

Open vSwitch

Open vSwitch

HOST-A

HOST-B

分散ルーティング

フローテーブル

98

さっそくやってみた。

99


・Apache Cloudstack 4.4・HOST :XenServer 6.2 SP1 x 2・Network Type

Advanced NetworkGuest NW Isolation method : GREVPCNW Service Provider : OVS , VPC-VR

100

Guest NW Isolation method : GRE

101

Network Service Provider : OVS

102

VPC Offerings

103

VPC Offerings

104

Network Offerings

105

Network Offerings

106

Global Configuration Parameters

・sdn.ovs.controller.default.label

VPC作成

107

VPC(WEB Network / DB Network)

108


109

WEB SERVERDB SERVER

VPC-VR


110

VPC-VR

10.50.2.5/24

10.50.1.225/24

10.50.2.1/24

10.50.1.1/24

10.50.1.225 → 10.50.2.5


112

VPC-VR

10.50.2.5/24

10.50.1.225/24

10.50.2.1/24

10.50.1.1/24

ここでパケットをキャプチャしてみる

VPC-VRの接続ポート[root@xen02 ~]# ovs-vsctl show

：Bridge "xapi1"

fail_mode: standalonePort "xapi1"

Interface "xapi1"type: internal

Port "vif6.2"Interface "vif6.2"

Port "t164-2-1"Interface "t164-2-1"

type: greoptions: {cloudstack-network-id="f8a37d0b-a3f2-4c32-b27c-6fbfd140e215", key="164",

remote_ip="10.0.0.5"}Port "vif6.3"

Interface "vif6.3"Port "t101-2-1"

Interface "t101-2-1"type: greoptions: {cloudstack-network-id="63a9706a-b878-427f-8b03-30d2c4a824e3", key="101",

remote_ip="10.0.0.5"}ovs_version: "1.4.6"

113

VPC-VRではパケットを処理していない[root@xen02 ~]# tcpdump -n -i vif6.2tcpdump: WARNING: vif6.2: no IPv4 address assignedtcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on vif6.2, link-type EN10MB (Ethernet), capture size 65535 bytes^C0 packets captured0 packets received by filter0 packets dropped by kernel

114

[root@xen02 ~]# tcpdump -n -i vif6.3tcpdump: WARNING: vif6.3: no IPv4 address assignedtcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on vif6.3, link-type EN10MB (Ethernet), capture size 65535 bytes^C0 packets captured0 packets received by filter0 packets dropped by kernel

Open vSwitchのフローテーブル(1)[root@xen01 ~]# ovs-ofctl dump-flows xapi1NXST_FLOW reply (xid=0x4):cookie=0x0, duration=9995.348s, table=0, n_packets=27, n_bytes=3356, priority=1100,in_port=1 actions=resubmit(,1)cookie=0x0, duration=9851.024s, table=0, n_packets=2496, n_bytes=3438281, priority=1100,in_port=3 actions=resubmit(,1)cookie=0x0, duration=9995.66s, table=0, n_packets=166, n_bytes=9972, priority=1200,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,2)cookie=0x0, duration=1073.561s, table=0, n_packets=8, n_bytes=718, priority=1200,ip,in_port=6,nw_dst=10.50.1.0/24 actions=resubmit(,1)cookie=0x0, duration=9851.034s, table=0, n_packets=0, n_bytes=0, priority=1000,ip,in_port=3,nw_dst=224.0.0.0/24 actions=dropcookie=0x0, duration=1073.563s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=4,nw_dst=10.50.2.0/24 actions=resubmit(,1)cookie=0x0, duration=9995.358s, table=0, n_packets=0, n_bytes=0, priority=1000,ip,in_port=1,nw_dst=224.0.0.0/24 actions=dropcookie=0x0, duration=9995.638s, table=0, n_packets=1400, n_bytes=99897, priority=0 actions=resubmit(,1)cookie=0x0, duration=1073.561s, table=0, n_packets=848, n_bytes=83104, priority=1100,ip,in_port=6,dl_dst=02:00:77:fd:00:02,nw_dst=10.50.0.0/16 actions=resubmit(,3)cookie=0x0, duration=1073.562s, table=0, n_packets=848, n_bytes=83104, priority=1100,ip,in_port=4,dl_dst=02:00:60:19:00:02,nw_dst=10.50.0.0/16 actions=resubmit(,3)cookie=0x0, duration=9851.044s, table=0, n_packets=0, n_bytes=0, priority=1000,in_port=3,dl_dst=ff:ff:ff:ff:ff:ff actions=dropcookie=0x0, duration=9995.368s, table=0, n_packets=0, n_bytes=0, priority=1000,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff actions=dropcookie=0x0, duration=9995.647s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,nw_dst=224.0.0.0/24 actions=resubmit(,2)cookie=0x0, duration=1073.562s, table=1, n_packets=864, n_bytes=85494, priority=1100,dl_dst=02:00:62:5d:00:03 actions=output:6cookie=0x0, duration=1073.56s, table=1, n_packets=25, n_bytes=1050, priority=1100,dl_dst=02:00:60:19:00:02 actions=output:3cookie=0x0, duration=1073.563s, table=1, n_packets=873, n_bytes=84154, priority=1100,dl_dst=02:00:55:18:00:01 actions=output:4

115

Open vSwitchのフローテーブル(2)cookie=0x0, duration=1073.561s, table=1, n_packets=10, n_bytes=802, priority=1100,dl_dst=02:00:77:fd:00:02 actions=output:1cookie=0x0, duration=1073.56s, table=1, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,2)cookie=0x0, duration=1073.822s, table=2, n_packets=0, n_bytes=0, priority=1100,in_port=3 actions=output:4,output:6cookie=0x0, duration=1073.822s, table=2, n_packets=0, n_bytes=0, priority=1100,in_port=1 actions=output:4,output:6cookie=0x0, duration=1073.821s, table=2, n_packets=7, n_bytes=1494, priority=1100,in_port=6 actions=output:3,output:1,output:4cookie=0x0, duration=1073.821s, table=2, n_packets=0, n_bytes=0, priority=1100,in_port=4 actions=output:3,output:1,output:6cookie=0x0, duration=1073.821s, table=2, n_packets=0, n_bytes=0, priority=0 actions=dropcookie=0x0, duration=9840.762s, table=3, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,4)cookie=0x0, duration=9840.763s, table=3, n_packets=935, n_bytes=91630, priority=1002,ip,nw_src=10.50.1.0/24 actions=resubmit(,4)cookie=0x0, duration=9840.762s, table=3, n_packets=951, n_bytes=93198, priority=1002,ip,nw_src=10.50.2.0/24 actions=resubmit(,4)cookie=0x0, duration=1073.563s, table=4, n_packets=848, n_bytes=83104, ip,nw_dst=10.50.2.5 actions=mod_dl_src:02:00:60:19:00:02,mod_dl_dst:02:00:55:18:00:01,resubmit(,5)cookie=0x0, duration=1073.562s, table=4, n_packets=848, n_bytes=83104, ip,nw_dst=10.50.1.225 actions=mod_dl_src:02:00:77:fd:00:02,mod_dl_dst:02:00:62:5d:00:03,resubmit(,5)cookie=0x0, duration=1073.561s, table=4, n_packets=0, n_bytes=0, ip,nw_dst=10.50.1.1 actions=mod_dl_src:02:00:77:fd:00:02,mod_dl_dst:02:00:77:fd:00:02,resubmit(,5)cookie=0x0, duration=1073.56s, table=4, n_packets=0, n_bytes=0, ip,nw_dst=10.50.2.1 actions=mod_dl_src:02:00:60:19:00:02,mod_dl_dst:02:00:60:19:00:02,resubmit(,5)cookie=0x0, duration=1073.559s, table=4, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,1)cookie=0x0, duration=9840.761s, table=5, n_packets=0, n_bytes=0, priority=0 actions=dropcookie=0x0, duration=9840.762s, table=5, n_packets=935, n_bytes=91630, priority=1001,ip,nw_dst=10.50.2.0/24 actions=resubmit(,1)cookie=0x0, duration=9840.763s, table=5, n_packets=946, n_bytes=92708, priority=1001,ip,nw_dst=10.50.1.0/24 actions=resubmit(,1)

116


117

VPC-VR

10.50.2.5/24

10.50.1.225/24

10.50.2.1/24

10.50.1.1/24

10.50.1.225 → 10.50.1.1


119

VPC-VR

10.50.2.5/24

10.50.1.225/24

10.50.2.1/24

10.50.1.1/24

ここでパケットをキャプチャしてみる

VPC-VRでパケット到着を確認[root@xen02 ~]# tcpdump -n -i vif6.2tcpdump: WARNING: vif6.2: no IPv4 address assignedtcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on vif6.2, link-type EN10MB (Ethernet), capture size 65535 bytes23:52:38.486712 IP 10.50.1.225 > 10.50.1.1: ICMP echo request, id 46346, seq 152, length 6423:52:38.487214 IP 10.50.1.1 > 10.50.1.225: ICMP echo reply, id 46346, seq 152, length 6423:52:39.486705 IP 10.50.1.225 > 10.50.1.1: ICMP echo request, id 46346, seq 153, length 6423:52:39.487047 IP 10.50.1.1 > 10.50.1.225: ICMP echo reply, id 46346, seq 153, length 6423:52:40.486372 IP 10.50.1.225 > 10.50.1.1: ICMP echo request, id 46346, seq 154, length 6423:52:40.486541 IP 10.50.1.1 > 10.50.1.225: ICMP echo reply, id 46346, seq 154, length 64^C6 packets captured6 packets received by filter0 packets dropped by kernel

120

To closeVirtual Router1.概要

VM , Network Service , External devices2.内部構造

Debian Linux , OSS , Scripts3.性能4.新機能


121

参考文献/Reference

CloudStack Administration DocumentationManaging Networks and Traffichttp://docs.cloudstack.apache.org/projects/cloudstack-administration/en/latest/networking_and_traffic.html

Distributed routing and network ACL with OVS plug-inhttps://cwiki.apache.org/confluence/display/CLOUDSTACK/OVS+distributed+routing+and+network+ACL

CloudStack Advanced Networking With GRE SDN Tunnelshttp://shankerbalan.net/blog/cloudstack-advanced-networking-with-gre-sdn-tunnels/

122

参考文献/Reference

CloudStack仮想ルータの謎に迫る / @MayumiK0http://www.slideshare.net/samemoon/cloud-stackadventcalendar-2012121201-15600230

CloudStackのアーキテクチャ / Kimihiko Kitasehttp://www.slideshare.net/kkitase/cloudstack-architecture-19886203

Virtual Router in CloudStack 4.4 / Sheng Yanghttp://www.youtube.com/watch?v=0lxaYOjvghQhttp://events.linuxfoundation.org/sites/events/files/slides/VR_4_4%20.pdf

123

124

ありがとうございましたThank you so much.

CloudStack_usersgroup_20_nakaya_20140801_pub

Software

Transcript of CloudStack_usersgroup_20_nakaya_20140801_pub