Post on 12-May-2015
description
MidoNet: Overlay-based Virtual Networking for IaaS Clouds
March 21, 2013
Adam Johnson General Manager, Midokura
@adjohn
Copyright ©2012 Midokura All rights reserved
Requirements
2
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
3
Isolated tenant network (virtual
data center)
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
4
L3 isolation (similar to VPC and VRF)
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
5
Isolated L2 networks
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
6
Redundant, optimized and fault-tolerant
paths to the Internet (e.g. via BGP)
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
7
Fault-tolerant devices and links
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
8
NAT, LB, and Filtering
NAT, LB, and Firewalls
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
9
L3 (and L2) VPNs
Copyright ©2012 Midokura All rights reserved
Tenant/Project A
Network A1
VM1 VM3
Network A2
VM5
Tenant/Project B
Network B1
VM2 VM4
uplink
Provider Virtual Router (L3)
Tenant AVirtual Router
Tenant BVirtual Router
VM6
Virtual L2 Switch B1
Virtual L2 Switch A1
Virtual L2 Switch A2
TenantB office
Tenant BVPN Router
Office Network
Requirements
10
Minimize ARP broadcasts by exploiting CMS config
RESTful API for CMS integration and direct
tenant access
Solid integration with leading open CMS: ���
OpenStack, CloudStack DHCP, DNS and other
services
Copyright ©2012 Midokura All rights reserved
Requirements: recap
11
l Multi-tenancy l Scalable, fault-tolerant
devices (or device-agnostic network services).
l L2 isolation l L3 routing isolation
u VPC u Like VRF (virtual
routing and fwd-ing) l BGP gateway l Scalable control plane
u ARP, DHCP, ICMP l Floating IP
l Stateful NAT
u Port masquerading u DNAT
l ACLs l Stateful (L4) Firewalls
u Security Groups l LB health checks l VPNs at L2 and L3
u IPSec l REST API l Integration with CMS
u OpenStack u CloudStack
Copyright ©2012 Midokura All rights reserved
How to build it?
12
1. Virtualized physical devices
2. Centrally controlled OpenFlow-based hop-by-hop switching fabric
3. Edge to edge overlays
Copyright ©2012 Midokura All rights reserved
Virtualized physical devices
13
l 4096 limit on number of unique tags l Large spanning trees terminating on many hosts l High churn in switch control planes due to MAC learning
l Each VM is separate virtual MAC! l Need MLAG for L2 multi-path (vendor specific)
1
VLAN VLAN1
VLAN2
Copyright ©2012 Midokura All rights reserved
Virtualized physical devices
14
l L2 isolation l What about L3 and Internet access? l Use VRF or virtual appliances?
1
VLAN (more) VLAN1
VLAN2
Copyright ©2012 Midokura All rights reserved
Virtualized physical devices
15
1
l Not scalable to cloud scale l Expensive hardware l Not fault tolerant (HSRP?) l L2 and L3 isolation. What about NAT, LB, FW?
出典:http://infrastructureadventures.com/tag/vrf-lite/
VRF
Core VLAN 10 VLAN11 VLAN12
Product VLAN 20 VLAN21 VLAN22
Sales VLAN 99
VRF VRF VRF
Copyright ©2012 Midokura All rights reserved
OpenFlow hop-by-hop switch fabric
16
2
OpenFlow Switches
OpenFlow Controller (Cluster)
l Fabric extends to the compute host software switch? • State in each switch is proportional to the virtual network state • Need to update all switches in path when provisioning new virtual devices or updating them. • Not scalable, slow and non-atomic switch updates.
Copyright ©2012 Midokura All rights reserved
OpenFlow hop-by-hop switch fabric (more)
17
2
OpenFlow Switches
OpenFlow Controller (Cluster)
l Flow rules for VM flows (microflows)? l Flow rules for virtual device simulation?
Copyright ©2012 Midokura All rights reserved
Edge-to-Edge Overlays
18
3
VM
VM Edge
Edge Edge
Edge Edge
Edge
Use scalable IGP (iBGP, OSPF) to build multi-path underlay
Copyright ©2012 Midokura All rights reserved
Edge-to-Edge Overlays
19
3
VM
VM Edge
Edge Edge
Edge Edge
Edge
IP encapsulation provides isolation
Copyright ©2012 Midokura All rights reserved
Edge-to-Edge Overlays
20
3
VM
VM Edge
Edge Edge
Edge Edge
Edge
Virtual network processing at ingress host, decoupled from
physical network
Copyright ©2012 Midokura All rights reserved
Edge-to-Edge Overlays
21
3
VM
VM Edge
Edge Edge
Edge Edge
Edge
Virtual network changes don't affect
underlay state
Copyright ©2012 Midokura All rights reserved
Edge-to-Edge Overlays
22
3
• Packet processing on x86 CPUs (at edge) • Intel DPDK facilitates packet processing • Number of cores in servers increasing fast
• Clos Networks (for underlay) • Spine and Leaf architecture with IP • Economical and high E-W bandwidth
• Merchant silicon (cheap IP switches) • Broadcom, Intel (Fulcrum Micro), Marvell • ODMs (Quanta, Accton) starting to sell directly • Switches are becoming just like Linux servers
Copyright ©2012 Midokura All rights reserved
Overlays are the right approach!
But not sufficient... We still need a scalable control plane.
23
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
24
Logical Topology
vPort
Provider Virtual Router
Tenant A Virtual Router
Tenant B Virtual Router
Virtual Switch A1
Virtual Switch A2
Virtual Switch B1
vPort
vPort
vPort
vPort
vPort
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
25
Logical Topology
Private IP Network
MN
MN
MN
Internet
BGP Multi
Homing
Physical Topology
MN VM
VM
MN VM
VM
MN VM
VM
BGP To ISP3
BGP To ISP2
BGP To ISP1
vPort
Provider Virtual Router
Tenant A Virtual Router
Tenant B Virtual Router
Virtual Switch A1
Virtual Switch A2
Virtual Switch B1
vPort
vPort
vPort
vPort
vPort
Network State Database
MN MN MN
Tunnel
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
26
Distributed State
MidoNet REST API
Dashboard
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
27
Distributed State
Linux Kernel + OVS KMOD
VM1 MidoNet
Ctrl
HW
Linux Kernel + OVS KMOD
VM2 MidoNet
Ctrl
HW
Host A Host B
Lazy state propagation
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
28
Distributed State
Linux Kernel + OVS KMOD
VM1 MidoNet
Ctrl
HW
Linux Kernel + OVS KMOD
VM2 MidoNet
Ctrl
HW
Host A Host B
VM sends first packet; table miss; NetLink
upcall to MidoNet
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
29
Distributed State
Linux Kernel + OVS KMOD
VM1 MidoNet
Ctrl
HW
Linux Kernel + OVS KMOD
VM2 MidoNet
Ctrl
HW
Host A Host B
MidoNet agent locally processes packet (virtual layer simulation); installs
local flow (drop/mod/fwd)
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
30
Distributed State
Linux Kernel + OVS KMOD
VM1 MidoNet
Ctrl
HW
Linux Kernel + OVS KMOD
VM2 MidoNet
Ctrl
HW
Host A Host B
Packet tunneled to peer host; decap; kflow table miss; Netlink notifies
peer MidoNet agent
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
31
Distributed State
Linux Kernel + OVS KMOD
VM1 MidoNet
Ctrl
HW
Linux Kernel + OVS KMOD
VM2 MidoNet
Ctrl
HW
Host A Host B
MN agent maps tun-key to kernel datapath port#; installs fwd flow
rule
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
32
Distributed State
Linux Kernel + OVS KMOD
VM1 MidoNet
Ctrl
HW
Linux Kernel + OVS KMOD
VM2 MidoNet
Ctrl
HW
Host A Host B
Subsequent packets matched by flow rules
at both ingress and egress hosts
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
33
• Distributed and scalable control plane Ø Handle all control packets at local MidoNet agent
adjacent to VM • Scalable and fault tolerant central database
Ø Stores virtual network configuration Ø Dynamic network state
² MAC learning, ARP cache, etc Ø Cached at edges on demand
• All packet modifications at ingress Ø One virtual hop
² No travel through middle boxes Ø Drop at ingress
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
34
• Scalable edge gateway interface to external networks • Multihomed BGP to ISP • REST API and GUI • Integration with popular open source cloud stacks • OpenStack • Removes SPOF of network node • Scalable and fault tolerant NAT for floating IP • Implements security groups efficiently
• CloudStack
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
35
Deep OpenStack Integration
• Quantum Plugin • L2 isolation, of course • Also… • L3 isolation (without VM / appliance) • Security groups (stateful firewall) • Floating IP (NAT) • Load balancing (L4)
Copyright ©2012 Midokura All rights reserved
HorizonWeb GUI
Quantum Plugin
MidoNet Manager (Web GUI)
Nova API
MidoNet API
RabbitMQ - Passing Queue
Compute Host
NovaCompute
Libvirt driver
Datapath
Inst
an
ce
A1
Inst
an
ce
B1
MidoNetAgent
Quantum
API
MidoNet Plugin
MidoNet Distributed
State
MidoNet Distributed
State
MidoNet Distributed
State
MidoNet EdgeAgent
MidoNet EdgeAgent
MidoNet EdgeAgent
INTERNET
MidoNetVIF Driver
KeystoneAuthentication
OpenStack Integration
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
37
Future Directions
• Scalable L7 virtual appliances • Content aware load balancer • MPLS VPN termination Ø Interconnect with carrier backbones • multiple data center federation Ø Virtual L2 between sites • LISP Ø Global IP mobility between sites
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
38
Conclusions
• IaaS clouds require new networking model • Edge to edge overlays are the right
approach • Servers are good enough at packet
processing Ø Can use them for edge gateways • Multipath IP network fabric is cheap and
easy to build
Questions?
info@midokura.com
We’re hiring http://midokura.com/careers/
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
40
Backup Slides
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
41
MN
Packet Encapsulated
Tunnel
Drop/Block
Packet from VM, VPN, or external BGP peer
enters kernel datapath
Copyright ©2012 Midokura All rights reserved
MidoNet SDN Solution
42
MN
Packet Encapsulated
Tunnel
Drop/Block
One flow rule reflecting the outcome of the virtual layer
simulation AND the mapping of egress vport to
peer host decides to drop or fwd
Copyright ©2012 Midokura All rights reserved
Spine and Leaf Network Architecture
43
L3 Switch Spine
Leaf
L3 Switch L3 Switch L3 Switch
L3 Switch L3 Switch L3 Switch x32
x4
48x10G
1536 x 10G
4x40G
e.g Force10 Z9000
IBGP and ECMP
e.g Arista 7050T
e.g Force10 Z9000