Post on 21-Jan-2018
KUBERNETES INTEGRATEWITH CORDHUNG-WEI CHIU
LINKER NETWORKS.INC
WHO AM I
• Hung-Wei Chiu (邱宏瑋)
• hwchiu@linkernetworks.com
• hwchiu.com
• Experience
• Software Engineer at Linker Netowrks
• Software Engineer at Synology (2014~2017)
• Co-Found of SDNDS-TW
• Open Source experience
• SDN related projects (mininet, ONOS, Floodlight, awesome-sdn)
OUTLINE
• What is CORD
• Challenge of kubernetes with CORD.
• What have we done now
• Next steps
What is CORD ?
CENTRAL OFFICE RE-ARCHITECTED AS A DATACENTER
5
SDN + NFV + Cloud
Open Source Software
Commodity Hardware
(Servers, White-Box Switches, I/O Blades)
Large number of
COs
Evolved over 40-50 years
300+ Types of
equipmentHuge source
of CAPEX/OPEX
DATA CENTER LEAF-SPINE FABRIC
6
Reliability 😃 Flexibility 😃 Bandwidth 😃
Spine
Leaf
Me
troR
ou
ter
White Box White Box
White Box
White Box
White Box White Box White Box White Box
White Box White Box White Box
White Box
White Box
White Box
Open SourceSDN-basedBare-metal
White Box
White Box
R,E,M-Access
7
ONOS Controller Cluster
vRouterControl
XOS (Orchestrator)
vSG
VNF
VNF VNFVNFvSG VNF
VNF
VNFVNF
vSG VNF
VNF VNF
VNF
OVS OVS OVS OVS OVS
Residential Mobile Enterprise
Underlay Control
Underlay
OverlayControl
Overlay
vOLTControl
Final CORD Architecture
Me
troR
ou
ter
White Box White Box
White Box
White Box
White Box White Box White Box White Box
White Box White Box White Box
White Box
White Box
White Box
Open SourceSDN-basedBare-metal
White Box
White Box
R,E,M-Access
8
ONOS Controller Cluster
vRouterControl
XOS (Orchestrator)
vSG
VNF
VNF VNFVNFvSG VNF
VNF
VNFVNF
vSG VNF
VNF VNF
VNF
OVS OVS OVS OVS OVS
Residential Mobile Enterprise
Underlay Control
Underlay
OverlayControl
Overlay
vOLTControl
Final CORD Architecture
Me
troR
ou
ter
White Box White Box
White Box
White Box
White Box White Box White Box White Box
White Box White Box White Box
White Box
White Box
White Box
Open SourceSDN-basedBare-metal
White Box
White Box
R,E,M-Access
9
ONOS Controller Cluster
vRouterControl
XOS (Orchestrator)
vSG
VNF
VNF VNFVNFvSG VNF
VNF
VNFVNF
vSG VNF
VNF VNF
VNF
OVS OVS OVS OVS OVS
Residential Mobile Enterprise
Underlay Control
Underlay
OverlayControl
Overlay
vOLTControl
Final CORD Architecture
SUMMARY
• VM-based NFV (Network Function Virtualization)
• Use ONOS (SDN controller) + Open VSwitch to control packets.
• Use XOS (Service Orchestration) to control all services (VNF)
• ONOS/XOS need to communicate with OpenStack component.
Why kubernetes CORD ?
WHO?
• Ciena
• Hate OpenStack
• Intel China
• Some Telecom service providers
• China unicom
CHANGE TO KUBERNETES
• VM-based NFV (Network Function Virtualization)
• vSG, vPGW, vSGW, etc
• Who owns the NFV?
• Vendors
• We can’t force them to convert all NFVs to container.
• It’s impossible to have a kubernetes solution for CORD now.
CHANGE TO KUBERNETES
• How about hybrid architecture?
• Support OpenStack + kubernetes
CHANGE TO KUBERNETES
• Use ONOS (SDN controller) + Open vSwitch to control packets.
• There’re so many CNI for kubernetes now.
• None of them are Open vSwitch based solution.
• Linen-CNI is Open vSwitch + Linux Bridge solution
• Same subnet traffics between POD are controlled by Bridge
NODE
POD POD
OVS
br0
ens0p3
CHENLAGE TO KUBERNETES
• Use ONOS (SDN controller) + Open vSwitch to control packets.
• Create our own CNI to support pure Open vSwitch version.
• Still some problem need to conquer future. NODE
POD POD
OVS
ens0p3
?
CHANGE TO KUBERNETES
NODE
POD POD
Bridge
ens0p3
many iptable rules.
many iptable rules.
CHANGE TO KUBERNETES
• Use XOS (Service Orchestration) to control all services (VNF)
• Kubernetes can handle most thing.
• XOS should communicate with Kubernetes via its API service.
CHANGE TO KUBERNETES
• ONOS/XOS need to communicate with OpenStack component.
• ONOS need to know the IP information of each host (VMs) from neutron component.
• In kubernetes, we need to provide the IP information of each POD.
• Since we implement our CNI, we can send the IP information after CNI assigns IP to POD.
• Send the information via Restful API/gRPC
PROBLEMS WE MET
• Deploy ONOS controller as container
• Multiple network interface for POD
• Centralized IP management
DEPLOY PROBLEM
• We need to deploy ONOS as container
• The chicken-and-egg conundrum!
• Hard to solve, we need to work-around now.
• We decide to move the ONOS out of POD.
• For each node, it should have multiple network interface including data network and control
network.
• Out of band.
DEPLOY PROBLEM
NODE
POD POD
OVS
ens0p3
ens0p4
NODE
POD POD
OVS
ens0p3
ens0p4
NODE
POD POD
OVS
ens0p3
ens0p4
Data network
Control network
DEPLOY PROBLEM
NODE
POD POD
OVS
ens0p3
ens0p4
NODE
POD POD
OVS
ens0p3
ens0p4
NODE
POD POD
OVS
ens0p3
ens0p4
Data network
Control network
MULTIPLE NETWORK INTERFACE
• For some NFV,(vSG) they need to multiple interface in its POD.
• For example.
NODE
POD POD
OVS
ens0p3
MULTIPLE NETWORK INTERFACE
• We found a open source project (multus-CNI)
• Provides the multi interface support in a pod
• We don’t figure out how to use it first.
• We thought it’s a global setting.
• We try to implement by ourself.
MULTIPLE NETWORK INTERFACE
• Multiple network interface means call CNI multiple times.
• For CNI, we need to know the namespace location for each POD.
• We want to provide a interface to dynamically call CNI for any existing POD.
• Input
• Pod Name, Network configuration (CNI name, configuration)
• Output
• Success (add another interface for existing POD)
• Error
MULTIPLE NETWORK INTERFACE
• For each CNI, it should know the namespace location of each POD.
• We also need to know the POD name
• In the CNI plugin, we can get above information via
• Args.Args (many information, separate by semi-colon)
• Args.Netns
• We store those information in ETCD.
MULTIPLE NETWORK INTERFACE
• After I have finished almost tasks.
• One intel guy tell us multus-cni support pod configuration.
• …..…..
• Ok, we use multus-cni.
CENTRALIZED IP MANAGEMENT
• Our CNI plugin use IPAM to handle the IP management.
• Official IPAM support two types.
• Host-local
• DHCP
IPAM DHCP
• Requirement
• You should run a IPAM DHCP daemon on each node.
• You should setup a DHCP server on your network.
• How it works.
• Start a DHCP client when a POD is be created.
• Forward the DHCP packet to DHCP server(it depends on your CNI forward L2 broadcast)
• Official recommend to use MacVLan as CNI.
• Limitation
• All node in same subnet.
• Simple configuration. You only specify “type=dhcp” in CNI configuration.
IPAM HOST-LOCAL
• Requirement
• None
• How it works
• Use a local file to record how many IP address has been used.
• Lookup the file and choose an available IP address for CNI.
• Limitation.
• You should prepare a configuration for each Node with different setting.
• Complex configuration. You need to specify which subnet the Node will use.
• Make sure no duplicate for each Node.
IPAM
• What we want ?
• Simple configuration
• Support multiple subnet
• We decided to create new IPAM
DHCP
• Refer to trellis (CORD network infrastructure)• Requirement
• DHCP server
• How it works• We set the Gateway Address for each dhcp request to support multiple subnet via L3 unicast.• It looks like
DHCP
NODE 1
POD POD
OVS
ens0p3
POD POD
OVS
ens0p3NODE 2
network
Master DHCPServer
DHCP relay DHCP relay192.168.1.1 192.168.2.1
Subnet 192.168.1.0/24 {
}Subnet 192.168.2.0/24 {
}
DHCP – PROBLEM
• The Problem is..• How to decide the IP address of each Open vSwitch?
• Gateway address of each subnet.
• Use ETCD• Maybe we can use ETCD to replace the DHCP
ETCD
• Implement a new IPAM which use etcd to record the subnet of each node.• Simple configuration
• Network: 10.12.0.0/16• Subnet length: 24• etcd address.
• The subnet will from 10.12.1.0/24 to 10.12.255.0/24• Simple and easy to implement.
NEXT STEP
• Integrate the ONOS (SDN controller) with our CNI.
• Make sure the ONOS can control the network.
• For the kubernetes internal communication, the ONOS should implement all of them by
Openflow, not iptables.NODE
POD POD
OVS
ens0p3
Use openflow rules
many iptable rules.
NEXT STEP
Fix Bugs
Q&A