OSDC 2014: Christopher Kunz - Software defined networking in an open-source compute cloud
description
Transcript of OSDC 2014: Christopher Kunz - Software defined networking in an open-source compute cloud
Heartbleed
...and why yours should, too
You are in the right session
_ This is an emergency service announcement _ Due to events that transpired on Tuesday _ I thought it‘d be good to have some info
10.04.14 OSDC 2014 2
About me
_ Dr. Christopher Kunz _ Studied CompSci in Hannover, PhD in 2012 _ Works as a hoster for 15 years
_ Some admin experience
_ Used to do a lot of PHP _ Author, „PHP-‐Sicherheit“, ed. 1-‐3
_ And don‘t get me started about swords!
10.04.14 OSDC 2014 3
About filoo
_ hQps://www.filoo.de _ Quickly-‐growing hosVng company _ Data center in Frankfurt, Germany _ Developed own IaaS middleware
_ QEMU/KVM, OVS, Ceph
_ Offer hosVng, co-‐locaVon, cloud services _ 100% subsidiary of Thomas-‐Krenn.AG _ Visit their booth!
10.04.14 OSDC 2014 4
Heartbleed in a nutshell
_ A bug with a cute name _ ...and not so cute effects _ Pre-‐auth, pre-‐logging universal TLS/SSL bug _ Introduced in OpenSSL 1.0.1a (2012) _ Allows to make 64kb memory dumps of the server‘s memory
10.04.14 OSDC 2014 5
Wait. What?
_ Yes, remote memory dumps _ Due to an unchecked buffer length, a TLS enabled server may dump memory contents to the client _ Limit of 64k per reply _ MulVple replies possible _ Memdump may contain...
_ URLs and GET / POST variables _ Random excerpts from whatever _ Source code of scripts/whatever else _ SSL cerVficate private keys
10.04.14 OSDC 2014 6
About DTLS heartbeats
_ RFC 6520, Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS) Heartbeat Extension _ Provides a heartbeat for TLS (TCP) and DTLS (mostly UDP) sessions _ Intended to add stability to unstable connecVons and prevent renegoVaVons _ Implemented in OpenSSL as part of a PhD thesis _ Patch commiQed Dec 15, 2011
10.04.14 OSDC 2014 7
What this bug is not
_ This is not a crypto bug _ At least not in its primary funcVon
_ This is not a fully arbitrary mem disclosure _ Only memory belonging to aQacked daemon can be dumped
_ This is not a remote root hole _ Hence the relaVvely low CVE score of 5.0
10.04.14 OSDC 2014 8
Anatomy of the bug 1
struct { HeartbeatMessageType type; uint16 payload_length; opaque payload[HeartbeatMessage.payload_length]; opaque padding[padding_length];
} HeartbeatMessage;
_ From RFC6520: _ payload_length: The length of the payload. _ payload: The payload consists of arbitrary content.
10.04.14 OSDC 2014 9
Anatomy of the bug 2
_ ssl/d1_both.c, line 1474+: buffer = OPENSSL_malloc(1 + 2 + payload + padding);
bp = buffer;
[..]
memcpy(bp, pl, payload);
_ From: https://github.com/openssl/openssl/commit/4817504d069b4c5082161b02a22116ad75f822b1
10.04.14 OSDC 2014 10
Anatomy of the bug
_ The heartbeat extension allocates payload+19 bytes of memory _ Copies pl bytes of arbitrary user-‐supplied data payload via memcpy() to construct response _ Client sets pl to 65535 _ Client sends only 1 byte of data in payload
_ Response contains 1 byte of client-‐supplied payload _ ...and 64K of RAM from the memcpy() call _ Analysis in: hQp://blog.existenValize.com/diagnosis-‐of-‐the-‐openssl-‐heartbleed-‐bug.html
10.04.14 OSDC 2014 11
Test vulnerability
_ Python script at: hQps://gist.github.com/takeshixx/10107280 _ Can test any SSL/TLS enabled TCP service
_ Has support for StartTLS (-‐s opVon) _ Conveniently dumps 64kb of memory for you
10.04.14 OSDC 2014 12
00d0: 10 00 11 00 23 00 00 00 0F 00 01 01 33 41 31 25 ....#.......3A1% 00e0: 32 43 25 32 32 5F 6D 6F 64 65 25 32 32 25 33 41 2C%22_mode%22%3A 00f0: 25 32 32 6A 73 6F 6E 25 32 32 25 32 43 25 32 32 %22json%22%2C%22 0100: 5F 69 64 25 32 32 25 33 41 25 32 32 70 5F 33 30 _id%22%3A%22p_30 0110: 33 34 35 38 31 38 25 32 32 25 32 43 25 32 32 5F 345818%22%2C%22_ 0120: 63 6F 6E 74 61 69 6E 65 72 25 32 32 25 33 41 30 container%22%3A0 0130: 25 32 43 25 32 32 5F 61 63 74 69 6F 6E 25 32 32 %2C%22_action%22 0140: 25 33 41 25 32 32 76 69 65 77 25 32 32 25 32 43 %3A%22view%22%2C
Memdump
_ From: hQps://twiQer.com/markloman/status/453502888447586304
10.04.14 OSDC 2014 13
Memdump
_ Memory contents is non-‐determinisVc _ SomeVmes exciVng, mostly boring
_ while true do python hb-test.py yahoo.com | grep -C 2 login >> /tmp/out; sleep 1; done"
_ Profit!
10.04.14 OSDC 2014 14
Detect exploitation
_ No logging on the machine _ All exploitaVon is pre-‐logging, pre-‐applicaVon _ IDS vendors are pushing out signatures already
10.04.14 OSDC 2014 15
Affected services
_ Above all, SSL-‐enabled web servers _ Any that uses OpenSSL, anyway
_ Mail servers _ IMAP over SSL, POP over SSL, SMTP over SSL, StartTLS
_ VPN tunnels _ OpenVPN when using cert auth (maybe?) _ PotenVally others
_ IRC servers, XMPP, FTP over TLS _ Android 4.1.1 is vulnerable _ OpenSSH is not vulnerable
10.04.14 OSDC 2014 16
Linux versions affected
_ OpenSSL 1.0.1 a thru f _ Debian Wheezy, Jessie, Sid
_ Fixed for Wheezy & Sid
_ Ubuntu 10.04, 12.04, 12.10, 13.10, 14.04 _ Fixed packages exist
_ RHEL 6 _ Patch exists
_ And all others that ship OpenSSL _ Clients are also vulnerable!
10.04.14 OSDC 2014 17
Other affected stuff
_ Cisco devices: „We use Cisco SSL which is not OpenSSL.“; SSL VPN products potenVally affected _ Juniper has released fixes for their SSL VPN, none for J-‐Web etc. yet _ Big IP? Kemp? Fritz.Box? Your home NAS? _ More info (hopefully) here: hQp://www.kb.cert.org/vuls/byvendor?searchview&Query=FIELD+Reference=720951&SearchOrder=4
10.04.14 OSDC 2014 18
Mitigation & cleanup
_ First, upgrade to fixed openssl _ apt-‐get install openssl libssl-‐1.0.0
_ Next, restart all services that load old lib _ Use checkrestart or lsof –n | grep DEL | grep ssl
_ If you use staVc binaries, recompile everything _ If you use Google‘s mod_spdy on Apache2.2, don‘t
_ It has its own staVcally linked mod_ssl which is shamefully out of date
10.04.14 OSDC 2014 19
What about certs?
_ It is possible that privkeys have leaked _ If so, you need to revoke&reissue certs _ Some CAs offer free reissue _ If you don‘t have PFS, you have a problem _ AQackers who sniffed your traffic might be able to decode it
10.04.14 OSDC 2014 20
Thank you
_ .Do not despair, there is hope!
_ ...and now, back to our regular scheduled programme!
10.04.14 OSDC 2014 21
hQp://xkcd.com/1353/
Software-defined Networking
In an open-source cloud
Agenda
_ High-‐Level overview: What is this about? _ The use case – virtualized networks for IaaS _ Intro to OpenVSwitch _ How-‐to: Deploy OpenVSwitch _ Frontnet, Backnet, public net _ Firewalling _ Tying it all together
10.04.14 OSDC 2014 23
So what‘s the hype?
_ Sovware-‐Defined Networking is the hype _ I‘m not good with hype
_ Networking is decoupled from bare metal _ EssenVally you virtualize parts of your network _ Control and data plane are decoupled
_ Many vendors jumped on the train _ HP, Cisco, VMWare, you name it
10.04.14 OSDC 2014 24
OpenFlow
_ ImperaVve control _ Switches are dumb – they only forward according to rules _ OpenFlow controllers make the rules _ First packet of each type is sent thru OpenFlow controller _ Subsequent ones go directly through switch
10.04.14 OSDC 2014 25
OpFlex
_ Cisco‘s answer to OpenFlow _ Other vendors on board: Citrix, MSFT, RHAT, Canonical _ Not on board: J, HP, Huawei, vmWare
_ Balance intelligence between switch and controller _ „DeclaraVve control“; just declare how you want it and the switch interprets that rule
_ IETF proposed standard _ Drav-‐smith-‐opflex _ Open APIs
_ AltruisVc goal: Eliminate SPOF (the controller) _ EgoisVc goal: Sell smarter (=$++) switches
10.04.14 OSDC 2014 26
The OSS Contender
_ OpenVSwitch _ Openvswitch.org
_ Open Source _ Apache 2.0 license, non-‐viral _ GPLv2
_ MulVlayer (2,3) virtual switch
_ Supports lots of interesVng features _ VLANs, Ne{low, sFlow, LACP, filtering, ...
10.04.14 OSDC 2014 27
OVS Overview
_ Shamelessly lived from [1]
10.04.14 OSDC 2014 28
ovs-vswitchd
OVS Kernel Module
Control Cluster
ovsdb-server
Off-box
User Kernel
Management Protocol (6632/TCP)OpenFlow (6633/TCP)Netlink
OSVDB
_ Database holds configuraVon items _ DefiniVons for bridges, tunnels, interfaces _ Controller addresses
_ ConfiguraVon is reboot-‐safe _ Custom database system, not MySQLiteMongoDB _ Speaks custom protocol (OSVDB) _ Log based
_ osvdb-‐tool show-‐log shows all changes _ Nivy for debug / change management!
10.04.14 OSDC 2014 29
How ovs works
_ ImperaVve control _ All intelligence is in the controller _ Data path only carries out instrucVons
_ Data Path _ Kernel module _ Licensed under GPLv2
_ Controller _ Lives in userland _ Licensed under Apache 2.0
10.04.14 OSDC 2014 30
Flow flow
_ Everything is a flow _ CombinaVon of input port, VLAN, MAC, IP, TCP/UDP port
10.04.14 OSDC 2014 31
OVS management
_ Command-‐line tools _ Ovs-‐vsctl for switch management _ Ovs-‐ofctl for flow management _ Ovsdb-‐tool for database management
10.04.14 OSDC 2014 32
What‘s our angle here?
_ filoo is a hoster.
_ We host VMs.
_ VMs need networking.
_ See where this goes?
10.04.14 OSDC 2014 33
What we wanted
_ Internet-‐facing front-‐net interface _ Private LAN for VMs _ VM isolaVon _ Firewalling _ Traffic shaping _ Fine-‐grained accounVng _ Live migraVon
10.04.14 OSDC 2014 34
Overview - physical
10.04.14 OSDC 2014 35
Back-‐end switch
Front-‐end switch
Overview - virtual
10.04.14 OSDC 2014 36
Firewall Firewall Firewall
Overview – OVS stack
10.04.14 OSDC 2014 37
OVS OVS OVS
Let‘s get started
_ We usually compile ovs ourselves _ There are also packages in apt _ Those might work or not
_ Download & compile OVS _ Latest stable: 2.1.0, latest LTS: 1.9.3 _ ./boot.sh && ./configure && make && make install
_ Kernel module from 3.3+ _ Enable in Kernel Networking -‐> OpVons -‐> Open Vswitch _ modprobe openvswitch
10.04.14 OSDC 2014 38
Let‘s get started 2
_ Set up ovs db _ Ovsdb-‐tool create conf.db vswitch.ovsschema _ Conf.db is in /usr/localetc/openvswitch _ /usr/src/openvswitch-‐1.9.3/vswitchd/vswitch.ovsschema
_ Make sure ovs-‐vswitchd and ovsdb-‐server start before networking _ Add startup entries to rc.local _ Remove networking from rc.d _ start networking in rc.local
10.04.14 OSDC 2014 39
Initial bridges
_ Front-‐net vlan: 199 _ Same procedure for back-‐net VLAN _ Add bridges
_ ovs-‐vsctl add-‐br vmbr1 _ ovs-‐vsctl add-‐port vmbr1 vlan199 tag=199 _ ovs-‐vsctl set interface vlan199 type=internal
_ Log in via IPMI _ ovs-‐vsctl add-‐port vmbr1 eth1 _ Machine is offline now
_ Modify physical switching
10.04.14 OSDC 2014 40
VM networking
_ We use KVM/QEMU _ Add the TAP interface
_ /sbin/ip tuntap add dev tap1i0d0 mode tap user fcms _ qemu-‐system-‐x86_64 ... -‐device rtl8139,mac=00:F1:70:00:00:10,netdev=vlan0d0 -‐netdev type=tap,id=vlan0d0,ifname=tap1i0d0
_ Bring up the port _ /usr/local/bin/ovs-‐vsctl add-‐port vmbr0 tap1i0d0 199 other_config:stp-‐enable=false
10.04.14 OSDC 2014 41
From TAP to port to flow
_ We have a tap interface tap1i0d0
_ Find the corresponding bridge port: _ ovs-‐ofctl show vmbr0 | grep tap1i0d0 _ 1820(tap1i0d0): addr:fa:7a:67:e3:5d:�
_ Now we have a port number: 1820
_ We use this port for flow management
10.04.14 OSDC 2014 42
Multiple interfaces
_ Add more TAP interfaces _ Assign one VLAN per customer _ Internal network across VMs on same node
_ Make VLAN known on inter-‐node switches _ Via whatever switch automaVon you have
_ Cross-‐node internal networking _ VLAN limits apply – hard cut at ~4090 _ Overlay networks to the rescue
10.04.14 OSDC 2014 43
Prevent MAC spoofing
_ PORT=1820 _ ovs-‐ofctl add-‐flow vmbr0 "in_port="${PORT}" arp idle_Vmeout=0 priority=39500 acVon=resubmit("${PORT}",2)“ _ ovs-‐ofctl add-‐flow vmbr0 "in_port="${PORT}" table=2 arp priority=200 idle_Vmeout=0 arp_sha=00:F1:70:00:00:10 nw_src=192.168.1.1 acVon=normal" _ ovs-‐ofctl add-‐flow vmbr0 "in_port="${PORT}" table=2 priority=100 idle_Vmeout=0 acVon=drop"
10.04.14 OSDC 2014 44
We know this MAC because we control the hypervisor!
We know this address too!
Caveats for MAC/ARP
_ SomeVmes you want customers to spoof _ HA soluVons that switch „cluster IP addresses“ _ You can cater for this in case you know the corresponding MACs _ Assign sequenVal MACs and wildcard _ Or set specific rules _ OpVonal „HA feature“ for VMs _ Never allow customers to wildcard here!
10.04.14 OSDC 2014 45
Firewalling with flows
_ ovs-‐ofctl add-‐flow vmbr0 "in_port="${PORT}" table=1 tcp idle_Vmeout=0 nw_dst=192.168.12.13/32 nw_src=192.168.1.123/32 tp_dst="80" priority=38000 acVon=drop“ _ From 192.168.1.123 _ To 192.168.12.13 _ Port 80 _ Drop
10.04.14 OSDC 2014 46
Port ranges
_ ovs-‐ofctl add-‐flow vmbr0 "in_port="${PORT}" table=1 tcp idle_Vmeout=0 nw_src=192.168.1.123/32 nw_dst=192.168.12.13/24 tp_src="0x05E8/0xFFFC" priority=37960 acVon=drop“ _ Source 192.168.1.123 _ DesVnaVon 192.168.12.0/24 _ Source port = 0x05E8/0xFFFC _ 0x05E8/0xFFFC = 1512/65532 _ Port 1512 – 1516
_ OVS 1.11 supports „Megaflows“, i.e universal wildcarding
10.04.14 OSDC 2014 47
Default accept
_ ovs-‐ofctl add-‐flow vmbr0 "in_port="${PORT}" table=1 priority=100 acVon=normal“ _ Fallthru rule _ Match everything else
10.04.14 OSDC 2014 48
Accounting
_ We grab interface counters from the tap interfaces _ You can also use Ne{low/sFlow or ipfix _ We didn‘t go there yet, experiences welcome
10.04.14 OSDC 2014 49
Shaping
_ Simple shaping: _ ovs-‐vsctl set Interface tap0 ingress_policing_rate=100000 _ ovs-‐vsctl set Interface tap0 ingress_policing_burst=1000
_ QoS policies: _ ovs-‐vsctl set port eth1 qos=@newqos \ id=@newqos create qos type=linux-‐htb \ other-‐config:max-‐rate=200000000 queues=0=@q0,1=@q1 \
_ We don‘t do QoS policies, shaping works mostly as intended
10.04.14 OSDC 2014 50
Live migration
_ We don‘t actually do OVS‘s own live migraVon _ Start VM on target host in suspend-‐to-‐RAM mode _ Stop VM on losing host; down interface _ Resume VM on target host
_ There are live migraVon mechanisms in OVS _ L2 based _ Inter-‐OVS GRE tunnel _ Honestly, I have no clue.
10.04.14 OSDC 2014 51
Thank you
_ I hope you learned something _ If not, I hope you had a laugh at my expense _ If neither, I‘m really sorry. Beer?
_ QuesVons?
10.04.14 OSDC 2014 52
Literature
_ [1] hQp://openvswitch.org/slides/OpenStack-‐131107.pdf – OVS Deep Dive _ OVS IntroducVon: hQp://horms.net/projects/openvswitch/2010-‐10/openvswitch.en.pdf
10.04.14 OSDC 2014 53