TCP Issues in DataCenter Networks
-
Upload
hemanth-kumar-mantri -
Category
Technology
-
view
506 -
download
0
description
Transcript of TCP Issues in DataCenter Networks
![Page 1: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/1.jpg)
TCP Issues in Virtualized Datacenter Networks
Hemanth Kumar MantriDepartment of Computer Science 1 of 27
![Page 2: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/2.jpg)
Selected Papers
• The TCP Outcast Problem: Exposing Unfairness in Data Center Networks.– NSDI’12
• vSnoop: Improving TCP Throughput in Virtualized Environments via Ack Offload.– ACM/IEEE SC, 2010
2 of 27
![Page 3: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/3.jpg)
Background and Motivation
• Data center is a shared environment– Multi Tenancy
• Virtualization: A key enabler of cloud computing– Amazon EC2
• Resource sharing– CPU/Memory are strictly shared– Network sharing largely laissez-faire
3 of 27
![Page 4: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/4.jpg)
Data Center Networks
• Flows compete via TCP• Ideally, TCP should achieve true fairness
– All flows get equal share of link capacity
• In practice, TCP exhibits RTT-bias– Throughput is inversely proportional to RTT
• 2 Major Issues– Unfairness (in general)– Low Throughput (in virtualized environments)
4 of 27
![Page 5: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/5.jpg)
Datacenter Topology (Hierarchical)
5 of 27
![Page 6: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/6.jpg)
Traffic Pattern: Many to One
6 of 27
![Page 7: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/7.jpg)
Key Find: Unfairness
Inverse RTT Bias?Low RTT = Low Throughput
7 of 27
![Page 8: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/8.jpg)
Further Investigation
Instantaneous Average
2-hop flow is consistently starved!!
TCP Outcast Problem• Some Flows are ‘Outcast’ed and receive very low
throughput compared to others• Almost an order of magnitude reduction in some
cases
8 of 27
![Page 9: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/9.jpg)
Experiments• Same RTTs• Same Hop Length• Unsynchronized Flows• Introduce Background Traffic• Vary Switch Buffer Size• Vary TCP
– RENO, MP-TCP, BIC, Cubic + SACK
• Unfairness Persists!9 of 27
![Page 10: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/10.jpg)
Observation
Flow differential at input ports is the culprit! 10 of 27
![Page 11: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/11.jpg)
Vary #flows at competing bottle neck switch
11 of 27
![Page 12: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/12.jpg)
Reason: Port Blackout
1. Packets are roughly same size2. Similar inter-arrival rates (Predictable Timing)
12 of 27
![Page 13: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/13.jpg)
Port Blackout• Can occur on any input port• Happens for small intervals of time• Has more catastrophic effect on
throughput of fewer flows!!– Experiments showed that “same number” of
packet drops affect the throughput of fewer flows much more than if there were several concurrent flows.
13 of 27
![Page 14: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/14.jpg)
Conditions for TCP Outcast
14 of 27
![Page 15: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/15.jpg)
Solutions?• Stochastic Fair Queuing (SFQ)
– Explicitly enforce fairness among flows– Expensive for commodity switches
• Equal Length Routing– All flows are forced to go through Core– Better interleaving of packets, alleviate PB
15 of 27
![Page 16: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/16.jpg)
• Multiple VMs hosted by one physical host• Multiple VMs sharing the same core
– Flexibility, scalability, and economy
VM Consolidation
Hardware
Virtualization Layer
VM 1 VM 3 VM 4VM 2
Observation:VM consolidation negatively impacts network performance!
16 of 27
![Page 17: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/17.jpg)
Sender
Hardware
Virtualization Layer
Investigating the ProblemServer
VM 1 VM 2 VM 3
Client
17 of 27
![Page 18: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/18.jpg)
40
60
80
100
120
140
160
180
5432
RT
T (
ms)
Number of VMs
RTT increases in proportion to VM scheduling slice
(30ms)
Effect of CPU Sharing
18 of 27
![Page 19: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/19.jpg)
Exact Culprit
Sender
Hardware
Driver Domain(dom0)
VM 1
Device Driver
VM 3
bufbuf
VM 2
buf
19 of 27
![Page 20: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/20.jpg)
Connection to the VM is much slower than dom0!
Impact on TCP Throughput
+ dom0 x VM
20 of 27
![Page 21: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/21.jpg)
Solution: vSnoop• Alleviates the negative effect of VM scheduling on
TCP throughput• Implemented within the driver domain to
accelerate TCP connections
• Does not require any modifications to the VM• Does not violate end-to-end TCP semantics• Applicable across a wide range of VMMs
– Xen, VMware, KVM, etc.
21 of 27
![Page 22: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/22.jpg)
Sender VM1 BufferDriver Domain
Time
SYN
SYN,ACK
SYN
SYN,ACK
VM1 buffer
TCP Connection to a VM
Scheduled VM
VM1
VM2
VM3
VM1
VM2
VM3
SYN,ACKSYN
VM Scheduling Latency
RTT
RTT
VM Scheduling Latency
Sender establishes a TCP connection to VM1
22 of 27
![Page 23: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/23.jpg)
Sender VM Shared BufferDriver Domain
Time
SYN
SYN,ACK
SYN
SYN,ACK
VM1 buffer
Key Idea: Acknowledgement Offload
Scheduled VM
VM1
VM2
VM3
VM1
VM2
VM3
SYN,ACK
w/ vSnoop
Faster progress during TCP slowstart
23 of 27
![Page 24: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/24.jpg)
• Challenge 1: Out-of-order/special packets (SYN, FIN packets)• Solution: Let the VM handle these packets
• Challenge 2: Packet loss after vSnoop• Solution: Let vSnoop acknowledge only if room in buffer
• Challenge 3: ACKs generated by the VM• Solution: Suppress/rewrite ACKs already generated by vSnoop
Challenges
24 of 27
![Page 25: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/25.jpg)
vSnoop Implementation in Xen
Driver Domain (dom0)
Bridge
Netfront
Netback
vSnoop
VM1
Netfront
Netback
VM3
Netfront
Netback
VM2
buf buf buf
Tuning Netfront
25 of 27
![Page 26: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/26.jpg)
Median
0.192MB/s
0.778MB/s
6.003MB/s
TCP Throughput Improvement• 3 VMs consolidated, 1000 transfers of a 100KB file• Vanilla Xen, Xen+tuning, Xen+tuning+vSnoop
30x Improvement
+ Vanilla Xen x Xen+tuning * Xen+tuning+vSnoop
26 of 27
![Page 27: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/27.jpg)
Thank You!
• References– http://friends.cs.purdue.edu/dokuwiki/doku.php
– https://www.usenix.org/conference/nsdi12/tech-schedule/technical-sessions
• Most animations and pictures are taken from the authors’ original slides and NSDI’12 conference talk.
27 of 27
![Page 28: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/28.jpg)
BACKUP SLIDES
28
![Page 29: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/29.jpg)
Conditions for Outcast• Switches use the tail-drop queue
management discipline
• A large set of flows and a small set of flows arriving at two different input ports compete for a bottleneck output port at a switch
29
![Page 30: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/30.jpg)
Why does Unfairness Matter?
• Multi Tenant Clouds– Some tenants get better performance than
others
• Map Reduce Apps– Straggler problems– One delayed flow affects overall job
completion
30
![Page 31: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/31.jpg)
State Machine Maintained Per-FlowStart
Unexpected Sequence
Active(online)
No buffer(offline)
Out-of-order packet
In-order pkt Buffer space available
Out-of-order packet
In-order pktNo buffer
In-order pkt Buffer space available
No buffer
Packet recvEarly acknowledgementsfor in-order packets
Don’t acknowledge
Pass out-of-order pkts to VM
31
![Page 32: TCP Issues in DataCenter Networks](https://reader033.fdocument.pub/reader033/viewer/2022060108/55515304b4c905e1708b4697/html5/thumbnails/32.jpg)
vSnoop’s Impact on TCP Flows• Slow Start
– Early acknowledgements help progress connections faster
– Most significant benefit for short transfers that are more prevalent in data centers
• Congestion Avoidance and Fast Retransmit– Large flows in the steady state can also benefit
from vSnoop– Benefit not as much as for Slow Start
32