Post on 08-Feb-2016
description
Sizing Router BuffersSizing Router Buffers
Nick McKeown
Guido Appenzeller & Isaac Keslassy
SNRC ReviewMay 27th, 2004
2
Routers need Packet BuffersRouters need Packet Buffers
It’s well known that routers need packet buffers It’s less clear why and how much
Goal of this work is to answer the question:
How much buffering do routers need?
Given that queueing delay is the only variable part of packet delay in the Internet, you’d think we’d know the answer already!
3
How much Buffer does a Router How much Buffer does a Router need?need?
Universally applied rule-of-thumb: A router needs a buffer size:
2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link
Context Mandated in backbone and edge routers. Appears in RFPs and IETF architectural guidelines.. Usually referenced to Villamizar and Song: “High Performance
TCP in ANSNET”, CCR, 1994. Already known by inventors of TCP [Van Jacobson, 1988] Has major consequences for router design
CTB 2
CRouterSource Destination
2T
4
ExampleExample
10Gb/s linecard Requires 300Mbytes of buffering. Read and write 40 byte packet every 32ns.
Memory technologies DRAM: require 4 devices, but too slow. SRAM: require 80 devices, 1kW, $2000.
Problem gets harder at 40Gb/s Hence RLDRAM, FCRAM, etc.
5
Outline of this WorkOutline of this Work
Main Results The rule of thumb is wrong for a core routers today
Required buffer is instead of CT 2n
CT 2
Outline of this talk Where the rule of thumb comes from Why it is incorrect for a core router in the internet today Correct buffer requirements for a congested router Buffer requirements for short flows (slow-start) Experimental Verification Conclusion
6
OutlineOutline
The Rule of Thumb Where does the rule of thumb comes from? (Answer: TCP) Interaction of TCP flows and a router buffers
The buffer requirements for a congested router Buffer requirements for short flows (slow-start) Experimental Verification Conclusion
7
TCPTCP
CC’ > C
Only W=2 packets may be outstanding
TCP Congestion Window controls the sending rate Sender sends packets, receiver sends ACKs Sending rate is controlled by Window W, At any time, only W unacknowledged packets may be outstanding
The sending rate of TCP is
RouterSource
RTT
WR
Dest
8
Single TCP FlowSingle TCP FlowRouter with large enough buffers for full link utilizationRouter with large enough buffers for full link utilization
B
DestCC’ > C
Source
maxW
2maxW
t
Window size RTT
For every W ACKs received, send W+1 packets
RTT
WR
13
Origin of rule-of-thumbOrigin of rule-of-thumb Before and after reducing window size, the sending rate of the
TCP sender is the same
Inserting the rate equation we get
The RTT is part transmission delay T and part queueing delay B/C . We know that after reducing the window, the queueing delay is zero.
newold RR
new
new
old
old
RTT
W
RTT
W
T
W
CBT
W oldold
2
2/
/2
CTB 2
)1( 1
noldnew WW CTBn
2)(1
1
14
Rule-of-thumbRule-of-thumb
Rule-of-thumb makes sense for one flow Typical backbone link has > 20,000 flows Does the rule-of-thumb still hold?
Answer: If flows are perfectly synchronized, then Yes. If flows are desynchronized then No.
15
OutlineOutline
The Rule of Thumb The buffer requirements for a congested router
Synchronized flows Desynchronized flows The 2T×C/sqrt(n) rule
Buffer requirements for short flows (slow-start) Experimental Verification Conclusion
16
If flows are synchronizedIf flows are synchronized
Aggregate window has same dynamics Therefore buffer occupancy has same dynamics Rule-of-thumb still holds.
2maxW
t
max
2
W
maxW
maxW
17
When are Flows When are Flows Synchronized?Synchronized?
Small numbers of flows tend to synchronize
Large aggregates of flows are not synchronized For > 200 flows, synchronization disappears Measurements in the core give no indication
of synchronization
18
If flows are not If flows are not synchronizedsynchronized
ProbabilityDistribution
B
0
Buffer Size
W
19
Quantitative ModelQuantitative Model Model congestion window of a flow as random variable
)(tWi model as )(][ xfxWP i iW where
For many de-synchronized flows We assume congestion windows are independent All congestion windows have the same probability distribution
2]var[][ WiWi WWE
Now central limit theorem gives us the distribution of the sum of the window sizes
)1,0()( NnntW WWn
i
20
Buffer vs. Number of FlowsBuffer vs. Number of Flowsfor a given Bandwidthfor a given Bandwidth
If for a single flow we have
Standard deviation of sum of windows decreases with n
)1,0(1
)( 11 Nn
tW nnn
i
Thus as n increases, buffer size should decrease
n
BB n 1
211 ]var[][ nn WWE
For a given C, the window W scales with 1/n and thus2
11 ]var[][
nW
nWE nn
22
SummarySummary Flows in the core are desynchronized
For desynchronized flows, routers need only buffers of
n
CTB
2
23
OutlineOutline
The Rule of Thumb The buffer requirements for a congested router Buffer requirements for short flows (slow-start)
Experimental Verification Conclusion
24
Short FlowsShort Flows
So far we were assuming a congested router with long flows in congestion avoidance mode. What about flows in slow start? Do buffer requirements differ?
Answer: Yes, however: Required buffer in such cases is independent of line
speed and RTT (same for 1Mbit/s or 40 Gbit/s) In mixes of flows, long flows drive buffer requirements
Short flow result relevant for uncongested routers
25
A single, short-lived TCP flowA single, short-lived TCP flowFlow length 62 packets, RTT ~140 msFlow length 62 packets, RTT ~140 ms
2
4
8
16
32
RTTsynfin ack
received
Flow Completion Time (FCT)
27
Queue DistributionQueue Distribution
We derived closed-form estimates of the queue distribution using Effective Bandwidth Gives very good closed form approximation
Buffer requirements for short flows Small & independent of line speed and RTT
In mixes of flows, long flows dominate buffer requirements
]E[S
E[S]
ρ
ρ)(κeb)P(Q bκ
2
12
28
OutlineOutline
The Rule of Thumb The buffer requirements for a congested router Buffer requirements for short flows (slow-start) Experimental Verification Conclusion
29
Experimental Evaluation Experimental Evaluation OverviewOverview
Simulation with ns2 Over 10,000 simulations that cover range of settings
Simulation time 30s to 5 minutes Bandwidth 10 Mb/s - 1 Gb/s Latency 20ms -250 ms,
Physical router Cisco GSR with OC3 line card In collaboration with University of Wisconsin
Experimental results presented here Long Flows - Utilization Mixes of flows - Flow Completion Time (FCT) Mixes of flows - Heavy Tailed Flow Distribution Short Flows – Queue Distribution
30
Long Flows - Utilization (I)Long Flows - Utilization (I)Small Buffers are sufficient - OC3 Line, ~100ms RTTSmall Buffers are sufficient - OC3 Line, ~100ms RTT
2T C
n
99.9%
98.0%
99.5%2T C
n
2×
31
Long Flows – Utilization (II)Long Flows – Utilization (II) Model vs. ns2 vs. Physical Model vs. ns2 vs. Physical
RouterRouterGSR 12000, OC3 Line CardGSR 12000, OC3 Line Card
TCP
Flows
Router Buffer Link Utilization
Pkts RAM Model Sim Exp
100 0.5 x
1 x
2 x
3 x
64
129
258
387
1Mb
2Mb
4Mb
8Mb
96.9%
99.9%
100%
100%
94.7%
99.3%
99.9%
99.8%
94.9%
98.1%
99.8%
99.7%
400 0.5 x
1 x
2 x
3 x
32
64
128
192
512kb
1Mb
2Mb
4Mb
99.7%
100%
100%
100%
99.2%
99.8%
100%
100%
99.5%
100%
100%
99.9%
2T C
n
32
OutlineOutline
The Rule of Thumb The buffer requirements for a congested router Buffer requirements for short flows (slow-start) Experimental Verification Conclusion
33
Impact on Router DesignImpact on Router Design 10Gb/s linecard with 200,000 x 56kb/s flows
Rule-of-thumb: Buffer = 2.5Gbits Requires external, slow DRAM
Becomes: Buffer = 6Mbits Can use on-chip, fast SRAM Completion time halved for short-flows
40Gb/s linecard with 40,000 x 1Mb/s flows Rule-of-thumb: Buffer = 10Gbits Becomes: Buffer = 50Mbits
For more details… “Sizing Router Buffers – Guido Appenzeller, Isaac Keslassy and
Nick McKeown, to appear at SIGCOMM 2004