Wresting Control from BGP: Scalable Fine-grained Route Control UCSD / AT&T Research Usenix —June 22, 2007 Dan Pei, Tom Scholl, Aman Shaikh, Alex C. Snoeren, Kobus van der Merwe Patrick Verkaik

What is route control? ISP Route control ? ? Traffic destination Traffic ISPs need to control selection of alternate paths … in response to dynamic network conditions ISPs need to control selection of alternate paths … in response to dynamic network conditions measurement Historically routing provides simple connectivity But demands are changing: gaming, DDoS defense Historically routing provides simple connectivity But demands are changing: gaming, DDoS defense However, demands are changing:

IRSCP: enabling route control IRSCP: Intelligent Route Service Control Point –Next step after RCP [Caesar et al.] Enable route control for inter-domain traffic : –Automated, based on network conditions –Scalable to demands of large ISP –No changes to existing ISP infrastructure

Simple example: load balancing Customer ISP Traffic Egress router R1Ingress router R5 Border Gateway Protocol (BGP) BGP defaults to hot potato routing Effectively: shortest path routing Overloaded link Traffic on customer access links is balanced R3 R2 R4 Simple route control objective Customers are asking for this Yet BGP can’t do it! Simple route control objective Customers are asking for this Yet BGP can’t do it!

Customer 1 Why does BGP do that? ISP 1 ISP 2 Customer 2 External BGP (eBGP) session Internal BGP (iBGP) session ISP router External router Selec t route dest d R1 R3 R2 R4 Hundreds of ISP routers making local decisions Hot potato default policy: –Ensures consistency –..but precludes control over routing Hundreds of ISP routers making local decisions Hot potato default policy: –Ensures consistency –..but precludes control over routing

IRSCP route control Abstraction: “for this destination, direct traffic from this ISP router to that ISP router” Automated, based on network conditions: route control application Backwards compatible, consistent and scalable Selective: allow default BGP decision where desired

Outline IRSCP architecture Routing consistency Implementation Evaluation

Customer 1 Route control abstraction ISP 1 ISP 2 Customer 2 Traffic Egress links RC App iBGP session RC App IRSCP Assign- ments RC App assigns egress links to ISP routers Speak BGP to routers …using IRSCP Assign Route ? ? R4 R1 R3 R2

Route failover What if a route for egress link fails? –RC application at relatively slow timescale –IRSCP must fail over instantly So application sends a list of egress links for each ISP router –We call this a ranking

Customer 1 Rankings ISP 1 ISP 2 Customer 2 Traffic RC App IRSCP Rankings R1 R4 R3 R R4 R3R1 Select route for R1 and R3 Select route for R2 and R4 Rankings map traffic from ingress to egress arbitrarily And allow route fail-over at routing time-scale Rankings map traffic from ingress to egress arbitrarily And allow route fail-over at routing time-scale

What about scalability? IRSCP talks to many thousands of routers Responsible for route decision for each ISP router: –Computation –Single point of failure Maintaining BGP session for each router : –State per session –Each eBGP session may add a route 2-3 GB sufficient Distributed IRSCP

Customer 1 Distributed IRSCP ISP 1 ISP 2 Customer 2 Multiple IRSCP servers: –To distribute BGP sessions –For geographic diversity Routers may peer with several IRSCP servers IRSCP servers are replicas: exchange all routes IRSCP IRSCP session R4 R1 R3 R2

Outline IRSCP architecture Routing consistency Implementation Evaluation

Consistency Forwarding anomalies: Deflection Traffic R1 R2 R3 R2 Looping R1 R3 Rankings must be “consistent”

R1 R3 R2 R4 Example of inconsistent rankings RC application checks consistency constraints on rankings No anomalies Traffic R1 R2 Rankings R3R4 Deflection

Outline IRSCP architecture Routing consistency Implementation Evaluation

Expl. ranked decision process Hot potato/BGP decision process Routing information baseRoutes Prototype implementation IRSCP server RC App Rankings R1 IRSCP server R2 IRSCP server Import policy Export policy I1 I2

Outline IRSCP architecture Routing consistency Implementation Evaluation –Can IRSCP handle routing load in a real ISP? –Both explicitly ranked and hot potato decision process

Methodology Emulation of realistic large ISP topology and routing load Connect IRSCP implementation to emulation of ISP Measure if our implementation handles the emulated load

Customer 1ISP 2 Deployment Scenario IRSCP 240 external routers per POP 15 ISP routers per POP 40 POPs POP 1 IRSCP server per POP IRSCP session BGP session Throughput of IRSCP server depends on how many of each kind of session it has

Finding maximum throughput IRSCP Input update rate Route update receiver Mix of BGP and IRSCP sessions based on ISP scenario IRSCP implementation: –3.6-GHz Xeon –4 GB memory Search for maximum sustainable input rate: –Gradually increase input rate, sustaining for twenty minutes –Compute expected output rate given input rate –Once measured output rate falls behind, we’ve reached maximum throughput Output update rate Route update generator Multiplier

Maximum throughput 3600 Output rate Input rate Hot potato Explicitly ranked % in real ISP Average in real ISP 220 updates/s Flexibility of rankings has a cost But IRSCP handles Tier-1 ISP routing load, and more Flexibility of rankings has a cost But IRSCP handles Tier-1 ISP routing load, and more

Conclusion IRSCP route control platform: –Feedback of network conditions into route selection –Scalable, robust against failures, backward compatible –Powerful, yet safe ranking abstraction Enables new class of “route control application”: –Security –Traffic engineering –Customer-oriented services Trials of IRSCP for several applications taking place in AT&T!

Alex Aman Questions? Kobus Tom Dan Patrick

BGP sessions As we saw, IRSCP speaks iBGP to ISP routers For full control, IRSCP must also speak eBGP to external routers IRSCP Multihop eBGP session iBGP session I E1 E2 eBGP session Selec t route Select route for I

Example consistency constraint If e 1 outranks e 2 at R1 then must also do so at router(e 1 )=R3 Two simple consistency constraints Checked by RC application before sending to IRSCP Example: R1 R3 e1e1 e2e2 e1e1 e2e2 R1 R2 R3R4 e1e1 e2e2 R2 R4

Throughput Achieved output rate Estimated 95 perc. required input rate Achieved input rate Out of 255 routers per POP