1. Shedding Light on the Glue Logic of the Internet Routing Architecture
ACM SIGCOMM 2008
August 19th 2008
Franck Le1, Geoffrey Xie2, Dan Pei3, Jia Wang3, Hui Zhang1
1 Carnegie Mellon University
2 Naval Postgraduate School
3 AT&T Labs – Research 1 1

2. IGP domains linked, not by BGP
Routing Design
Routing designs of operational networks are complex1
Multiple IGP domains per network
BGP
IGP
OSPF
EIGRP
ISIS
RIP
IGP domains linked, not by BGP
1Maltz, et al. Routing design in operational networks: A look from the
inside. SIGCOMM 04

3. Routing Glue Logic
Recent study1 revealed the existence of a lower level glue logic to interconnect routing domains
Route redistribution allows exchange of routing information among routing domains
FIB
route selection
RIP
OSPF B
OSPF
RIP A B D C
Route redistribution provides required “glue logic” between routing domains
1Le, Xie, Zhang, Understanding Route Redistribution, ICNP 07

4. How does the Glue Logic compare to BGP?
Glue logic can implement policies, like BGP
Unlike BGP, glue logic is NOT a protocol
RR is just configuration mechanism, used separately at each router, and extremely vulnerable to anomalies1
Our discussions with operators revealed glue logic, not BGP, is often used to interconnect network domains
Even when BGP is used, glue logic is required to specify the routes to advertise at the BGP level
Glue logic seems more commonly used than BGP, but is much less understood and much more error-prone
1Le, Xie, Zhang, Understanding Route Redistribution, ICNP 07 4

5. Contributions
Developed a model for characterizing interconnections between routing domains
Analyzed configurations of networks
Show the glue logic is fundamental component of Internet routing architecture
Show insufficiencies of glue logic lead to complex configurations and instability concerns
Discuss potential role of glue logic as the Internet architecture evolves
Glue logic, a critical component of Internet architecture, that needs more research 5

6. Dataset 1600+ operational networks Networks
Tier-1 Service Provider
Enterprise networks
University campuses
Number of routers per network: 1 to 3000+
Number of lines per router:
Average: 675 lines
Maximum: lines

7. Routing Domains 10% 21% 34% % of networks with ≤ n routing domains
Number of Routing Domains (n)

8. Prevalence of Route Redistribution
99.9% networks rely on route redistribution
From IGP and local routes to BGP
From BGP into IGP (78% of networks with 15+ routers)
From IGP into IGP (35% of networks with 15+ routers)
BGP
BGP
BGP
BGP
IGP, local
IGP, local
IGP, local
Office 1
IGP
Office 1
IGP
BGP
Backbone
BGP
Backbone
Office 2
IGP
Office 2
IGP
Office 1
OSPF
Office 1
IGP
Office 2
RIP
Office 2
IGP

9. Route Redistribution between IGP Domains
Why using route redistribution, rather than BGP, to interconnect IGP domains?
BGP cannot support some of the existing design objectives
Domain Backup
Router-level Shortest Path Routing

10. Domain Backup X RIP OSPF RIP OSPF RIP OSPF X Y BGP 65001 BGP 65002 X Y
X, AS PATH: 65001, 65002
X, AS PATH: 65002
Domain Backup
X, AS PATH: 65002
RIP
OSPF X Y
BGP 65001
BGP 65002 X
X, AS PATH: 65001, 65002 /0
other routes
RIP
Site
OSPF
Backbone A B X Y

11. Router-level Shortest Path Routing
BGP 65001
BGP 65002
BGP 65003
America
(OSPF3)
Europe
(OSPF2)
Asia
(OSPF1)
Receiver
Sender
Sender
BGP cannot support router-level shortest path routing

12. ? Router-level Shortest Path Routing
router ospf 1
redistribute ospf 3 metric type 1 subnets route-map UStoAsia
... !
route-map UStoAsia permit 10
match ip address prefix-list US
set tag 1
route-map UStoAsia permit 20
match tag 4
set tag 5
route-map UStoAsia permit 30
match tag 8
set tag 9
route-map UStoAsia deny 100 10 10
Europe
(OSPF2)
Receiver 5 5 1
Asia
(OSPF1) 7 1 12 ?
America
(OSPF3) 12 8 5
Sender 20 17
With route redistribution, cost of routes can be preserved across routing domains

13. Routing Policies
Configurations of route redistributions have complex policies
Tags, prefix filters, etc.
Rationale
Route redistribution does not include any mechanism to thwart routing anomalies
As such, each network designs own ad-hoc solution to prevent routing anomalies

14. Illustration of Deployed Solutions
Asia
(OSPF1)
X…11
Europe
(OSPF2)
b31…b1b0
X…01
b31…b2b1b0
route-map UStoAsia permit 10
match ip address prefix-list US
set tag 1 !
route-map UStoAsia permit 20
match tag 4
set tag 5
...
America
(OSPF3)
route-map UStoAsia permit 30
match tag 8
set tag 9 !
...
route-map UStoAsia deny 100
X…00
Count to infinity
problem
Dest. X
b31…b1b0
bitmap
32 bits tag

15. Routing Anomalies Clever solutions to prevent routing anomalies
Yet, ensuring safety of route redistribution, proven to be very difficult1
Indeed,
Analyzed configurations, still vulnerable to route oscillations
Route redistribution, long suspected to be at the origins of reported long-lived inter-AS loops
1Le, Xie, Zhang, Understanding Route Redistribution, ICNP 07

16. Route redistribution (RR)
Concluding Remarks
Glue logic, a fundamental component of Internet routing architecture
Implements a necessary function
Widely used in operational networks
Used to fulfill important design objectives
Existing glue logic, powerful tool, but severe limitations
Introduced by router vendors in an ad-hoc manner
No consideration of safety properties
Glue logic
Route selection (RS)
Route redistribution (RR)

17. Concluding Remarks (continued)
Glue logic’s functions are necessary but how to achieve them safely?
Level of abstraction?
Definitions of primitives?
Correctness of routing protocols, not sufficient to ensure robustness of networks
Except few exceptions1,2, most work has focused on individual routing protocols
Yet, glue logic can result in routing anomalies
1Griffin et al., On the correctness of IBGP configuration, SIGCOMM 02
2Teixeira et al., Dynamics of Hot-Potato Routing in IP Networks, SIGMETRICS 04

18. Acknowldegment Jay Borkenhagen Appanna Chottera Mike Donoghue
Alex Gerber
Timothy Griffin
Seungjoon Lee
Steve Legget
Mark Lyn
Jason Philippon
Mike Satterlee
Tom Scholl
Aman Shaikh
Philip Taylor
Kobus van der Merwe
Others (anonymity)

19. Thank you

20. Appendix: Limitations of BGP Local Pref
Europe
BGP 65001 Q
cost: 13 2 2
Asia R 1 1
Receiver X
cost: 10 2
cost: 12 1 Y M N X S D 5 5
Sender A
America
BGP 65003
Shortest path is not selected

21. Appendix: Limitations of BGP communities
Europe
BGP 65001 Q 2 2
Asia Y R 1 1
Receiver 2 1
comm: 1 M N X
comm: x S 5
BGP 65003 5
Sender A
America D
BGP communities to carry an indication of the cost
BGP Local Pref set depending on the BGP communities

22. Long-lived inter-AS loops
BGP 1
BGP 5
BGP 4
BGP 2
BGP 3
IGP
OSPF
ISIS 1
2, 1
2, 5 5
3,2,1
3,2,5
RR can cause persistent loops between BGP ASes
V. Paxson. End-to-end routing behavior in the Internet. SIGCOMM, 1996