1. Practical Searches for Stability in iBGP
Ashley Flavel Matthew Roughan Nigel Bean Aman Shaikh

3. Routing Protocol
A distributed mechanism to determine the best route to a destination.
Routers propagate their own best route to their neighbors.
Best route may not be the shortest-path
Non-shortest path routing protocols can oscillate!
Routers persistently altering their decision in response to one-another.
Nemesis to stability
Waldo’s nemesis is Odlaw

4. What is Routing Oscillation?
Preference to
destination 1 2

5. What is Routing Oscillation?
Preference to
destination 1 2

6. What is Routing Oscillation?
Preference to
destination 1 2

7. What is Routing Oscillation?
Preference to
destination 1 2

8. What is Routing Oscillation?
Preference to
destination 1 2

9. What is Routing Oscillation?
Preference to
destination 1 2

10. What is Routing Oscillation?
Preference to
destination 1 2

11. What is Routing Oscillation?
Preference to
destination 1 2

12. What is Routing Oscillation?
Preference to
destination 1 2

13. What is Routing Oscillation?
Preference to
destination
Same as earlier step! 1 2

14. Motivation Why is oscillation bad?
Network reliability is impossible
Network predictability is impossible
Debugging a network is impossible
Degrades router performance
Routing oscillation should never occur!
Full control over network
Internal Routing Oscillation Causes…..
MED
iBGP topology
Network Reliability is impossible
Case where there is no oscillaiton – we prove there is none!

15. Motivation Oscillation does occur in practice!
How do we detect oscillations?
What are the causes of oscillation inside an AS?
MED oscillation
Filtering or resetting this attribute prevents this form of oscillation
iBGP topology oscillation
Caused by the interaction between iBGP and the IGP.
Internal Routing Oscillation Causes…..
MED
iBGP topology
Network Reliability is impossible
Case where there is no oscillaiton – we prove there is none!

16. iBGP Topology Oscillation
iBGP and IGP run on different topologies!
iBGP is overlaid on physical topology
iBGP used to determine route to external destination
IGP used to determine route to internal destination
They interact
Hot-potato routing
Lead to oscillation

17. BGP Decision Process
Each router’s decision is independent and based on its set of available routes
Set of available routes depends on iBGP topology
1. Highest Local-Preference
2. Shortest AS Path
3. Best Origin
4. Lowest MED
5. Shortest IGP distance to egress router (hot-potato)
6. Tie-break
AS-wide steps
Individual router steps

18. Our Approach
We create an abstract model of the interaction between iBGP and the IGP.
Prove oscillatory properties of the abstract model
Localize where oscillation can occur.
Where a configuration is not oscillatory we mathematically prove this is the case.

19. iBGP Route Reflection
Used in preference to full-mesh of iBGP sessions in large networks.
Due to scalability concerns
Two classes of routers
Clients
Route-reflectors
Multi-level hierarchy possible
We focus our attention to two levels

20. iBGP Route Reflection
Clients propagate externally learned routes to parents 1 2 3 8 9 10 4 5 6 7 4
Route-reflector (RR)
Client
Client-parent
iBGP sesssion
Pink, Blue and Green are a single destination.
Not all routers have to make the same decision.
RR-RR
iBGP sesssion

21. iBGP Route Reflection
Clients propagate externally learned routes to parents 1 2 3 8 9 10 4 5 6 7 4
Route-reflector (RR)
Client
Client-parent
iBGP sesssion
RR-RR
iBGP sesssion

22. iBGP Route Reflection
Clients propagate externally learned routes to parents 1 2 3 8 9 10 4 5 6 7 4
Route-reflector (RR)
Client
Client-parent
iBGP sesssion
RR-RR
iBGP sesssion

23. iBGP Route Reflection
Clients propagate externally learned routes to parents 1 2 3 8 9 10 4 5 6 7 4
Route-reflector (RR)
Client
Client-parent
iBGP sesssion
RR-RR
iBGP sesssion

24. iBGP Route Reflection
Route-reflectors propagate client routes to all other iBGP neighbors. 1 2 3 8 9 10 4 5 6 7 4
Route-reflector (RR)
Client
Client-parent
iBGP sesssion
RR-RR
iBGP sesssion

25. iBGP Route Reflection
Route-reflectors propagate client routes to all other iBGP neighbors. 1 2 3 8 9 10 4 5 6 7 4
Route-reflector (RR)
Client
Client-parent
iBGP sesssion
RR-RR
iBGP sesssion

26. iBGP Route Reflection
Route-reflectors propagate client routes to all other iBGP neighbors. 1 2 3 8 9 10 4 5 6 7 4
Route-reflector (RR)
Client
Client-parent
iBGP sesssion
RR-RR
iBGP sesssion

27. iBGP Route Reflection
Route-reflectors reflect routes learned from other route-reflectors to clients 1 2 3 8 9 10 4 5 6 7 4
Route-reflector (RR)
Client
Client-parent
iBGP sesssion
RR-RR
iBGP sesssion

28. iBGP Route Reflection
Route-reflectors reflect routes learned from other route-reflectors to clients 1 2 3 8 9 10 4 5 6 7 4
Route-reflector (RR)
Client
Client-parent
iBGP sesssion
RR-RR
iBGP sesssion

29. Reliance Graph What happens when routers’ decisions oscillate?
Multiple routers persistently alter their decision in response to the decision of others
If a router can learn of its best route from another router, then it is reliant on it
The reliance graph captures all possible reliances.
Earlier we have refered to ‘relies on’
REMOVE THE WHERE CAN A ROUTER learn

30. Reliance Graph Basics
A reliance graph is on a per egress instance basis
Set of routers which have a direct egress
Equally attractive over AS-wide decision steps
A directed edge in the reliance graph exists if a router’s decision is reliant on another.
Oscillation requires a cycle in the reliance graph 1 2 3 4 5
Get rid of downstream egress set blob
EGRESS INS 6 7 8 9 10

31. Reliance Graph
All iBGP sessions can be bi-directional edges in reliance graph.
We can prune most of these edges!
iBGP pruning
Some neighbors will never propagate a route
Based on route-reflection
IGP pruning
Some neighbors will never propagate a best route
Based on IGP distances 1 2 3 4 5
PUT IN PRUNING HERE! 6 7 8 9 10

32. iBGP Pruning Routers in the egress instance Will select its own route
Not reliant on any other router 1 2 3 4 5 6 7 8 9 10

33. iBGP Pruning Routers in the egress instance
Will select its own route
Not reliant on any other router
Clients can only select a route learned from their parent.
Do not propagate any information to their parents 1 2 3 4 5
First where’s wally picture here 6 7 8 9 10

34. iBGP Pruning
A route-reflector can only be reliant on another route-reflector if that route-reflector has a client in the egress instance. 1 2 3 4 5
Successively cut the blob
Change colours of input routes 6 7 8 9 10

35. iBGP Pruning
A route-reflector can only be reliant on another route-reflector if that route-reflector has a client in the egress instance.
We can prune further using IGP distances
Route-reflectors only reliance on their ‘best’ client 1 2 3 4 5
Successively cut the blob
Change colours of input routes 6 7 8 9 10

36. Co-reliance groups
We partition the reliance graph into strongly connected components
Termed co-reliance groups
Oscillation can only occur within a non-singleton co-reliance group.
The only possible location for a non-singleton co-reliance group
route-reflectors with clients in the egress instance 1 2 3 4 5
Successively cut the blob
Change colours of input routes 6 7 8 9 10

37. Split picture

38. Will a co-reliance group oscillate?
Maybe.
Not everything in a striped sweater is Waldo
Analyze each co-reliance group independently
Oscillation can only occur within a co-reliance group.
Not everything in a striped jumper is Waldo
Just because a group is a co-reliance group – doesn’t mean that it is bad.
Put in odlaw in pic

39. Co-reliance group algebra
A co-reliance group only contains route-reflectors with clients in the egress instance
Each route-reflector has egresses which it can learn via
a client (direct route), and
another route-reflector (indirect)
Irrelevant which indirect route is chosen
Each router in co-reliance group only has two route choices!
Direct (d) or Indirect (i)
PICTURE – NODES 0-5
Each rout-rflector in co-reliance group can have two choices
Each router has two choices
Select its client
Select another routers client
Explain the choices 1 3 4 5

40. Example Labels: direct (d), indirect (i)
Preference Relationship: i > d
Outbound Arc Labels: i if node is d, nothing if node is i d d
Get rid of blue
Get rid of ring 4 5

41. Example Labels: direct (d), indirect (i)
Preference Relationship: i > d
Outbound Arc Labels: i if node is d, nothing if node is i d d
Get rid of blue
Get rid of ring d 4 5 d

42. Example Labels: direct (d), indirect (i)
Preference Relationship: i > d
Outbound Arc Labels: i if node is d, nothing if node is i d d
Get rid of blue
Get rid of ring d 4 5 d i

43. Example Labels: direct (d), indirect (i)
Preference Relationship: i > d
Outbound Arc Labels: i if node is d, nothing if node is i d d
Get rid of blue
Get rid of ring d 4 5 i i

44. Example Labels: direct (d), indirect (i)
Preference Relationship: i > d
Outbound Arc Labels: i if node is d, nothing if node is i
One stable solution d d
Get rid of blue
Get rid of ring d 4 5 i i

45. Example Alternative stable solution
If all possible message orderings result in the group settling to a solution - the system is stable. d i d
Get rid of blue
Get rid of ring i 4 5 d

46. State Machine dd ii di id d i d i 4 5 d

47. Three Node Co-reliance Group
Will this co-reliance group oscillate?
Show state diagram 1 2 3

48. State Machine
ddd
idd
idi
ddi
Show state diagram
iid
did
dii
iii

49. How many reliance graphs?
So far we have focused on a single egress instance
Analyzing all egress instances results in a combinatorial explosion.
Some border routers never used in combination.
Filters on border routers
We can further scope the problem
Prioritize the checking of current egress instances

50. Practical Analysis Based on topology and routing state of a Tier 2 AS
Approx 500 routers.
954 egress instances found (maximum number of egress routers = 17).
Power set of these egresses raised the number of egress instances to 204,621
Results
Stable
Under 15 minutes to run
Further optimizations possible
Can run in parallel
Can reduce co-reliance groups to equivalent forms
Can keep a library of co-reliance groups
Topology and routing of a tier 2 AS
Look at al possible scenarios we need to analyze
Get rid of part
No co-reliance groups
Hence Stable
Build up a library
Collapsing Co-reliacne groups
Omit ‘no co-reliance groups present’
Without any optimization
Parallelizable
Computer based proof that this network does not oscillate
Not a simulation

51. What does this mean?
Proof a large practical iBGP configuration’s stability.
Not a simulation.
Can determine the oscillatory properties of
configuration changes prior to their implementation
the current network state
the network as it evolves (e.g. due to failures)
Guarantee the stability of a configuration.
Pinpoint the routers responsible for oscillatory modes.
Topology and routing of a tier 2 AS
Look at al possible scenarios we need to analyze
Get rid of part
No co-reliance groups
Hence Stable
Build up a library
Collapsing Co-reliacne groups
Omit ‘no co-reliance groups present’
Without any optimization
Parallelizable
Computer based proof that this network does not oscillate
Not a simulation

52. Conclusion and Future Work
We can determine the oscillatory properties of a network configuration.
Fast
Scalable for realistic networks
It is a proof of stability or you find where the oscillation is.
Future Work
How do you fix oscillation?

53. Where’s Waldo? Longer talk – spend logner on examples More background
More connection with algebraic approach
High level stuff
Change the abstract to be interesting