2. 1 BGP Convergence Measurement Issues Susan Hares, NextHop Padma Krishnaswamy, NextHop Marianne Lepp, Juniper Networks Alvaro Retana, Cisco Howard Berkowitz, Gett Communications Elwyn Davies, Nortel Networks

3. 2 AS Convergence? Tester Internet-wide AS FlappingAS He's Dead, Jim

4. 3 Single AS Convergence: Within an AS R R R Tester AS-wide iBGP

5. 4 Convergence: Within a Box Single Box Tester Routing/Control Forwarding

6. 5 Convergence for BMWG Box wide —eBGP initially —Control Plane initially —Black box —Specify begin and end of convergence measurement —Specify measurement point

7. 6 Send a packet stream from TR – 3 Measurements Convergence 1: 1 st packet sent from Test Generator to 1 st packet received by Test Collector —Transmission in and out plus process time of 1 st packet Convergence 2: Last Packet sent from Test Generator to last packet received by Test Collector —Transmission in, queuing, processing of preceding updates, tail end processing, transmission out of last packet Convergence 3: 1 st packet sent from Test Generator to last packet received by Test Collector —Transmission in and out (relative to DUT), plus back-up in BGP update and processing of entire stream

8. 7 Measurement 1-3: Factors Packing matters —Influences number of packets in the train —Attribute packing –Classification speed –Packetization triggers IBGP synchronization turned off Turn off Minimum Route Advertisement Interval Timers Smoothing in BGP to avoid self-synchronization in the Network

9. 8 BGP Convergence Depends On … Route mixtures Packet packing Timers TCP implementations Peers types, number of peers, and connectivity BGP-specific functionality —Eg. Confederations, use of route reflectors, etc. Topology Vantage point within the network Policy

10. 9 Benchmarking Convergence Approach Must be repeatable Must be consistent Must be specifiable Must take into account —Route mixture (data) —Peers types and connectivity —BGP-specific functionality —Topology

11. 10 Goals Provide a baseline of expected performance in today’s network. Test different vendor implementations fairly Design tests that can be replicated Good results require good data —The amount, type and composition of the information advertised to the DUT has an impact on the convergence.

12. 11 Route Mixtures

13. 12 Modeling Route Mixtures: Why not just use a feed? The route mixture is highly dependent on the vantage point – Tier 1 ISP, Enterprise, Access, etc. Problems with Looking Glass —Vantage point Need to test tables larger than current live tables Needs to be repeatable, consistent, and specifiable

14. 13 Route Mixture Factors that describe the BGP Table: composition and timing —Prefix distribution –Node distribution and levels on tree —AS mixtures and path lengths —Attribute distribution (nexthop, communities,MED, localpref) —Packet packing —Update sequencing (timing) –Packet trains

15. 14 Prefix Distribution Example: A table of all /32s is not representative of the real world Prefixes are distributed across dozens of prefix lengths —For IPv4, the distribution is spread out through the Class A, B, and C address spaces. —For IPv6, there is no data Need to describe prefix distribution per prefix length —Better characterization for IPv4 if Class also taken into account Analyze current Internet table to determine prefix distribution characteristics

16. 15 Prefix Distribution Example percentages of prefix distribution: Mask Overall Class A Class B Class C 16 0.08114 0.00076 0.06637 0.01401 17 0.00912 0.00030 0.00142 0.00741 18 0.01813 0.00093 0.00113 0.01607 19 0.05910 0.00378 0.00196 0.05336 20 0.03372 0.00152 0.00151 0.03070 21 0.04128 0.00085 0.00127 0.03915 22 0.05574 0.00171 0.00226 0.05176 23 0.07878 0.00235 0.00450 0.07193 24 0.53355 0.00892 0.02366 0.50097 Total prefix length distribution. IPv4 sample distribution across classes.

17. 16 IP v6 Prefix Distribution Example percentages of prefix distribution: Mask Overall 3FEE 2001 other 0-10 0.08114 0.00076 0.06637 0.01401 11-20 0.00912 0.00030 0.00142 0.00741 21-30 0.01813 0.00093 0.00113 0.01607 31-40 0.05910 0.00378 0.00196 0.05336 41-50 0.03372 0.00152 0.00151 0.03070 51-60 0.04128 0.00085 0.00127 0.03915 61-70 0.05574 0.00171 0.00226 0.05176 71-80 0.07878 0.00235 0.00450 0.07193 81-90 0.53355 0.00892 0.02366 0.50097 91-100 0.53355 0.00892 0.02366 0.50097 100-110 0.53355 0.00892 0.02366 0.50097 111-128 0.53355 0.00892 0.02366 0.50097 Total prefix length distribution. IPv6 sample distribution across currently routed Addres space

18. 17 Node Distribution Is tree dependent Width and depth of table are important Route mixtures should exercise various choices of trees —A route mixture that minimizes the number of nodes is not accurate —A route mixture that maximizes the spread of prefixes creates is not accurate

19. 18 Node Distribution Levels Nodes

20. 19 IP v6 Node Distribution ROOT 3FEE:: 2001::012001:02 2001:0201 2001:: 3FEE:0101 3FEE:0100 3FEE:2000:: Levels Nodes 3FEE:0101:01 3FEE:0101 2001:0201:01 2001:0201:02

21. 20 Node Distribution For example, the following tables both contain three Class A /32 prefixes —Table A 1.1.1.1/32, 1.1.1.2/32, 1.1.1.3/32 —Table B 1.1.1.1/32, 2.1.1.1/32, 3.1.1.1/32 —Their distribution in a tree will be different. –Table A represents a narrow distribution, while Table B represents a wide distribution. 1.0.0.0 1.1.0.0 1.1.1.2 1.1.1.0 1.1.1.21.1.1.1 ROOT Table A 1.0.0.0 1.1.1.1 1.1.1.0 1.1.0.0 2.0.0.0 2.1.1.1 2.1.1.0 2.1.0.0 3.0.0.0 3.1.1.1 3.1.1.0 3.1.0.0 ROOT Table B

22. 21 Node Distribution Summary The width of the table must be measured per prefix distribution and length Need to determine how many nodes each address/prefix length combination use in a real table Solution: Analyze current Internet table to determine node distribution characteristics

23. 22 Route Components

24. 23 BGP Attribute Distribution A BGP table contains many “attribute combinations” Analysis shows: —11.75% of the routes have a unique AS_PATH —2.5% of the routes have some other unique attribute. —0.25% of the table have both a unique AS_PATH and some other unique attribute

25. 24 BGP Attribute Distribution Prefixes that share an attribute are not necessarily grouped together Analysis shows an average of two consecutive NLRI share the same attribute combination —1.0.0.0/8AS_PATH 100 200 —2.0.0.0/8AS_PATH 100 200 —3.0.0.0/8AS_PATH 200 300 —4.0.0.0/8AS_PATH 200 300 —5.0.0.0/8 AS_PATH 200 300 —6.0.0.0/8 AS_PATH 100 200

26. 25 Planes (control), Trains, and no Automobiles

27. 26 Packet Packing Each packet has attributes and NLRIs Attribute packing is the ability to detect and pack NRLIs with the same attributes into a packet NLRI packing is: — the number of NLRIs per packet —MPBGP not considered for 1 st draft –IPv6 packing is not different than IPv4 –Multicast packing (IP v4 and IP v6) may impact packing Specifics are affected by implementation

28. 27 Update Sequencing (Timing) Parameters are: —Number of packets in a train —Interval between packets in a train —TCP parameters, traffic and implementations affect this Packet 1Packet 2Packet 3Packet 4 Packet train

29. 28 Timers Key timers —Min-Route Advertisement Interval, Min AS Originations Interval -- best setting still in debate —Route Flap damping mechanisms –Implementations vary –Shorter prefixes get less damping –RIPE 229 suggest parameters –1 st Bgp Conv draft mandates route flap damping off —TCP settings Operators need to give feedback

30. 29 Peers, not Beers

31. 30 Peer type matters EBGP vs IBGP EBGP —3 rd party versus 1 st party nexthop —promiscuous versus specific peering IBGP - Route Reflection client and Confederations affect convergence patterns —See ietf-idr-route-oscillations-01.txt Still single box but these affect work done by box

32. 31 Multiple Peers in test Environment Peers can have staggered starts —Most realistic Peers can all send simultaneously —Most load on the router Peers can have staggered starts in groups

33. 32 Sample topology with 4 Peers TG1 DUT TC tcpdump TG2 TG3 TG4 tcpdump

34. 33 Peer Specifics Type of Peer —Promiscuous/Specific Sequence —Connection establishment —Sending 1 st data —Spacing of updates —Connection up/down

35. 34 Timing & Synchronization Consistency among timestamps taken by different devices is a requirement Should be at least 1 order of magnitude better than measured quantity —For BGP convergence, we are time-stamping packets NTP? GPS? Other? Synchronization between measurements can a significant factor

36. 35 Some Boxes work Harder than Others

37. 36 BGP Protocol functions will impact convergence Route Reflections, Confederations Add/delete communities RFC 2547, Label switching Multi-protocol Route flap damping Min Route Advertisement

38. 37 Parameters we suggest for Protocol Functions for 1 st Document No Route Reflectors (no IBGP this version) No confederations No Add/Delete communities No 2547 VPNS or multicast Route flap damping OFF Min Route Advertisement Interval specified Min AS Origination Interval specified

39. 38 Topology

40. 39 Topology matters Exchange point topology N star topologies meshed for Route Reflection Confederations with particular topologies IBGP/EBGP mesh overlay Building blocks —single link, line, mesh, partial mesh, star, wheel

41. 40 Single link: 1 st Document …… DUT TRes TR1 TR2 TRn n >= 1 line

42. 41 Line DUT TRes TR1 Longer line

43. 42 Mesh DUT TResTR1 DUT TRes mesh

44. 43 Partial Mesh DUT TResTR1 DUT TRes

45. 44 References IETF51 BMWG talk: http://www.ietf.org/proceedings/01aug/slides/bmwg-4/ http://www.ietf.org/proceedings/01aug/slides/bmwg-4/ NextHop IETF51 talk: http://www.ietf.org/proceedings/01aug/slides/bmwg- 5/index.html http://www.ietf.org/proceedings/01aug/slides/bmwg- 5/index.html Howard’s IETF51 talk: http://www.ietf.org/proceedings/01aug/slides/bmwg- 6/index.html http://www.ietf.org/proceedings/01aug/slides/bmwg- 6/index.html Recommendations for flap damping, Ripe 229: http://www.ripe.net/ripe/docs/ripe-229.html http://www.ripe.net/ripe/docs/ripe-229.html BGP Convergence Terminology ID: http://www.ietf.org/internet-drafts/draft-ietf-bmwg- conterm-00.txt http://www.ietf.org/internet-drafts/draft-ietf-bmwg- conterm-00.txt BGP Convergence Methodology: http://www.ietf.org/internet- drafts/draft-ietf-bmwg-bgpbas-00.txthttp://www.ietf.org/internet- drafts/draft-ietf-bmwg-bgpbas-00.txt

46. 45 Thank You Questions?

47. 46 Route Mixtures Matter! The amount, type and composition of the information advertised to the DUT has an impact on the convergence. Goal is to provide a baseline of expected performance in today’s network. Test different vendor implementations fairly Design tests that can be replicated Good results require good data