1. Abilene and Internet2 Engineering Update Guy Almes Terena Networking Conference 2002 Limerick, Ireland Guy Almes Terena Networking Conference 2002 Limerick, Ireland

2. 6 May 2002 2 Outline Abilene Update Engineering Update »Multicast »IPv6 »QoS »End-to-End Measurements »Transport for Bulk Data Flows

3. 6 May 2002 3 Internet2 Engineering Objectives Provide our universities with superlative networking: »Performance »Functionality »Understanding Make superlative networking strategic for university research and education

4. 6 May 2002 4 Abilene Update Current 2.5 Gb/s Abilene Network Plans for 10 Gb/s Upgrade

5. 6 May 2002 5 Abilene is a Partnership To build/operate Abilene, Internet2 partners with: »Cisco Systems (routers, switches, and access) »Juniper Networks (routers) »Nortel Networks (SONET kit) »Qwest Communications (circuits and collocation) »Indiana University (network operations center) »Internet2 Test & Evaluation Centers (ITECs) –North Carolina –Ohio

6. 6 May 2002 6 Current Abilene Status IP-over-SONET backbone (2.5 Gb/s) »53 direct connections »4 2.5 Gb/s connections »1 Gigabit Ethernet trial »23 will connect via at least 622 Mb/s by 1Q02 »Number of ATM-based connections decreasing 215 participants: universities and labs »All 50 states, District of Columbia, and Puerto Rico »15 regional gigaPoPs support ~70% of participants Expanded access »50 sponsored participants »23 state education networks (SEGPs)

7. 6 May 2002 7 Abilene international connectivity Transoceanic R&E bandwidths growing !? »GÉANT: 5 Gb/s between Europe and New York City Key international exchange points: »StarTap and StarLight: Chicago (GigE) »AmPath: Miami (155 Mb/s ATM) »Pacific Wave: Seattle (GigE) »MAN LAN: New York City (GigE/10GigE planned) »CA*net3/4: Seattle, Chicago, and New York »CUDI: CENIC and Univ Texas El Paso International transit service »Collaboration with CA*net3 and StarTap

8. Sacramento Los Angeles Washington Abilene International Peering STAR TAP/Star Light APAN/TransPAC, Ca*net3, CERN, CERnet, FASTnet, GEMnet, IUCC, KOREN/KREONET2, NORDUnet, RNP2, SURFnet, SingAREN, TAnet2 NYCM BELNET, CA*net3, GEANT*, HEANET, JANET, NORDUnet Pacific Wave AARNET, APAN/TransPAC, CA*net3, TANET2 SNVA GEMNET, SINET, SingAREN, WIDE LOSA UNINET AMPATH REUNA, RNP2 RETINA, ANSP, (CRNet) OC3-OC12 El Paso (UACJ-UT El Paso) CUDI San Diego (CALREN2) CUDI * ARNES, CARNET, CESnet, DFN, GRNET, RENATER, RESTENA, SWITCH, HUNGARNET, GARR-B, POL-34, RCST, RedIRIS 09 March 2002

9. 6 May 2002 9 Packetized Raw HDTV Raw HDTV/IP: single 1.5 Gb/s UDP flow DARPA-funded project of USC/ISIe, Tektronix, and Univ Washington »6 Jan 2002: Seattle to Washington DC via Abilene »18 hours: no packets lost, 15 resequencing episodes »End-to-end network performance (includes P/NW & MAX) – Loss: <0.8 ppb (90% c.l.) – Reordering: 5 ppb »Transcontinental 1-Gb/s TCP requires loss of – <30 ppb (1.5 KB frames) – <1 ppm (9 KB jumbo)

10. 6 May 2002 10 End-to-End Performance: High bandwidth is not enough Bulk TCP flows »Current median flow over Abilene: 1.9 Mb/s –95th percentile: 7.0 Mb/s

11. 6 May 2002 11 Future of Abilene Internet2/Qwest agreement amended, and extended to Oct-06 Upgrade now underway to shift »from OC-48c (using Nortel OC-192 Sonet) »to 10-Gb/s lambda (unprotected) »x4 increase in core backbone bandwidth

13. 6 May 2002 13 Next generation Abilene router selection Extensive router specification and testing Tests focused on next gen advanced services »High performance TCP/IP throughput »High performance multicast »IPv6 functionality and throughput »Classification for QoS and measurement 3 router platforms tested and commercial ISPs referenced Juniper T640 platform selected

14. 6 May 2002 14 Deployment timing Ongoing: Backbone router procurement, detailed deployment planning July: Rack assembly (Indiana Univ.) Aug/Sep: New rack deployment Fall: First Wave lambdas commissioned Fall meeting demonstration events »Internet2 Fall Member Meeting (Los Angeles): late Oct. »SC2002 (Baltimore): mid Nov. 2003: Remaining lambdas commissioned

15. 6 May 2002 15 Key next-gen Abilene emphases: I Native IPv6 »Motivations –Resolving IPv4 address exhaustion issues –Preserving original End-to-End Architecture model –International collaboration –Router and host OS capabilities »Run native IPv6, concurrent with IPv4 »Replicate Abilenes 1999 multicast deployment strategy »Close collaboration with Internet2 IPv6 Working Group

16. 6 May 2002 16 Key next-gen Abilene emphases: II Network resiliency »Abilene lambdas will not be ring protected »Increasing use of videoconferencing/VoIP impose tighter restoration requirements (<100 ms) »Options: –MPLS/TE fast reroute (initially) –IP-based IGP fast convergence (preferable)

17. 6 May 2002 17 Key next-gen Abilene emphases: III Deeper measurement capabilities »Significant factor in NGA rack design –4 dedicated servers at each nodes –Additional provisions for future servers –Local data collection to capture data at times of network instability »Enhance active probing –Now: Latency and jitter, loss, reachability (Surveyor) –Regular TCP/UDP throughput tests: ~1 Gbps Separate server for E2E performance beacon »Enhance passive measurement –Now: SNMP (NOC) and traffic matrix/type (Netflow) –Routing (BGP and IGP) –Optical splitter taps on backbone links at select location(s)

18. 6 May 2002 18 Abilene Observatories Currently a sketch of a program for better support of computer science research 1) Improved and accessible data archive »Need coherent database design »Unify & correlate 4 separate data types –SNMP, active measurement data, routing, Netflow 2) Provision for direct network measurement and experimentation »Resources reserved for two additional servers –Power (DC), rack space (2RU), router uplink ports (GigE) »archive/measurement/experiment

19. 6 May 2002 19 Abilene Upgrade Summary Backbone upgrade project underway »Partnership with Qwest extended through 2006 »Juniper T640 routers selected for backbone »10-Gb/s backbone lambda deployment starts this fall Advanced service foci »Native, high-performance IPv6 »Enhanced, differentiated measurement »Network resiliency Incremental, non-disruptive transition

20. 6 May 2002 20 Multicast By 1998, »Routing protocols existed »Deployment of native IP multicast quite rare »Early MBone no longer scalable »Considered key to new conferencing and streaming applications Current native multicast support »PIM-Sparse, MBGP, and MSDP Emphases on »Deployment and support for operations »Applications »Working to make it scalable

21. 6 May 2002 21

22. 6 May 2002 22 Current Multicast Emphases Pressing ahead on Deployment »What are the current inhibitors to progress? Applications / Content »Make it useful for your campuses »Explore the role of multicast in the future Internet Improve Scalability »Press deployment of SSM »Explore the role of SSM

23. 6 May 2002 23 Could SSM be Enough? 'Classic' Multicast »Group has global significance »A user creates, joins, sends to g »Others can join, then send to and/or listen to g »MBGP, PIM-SM, MSDP triad Source Specific Multicast »Group has local significance »A user 's' creates, sends to »Others can subscribe to, then listen to »No need for MSDP (or allocation of values)

24. 6 May 2002 24 Implications of SSM Simplify Multicast Routing / Addressing »No need for global class-D address allocation »No need for source discovery Complicates 'few-to-few' applications »Define all the members of the application-level group »Both a burden and an opportunity Allows better Security, Scalability Requires new version of IGMP

25. 6 May 2002 25 IPv6 Clarify motivation for IPv6 »End-to-end transparency and global addressability »Supports application innovation, e.g., peer-to-peer Support deployment and engineering expertise on networks, especially on campus Anticipate need for first-class support »E.g., 10 Gb/s Abilene upgrade »E.g., Linux, Windows XP

26. 6 May 2002 26

27. 6 May 2002 27 Current IPv6 Emphases IPv6 Training Workshops »About 8-10 workshops this year »First: in Los Angeles, hosted by CENIC, in February Get some IPv6 on each campus/gigaPoP Prepare for native peering »Abilene to gigaPoP »gigaPoP to campus »continue within campuses to key departmental LANs Explore applications, DNS, operational stability

28. 6 May 2002 28 QoS Premium Service Retrospective »Inter-AS Premium Service proved too ambitious »Too great a demand on all routers to be able to police and to shape »Too great a demand on ability of peering networks to coordinate Implicit claims »Over-provisioning plus removal of non-congestive loss »Adaptive applications

29. 6 May 2002 29 Non-Elevated Services Scavenger »Less than best effort »Easy to deploy »Applications: –Massive file transfers –Marking non-performance-sensitive applications Alternative Best Efforts »Active area of research »Avoid gaming by users, while avoiding need for policing

30. 6 May 2002 30 Localized Elevated Services Nature of Congestion as a threat »Less on national/international backbones »More at hard-to-upgrade local/metro networks Decentralized Experimentation Coordinates with some Scavenger deployment Active area of study by the working group

31. 6 May 2002 31 The Current Situation Our universities have access to an infrastructure of considerable capacity »examples of multi-hour 1.6 Gb/s flows with no loss and very little reordering End-to-end performance varies widely »but 40 Mb/s flows not always predictable »users don't know what their expectations should be A well-known mismatch

32. 6 May 2002 32 What are our Aspirations? Candidate Answer #1: Switched 100BaseT + Well-provisioned Internet2 networking at 80 Mb/s But user expectations and experiences vary widely

33. 6 May 2002 33 What are our Aspirations? Candidate Answer #2: Lower user expectations and minimize complaining phone calls There is a certain appeal I suppose...

34. 6 May 2002 34 What are our Aspirations? Candidate Answer #3: Raise expectations, encourage aggressive use, deliver on performance/functionality to key constituencies. Not the easy way, but necessary for success

35. 6 May 2002 35 Threats to End to End Performance Fiber problems »dirty fiber »dim lighting »'not quite right' connectors

36. 6 May 2002 36 Threats to End to End Performance Fiber problems Switches »horsepower »full vs half-duplex »head-of-line blocking

37. 6 May 2002 37 Threats to End to End Performance Fiber problems Switches Inadvertently stingy provisioning »mostly communication »happens also in international settings

38. 6 May 2002 38 Threats to End to End Performance Fiber problems Switches Inadvertently stingy provisioning Wrong Routing »asymmetric »best use of Internet2 »distance

39. 6 May 2002 39 Threats to End to End Performance Fiber problems Switches Inadvertently stingy provisioning Wrong Routing Host issues »NIC »OS / TCP stack »CPU

40. 6 May 2002 40 Perverse Result 'Users' think the network is congested or that the Internet2 infrastructure cannot help them 'Planners' think the network is underutilized, no further investment needed, or that users don't need high performance networks

41. 6 May 2002 41 Transport Protocol Issues Improved TCP Implementation »Web100 Project »SACK, Window Scaling »ECN But, still subject to fundamental limits »Mathis et al. Theoretical result »TCP-throughput = (C x MTU) / (RTT x sqrt(loss)) »Prospects for raising MTU, reducing RTT, loss

42. 6 May 2002 42 Need to supplement TCP Bulk Data Transfers »Absolute need to include congestion control »But include flow control in a more aggressive way Related Ideas »Transport-level gateways? »Known-Capacity pipes vs Groping for available Capacity

43. www.internet2.edu