Internet2 Engineering and Abilene Update Westnet Meetings :: Boulder Guy Almes 29 June 2001

Outline of Talk A Internet2 Engineering Objectives A Abilene History and Status A Engineering Update • Multicast, IPv6, QoS • Issues in End-to-End Performance

Internet2 Engineering Objectives A Provide our universities with superlative networking: • Performance • Functionality • Understanding A Make superlative networking strategic for university research and education

The End to End Challenge A Support advanced networking end to end A Performance • 100 Mb/s across the country normative • several multiples possible in some cases A Functionality • Multicast • Quality of Service • IPv6 • Measurements

Abilene Update

Original Abilene Objectives Apr-98 A Provide high-quality, widely available Interconnect among participating gigaPoPs/universities A Connect to Internet2 members via the vBNS and to other key research/ education sites via Internet2/NGI-class federal and non-US nets

Abilene Objectives, cont. A [Support] advanced functionality A Maximize Robustness A Minimize Latency A Provide Capacity to Avoid Congestion

Key Attributes A 12 Router Nodes • Cisco Routers • Qwest collocation A OC48 Interior Circuits connect them • Packet over Sonet in all cases A Access: 52 total • OC3, OC12, and some OC48 • via any Qwest Sonet PoPs (Access Nodes) • ATM and POS both supported

Abilene Partners A Qwest: Sonet and Collocation A Nortel: OC192 Sonet ADMs A Cisco: Routers A Indiana University: NOC

Abilene core Seattle Kansas City Denver Cleveland New York Atlanta Houston Sunnyvale Los Angeles Indianapolis Washington

The Qwest plow laying: - two conduits - one with 96 fibers - one empty

Abilene Router Nodes being assembled Summer 1998

Qwest 'right of way' PoP

60 Hudson an early carrier hotel

Abilene Connections by (roughly) summer 2001

Current Status A About 50 Connections A Growing number of OC3-to-OC12 POS upgrades • e.g., NYSERnet Buffalo • but also NoX, Univ Pennsylvania, Florida A 185 Participants in 50 States and DC • Puerto Rico soon

Sponsored Education Group Participants – Other Possible

Increasing Routing Complexity A Abilene International Transit Network • with CA*net3 and StarTap A Abilene policy on Federal Labs • differentiate Federal Labs from BB agencies

Abilene Traffic Matrix NNN N YY Federal Network NNNYYY Non-AITN Peer NN Y YYY Abilene ITN Peer Source N YY N YY Federal Lab/BB YYYY N Y Corporate Participant YYYYYY Regular Participant Federal Network Non-AITN Peer Abilene ITN Peer Federal Lab/BB Corporate Participant Regular Participant eringPeransitT inationDest

International Peering STAR TAP APAN/TransPAC, CA*net3, IUCC, RENATER, REUNA, SURFnet, SINET, TAnet2 CERnet, (HARnet) OC12 New York DANTE*, JANET, NORDUnet, SURFnet CA*net3 Seattle CA*net3, (AARnet) Sunnyvale (SINET) Los Angeles SingAREN, SINET Miami (REUNA, RNP2, RETINA) OC3-12 El Paso (CUDI) San Diego CUDI

Early NoF Planning A Commitment to ongoing needs of the Internet2 infrastructure beyond 2003  Leverage growing DWDM/fiber provisioning with many 10-Gb/s l s A Needs: • Leverage Backbone/GigaPoP/Campus structure • Serious attention to international/federal peering • Current advanced services now normative

The Houston Flood A Tropical storm Allison hit Houston hard • 26 inches in 24 hours A Abilene was effected by this • Houston Router Node went down Saturday morning • No news until mid-day Monday • "Technicians could not access this facility until late this afternoon when it was determined that the Abilene equipment there had been damaged beyond repair."

The Houston Flood, cont. A Then (!), on Tuesday morning, it came up • First, just a few circuits • Operational for most Texas/Louisiana connectors • Last circuit came up Thursday A Networking, like baseball, is a "game of inches"

Advanced Functionality A Multicast A IPv6 A QoS

Internet2 Multicast A Multicast Working Group • Kevin Almeroth, Univ California Santa Barbara, chair A Encouraging more pervasive high-quality deployment of native IP multicast throughout the Internet2 infrastructure A Fighting fires A Keeping an eye on SSM A Clarifying the application story

Internet2 Multicast Architecture A PIM-SparseMode • multicast routing within an Autonomous System • quite scalable • notion of rendezvous points A MBGP • between Autonomous Systems A MSDP • Source Discovery

Longer-term WG Issues A Scalability (what happens if it does catch on?) A Exploring the role of Source-Specific Multicast

Could SSM be Enough? A 'Classic' Multicast • Group has global significance • A user creates, joins, sends to g • Others can join, then send to and/or listen to g • MBGP, PIM-SM, MSDP triad A Source Specific Multicast • Group has local significance • A user 's' creates, sends to • Others can subscribe to, then list to • No need for MSDP (or allocation of values)

Implications of SSM A Simplify Multicast Routing / Addressing • No need for global class-D address allocation • No need for source discovery A Complicates 'few-to-few' applications • Define all the members of the application-level group • Both a burden and an opportunity A Allows better Security, Scalability A Requires new version of IGMP

Multicast Summary A Full functionality supported now A Deployment steadily increasing A Some international peering, e.g., CA*net3 A Performance excellent A Scalability? A Applications?

Internet2 IPv6 A IPv6 Working Group • Dale Finkelson, Univ Nebraska, chair A Build the Internet2 IPv6 infrastructure A Educate campus network engineers to support IPv6 A Explore the Motivation for IPv6 within the Internet2 community

IPv6 Infrastructure A vBNS and Abilene both support IPv6 A Abilene IPv6 with IPv6/IPv4 • Four 'backbone' nodes: Cisco 7200 " Atlanta, Pittsburgh, Denver, and Indianapolis • Managed by the Abilene NOC A IPv6 WG: address allocation and engineering coordination

Education / Training Goals A IPv6 hands-on workshop • Lincoln, Nebraska; 17 May 2001 • starting from scratch, build an IPv6 network, including routers, hosts, DNS tools and various transition tools, ending up with a functional IPv6 network fully interconnected to the global Internet. A Materials from this workshop will be available to enable gigaPoPs and others to use in their own workshops.

Explore IPv6 Motivation A Why should our users, campus decision- makers, and community generally care about IPv6? • we like Steve Deering • IPv6 preserves the classic end-to-end transparency of the Internet architecture • improved support for mobility • key for IPsec • key for the scalability of the Internet A The answers must be pragmatic.

Internet2 QoS A Quality of Service Working Group • Ben Teitelbaum, Internet2 staff, chair A QBone Premium Service A Scavenger Service A Architectural and ad-hoc projects

QBone Premium Service A For a given bit/second rate, minimize: • Delay and variation in delay, and • Loss A And support • Interoperability of separately designed/managed IP networks (e.g., Abilene, gigaPoP, ESnet, campus) • Interoperability of different (compliant) equipment A This is hard and very important

Abilene Premium Service A Goal • Make APS a reference implementation of QBone A Status • CAR policing + uncongested ABES A Seven participating connectors A Summer 2001: • Turn on PQ and stochastically detect illegal EF traffic via NetFlow monitoring A Fall 2001: • Engine-3-based real policing

Scavenger Service A Suppose there were a less-than-best- efforts IP service within Internet2? • users can mark their packets LBE • best-efforts traffic generally routed before LBE traffic • what bottom-feeding applications would emerge? • much easier than Premium Service

Other Abilene items A DDoS Detection Technique Development • collaboration of Asta Networks and the NOC • reflects open measurement/management stance • similar relation anticipated with Arbor Networks A High Performance Demos • Fall Member Meeting; Austin in October • SC'2001; Denver in November

Issues in End-to-End Performance

The Current Situation A Our universities have access to an infrastructure of considerable capacity • examples of 240 Mb/s flows A End-to-end performance varies widely • but 40 Mb/s flows not always predictable • users don't know what their expectations should be A Note the mismatch

What are our Aspirations?  Candidate Answer #1: Switched 100BaseT + Well-provisioned Internet2 networking ® 80 Mb/s A But user expectations and experiences vary widely

What are our Aspirations? A Candidate Answer #2: Lower user expectations and minimize complaining phone calls A There is a certain appeal I suppose...

What are our Aspirations? A Candidate Answer #3: Raise expectations, encourage aggressive use, deliver on performance/functionality to key constituencies. A Not the easy way, but necessary for success

Why should we Care? A "We" as the university community. A "We" as campus networking specialists. A "We" as networking professionals. A "We" as the (broad) Internet2 project. A Low aspirations are dangerous to us.

End to End Performance Initiative A Goal: • To create a ubiquitous, predictable, and well- supported environment in which Internet2 campus network users have routinely successful experiences in their development and use of advanced Internet applications, by focusing resources and efforts on improving performance problem detection and resolution throughout campus, regional, and national networking infrastructures.

Threats to End to End Performance A BW = C x packet-size / ( delay x sqrt(packet-loss )) (Mathis, Semke, Mahdavi, and Ott, CCR, July 1997) A Context: • Network capacity • Geographical distance • Aggressive application

Threats to End to End Performance A Fiber problems • dirty fiber • dim lighting • 'not quite right' connectors

Threats to End to End Performance A Fiber problems A Switches • horsepower • full vs half-duplex • head-of-line blocking

Threats to End to End Performance A Fiber problems A Switches A Inadvertently stingy provisioning • mostly communication • happens also in international settings

Threats to End to End Performance A Fiber problems A Switches A Inadvertently stingy provisioning A Wrong Routing • asymmetric • best use of Internet2 • distance

Threats to End to End Performance A Fiber problems A Switches A Inadvertently stingy provisioning A Wrong Routing A Host issues • NIC • OS / TCP stack • CPU

Perverse Result A 'Users' think the network is congested or that the Internet2 infrastructure cannot help them A 'Planners' think the network is underutilized, no further investment needed, or that users don't need high performance networks

Promising Approaches A Work with key motivated users A 'Shining a flashlight' on the problem A Measurements A Divide-and-Conquer A Understanding Application Behavior A Getting it right the first time

Internet2 End-to-End Performance Initiative A Very recently hired / deployed staff • Cheryl Munn-Fremon, initiative director • Russ Hobby, chief technical architect • George Brett, chief information architect A $1.5M budgeted by Internet2

Internet2 End-to-End Performance Initiative A Distributed measurement infrastructure • Enable rapid effective understanding of why an instance of end-to-end performance is limited • Make the work of PERT members rewarding • Enable initiation of tests by PERT members A Teams of performance analysis specialists (PERTs) A Dissemination of best practices

Internet2 End-to-End Performance Initiative A Distributed measurement infrastructure A Teams of performance analysis specialists (PERTs) • members at campuses, gigaPoPs, backbones • socially and technically coordinated • committed to effecting radical change A Dissemination of best practices

Internet2 End-to-End Performance Initiative A Distributed measurement infrastructure A Teams of performance analysis specialists (PERTs) A Dissemination of best practices • Identify key techniques, tools, and 'best practices' • Make them common • Work toward widespread / routine excellent user experiences • Improve the reputation / status of network engineers

Anticipated Partners A NLANR: DAST, MOAT, and NCNE A Web100 Project A Abilene partners A Leading campuses and gigaPoPs A Internet2 corporate members

Internet2 Measurements A Measurement Working Group • Matt Zekauskas, Internet2 Staff A Define architecture: • Usage • Active Measurements of Performance • Passive Measurements A Uniform Access to Results A Contributing to Measurement Infrastructure for the E2EPerf

Applications for Measurements A End-to-end Performance Debugging A Verification of QoS Performance Characteristics A Support for Operations A Forward engineering of new infrastructure A Supporting research, e.g., by university computer scientists

Active Measurements within Abilene Surveyors with: Active delay/loss measurements Ad hoc throughput tests

Application to Performance Debugging

Divide and Conquer A Systematically identify/isolate the network segment at fault A Can we make this systematic and (eventually) automated?

Access to Key Resources A Optical telescopes in Hawaii A CRAFT Project A PACI Supercomputer Facilities A CERN

Working Groups as Opportunities A We intend the WGs to be effective as: • means for interested engineers to 'sink their teeth into' hard Internet2 engineering problems • means for disseminating best practices etc to the Internet2 membership A New Engineering Area of Internet2 web site due up by 14-Feb-01

Internet2 and Stephen F Austin A Can we defeat distance as a barrier to: • human collaboration? • effective access to key instruments / data sources? A For very large research universities, this is somewhat important, but it is key for smaller ones!

Applications Communities A General notion: distributed sets of researchers who collaborate at a distance • High Energy Physics (CERN, MIT, Caltech) • Space Physics & Aeronomy Research Collaboratory • Geospatial Information Systems community A These groups explore why advanced Internet2 infrastructure is important