1 Status Report on US networks at the Turn of the Century Les Cottrell – SLAC & Stanford U. Presented at CHEP00, Padua Italy, February 9, 2000 Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP

2 Overview U.S. Networks: –Internet 2Internet 2 –Federal networks, in particular ESnetFederal networks, in particular ESnet –CommercialCommercial Performance seen from U.S.: –Compare Internet2 vs. ESnet vs CommercialCompare Internet2 vs. ESnet vs Commercial –Performance to Europe and rest of worldPerformance to Europe rest of world –TrendsTrends Summary

3 US National Networks Internet 2 - universities –Abilene & vBNS backbones Federal backbone networks –ESnet (DoE), DREN (DoD), NREN (NSF), NSI (NASA)... Commercial Internet Service Providers (ISPs) Interconnection points (MAEs, NAPs, NGIXs, & colocation points …)

4 Internet 2 A project by consortium of universities (UCAID) to: –foster development of advanced internet applications; –foster development internet technology itself; –provide a high performance network for general research. Not a Government project; no direct Federal subsidy Not a network itself: –The NSF-funded vBNS evolved into a ‘pre’ Internet-2 backbone; –Abilene is the UCAID-sponsored Internet-2 backbone.

5 Internet 2 Membership Internet 2 –170 universities –~ non-university members including CERN –other networks can connect as affiliates; –U.S. National Labs are not members of UCAID: National Lab Internet-2 participation assumed thru ESnet vBNS –101 institutions connected –22 peer networks Abilene –96 participants, 75 connected –15 peer networks

6 Internet 2 AUP Internet 2 Acceptable Use Policy (AUP) prohibits transit traffic between affiliates: –ESnet site with Internet-2 member site is ‘OK’ to use I2; –ESnet site to other affiliated network site via Internet-2 backbone is NOT permitted. Internet 2 AUP also prohibits “commodity” internet traffic! –Traffic to another Internet-2 site routed via Internet-2 backbone; –Traffic to a non-Internet-2 site must be routed via commercial ISP; –Requires universities to have a separate Internet connection for commodity traffic.

miles deployed, backbone operates at speeds up 2.4Gbps. Interconnections at 155 & 622Mbps Abilene Seattle San Fran. CalRen UCB Stanford UCSF UCDavis UCSC Universities connect to GigaPoPs, GigaPoPs connect to backbone Peer with: APAN/Transpac, CA*net-2, DANTE, DFN, DREN, ESnet, ILAN, INFN, JANET, NACSIS, NORDunet, NISN, NREN, RENATER, SingAREN, SURFnet, vBNS … Red=N.America, Blue=Europe, Green=Asia ESnet

8 Abilene Network, Jan-2000

9 vBNS ATM-based OC12 (622Mbps) backbone, with 2.4 Gbps IP- over-SONET segments on parts of the backbone: –45Mbps is the minimum connection line speed; –98 connections as of 6/15, with 7 more pending; Run by MCI; subsidized out of NSF NGI funds; –NSF vBNS contract expires April 1st, 2000; –June 99 MCI announces vBNS+ a 5 year agreement with EDUCAUSE 1750 higher education institutions members vBNS+ no restrictive NSF imposed AUP, Connection speeds from 1.5Mbps - 2.5Gbps peering: with vBNS requires NSF authorization, none at moment with Abilene, will offer to FEDnet and International nets

10 vBNS (101 institutions)

11 ESnet DOE Energy Sciences Research Network Connects DOE ER labs and universities with major DoE funded projects –~ 50 sites –Mainly 155 Mbps backbone with some 622Mbps links Peers with other major networks –13 Internet Interconnect points –Peering exchange at MAE-West, MAE-East, Sprint NAP, Ameritech NAP, PacBell NAP International connections –CERN, DFN, INFN, JAERI, KEK, Moscow, NIFS

12 ESnet 100% growth/year since 1990 New contract, Sprint did not bid –New contract with Qwest announced January 4, 2000 –1 year transition, 2 concurrent contracts for coming year means funding tight –Same supplier as US-CERN link (KPN-Qwest) –Initial deployment ATM based ESnet3 backbone to be Tbit/sec by –5 major hubs see next transparency...

13 ESnet-3 Initial Configuration SNV ALB ORN NYC CHI LANL SNLA BNL TELEHOUSE OC48-ATM OC12-ATM OC3-ATM OC48-SONET OC3-SONET T3-ATM T3 MIT CHI-NAP PPPL ORNL ATL SRS ANL FNAL AMES PANTEX JLAB GTN ASIG 60-HUD DC Offices DC SNLL LLNL LBNL NERSC FIX-W PB-NAP MAE-W OC12-SONET SLAC MAE-E GA JGI PNNL GA (SDSC) YUCCA-MT (BECHTEL) SEA (SAIC) FULL-MESHED ATM CORE INEEL JAnet SURFnet NORDUnet Abilene DFN INFN DANTE OC3? Canada France CERN KEK/China Japan/Russia Courtesy of Jim Leighton/ESnet

14 Testbeds for U.S. NRENs “Foster Development of Internet Technology” –IPv6 = Next Generation Internet volunteers see –QoS - VoIP, multimedia and data transfer –Computing & Data grids –Collaboratories - video, virtual reality, electronic notebooks, multicast.. –Middleware - PKI, directories...

15 Commercial Internet Important to HENP –~ 18% of US HEP universities still rely solely on commercial ISPs for internet access: Internet-2 participation, even subsidized, isn’t cheap… –Critical information needed from commercial sites Quality of US commercial Internet Service is improved: –Commercial ISPs have been keeping their backbone capacity in line with (or ahead of…) demand; –Network Access Point (NAP) congestion is down. ISPs match research networks technologically: –ISPs are ahead in rollout of high bandwidth links; –ISPs are pursuing Quality of Service solutions

16 Commercial traffic: Ames Internet Exchange (AIX) TCP 90%, UDP 10%, Web ~ 55%, FTP ~ 5%, mail ~ 3% Game traffic represents noticeable portion –Quake & Starcraft account for 5% in summer, 2-3% term-time –Fairly constant (0.5%) Real audio declining (factor 2 in 6 months, now 1%) IPSEC traffic, small (< 0.2%) but growing (factor 3 in 6 months) Spikes in ICMP (security scans?)

17 Performance Measurements from –28 monitors in 15 countries –Over 500 remote hosts –72 countries (covers all 56 PDG booklet countries) –Over 1200 monitor-remote site pairs Over 50% of HENP collaborator sites are explicitly monitored as remote sites by PingER project –Atlas (37%), BaBar (68%), Belle (23%), CDF (73%), CMS (31%), D0 (60%), LEP (44%), Zeus (35%), PPDG (100%), RHIC(64%)

18 How are the U.S. Nets doing? In general performance is good (i.e. <= 1%). Edu (vBNS/Abilene) is catching up with ESnet XIWT (70%.com) 3-5 times worse than ESnet | I2

19 Europe seen from U.S. 650ms 200 ms 7% loss 10% loss 1% loss Monitor site Beacon site (~10% sites) HENP country Not HENP Not HENP & not monitored

20 Asia seen from U.S. 3.6% loss 10% loss 0.1% loss 640 ms 450 ms 250ms

21 Latin America, Africa & Australasia 4% Loss 2% Loss 350 ms 700ms 170 ms 220 ms

22 Bulk transfer - Performance Trends Bandwidth TCP < 1460/(RTT * sqrt(loss)) Note: E. Europe NOT catching up

23 Summary US HEP research network research environment is improving… The Internet-2 project is already having positive results for collaborative research: –More research universities have high bandwidth, low latency access to major U.S. research facilities; –Mission-specific research networks have something to direct improving university access efforts at…; –Testbed projects emerging for new network technologies. International performance from US to sites outside W. Europe, Japan, Korea is generally poor to bad

24 More Information ESnet home page – Internet 2 home page – vBNS+ home page – IEPM/PingER home site – ICFA-SCIC Homepage –