1. Internet-2, NGI and TEN-155 Lessons for the (European) Academic and Research Community David Williams CERN - IT Division Supercomputer’98 Mannheim David.O.Williams@cern.ch Slides: http://nicewww.cern.ch/~davidw/public/Mannheim/Mannheim.ppt

2. What about me? n Other side of the Röstigraben n CERN n Never a networking specialist n No longer a manager n ICFA-NTF n EU n So this talk is all my own views, but not all my own work n Too much to tell you about

3. Outline of this Talk n Does the Internet work? n Applications n Technical changes n America –Internet-2-- NGI n Europe –TEN-34 and successors-- 5FP n What are the lessons? n Networks for supercomputer users n The (general-purpose) Internet in Europe

4. (How well) Does the (academic) Internet work? Not as well as it should

5. April 1998 Packet Loss (extract) US universityfrom Fermifrom CERN brown.edu16.1517.29 umass.edu14.9215.77 pitt.edu10.7115.52 harvard.edu8.069.48 uoregon.edu4.180.33 washington.edu2.970.30 princeton.edu1.505.14 cmu.edu1.4914.61 hawaii.edu1.180.70 duke.edu0.306.60 umd.edu0.165.07 indiana.edu0.1415.25 mit.edu0.020.27

6. NSF Awards NLANR Group at UCSD $2.08 Million for Measurement and Analysis of Internet Infrastructure (1/2) n UNIVERSITY OF CALIFORNIA, SAN DIEGO -- May 27, 1998 n The National Science Foundation has awarded $2.08 million over 30 months to the University of California, San Diego to monitor and analyze the continent-wide research network that is a key component of the next generation of Internet technologies.

7. NLANR Award (2/2) n The award establishes the Measurement and Operations Analysis Team (MOAT) as a formal group within the National Laboratory for Applied Networking Research to analyze network traffic patterns and traffic behavior, evaluate service models, and conduct research to enhance the NSF's very high performance Backbone Network Service n See http://www.npaci.edu/online/ and http://moat.nlanr.net/ for more info

8. CERN to SLAC Monitoring over 1 week in December 1997 Above (left) is Round trip time (about 180 ms) Below (right) is packet loss rate (very low)

9. CERN to Uni Tokyo - same week RTT 350-500ms Big daily peaks Packet loss worse Some 20-30% samples One bad peak SatSun

10. Things do improve - CERN to Tokyo May 98 RTT at 300-400ms Daily peak effect much less Packet loss quite reasonable

11. SatSun Sun Mon Tue Wed Thurs Fri Daily packet loss structure on a congested route 50% peaks 8 quiet hours at night From ~01.00 to 09.00 CET

12. Packet loss rate - Fermilab to Brown University - April 98 Performance inside the US can be bad too!

13. Same period - CERN to Brown University - packet loss

14. With RTT - same period - CERN to Brown RTT = 250 to 400+ msec Compare 180 msec to SLAC

15. CAnet ESnet DFN TEN-34 HEP networking seen from Canada NACSIS

16. Lessons so far? n Not everything is wonderful in the USA n A few universities badly connected, even some very rich and famous ones n Everything depends on the detailed routing n InterNet = Interconnects + Nets n We know how to build the nets well n We have not yet learned how to build the interconnects as well

17. Three factors for the future TechnologyEconomicsOrganisation

18. Applications

19. Application generations n First = what we are used to n Second = what is about to enter general production n Third = interesting, but needs lots of bandwidth n PLUS n Interactive = human intimately in loop

20. First generation apps n e-mail –few 10s of packets; tolerant of high packet loss; non- interactive n Web access –quite similar; not really very interactive n manually initiated file transfer –more packets, but not very interactive n telnet, X-window –interactive; people start to get very disturbed with delays ~>5 sec; sensitive to packet loss rates

21. First generation apps (summary) n So far, so good n But not very adventurous n Fortunately only the interactive telnet and X traffic is very sensitive to packet loss

22. Second generation apps (1) n streaming video and audio for individuals –lots of data –quite strong RT constraints –intolerant of packet loss (esp. audio) n groupware for collaboration at a distance –fundamentally important for collaborative science –shared software development (here or next?) –brainstorming (shared whiteboards, good access to Web, incl. graphics, video and audio,...) –weekly meetings (10 people, 5 locations, …) –must be easy to use, reliable, good performance

23. Second generation apps (2) n automated data access and transfer –contrast with manually initiated file xfer –automated file transfer systems for ‘production’ –basic remote super-computer services –general client-server systems n shared file systems (e.g. AFS) –form a special subset –starting to become part of the “normally expected” infrastructure in HEP –need reasonable bandwidth and reliability –so none to Japan, quite a bit across Atlantic, but less than inside Europe and inside USA

24. Second generation apps (summary) n When they can be widely deployed they will bring fundamental improvements to collaborative science on national, European, and inter-national levels n But they do require significant improvements in the reliability of the links being used (more bandwidth, lower packet loss)

25. Third generation apps n Collaboratories and Advanced groupware –trying to break the distance barrier for individuals who work together n Remote control rooms –telescopes or physics experiments or ….. –trying to break the distance barrier for teams looking after complex technical equipment n Remote VR or other very advanced graphics –trying to break the distance barrier for people working with computers

26. Third generation apps (summary) n No shortage of interesting ideas n Probably? a shortage of bandwidth n Or of money to pay for it n Or of means to make everything work on an inter-continental basis

27. Technical changes

28. Bandwidth (1/2) n Presently drive laser signals down fibre optic cables n (Optically) amplify them every ~100 km n Lay cables under sea, on power lines, in conduits along roads and railways n Fibre itself not expensive -- O(1 DEM/m) n Undersea amplifiers have to consume very little power, which comes down cable (10 kV DC across the Atlantic) and operate unattended for 25 years. So only 4 pairs per cable, cf 24, 48, and more overland. This is the reason why undersea bandwidth is inherently more expensive today.

29. Bandwidth (2/2) n Signal processing today is electronic. 2.5 Gbps is standardly available, 10 Gbps a little expensive, but coming into use. n Big movement is to use multiple wavelengths. 4x in regular use by carriers, 32x seems almost here, some people guess that 1024x is feasible n Can you do everything optically?? Who knows?

30. Switches and routers n These are basically specialised computers, and benefit from the overall improvement in computer technology n Critical functions being incorporated in specialised electronics n High bandwidth backplanes/switching fabrics n There seem to be no barriers to progress n And a lot of very smart people and companies hoping to make money from Tbps routers

31. Service levels n Perhaps the next “religious war” n Telecoms people talk about “quality of service” and know that ATM provides it, while Internet fans now talk about “differentiated service” (instead of “integrated services”), and are thinking about who can tell what makes a service different, and how you can profit from the knowledge

32. Internet-2

33. Some history n Agencies and universities n NSFnet and 1994-95 n No NRN n UCAID

34. Agencies and universities n It is important (for Europeans) to understand that in the USA the financing of the various agencies of the federal government (such as NSF, DoE, DOD, NIH,..) is entirely separate from the funding of the universities, which is either on a private or state basis. n Although you can find some parallels in Europe (Germany in particular) the separation in the US seems to be very strong. n This all means that until now there has been no national A&R network in the US

35. NSFnet and 1994-95 n The NSFnet played a key role in the development of the Internet during the period from ~1986 until 1994/95. It was effectively the overall backbone, interconnecting the various agency networks, and its capacity was upgraded from 56 kbps to 1.5 Mbps to 45 Mbps during that time. n It was decided to “privatise” this function, and that took place during 1994/95, and NSFnet had been decommissioned by 31 March 1995.

36. No NRN n The NSFnet backbone was “replaced” by whichever Internet Service Provider (ISP) the university took its business to. n Most (and essentially all of the research universities) chose the InternetMCI service of MCI, who had run the NSFnet. n 95 and 96 were years of very fast growth of the general purpose Internet in the US (home and business use was booming). n Lots of congestion and frustration, leading in October 1996 to the first Internet-2 meetings

37. Internet-2 n This statement may not be PC (in the US), but Internet-2 is a good approximation to the NRN that the US has never had n Though Internet-2 per se has no (direct) relations with or connection to the national labs n Top priority is on production services, and not on research into networking n Though they do have plans to encourage advanced apps and to investigate and deploy QoS features

38. UCAID n The University Corporation for Advanced Internet Development is, to all intents and purposes, the supervisory board of a national university network n Responsible for Internet-2 n 122 (research) universities are members n 14 corporate partners (AT&T, 3Com, ANS, Bay, Cabletron, Cisco, Fore, IBM, Lucent, MCI, Newbridge, Nortel, Qwest, StarBurst) n Chief executive is Doug van Houweling

39. GigaPoPs (1/2) n One theme of Internet-2, which I personally find very interesting, is to construct GigaPoPs (which officially stands for Gbps point-of-presence) n All sites wanting to connect in a given region (city, state, …) connect to a commonly located, operated and funded Point of Presence n Makes it far simpler, and far more competitive, for different backbone providers to connect up GigaPops n Also simplifies inter-connection with “other networks” (such as ESnet) with decent performance

41. GigaPoPs (2/2) n Instead of universities and research institutes needing to get to each carrier’s Point of Presence, GigaPoPs allow the universities to specify where the carriers must come to n It would be a good idea (according to me) for the (A&R community in) European countries, regions and cities to invest in such EuroPoPs or UserPoPs

42. vBNS backbone n Until recently the de facto backbone for I2 was the NSF’s vBNS, provided by MCI n The very-high-performance Backbone Network Service was initially created as a fast interconnect between NSF’s Supercomputer Centres n It basically provides 622 Mbps links n During 1997/98 it has been transformed into the Connections Program and some 100? universities are now connected to it

44. More competition n Recently (15 April 98) UCAID announced that Abilene will form an alternative backbone for Internet-2 n Abilene is based on the use of the Qwest fibre optic network, with equipment from Cisco and Nortel (Northern Telecom) n I have seen essentially no discussion of the financial terms, but it is clear that some of this is supported financially by the three companies concerned

47. More on Qwest n Qwest Teams with Cisco to Build the Next Generation of High-Speed Voice/Data Networks n Dr. Shafei explained that Qwest's network starts with 48 fibers (with the capability to add ten times as many fibers through additional in-place conduits), bidirectional, line switching OC-192 ring SONET nationwide network. Each fiber can carry 8 wave division multiplexing (WDM) windows, where each WDM window has a bandwidth of 10 gigabits per second, providing Qwest with the potential of a multi terabit-per-second capability. n From a 30 Sept 1997 Press Release

48. Next Generation Internet (NGI)

49. Overview (1/2) n NGI initiative = multi-agency federal agency R&D programme for:- –developing advanced network technologies –developing revolutionary apps needing advanced nets –demonstrating via testbeds 100x-1000x faster end-to-end than today’s Internet n i.e. not the universities (directly) n Started 1 October 1997 (FY’98) n Normally said to be 3-year program (I have seen 5) n DARPA, NASA, NIH, NIST, NSF all involved n DoE from FY’99??

50. Overview (2/2) n This is the real place where the “leading-edge” R&D is being performed n But… insofar as NSF and hence vBNS are part of NGI, the project plays an important role in getting Internet-2 off the ground n Impressive (to me) how far the politicians (Gore et al, but not only him) have understood the economic importance of Internet evolution n One of my worries in Europe….

52. A first NGI vision n The Next Generation Internet will: –Accelerate mission-critical and time-sensitive research for Federal technology programs –Expedite the introduction of advanced networking services and applications –Ensure and strengthen the technological and scientific leadership of the United States –Foster stronger technology research partnerships among government, academia and industry (From talk by Toole on 13 June 97, at www.ngi.gov/talks)

53. A second NGI vision n Imagine an Internet a thousand times faster than today –An Internet so ubiquitous that it interconnects all Americans regardless of location, age, income or health –An Internet so safe and reliable that Americans confidently use it for most of their important communications –An Internet so intelligent that it can be used effortlessly to help us preserve our environment, improve our productivity, and get first rate medical care (From talk by Howell on 9 April 98, same location)

54. First NGI testbed n At least 100 sites (universities, federal labs, other research partners) connected with end-to- end speeds 100x faster than today’s Internet n Today’s end-to-end speed (such as available between two workstations) is assessed as 10 Mbps at most n Led by NSF, NASA and DoE (from FY 99)

55. Second NGI testbed n About 10 sites with end-to-end speeds 1000x faster than today’s Internet n Development of ultra-high speed switching and transmission technologies, and end-to-end speeds of 1+ Gbps n Speaks of laying groundwork for Tbps, with net management, control and QoS guarantees n Led by DARPA, with participation by NASA, NSF, DoE (from FY 99) etc.

56. Also n Experimental research for advanced network technologies n Developments of “revolutionary” applications n Everything will, as far as possible, be tested on the testbeds

59. SuperNet (1/2) n SuperNet will lay the groundwork for Tbps networks n Coordinated by DARPA IT Office n Wide-Area Broadband Core –It is DARPA’s intention that one or more metropolitan networks with links capable of at least 40 Gbps transmission rates be deployed. –In addition some or all of the metro nets will be connected to form a national ultra-high-capacity net –The elements in these nets will be largely all-optical with no electronic conversion (from www.darpa.mil/ito/Solicitations)

60. SuperNet (2/2) n Broadband Local Trunking –Searching for cost-effective ways for delivering really high-performance services to users, everywhere –Near-transparent (?) and service-independent connectivity from customer premises to all-optical backbone –20-40 Gbps fibre-based access? or –Gbps RF access (including satellite access)

61. Timetable (selected items) n Starting in 1999 –First testbed with >100 sites connected to 622 Mbps infrastructure over 155 Mbps connections n Starting in 2000 –Second testbed connecting ~10 sites with 2.5 Gbps connections n Starting in 2001 –Integrate QoS over a variety of technologies and carriers n Starting in 2002 –Tbps packet switching demo. Advanced apps tested over second testbed

62. Budgets FY’98 and FY’99 (proposal) AgencyFY’98FY’99 42 (All in MUSD) 40 DARPA42 (All in MUSD) 40 2325 NSF2325 025 DoE025 1010 NASA1010 55 NIST55 5 5 NIH5 5 85 110 Total85 110

63. The relationship between Internet-2 and NGI

64. Not easy to understand … (1/2) n Some aspects of telling politicians what they want to hear n Internet-2 grew out of university frustration with poor Internet performance. Above all it is about a better production network for universities. n The federal agencies have had their own networks, with little or nothing to be frustrated about n NGI is basically advanced R&D triggered and largely funded by the US government, with the aim of keeping US industry in a dominant position in Internet technologies

65. Not easy to understand … (2/2) n Internet-2 and NGI do not intrinsically have much to do with each other n But, as I already pointed out, Internet-2 starts out by depending on vBNS for its backbone n What change does Abilene make?? n The first NGI testbed looks awfully like Internet-2 with good connections from the various federal nets via vBNS?? n If things go well, there might be a big opportunity for the US universities and federal agencies (and industry) to make progress fast together……

66. TEN-34 and Ten-155

67. What is TEN-34? n Trans-European Network at 34 Mbps n A distributed switch interconnecting Europe’s national A&R networks (NRNs) n A true InterNet (remember I said that building these InterConnects is the hardest job) n A truly major technical and political achievement n Started (after almost two years preparatory work) in ~April 97 n Funded 40% by EU and 60% by NRNs n 4FP project which formally ends 31 July 1998

70. Some comments n You can see that in fact not many of the lines in TEN-34 are operated at 34 Mbps n And that there are in fact two basic networks - an IP network from Unisource and an ATM network from various PTTs. There was no credible single supplier in mid-96. n Administrative structure is very complex

72. More comments n Slightly more capacity installed between Europe and US than between European countries n About 100 Mbps added (to ~380 Mbps) to/from the US in the last 2 months n The US capacity is very heavily used, so total volume of A&R traffic transmitted from Europe to US is much higher than inside Europe

74. Moving to TEN-155 n Formal decisions will be made during the coming weeks. The following information is believed to be accurate, but should be treated as provisional n Arrangements have been made to extend the TEN-34 project, so that it can run down until 31 December 98, at the same time as TEN-155 runs up n TEN-155 may/might be the name of the production network which forms part of the Quantum Project, one of the last projects to be funded from the 4FP. Quantum also involves R&D into QoS issues

75. TEN-155 (more) n Should be “fully” operational by 1 Jan 99 n Likely to last about 1 year, when an FP5 takes over (see below) n Many of the major countries will have 155 Mbps access to the TEN-155 backbone

76. The Fifth Framework Programme

77. Fifth Framework Programme n What is it? n When will it happen? n What is the structure? n What about A&R networking Caveat 1: FP5 has not yet been finally defined or approved. Everything we say about it has to be taken conditionally. Caveat 2: My information may be out-of-date

78. What are Framework Programmes? My explanation n Every 5 years (in principle) the EU makes a plan concerning all of the research and technical development which it intends to carry out n These plans are called Framework Programmes n The basic idea is that the EU only does things which cannot be done better nationally n The planning exercise is quite complex

79. What is it? (from EU documentation) n Framework programmes are instruments which reflect the scientific and technological priorities of their particular period, as well as the prevailing economic and political circumstances. n Taking the form of a European Community legislative decision, framework programmes set out, for their period of application, the global objectives of Community RTD activities, the specific priorities and research themes to be addressed, the rules and procedures for implementation, the general conditions for participation, the indicative budget and the allocation of financial resources to the various research themes. n The research themes identified in the framework programme decision are then implemented by a number of "specific programmes" (e.g. Biomedicine and Health, Telematics Applications, Innovation, etc.)

80. When will it happen? n FP4 covers the period 1994-1998. It was originally allocated funding of 12.3 Gecu (for the five years), which has since been increased to 13.2 Gecu. n FP5 should cover the period 1999-2002. Present proposal is that the funding should be 16.3 Gecu (for the four years). Of this 3.925 Gecu, or 1 Gecu annually, would be allocated to “Creating a user-friendly information society”. This is about 0.12‰ of Europe’s GDP. n Pessimists expect the approval process for FP5 to last into 1999, whereas optimists hope that Calls for Proposals for specific programmes will already be issued by end-1998.

81. Structure of FP5 - Thematic n There are four Thematic Programmes, one of which is “Creating a user-friendly Information Society”. n This should have four “key actions”:- –Systems and services for the citizen –New methods of work and electronic commerce –Multimedia content and tools –Essential technologies and infrastructures n These activities are all about R&D (and nothing to do with production services)

82. Structure of FP5 - Horizontal n There are also three Horizontal Programmes, one of which is “Improving Human Potential”. n This has a “general objective, to be realised in concert with related actions elsewhere in the FP” –Support for research infrastructures n Among other things, this is the label under which support for “TEN-155” and successors, including better connections to/from outside Europe, might be provided.

83. The Information Society Technologies (IST) Programme n The “convergence” of computing, telecommunications and consumer electronics is well recognised in Brussels. n The EU has woken up to the importance of the Internet. It is starting to understand both the speed of change and the economic impact, and wants to put in place a programme for Internet development in Europe which has some cohesion and identity. n In FP4, ESPRIT, ACTS and Telematics were separate programmes. In FP5, they will be defined as part of the same programme (the IST Programme) and coordinated through a single EU management structure.

84. Networking in FP5 n Funding levels available will depend on discussions about whether A&R networking should be funded only by the IST programme, or also by the other FP5 programmes n Intention is to keep working on the interconnection of the NRNs, as long as there is a need, and as long as that need cannot be met sensibly by commercial services

85. What are the lessons?

86. Need for coherent A&R networking n A&R community forms a natural grouping n It needs reliable high-bandwidth services, both nationally and internationally n At the moment the needs and the traffic are differentiated enough to keep them separate from business, commercial, school or home use n [Good interconnections assumed] n Basically we need NRNs

87. Need for coherent European networking n Complexity - countries are different - different structures for A&R - different ideas on industrial policy n But all of the countries need to interconnect their NRNs well (besides getting good connectivity to the US) n Organisationally we need projects like TEN-34, TEN- 155 and successors n They will change over time - involving more open competition for the supply of infrastructure and services

88. Internet and the European economy n It is going to change society in many ways n It is going to be economically very important - both for the telecoms suppliers, but also for “normal” business n I believe that it will change the whole economics of voice telephony and fax n As a group we understand quite a bit about this and have a duty to explain it to our governments and fellow citizens

89. Understanding Internet better n We need better facts n About traffic, performance, reliability n Who is using our networks - what for - whether they are hppy (getting their job done) n We (users, NRNs, ISPs) must be more open about these facts

90. QoS and traffic flow separation n Differentiated services offer a big opportunity to understand which community generates which sort of traffic n Might be useful for all sorts of reasons n Including sending critical flows over separate lines (physically separate - maybe separately funded)

91. Networks for supercomputer users

92. Basics n Supercomputers can generate information faster than “normal” computers n Their users do not (normally) want or need to live close to the supercomputer n They need shared file systems to prepare programs and data n Plus collaborative software development environments n But … more info == more bandwidth n How much (semi) interaction??

93. We all need... n Like all serious network users, supercomputer users need:- –More bandwidth –Smooth reliable operation –No boundaries (between countries, between service providers) n When they obtain those they will be able to do better science and engineering

94. The (general-purpose) Internet in Europe

95. Everything so far n Was about A&R networking n But the Internet is also about people, companies, economics, and society as a whole n Europe has a tendency to treat the academic and commercial worlds as quite distinct n Carried to extremes this can be dangerous

96. Reasons for European pessimism (1/2) n Others (in this context, especially Americans) are at least as smart as we are. n And they have 2-3 years more experience than us in understanding the impact of the Internet on the economy in particular, and society in general. That is a long time! n The worlds of European business, commerce and banking completely fail (in my opinion) to understand the speed with which the Internet is transforming our basic economic assumptions. n There are not many signs that the ex-PTTs will be able to meet the double challenge of privatisation, and the Internet revolution, hitting them at the same time. There is a strong risk that they will stay national in their thinking for far too long.

97. Reasons for European pessimism (2/2) n We never (really) created a strong European computing industry n We lack the venture capital “philosophy”??? n In Europe we did not succeed to build the alliance between universities, research laboratories, the IT industry, telecoms carriers, and politicians which, in the USA, moved the Internet out from the labs and into a mass market. Fuelling a complete and virtuous economic cycle from research through to production deployment and back to research n Or, where we built the alliance in Europe, it was not based around IP

98. Reasons for European optimism (1/2) n We are at least as smart as anyone else n We can boast of many really excellent world class companies n Both in general, and in particular in (traditional) telecoms equipment supply n We have a vibrant mobile telephony industry n We (largely) liberalised the European telecoms market in January 1998. Even Greece liberalises at end 2000. n At this moment in time our national telecoms carriers are cash-rich

99. Reasons for European optimism (2/2) n Many (not all) of our national A&R networks are excellent n So is TEN-34 (if still too expensive) n (You can argue that) we have been good, historically, at understanding the need for infrastructures, at planning for their implementation, and at keeping them in good repair. [Roads, rail, airports, TGV et. al., motorways, city centre transport, schools, health care, mobile telephony,…]