Internet2 Networks in Support of e-Science Rick Summerhill Director Network Research, Architecture, and Technologies Russ Hobby Program Manager for Science and Engineering Internet2 NSF Internet2 Day 19 October 2006

Agenda The Existing Internet2 Network (Abilene) History The Network Peering with International Networks Science Applications The New Challenge The new Internet2 Network A Broader Design Topology New Capabilities Connections and Peerings

History With the end of NSFnet and beginning of privatization, the ability to support large science applications was limited on the commodity network The vBNS days brought much better connectivity, especially to super computer centers NSF connections program was of great benefit to the community Initial meetings of what was to become Internet2 in 1996 There was a clear need to support the university research community The Internet, even for “small” applications was not as robust as it is today The university networking needs for science are not met today by the commodity Internet and likely will not be met in the near future.

The Internet2 Network Abilene First Internet2 network began in mid 1998 with first backbone (Abilene) at 2.5 Gbps (OC-48 SONET backbone) Partnership with Qwest, Cisco, and Nortel NSF connections program played a crucial role in this period Backbone upgrade to 10 Gbps started in 2001 and completed in early 2003 Juniper became a partner at that time. Supported advanced services like IPv6 and Multicast Focus on performance and reliability primarily for e-Science applications

Rick Summerhill Current Internet2 Network (Abilene)

Rick Summerhill National Architecture (Current)

Internet2 Connectors and Peers Connectors 35 direct connections (OC-3c  10 Gbps) 246 Primary Participants – research universities and labs Cost recovery model Regional to National Campus to National and Regional Campuses and Regionals typically also connect to the commodity network Connections are Transparent - Many users don’t know they use Internet2 Peerings Connections through exchange points 48 International agreements and 80+ International networks reachable through Abilene Abilene supports transit for International peers Peerings with other Federal Research and education networks in the US See:

Science and Engineering Applications Russ Hobby Program Manager for Science and Engineering

More Than High-End Computing and Connectivity Focused on sharing and making greater capabilities available across the science and engineering research communities Allows applications to interoperate across institutions and disciplines Ensures that data and software acquired at great expense are preserved and easily available to all Empowers enhanced collaboration over distance and across disciplines. Report of the National Science Foundation Blue-Ribbon Advisory Panel on Cyberinfrastructure

CI Functions and Interactions Researcher Control Program Viewing Instrumentation Security Control Data Generation 3D Imaging Display and Visualization. Display Tools Security Data Sets Storage Security Search RetrievalInput Schema Metadata Computation Analysis Simulation Program Security Middleware Security and Access Authentication Access Control Authorization Security Archive Data Input Data Directories Collab Tools Education And Training Ontologies

The Network CI Functions and Interactions Researcher Control Program Viewing Instrumentation Security Control Data Generation 3D Imaging Display and Visualization. Display Tools Security Data Sets Storage Security Search RetrievalInput Schema Metadata Computation Analysis Simulation Program Security Middleware Security and Access Authentication Access Control Authorization Security Archive Data Input Data Directories Collab Tools Education And Training Ontologies

The Applications Area is part of the Community side of Internet2 Program managers are one channel through which research disciplines can interact with Internet2 and the broader Internet2 community Disciplines that do not normally interact can share experiences and solutions to cyberinfrastructure problems Joint projects can be created to accomplish more than one discipline can do by itself

Application Communities One way to organize is by their stage of CI development: Mature-Helping Develop CI Using CI Implementing Planning Thinking about it Those with more experience teach those with less

High Energy and Nuclear Physics (HENP) Physics has traditionally been one of the “power users” of all networks Physicists are generating Terabytes of data (1,000,000,000,000 or 1x10 12 ) per experiment from the CERN lab in Switzerland Types of network usage: Bulk data transfers that are extremely resistant to data loss. VRVS expects multicast and low- latency/jitter networks for effective video conferencing As a mature community, we learn as much from the HENP community as they do from us (it could be argued that we are the students).

Network for Earthquake Engineering Simulation NEES has multiple labs at various geographical locations NEES has it’s own IT organization, NEESit, that provides support for the community Internet2 works with NEESit and the NEES headquarters

Electronic Very-Long-Baseline Interferometry (eVLBI) Astronomers collect data about a star from many different earth based antennae and send the data to a specialized computer for analysis on a 24x7 basis. eVLBI is not as concerned with data loss as they are with long term stability. The end goal is to send data at 1Gb/s from over 20 antennae that are located around the globe. Internet2 works closely with the eVLBI researchers and assists where we can. We are the teachers, but probably not for long.

NEON and Earthscope Both NSF sponsored programs are in the implementation stages Their research goals and science plans are fairly well understood. Using CI and advanced networks to connect researchers, data and sensors is assumed. The specific ways in which advanced networking will be integrated into their project still needs to be investigated As newer groups to the Internet2 community, Program Managers help identifying areas in which advanced networking experience of others can be used to ease their efforts

Sensor Networks NEON and EarthScope are just two of many sensor network projects. Many of these projects are in the early planning stages and can benefit from sharing experiences. They will need all the CI resources AND Many will want to connect instrumentation at remote sites that lack any network connectivity There is a need for national wireless coverage for sensor networks (connecting the last square kilometer)

Oceanography The Ocean Research Interactive Observatory Networks (ORION) is a program that focuses the science, technology, education and outreach of an emerging network of science driven ocean observing systems As a planning community, we help them learn from the experiences of others. We also observe their planning process so others may benefit.

Security Spiderman’s Uncle said, “with great power comes great responsibility.” Security is a inter- community, cross-application concern. Cutting edge applications are often indistinguishable from security breaches. Implementing systems to support advanced applications while addressing security is a requirement for all future applications.

The Dynamic Virtual Global Collaboratory Collaborating Virtual Organizations will become increasingly important Consider the emerging e-science paradigm… Global science For example, astrophysics, astronomy, earth sciences, climate modeling, etc. Global shared resources Large Hadron Collider, radio telescopes, polar research stations, computational resources, etc. Global collaborating science teams E-VLBI, HEP, Genomic Research, etc These “affinity groups” combine resources and people into a globally distributed virtual collaborating organizations to pursue a common discipline or objective.

The Coming Challenge “Grid applications will incorporate in excess of 100,000 processors within 5 years.” Dr. Larry Smarr, “On Vector” Workshop, UCSD Feb 2006 “The Global Information Grid will need to store and access exabytes of data on a real-time basis by 2010” Dr. Henry Dardy, Optical Fiber Conference, Los Angeles, CA USA, Mar 2006 “Each LHC experiment foresees a recorded raw data rate of 1 to several PetaBytes/year” Dr. Harvey Neuman (Cal Tech) “US Bancorp backs up 100 TB financial data every night – now.” David Grabski (VP Information Tech. US Bancorp), Qwest High Performance Networking Summit, Denver, CO. USA, June “The VLA facility is now able to generate 700 Gbps of astronomical data and will reach 3.2 Terabits per second by 2009.” Dr. Steven Durand, National Radio Astronomy Observatory, E-VLBI Workshop, MIT Haystack Observatory., Sep 2006.

The Networking Challenge 10,000 processor compute cluster 4 Gbyte/processor memory, 1 GigE NIC Burst capability = 10 Tbps Parallel and distributed clusters are incorporating nodes faster than Moore’s Law is reducing their size.. Power requirements for single clusters will be too large to support in a single location - moved to geographically distributed clusters How will they communicate? These reflect some fundamental design decisions/assumptions of the existing (and original) internet architecture that may not be applicable today, especially for e-Science applications The GENI initiative (NSF) hopes to construct a Global Environment for Network Innovation Need to examine new architectures for e-Science applications now

The New Internet2 Network Agreement with Qwest for the current Abilene Network ends in October of 2007 The new Intenet2 network must be capable of supporting e-Science applications Strategic Objectives Ensure community control of the underlying (optical) network infrastructure Leverage the capabilities of a global telecommunications leader Providing carrier class reliability and expanded breadth of services, along with a broad set of partnership options Capitalize on the latest technological advancements in networking Create an asset that benefits the entire community - researchers, universities, regional optical networks, industry, government, K-12, and the international community

Architecture Goals Develop an innovative optical system on a national footprint to serve the broad research and education community - a hybrid network capable of providing point-to-point services (circuits) together with an IP network Community should have complete control of the lower layer optical system including provisioning and switching of wavelengths The system should be capable of supporting network research in wide variety ways Platform support for highly experimental projects to production services Provide capabilities to support e-Science using new paradigms in networking

New Network Paradigms to Support e- Science Example: eVLBI The ability to dynamically create the necessary network to support particular experiments

Rick Summerhill The New Internet2 Network

Internet2 ESnet Partnership Internet2 and ESnet have formed a partnership to build their respective networks on this DWDM footprint ESnet hybrid network An IP network connecting the labs An lower layer network for deterministic services - SDN Internet2 hybrid network An IP network similar to the existing Abilene network A layer 1 dynamically provisioned network providing deterministic services Static and Dynamic services will be available to other partners

The New Internet2 Network - What’s Different? Hybrid IP and Optical System utilizing Level3 fiber platform; equipment and fiber dedicated to Internet2, sparing and equipment maintenance by Level3, including SLA for wave system Initially provisioned with ten times the capacity scalable to much larger Dynamic provisioning of circuits and waves across the network within seconds Connections and Peerings through IP and circuits (lightpaths)! The ability to create circuits between between researchers and facilities such as international radio telescopes and particle accelerators Geographically distributed Instruments appear as on a LAN rather than use the public network

Rick Summerhill National Architecture (New)

More Info Rick Summerhill Russ Hobby science.internet2.edu