1. ATLAS Tier 2 Paths Within ESnet
Mike O’Connor
ESnet Network Engineering Group
Lawrence Berkeley National Lab

2. ATLAS Tier 2 Connectivity
Typical ATLAS tier 2 site connectivity should be via Abilene and ESnet in order to reach Brookhaven Lab.
“ESnet’s Goal is that DOE Lab ↔ Univ. connectivity should be as good as Lab ↔ Lab and Univ. ↔ Univ.” W. E. Johnston
Speaking to the ATLAS tier 2 sites.
Assuming that your institute has attained sufficient connectivity to the Abilene network, I will discuss how these paths through the ESnet are engineered today and the options that exist for the future.

3. ATLAS Tier 2 Support
The role of ESnet is to provide networking that supports and anticipates the needs of the Office of Science Labs and their collaborators.
ESnet support for ATLAS tier 2 site connectivity makes high bandwidth and fault tolerant paths between BNL and Abilene an absolute priority.
Planning is underway to upgrade BNL’s ESnet OC48 circuit to a multi 10G lambda fault tolerant Metropolitan Area Network (MAN). (Targeted for Q1 2006)
Today ESnet and Abilene peer in Atlanta, Chicago, New York and Sunnyvale with a combined bandwidth of 18.2Gbps.These four locations comprise a high bandwidth and fault tolerant interconnect between these two networks.

4. ESnet Provides Global High-Speed Internet Connectivity for DOE Facilities and Collaborators (Spring, 2006 Configuration)
Japan (SINet)
Australia (AARNet)
Canada (CA*net4
Taiwan (TANet2)
Singaren
SINet (Japan)
Russia (BINP)
GÉANT
- France, Germany,
Italy, UK, etc
CA*net4 France
GLORIAD (Russia, China) Korea (Kreonet2
MREN
Netherlands
StarTap Taiwan (TANet2, ASCC)
PNWGPoP/ PAcificWave
CERN
(USLHCnet CERN+DOE funded)
ESnet Science Data Network (SDN) core
SEA
LIGO
MAN LAN Abilene
PNNL
ESnet IP core
Starlight
CHI-SL
NYC
MIT
Abilene AU
INEEL
Chi NAP
LVMR-NOC
BNL
JGI
Abilene
FNAL
LLNL
Lab DC
Offices
LBNL
AMES
ANL
PPPL
SNLL
NERSC
SNV
CHI
MAE-E
SLAC
PAIX-PA
Equinix, etc.
NREL
OSC GTN NNSA
Equinix
SNV SDN
BECHTEL-NV
KCP
JLAB
NASA Ames DC
ORNL
YUCCA MT
OSTI
ORAU
LANL
MAXGPoP
ARM
ATL
SDSC
SNLA
NOAA AU GA
Abilene
SRS
ALB
Allied
Signal
ESnet IP core: Packet over SONET Optical Ring and Hubs
PANTEX
SoXGPoP
DOE-ALB
South America West
ELP
South America East
International (high speed)
10 Gb/s SDN core
10G/s IP core
2.5 Gb/s IP core
MAN rings (≥ 10 G/s)
OC12 ATM (622 Mb/s)
OC12 / GigEthernet
OC3 (155 Mb/s)
45 Mb/s and less
42 end user sites
international R&E peers
Office Of Science Sponsored (22)
commercial and R&E peering points
NNSA Sponsored (12)
Joint Sponsored (3) IP
SDN
ESnet core hubs
Abilene
Other Sponsored (NSF LIGO, NOAA)
high-speed peering points with Internet2/Abilene
Laboratory Sponsored (6)

5. ESnet & Abilene Peering points
*paths do not include all router hops
**Example site logos chosen at random.

6. Abilene Routing Toward ESnet
Abilene Interior Gateway Protocol (IGP) mileage based metrics determine the best exit path toward the ESnet. This common method also known as “hot potato routing” it will always compute an exit path to the nearest peering point.

7. ESnet Routing Toward Abilene
Abilene Multi Exit Discriminators (MEDs) are used by ESnet to choose the best exit path toward Abilene. These MEDs are based on the same IGP metrics used in the interior of their network. This creates a symmetric path back to Abilene.

8. 32 Avenue Of the Americas NYC

9. Chicago

10. Current Load in Chicago

11. Sunnyvale & Atlanta

12. Future Directions
Long Island MAN Ring
ESnet Lambda Infrastructure

13. Circuits based on WDM rings
ESnet LI MAN Ring
2 λs (10 G circuits) delivered as two 10 GigEther circuits
8 X current BNL bandwidth
Physically diverse fiber paths forming a ring topology.
Will be used as a10 Gb/s production IP and a10 Gb/s path for SDN circuit services
Services switch over to the other service path during a circuit failure.
λ2 SDN/circuit
Circuits based on WDM rings
10 Gb/s optical channels
λ1 production IP
*SDN – Science Data Network

14. ESnet 2010 Lambda Infrastructure and LHC T0-T1 Networking
TRIUMF
Vancouver
CERN-1
CANARIE
Seattle
Toronto
CERN-2
Boise
CERN-3
BNL
Chicago
Clev
New York
Denver
Sunnyvale KC
Pitts
GÉANT-1
FNAL
Wash DC
Raleigh LA
Albuq.
Tulsa
Phoenix
GÉANT-2
San Diego
Atlanta
Dallas
El Paso - Las Cruces
Jacksonville
NLR PoPs
Pensacola
Houston
Baton Rouge
ESnet IP core hubs
San Ant.
ESnet Production IP core (10-20 Gbps)
ESnet Science Data Network core (10G/link) (incremental upgrades, )
Other NLR links
CERN/DOE supplied (10G/link)
International IP connections (10G/link)
ESnet SDN/NLR hubs
Tier 1 Centers
Cross connects with Internet2/Abilene
New hubs

15. Conclusions
ESnet provides an infrastructure and services that are critical to ATLAS science
ESnet is well positioned to provide essential network services to the LHC tier 2 community
ESnet is working on providing the DOE mission science networking requirements with several new initiatives and a new architecture