1. CERN external networking update & DataTAG project
Presented at the Internet2 Spring member meeting
6th May 2002, Arlington
Olivier H. Martin
CERN - IT Division

2. Main Internet connections at CERN
Mission Oriented &
World Health Org.
Swiss National Research Network
IN2P3
General purpose
A&R and commodity
Internet connections
(Europe/USA/World)
WHO
2.5Gbps
SWITCH
155Mbps
45Mbps
Europe
GEANT (1.25/2.5Gbps)
1Gbps
USLIC 622Mbps
USA
1Gbps
CERN
CIXP
2.5Gbps
Commercial
DataTAG
NetherLight
Network Research

3. CERN’s Distributed Internet Exchange Point (CIXP)
Telecom Operators & dark fibre providers:
Cablecom, COLT, France Telecom, Global Crossing, GTS/EBONE, KPNQwest, LDCom(*), Deutsche Telekom/Multilink, MCI/Worldcom, SIG, Sunrise/diAx, Swisscom (Switzerland), Swisscom (France), SWITCH (**), Thermelec.
Internet Service Providers include:
3GMobile (*), Infonet, AT&T Global Network Services (formerly IBM), Cablecom, Callahan, Carrier1, Colt, DFI, Deckpoint, Deutsche Telekom, Easynet, GTS/Ebone, Eunet/KPNQwest, France Telecom/OpenTransit, Global-One, InterNeXt, IS Internet Services (ISION), IS-Productions, Nexlink, PSI Networks (IProlink), MCI/Worldcom, Petrel, Renater, Sita/Equant(*), Sunrise, Swisscom IP-Plus, SWITCH, GEANT, VTX/Smartphone, UUnet.
isp
isp
Telecom
operators c i x p
isp
isp
isp
isp
isp
isp
CERN
firewall
Telecom
operators
Cern Internal Network

4. The CIXP is a combination of Starlight & AADS/StarTAP
Multiple Telecom Operators, Internet Service Providers & dark fiber providers
Neutral location
1 Gigabit Ethernet based, 10GBE soon.
Service extension to:
Telehouse Geneva (2*GBE/dark fiber)
Future prospects:
Close 2.5 Gbps Amsterdam-Chicago-Geneva triangle in time for iGRID’2002
Metropolitan optical testbed for demonstration at Telecom’2003 (Geneva)
isp
isp
Telecom
operators c i x p
isp
isp
isp
isp
isp
isp
ISPs
CERN
firewall
Telecom
operators
Cern Internal Network

5. The DataTAG Project

6. Funding agencies
Cooperating Networks

7. EU partners

8. Associated US partners

9. The project CERN is the project coordinator.
European partners: INFN (IT), PPARC (UK), University of Amsterdam (NL) and INRIA (FR) soon (June/July2002).
ESA/ESRIN (IT) will provide Earth Observation demos together with NASA.
Budget: 3.98 MEUR
Start date: January, 1, 2002
Duration: 2 years (aligned on DataGrid)
Funded manpower: ~ 15 persons/year
11/12/2018
The DataTAG Project

10. US Funding & collaborations
US NSF support through the existing collaborative agreement with CERN (Eurolink award).
US DoE support through the CERN-USA line consortium.
Significant contributions to the DataTAG workplan have been made by Andy Adamson (University of Michigan), Jason Leigh of Illinois), Joel Mambretti (Northwestern University), Brian Tierney (LBNL).
Strong collaborations already in place with ANL, Caltech, FNAL, SLAC, University of Michigan, as well as Internet2 and ESnet.
11/12/2018
The DataTAG Project

11. In a nutshell Two main focus:
Grid related network research (WP2, WP3)
Interoperability between European and US Grids (WP4)
2.5 Gbps transatlantic lambda between CERN (Geneva) and StarLight (Chicago) around July 2002 (WP1).
Dedicated to research (no production traffic)
Fairly unique multi-vendor testbed with layer2 and layer 3 capabilities
In principle open to other EU Grid projects as well as ESA for demonstrations
11/12/2018
The DataTAG Project

12. Multi-vendor testbed with layer3 & layer2 capabilities
INFN (Bologna)
STARLIGHT (Chicago)
Abilene
CERN (Geneva)
GEANT
ESnet
1.25Gbps
Juniper
Juniper
Research2.5Gbps
Cisco 6509 M M
Alcatel
Alcatel
Starlight
GBE
Production
622Mbps
Cisco
Cisco
11/12/2018
M= Layer 2 Mux
The DataTAG Project

13. « Distributed layer2 exchange point »
G Eth switch
G Eth switch
2.5Gb
(STM-16
2.5Gb
(STM-16
1670SM
1670SM
Operator
Trans-Atlantic
Link (KPNQwest)
CERN
Chicago
DWDM “POP”
DWDM “POP”
Alcatel 7770
Alcatel 7770
= equivalent GE GE
G Eth switch GE
G Eth switch
Starlight
CERN
« Distributed layer2 exchange point »
7770
7770

14. Goals
End to end Gigabit Ethernet performance using innovative high performance transport protocols.
Assess & experiment inter-domain QoS and bandwidth reservation techniques.
Interoperability between some major GRID projects in Europe and North America
DataGrid, possibly other EU funded Grid projects
PPDG, GriPhyN, Teragrid, iVDGL (USA)
11/12/2018
The DataTAG Project

15. Testbed Sites (>40) HEP sites ESA sites Dubna Moscow RAL Lund
Lisboa
Santander
Madrid
Valencia
Barcelona
Paris
Berlin
Lyon
Grenoble
Marseille
Brno
Prague
Torino
Milano
BO-CNAF
PD-LNL
Pisa
Roma
Catania
ESRIN
CERN
HEP sites
ESA sites
IPSL
Estec KNMI
(>40) -

16. DataTAG/WP4 framework and relationships
Grid projects:
DataGrid
PPDG
Griphyn
LCG
Globus
Condor
…..
input
feedback
…..
Grid
Interoperability
Activities:
DataTAG/WP4
iVDGL
HICB/HIJTB
GGF
Integration
stardardization
Proposals
…..
Applications:
LHC experiments
CDF
Babar
ESA
11/12/2018
The DataTAG Project

17. Major 2.5 Gbps circuits between Europe & USA
DataTAG project
NewYork
Abilene UK
SuperJANET4 IT
GARR-B
STAR-LIGHT
ESNET
GEANT
CERN
MREN NL
SURFnet
STAR-TAP
Major 2.5 Gbps circuits between Europe & USA

18. Project positioning Why yet another 2.5 Gbps transatlantic circuit?
Most existing or planned 2.5 Gbps transatlantic circuits are for production, i.e. not suitable for advanced networking experiments requiring operational flexibility, e.g.:
deploying new equipment (routers, G-MPLS capable multiplexers),
activating new functionality (QoS, MPLS, distributed VLAN)
The only known exception to date is the Surfnet circuit between Amsterdam & Chicago (Starlight)
Concerns:
How far beyond Starlight can DataTAG extend?
11/12/2018
The DataTAG Project

19. Major Grid networking issues (1)
QoS (Quality of Service)
still largely unresolved on a wide scale because of complexity of deployment
End to end performance in the presence of firewalls
There is a lack of high performance firewalls, can we rely on products becoming available or should a new architecture be evolved?
Evolution of LAN infrastructure to 1Gbps then 10Gbps
Uniform end to end performance
11/12/2018
The DataTAG Project

20. CERN’s new firewall architecture
Regular flow
Gbit Ethernet
Backup
Bypass
CERNH2
(Cisco OSR 7603)
Gbit
Ethernet
Fast Ethernet
FastEthernet
Dxmon FE and FDDI+bridge
Cisco PIX
Cisco RSP7000
FastEthernet
100/1000 Ethernet
Fast Ethernet
Cabletron SSR
Security monitor
Gbit Ethernet

21. Major Grid networking issues (2)
TCP/IP performance over high bandwidth, long distance networks
The loss of a single packet will affect a 10Gbps stream with 200ms RTT (round trip time) for 5 hours. During that time the average throughput will be 7.5 Gbps.
On the 2.5Gbps DataTAG circuit with 100ms RTT, this translates to 38 minutes recovery time, during that time the average throughput will be 1.875Gbps.
Line Error rates
A 2.5 Gbps circuit can absorb 0.2 Million 1500 Bytes packets/second
Bit error rates of 10E-9 means one packet loss every 250 milliseconds
Bit error rates of 10E-11 means one packet loss every 25 seconds
11/12/2018
The DataTAG Project

22. Single stream vs Multiple streams effect of a single packet loss (e. g
Single stream vs Multiple streams effect of a single packet loss (e.g. link error, buffer overflow)
Streams/Throughput 10 5 10
Avg. 7.5 Gbps
Throughput Gbps
7 5
Avg Gbps
Avg Gbps 5
2.5
Avg Gbps
T = 2.37 hours!
(RTT=200msec, MSS=1500B) T T T
Time T
11/12/2018
The DataTAG Project

23. Concluding remarks
The dream of abundant bandwith has now become a hopefully lasting reality!
Major transport protocol issues still need to be resolved.
Large scale deployment of bandwidth greedy applications still remain to be done,
Proof of concept has yet to be made.
11/12/2018
The DataTAG Project