1. The DataTAG Project Olivier H. Martin http://www.datatag.org
INFN Grid workshop
17th June 2002
Olivier H. Martin

2. Funding agencies
Cooperating Networks

3. EU partners

4. Associated US partners

5. The project CERN is the project coordinator.
European partners: INFN (IT), PPARC (UK), University of Amsterdam (NL) and INRIA (FR) soon (July/August 2002).
ESA/ESRIN (IT) will provide Earth Observation demos together with NASA.
Budget: 3.98 MEUR
Start date: January, 1, 2002
Duration: 2 years (i.e. terminate at the same date as DataGrid)
Funded manpower: ~ 15 persons/year
9/21/2018
The DataTAG Project

6. US Funding & collaborations
US NSF support through the existing collaborative agreement with CERN (Eurolink award).
US DoE support through the CERN-USA line consortium.
Significant contributions to the DataTAG workplan have been made by Andy Adamson (University of Michigan), Jason Leigh of Illinois), Joel Mambretti (Northwestern University), Brian Tierney (LBNL).
Strong collaborations already in place with ANL, Caltech, FNAL, SLAC, University of Michigan, as well as Internet2 and ESnet.
9/21/2018
The DataTAG Project

7. In a nutshell Two main focus: Grid related network research (WP2, WP3)
Interoperability between European and US Grids (WP4) – See separate presentation by A. Ghiselli
2.5 Gbps transatlantic lambda between CERN (Geneva) and StarLight (Chicago) around September 2002 (WP1).
Dedicated to research (no production traffic)
Unique multi-vendor testbed with layer2 and layer 3 capabilities
In principle open to other EU Grid projects as well as ESA for demonstrations
9/21/2018
The DataTAG Project

8. Major 2.5 Gbps circuits between Europe & USA
DataTAG project
NewYork
Abilene UK
SuperJANET4 IT
GARR-B
STAR-LIGHT
ESNET
GEANT
CERN
MREN NL
SURFnet
STAR-TAP FR
INRIA
ATRIUM/VTHD
Major 2.5 Gbps circuits between Europe & USA

9. Project positioning Why yet another 2.5 Gbps transatlantic circuit?
Most existing or planned 2.5/10 Gbps transatlantic circuits are for production,
not suitable for advanced networking experiments requiring operational flexibility:
deploying new equipment (routers, G-MPLS capable multiplexers),
activating new functionality (QoS, MPLS, distributed VLAN)
Concerns:
KPNQwest bankruptcy
emergency re-procurement under way
DataTAG reach beyond Starlight?
9/21/2018
The DataTAG Project

10. Multi-vendor testbed with layer3 & layer2 capabilities
INFN (Bologna)
STARLIGHT (Chicago)
Abilene
CERN (Geneva)
GEANT
ESnet
1.25Gbps
Juniper
Juniper
Research2.5Gbps
Cisco 6509 M M
Alcatel
Alcatel
Starlight
GBE
Production
622Mbps
Cisco
Cisco
9/21/2018
M= Layer 2 Mux
The DataTAG Project

11. « Distributed layer2 exchange point »
G Eth switch
G Eth switch
2.5Gb
(STM-16
2.5Gb
(STM-16
1670SM
1670SM
Operator
Trans-Atlantic
Link
CERN
Chicago
DWDM “POP”
DWDM “POP”
Alcatel 7770
Alcatel 7770
= equivalent GE GE
G Eth switch GE
G Eth switch
Starlight
CERN
« Distributed layer2 exchange point »
7770
7770

12. The STAR LIGHT
Next generation STAR TAP with the following main distinguishing features:
Neutral location (Northwestern University)
1/10 Gigabit Ethernet based
Multiple local loop providers
Optical switches for advanced experiments
The STAR LIGHT will provide 2*622 Mbps ATM connection to the STAR TAP
Started in July 2001
Also hosting other advanced networking projects in Chicago (OMNINET) & State of Illinois (I-WIRE)
9/21/2018
The DataTAG Project

13. OMNInet Technology Trial
West Taylor
Evanston GE
10GE l
10GE l GE
OPTera
Metro
5200
Optical
Switching
Platform
Optical
Switching
Platform
Application
Cluster
Passport
8600
Passport
8600
Application
Cluster
OPTera
Metro
5200
OPTera
Metro
5200
Lakeshore
S. Federal
10GbE WAN
To Ca*Net4
(future) GE
10GE l
10GE l GE
OPTera
Metro
5200
Optical
Switching
Platform
Application
Cluster
Optical
Switching
Platform
Passport
8600
Passport
8600
Application
Cluster
OPTera
Metro
5200
A four site network in Chicago -- the first 10GE service trial!
A test bed for all-optical switching, advanced high-speed services, and and new applications including high-performance streaming media and collaborative applications for health-care, financial, and commercial services.
Partners: SBC, Nortel, International Center for Advanced Internet Research (iCAIR) at Northwestern, Electronic Visualization Lab at Univ of Illinois, Argonne National Lab, CANARIE

14. The CIXP is a combination of Starlight & AADS/StarTAP
Multiple Telecom Operators, Internet Service Providers & dark fiber providers
Neutral location
Gigabit Ethernet based, 10GBE soon (Beta test with Cisco 3Q02).
Service extension to:
Telehouse Geneva (2*GBE/dark fiber)
Future prospects:
Close 2.5 Gbps Amsterdam-Chicago-Geneva triangle in time for iGRID’2002
Metropolitan optical testbed for demonstration at Telecom’2003 (Geneva)
isp
isp
Telecom
operators c i x p
isp
isp
isp
isp
isp
isp
ISPs
CERN
firewall
Telecom
operators
Cern Internal Network

15. Major Grid networking issues (1)
QoS (Quality of Service)
Still largely unresolved on a wide scale because of complexity of deployment
Non elevated services like “Scavenger/LBE” (lower than best effort) or Alternate Best Effort (ABE) are very fashionable!
End to end performance in the presence of firewalls
There is (will always be) a lack of high performance firewalls, can we rely on products becoming available or should a new architecture be evolved?
Some f/w are not completely transparent (e.g. PIX & Window scaling)
Evolution of LAN infrastructure to 1Gbps then 10Gbps
Uniform end to end performance (LAN/WAN)
9/21/2018
The DataTAG Project

16. CERN’s new firewall architecture
Regular flow
Gbit Ethernet
Backup
HTAR
(High Throughput Access Route)
CERNH2
(Cisco OSR 7603)
Gbit
Ethernet
Fast Ethernet
FastEthernet
Dxmon FE and FDDI+bridge
Cisco PIX
Cisco RSP7000
FastEthernet
100/1000 Ethernet
Fast Ethernet
Cabletron SSR
Security monitor
Gbit Ethernet

17. Major Grid networking issues (2)
TCP/IP performance over high bandwidth, long distance networks
The loss of a single packet will affect a 10Gbps stream with 200ms RTT (round trip time) for 5 hours. During that time the average throughput will be 7.5 Gbps.
On the 2.5Gbps DataTAG circuit with 100ms RTT, this translates to 38 minutes recovery time, during that time the average throughput will be 1.875Gbps.
Line Error rates
A 2.5 Gbps circuit can absorb 0.2 Million 1500 Bytes packets/second
Bit error rates of 10E-9 means one packet loss every 250 milliseconds
Bit error rates of 10E-11 means one packet loss every 25 seconds
9/21/2018
The DataTAG Project

18. Single stream vs Multiple streams effect of a single packet loss (e. g
Single stream vs Multiple streams effect of a single packet loss (e.g. link error, buffer overflow)
Streams/Throughput 10 5 10
Avg. 7.5 Gbps
Throughput Gbps
7 5
Avg Gbps
Avg Gbps 5
2.5
Avg Gbps
T = 2.37 hours!
(RTT=200msec, MSS=1500B) T T T
Time T
9/21/2018
The DataTAG Project

19. TCP/IP based transport
Parallel TCP/IP streams
BBFTP
GridFTP
PSockets
Striping data across several socket connections (network striping)
Improved TCP/IP stacks
Web100 (NSF) / Net100 (DoE)
Auto-tuning
Visualization
High Speed TCP (Internet draft, Sally Floyd)
Basic principle is to adapt the congestion avoidance to very high speed environments by changing the AIMD (additive increase, multiplicative decrease) parameters
FAST (Fast AQM (Active Queue Management) Scalable TCP-NSF/STI – Caltech)
ECN (Early Congestion Notification)
New set of TCP/IP options used to signal congestion instead of packet drops
9/21/2018
The DataTAG Project

20. UDP based transport
Form of “rate based” TCP using a combination of UDP and TCP as a return channel to signal packet losses back to the sender.
RBUDP (Reliable Blast UDP), Electronic Visualization Lab of Illinois)
SABUL (Simple Available Bandwidth Utilization Library for High-Speed Wide Area Networks)
Tsunami (University of Indiana) ….
Can be combined with Scavenger/LBE type service in order to minimize the impact on other flows
Can be perceived as a Denial of Service (DoS) attack, therefore definitely not a mature solution on the public Internet!
Impact could be minimized if all routers were capable of running the “TCP friendly” test:
i.e. no flow should receive a better treatment that a network congestion aware TCP/IP flow
9/21/2018
The DataTAG Project

21. The Tsunami Protocol
Developed specifically to address extremely high-performance batch file transfer over global-scale WANs.
Transport is UDP using 32K datagrams/blocks superimposed over standard 1500-byte Ethernet packets.
No sliding window (a-la TCP), each missed/dropped block is re-requested autonomously (similar to smart ACK)
Very limited congestion avoidance compared to TCP. Loss behavior is similar to Ethernet collision behavior, not TCP congestion avoidance.
9/21/2018
The DataTAG Project

22. Tsunami Protocol UDP Data Flow … 9 4 3 Data Type Seq Data Type Seq
Server
(retransmit request)
(shutdown request)
Client 5 6 7 8
TCP Control Flow
9/21/2018
The DataTAG Project

23. Tsunami Performance
Link between Seattle and Brussels (Abilene/GEANT/Belnet (GTRN) approximately 10,000 kilometers.
At gigabit speeds, pipe has approximately 12MBytes of capacity (remember old mercury-delay memories?)
During testing, it was discovered that routine packet loss across the link was %
TCP cannot maintain reasonable performance with even that small loss. Approximate 40Mbit/s throughput over link with 0.2% packet loss.
Tsunami able to achieve sustained 850Mbit/s for over 17 hours
9/21/2018
The DataTAG Project

24. Concluding remarks
The dream of abundant bandwith has now become a hopefully lasting reality!
Major transport protocol issues still need to be resolved.
Large scale deployment of bandwidth greedy applications still remain to be done,
Proof of concept has yet to be made.
9/21/2018
The DataTAG Project