1. The LHCONE network and L3VPN status update
Enzo Capone (with support from E. Martelli)
HEPIX2017/LHCONE-OPN Meeting
KEK (Japan), 16th October 2017

2. Enter the LHC Open Network Environment
A long time ago…
In the beginning was MONARC
Strictly hierarchical
Restricted T0-T1, T1-T1 transfers (the LHCOPN realm)
Pre-defined T1-T2 communication paths (region/country based)
LHCOPN
LHCOPN
Then the Bos-Fisk report came
“Unstructured” traffic (T1-T2 and T2-T3)
New model for data distribution (T2-T2)
No more country/region boundaries
Enter the LHC Open Network Environment

3. A new hope…
Serving any LHC sites according to their needs and allowing them to grow
Sharing the cost and use of expensive resources
A collaborative effort among Research & Education Network Providers
Traffic separation: no clash with other data transfer, resource allocated for and funded by the HEP community
Trusted peers: common security policies

4. The prototype
Based on the concept of exchange point

5. The first implementation
L2VPN-based (VPLS, mostly)
Route servers

6. L3 strikes back… Layer3 (routed) Virtual Private Network
Dedicated multi-domain service connecting Tier1s, T2s and T3s at high bandwidth
Possible to reserve bandwidth for LHC data transfers, no contention with other research projects
Trusted traffic that can bypass slow perimeter firewalls

7. LHCONE L3VPN model TierXs TierXs TierXs TierXs TierXs National VRFs
- TierX sites connected to National-VRFs or Continental-VRFs
- National-VRFs interconnected via Continental-VRFs
- Continental-VRFs interconnected by trans-continental/trans-oceanic links
Acronyms: VRF = Virtual Routing Forwarding (virtual routing instance)
Continental
VRFs
Continental
VRFs
Continental
VRFs
Continental
VRFs
LHCONE

8. The state-of-art
Asia
North America
Europe
South America

9. LHCONE services L3VPN (VRF) perfSONAR P2P
A routed virtual private network
perfSONAR
Monitoring infrastructure
P2P
Dedicated point-to-point circuits, with guaranteed bandwidth (in development)

10. Acceptable Use Policy
The LHCONE AUP has been defined to regulate the utilization of the L3VPN service (

11. From an Empire to a Republic…

12. L3VPN status update

13. World regions
Asia
Europe
North America
Latin America

14. SINET
GEANT, North America

15. TIFR
Top is with GEANT, bottom is total monthly (including Top) from CERN

16. Asia-Pacific

17. AARNET

18. LHCONE in Europe LHCONE VPN Service in Europe is deployed in:
ARNES (Slovenia)
CERN (Switzerland)
CESnet (Czech Republic)
DFN (Germany)
GARR (Italy)
GÉANT
JISC (UK)
NORDUnet
RENATER (France)
RedIRIS (Spain)
RoEduNet (Romania)
SURFnet (Netherlands)
Belnet (Belgium) – temporarily not using LHCONE
URAN (Ukraine)
PIONIER (Poland)
GRNET (Greece)
RENs in these countries connect:
Six Tier 1s in Europe to LHCONE
34 Tier 2s in Europe to LHCONE
Indian T2s (via TIFR)
Russian T1s and T2s (direct peering with GÉANT)
The only T2 in Belgium has asked BELNET to put their connection on hold.

19. CERN
Last month

20. ARNES - CESNET New upgrade to 20G
ARNES having reached the limit of their 10G access capacity has upgraded to 20G
CESNET is now connected to 100G

21. DFN
They have two 100GE links one in Frankfurt and other in Hamburg. In the process of upgrading to 2x100G

22. GARR - JISC
Only Imperial College T2

23. RedIRIS - RoEduNet

24. RENATER
RENATER has two connections one in Geneva and the other in Paris. Both are being used in parallel and both have regular peaks of over 15Gbps.

25. SARA-Matrix – GRNET

26. URAN - PIONIER New 10G peering
URAN now peering directly to GÉANT in Vienna at 10G (used to have 1G via PIONIER)
Connection to PIONIER delivered over their 100G IP access port (recently upgraded)

27. NORDUNet
The graph shows total LHCONE traffic in NORDUnet which include links to CERN, GÉANT, Esnet and other connectors.

28. RU-VRF
Top GÉANT
Bottom CERN
Added a new T2 (INR)

29. GÉANT overall Last day 3 months
Same time last year the peak traffic was around 40Gbps and average traffic closer to 30Gbps now we see regular peaks of over 60Gbps.
Overall traffic seems to be growing as you can see from GARR traffic their traffic is also growing. Some of NRENs and sites need to be upgraded as they seem to be flat-lining.
We expect to see a lot more traffic after these NRENs and sites have upgraded.
3 months

30. ESnet
LHCONE traffic in ESnet have almost doubled as well, almost 90% increase in traffic from same time last year
July – 2016
August – 2016
Feb – 2016

31. ESnet ATLAS Great Lakes Tier 2 Indiana University
University of Florida
ATLAS Northeast Tier 2
Oak Ridge National Laboratory
University of Illinois at Urbana-Champaign
Argonne National Laboratory
Pacific Northwest National Laboratory
University of Nebraska Lincoln
Brookhaven National Laboratory
Purdue University
University of Notre Dame
CERN
SLAC National Accelerator Laboratory
University of Oklahoma
California Institute of Technology
Texas Advanced Computing Center
University of Texas-Arlington
Duke University
UCSB - University of California Santa Barbara
University of Wisconsin Madison
Fermi National Accelerator Laboratory
University of California San Diego
Vanderbilt University
Indiana GigaPOP
University of Chicago

32. Internet2

33. CANARIE

34. CANARIE

35. Latin America
GÉANT
LHCONE
VRF
RNP
LHCONE
VRF
VLAN 2015
mx1.lon.uk