1. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Status and Plans TERENA 2010 Vilnius, Lithuania John Shade /CERN

2. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Large Hadron Collider LHCOPN Presentation, TERENA 2010 – Slide 2

3. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t CERN Accelerator Complex LHCOPN Presentation, TERENA 2010 – Slide 3

4. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 26659m in Circumference 5000 SC Magnets pre ‑ cooled to -193.2°C (80 K) using 10 080 tonnes of liquid nitrogen 60 tonnes of liquid helium bring them down to - 271.3°C (1.9 K). In 1 second, a proton will circulate the LHC 11245 times The internal pressure of the LHC is 10 -13 atm, ten times less than the pressure on the Moon 600 Million Proton Collisions/second LHC (Some) Facts and Figures LHCOPN Presentation, TERENA 2010 – Slide 4

5. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t CERN’s LHC Detectors LHCOPN Presentation, TERENA 2010 – Slide 5 ALICE 26 m long 16 m high 16 m wide 10 000 tonnes CMS 21 m long 15 m high 15 m wide 12 500 tonnes LHCb 21m long 10m high 13m wide 5600 tonnes 46 m long 25 m high 25 m wide 7000 tonnes ATLAS

6. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 30-MAR-2010 – collisions at 7TeV LHCOPN Presentation, TERENA 2010 – Slide 6 On 30 March 2010, beams collided in the LHC at 7 TeV, the highest energy ever achieved in a particle accelerator, marking a new world record and the start of the LHC research programme. More than half a billion collisions observed to date. Physicists from all over the world are analysing the new data and retracing the particles discovered in past experiments (e.g. W particle and the B-meson).

7. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t ALICE Events LHCOPN Presentation, TERENA 2010 – Slide 7

8. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t ATLAS Collisions LHCOPN Presentation, TERENA 2010 – Slide 8

9. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t CMS Collisions LHCOPN Presentation, TERENA 2010 – Slide 9

10. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LHCb Events LHCOPN Presentation, TERENA 2010 – Slide 10

11. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Detectors  ~150 million electronic channels 40 MHz collision rate Fast response electronics, FPGA, embedded processors, very close to the detector N x 10 Gbit links to the Computer Center LHC Accelerator and 4 Experiments Level 1 Filter and Selection Level 2 Filter and Selection 1 Pbytes/s 150 Gbytes/s 0.7 Gbytes/s CERN Computer Center Event selection is based on the physics model, (“prejudice”, expectations) Will change over time, “limits” the physics LHCOPN Presentation, TERENA 2010 – Slide 11 Raw Data Rates

12. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 4 Experiments 3 PBytes/s We are looking for 1 ‘good’ snapshot in 10 000 000 000 000 ‘photos’ We are looking for 1 ‘good’ snapshot in 10 000 000 000 000 ‘photos’ 2 GBytes/s to the CERN computer center Store on disk and tape World-Wide Analysis Export copies Create sub-samples Physics Explanation of nature Physics Explanation of nature 10 GBytes/s4 GBytes/s 1 TByte/s ? Distributed + local The Dataflow Filter and first selection The Data Flow LHCOPN Presentation, TERENA 2010 – Slide 12

13. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t WLCG Project The Worldwide LHC Computing Grid Project, WLCG, (http://lcg.web.cern.ch/LCG/) is a global collaboration of more than 140 computing centres in 34 countries.http://lcg.web.cern.ch/LCG The mission of the WLCG project is to build and maintain a data storage and analysis infrastructure for the entire high energy physics community that will use the Large Hadron Collider at CERN. LHCOPN Presentation, TERENA 2010 – Slide 13

14. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Tier0 center, CERN very large center; tape storage; 1st-level processing and meta-data storage; quality service (24*7) Tier1 center, 11 world-wide large capacity; tape storage; quality service Tier2 center, 129 world-wide medium size, some large; no 24*7 service; no custodial storage Tier3 center very small to medium size; no availability guarantee; focus on end-user analysis activity Tier Structure LHCOPN Presentation, TERENA 2010 – Slide 14

15. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t The Beginning... Essential for Grid functioning to distribute data out to the T1’s. –Capacity had to be large enough to deal with most situations including “Catch up” LHCOPN proposed in 2004 by D. Foster (CERN) as a “Community Network” –Renamed as “Optical Private Network” as a more descriptive name after the initial meeting of stakeholders in Amsterdam. –Based on 10G as the best choice for affordable adequate connectivity by 2008. Considered by some as too conservative - can fill a 10G pipe with just (a few) PC’s! 10G is commodity now! Simple end-end model –This was not a research project, but, an evolving production network relying on emerging facilities. LHCOPN Presentation, TERENA 2010 – Slide 15

16. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Issues, Risks, Mitigation It is a complex multi-domain network relying on infrastructure provided by: –(links) NREN’s, Dante and commercial providers –(L3) T1’s and CERN –(operations) T1’s, CERN, EGI and USLHCNet Managed by the community –“Closed Club” of participants –Simple L3 model, routers at the end points –Federated operational model Design separated from implementation Need to combine innovation and operation LHCOPN Presentation, TERENA 2010 – Slide 16

17. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LHCOPN L2 Network Map LHCOPN Presentation, TERENA 2010 – Slide 17

18. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Current Situation T0-T1 Network is operational and stable. Several areas of focus: –Physical Path Routing –Operational Support –Monitoring LHCOPN Presentation, TERENA 2010 – Slide 18

19. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Operational Model GGUS used as ticket & tracking system Twiki used for collaboration documents Incident Management procedures Change Management procedures Conference calls used for Operations and Monitoring, with F2F meetings on a regular basis LHCOPN Presentation, TERENA 2010 – Slide 19

20. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t T0-T1 links’ availability LHCOPN Presentation, TERENA 2010 – Slide 20 Explanation Link availability calculated on the uptime of the BGP routing protocol relationship with each neighbor. It says for how long the T0's routers were able to route traffic to/from every T1 over the LHCOPN link. It doesn't say anything about Data-Centre to Data-Centre connectivity. FNAL-Sec and BNL-sec are the additional 10G links recently deployed, thus the high “Time Undertermined”.

21. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Basic Link Layer Monitoring perfSONAR - Integrated into the “End to End Coordination Unit” (E2ECU) run by DANTE –Provides simple indications of “hard” faults. –Insufficient to understand the quality of the service Dante MDM deployed (but is it useful?) – Configuration audit underway. Specifications for visualisation dashboard(s) being developed –Access to the MDM data is needed LHCOPN Presentation, TERENA 2010 – Slide 21

22. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LHCOPN Traffic LHCOPN Presentation, TERENA 2010 – Slide 22 http://network-statistics.web.cern.ch/network-statistics/ext/?p=sc

23. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LHCOPN Core Weekly Traffic LHCOPN Presentation, TERENA 2010 – Slide 23

24. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LHCOPN Yearly Traffic Volume LHCOPN Presentation, TERENA 2010 – Slide 24 Traffic exchanged among the T0 and the Tier1s and transited through the CERN routers. Period: 05-2009 to 05-2010 (source: http://network-statistics.web.cern.ch/network- statistics/ext/?p=sc&q=LHCOPN%20Total%20Traffic&m=LHCOPN-Total)

25. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Link capacity LHCOPN Presentation, TERENA 2010 – Slide 25 LHCOPN uses 10Gb/s links between Tier0 and all Tier1s For BNL, FERMI and RAL, two 10Gbps links are configured in round-robin, thus providing 20Gbps Renater dark fibre to Lyon would allow IN2P3 to upgrade We await (next year?) to see the prices of 40Gb interfaces. Will they be cheaper than 4*10?

26. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Future Plans LHCOPN Presentation, TERENA 2010 – Slide 26 LHCOPN “core” (Tier0, Tier1s): Extend capacity for Tier1-Tier1? Extended deployment of Cross-Border Fibre? Dedicated Tier1 Exchange Point (T1XP)? Use LHCOPN operational model for Tier2s? Hard to impose a central model on Tier2s

27. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Conclusion LHCOPN Presentation, TERENA 2010 – Slide 27 Current LHCOPN (core) does what it was designed for - shouldn’t be modified (extend – not redesign) We should care about the “broader picture” -data distribution and movement does not stop at Tier1s and is important for LHC experiments’ operation Tier1-Tier1 and Tier1-Tier2 data movements will need to be addressed, probably sooner rather than later

28. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Conclusions (continued) LHCOPN Presentation, TERENA 2010 – Slide 28 Experiments’ use of the OPN is exceeding initial estimates We need to foresee growth of connectivity to 40G and 100G and explore the paradigms of dynamic circuit provisioning Transatlantic connectivity will be a challenge!