CERN Campus Network Infrastructure Specificities Jean-Michel Jouanigot Campus Network Leader CERN EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCH EUROPEAN LABORATORY FOR PARTICLE PHYSICS

2 CERN - European Organization for Nuclear Research Networks at CERN Why are they important? Why are they important? Carry all organization “business” Carry all organization “business” What for? What for? Support wide range of equipment (supercomputers to PDAs) Support wide range of equipment (supercomputers to PDAs)  Must accommodate any equipment  Date, Voice, Video  We are operating very much like an ISP For complex population (from safety guard to computer expert) For complex population (from safety guard to computer expert) Low performance (i.e. wireless) to very high performance (i.e. computer center & experiments) Low performance (i.e. wireless) to very high performance (i.e. computer center & experiments) The users expect The users expect 100% availability 100% availability Global connectivity Global connectivity The network to be part of the internet The network to be part of the internet Security Security

3 CERN - European Organization for Nuclear Research A large infrastructure Two distinct multi-gigabit backbones Two distinct multi-gigabit backbones About 85 very high performance gigabit routers About 85 very high performance gigabit routers 1’200 subnets 1’200 subnets 800 switches, ~40’000 ports 800 switches, ~40’000 ports 600 Ethernet hubs, ~15’000 ports 600 Ethernet hubs, ~15’000 ports 20’000 active user devices 20’000 active user devices 40’000 sockets – 2’000 km of UTP cable 40’000 sockets – 2’000 km of UTP cable 200+ starpoints (from 20 to 1’000 outlets) 200+ starpoints (from 20 to 1’000 outlets) 2’500 km of fibers 2’500 km of fibers Extremely Dynamic Extremely Dynamic 1’500+ requests for Moves-Adds-Changes per month 1’500+ requests for Moves-Adds-Changes per month Multi vendor site using only standards Multi vendor site using only standards ~10 Tons ~150 meters

CERN Network infrastructure Specificities

5 CERN - European Organization for Nuclear Research Product selection and validation International open call for tenders International open call for tenders Understanding market evolution Understanding market evolution Product evaluation and certification Product evaluation and certification  Network “lab” with traffic generators, etc  Large computer center testbeds: real life tests Leading Edge products Leading Edge products Constant evaluation of new technologies and early adopters Constant evaluation of new technologies and early adopters  Gigabit Ethernet in 1998  Ten Gigabit Ethernet in 2002 Strong, direct interaction with manufacturers Strong, direct interaction with manufacturers  Technology and product feedback (i.e. Openlab) We have broken almost all the products tested so far We have broken almost all the products tested so far  Some manufacturers were not able to reproduce  Interest of many manufacturers to validate their equipment at CERN

6 CERN - European Organization for Nuclear Research Example: LCG cluster network gbps to 88*1gbps ~6000 CPU servers ~2000 Tape and Disk servers Campus Network Experiment al areas Experiment al areas WAN One double Link not shown Gigabit Ethernet Ten Gigabit Ethernet link Double Ten gigabit Ethernet link 10gbps to 88*1gbps ….…. ….…. 2.4 Tbps CORE Distribution layer 10gbps to 32*1gbps10gbps to 10*10gbps

7 CERN - European Organization for Nuclear Research The role of automation Investment in Automation to save manpower Investment in Automation to save manpower Very large investment in definition & implementation of a global multi- manufacturer management framework Very large investment in definition & implementation of a global multi- manufacturer management framework  Well defined service definition (WEB interfaces, etc)  Well defined models and procedures  The right tools prevent mistakes and improve reliability The Network is heterogeneous, high bandwidth and large The Network is heterogeneous, high bandwidth and large Enterasys, 3COM, BATM, Cisco, HP, etc Enterasys, 3COM, BATM, Cisco, HP, etc The network is operated by The network is operated by First line: 5 FTE (SLA) First line: 5 FTE (SLA) Second line: 1 FTE Second line: 1 FTE Infrastructure Infrastructure new installations: 7 FTE new installations: 7 FTE Software and tools: 6 FTE Software and tools: 6 FTE

8 CERN - European Organization for Nuclear Research Summary Strong in house knowledge in Strong in house knowledge in Network technologies Network technologies Product evaluation and validation Product evaluation and validation Design & Operation of very large high performance network infrastructures Design & Operation of very large high performance network infrastructures Network Management Automation Network Management Automation Established partnership with some key manufacturers Established partnership with some key manufacturers Early adopters of new technologies Early adopters of new technologies

9 CERN - European Organization for Nuclear Research Thank you!