Run - II Networks Run-II Computing Review 9/13/04 Phil DeMar Networks Section Head

LAN Overview Centralized management of the facility network  Includes design, installation, & operation of Run-II networks LAN architecture based on work groups  Run-II computing resources reside within its own LAN General philosophies on network design  “Over-provisioning” capacity reduces problems  Homogeneity & simplicity help management & operations Technical directions:  Capacious switch fabric w/ high port density  10 GE backbone links; 1000B-T for host systems

FNAL Networks: A Work Group Model Minos- Soudan Core Network Off Site On Site Fast routing & switching of workgroup LANs, each with a 1000Mb/s link into the core network OC12 link {622 Mb/s} Workgroup internal LANs have switched 10/100 Mb/s desktop links & 100/1000 Mb/s server links. CDF D0 Run-II work groups: 10GE & (n x 1GE) backbone links 1000 Mb/s server & farms support switched 100 Mb/s desktop support R&D WANs StarLight dark fiber: 10GE & (2 x 1GE) channels 10 GE links to CDF, CMS, and (soon) D0 Intended use: R & D Production WANs CMS MINOS Mini- Boone SDSS

CDF Run-II Network

D0 Run-II Network

FY05 Planned LAN Upgrades Core Network Upgrades:  10 Gb/s backbone links, including border router  Wireless LAN upgrade to b (54Mb/s) CDF Run-II Network:  CAF expansion – 160 workers; 20 servers; 10GE uplink  Off-line LAN upgrade – 10 GE uplink; GE for desktops D0 Run-II Network:  CAS switch fabric upgrade, incl. addtl 1000B-T ports  10 GE backbone, incl. HDCF & StarLight  Farms expansions – 280 CAB & 80 Recon nodes  On-line switch fabric upgrade & 1000B-T ports

Current WAN Status ESnet-funded off-site link carries production traffic  Currently, OC12 (622 Mb/s)  CMS robust data challenge: 10,000 Mb/s FNAL-funded StarLight dark fiber  Initial configuration: one 10 GE & two 1 GE channels  Theoretical capacity = > Thirty-three 10GE channels  Intended uses:  WAN network R & D projects  Overflow link for production WAN traffic  Redundancy for ESnet link

ESnet Off-Site link utilization Current link is “stressed”:  Carries all production traffic  Only an OC12 (622 Mb/s)  Traffic doubles every year  Outbound: blue = average; magenta = peak  Inbound: green = average; dark green = peak Traffic mostly Run-II data  CMS traffic beginning to appear ESnet upgrade of existing link is problematic

ESnet Networking Directions Three-tiered network architecture  Production (what we have today…)  High-impact (very large scale data movement)  Research (next generation networking & apps) Cost-sharing model for site connectivity  Labs expected to contribute to site link upgrades Metropolitan Area Network (MAN) initiative  Metro fiber rings supporting multiple Labs  ESnet proposing MANs for Bay Area & Chicago  Bay Area MAN partially funded in FY04  Chicago MAN not funded in FY04…

ESnet Chicago MAN proposal

FNAL Starlight dark fiber project Dark fiber to Starlight in place  StarLight = international optical network exchange pt.  Link optics w/ DWDM gear soon  Capable of multiple λs  Each λ can carry 2-10 Gb/s  But expensive, (O) $100k/ 10GE λ  Initially, one 10GE & one 2 GE λs  Currently: 1 Gb/s ‘til DWDM turn-on  Separate network infrastructure from production LAN path

Off-Site Link Upgrade Planning Primary emphasis is to get ESnet MAN deployed  Would provide both production & high impact paths  Pursuing fiber availability options for missing segment  In the end, the MAN needs DOE approval & funding Alternatives involve using StarLight link for production traffic  Carrying overflow traffic would be one possible mode  Keeping ESnet as our service provider remains long term goal  Want to avoid being in the production network WAN business

WAN Network R & D Projects Primary use of StarLight link intended to be R&D Practical benefits:  Environment for collaborators to develop & refine their applications for high rate data movement  Non-production (< five 9s…) facility for testing & introducing advanced network technologies  Can be utilized to support production network facilities  Redundancy; overflow traffic… Research benefits:  Develop services & tools for HEP research environment  LambdaStation: 2-3 year SciDac Project to facilitate selective path forwarding across advanced WAN research networks

StarLight Link R&D Projects Build it and they will come….. Project Local Affiliation Collaborating Site(s)StatusBW Needs Usage Pattern UKlightCDFUCLActive4 Gb/speriodically CMS/CERNCMS (OSG?)CERNActiveUp to 10 Gb/speriodically CMS/ LambdaStation CMSCal Tech/UCSDStart 10/2004Up to 10 Gb/ssustained Toronto HEPCDFU. TorontoInactive{undefined} SC2004 BW Challenge CDSC200411/06/2004Up to 10 Gb/sOne-time VanderbiltOSG(?)VanderbiltPending{undefined} WestGrid (Ca)D0Simon Fraser UPending4 Gb/ssustained LHC Tier 1CMSBNLPending< 622 Mb/s{undefined} UltraLightCMSCal Tech, UF, others PendingUp to 10 Gb/s{undefined}

Network support at FNAL - Effort Network FTE level has been static for past decade  Greater complexity & added services = increased demands on personnel resources  Continually working toward more efficient, automated procedures ~1.5 FTE to upgrade/service each Run-II network/year  Also a relatively static number for the past decade  Core network, services, WAN, security, etc. efforts not included…  Each experiment has liaison to coordinate network support… Emerging or significantly growing areas of effort  Advanced WAN (StarLight…) infra. & ops support  Network R&D (LambdaStation)  Computer security demands on network support