Heavy Ion Physics Program of CMS Proposal for Offline Computing Charles F. Maguire Vanderbilt University for the US CMS HI Collaboration April 25, 2009 CMS-HI Meeting at Santa Fe

CMS-HI Meeting at Santa Fe Outline Impact of CMS-HI Research Plan CMS-HI Compute Model Guiding principles Actual implementation Computing Requirements Wide area networking Compute power and local area networking Disk and tape storage Capital Cost and Personnel Summary Offline Organization, Operations, and Oversight April 25, 2009 CMS-HI Meeting at Santa Fe

Impact of the CMS-HI Research Plan CMS-HI Research Program Goals Focus is on the unique advantages of the CMS detector for detecting high pT jets, Z0 bosons, quarkonia, D and B mesons Early studies at low luminosity will concentrate on establishing the global properties of heavy ion central collisions at the LHC Later high luminosity runs with sophisticated high level triggering will allow for in-depth rare probe investigations of strongly interacting matter Projected luminosity growth Only one HI run is known for certain, at the end of 2010 After 2010 run LHC may shut down for an extended period Conditioning work needed to achieve design beam energy This proposal assumes a simple 3-year luminosity growth model Computer hardware resource acquisition is tailored to the model April 25, 2009 CMS-HI Meeting at Santa Fe

Impact of the CMS-HI Research Plan Projected HI Luminosity and Data Acquistion for the LHC 2010-2012 CMS-HI Run Ave. L (cm-2s-1) Uptime (s) Events taken Raw data (TB) 2010 (FY11) 2.5 x 1025 105 1.0 x 107 22 2011 (FY12) 2.5 x 1026 5 x 105 2.5 x 107 110 2012 (FY13) 5.0 x 1026 106 5.0 x 107 225 Caveats 1) First year running may achieve greater luminosity and uptime resulting in factors of 2 or 3 more events taken than assumed here 2) Second year running may not occur in 2011, but may shift to 2012 3) Third year running is the planned “nominal” year case when the CMS DAQ writes at the design 225 MB/s for the planned 106 s of HI running April 25, 2009 CMS-HI Meeting at Santa Fe

CMS-HI Meeting at Santa Fe CMS-HI Compute Model CMS-HI Compute Model Guiding Principles CMS-HI computing will follow, as much as feasible, the existing design and framework of CMS computing TDR (2005) CMS-HI community is much too small to embark on independent software development outside the mainstream of rest of CMS Size of CMS-HI community also mandates that we adapt the CMS multi-tiered computing grid to be optimum for our work CMS-HI Compute Model Implementation Raw data to be tranferred to and tape-archived at Vanderbilt site Reconstruction passes will also be done at Vanderbilt site Some reconstruction output will be copied to overseas sites (Moscow, Paris, Budapest, Seoul) as is practical Analysis passes will be done by all CMS-HI institutions using CMS’s remote job batch submission system (CRAB) Simulation production and support will be done at MIT site April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Wide Area Networking Nominal Year Running Specifications for HI CMS DAQ writes at 225 MB/s for 106 s = 225 TB Calibration and fast reconstruction at Tier0 = 75 TB Total nominal year data transfer from Tier0 = 300 TB Note: DAQ may be able to write faster eventually Nominal Year Raw Data Transport Scenario Raw data can reside on Tier0 disk buffers for only a brief time Tier0 holds data only a few days: calibrations, preliminary reco Data are written to Tier0 tape archive, not designed for re-reads Above mandates a continuous transfer of data to a remote site 300 TB x 8 bits/Byte / (30 days x 24 hours/day x 3600 sec/hour) = 0.93 Gbps DC rate (no outages) Same rate calculation as for pp data except pp runs for ~5 month A safety margin must be provided for outages, e.g. 4 Gbps burst April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Wide Area Networking Nominal Year Raw Data Transport Scenario Tier0 criteria mandate a continuous transfer of data to a remote site 300 TB x 8 bits/Byte / (30 days x 24 hours/day x 3600 sec/hour) = 0.93 Gbps DC rate (no outages) A safety margin must be provided for outages, e.g. 4 Gbps burst Plan is to transfer this raw data to the Vanderbilt tape archive site Raw Data Transport Network Proposal for CMS-HI CMS-HEP and ATLAS will use USLHCNet to FNAL and BNL FNAL estimates that CMS-HI traffic will be ~2% of all USLHCNet Propose to use USLHCNet with modest pro-rated cost to DOE-NP Have explored Internet2 alternatives to the use of USLHCNet CMS management strongly discourages use of a new raw data path A new raw data path would have to be solely supported by CMS-HI It is not obvious that there would be any cost savings to DOE-NP April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Annual Compute Power Budget for All Tasks Annual Compute Tasks for Available Compute Power Reconstruction passes (CMS standard is 2) Analysis passes (scaled to take 50% of reconstruction time) Simulation production and analysis (takes 50% of reco time) Simulation event samples at 5% of real event totals (RHIC experience) Simulation event generation and processing takes 10x that of real event Constraint: Accomplish All Processing in One Year Offline processing must keep up with the annual data streams Would like to process all the data within one year, on average Essential to have analysis guidance for future running Computer Power 12 Month Allocation A single, complete reconstruction pass will take 4 months This proposal will allow for 1.5 reconstruction passes = 6 months Analysis passes of these reconstruction passes in 3 months Simulation production and analysis in 3 months April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Compute Time for Single Reconstruction Pass Nominal year DAQ HI Bandwidth Partition and Associated CPU Times Channel BW (MB/s) Size/Evt (MB) Rate (Hz) CPU time/evt (s) Annual total events Annual CPU time (s) Min Bias 33.75 2.5 13.5 100 1.35 x 107 1.35 x 109 Jet-100 24.75 5.8 4.27 450 4.27 x 106 1.92 x 109 Jet-75 27.00 5.7 4.74 4.74 x 106 2.13 x 109 Jet-50 27.50 5.4 5.00 5.00 x 106 2.25 x 109 J/y 67.50 4.9 13.78 1000 1.38 x 107 1.38 x 1010 Y 2.25 0.46 4.59 x 105 4.59 x 108 e-g-10 40.50 6.98 6.98 x 106 3.14 x 109 Ultra-per 1.0 2.25 x 106 1.25 x 109 Sum 225 51 x 106 Evts/yr 2.53 x 1010 seconds/yr CPU times from the jet-g simulation study, and scaled to a 1600 SpecInt2000 processor April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Net Compute Power Requirement Determination of the Total CPU Number Ncpu A complete reconstruction pass takes 25 x 109 seconds (scaled to a 1600 SpecInt2000 processor) A single reconstruction pass must be completed in 4 months Assume that there is an effective duty cycle of 80% The US CMS-HI Compute Center is Set for 3,000 CPUs Real data reconstruction and analyses will consume 75% of the annual integrated compute power, i.e. ~2,250 CPUs Simulation production and analyses will consume 25% of the annual integrated compute power, i.e. ~750 CPUs A satellite simulation center is proposed at the MIT Tier2 CMS faciltiy, taking advantage of that local support and opportunistic cycles April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Annual Allocations of CMS-HI Compute Power Distribution of Compute Power For 3,000 CPUs Processing Task Duration (months) Total CPU Use (CPU-months) Partial reconstruction (50% of events) 2 6,000 Partial analysis pass (50% of events) 1 3,000 Complete reconstruction pass 4 12,000 Complete analysis pass Simulation generation and analysis 3 9,000 Total 12 36,000 Simulation generation and analysis can be done asynchronously with real data analysis, i.e. 750 CPUs x 12 months = 9,000 CPU-months Comparison of CMS-HI Compute Center with CMS-HEP Compute Centers 1) CMS-HI with 3000 CPUs @ 1600 SpecInt2000 = 4.8 MSpecInt2000 2) CMS-HEP with 7 Tier1 and 36 Tier2 has a quota of 49.2 MSpecInt2000 (from TDR-2005) 3) 10% relative size scales with 10% of running and raw data output April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Local Disk Storage April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Tape Storage April 25, 2009 CMS-HI Meeting at Santa Fe

Capital Cost and Personnel Summary Impact of CMS-HI Research Plan CMS-HI Compute Model Integration into existing CMS compute model Service role to the total CMS-HI community Computing Requirements Wide area networking Compute power and local area networking Disk and tape storage Capital Cost and Personnel Summary Offline Organization, Operations, and Oversight April 25, 2009 CMS-HI Meeting at Santa Fe

Organization, Operations, and Oversight Impact of CMS-HI Research Plan CMS-HI Compute Model Integration into existing CMS compute model Service role to the total CMS-HI community Computing Requirements Wide area networking Compute power and local area networking Disk and tape storage Capital Cost and Personnel Summary Offline Organization, Operations, and Oversight April 25, 2009 CMS-HI Meeting at Santa Fe