1. Heavy Ion Physics Program of CMS Proposal for Offline Computing
Charles F. Maguire
Vanderbilt University
for the US CMS HI Collaboration
April 25, 2009
CMS-HI Meeting at Santa Fe

2. CMS-HI Meeting at Santa Fe
Outline
Impact of CMS-HI Research Plan
CMS-HI Compute Model
Guiding principles
Actual implementation
Computing Requirements
Wide area networking
Compute power and local area networking
Disk and tape storage
Capital Cost and Personnel Summary
Offline Organization, Operations, and Oversight
April 25, 2009
CMS-HI Meeting at Santa Fe

3. Impact of the CMS-HI Research Plan
CMS-HI Research Program Goals
Focus is on the unique advantages of the CMS detector for detecting high pT jets, Z0 bosons, quarkonia, D and B mesons
Early studies at low luminosity will concentrate on establishing the global properties of heavy ion central collisions at the LHC
Later high luminosity runs with sophisticated high level triggering will allow for in-depth rare probe investigations of strongly interacting matter
Projected luminosity growth
Only one HI run is known for certain, at the end of 2010
After 2010 run LHC may shut down for an extended period Conditioning work needed to achieve design beam energy
This proposal assumes a simple 3-year luminosity growth model
Computer hardware resource acquisition is tailored to the model
April 25, 2009
CMS-HI Meeting at Santa Fe

4. Impact of the CMS-HI Research Plan
Projected HI Luminosity and Data Acquistion for the LHC
CMS-HI Run
Ave. L (cm-2s-1)
Uptime (s)
Events taken
Raw data (TB)
2010 (FY11)
2.5 x 1025
105
1.0 x 107 22
2011 (FY12)
2.5 x 1026
5 x 105
2.5 x 107
110
2012 (FY13)
5.0 x 1026
106
5.0 x 107
225
Caveats
1) First year running may achieve greater luminosity and uptime resulting in factors of 2 or 3 more events taken than assumed here
2) Second year running may not occur in 2011, but may shift to 2012
3) Third year running is the planned “nominal” year case when the CMS DAQ writes at the design 225 MB/s for the planned 106 s of HI running
April 25, 2009
CMS-HI Meeting at Santa Fe

5. CMS-HI Meeting at Santa Fe
CMS-HI Compute Model
CMS-HI Compute Model Guiding Principles
CMS-HI computing will follow, as much as feasible, the existing design and framework of CMS computing TDR (2005)
CMS-HI community is much too small to embark on independent software development outside the mainstream of rest of CMS
Size of CMS-HI community also mandates that we adapt the CMS multi-tiered computing grid to be optimum for our work
CMS-HI Compute Model Implementation
Raw data to be tranferred to and tape-archived at Vanderbilt site
Reconstruction passes will also be done at Vanderbilt site
Some reconstruction output will be copied to overseas sites (Moscow, Paris, Budapest, Seoul) as is practical
Analysis passes will be done by all CMS-HI institutions using CMS’s remote job batch submission system (CRAB)
Simulation production and support will be done at MIT site
April 25, 2009
CMS-HI Meeting at Santa Fe

6. Computing Requirements: Wide Area Networking
Nominal Year Running Specifications for HI
CMS DAQ writes at 225 MB/s for 106 s = 225 TB
Calibration and fast reconstruction at Tier0 = 75 TB
Total nominal year data transfer from Tier0 = 300 TB
Note: DAQ may be able to write faster eventually
Nominal Year Raw Data Transport Scenario
Raw data can reside on Tier0 disk buffers for only a brief time
Tier0 holds data only a few days: calibrations, preliminary reco
Data are written to Tier0 tape archive, not designed for re-reads
Above mandates a continuous transfer of data to a remote site 300 TB x 8 bits/Byte / (30 days x 24 hours/day x 3600 sec/hour) = Gbps DC rate (no outages)
Same rate calculation as for pp data except pp runs for ~5 month
A safety margin must be provided for outages, e.g. 4 Gbps burst
April 25, 2009
CMS-HI Meeting at Santa Fe

7. Computing Requirements: Wide Area Networking
Nominal Year Raw Data Transport Scenario
Tier0 criteria mandate a continuous transfer of data to a remote site 300 TB x 8 bits/Byte / (30 days x 24 hours/day x 3600 sec/hour) = Gbps DC rate (no outages)
A safety margin must be provided for outages, e.g. 4 Gbps burst
Plan is to transfer this raw data to the Vanderbilt tape archive site
Raw Data Transport Network Proposal for CMS-HI
CMS-HEP and ATLAS will use USLHCNet to FNAL and BNL
FNAL estimates that CMS-HI traffic will be ~2% of all USLHCNet
Propose to use USLHCNet with modest pro-rated cost to DOE-NP
Have explored Internet2 alternatives to the use of USLHCNet
CMS management strongly discourages use of a new raw data path
A new raw data path would have to be solely supported by CMS-HI
It is not obvious that there would be any cost savings to DOE-NP
April 25, 2009
CMS-HI Meeting at Santa Fe

8. Computing Requirements: Annual Compute Power Budget for All Tasks
Annual Compute Tasks for Available Compute Power
Reconstruction passes (CMS standard is 2)
Analysis passes (scaled to take 50% of reconstruction time)
Simulation production and analysis (takes 50% of reco time)
Simulation event samples at 5% of real event totals (RHIC experience)
Simulation event generation and processing takes 10x that of real event
Constraint: Accomplish All Processing in One Year
Offline processing must keep up with the annual data streams Would like to process all the data within one year, on average
Essential to have analysis guidance for future running
Computer Power 12 Month Allocation
A single, complete reconstruction pass will take 4 months This proposal will allow for 1.5 reconstruction passes = 6 months
Analysis passes of these reconstruction passes in 3 months
Simulation production and analysis in 3 months
April 25, 2009
CMS-HI Meeting at Santa Fe

9. Computing Requirements: Compute Time for Single Reconstruction Pass
Nominal year DAQ HI Bandwidth Partition and Associated CPU Times
Channel
BW (MB/s)
Size/Evt (MB)
Rate (Hz)
CPU time/evt (s)
Annual total events
Annual CPU time (s)
Min Bias
33.75
2.5
13.5
100
1.35 x 107
1.35 x 109
Jet-100
24.75
5.8
4.27
450
4.27 x 106
1.92 x 109
Jet-75
27.00
5.7
4.74
4.74 x 106
2.13 x 109
Jet-50
27.50
5.4
5.00
5.00 x 106
2.25 x 109
J/y
67.50
4.9
13.78
1000
1.38 x 107
1.38 x 1010 Y
2.25
0.46
4.59 x 105
4.59 x 108
e-g-10
40.50
6.98
6.98 x 106
3.14 x 109
Ultra-per
1.0
2.25 x 106
1.25 x 109
Sum
225
51 x 106
Evts/yr
2.53 x 1010
seconds/yr
CPU times from the jet-g simulation study, and scaled to a 1600 SpecInt2000 processor
April 25, 2009
CMS-HI Meeting at Santa Fe

10. Computing Requirements: Net Compute Power Requirement
Determination of the Total CPU Number Ncpu
A complete reconstruction pass takes 25 x 109 seconds (scaled to a 1600 SpecInt2000 processor)
A single reconstruction pass must be completed in 4 months
Assume that there is an effective duty cycle of 80%
The US CMS-HI Compute Center is Set for 3,000 CPUs
Real data reconstruction and analyses will consume 75% of the annual integrated compute power, i.e. ~2,250 CPUs
Simulation production and analyses will consume 25% of the annual integrated compute power, i.e. ~750 CPUs
A satellite simulation center is proposed at the MIT Tier2 CMS faciltiy, taking advantage of that local support and opportunistic cycles
April 25, 2009
CMS-HI Meeting at Santa Fe

11. Computing Requirements: Annual Allocations of CMS-HI Compute Power
Distribution of Compute Power For 3,000 CPUs
Processing Task
Duration (months)
Total CPU Use
(CPU-months)
Partial reconstruction (50% of events) 2
6,000
Partial analysis pass (50% of events) 1
3,000
Complete reconstruction pass 4
12,000
Complete analysis pass
Simulation generation and analysis 3
9,000
Total 12
36,000
Simulation generation and analysis can be done asynchronously with real data analysis, i.e. 750 CPUs x 12 months = 9,000 CPU-months
Comparison of CMS-HI Compute Center with CMS-HEP Compute Centers
1) CMS-HI with SpecInt2000 = 4.8 MSpecInt2000
2) CMS-HEP with 7 Tier1 and 36 Tier2 has a quota of 49.2 MSpecInt2000 (from TDR-2005)
3) 10% relative size scales with 10% of running and raw data output
April 25, 2009
CMS-HI Meeting at Santa Fe

12. Computing Requirements: Local Disk Storage
April 25, 2009
CMS-HI Meeting at Santa Fe

13. Computing Requirements: Tape Storage
April 25, 2009
CMS-HI Meeting at Santa Fe

14. Capital Cost and Personnel Summary
Impact of CMS-HI Research Plan
CMS-HI Compute Model
Integration into existing CMS compute model
Service role to the total CMS-HI community
Computing Requirements
Wide area networking
Compute power and local area networking
Disk and tape storage
Capital Cost and Personnel Summary
Offline Organization, Operations, and Oversight
April 25, 2009
CMS-HI Meeting at Santa Fe

15. Organization, Operations, and Oversight
Impact of CMS-HI Research Plan
CMS-HI Compute Model
Integration into existing CMS compute model
Service role to the total CMS-HI community
Computing Requirements
Wide area networking
Compute power and local area networking
Disk and tape storage
Capital Cost and Personnel Summary
Offline Organization, Operations, and Oversight
April 25, 2009
CMS-HI Meeting at Santa Fe