1. S.Jarp CERN openlab 2003 1 CERN openlab Total Cost of Ownership 11 November 2003 Sverre Jarp

2. S.Jarp CERN openlab 2003 2 Agenda Day 1 (IT Amphitheatre) Introductory talks: 09:00 – 09:30 Welcome. (W. von Rüden) 09:30 – 10:15 Setting the scene: A Global View of the plans for the LHC Computing Grid – (L.Robertson) 10:15 – 10:45 Coffee break Part 2: 10:45 – 11:15 Review of calculated cost summary for 2003 (S. Jarp) 11:15 – 12:00 Estimated LCG Personnel and Materials costs (2004 – 2008) (B.Panzer) 12:30 – 14:00 Lunch Part 3 14:00 – 14:45 Planning the LCG Fabric at CERN (T. Cass) 14:45 – 15:30 External Testimony: Michael Levine (Pittsburgh Supercomputer Center) 15:30 – 16:00 Coffee break 16:00 – Discussions and conclusion of day 1 (All) Day 2 (IT Amphitheatre MORNING, 513 1-024 afternoon) Discussions with HP: 08:45 – 10:45 One-on-one with HP Discussions with Intel: 11:00 – 13:00 One-on-one with Intel Discussions with IBM: 14:00 – 16:00 One-on-one with IBM

3. S.Jarp CERN openlab 2003 3 AIM Understand Total Cost of Ownership “Linux fabric” in the Computer Centre CPU servers Disk servers, Tape servers + Silos To a “sufficient” level of detail 2003 (July/August) Budget situation Rather than actual sums across all cost centres Personnel + Materials No direct measurement of quality Nor the fraction of capacity actually in use at a given time

4. S.Jarp CERN openlab 2003 4 Cost of manpower Complicated issue Approach chosen: Take IT salary budget for staff and fellows Including direct benefits (pension fund, health care, etc.) Without counting management overhead Neither inside IT nor CERN Actual number: 168.8 KCHF Excluded: Associates + Students In general, paid from the materials budget Associates: Some do not necessarily spend most of their time at CERN Students: Avoid very short stays (Summer students)

5. S.Jarp CERN openlab 2003 5 IT budget categories In the TCO exercise the total IT budgets were initially split into 9 categories: Computer Centre Services: Cost of the Physics Fabric and closely related services, such as internal and external networking, etc. Grid Development/Deployment Openlab Computing Infrastructure (non-physics oriented): Windows, Mail, Web services, Phone exchange, etc. Product Support (mainly for Engineers): Digital electronics, CAD, license office, Solaris support. etc. Controls for physics experiment so-called “on-line computing” Database Services and Applications Communications Infrastructure for the LHC Accelerator Divisional Management

6. S.Jarp CERN openlab 2003 6 IT Division (total) Materials (KCHF) FTE% of IT total (Materials + FTEs) (Basic) Computer Centre Services12,48679.4540.4 Grid Development and Deployment 15032.63 8.8 Openlab --- 3.75 1.0 Infrastructure (non-physics) 4,71434.6016.4 Product Support 1,95417.73 7.7 Controls 68618.95 6.1 Databases and Applications 1,66215.65 6.7 Accelerator Communications 3,294 3.50 6.1 Divisional Management 79621.35 6.9 Total25,742227.6 100.0

7. S.Jarp CERN openlab 2003 7 Box count Central Linux Batch “white box” servers1490 Central IDE Disk servers 250 Central PC Tape servers 60 Linux systems in the Computer Centre1800 Linux systems outside the computer centre 1600 Central Windows servers 150 Central Solaris servers 50 IP devices on Campus:28K

8. S.Jarp CERN openlab 2003 8 Capacity Numbers Other relevant capacity numbers are: Total installed PC CPU capacity: ~1.2 Million SpecInt2000 (SI2K) CPU capacity purchased by August: 400 * 1.6 KSI2K = 640 KSI2K Total disk space: 200 TB Disk space acquired by August: 50 TB at required performance (actually: 65 TB) Tape silos installed: 10 (with 6000 slots each) Tape drives: 50 STK9940B (+ 20 STK9840 with low usage)

9. S.Jarp CERN openlab 2003 9 Cost of Central Services Batch/Interactive services: Acquisition costs for batch/interactive servers, software licenses and maintenance Managed storage: Tape and disk servers (purchase and maintenance) Computer Centre Infrastructure: Various contract staff contracts; Monitoring and Supervision; Remedy Fabrics: The budget line only covers the CC upgrade this year. The two "Operations" lines: Cover costs associated with the various groups (travel, associates/consultants, workstation purchases, etc.)

10. S.Jarp CERN openlab 2003 10 Cost of Central Services

11. S.Jarp CERN openlab 2003 11 Cost of Fabric

12. S.Jarp CERN openlab 2003 12 Costs beyond the fabric In many cases non-fabric-related costs dominate Campus Network: 2991 KCHF User Support and Quality Insurance: 1265 KCHF Computer security: 741 KCHF Etc. In other words: The fabric benefits from large demands on other services

13. S.Jarp CERN openlab 2003 13 CPU part of fabric

14. S.Jarp CERN openlab 2003 14 Disk/tape part of fabric

15. S.Jarp CERN openlab 2003 15 Summary All-in-all: IT estimates (on average) that it has spent ~(1 + 1) MCHF on basic hardware across CPU servers Disk servers + Tape servers Our calculation shows: Linux fabric in total: 18.6 MCHF CPU part: 10.5 MCHF Disk/tape part: 6.3 MCHF External networking: 1.8 MCHF Cost on top of basic hardware: 14.8 / 2 = 7.4

16. S.Jarp CERN openlab 2003 16 Backup

17. S.Jarp CERN openlab 2003 17 2003 (replacement) cost IT main entities Acquisition (CHF) Installation, Maintenance (CHF) Quantity (per unit) CERN totalAcquisition (so far) 2003 CPU server2,0003001600 SI2K1.2 MSI2K400 Disk server10,0002,0001 TB (1.3)200 TB50 Tape cartridge 130-200 GB Tape silo150,000 +6,000 slots10- Tape drives (9940B/9840) 30,000/N.A.4500 3000 30/20 MB/s50 + 20- AFS home/project 80,0001 TB4 TB AFS scratch10,0002,0001 TB