1. CERN - European Laboratory for Particle Physics HEP Computer Farms Frédéric Hemmer CERN Information Technology Division Physics Data processing Group

2. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 2 Outline n Offline analysis (current) n Quasi-on line analysis (2000) n Online analysis (2005)

3. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 3 Physics Data Processing evolution n 1990->1997 Migration from mainframe computing to Distributed RISC/Unix computing  Reduce Acquisition/Maintenance Costs  Decrease Price/Performance ratio but … system management costs becoming a serious issue n 1997->... Migration from Distributed RISC/Unix to PC (NT & Linux) technology  Reduce even further Acquisition/Maintenance Costs  Possible only starting with Ppro performance but new issues : OS, management model, stability, performance, technology evolution, etc..

4. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 4 Physics Data Processing evolution (II) n Before 1992 Manual tape transfer n 1992-1994 Central Data Recording (< 1 MB/s) n 1998 Computer Center part of the experiment CDR (20 MB/s) and online tagging n 2000 CDR (35 MB/s) and online filtering n 2005 CDR (100-1000 MB/s) and online filtering

5. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 5 Offline analysis NT PC Network Unix RFIO Server Server Server Server Unix Tape Server stagexxx commands RFIO

6. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 6 NT Simulation Facility : Goals n Make PC+NT a standard option for Physics Data Processing, starting with simulation n Establish a minimum management model for NT farm management n Address scalability issues n Gain Windows NT experience

7. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 7 Physically... 1997 1998

8. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 8 PCSF Usage

9. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 9

10. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 10 Key Issues n AFS access n LSF support n Boot proms, equipment interoperability n CODE reintegration (Physics & CERNLIB) n Think Windows n Scalability & Management (home grown solution vs. commercial apps.)

11. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 11 Next Steps n Finish and understand remote boot issues n Complete remote boot - remote install n AFS Integration n Build up resilience n Investigate how to use the new WfM, DMI, PXE, ACPI, etc. initiatives n Investigate whether WSH is an alternative n Investigate NT’s I/O capabilities

12. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 12 Conclusions n PC+NT has proven to work in batch environment, and is now an option for Physics Data Processing n Farm management is less of a concern after have built a few tools (alternatives would be to use SMS or TNG), but some work is still needed n Scalability has started to be addressed, but the relatively small number of nodes does not help here n Considerable NT experience has been gained

13. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 13 NCF Initial conclusions (Feb. 1998) n PC’s can be used for low I/O tasks n Confirmed with Simulation on PCSF NOMAD reconstruction on Linux NA45 reconstruction on NT n PC’s are not adequate now as disk servers  Mixed PC/RISC Unix clusters will be used for 1998

14. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 14 PC NT based Tape server n Goal: Attach SCSI/FC-AL tape drives to PC running NT and provide access to 100’s of TB through Gbit Ethernet/HiPPI in order to reduce server acquisition prices. Obtain good enough performance (Linux has been already demonstrated) on this platform. Is HPSS out of the game here ?

15. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 15 Disk performance (Feb. 1998) Linux striping has no effect 1 stream 2 stripes : 21 MB/s (22 max)1 stream 2 stripes : 21 MB/s (22 max) 1 stream 3 stripes : 21 MB/s (33 max)1 stream 3 stripes : 21 MB/s (33 max)

16. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 16 Memory bandwidth (lmbench)

17. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 17 PC NT based Disk server n Goal: Attach (RAID) disk drives to PC’s running NT and provide access to 10’s of TB through Gbit Ethernet/HiPPI in order to reduce server acquisition prices. Obtain good enough performance on this platform (including using Objectivity databases). Issues –Scalability, disk & network performance

18. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 18 Current & Future Data rates at CERN

19. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 19 NA48 setup (1999) Fast Ethernet Gigabit Ethernet HiPPI 4 * SUN E450 4 * SUN E450 4.5 TB Disk space EventBuilder Sub detector VME crates 7 KM 3Com 3900 HiPPI 3Com 9300 Gigabit Ethernet Fast Ethernet Cisco 5505 On/Offline On/Offline PC Farm

20. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 20 Compass (1999-2008) n Parameters 300 TB/year 5-20 TB disks 20000 CU = 200 PII@450 MHz 35 MB/s Objectivity n NT is considered for both computing and data serving.

21. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 21 CMS Trigger DAQ

22. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 22 Main challenges n Management/Control/Monitoring of the filter applications how to distribute the work (Symera, Clustor ?) how to manage 1000’s of tasks at every moment n Management/Control/Monitoring of the “computer system” itself could be 1000’s computer systems or one very large SMP or a combination

23. CERN - European Laboratory for Particle Physics MS Research, 27 January 1999 Frédéric Hemmer CERN-IT/PDP 23 NT Prototype n Modest farm exists in CPPM (Marseille): 4 Dual Ppro’s @ 200 Mhz Small SUN as injector Fast Ethernet switch n Regularly being scale-up at CERN using larger configurations : Gbit Ethernet 30 Processors