1. CERN - European Laboratory for Particle Physics DESY November 2, 1998 PC Farms at CERN Frédéric Hemmer CERN-IT/PDP

2. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 2 Disclaimer n This will cover farms which imply an involvement of CERN’s computer center. n There are other farms in strict online environments or “private” farms in building.

3. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 3 Overview n Off line farms Linux farms NT farms Issues n PC Technology & Performance n Online Farms & quasi online farms n Cost of ownership n Conclusions

4. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 4 Linux Farms - Nomad n Proof of concept in Summer 97 n Straight NQS port n SHIFT SW client port n CERNLIB port n NOMAD observed a quasi linearity with clock frequency compared to Alpha’s !!! I.e. Alpha@266 MHz = PII@266 MHz n Now 17 PC’s dual, 3 types of MB

5. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 5 Linux Farms - NA49 n NA49 already deployed privately a PC farm in their premises n Request a new farm to be deployed in order to benefit from the computer center infrastructure (people and equipment …) in 1 H98 n Trivial deployment, running with NQS n Most PC’s are branded PC’s (HP) n Now completely off RISC for CPU n 18 DUALS @ 300->400 MHz

6. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 6 NA49 Analysis - data access SONY DMS UnixServer UnixServer UnixServer CORETapeServers HiPPI From experiment 10-12 TB / month 1 month/year Manual Feed 100 GB Cartridges HPK260 HPK260 HPK260 HPK260 HPK260 FDDI 600 GB 1 Run PC PC PC PC PC PC 100BT SGIChallenge

7. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 7 Linux Farms (NA48) n NA48 was using the QSW CS/2 (128 proc.) n CS/2 overload -> investigate PC’s in late 97 n Installation of 12 Dual machines in 1Q98 and more...

8. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 8 Linux Issues n EEPRO 100 B MP crashes n AFS support (MP) n NFS support (MP) n Commercial software n Manufacturer support for Linux n Very few Linux experts

9. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 9 NT offline Farms n PCSF Simulation facility but … n COMPASS Evaluating & benchmarking technology

10. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 10 PCSF - Overview n Configuration n Applications n Data access n Specific work & solutions n Key issues n Conclusions

11. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 11 PCSF - Goals n Make PC+NT a standard option for Physics Data Processing, starting with simulation n Establish a minimum management model for NT farm management n Address scalability issues n Gain Windows NT experience

12. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 12 PCSF Milestones n Joined RD47 in Autumn 96 n Price inquiry issued in 12/96 n Hardware delivered 4/97 n Ready to use 6/97 n RD47 report 10/97 n Expansion 5/98

13. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 13 PCSF Configuration (1) n Server running NT 4.0 Server SP3 1 dual capable Ppro @ 200 MHz, 96 MB, with 9 GB data disk (with mirroring). LSF central queues. n Server running NT Terminal Server Beta 2 1 dual Ppro @ 200 MHz, 128 MB, with 4 GB data disk. Runs IIS 3.0 and is accessible from outside CERN. It also host the asp’s for Web access n Servers running NT 4.0 Workstation SP3 9 dual Ppro’s @ 200 MHz, 64 MB, 2*4GB 25 dual PII’s @ 300 MHz, 128 MB, 2*4GB All equipped with boot proms

14. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 14 PCSF Configuration (2) n Machines interconnected with 4 3com 3000 100BaseT switch n Display/Keyboard/Mouse connected to a Raritan multiplexor n PC Duo for remote admin access  There were problems with other products n All running LSF 3.0.  LSF 3.2 does not work, support weak n Completely integrated with NICE

15. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 15 Applications on PCSF n ATLAS Dice simulation n NA45 1996 reconstruction n CMS reconstruction with Objectivity being tested n LHCB simulation code ready n ATLAS reconstruction being ported n ATLAS/Marseille event filter prototype scalability tests

16. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 16 Data access NT PC Network Unix RFIO Server Server Server Server Unix Tape Server stagexxx commands RFIO

17. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 17 ATLAS Level 3 DAQ Processor Farm Event Builder  SFISFISFI Storage (100 MB/s) Readout Buffers 1 GB/s

18. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 18 ATLAS Event Filter n Testbed for evaluating algorithms & sizing n Architecture & simulation studies n Monitoring, system management, feedback, etc… n Interface prototypes (SFI, SFO) n Timescale : prototype -1 (I.e. end 98) n Status : sizing of an initial farm

19. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 19 PCSF Usage

20. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 20

21. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 21 Specific work so far n Installation (Remote Boot, Winstall, NICE replica’s, Install Server) n User codes, CERNLIB, SHIFT n Job Starter n PC MGR n WNTS n Web Interface

22. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 22 Installation n Disk cloning + change SID  Fastest method, but not very automated n Remote boot Remote boot install procedures with virtual disk Use unattended setup, installs Winstall and other things Third party packages installed through Winstall  boot prom support on some hardware

23. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 23 Porting n Usually porting code from Unix to NT is easy (NA45 code ported in 1 week) n Usually porting production environment from Unix to NT is difficult (shell scripts) n Porting build environment is difficult, better to use native tools (Dev Studio)  Mixing Unix and NT build environment, revision control, etc.

24. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 24 Jobstarter n Initially inherited from Unix LSF CERN JobStarter n Rewritten in C++, using PcMgrSvc for drive mapping n Check execution preconditions n Clean up normal and abnormal job end n Kill popup dialog windows  Excel & Winzip in batch

25. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 25 PcMgrSvc/Ctl n Checks Status of monitored processes/services Amount of scratch space Drive mapping(s) n Map/Unmap drives n Sync. with time servers n Generate alarms on request n Gets all parameters from registry

26. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 26 Web Interface n As a solution to Remote access from outside CERN Access from non NT hosts n Implemented as ASP’s with VB n Requires IIS on the server

27. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 27 Web Interface - authentication

28. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 28 Web Interface - Overview

29. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 29 Web Interface - bjobs

30. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 30 Web interface - bjobs result

31. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 31 Windows NT Terminal Server

32. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 32 Next Steps n Finish and understand remote boot issues n Complete remote boot - remote install n AFS Integration n Build up resilience n Investigate how to use the new WfM, DMI, PXE, ACPI, etc. initiatives n Investigate whether WSH is an alternative n Investigate NT’s I/O capabilities

33. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 33 Key Issues n AFS access n LSF support n Boot proms, equipment interoperability n CODE reintegration (Physics & CERNLIB) n Think Windows n Scalability & Management (home grown solution vs. commercial apps.) n Remote & external access

34. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 34 PC with NT n PC+NT has proven to work in batch environment, and is now an option for Physics Data Processing n Farm management is less of a concern after have built a few tools (alternatives would be to use SMS or TNG), but some work is still needed n Scalability has started to be addressed, but the relatively small number of nodes does not help here n Considerable NT experience has been gained

35. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 35 Issues so far n Linux EEPRO 100 B MP support Commercial software Manufacturer support Very few local Linux experts n NT AFS access LSF support Think Windows Remote and external access n PC Interoperability (cards/MB combination Remote Boot support

36. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 36 PC Technology evolution in 97 n Pentium Pro  Pentium II 50 % raw performance increase but 50 % cache performance reduction n SEC  new motherboards n 440 FX  440 LX (SDRAM, AGP) n Recent MB’s  embedded SCSI, E’net, VGA n 100 Mbit E’net switches standard, 1000 Mbit arriving

37. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 37 PC Technology evolution in 98 n Pentium II @300 MHz  Pentium Xeon @ 450 MHz MP support 50 % cache performance increase n Slot 2  new motherboards n 440 LX  440 BX, 440 NX (100 MHz, EDO) n Recent MB’s  No more available through Intel, TYAN n 1000 Mbit/s E’net switches standard, >> 1000 Mbit/s arriving

38. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 38 Racking evolution 1997 1998

39. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 39 At the back...

40. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 40 Console multiplexors

41. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 41 Fast Ethernet switches (Sep. 98)

42. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 42 Fast Ethernet Switches (Oct. 98)

43. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 43 At the back of Fast Ethernet Switches (Oct. 98)

44. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 44 Gigabit Ethernet Switches

45. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 45 Network performance: Results n PC’s interconnected through 100 BaseT 3Com 3000 switch n Repeated with other H/W n Half duplex behavior n Block size does not matter n Linux uses less CPU than NT  Good unidirectional performance  Disappointing CPU consumption on NT  Disappointing bi-directional performance

46. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 46 PC to PC Network performance

47. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 47 Network performance: issues n Unexplained 0.5 MB/s observed with some eepro100 versions on PCRD hardware, but OK on PCSF n Recent DEC E'net boards with chipset > 21140 give poor performance on Linux n Surprising results PC/Alpha

48. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 48 PC/Alpha Network performance

49. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 49 PC High Performance Networking HiPPI (5/98) n PII, 300 MHz, 440LX, SDRAM, Roadrunner to SGI O2000, 4 CPU, IRIX 6.4 n Transmit: 50 MB/s n Receive: 50 MB/s (53 MB/s with SMP) Gigabit Ethernet (10/98) n n PII, 400 MHz, 440 BX, 100 MHz SDRAM, PCI 32/33, Tigon I n n 1500 bytes/packet: 28 MB/s, 40% CPU n n 9000 bytes/packet, 90 MB/s, 90% CPU

50. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 50 Disk performance n PC’s connected to SEAGATE ST19171W using two Adaptec 2940 UW n NT needs a lot of tuning (default behavior is to swap data out!) n Block size, BIOS settings, EDO/FPM does not matter  Poor performance  Windows NT even worse  Memory bandwidth is suspected

51. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 51 Disk performance Striping has no effect 1 stream 2 stripes : 21 MB/s (22 max)1 stream 2 stripes : 21 MB/s (22 max) 1 stream 3 stripes : 21 MB/s (33 max)1 stream 3 stripes : 21 MB/s (33 max)

52. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 52 Disk performance: issues n Memory bandwidth suspected n Need to test with LX/SDRAM, BX SDRAM@100 Mhz n RISC PCI does not support variety of boards n Combined disk/network performance even worse : 5-6 MB/s on Linux

53. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 53 Memory bandwidth (lmbench)

54. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 54 Memory bandwidth (lmbench)

55. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 55 Technology issues n Technology evolves too fast (processors, chipsets, memory, motherboards, networking,...) Changing environment/interoperability issues Hard to maintain (obsolescence) New NIC’s, drivers Measurements valid only a few months  Difficult to establish stable environments n Wide variety of solutions  Some combinations work, other not n Local suppliers cannot help to solve problems

56. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 56 PC Performance summary n CPU performance fine n Network performance Some configurations do not work Some configurations can saturate Fast Ethernet Recent tests show excellent performance n Memory performance Now better than low-end RISC n Disk Performance disappointing n Linux better than NT

57. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 57 Online and quasi online farms n NA48 Data Recording n NA45 Data Recording in Objectivity

58. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 58 NA48 Central Data Recording Cisco 5505 3Com 3900 FDDI Fast Ethernet XLNT Gbit FDDI HiPPI GigaRouter 3Com 9300 Gigabit Ethernet HiPPI CS/2 2.5 TB Disk space SUN E450 500 GB Disk space Event Builder Online PC Farm Sub detector VME crates 7 KM Offline Offline PC Farm

59. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 59 NA 48 Data Recording in 98 n May  September 1998 n Raw Data on Tape 68 TB (1450 tapes, mainly 50 GB tapes) 12.5 TB Selected Reconstructed Data Total with 97 data : 96 TB n Average Data Rate : 18 MB/s (peaks @ 23 MB/s) n CDR system can do 40-50 MB/s; limitation is CPU Time available n Data recorded as files (4 million)

60. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 60 NA48 On Line Farm n 11 Subdetector PC’s (dual PII-266, 128 MB) n 8 Event Building PC’s (dual PII-266, 128 MB, 18 GB SCSI) n 4 CDR routing PC’s (dual PII-266, 64 MB, FDDI) n All running Linux n Software event building in the interburst gap n Optional Software Filter (tags data) n Send data to computer center (local disk buffers : 144 GB, 2 hours) n On CS/2 : L3 Filtering and tape writing

61. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 61 NA48 Plans for 1999 Fast Ethernet Gigabit Ethernet HiPPI 4 * SUN E450 4 * SUN E450 4.5 TB Disk space EventBuilder Sub detector VME crates 7 KM 3Com 3900 HiPPI 3Com 9300 Gigabit Ethernet Fast Ethernet Cisco 5505 On/Offline On/Offline PC Farm

62. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 62 NA45 Data Recording Fast Ethernet Gigabit Ethernet HiPPI 2 * SUN E450 2 * SUN E450 500 GB Disk space Event Builder On Line PC Farm Sub detector VME crates 7 KM 3Com 3900 HiPPI Gigabit Ethernet Fast Ethernet SCI 3Com 3900 3Com 9300 NA48 PCSF

63. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 63 NA45 Raw Data recording in Objectivity n October 98 ; November 98 n Estimated bandwidth : 15 MB/s n Processes translate Raw Data format to Objectivity n Database files (1.5 GB) are closed, then written on tape n Steering done using a set of perl scripts on the disk servers n On line filtering/reconstruction/calibration possible n Farm is running Windows NT n Reconstruction can use PCSF

64. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 64 Current & Future Data rates at CERN

65. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 65 Summary n On line PC farms are being used to record data at sensible rates (Linux) n Off line PC farms are being used for reconstruction/filtering/analysis (Linux/NT) n Still a lot to do on scalable farm management, global steering, CDR monitoring, etc..

66. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 66 PC Total Cost of Ownership Software not included Install labor not included Assumes 3 years lifetime

67. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 67 DEC 8400 (12-Way) Cost of Ownership Software & SW maintenance not included Assumes 5 years lifetime

68. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 68 General Conclusions (1) n PC’s are now used for online, quasi online and offline environments n The “offline” is now part of the online n The I/O is still done using RISC/Unix but recent MP Xeon may change this …

69. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 69 General Conclusions (2) n PC technology is moving very fast Good for performance Not so for stability, interoperability Not so for understanding issues n The general management of large farms is not solved but … Number of initiatives/standards/tools may help us here : WfM, DMI, PXE, ACPI, SMS, TNG, etc.

70. CERN - European Laboratory for Particle Physics DESY November 2, 1998 Frédéric Hemmer CERN-IT/PDP 70 General Conclusions (3) n Linux vs. NT … the battle is over Choose the one suitable to your application NT can be used Linux is usable (and offers more performance). n PC real costs are usually not well understood