1. T. Bowcock A.Moreton, M.McCubbin CERN-IT 5/00

2. 29 May 2000CERN-IT T. Bowcock2 University of Liverpool MAP System COMPASS Grid Summary

3. 29 May 2000CERN-IT T. Bowcock3 MAP@Liverpool LHCb Experiment –CP violation –Rare B decays –signals of 10 3 to 10 6 Backgrounds –Potentially all 10 14 collisions/year! 32 2 About 1 10 12 BB produced/year

4. 29 May 2000CERN-IT T. Bowcock4 LHCb Experiment Vertex detector

5. 29 May 2000CERN-IT T. Bowcock5 LHCb Experiment Optimize the Detector Study the Backgrounds

6. 29 May 2000CERN-IT T. Bowcock6 Simulation Full GEANT3 simulation –Event takes of order 120-200s on a 400MHz PC Put together a simulation facility –Samples of 10 7 to 10 8 / year –Many times more passed through GEANT –Monte Carlo Array Processor –Similar or larger samples –10 9 institute/year Analysis, reprocessing

7. 29 May 2000CERN-IT T. Bowcock7 Philosophy Fixed Purpose (MC): simplicity Low Cost –No Gbit ethernet until price falls –Dont buy top of range processors –No SMP boards 1998/1999 –No tapes Develop architecture with future in mind –Minimum maintenance/development

8. 29 May 2000CERN-IT T. Bowcock8 Using MAP Disposable MC(throwaway!) Cost Write out ntuple/summary information I/O not really limited by architecture Events may be written out Small internal disks

9. 29 May 2000CERN-IT T. Bowcock9 Hardware 300 processors –400MHz PII –128 Mbytes memory –3 Gbytes disk/processor (IDE) –D-Link 100BaseT ethernet +hubs –commercial units custom boxes for packing and cooling –Total 600kChF inc 17.5% VAT 1998/1999 (Funding Jan 99). ITS Including installation and 3-yr next day on-site maintenance.

10. 29 May 2000CERN-IT T. Bowcock10 MAP-OS Linux –Originally RH5.2 (also tested 6.1) –Stripped to minimum On disk 180MBytes! –Will (with FCS) reinstall/upgrade itself –Access/security

11. 29 May 2000CERN-IT T. Bowcock11 View High Gflops/m 2 Old Mainframe Room Power supply (3 phase) 0.1MW max 50kw cooling

12. 29 May 2000CERN-IT T. Bowcock12 Architecture Master External Ethernet MAP Slaves Hub (Switch- 00) Hub (Switch - 00) 100BaseT

13. 29 May 2000CERN-IT T. Bowcock13 Design Features Mother boards/bios –No keyboard etc required on boot! Front panels –All connections except power Access to each PC via trolley on wheels Cheaper than patch panel! Very convenient. –Cooling (room air flow) 30kW required 50kW capacity Power cutoff installed Rack Mount –30/rack, easy to extract

14. 29 May 2000CERN-IT T. Bowcock14 Learned… Prototype Cables –Cheaper ethernet cables seem OK Would have been nice to have –On board power/heat sensing Dont really need power system –Daisy chain in groups of 5 –Transients can be huge!

15. 29 May 2000CERN-IT T. Bowcock15 Bad things happen… Catastrophic power failure –No UPS (original design had one) –4% needed manual intervention but no hardware failure Burn-in & 4 months of operation –1 power supply exploded –4 PCs with mother-board problems –5 HD failures (within 1 week of turn on) –NIC cards fail –Typically 1% nodes may have a problem

16. 29 May 2000CERN-IT T. Bowcock16 Flow Control System MAP-FCS –UDP level (frames) –solve packet-loss problem Bad hubs(D-Link) NIC Realtek clones with high failure rate –Broadcast system 4 Mbytes/s 300 (Master to Slaves) –Point to point on fail –Standard Mode Communication only with master –Control up to 10,000 PCs

17. 29 May 2000CERN-IT T. Bowcock17 Performance Jan/May 00 –15 million GEANT events for optimization –cf 250,000 possible at CERN –DELPHI events 500,000/day Trilinear Gauge Couplings, W-mass systematics –ATLAS, CDF, H1

18. 29 May 2000CERN-IT T. Bowcock18 User Interface to master only –Web/Grid interface –Security Submission script –Job Control File Sequential jobs, files to keep etc Quick and easy to use Statically linked executable Toolkit –Enables assembly/merging of 300 outputs

19. 29 May 2000CERN-IT T. Bowcock19 Search Analysis As a search-engine MAP architecture is ideal –Low search and recovery times –Chemistry Centre for Innovative Catalysis (JIF 00), promises world lead for Liverpool. –Bio-informatics Compute/search farms

20. 29 May 2000CERN-IT T. Bowcock20 Extending MAP Wish to store events –Part of our mindset (reevaluate?) With existing system –Build an analysis and storage system –Add on disk servers

21. 29 May 2000CERN-IT T. Bowcock21 COMPASS

22. 29 May 2000CERN-IT T. Bowcock22 COMPASS-99 DELL ITS

23. 29 May 2000CERN-IT T. Bowcock23 COMPASS-00 3Tbytes –On top of 1TByte MAP internal Rack Mounted Prototype of 40Tbyte system

24. 29 May 2000CERN-IT T. Bowcock24 COMPASS Low cost(25KCHf/Tbyte inc 17.5%VAT) –SCSI disks(10 50GByte) –Dual Redundant Power Supplies –No RAID backplane –No hotswap –750 MHz processors + 512MBytes memory –Linux –Act as MAP masters

25. 29 May 2000CERN-IT T. Bowcock25 COMPASS Have 3Tbytes of store for R&D on GRID and exploitation of MAP MAP & COMPASS are complementary… Originally requested 40TBytes of store –For H1, BaBar, ATLAS, DELPHI

26. 29 May 2000CERN-IT T. Bowcock26 MAP &COMPASS DST or processed data stored –From MAP Reprocessed/analysed locally –COMPASS Limit data movement off site –COMPASS farm in own right –Powerful analysis engines –Access from remote sites –Designed to, in parallel, analyse very large data sets (Data split nodes – June 00)

27. 29 May 2000CERN-IT T. Bowcock27 Data Transfer 2000-2003 Data transfer to/from –Liverpool-CERN/RAL –Liverpool-SLAC/FNAL High Speed link may be a waste of money –3MCHF for 2MBs line! –Quality of service –Probably not true in long term t Transfer disks

28. 29 May 2000CERN-IT T. Bowcock28 MAP-2001 Extension of existing architecture –Vast underestimate of amount of MC required –Extend to 1000 PCs 720 800MHz PIII with 72Gbyte disks 128MBytes memory Switched network (&higher quality!) Better NICs/(onboard?)

29. 29 May 2000CERN-IT T. Bowcock29 MAP-2001 Companies more willing to discuss COTS type architecture –Many selling BEOWULF systems –Even IBM! –ITS will provide a turnkey system including our version of MAP control

30. 29 May 2000CERN-IT T. Bowcock30 MAP-2001 Capability –Standard MAP mode –DST transfer –Search Engine –Interprocess communication –Large Internal Store Minimize network traffic Reprocessing

31. 29 May 2000CERN-IT T. Bowcock31 MAP-2001 Increase power by factor of 5 Aim for 1.5M LHCb events/day –Non-volatile 1 Tbyte/day –50Days internal store Use for reprocessing data Disk size will increase by calendar 2001 Multi-user and projects

32. 29 May 2000CERN-IT T. Bowcock32 Issues Authentication and Security Quality of Service Resource Allocation

33. 29 May 2000CERN-IT T. Bowcock33 Grid Adding Globus (June 2000) Access from CERN & –Cambridge University, JMU, Liverpool, RAL Remote submission

34. 29 May 2000CERN-IT T. Bowcock34 Grid 2005 Tier 1 T2 3 3 3 3 3 3 3 3 3 3 3 3 Tier 0 (CERN) 4444 3 3 ?????? T2

35. 29 May 2000CERN-IT T. Bowcock35 Grid-LHCb Aim to use MAP as an LHCb testbed –MC production –Data access –Analysis –UK and CERN sites –Interaction with RAL

36. 29 May 2000CERN-IT T. Bowcock36 Health Grid Virtual Population Laboratory –Co-proposed by Liverpool for a world scale met office for disease prediction in collaboration with WHO –Analysis power based on MAP 5000 PC system

37. 29 May 2000CERN-IT T. Bowcock37 Health Grid Community Health Surveillance –WAP, local data bases Information –statistics, Analysis –MAP like centres for Health Policy WHO Med Centre

38. 29 May 2000CERN-IT T. Bowcock38 Comments High Power MC systems vital for HEP –Do we have/plan enough for LHC? Cost and Techniques of Storage –Small groups cant afford/want HSM –Is tape obsolete? Problems for institutes not the same as for Tier 0/1 centres

39. 29 May 2000CERN-IT T. Bowcock39 Summary MAP fulfils its design goals (works!) –MAP-FCS control up to 10,000PCs Minimum manpower 0.5FTE to date –Maintenance and development COTS architecture a success –Low cost has its ups and downs! –MAP available off the shelf for HEP-MC Low cost high density storage server farm in prototype Grid enabled –Access from CERN – and UK HEP institutes soon MAP-2001 –Test a 1000PC farm for LHC