1. AMS Computing Y2001-Y2002 AMS Technical Interchange Meeting MIT Jan 22-25, 2002 Vitali Choutko, Alexei Klimentov

2. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 20022 Outline AMS Production Farm  requirements  architecture  prototyping  test of HW and SW components HW and SW evaluation for AMS02 Ground Segment Data Transmission SW Y2002 Milestones

3. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 20023 AMS Ground Centers Science Operations Center POCC POIC@MSFC AL AMS Remote center RT data Commanding Monitoring NRT Analysis NRT Data Processing Primary storage Archiving Distribution Science Analysis MC production Data mirror archiving External Communications Science Operations Center XTerm HOSC Web Server and xterm TReK WS commands Monitoring, H&S data Flight Ancillary data AMS science data (selected) TReK WS “ voice”loop Video distribution Production Farm Analysis Facilities PC Farm Data Server Analysis Facilities GSE D S A e T r A v e r GSE Buffer data Retransmit To SOC AMS Station AMS Station AMS Station GSE MC production cmds archive AMS Data, NASA data, metadata

4. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 20024 AMS Production Farm (requirements) Reliability – High (24h/day, 7days/week) Performance goal – process data “quasi-online” (with typical delay < 1 day) Disk Space – 12 months data “online” Minimal human intervention (automatic data handling, job control and book-keeping) System stability – months Scalability Price/Performance Complex system that consists of computing components including I/O nodes, worker nodes, data storage and networking switches. It should perform as a single system. Requirements :

5. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 20025 AMS Production Farm (considerations) Uniform node architecture ( dual-CPU Pentiums and AMDs) Uniform Operating System (RedHat Linux) Computing capacity equivalent to 400x450MHz PII processors (including 20% contingency and reprocessing) Total of 10 Tbyte data stored online Two types of computers : Considerations based on AMS01 data processing experience and MC production Y2000-2001 : “ Processing node” with cheap IDE disks used for transient data storage “Server node” with IDE and SCSI RAID disks for persistent data storage

6. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 20026 Y2001 milestones HW evaluation to make a choice of platform and architecture (“official” AMS02 simulation/reconstruction code been used for the benchmarking ) Functional Goal : AMS01 STS91 Data Rerun and AMS02 MC production using production farm prototype and SW

7. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 20027 AMS02 Benchmarks Executive time of AMS “standard” job compare to CPU clock 1) V.Choutko, A.Klimentov AMS note 2001-11-01 1) Brand, CPU, Memory Intel PII dual-CPU 450 MHz, 512 MB RAM OS/Compiler RH Linux 6.2 / gcc 2.95 “Sim” 1 “Rec” 1 Intel PIII dual-CPU 933 MHz, 512 MB RAMRH Linux 6.2 / gcc 2.950.54 Compaq, Quad α-ev67 600 MHz, 2 GB RAMRH Linux 6.2 / gcc 2.950.580.59 AMD Athlon,1.2GHz, 256 MB RAMRH Linux 6.2 / gcc 2.950.390.34 Intel Pentium IV 1.5GHz, 256 MB RAMRH Linux 6.2 / gcc 2.950.440.58 Compaq dual-CPU PIV Xeon 1.7GHz, 2GB RAMRH Linux 6.2 / gcc 2.950.320.39 Compaq dual α-ev68 866MHz, 2GB RAMTru64 Unix/ cxx 6.20.230.25 Elonex Intel dual-CPU PIV Xeon 2GHz, 1GB RAMRH Linux 7.2 / gcc 2.950.290.35 AMD Athlon 1800MP, dual-CPU 1.53GHz, 1GB RAM RH Linux 7.2 / gcc 2.950.240.23 8 CPU SUN-Fire-880, 750MHz, 8GB RAMSolaris 5.8/C++ 5.20.520.45 24 CPU Sun Ultrasparc-III+, 900MHz, 96GB RAMRH Linux 6.2 / gcc 2.950.430.39 Compaq α-ev68 dual 866MHz, 2GB RAMRH Linux 7.1 / gcc 2.950.220.23

8. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 20028 AMS01 STS91 Data Rerun (performance)

9. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 20029 AMS02 Benchmarks (summary) α-ev68 866MHz and AMD Athlon MP 1800+ have nearly the same performance and are the best candidates for “AMS processing node” (the price of system based on α-ev68 is twice higher than the similar one based on AMD Athlon) Though PIV Xeon has lower performance, resulting 15% overhead comparing with AMD Athlon MP 1800+, the requirements of high reliability for “AMS server node” dictates the choice of Pentium machine. SUN and COMPAQ SMP might be the candidates for AMS analysis computer (the choice is postponed up to L-12 months) Conclusion : The total power of AMS02 processing farm must be equivalent to 50 AMD Athlon MP 1800+ computers.

10. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200210 Production Farm (“AMS processing node” architecture) Processor Chip set Memory System Disk Disk Controller Disks (transient storage) Ethernet Adapters “public” “AMS private” dual-CPU 1.5+GHz currently AMD 1 GB RAM LVD SCSI 3Ware IDE RAID 6x120+GB IDE 100 Mbit/sec 2x1 GBit/sec

11. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200211 Production Farm (“AMS server node” architecture) Processor Chip set Memory System Disk Disk Controller Disks (permanent storage) Disk Controller Disks (transient storage) Ethernet Adapters “public” “AMS private” dual-CPU 1.4+GHz currently Intel 1 GB RAM LVD SCSI IPC SCSI RAID 8x180+GB SCSI 3Ware IDE RAID 7x120+GB IDE 100 Mbit/sec 2x1 GBit/sec

12. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200212 Production Farm HW  Tape Drive (“raw” data backup) IBM LTO Ultrium ( connected to “server node” prototype ) data transfer (write) RAID 5 array -> tape 11MByte/sec data transfer (read) tape -> Null device 19MByte/sec tape -> RAID 5 array 11MByte/sec tape capacity 200GB (see also http://cscct.home.cern.ch/cscct/ultrium)

13. 13 Archiving and Staging Analysis Facilities Data Server Cell #1 #2 #8 PC Linux 2x2GHz+ PC Linux 2x2GHz+ PC Linux 2x2GHz+ PC Linux 2x2GHz+ PC Linux 2x2GHz+ Tape Server PC Linux 2x2GHz+ PC Linux Server 2x2GHz, SCSI RAID Tape Server Disk Server Disk Server Disk Server Gigabit Switch (1 Gbit/sec) PC Linux 2x2GHz+ 2xSMP, (Q, SUN) AMS data NASA data metadata AMS Science Operation Center Computing Facilities Production Farm Disk Server Disk Server Simulated data A.Klimentov Jan 15,2002 MC Data Server

14. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200214

15. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200215 AMS Computing Y2001 (SW)  AMS production process/process communication and control SW (PPCC) and monitoring  Data Handling ORACLE DB to store metadata and catalogues (M.Boschini, A.Klimentov)  Data transmission package Client/Server Corba technology (V.Choutko) Process Monitoring package (M.Boschini, V.Choutko, A.Klimentov) Based on bbftp (A.Elin, A.Klimentov AMS note 2001-11-02)

16. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200216 AMS Production Highlights Excellent HW stability ( uptime more than 3 months) AMS01 STS91 data rerun (10 Linux boxes, 19 CPUs) Average efficiency 95% (cpu time/elapsed time) Processes communication and control via Corba LSF for process submission Oracle server on AS4100 Alpha and Oracle clients on Linux. Oracle RDBMS  Tag DB with 100M entries  Conditions DB with 100K entries  Bookkeeping  Production status  Runs history  File catalogues

17. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200217 Data Transmission SW High Rate Data Transfer between MSFC and POCC/SOC, POCC and SOC, SOC and MasterCopy repositary(s) will become a paramount importance (tests with TReK between MIT and CERN, TReK is the best candidate for AMS commanding and transferring of data samples) What should be used for the bulk data transfer ? Why not FileTransferProtocol (ftp) or ncftp, etc ? to speed up data transfer to encrypt sensitive data and not encrypt bulk data to run in batch mode with automatic retry in case of failure … starting to look around and came up with bbftp in September (bbftp developed in BaBar and used to transmit data from SLAC to IN2P3@Lyon) adapted it for AMS, wrote service and control programs 1) 1) A.Elin, A.Klimentov AMS note 2001-11-02 2) P.Fisher, A.Klimentov AMS Note 2001-05-02

18. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200218 Data Transmission SW (the inside details) Server  copy data files between directories (optional)  scan data directories and make list of files to be transmitted  purge successfully transmitted files and do book-keeping of transmission sessions Client  periodically connect to server and check if new data available  bbftp new data and update transmission status in the catalogues.

19. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200219 Data Transmission SW (tests) LocationLine Mbit/sec ProgramRate Mbit/sec Prevessin ->Meyrin10ftp bbftp bbcp 5.8 7.8 8.0 Prevessin -> Prevessin100ftp bbftp bbcp 21.0 40.0 42.0 Prevessin -> Milano16bbftp6.0 Server and client – dual-CPU Intel PIII, Linux OS. bbftp release 2.1.2 Transmit AMS01 “raw” data and AMS01 data summary files (Ntuples) Duration 12-24h 1) 1) M.Boschini installed bbftp in INFN Milano

20. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200220 AMS Computing Y2001 Y2001 milestones are fulfilled

21. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200221 AMS Computing Y2002 Build AMS02 “ production cell ” and use it for MC production Build AMS02 “ analysis cell ” AMS02 process and data control SW (migrate from OpenSource Corba to the licensed version) “bbftp” tests between MIT and CERN, GSC@MSFC and MIT/CERN Evaluate archiving and staging system for AMS (Jan 2002 - 4TB)

22. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200222 “ Server Node #1 ” “Processing Node #1” AMS Computing Y2002 (“production cell”) “ Processing Node #5” IDERAIDIDERAID IDERAIDIDERAID Dual-CPU AMD Processing Nodes 1-5 Dual-CPU Athlon 1900+ 1GB RAM 3Ware IDE Raid 6x120GB Western Digital 1 Gbit/sec ethernet 2x100MBit/sec ethernet Server Node 1 Dual-CPU Xeon or PIII 1GB RAM 3Ware IDE Raid 7x120GB Western Digital IPC SCSI Raid 8x160GB WD disks 1 Gbit/sec ethernet 2x100MBit/sec ethernet 1Gbit/ses AMS private 100 Mbit/sec CERN backbone Dual-CPU Intel Dual-CPU AMD IDERAIDIDERAID IDERAIDIDERAID Analysis programs SCSIRAIDSCSIRAID

23. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200223 AMS Computing Y2002 (“analysis cell”) 2 dual-CPU AMD Athlon dedicated for AMS analysis and Geant4 simulation. Architecture is similar to “AMS processing node” (but 4 channels IDE RAID controller with 4x120GB WD HDD)

24. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200224 Y2002 Milestones AMS computers upgrade (1Q) AMS “production cell” (1Q) AMS “analysis cell” (2Q) Data transmission tests (2Q) Evaluation of archiving and staging systems (technical meeting with CASPUR Feb/Mar, system choice 3Q) AMS data handling and PPCC SW, Licensed CORBA package (3Q)

25. V.Ch, A.K.AMS TIM, MIT, Jan 22-25 200225 Growth of computers and data storage in Science Operation Center