AMS COMPUTING VIII International Workshop Advanced Computing And Analysis Techniques in Physics Research Moscow, June 24-29, 2002 Vitali Choutko, Alexei Klimentov MIT Cambridge, ETH Zurich

A.KlimentovAMS Computing ACAT'20022 Outline AMS – particle physics experiment on the international space station : AMS Data flow and ground centers Processing Farm HW components choice AMS HW evaluation Software development Distributed Data Processing STS91 precursor flight AMS ISS mission

A.KlimentovAMS Computing ACAT'20023 AMS : a particle physics experiment in space PHYSICS GOALS : Accurate, high statistics measurements of charged, cosmic ray spectra in space > 0.1GV Nuclei and e- spectra measurement The study of dark matter (90% ?) Determination of the existence or absence of antimatter in the Universe Look for negative nuclei The study of the origin and composition of cosmic rays Measure isotopes D, He, Li, Be… +

A.KlimentovAMS Computing ACAT'20024 Magnet : Nd 2 Fe 14 B TOF : trigger, velocity and Z Si Tracker : charge sign, rigidity, Z Aerogel Threshold Cerenkov : velocity Anticounters : reject multi particle events Results :  Anti-matter search : He / He = 1.1x 10  Charged Cosmic Ray spectra Pr, D, e-, He, N  Geomagnetic effects on CR under/over geomagnetic cutoff components 10 events recorded Trigger rates 0.1-1kHz DAQ lifetime 90% 8 -6 _ Precursor flight : +

A.KlimentovAMS Computing ACAT'20025 AMS Detector in shuttle payload bay (STS91 flight)

A.KlimentovAMS Computing ACAT'20026 AMS on ISS, 3 years in space Separate e- from p,p + _ up to 300 GeV He, He, B, C… 3 4 e-,   up to 1000 GeV +

A.KlimentovAMS Computing ACAT'20027

A.KlimentovAMS Computing ACAT'20028 AMS02 Data Volume Average event rate Hz (after Level-3 trigger) “Raw” Data : StreamAv.BandwidthGB/dayTB/year Scientific and Calibration 3-4 MBit/s (orbit average) House Keeping 16 Kbit/s (continuously) NASA ancillary data Event Summary Data : 45 TB/year Monte-Carlo Data : 45 TB/year The Total Data Volume of AMS PByte

A.KlimentovAMS Computing ACAT'20029 Monitoring & science data Stored data Real-time Data H&S Real-time & “Dump” data Real-time, “Dump”, & White Sand’s LOR playback AMS Crew Operation Post High Rate Frame MUX White Sand, NM facility MSFC, Al Payload Data Service system Telescience centers External Communications GSE Long Term Short Term Payload Operations Control Center Science Operations Center ISS to Remote AMS Centers Data Flow ISS NASA Ground Infrastructure Remote AMS Sites H&S Monitoring Science Flight ancillary data Real-time & “dump” NearReal-time File transfer playback

A.KlimentovAMS Computing ACAT' AMS Ground Centers Science Operations Center POCC AL AMS Remote center RT data Commanding Monitoring NRT Analysis NRT Data Processing Primary storage Archiving Distribution Science Analysis MC production Data mirror archiving External Communications Science Operations Center XTerm HOSC Web Server and xterm TReK WS commands Monitoring, H&S data Flight Ancillary data AMS science data (selected) TReK WS “ voice”loop Video distribution Production Farm Analysis Facilities PC Farm Data Server Analysis Facilities GSE D S A e T r A v e r GSE Buffer data Retransmit To SOC AMS Station AMS Station AMS Station GSE MC production cmds archive AMS Data, NASA data, metadata

A.KlimentovAMS Computing ACAT' AMS Ground Centers (Summary) Ground Support Computers at MSFC Huntsville Al  receives data from NASA  buffers until retransmission to SOC  runs unattended 24/day, 7 days/week Payload Operations and Control Center  single commanding point  monitor data and run quality control program  process about 10% of data in near real-time mode Science Operations Center (CERN)  Data production and MC simulation  Reconstructs 100% of events, typical delay less than 1 day  Central analysis system provides most of required analysis power  Complete data and MC repositary Remote Centers (Italy, USA)  Mirror data and metadata repositary  Monte Carlo production  Analysis facility to support local user’s community

A.KlimentovAMS Computing ACAT' AMS02 HW Choice Processing Farm (poor man solution) To build high-performance quality computing farm with cheap computing elements  The choice of the operating system is easy (Linux is de facto standard for the farms)  Networking components are also well identified (Gigabit switch)  CPU and Disks nodes processor and server nodes have the identical architecture (dual-CPU, 1GB RAM, 3 ethernet interfaces), computers equipped with IDE RAID arrays, servers have IDE and SCSI RAID arrays (data processing is done on both)

A.KlimentovAMS Computing ACAT' AMS02 HW Choice (cont’d) AMS02 Farm Prototype :  processors : 8 nodes (PIII, AMD, PIV) equipped with 3-ware escalade controller and TB RAID5 array of IDE disks. Events processing is done localy and the results are stored on “local disk”. 3 production jobs per machine.  servers : 2 nodes (AMD and PIV Xeon), each node has 0.8 TB RAID5 array of IDE disks and in addition external UW SCSI tower of 1.4TB, nfs’ed via dedicated ethernet segment to processing nodes. 2 production jobs per machine. We couldn’t reach declared 160 MB/s for UW SCSI controller, but 80 MB/sec). Data moved from processor nodes to server on run-by-run basis (6 months of extensive disks usage, SCSI RAID I/O performance is 20% better) 2 USD/Gbyte for “IDE RAID” and 10 USD/Gbyte for “SCSI RAID”

A.KlimentovAMS Computing ACAT' Evaluation of CPU performance Use AMS Offline simuation/reconstruction code compiled with CERNlibs, Geant, Naglib, PAW, ROOT, etc

A.KlimentovAMS Computing ACAT' AMS02 Benchmarks Executive time of AMS “standard” job compare to CPU clock 1) V.Choutko, A.Klimentov AMS note ) Brand, CPU, Memory Intel PII dual-CPU 450 MHz, 512 MB RAM OS/Compiler RH Linux 6.2 / gcc 2.95 “Sim” 1 “Rec” 1 Intel PIII dual-CPU 933 MHz, 512 MB RAMRH Linux 6.2 / gcc Compaq, Quad α-ev MHz, 2 GB RAMRH Linux 6.2 / gcc AMD Athlon,1.2GHz, 256 MB RAMRH Linux 6.2 / gcc Intel Pentium IV 1.5GHz, 256 MB RAMRH Linux 6.2 / gcc Compaq dual-CPU PIV Xeon 1.7GHz, 2GB RAMRH Linux 6.2 / gcc Compaq dual α-ev68 866MHz, 2GB RAMTru64 Unix/ cxx Elonex Intel dual-CPU PIV Xeon 2GHz, 1GB RAMRH Linux 7.2 / gcc AMD Athlon 1800MP, dual-CPU 1.53GHz, 1GB RAM RH Linux 7.2 / gcc CPU SUN-Fire-880, 750MHz, 8GB RAMSolaris 5.8/C CPU Sun Ultrasparc-III+, 900MHz, 96GB RAMRH Linux 6.2 / gcc Compaq α-ev68 dual 866MHz, 2GB RAMRH Linux 7.1 / gcc

A.KlimentovAMS Computing ACAT'200216

A.KlimentovAMS Computing ACAT'200217

A.KlimentovAMS Computing ACAT' AMS02 Benchmarks (summary) α-ev68 866MHz and AMD Athlon MP have nearly the same performance and are the best candidates for “AMS processing node” (the price of system based on α-ev68 is twice higher than the similar one based on AMD Athlon) Though PIV Xeon has lower performance, resulting 15% overhead comparing with AMD Athlon MP 1800+, the requirements of high reliability for “AMS server node” dictates the choice of Pentium machine. SUN and COMPAQ SMP might be the candidates for AMS analysis computer (the choice is postponed up to the fall of 2004)

A.KlimentovAMS Computing ACAT' AMS Software Strategies, choices and basic decisions :  new code C++ only (though we had a large part of legacy SW written on Fortran)  Existing Fortran libraries (Cernlib, Geant, Naglib, etc) incorporated via C/Fortran interface (R.Burow)  Simualation code compatible with both “Geants”  Geant3 vs Geant4, Root vs Ntuples are selected by Makefile options  transient and persistent classes are separated with implementing of copy member functions  Use Root and HBOOK for histogramming and data visualization

A.KlimentovAMS Computing ACAT' AMS Software (cont’d)  Use different persistency solutions for various type of data : Flat files for “static”data like geometry and magnetic field map Flat files for raw data Ntuples and Root files for ESD Relational Database (Oracle) [Objectivity up to Sep 1998] o Calibration data o Event Tags o Slow control data o NASA ancillary data o Full processing history o Input paramaters o Computing and storage facilities o Files location catalogues  Keep the data store and data catalogues separate

A.KlimentovAMS Computing ACAT' AMS Software (cont’d)  TReK for commanding and “online” monitoring (Telescience Resource Kit, NASA certified SW)  AMS adopted BBFTP version for data transferring between MSFC Al and CERN ( bbftp was designed in BaBar to transfer big files between SLAC and Lyon)  AMS data processing system based on Corba client- server model and Oracle RDBMS  Distributed MC Data Production System

A.KlimentovAMS Computing ACAT' server producers Raw data Oracle RDBMS Conditions DB Tag DB Active Tables : Hosts, Interfaces, Producers, Servers Catalogues Nominal Tables Hosts, Interfaces Producers, Servers… ESD Raw data {I} {II} {III} {IV} {V} {VI} {I} submit 1 st server {II} “cold” start {III} read “active” tables (available hosts, number of servers, producers, jobs/host) {IV} submit servers {V} get “run”info (runs to be processed, ESD output path) {VI} submit producers (LILO, LIRO,RIRO…) Notify servers AMS Production server

A.KlimentovAMS Computing ACAT' AMS Production Highlights Stable running for more than 9 months Average efficiency 95% (98% without Oracle) Processes communication and control via Corba LSF and remote shell for process submission Run Oracle server on AS4100 Alpha and Oracle clients on Linux. Oracle RDBMS (for what it is good)  Tag DB  Conditions DB  Bookkeeping  Production status  Runs history  Files location catalogues

A.KlimentovAMS Computing ACAT' Distributed MC Production Central repositary with binaries, execs, input parameters Static version of all exec files Central Relational Database to keep description of MC centers and computing facilities MC users Production jobs status and processing history Files location catalogues Production status  Central server (use CORBA) to control local and remote clients  Central Database Server (initially C++, now Perl)  WEB interface to control and to monitor production status to download binaries, geometry, magnetic field map, etc

A.KlimentovAMS Computing ACAT' Conclusions AMS02 production farm prototype concept is successfully tested. The prototype is in use. Data production SW has proven to be stable and to deliver 95% performance for the period of weeks. MC distrubuted production system is under testing with the remote sites in Europe (France, Italy, Suisse) and US (MIT)