1. Computing Strategy of the AMS-02 Experiment
V. Choutko1, B. Shan2 A. Egorov1, A. Eline1
1MIT, USA
2Beihang University, China
CHEP 2015, Okinawa, April 13, 2015

2. Outline AMS Experiment AMS Data/Computing Model
Parallelization of Reconstruction Software
Parallelization of Simulation Software
Production Supporting Software
Summary

3. Z, P are measured independently by the Tracker, RICH, TOF and ECAL
AMS: A TeV precision, multipurpose particle physics spectrometer in space.
TRD
Identify e+, e-
Particles and nuclei are defined by their charge (Z) and energy (E ~ P)
TOF
Z, E
TRD
TOF
Tracker
RICH
ECAL 1 2
7-8
3-4 9
5-6
Magnet ±Z
Silicon Tracker
Z, P
RICH
Z, E
ECAL
E of e+, e-, γ
Z, P are measured independently by the Tracker, RICH, TOF and ECAL 3 3 3

4. AMS Experiment
The Alpha Magnetic Spectrometer (AMS) is a high energy physics experiment on board the ISS, featured:
TOF: Four Layers of Scintillators (120 ps);
Tracker: Nine Layers of Silicon Detectors (10 μ);
Gaseous TRD Detector: h/e Rejection O(102);
Pb/Sc EM Calorimeter: h/e Rejection O(104);
RICH Detector: Velocity Measurement O(10-4);
Geometrical Acceptance: 0.5 m2sr;
Number of ReadOut Channels: ≈200K
TRD
TOF
Tracker
RICH
ECAL 1 2
7-8
3-4 9
5-6

5. AMS Experiment
The Alpha Magnetic Spectrometer (AMS) is a high energy physics experiment on board the ISS, featured:
TOF: Four Layers of Scintillators (120 ps);
Tracker: Nine Layers of Silicon Detectors (10 μ);
Gaseous TRD Detector: h/e Rejection O(102);
Pb/Sc EM Calorimeter: h/e Rejection O(104);
RICH Detector: Velocity Measurement O(10-4);
Geometrical Acceptance: 0.5 m2sr;
Number of ReadOut Channels: ≈200K

6. AMS Data Model Average 500 cosmic ray events per second
Data collection rate ~1.2MB/s
Transferred via relay satellite, Marshall Space Flight Center, to CERN
Recording original electron signals
Organized to one RUN every ¼ orbit (~23 minutes)
First Production
Runs 24/7 on freshly arrived data
Initial data validation and indexing
Produces Data Summary Files and Event Tags for fast events selection
Requires ~ 80 CPU cores to cope with data rate
Usually be available within 2 hours after flight data arrived
used to produce various calibrations for the second production as well as quick performance evaluation.
Second Production
To use all the available calibrations, alignments, ancillary data from the ISS, and slow control data (temperatures, pressures, voltages) to produce physics analysis ready set of reconstructed data.

7. Remote Computing Facilities
AMS Computing Model
DATABASE
ORACLE(PDBR)
Frames
Web Server
Preproduction
Production Server
Production platform
RAW
AMS
Cluster
(PBS)
CERN
lxbatch
(LSF)
Remote Computing Facilities
(Producer)
ROOT(DST)
Remote Centers
Validator
Uploader
EOS, CASTOR
Data
Meta-data / Control
CERN
Facilities

8. Highlights in AMS Offline Software
Parallelization of Reconstruction
Parallelization of Simulation
Production Supporting Software

9. Parallelization of Reconstruction
Motivations
To shorten the (elapsed) time of data reconstruction
To increase productivity by using multi-core/hyper-thread enabled processors
To ease the production jobs management;
Tool: openmp
No explicit thread programming
Nearly no code change except few pragmas
Possible to change thread number during execution

10. Sequential Procession

11. Parallel Procession

12. Parallel Reconstruction Performance (Xeon?)

13. Parallel Reconstruction Performance (Intel MIC)

14. Parallel Reconstruction Performance (Intel X5482)

15. Parallel Reconstruction Performance (Intel Old Xeon HT)

16. Parallel Reconstruction Performance (Intel i7 HT)

17. Parallel Reconstruction Performance (Efficiency)

18. Parallel Reconstruction Performance (Xeon Nehalem HT)

19. Parallel Reconstruction Performance (Elapsed time change)

20. Parallelization of Simulation Software
Motivations
To decrease required memory amount per simulation job
To allow faster Monte-Carlo parameter/software tuning
Minimal software change: Geant MT model + openmp (icc 15)
Thread ID, thread number, and core number are taken from Geant , instead of from openmp
barrier, master, parallel from openmp do not work, so barriers are implemented by emulation
All other openmp pragmas work

21. Parallelization of Simulation Software
Memory optimization -- G4AllocatorPool modification
Garbage Collection with preserving of n chunks  (1 by default) with modes:
Fully automatic
At the end of event
Fast unsafe by name of G4Allocator at the end of event
Decreased the memory consumption by factor 2 and more for long jobs (W.R.T. original G4AllocatorPool)
Decreased the memory consumption by factor ?? W.R.T. sequential simulation job

22. Parallel Simulation Performance (Memory consumption)

23. Parallel Simulation Performance (Xeon?)

24. Parallel Simulation Performance (Intel MIC)

25. Production Supporting Software
Fully automated production cycle for:
Production in local (CERN) and remote computing centers
Data reconstruction and Monte-Carlo simulation
Local production
Production server/monitor
Remote computing centers
Light-weight production platform

26. Local production management
Centralized job management by production server/monitor
All production database reading is through server
Load balance between different kinds of computing resources
CORBA-based communication between producers and server
Dynamic thread number adjusting
Graphical production monitoring

27. Light-weight production platform
Fully-automated production cycle
Job acquiring, submission, monitoring, validation, transferring, and (optional) scratching
Easy to deploy
Based on perl/python/sqlite3
Installation-free
Customizable
Batch system, storage area, transferring method, etc.
Running at:
JUROPA, CNAF, IN2P3, NLAA

28. Latest Full Data Reconstruction
Second half of Feb. 2015
Performed at CERN, JUROPA, CNAF, ACAD. SINICA
41 months’ flight data
Reconstruction completed essentially in two weeks
100 times data collection speed
10-5 failing rate

29. Latest Full Data Reconstruction
(I will add one plot here to show the completing status of the pass6 reconstruction)

30. Summary
Parallelization of the AMS reconstruction software allows to increase the productivity of the reconstruction program up to 45% using the HT enable processors as well as to short per run reconstruction time and calibration jobs execution time by factor of 10.
Parallelization of the AMS simulation software allows to decrease the total memory consumption up to ??, which makes it much easier to run long simulation jobs for light nuclei.
With production supporting software, data reconstruction rate reached 100 times of data collection.