1. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD The DataFlow of the ATLAS Trigger and Data Acquisition System Giovanna Lehmann On Behalf of the ATLAS Trigger/DAQ DataFlow Subsystem

2. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD Outline  ATLAS  Interaction rates and event sizes  The Trigger/DAQ architecture  The DataFlow  ROS Design & Performance  LVL2 dataflow Design & Performance  Event Builder Design & Performance  Conclusions & Outlook

3. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD CERN Accelerators’ Complex Colliding particles: protons Center of mass Energy: 14 TeV Bunch crossing rate: 40 MHz Interaction rate: 10 9 Hz Event size: 1-2 Mbytes

4. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD DATAFLOWDATAFLOW EB 120 GB/s HLTHLT L1L1 DET DET ROD LVL2 TriggerDAQ ARCHITECTURE 2.5  s ~ 10 ms Calo MuTrCh Other detectors SFI SFO EFN FE Pipelines ROIB L2P L2SV L2N Event Filter EFP RoI RoI data = 2% RoI requests Lvl2 acc= ~2 kHz ~ sec Lvl1 acc = 75 kHz 40 MHz ~3 GB/s EFacc= ~0.2 kHz EB req/clears EBN ROS ROB IOManager data DFM RRM 40 MHz 75 kHz ~2 kHz ~ 200 Hz 120 GB/s ~ 300 MB/s ~3+3 GB/s O(100) L2P O(1000) EFP 1628 ROLs O(100) ROS O(100) SFI

5. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD The ROS  Receive & buffer event fragments from the 1628 detector ROLs  Up to 160 MB/s per ROL  Send selected event fragments on request  ROI requests : high rate, low data volume Rate: LVL1 rate (75 kHz), volume: ~2% of ROLs  EB requests : low rate, high data volume Rate: ~3% of LVL1 rate (~2 kHz), volume: complete event data  Provide fragment sampling for data monitoring

6. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD ROS High Level Components Data requests Event Fragments Monitoring data Control/Configuration RODs ROS subsystem L2 & EB Online SW I/O Manager (SW process) RobIn (custom module) Local Controller (SW process) 500x 150x

7. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD Test Setup: ROS performance  ROS implemented on a 2 GHz PC, with 4 PCI busses (64 bit/66 MHz)  3 RobIn emulators on PCI  On-board “local” bus limited to 266MB/s  Each simulates 4 input channels  12 ROLs per ROS  I/O to/from L2 & EB emulator  Connected to the ROS through a GE switch  Sends ROI/EB requests and clears to the ROS  Receives data fragments back  Uses TCP as communication protocol (maximum possible overhead for message passing)

8. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD ROS Performance ATLAS baseline conditions (from paper model: contains safety factor 4 with respect to physics simulation )

9. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD The LVL2 Dataflow  Receive RoI information from LVL1  8 ROLs @ LVL1 rate (75 kHz)  Form a LVL1 result record  Build 1 record out of 8 @ LVL1 rate  Retrieve RoI data from ROSs  2 % of full Event (~30 kB)  Forward the LVL2 decision to the EB  @ LVL2 accept rate (rejects are grouped)  Forward the LVL2 decision record to the EB  @ LVL2 accept rate (~2 kHz)

10. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD RoI req./data Decisions Control/Configuration EB LVL2 subsystem LVL1 Online SW RoIBuilder (custom module) L2SV (SW process) L2PU (SW process) pROS (SW process) L2 record ROS RoI information 1x10x 200-500x 1x LVL2 High Level Components DC Controller (SW process)

11. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD Performance of RoI Builder, L2SV and pROS  Performance of RoI Builder  Custom built 12U VME prototype has achieved required performance  Performance of each LVL2 supervisor  Measured to be ~30 kHz on a 2.4 GHz dual CPU PC  Is insensitive to the number of L2PUs  Performance of pROS  Not a demanding application  Requirement to receive <10 kB at LVL2 accept rate (~3 kHz) and forward them to the EB is largely satisfied.

12. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD Test Setup: Performance of L2PU  ROS emulators used to send data over Gbit Ethernet.  RoI data collection takes always a small fraction of the time requested by the LVL2 event processing (~10 ms).  From a dataflow point of view << 100 L2PUs could sustain already the LVL1 rate.

13. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD The Event Builder  Receive LVL2 decisions  @ LVL2 accept rate (~2 kHz; rejects are grouped)  Request data from ROS and pROS  Build complete events  Depending on ROS implementation merge 140-1600 fragments into one.  ~70 MB/s at every SFI  Distribute clears to ROS and pROS  @ rate < LvL2 accept rate  Forward complete events to EF  ~70 MB/s at every SFI  Provide fragment sampling for data monitoring

14. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD EB req./data LVL2 Decisions Control/Configuration L2 EB subsystem pROS Online SW DFM (SW process) SFI (SW process) Clears ROS 1x 50x EB High Level Components EF Monitoring DC Controller (SW process) 10x Complete Event

15. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD Test Setup: EB Performance DFM 16x ROS em... Switch SFI... 8x ROS em SFI applications were run on 2.4 GHz dual CPU PCs Many ROSs to many SFIs ROS Emulators:  ALTEON programmable GE NICs  Raw ethernet communication protocol  Simulating n sources

16. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD EB Performance Limit of 16 ROS emulators for single frame messages EB rate with 8 SFIs ~ 350Hz (17% of ATLAS EB rate) 8 ROLs/ROS Flow Control 1 ROL/ROS No Flow Control

17. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD Conclusions & Outlook  All elements of the DataFlow system have shown that they can satisfy the ATLAS requirements already with the present implementations and with today’s technology.  From now on emphasis will be put on the performance of the integrated DataFlow system.  Testbeds are being setup to measure its behaviour and the first results are encouraging.

18. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD Spares

19. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD ATLAS baseline conditions (from paper model) Results of Test 1 (no I/O to LVL2 & EB)

20. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD Test Setup 2: Scaling of LVL2 Network From a dataflow point of view a few L2PUs sustain already a large fraction of the LVL1 rate.

21. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD Test Setup 1: Performance of the DFM Tester: L2SV + n SFIs DFM LVL2 decision (group) SFI EOE DFM_Decision Clears (group 300)  Tester Application emulates L2SV and many SFIs  DFM handling full I/O as for real ATLAS  DFM exposed to full input message rate from tester  DFM sending to non existing destinations (Connectionless protocol used)

22. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD DFM Performance ATLAS event building rate raw ethernet frames udp Test on a 2.2 GHz dual CPU PC: Rate= function of CPU clock ->

23. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD Test Setup 1: SFI Performance DFM 1 Gbit/s Ethernet 16x ROS em... EF Switch ROS em SFI application was run on a 2.4 GHz dual CPU PC Many ROSs to 1 SFI ROS Emulators:  ALTEON programmable GE NICs  Raw ethernet communication protocol  Simulating n sources SFI

24. 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD SFI Performance 95 MB/s – IO limited #ROLs/ROS EB only With output to EF CPU limited (2.4 GHz CPU)  Reaching I/O limit at 95 MB/s otherwise CPU limited  35% performance gain with at least 8 ROLs/ROS  Will approach I/O limit for 1 ROL/ROS with faster CPU