1. CHEP 24-28 March 2003 Sarah Wheeler 1 Supervision of the ATLAS High Level Triggers Sarah Wheeler on behalf of the ATLAS Trigger/DAQ High Level Trigger group

2. CHEP 24-28 March 2003 Sarah Wheeler 2 Read-Out Sub-systems HLTHLT LV L1 D E T RO ROD LVL2 TriggerDAQ 2.5  s ~ 10 ms DATAFLOWDATAFLOW Calo MuTrCh Other detectors ROB ROS SFI SFO RRC RRM EBN EFN FE Pipelines Read-Out Drivers ROD-ROB Connection Read-Out Buffers ROD-ROS Merger Dataflow Manager Sub-Farm Input Sub-Farm Output Event Filter N/work ROIB L2P L2SV L2N Event Filter DFM EFP RoI Builder L2 Supervisor L2 N/work L2 Proc Unit RoI RoI data = 2% RoI requests Lvl2 acc = ~2 kHz Event Building N/work ~ sec Lvl1 acc = 75 kHz 40 MHz 120 GB/s ~ 300 MB/s ~3+3 GB/s Event Filter Processors 120 GB/s ~3 GB/s EFacc = ~0.2 kHz 40 MHz 75 kHz ~2 kHz ~ 200 Hz ATLAS Trigger and Data Acquisition High Level Triggers

3. CHEP 24-28 March 2003 Sarah Wheeler 3 Supervision of the HLT Supervision is one of the areas where commonality between Level-2 and Event Filter can be effectively exploited HLT implemented as hundreds of software tasks running on large processor farms For reasons of practicality farms split into sub-farms Supervision is responsible for all aspects of software task management and control Configuring Controlling Monitoring SFI EFP SFI EFP SFI EFP EVENT BUILDING Event Filter Farm

4. CHEP 24-28 March 2003 Sarah Wheeler 4 Prototype HLT supervision system Prototype HLT supervision system has been implemented using tools from the ATLAS Online Software system (OnlineSW) OnlineSW is a system of the ATLAS Trigger/DAQ project Major integration exercise: OnlineSW provides generic services for TDAQ wide configuration, control and monitoring Successfully adapted for use in the HLT For HLT control activities following OnlineSW services are used: Configuration Databases Run Control Supervisor (Process Control) Controllers based on a finite-state machine are arranged in a hierarchical tree with one software controller per sub-farm and one top-level farm controller Controllers successfully customised for use in HLT

5. CHEP 24-28 March 2003 Sarah Wheeler 5 SubFarm 1SubFarm 2 Supervisor setup boot Supervisor Control Supervisor Control Controlling a Farm load start configure

6. CHEP 24-28 March 2003 Sarah Wheeler 6 Monitoring Aspects Monitoring has been implemented using tools from OnlineSW Information Service Statistical information written by HLT processes to information service servers and retrieved by others for e.g. display Error Reporting system HLT processes use this service to issue error messages to any other TDAQ component e.g. the central control console where they can be displayed

7. CHEP 24-28 March 2003 Sarah Wheeler 7 Monitoring a Farm Example of Event Filter monitoring panel

8. CHEP 24-28 March 2003 Sarah Wheeler 8 Scalability Tests (January 2003) Series of tests to determine scalability of control architecture Carried out on 230 node IT LXPLUS cluster at CERN Configurations studied: Constant total number of nodes split into a varying number of sub-farms Constant number of sub- farms with number of nodes per sub-farm varied Tests focused on times to startup, prepare for data- taking & shutdown of configurations Control Tree Event Filter Farm

9. CHEP 24-28 March 2003 Sarah Wheeler 9 Generation of Configuration Database Custom GUI written to create configuration database files

10. CHEP 24-28 March 2003 Sarah Wheeler 10 Results – Constant number of Nodes Graph shows times to start and stop control infrastructure Increase in times seen with number of sub-farms More sub-farms mean more controller and supervisor processes Constant Farm Size (230 nodes) 0 2 4 6 8 10 3821 Number of Sub-Farms Time (s) setup boot shutdown

11. CHEP 24-28 March 2003 Sarah Wheeler 11 Results – Constant number of Nodes Graph shows times to cycle through run control sequence Decrease seen with number of sub-farms More sub-farms imply fewer nodes, therefore fewer trigger processes to control per sub-farm Constant Farm Size (230 nodes) 0 2 4 6 8 10 12 14 16 3821 Number of Sub-Farms Time (s) load configure start stop unconfigure unload

12. CHEP 24-28 March 2003 Sarah Wheeler 12 Results – Constant number of Sub-Farms Times increase with increasing numbers of nodes and processes to control as expected Constant Number of Sub-Farms (10) 0 1 2 3 4 5 6 7 51020 Number of Nodes per Sub-Farm Time (s) load configure start stop unconfigure unload

13. CHEP 24-28 March 2003 Sarah Wheeler 13 Conclusions and future Results are very promising for the implementation of the HLT supervision system for the first ATLAS run All operations required to startup, prepare for data- taking and shutdown configurations take of the order of a few seconds to complete Largest tested configurations represent 10-20% of final system Future enhancements of supervision system to include: Combined Run Control/Process Control component Parallelised communication between control and trigger processes Distributed configuration database