1. 1 Architecture of the ATLAS High Level Trigger Event Selection Software Monika Grothe, CERN EP, on behalf of the ATLAS Trigger/DAQ High Level Trigger group CHEP03 March 24-28 2003 La Jolla, California Outline: General strategy of the HLT event selection The components of the HLT event selection software Using offline code in the HLT online context

2. 2 LHC: proton-proton collisions @ E CM = 14 TeV starting 2007 L = 10 34 cm -2 s -1  23 collisions per bunch crossing @ 25 ns interval Challenge for the ATLAS trigger:  Interaction rate 10 9 Hz, offline computing can handle O(10 2 Hz)  ATLAS has O(10 8 ) read-out channels  Physics: e.g. SM Higgs (m = 120 GeV) H   occurs at 10 -13 x 10 9 Hz For high rejection power while preserving sensitivity for rare processes: 3 trigger levels: Level1 hardware trigger Level2 software trigger EventFilter software trigger On-going validation of HLT software design and implementation for the HLT Technical Design Report in summer 2003 and in preparation of the first stage of ATLAS commissioning High Level Trigger HLT } High Level Trigger HLT The stage Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03

3. 3 Level 1 trigger High Level Trigger Level 2 trigger EventFilter Region of Interest RoI 40 MHz 75 kHz ~1 kHz ~100 Hz ~1 sec ~10 ms 2.5  s Rate Latency ATLAS data flow diagram Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03

4. 4 EventFilter EventFilter: software trigger, 1 kHz  100 Hz, 1 sec average latency  Full event from EventBuilder passed to EventFilter farm (PCs, Linux)  Independent Processing Applications run selection algorithms on farm nodes  Selection software consists of offline-type algorithms that have access to latest calibration and alignment data Level1 Level1: hardware trigger, 40 MHz  75 kHz, 2.5  s maximum hardware latency  Looks for regions of potentially interesting activity, with high p T objects Region of Interest (RoI): muon, electromagnetic, tau/hadronic, jet clusters  Looks in calorimeters, muon spectrometer  Does not combine information of more than one detector Level2 Level2: software trigger, 75 kHz  1 kHz, 10 ms average latency  Data are held in readout buffers (ROB) during Level2 processing  Selection software run by Processing Application on one node of Level2 farm (PCs, Linux)  Input (seed) is Level1 RoI (its type, position, p t threshold passed) so that typically only few % of full event information in the ROBs need to be transferred to Level2  Feature extraction in RoI region by specialized algorithms that are optimized for speed and cover all sub-detectors sequentially  For events accepted by Level2 EventBuilder builds full event ATLAS trigger overview Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03

5. 5 Level1 finds 2 isolated em clusters with each p T >20 GeV possible signature for Z  e + e - Method:  Validate step-by-step  Check intermediate signatures  Reject at earliest possible moment Managed by HLT Steering package Alg Iso lation p T > 30GeV Cluster shape track finding Iso lation p T > 30GeV Cluster shape track finding EM20i + e30i + e30 + e e + ecand + Signature  Level1 seed  STEP 1 STEP 4 STEP 3 STEP 2 How the HLT works Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03 t i m e

6. 6 HLTSSW Steering ROBData Collector Data Manager HLT Algorithms Processing Application Event DataModel Processing Application Interface Dependency Package Event Filter HLT Core Software HLT Algorithms Main components of HLT selection software Event selection: how ? – Steering what ? - HLT ALgorithms Data access: how ? - DataManager what ? - EventDataModel Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03 Level2 HLT Selection Software HLT DataFlow Software

7. 7 StepController StepHandler StepSequencerStepDecision HLTAlgo STEP 1 STEP i For each step there exists a TriggerMenu: List of signatures for which an event can be accepted, e.g. SequenceTable: Specifies which algorithms are to be run given the input objects to the step, e.g. run on each input object algo ecand + track finding Different from the typical find-all-electrons/tracks/… in event: Localized reconstruction, guided by seed Steering calls relevant algorithm once per seed, in same event Event selection - how ? HLT steering Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03

8. 8 Steering hands over seed to algorithm via TriggerElements Algorithm returns result of processing as another TriggerElement TriggerElement has a label doesn’t hold any data content is related to DataObjects and other TriggerElements in a navigable way Signatures in a TriggerMenu consist of logical AND/OR of TriggerElements Event selection - how ? HLT steering (2) Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03

9. 9 Iso lation pt> 30GeV Cluster shape track finding EM20i e30i e30 e ecand TriggerElement “EM20i” TriggerElement “ecand” DataObject Level1 EMRoI HLT algorithm reconstruction realm Supporting information Steering realm Hypotheses DataObject CaloCluster uses uses DataObject CaloCells uses 1..* TriggerElement “e” DataObject track DataObject cluster uses uses 1..* seeded by etc. Event selection - how ? HLT steering (3)

10. 10 Data Access – What ? Raw Event Data Model DataFlow system ships raw event data to HLT in bytestream format, for Level2 fragments on demand, for EventFilter complete event Raw data objects (RDO) = object representation of raw event data (bytestream) Algorithms access raw event data by means of RDOs and transient event store Importance of raw Event Data Model (EDM): Different granularity of data as provided by DataFlow system ( min. granularity ROB) and data as deployed by algorithms (per geometric unit = DetectorElement) Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03Level1Level2EventFilter Level1 Result Level2 Algorithm Level2 Result EventFilter Algorithm EventBuilder bytestream ~1600 ROBs on demand

11. 11 Example: ATLAS Muon Spectrometer Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03 Transient Event Store Container DetElem … etc ID RDO etc RDO etc … DetectorElement DetectorElement Chamber = collection of raw data of readout objects (RDOs) channels Each collection can be retrieved from the transient store with the help of an identifier Raw EDM (2)

12. 12 Data Access – How ? ByteStreamConverters Algorithm Region Selector HLT Algorithm Region Selector Trans. Event Store Data Access Byte Stream Converter Data source organized by ROB Transient EventStore Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03 ByteStreamConverter maps DetectorElements onto minimum readout granularity Geometry  ROBs x RoI position Example: Muon Spectrometer region list DetElem IDs ROB ID raw event data DetElems list DetElem IDs DetElems

13. 13 HLT steering calls an HLT algorithm by passing to it a TriggerElement with a navigable link to a RoI DataObject that contains the RoI position The HLT algorithm typically needs to access the raw event data in the DetectorElements in a certain region around the RoI position The HLT algorithm requests from the RegionSelector the identifiers of all DetectorElements within that region The HLT algorithm passes the list of identifiers to the transient event store At this stage 2 cases can occur: 1. The requested DetectorElements are already available and are just returned 2. The information first needs to be retrieved from the DataFlow system  The transient event store invokes a ByteStreamConverter which requests the bytestream data from the correct ROB  The ByteStreamConverter extracts from the succession of 32-bit words in the ROB bytestream the relevant event data and fills the collections of RDOs  The thus obtained collections of RDOs are stored in the transient event store  The extraction of event data can be supplemented by invoking additional data preparation algorithms (e.g. cluster finders for the Cal) x RoI position Example: Muon Spectrometer Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03 ByteStreamConverters (2)

14. 14 Design aim: Use same environment for development & maintenance (offline) and deployment (online) of HLT selection algorithms Advantage: + Ease of developing HLT algorithms and studying their performance on simulated and real data in well-tested user-friendly offline environment + Ease of code migration offline  EventFilter  Level2 ATLAS offline framework ATHENA, based on a GAUDI (developed by LHCb) core Basic principles: dataalgorithms Separation of data and algorithms which produce and consume data transientpersistent Separation of transient and persistent representation of data Converters Converters convert one data representation into another Consequences for HLT design e.g.: Algorithms forced to communicate via transient event store, persistent data made available by converting into transient representation stored in transient event store, common database access tools, etc Importance of these offline-imposed elements to meet stringent online performance requirements Offline code in online context Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03

15. 15 HLT selection software: offline dependencies HLTSSW Steering Monitoring Service 1..* MetaData Service 1..* ROBData Collector Data Manager HLT Algorithms Processing Application EventData Model Processing Application > Offline EventDataModel Offline Reconstruction Algorithms > StoreGate Athena/ Gaudi > I nterface Dependency Package Event Filter Level2 HLT Core Software Offline Architecture & Core Software Offline Reconstruction HLT Algorithms HLT selection software HLT DataFlow software Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03

16. 16 Complications from special Level2 environment Example multi-threading: Used to keep idle time of CPUs in Level2 farm to minimum Implications:  Control over event loop needs to be with the HLT DataFlow software  All algorithms/services have to be configured/initialized outside event loop ATHENA controls event loop in the offline environment How to reconcile with common online/offline environment for HLT select. software ? Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03 Initialize Finalize Execute Stringent requirements on framework in Level2: Speed Robustness Multi-threading Initialize Execute Finalize Execute Athena Level2

17. 17 Solution: SteeringController as interface layer between HLT selection & HLT DataFlow software: 1. Uses GAUDI light, i.e. SteeringController uses only minimum set of GAUDI features 2. Uses Modification to GAUDI base libraries such that for every thread: EventLoopMgr_(threadID)->executeEvent() AnyServiceNeededByAlgo_(threadID) AnyHLTAlgorithm_(threadID) Modified GAUDI can be used in offline environment as ATHENA is based on GAUDI (suffix collapses to blank) Modifications since included in GAUDI as standard features Unexpected pitfall of using off-the-shelf software in Level2: STL locks Thread-safe but thread-inefficient implementation of STL containers Use different implementation of allocator Due to different way DataFlow and HLT selection software interact in EventFilter ATHENA can be used as framework for running HLT algorithms Timing issue: 30% of processing time spent in the raw EDM available at that time Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03 Complications from special Level2 environment (2)

18. 18 Conclusions and outlook For HLT Technical Design Report in summer 2003: Architecture of the HLT selection software is currently in validation stage Integration of HLT selection software with other components of the HLT software on-going Setting-up of full trigger vertical slices (electron/  and  in testbeds Detailed measurements of timing and physics performance on-going Re-use of offline code in online context currently experimental News from testbeds encouraging, though fair amount of code hand-tuning by online required But also very fruitful dialogue offline-online with concrete results “Why don’t you program your trigger in assembler ?” (offline to online) Improvement of CPU speed allows ever higher level of abstraction Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03

19. 19  The StoreGate: a Data Model for the Atlas Software Architecture P. Calafiura, Monday StoreGate  The Algorithm Steering & Trigger Decision mechanism of the Atlas High Level Trigger G. Commune, Tuesday HLT Steering package  The Region of Interest Strategy for the ATLAS Second Level Trigger V. Boisvert, Tuesday RegionSelector  Configuration of event-selection criteria in the ATLAS trigger system M.Elsing, Thursday Use of TriggerMenus  Use of a generic identification scheme connecting events and detector description in the ATLAS experiment DetectorElement Collection identifiers R.D. Schaffer, Thursday More details in the ATLAS talks @ parallel sessions Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03 Further ATLAS Trigger/DAQ talks: The DataFlow system of the ATLAS Trigger & DAQ, G. Lehmann, Monday The use of Ethernet in the DataFlow of the ATLAS Trigger & DAQ, S. Stancu, Monday Experience with multi-threaded C++ applications in the ATLAS DAQ system, S. Gadomski, Monday Supervision of the Atlas High Level Trigger System, S. Wheeler, Tuesday Online Monitoring software framework in the ATLAS experiment, S. Kolos, Thursday

20. 20 Monika Grothe Architecture of the ATLAS High Level Trigger Event Selection Software CHEP03 Level2 testbed Purpose: Integration of all components of the HLT selection software Only minimum of components of the DataFlow system necessary to run the HLT selection software Switch with 8 fast Ethernet ports (for single processor PCs) and 2 Gigabit Ethernet ports (for dual processor PCs)