March 2003 CHEP Online Monitoring Software Framework in the ATLAS Experiment Serguei Kolos CERN/PNPI On behalf of the ATLAS Trigger/DAQ Online Software group

March 2003Online Monitoring Software Framework in the ATLAS Experiment2 Contents ATLAS Detector layout ATLAS HLT/DAQ/DCS system What to monitor and where Performance and Scalability Requirements Monitoring framework architecture Monitoring framework implementation Test results Summary

March 2003Online Monitoring Software Framework in the ATLAS Experiment3 ATLAS Detector Layout Each partition may be operated independently Some partitions may be operated in parallel ~1000 Read-Out Drivers (RODs) in ~100 VME crates 33 sub-detector Partitions PixelTileCalLArMDTCSCSCTTRT Calorimeter Inner Detector Muon Spectrometer RPC TGC

March 2003Online Monitoring Software Framework in the ATLAS Experiment4 Data Storage Event Filter (EF) Event Builder (EB) HLT/DAQ/DCS system Detector Control System (DCS) LVL2 Trigger Read Out Systems (ROSs) Online Software: Configure Control Monitoring PixelTileCalLArMDTCSCSCTTRT Calorimeter Inner Detector Muon Spectrometer RPC TGC

March 2003Online Monitoring Software Framework in the ATLAS Experiment5 Where and what will be monitored Detector and Physics monitoring: ROD Crate, ROS: data quality and integrity DCS: detector hardware status and conditions Event Builder: correlation between sub-detectors, consistency of LVL1 information Event Filter: monitoring of reconstructed events DAQ monitoring: ROS, EB: operational monitoring (buffer occupancies, throughput, s/w and h/w status, errors, etc.) Trigger Monitoring: LVL1, LVL2: sample rejected events to check the trigger decision Event Filter: information attached to a sub-set of accepted and rejected events

March 2003Online Monitoring Software Framework in the ATLAS Experiment6 Monitoring data TypeFormatProductionAccess Samples of physics events Vector of 4-byte integer values On request Errors Error ID + Error Severity + Text In case of faults Via subscription Histograms One (or several) standard histogram formats AlwaysOn request and via subscription Other information (status, operational information, etc.) User-definedAlwaysOn request and via subscription

March 2003Online Monitoring Software Framework in the ATLAS Experiment7 Online Monitoring Framework: Functionality and Performance Requirements The functionality: Transporting monitoring data requests from consumers to providers Transporting monitoring data from providers to consumers The required performance: There are O(1000) sources of the monitoring data There are O(1) consumers for each monitoring data item Transfer rate between one provider and one consumer: For samples of physics events is O(1) MB/s For other monitoring data types O(1) kB/s Monitoring Framework Monitoring Framework Monitoring Data Provider Monitoring Data Consumer command data

March 2003Online Monitoring Software Framework in the ATLAS Experiment8 Online Monitoring Framework: Architecture Specific service for each monitoring data type IPC: Common communication abstraction layer: implements communication domains to support partitioning CORBA: Common communication implementation layer Common Object Request Broker Architecture (CORBA) Inter Process Communication Event Monitoring Service Message Reporting Service Information Service Online Histogramming Service

March 2003Online Monitoring Software Framework in the ATLAS Experiment9 Event Monitoring Service: Interfaces Provide unified way of getting samples of physics events from any point in the data-flow chain Implementation exists for C++ and Java Event Monitoring Service Event Monitoring Service Event Consumer EventIterator next_event Event Provider EventAccumulator add_event EventSampler start_sampling stop_sampling

March 2003Online Monitoring Software Framework in the ATLAS Experiment10 DAQ Workstation Event Distributor ROS ROD Crate Event Sampler ROD Crate Event Monitoring Service: Deployment EB Event Sampler ROD Crate Event Sampler Event Sampler Event Sampler Event Buffer Event Consumer 2. add_event 3. next_event 1. select 1.1 start_sampling There is no direct connection between Event Consumer and Event Sampler

March 2003Online Monitoring Software Framework in the ATLAS Experiment11 Message Reporting Service: Interfaces Each error has unique ID, severity and text, and optionally may have custom parameters and qualifiers. Error Consumer may subscribe to a messages by defining a range of values for any field of the error message Implementation exists for C++ and Java Message Reporting Service Message Reporting Service Error Consumer Error Provider MRSStream subscribe send_error notify MRSReceiver MRSCallback

March 2003Online Monitoring Software Framework in the ATLAS Experiment12 DAQ Workstation Message Reporting Service: deployment DAQ Workstation MRS Server ROS EB DAQ Application DAQ Application 2. send_error 3. notify 1. subscribe ROD Crate DAQ Application ROD Crate DAQ Application ROD Crate DAQ Application DAQ Application

March 2003Online Monitoring Software Framework in the ATLAS Experiment13 Information Service: Interfaces Allows applications to exchange user-defined information Information structure is defined in XML Information description is available at run-time Implementation exists in C++ and Java Information Service Information Service Info Consumer Info Provider InfoDistionary subscribe insert update remove notify InfoReceiver InfoCallback get_value get_description InfoDocument

March 2003Online Monitoring Software Framework in the ATLAS Experiment14 Information Service: Deployment DAQ Workstation DAQ Control Workstation IS Server ROS EB DAQ Application DAQ Application insert update remove notify subscribe ROD Crate DAQ Application ROD Crate DAQ Application ROD Crate DAQ Application DAQ Application DAQ Control Workstation IS Server DAQ Workstation IS Server get_value

March 2003Online Monitoring Software Framework in the ATLAS Experiment15 Histogramming Service Histogramming Service Online Histogramming Service: Interfaces Specialization of the Information Service for transporting histograms Current implementation supports ROOT histograms and also raw histograms (vectors of data) Has abstract interface layer which allows to add support for other types of histograms Information Service Histogram Provider RootHistoProvider RawHistoProvider Histogram Consumer CustomHistoProvider RootHistoReceiver RawHistoReceiver CustomHistoReceiver

March 2003Online Monitoring Software Framework in the ATLAS Experiment16 Histogramming Service: Deployment DAQ Workstation DAQ Control Workstation IS Server ROS EB DAQ Application DAQ Application insert update remove DAQ Application DAQ Control Workstation IS Server DAQ Workstation IS Server get_histogram ROD Crate DAQ Application ROD Crate DAQ Application ROD Crate DAQ Application Implementation is based on the Information Service

March 2003Online Monitoring Software Framework in the ATLAS Experiment17 IS Monitor (Motif, C++) Event Dump (Java) Histogram Display (ROOT,C++) Graphical User Interfaces

March 2003Online Monitoring Software Framework in the ATLAS Experiment18 Information Service Performance Tests receiver 5 receivers 10 receivers 15 receivers Number of information providers Mean time for one information update (ms) 222 Dual Pentium III PC ( Mhz) connected via Fast Ethernet Linux RedHat 7.3 Single IS server was running on a dedicated PC Up to a 15 IS receivers were running on 15 dedicated PCs Up to 1000 IS information providers where equally distributed over 200 PCs Each provider published one information and then updated it once per second

March 2003Online Monitoring Software Framework in the ATLAS Experiment19 Summary The Online Monitoring framework in the ATLAS experiment is responsible for transportation different type of monitoring data from provider to consumers The framework consists of 4 services to handle different types of monitoring data All the services have APIs in both C++ and Java The tests show that the performance and scalability of the current implementation is very close to the ATLAS requirements