1. Karol Buńkowski, University of Warsaw Control software in the current RPC trigger and DAQ system CMS GEM 2 day Electronics Meeting 10 October 2012

2. 2 CMS control system CMS detector: ~10 subsystems, dozens of thousands of electronics boards controlled by hundreds of computers. Developed by many groups  not so homogenous All of that must be properly setup for successful data taking  centralized system is needed which is able to steer this distributed and inhomogeneous structure Karol Buńkowski, UW, CMS GEM 2 day Electronics Meeting, 10 October 2012

3. 3 Karol Buńkowski, UW, CMS GEM 2 day Electronics Meeting, 10 October 2012 http://cmsdoc.cern.ch/cms/TRIDAS/RCMS/ http://cmsdoc.cern.ch/cms/TRIDAS/RCMS/Docs/RCMSPapers/talks/2010_05_27_RealTime2010_HS_7_final.pdf http://cmsdoc.cern.ch/cms/TRIDAS/RCMS/Docs/RCMSPapers/proceedings/2010_RealTime_Proceedings_HannesSakulin_CR.pdf Controls HV, LV, gas, etc., follows the LHC states changes One person (DAQ shifter) from one application is able to setup the CMS for the data taking, start and control the run, monitor the data taking process

4. 4 The Run Control and Monitor System (RCMS) is the collection of hardware and software components responsible for controlling and monitoring the CMS experiment during data taking. It provides physicists with a single point of entry to operate the data acquisition of the experiment. http://cmsdoc.cern.ch/cms/ TRIDAS/RCMS/ https://twiki.cern.ch/twiki/bin/view/XdaqWiki/WebHome XDAQ is a software platform designed specifically for the development of distributed data acquisition systems.. XDAQ is a middleware that eases the tasks of designing, programming and managing data acquisition applications by providing a simple, consistent and integrated distributed programming environment. Trigger Supervisor – extension of the XDAQ. Its purpose is to provide a framework to set up, test, operate and monitor the trigger components on one hand and to manage their interplay and the information exchange with the run control part of the data acquisition system on the other. The Trigger Supervisor is conceived to provide a simple and homogeneous client interface to the online software infrastructure of the trigger subsystems. https://twiki.cern.ch/twiki/bin/view/CMS/TriggerSupervisor https://twiki.cern.ch/twiki/bin/view/CMS/TriggerSupervisor

5. 5 Karol Buńkowski, UW, CMS GEM 2 day Electronics Meeting, 10 October 2012

6. 6 RPC trigger system – hardware and software schema Karol Buńkowski, UW, CMS GEM 2 day Electronics Meeting, 10 October 2012 TS Worker TC TS Worker TC TS Worker SC TS Worker SC … DB service (Tomcat) RPC FM RPCT TS Cell Trigger and Sorter Crates DCC/CCS crate … To RCMS Top FM To TS Central Cell … LBox’es FEB FECFEC VME FECFEC CCU rings x 18 DCC access XDAQ DCC access XDAQ … DCCDCC I2C rings … VME Config DB TS Worker LBOX TS Worker LBOX TS Worker LBOX TS Worker LBOX x13 x18

7. 7 XDAQ and TS for subsystem hardware control in practice XDAQ and Trigger Supervisor – C++ framework, set of libraries developed by the “central” CMS team(s) The XDAQ and TS framework provides: –Hardware Access Library VME driver and interface classes to the driver XML based HardwareAddressTable – definition of the registers – in the RPC trigger system replaced by the custom solution (Internal Interface, see next slides) –SOAP interface for communication between the XDAQ (TS) applications, Function Manager or any other custom application SOAP - Simple Object Access Protocol - standard protocol specification for exchanging structured information in the implementation of Web Services in computer networks. –Tools for exchanging the monitoring information between the application (Flash Lists) –Tools facilitating the GUI development – in case of the TS based on the AJAX technology –Tools for (remote) logging, monitoring the application state, and many, many others things To create the custom XDAQ application for controlling the given hardware component (e.g. Trigger Supervisor worker for the Trigger Crate) subsystem developers must: –customize some framework classes (e.g. to implement the stat machine) –define needed SOAP interface –define GUI –create the custom code for controlling and monitoring the hardware. Karol Buńkowski, UW, CMS GEM 2 day Electronics Meeting, 10 October 2012

8. 8 Control software layers - RPCtrigger Karol Buńkowski, UW, CMS GEM 2 day Electronics Meeting, 10 October 2012 HAL::VMEBusAdapterInterface, e.g.: virtual void write( DeviceIdentifier* deviceIdentifierPtr, uint32_t address, uint32_t addressModifier, uint32_t dataWidth, uint32_t data) virtual void read( DeviceIdentifier* deviceIdentifierPtr, uint32_t address, uint32_t addressModifier, uint32_t dataWidth, uint32_t* result) HAL::VMEBusAdapterInterface, e.g.: virtual void write( DeviceIdentifier* deviceIdentifierPtr, uint32_t address, uint32_t addressModifier, uint32_t dataWidth, uint32_t data) virtual void read( DeviceIdentifier* deviceIdentifierPtr, uint32_t address, uint32_t addressModifier, uint32_t dataWidth, uint32_t* result) CaenVmeDriver CAEN PCI VME bridge Devices Registers Access - IIAccess Idevice, e.g.: uint32_t readWord(TID id, int idx) = 0; void writeWord(TID id, int idx, uint32_t data) Devices Registers Access - IIAccess Idevice, e.g.: uint32_t readWord(TID id, int idx) = 0; void writeWord(TID id, int idx, uint32_t data) Classes for each type of the FPGA device with higher level functions: e.g. Pac: void reset(); void configure(DeviceSettings* settings, int flags); void enable(); Classes for each type of the FPGA device with higher level functions: e.g. Pac: void reset(); void configure(DeviceSettings* settings, int flags); void enable(); Classes for each type of Board and Crate Generic classes for diagnostic modules control System – structure of all hardware controlled by the application XDAQ/Trigger Supervisor: SOAP interface, GUI, State machine XDAQ/Trigger Supervisor: SOAP interface, GUI, State machine TS Worker (application controlling e.g. Trigger Crate) VME crate

9. 9 Trigger Crate TS Worker – configuration panel Karol Buńkowski, UW, CMS GEM 2 day Electronics Meeting, 10 October 2012

10. 10 Hardware structure description The workers controlling the hardware must know the structure of the hardware. The structure of the hardware is described in the dedicated database (in the CMS OMDS): –Tables for: chambers, strips, chips, boards, crates, interconnections between the boards, XDAQ application setup (host, port, etc.), etc. During the application initialization the worker ask the DB for the information about the hardware it will control, and than builds the objects corresponding to each hardware item (chip, board, crate). Karol Buńkowski, UW, CMS GEM 2 day Electronics Meeting, 10 October 2012

11. 11 Internal Interface The most common way of describing the registers of a chip is such that the ASIC or FPGA developer provides a txt or xml file in some defined format containing the register names, addresses, sizes, etc. This file is then parsed when the application starts. In the software the registers are accessed by the string identifiers. This approach is used in the HAL: void HAL::HardwareDevice::write(string item, unsigned long data) The disadvantage of this approach is such that in the software you can try to access the register which was removed in the new version of the firmware, but you will learn that only during the runtime. This is very painful during the firmware development phase. Alternative approach - Internal Interface Framework for integrating the firmware and control software, developed by Krzysztof Pozniak (Warsaw University of Technology) and Michal Pietrusinski (Warsaw of University) Firmware developer creates a file in a special format defining registers of a given device, i.e. meaning of bits, size of memory areas, and type of access (read or write). From this file, a dedicated application generates the C++ and VHDL code, which is then included in the software and firmware projects, respectively. the file contains C++ constants, which are included to the definition of the class corresponding to the FPGA device; some of that constants become the identifiers of the hardware registers, others define the registers or memory areas size,, etc.). A dedicated function calculates the hardware addresses corresponding to the register identifiers. the structure of the firmware is build directly into the software source. The register access is then: void TIIDevice::writeWord(TID id, int idx, uint32_t data) where id is e.g. const TN IID_CLASS :: WORD_BCN0_DELAY = 133; Karol Buńkowski, UW, CMS GEM 2 day Electronics Meeting, 10 October 2012

12. 12 IIAccess Web GUI for accessing registers in the FPGA devices implementing Internal Interface In graphical way presents the hardware structure: crate/board/chip/re gister Connects to many TSWorkers running on different computers Java web service running on tomcat Karol Buńkowski, UW, CMS GEM 2 day Electronics Meeting, 10 October 2012

13. 13 Hardware configuration The hardware configuration consist of two steps: –Hardwar reset (QPLLs, TTCrxs, GOLs) – must be done in a proper sequence –Applying the configuration parameters into the hardware (delays, mask of t disabled transmission channels, parameters configuring the algorithms) The hardware configuration data are stored in the database (the same as the hardware structure description). There is a separate table for the configuration data of each type of the chip. The set of the configuration data is identified by the configuration key – many different configurations data sets can be used to configure the hardware During the configure transition a TS worker asks for the configuration data set for the hardware items (chips) that it controls, and apply it into the hardware. Karol Buńkowski, UW, CMS GEM 2 day Electronics Meeting, 10 October 2012

14. 14 Hardware monitoring Monitoring is performed directly by the TS Workers Monitoring of the hardware status– status monitorables: –Device status (e.g. TTCrx ready, QPLL locked) – simple check, if not ok– ERROR is issued, –Transmission error counters: WARNIG or ERROR is issued depending on the error rate, The process in worker reads repetitively the values of selected registers: –The results are printed on the worker monitoring panel. The monitoring results are analyzed, and in case of problems being detected, the warning message is send to the “L1 page” to inform the trigger shifter. Additionally, email and SMS with the warning is send to the experts. Monitoring of the trigger data flowing through the hardware – statistic monitorables (counters and histograms analyzing the data stream): –Chamber strip rates –Muon candidate rates The plots with histograms are created in the real time and displayed by the panels in the workers and cell. In case of abnormal rate the warnings are generated. The data are stored in the root files for offline analysis. CMS L1 Trigger Online Software Review, 5 February 2009 Karol Buńkowski, Warsaw University

15. 15 Summary The subsystem control software must be robust and running smoothly as much as possible: –Software crashes not allowed –The transitions (configure, enable, stop, etc.) must be ~always successful  the software must handle the hardware problems as much as possible, and if the problem is to big to handle that – give clear message what is wrong –The software must provide tools for: monitoring the hardware status and data integrity, Diagnose and fix (if possible) the problems The software development is very big job - RPC online control software is 200 000 lines of code, developed since ~15 years by 1-2 persons. It is never-ending-story: there are always new hardware problems which must be handled, or the changes in the XDAQ/TS framework must be included, etc. More: K. Bunkowski PhD thesis: http://cms.cern.ch/iCMS/jsp/openfile.jsp?type=TS&year=2010&file s=TS2010_009.pdf http://cms.cern.ch/iCMS/jsp/openfile.jsp?type=TS&year=2010&file s=TS2010_009.pdf Karol Buńkowski, UW, CMS GEM 2 day Electronics Meeting, 10 October 2012