GSI, Oct 2005Hans G. Essel DAQ Control1 H.G.Essel, J.Adamczewski, B.Kolb, M.Stockmeier

GSI, Oct 2005Hans G. Essel DAQ Control2 Example CMS: event building and filtering 40 MHz Collision rate 100 kHz output from trigger ports switching fabric 5-10 TeraOps processing sub-cluster 1 Terabit/sec channels ~800 Gb/sec full connectivity 100 Mbytes/sec output to Petabyte archive and output to world- wide data grid Detectors 500 Readout Units Buffer incoming data and Serve it upon request 500 Builder Units combine the data from the RUs DAQ Cluster

GSI, Oct 2005Hans G. Essel DAQ Control3 CMS: blueprint for clustered DAQ

GSI, Oct 2005Hans G. Essel DAQ Control4 CMS DAQ: requirements Communication and Interoperability –Transmission and reception within and across subsystem boundaries without regard of the used protocols –Addition of protocols without a need for modifications in the applications Device Access –Access to custom devices for configuration and readout –Access to local and remote devices (bus adapters) without the need for modifications in applications Configuration, control and monitoring –Make parameters of built-in or user defined types visible and allow their modification –Allow the coordination of application components (define their states and modes) –Allow the inspection of states and modes –Provide services for recording structured information Logging, error reporting Interface to persistent data stores (preferrably without the need to adapt the applications) Publish all information to interested subscribers –Device allocation, sharing and concurrent access support Maintainability and Portability –Allow portability across operating system and hardware platforms –Support access for data across multiple bus systems –Allow addition of new electronics without changes in user software –Provide memory management functionality to Improve robustness Give room for efficiency improvements –Application code shall be invariant with respect to the physical location and the network –Possibility to factorise out re-usable building blocks Scalability –Overhead introduced by the software environment must be constant for each transmission operation and small with respect to the underlying communication hardware in order not to introduce unpredictable behaviour –Allow applications to take advantage of additional resource availability Flexibility –Allow the applications to use multiple communication channels concurrently –Addition of components must not decrease the system’s capacity

GSI, Oct 2005Hans G. Essel DAQ Control5 CMS XDAQ XDAQ is a framework targeted at data processing clusters –Can be used for general purpose applications –Has its origins in the I 2 O (Intelligent IO) specification The programming environment is designed as an executive –A program that runs on every host –User applications are C++ programmed plug-ins –Plug-ins are dynamically downloaded into the executives –The executive provides functionality for Memory management Systems programming queues, tasks, semaphores, timers Communication Asynchronous peer-to-peer communication model Incoming events (data, signals, …) are demultiplexed to callback functions of application components Services for configuration, control and monitoring Direct hardware access and manipulation services Persistency services

GSI, Oct 2005Hans G. Essel DAQ Control6 XDAQ Availability Platform OSCPUDescription Linux (RH)x86Baseline implementation Mac OS XPPC G3, G4no HAL, no raw Ethernet PT SolarisSparcno HAL, ro raw Ethernet PT VxWorksPPC 603, Intel x86no GM Current version:1.1 Next releases:V 1.2 in October 2002 (Daqlets) V 1.3 in February 2003 (HAL inspector) Change control:Via sourceforge: Version control:CVS at CERN License:BSD style

GSI, Oct 2005Hans G. Essel DAQ Control7 XDAQ: References J. Gutleber, L. Orsini, « Software Architecture for Processing Clusters based on I 2 O », Cluster Computing, the Journal of Networks, Software and Applications, Baltzer Science Publishers, 5(1):55-64, 2002 (goto for a draft version or contact me) The CMS collaboration, “CMS, The Trigger/DAQ Project”, Chapter 9 - “Online software infrastructure”, CMS TDR-6.2, in print (contact me for a draft), also available at G. Antchev et al., “The CMS Event Builder Demonstrator and Results with Myrinet”, Computer Physics Communications 2189, Elsevier Science North- Holland, 2001 (contact E. Barsotti, A. Booch, M. Bowden, “Effects of various event building techniques on data acquisition architectures”, Fermilab note, FERMILAB- CONF-90/61, USA, 1990.

GSI, Oct 2005Hans G. Essel DAQ Control8 XDAQ event driven communication Dynamically loaded application modules (from URL, from file) Inbound/Outbound queue (pass frame pointers, zero-copy) Homogeneous frame format Readout component Generates a DMA completion event Application component Implements callback function foo( ) Peer transport Receives messages from network Executive framework Demultiplexes incoming events to listener application component Computer

GSI, Oct 2005Hans G. Essel DAQ Control9 XDAQ: I 2 O peer operation for clusters Application component device Processing node IOP Controller node host Homogeneous communication –frameSend for local, remote, host –single addressing scheme (Tid) Application framework Executive Messaging Layer Peer Transport Agent Messaging Layer Executive Peer Transport Agent   ‚  „     Peer Transport Application I 2 O Message Frames

GSI, Oct 2005Hans G. Essel DAQ Control10 XDAQ: available peer transports TransportDescriptionMulticast TCP/IPReliable binary messages over various physical networksemulated UDPUnreliabe binary message over various physical networksnative FIFOLocal communication among components within the same executiveemulated GMReliable binary messages over Myrinetemulated MAZEReliable barrel shifter binary messages over Myrinetno EthernetUnreliable binary messages using raw Ethernet frames (only on VxWorks OS) emulated UNICUnreliable binary messages using raw Ethernet frames (only on Linux OS, planned in V1.3) emulated SOAP/HTTPReliable delivery of XML documents over various physical networksNo (emulated planned) SOAP/MailReliable delivery of XML documents over SMTP (planned for V1.3) no

GSI, Oct 2005Hans G. Essel DAQ Control11 XDAQWin client Daqlet window Daqlets are Java applets that can be used to customize the configuration, control and monitoring of all components in the configuration tree Configuration tree XML based configuration of a XDAQ cluster

GSI, Oct 2005Hans G. Essel DAQ Control12 XDAQ: component commands Configuration tree Predefined commands can be performed on selected XDAQ components (multiple selection allowed) Commands can be issued to groups of components: executives, applications, peer-transports; this enables true cluster operation.

GSI, Oct 2005Hans G. Essel DAQ Control13 XDAQ: component properties Component Properties Allows the inspection and modification of components’ exported parameters.

GSI, Oct 2005Hans G. Essel DAQ Control14 XDAQ: interface export User Commands Allows the execution of any exported command, including the command to reflect the interface Interface Query Returns all exported commands. Transition to WSDL planned Cluster commands Results from multiple components can be displayed.

GSI, Oct 2005Hans G. Essel DAQ Control15 BTeV: a 20 THz real-time system Input: 800 GB/s (2.5 MHz) Level 1 –Lvl1 processing: 190  s rate of 396 ns –528 “8 GHz” G5 CPUs (factor of 50 event reduction) –high performance interconnects Level 2/3: –Lvl 2 processing: 5 ms (factor of 10 event reduction) –Lvl 3 processing: 135 ms (factor of 2 event reduction) –1536 “12 GHz” CPUs commodity networking Output: 200 MB/s (4 kHz) = 1-2 Petabytes/year

GSI, Oct 2005Hans G. Essel DAQ Control16 BTeV: The problem Monitoring, Fault Tolerance and Fault Mitigation are crucial –In a cluster of this size, processes and daemons are constantly hanging/failing without warning or notice Software reliability depends on –Physics detector-machine performance –Program testing procedures, implementation, and design quality –Behavior of the electronics (front-end and within the trigger) Hardware failures will occur! –one to a few per week Given the very complex nature of this system where thousands of events are simultaneously and asynchronously cooking, issues of data integrity, robustness, and monitoring are critically important and have the capacity to cripple a design if not dealt with at the outset… BTeV [needs to] supply the necessary level of “self-awareness” in the trigger system. Real Time Embedded System

GSI, Oct 2005Hans G. Essel DAQ Control17 BTeV: RTES goals High availability –Fault handling infrastructure capable of Accurately identifying problems (where, what, and why) Compensating for problems (shift the load, changing thresholds) Automated recovery procedures (restart / reconfiguration) Accurate accounting Extensibility (capturing new detection/recovery procedures) Policy driven monitoring and control Dynamic reconfiguration –adjust to potentially changing resources Faults must be detected/corrected ASAP –semi-autonomously with as little human intervention as possible –distributed & hierarchical monitoring and control Life-cycle maintainability and evolvability –to deal with new algorithms, new hardware and new versions of the OS

GSI, Oct 2005Hans G. Essel DAQ Control18 RTES deliverables A hierarchical fault management system and toolkit: –Model Integrated Computing GME (Generic Modeling Environment) system modeling tools –and application specific “graphic languages” for modeling system configuration, messaging, fault behaviors, user interface, etc. –ARMORs (Adaptive, Reconfigurable, and Mobile Objects for Reliability) Robust framework for detection and reaction to faults in processes –VLAs (Very Lightweight Agents for limited resource environments) To monitor/mitigate at every level –DSP, Supervisory nodes, Linux farm, etc.

GSI, Oct 2005Hans G. Essel DAQ Control19 RTES Development The Real Time Embedded System Group –A collaboration of five institutions, University of Illinois University of Pittsburgh University of Syracuse Vanderbilt University (PI) Fermilab NSF ITR grant ACI Physicists and Computer Scientists/Electrical Engineers at BTeV institutions

GSI, Oct 2005Hans G. Essel DAQ Control20 RTES structure Analysis Runtime Design and Analysis Algorithms Fault Behavior Resource Synthesis Performance Diagnosability Reliability Experiment Control Interface Synthesis Feedback Modeling Reconfigure Logical Data Net Region Operations Mgr L2,3/CISC/RISCL1/DSP Region Fault Mgr Logical Data Net Logical Control Network Global Fault Manager Global Operations Manager Soft Real Time Hard

GSI, Oct 2005Hans G. Essel DAQ Control21 GME: simple meta model example

GSI, Oct 2005Hans G. Essel DAQ Control22 GME: simple model made from this meta model

GSI, Oct 2005Hans G. Essel DAQ Control23 GME: data type modeling Modeling of Data Types and Structures Configure marshalling- demarshalling interfaces for communication

GSI, Oct 2005Hans G. Essel DAQ Control24 RTES: GME modeling environment  Fault handling  Process dataflow  Hardware Configuration

GSI, Oct 2005Hans G. Essel DAQ Control25 RTES: system architecture RunControl Manager Router Information How many regions ? How many worker nodes inside the region? Node Identification information

GSI, Oct 2005Hans G. Essel DAQ Control26 Configuration of ARMOR Infrastructure (A) Modeling of Fault Mitigation Strategies (B) Specification of Communication Flow (C) A B C RTES: GME fault mitigation modeling language (1)

GSI, Oct 2005Hans G. Essel DAQ Control27 RTES: GME fault mitigation modeling language (2) Model translator generates fault-tolerant strategies and communication flow strategy from FMML models Strategies are plugged into ARMOR infrastructure as ARMOR elements ARMOR infrastructure uses these custom elements to provide customized fault-tolerant protection to the application ARMOR ARMOR Microkernel Fault Tolerant Custom Element FMML Model – Behavior Aspect Communication Custom Element Switch(cur_state) case NOMINAL: I f (time<100) { next_state = FAULT; } Break; case FAULT if () { next_state = NOMINAL; } break; class armorcallback0:public Callback { public:ack0(ControlsCection *cc, void *p) : CallbackFaultInjectTererbose>(cc, p) { } void invoke(FaultInjecerbose* msg) { printf("Callback. Recievede dtml_rcver_LocalArmor_ct *Lo; mc_message_ct *pmc = new m_ct; mc_bundle_ct *bundlepmc->ple(); pmc->assign_name(); bundle=pmc->push_bundle();mc); } }; Translator

GSI, Oct 2005Hans G. Essel DAQ Control28 RTES: user interface modeling language Enables reconfiguration of user interfaces Structural and data flow codes generated from models User Interface produced by running the generated code Example User Interface Model

GSI, Oct 2005Hans G. Essel DAQ Control29 RTES: user interface generation Generato r

GSI, Oct 2005Hans G. Essel DAQ Control30 ARMOR Adaptive Reconfigurable Mobile Objects of Reliability: –Multithreaded processes composed of replaceable building blocks –Provide error detection and recovery services to user applications Hierarchy of ARMOR processes form runtime environment: –System management, error detection, and error recovery services distributed across ARMOR processes. –ARMOR Runtime environment is itself self checking. 3-tiered ARMOR support of user application –Completely transparent and external support –Enhancement of standard libraries –Instrumentation with ARMOR API ARMOR processes designed to be reconfigurable: –Internal architecture structured around event-driven modules called elements. –Elements provide functionality of the runtime environment, error-detection capabilities, and recovery policies. –Deployed ARMOR processes contain only elements necessary for required error detection and recovery services. ARMOR processes resilient to errors by leveraging multiple detection and recovery mechanisms: –Internal self-checking mechanisms to prevent failures from occurring and to limit error propagation. –State protected through checkpointing. –Detection and recovery of errors. ARMOR runtime environment fault-tolerant and scalable: –1-node, 2-node, and N-node configurations.

GSI, Oct 2005Hans G. Essel DAQ Control31 ARMOR system: basic configuration Heartbeat ARMOR Detects and recovers FTM failures Fault Tolerant Manager Highest ranking manager in the system Daemons Detect ARMOR crash and hang failures ARMOR processes Provide a hierarchy of error detection and recovery. ARMORS are protected through checkpointing and internal self-checking. Execution ARMOR Oversees application process (e.g. the various Trigger Supervisor/Monitors) Daemon Fault Tolerant Manager (FTM) Daemon Heartbeat ARMOR Daemon Exec ARMOR App Process network

GSI, Oct 2005Hans G. Essel DAQ Control32 EPICS overview EPICS is a set of software components and tools to develop control systems. The basic components are: OPI (clients) –Operator Interface. This is a UNIX or Windows based workstation which can run various EPICS tools (MEDM, ALH, OracleArchiver). IOC (server) –Input Output Controller. This can be VME/VXI based chassis containing a Motorola 68xxx processor, various I/O modules, and VME modules that provide access to other I/O buses such as GPIB, CANbus. LAN (communication) –Local area network. This is the communication network which allows the IOCs and OPIs to communicate. EPICS provides a software component, Channel Access, which provides network transparent communication between a Channel Access client and an arbitrary number of Channel Access servers.

GSI, Oct 2005Hans G. Essel DAQ Control33 Basic attributes of EPICS Tool Based: –EPICS provides a number of tools for creating a control system. This minimizes the need for custom coding and helps ensure uniform operator interfaces. Distributed: –An arbitrary number of IOCs and OPIs can be supported. As long as the network is not saturated, no single bottle neck is present. A distributed system scales nicely. If a single IOC becomes saturated, it's functions can be spread over several IOCs. Rather than running all applications on a single host, the applications can be spread over many OPIs. Event Driven: –The EPICS software components are all designed to be event driven to the maximum extent possible. For example rather than having to poll IOCs for changes, a channel access client can request that it be notified only when changes occur. This design leads to efficient use of resources as well as to quick response times. High Performance: –A SPARC based workstation can handle several thousand screen updates a second with each update resulting from a channel access event. A IOC can process more than 6,000 records per second including generation of any channel access events.

GSI, Oct 2005Hans G. Essel DAQ Control34 IOC software components DATABASE: –The memory resident database plus associated data structures. Database Access: –Database access routines. With the exception of record and device support, all access to the database is via the database access routines. Scanners: –The mechanism for deciding when records should be processed. Record Support: –Each record type has an associated set of record support routines. Device Support: –Each record type has one or more sets of device support routines. Device Drivers: –Device drivers access external devices. A driver may have an associated driver interrupt routine. Channel Access: –The interface between the external world and the IOC. It provides a network independent interface to database access. Monitors: –Database monitors are invoked when database field values change. Sequencer –A finite state machine.

GSI, Oct 2005Hans G. Essel DAQ Control35 Hierarchy in a flat system tasks IOC Client IOCs –One IOC per standard CPU (Linux, Lynx, VxWorks) clients –on Linux, (Windows) Agents –Segment IOCs beeing also clients Name space architecture!

GSI, Oct 2005Hans G. Essel DAQ Control36 Local communication (node) status segment IOC Task intertask commands messages memory Task command thread working thread message thread Commands handled by threads Execution maybe in working thread Message thread maybe not needed Node

GSI, Oct 2005Hans G. Essel DAQ Control37 MBS node and monitor IOC External control asynchronous IOC status segment Dispatcher Task commands (text) messages (text) Task Messageserver Statusserver asynchronous on request

GSI, Oct 2005Hans G. Essel DAQ Control38 Screen shot FOPI

GSI, Oct 2005Hans G. Essel DAQ Control39 Kind of conclusion RTES: Very big and powerful. Not simply available! –Big collaboration –Fully modelled and simulated using GME –ARMORs for maximum fault tolerance and control XDAQ: Much smaller. Installed at GSI. –Dynamic configurations (XML) –Fault tolerance? EPICS: From accelerator controls community. Installed at GSI –Maybe best known –No fault tolerance –Not very dynamic