1. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 20071 Computer Instrumentation Introduction Jos Vermeulen, UvA / NIKHEF Topical lectures, 28 June 2007

2. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 20072 Computing essential ingredient of high-energy physics instrumentation for: trigger data-acquisition on-line monitoring experiment control calibration These lectures: Relevant hardware and software techniques, the ATLAS T/DAQ/DCS system serves as illustration

3. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 20073 A Toroidal LHC ApparatuS (ATLAS): One of two general-purpose experiments at the Large Hadron Collider at CERN. Under construction in Point 1, opposite to the main entrance of CERN

4. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 20074 SDX1 USA15 UX15 US15 Main control room Main Entrance CERN Point 1 A side (Geneva) C side (Jura) x y z

5. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 20075 Cut-away view of the ATLAS Detector

6. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 20076

7. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 20077

8. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 20078

9. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 20079 June 13, 2007: Lowering of side A End Cap Toroid (ECT)

10. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200710 Descent through the access shaft

11. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200711

12. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200712 Descent in the cavern

13. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200713 Landed Vessel produced by Schelde Exotech, cold mass components by Brush HMA BV

14. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200714 The relation with computing …. June issue of ATLAS e-news (http://aenews.cern.ch): Richard Stallman, the president and founder of the Free Software Foundation ….. went on a pit tour, where he had a chance to admire our freshly installed end-cap toroid!

15. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200715 Computer instrumentation in high-energy physics: Detector Control System (DCS) Trigger Data-AcQuisition (DAQ)

16. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200716 ATLAS Detector Control System (DCS) LCS USAL1 LCS USAL2 LCS US15 LCS SDX1 LCS 1LCS 2LCS 3LCS 4 CoolingRacksEnvironELMB HEC HV Temp Barrel HV FE Crates HV LV Purity Front-End Systems Magnet CERN LHC DSS Data Viewer AlarmStatusWeb Operator Interface DCS_IS WAN CIC PixelSCTTRTLArMDTTGCRPCCSC LAN Tile Common Infrastructure Controls Local Control Stations Detector Safety System DCS Information Service Underground Subdetector Control Stations Embedded Local Monitor Box Global Control Stations

17. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200717 SDX1 USA15 UX15 ATLAS Trigger / DAQ DataFlow Overview

18. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200718 Event data pushed @ ≤ 100 kHz, 1600 fragments of ~ 1 kByte each ATLAS detector Read- Out Drivers ( RODs ) First- level trigger Read-Out Subsystems ( ROSs ) UX15 USA15 SDX1 USA15 UX15 Dedicated links Timing Trigger Control (TTC) 1600 Read- Out Links 10 Gigabit Ethernet ATLAS Trigger / DAQ DataFlow Overview RoI Builder Regions Of Interest VME ~160 PCs Data of events accepted by first-level trigger Event data requests Delete commands Requested event data ~40x10 ~320x1 Gbit/s ~20 switches

19. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200719 Event data pushed @ ≤ 100 kHz, 1600 fragments of ~ 1 kByte each ATLAS detector Read- Out Drivers ( RODs ) First- level trigger Read-Out Subsystems ( ROSs ) LVL2 Super- visor UX15 USA15 SDX1 CERN computer centre SDX1 USA15 UX15 Dedicated links Timing Trigger Control (TTC) 1600 Read- Out Links 10 Gigabit Ethernet ATLAS Trigger / DAQ DataFlow Overview RoI Builder DataFlow Manager Event Filter (EF) pROS ~ 870~1500 Regions Of Interest VME Data of events accepted by first-level trigger Event data requests Delete commands Requested event data stores LVL2 output dual 1, 2 or 4-core CPU nodes ~100~30 Network switches Event data pulled: partial events @ ≤ 100 kHz, full events @ ~ 3 kHz Event rate ~ 200 Hz Data storage Local Storage SubFarm Outputs (SFOs) LVL2 Farm + switches Network switches Event Builder SubFarm Inputs (SFIs) Second- level trigger ~40x10 ~320x1 Gbit/s ~20 switches ~160 PCs Gigabit Ethernet

20. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200720 Event data pushed @ ≤ 100 kHz, 1600 fragments of ~ 1 kByte each ATLAS detector Read- Out Drivers ( RODs ) First- level trigger Read-Out Subsystems ( ROSs ) UX15 USA15 Dedicated links Timing Trigger Control (TTC) 1600 Read- Out Links RoI Builder VME Data of events accepted by first-level trigger RODs, ROS PCs and ROBINs 10 Gigabit Ethernet Read-Out Drivers (ROD): subdetector-specific, collect and process data (no event selection) output via Read-Out Links (ROL, 200 MByte/s optical fibers) to buffers on ROBIN cards in Read-Out Subsystem (ROS) PCs Same type of ROLs, ROBINs and ROS PCs used for all sub-detectors ROBINs: 64-bit 66 MHz PCI-X cards 3 ROL inputs ROS PCs: 4U rack-mounted PCs with 4 ROBINs => 12 ROLs per ROS PC ~40x10 ~320x1 Gbit/s ~20 switches ~150 PCs

21. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200721 Region of Interest (RoI) Builder receives for each first-level accept information from first-level trigger and passes formatted information to one of the LVL2 supervisors. LVL2 supervisor decides for one of the processors in the LVL2 farm and sends it the RoI information. LVL2 processor requests data from the ROSs as needed (possibly in several steps), produces an accept or reject and informs the LVL2 supervisor. Result of processing is stored in pseudo-ROS (pROS) for an accept. LVL2 supervisor passes decision to the DataFlow Manager. Trigger/DAQ DataFlow associated with second-level (LVL2) trigger First- level trigger Read-Out Subsystems ( ROSs ) LVL2 Super- visor USA15 Timing Trigger Control (TTC) 1600 Read- Out Links 10 Gigabit Ethernet RoI Builder Regions Of Interest Event data requests Delete commands Requested event data ~40x10 ~320x1 Gbit/s ~20 switches ~150 PCs SDX1 DataFlow Manager Event Filter (EF) pROS ~ 870~1500 dual 1, 2 or 4-core CPU nodes ~100~30 Network switches Event rate ~ 200 Hz Local Storage SubFarm Outputs (SFOs) LVL2 Farm + switches Network switches Event Builder SubFarm Inputs (SFIs)

22. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200722 For each accepted event the DataFlow Manager decides for a Sub-Farm Input (SFI) and sends it a request to take care of the building of a complete Event. The SFI sends requests to all ROSs for data of the event to be built. Completion of building is reported to the DataFlow Manager. For rejected events and for events for which event Building has completed the DataFlow Manager sends "clears" to the ROSs for up to 100 events together. On request the event data are passed from SFI to an Event Filter processor. Trigger/DAQ DataFlow associated with Event Building Event Building rate ~ 3 - 3.5 kHz First- level trigger Read-Out Subsystems ( ROSs ) LVL2 Super- visor USA15 Timing Trigger Control (TTC) 1600 Read- Out Links 10 Gigabit Ethernet RoI Builder Regions Of Interest Event data requests Delete commands Requested event data ~40x10 ~320x1 Gbit/s ~20 switches ~150 PCs SDX1 DataFlow Manager Event Filter (EF) pROS ~ 870~1500 dual 1, 2 or 4-core CPU nodes ~100~30 Network switches Event rate ~ 200 Hz Local Storage SubFarm Outputs (SFOs) LVL2 Farm + switches Network switches Event Builder SubFarm Inputs (SFIs)

23. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200723 Simple model ("paper model") used to predict average number of ROS PCs and ROLs from which data is needed for LVL2 trigger processing, for example for design luminosity trigger menu, per first-level accept: 16.2 ROLs or 8.4 ROS PCs -> RoI-driven processing is a key property of the ATLAS LVL2 system, but also makes the system more complex and its performance not so straight- forward to predict. With 1 - 1.5 kByte per fragment need network bandwidth of ~ 2 GByte/s at 100 kHz first-level trigger accept rate (instead of ~ 150 GByte/s for full read-out at 100 kHz) for LVL2 traffic Second-Level (LVL2) Trigger and Event Building: rates Event Building rate ~ 3 - 3.5 kHz (bandwidth ~ 3 - 5 GByte/s) First- level trigger Read-Out Subsystems ( ROSs ) LVL2 Super- visor USA15 Timing Trigger Control (TTC) 1600 Read- Out Links 10 Gigabit Ethernet RoI Builder Regions Of Interest Event data requests Delete commands Requested event data Event rate ~ 200 Hz ~40x10 ~320x1 Gbit/s ~20 switches ~150 PCs SDX1 DataFlow Manager Event Filter (EF) pROS ~ 870~1500 dual 1, 2 or 4-core CPU nodes ~100~30 Network switches Event rate ~ 200 Hz Local Storage SubFarm Outputs (SFOs) LVL2 Farm + switches Network switches Event Builder SubFarm Inputs (SFIs)

24. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200724 Application Specific Integrated Circuits (ASICs) Field Programmable Gate Arrays (FPGAs) Digital Signal Processors (DSPs) Microcontrollers Embedded microprocessors Crate processors Personal Computers (PCs) Compute servers Linux, multi-threaded C++, Java for GUIs usually no OS, cross development of software written in C or C++, exceptionally in assembler "programmed" in VHDL or Verilog + "fitting", specialists only, mostly electronic engineers, "firmware" can be changed in-circuit in many designs expensive, design by specialists, cannot be changed after production Computing hardware

25. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200725 VME bus: parallel bus in crates PCI, PCI-X bus : parallel bus in PCs PCI-Express: serial point-to-point connections inside PC CAN: serial bus system used for controls JTAG: serial connections between integrated circuits SPI, I2C: short distance serial connections RocketIO links of Xilinx FPGAs: up to 3.125 Gbit/s serial point-to-point links for connecting FPGAs GOL: Gigabit Optical Link, developed by CERN (radiation hard sender) S-link: protocol for point-to-point links, developed by CERN TTC: Timing and Trigger Control system: for broadcasting trigger decisions, but can also transmit control information, developed by CERN Switched Ethernet: 100 Mbit, 1 Gigabit, 10 Gigabit Other switched network technology (not in ATLAS, but e.g. in CMS: Myrinet) Connection technology

26. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200726 Interrupts Direct Memory Access (DMA) Memory-mapped I/O Error detection using parity bits or a Cyclic Redundancy Check (CRC) -> See presentation Sander Klous Memory management Drivers Multi-threaded programming Inter-process communication Remote process invocation Traffic shaping to avoid queueing Databases Techniques

27. Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 200727 Tomorrow: examples and discussion of different types of computing hardware and interconnection technology and of relevant techniques in the context of: I.ATLAS DCS and front-end readout (TTC, on-detector, RODs) II.ATLAS Triggering and DAQ