Control and operation of detector systems and their readout electronics in a complex experiment control system Stefan Koestner (CERN) on behalf of the LHCb Online Group Cavern-side: Dose outside detector up to 10 Gy in 10 yrs The LHCb experiment: Single-arm forward spectrometer dedicated to B-physics TeV, luminosity=2·10 32 cm -2 s -1 acceptance: (250)mrad bb/year, full B spectrum Counting house-side: No radiation hard electronics required e.g. Inner Tracker CCPC – Credit Card sized PC 15 different types of DAQ and trigger boards (in total some 400) a few hundred registers each (monitor) and memory blocks (upload) common readout board (Tell1) based on FPGA technology to adopt specific needs DIM – Distributed Information Management System portable lightweight communication layer to interface hardware from ECS if server crashes it can easily republish on DNS node (Robustness) clients can be installed on any machine just specifying DNS node (Portability) – no need to take care of connectivity Long distance I2C Analog Data (twisted pair ~5m) SPECS mezzanine card: Hosted on controller cards in radiation area Contains the SPECS slave – a radiation tolerant ACTEL ProASIC FPGA which translates the SPECS protocol into the required I2C, JTAG or local bus SPECS is a custom protocol made for LHCb to work in radiation and B-field environment SPECS is used for control and readout of ASICs (e.g. preamps) and sensors (e.g. temperature) For infrastructure the ELMB (CAN bus) is used Optical fibres ~100m SPECS ~100m (4 different unidirectional lines) L0 trigger ~ 1 MHz, 4 µs fixed latency Time synchronized to FE-chips L1 electronics: 1 MHz readout (~35 GB/s) collection and preprocessing push-protocol supposes enough buffer in next stage Tell1 Readout Supervisor Throttle Or Multi-Event-Packages (MEPs) sent to PC farm (Gigabit Ethernet) PC-Farm: High Level Triggers Reconstruction Storage Trigger & Fast Control: distributes clock sends trigger and broadcasts assigns farmnode IP sends throttle and resets SPECS Master: PCI card which can talk with up to 32 SPECS slaves generic server running on same PC to communicate with ECS (DIM) client Experiment Control System based on Industrial SCADA (PVSSII) LHC common JCOP framework distributed and scalable system coherent interface platform independend (CONTROL interpreter) Service box Inside concrete bunker close to detectors – requires radiation tolerant electronics: frontend configuration (SPECS mezzanine) and analog to digital conversion DIM protocol Embedded Microcontrollers (CCPC): readout boards in the radiation save area host embedded microcontrollers (credit card PC) access to I2C, JTAG and general purpose bus isolated access path  robustness controls network (10/100 Ethernet) separated from data network generic server to communicate with ECS (DIM) client Configuration DB: stores register settings uploaded at start up Oracle/SQL I2C like protocol without acknowledgements slave can send interrupt and master repeats clock only active during transfer (10 MHz) variable number of words (<256) with 10 bits SPECS – Serial Protocol for the Experiment Control System 10 Mbit/s serial link for configuration of remote electronics single master multi-slave (up to 32) bus – simple, fast and cheap serial bus with two signals in each direction (2 times clk & data) BLVDS on shielded twisted pair (8-bin RJ45, cat 5 Ethernet) SPECS multi-master board – 4 different busses standard 32-bit 33 MHz PCI board inserted on the same PC where SPECS server is running SPECS slave designed as portable VERILOG code implemented in ACTEL APA150 FPGA (tested up to 40 krad) SEU and SEL immune due to triple voting and one-hot state used SPECS mezzanine board houses the SPECS slave provides 16 long distance JTAG chains (4 signals each) provides 15 long distance I2C busses with selectable frequency local i2C bus and 16 bit parallel bus (8 bit address range) DCU chip with 6 ADC channels (12 bit) and 1 temp sensor 32 configurable I/O lines FPGA-based readout board ADC conversion or optical conv. synchronisation, reordering, pedestal subtraction, common mode sub., zero suppression (PP) MEP packing, IP framing (SL) Gigabit Ethernet Link Separated ECS interface (CCPC) 4” – SM520PCX from Digitallogic i486 compatible microcontroller (AMD ELAN 520) Linux Kernel at 133 MHz reading from local bus 20 MB/s filesystem shared over server access to board via gluecard over a PLXPCI9030 bridge: Parallel bus (8/16/32), 3 fast JTAG chains (2 MHz) to load firmware, 4 I2C lines and 9 GPIO lines. low level libraries under SLC4 DIM server running on CCPC or SPECS master PC publishes services to DIM Name Server (DNS) from where the client (ECS) can subscribe to it. Data exchange peer to peer from server to client Client sends commands (write/read) – server updates services (data/status) PVSS Event manager Database manager PVSS Database Control manager API GUIs Distribution manager Driver manager DIM client PVSS00dim User interface layer Processing layer Communication and memory layer Driver layer Distributed System Support PC running DNS node SPECS master PC running SPECS server or embedded controller running CCPC server DNS node server Ethernet The PVSS software architecture comprises a number of managers which may run on independent machines and communicate with the event manager via a PVSS internal protocol on top of TCP/IP a DIM driver is implemented which allows to communicate with hardware over ethernet Representation of the hardware inside the Experiment Control System Modelling of the entire experiment with Finite State Machines in order to model the experiment at an expert system level states and actions have to be defined From the top node (run control) commands can be propagated downwards a hierarchical tree Status and alarms can be propagated upwards the hierarchical tree to the run control The final branch of a tree is always a device unit acting on the hardware (connected to datapoint!) Additional control units can be implemented to give a better logical structure (domains) PVSS datapoints each type of hardware has to be represented as a PVSS datapoint type, from which the various instances can be derived (datapoints) a datapoint is a complex structure connected to the internal PVSS database if the data inside a datapoint changes a callback function or alert can be launched registers can be organized as substructures to mimic the organization of a hardware each register consists of various datapoints holding the settings of each register or allowing to launch DIM commands or receive DIM services FwHw tool a tool (GUI) was developed which eases the development of hardware types scripts using the framework functions can be automatically generated the tool takes care of the connection of the registers to the DIM driver abstraction hides diversity and complexity of the various hardware/bus types registers can be subscribed  register settings (address, etc.) sent to server to be stored in list  DIM services and commands created with unique register names write/read functions by calling just register names server takes care to treat it in the appropriate way according to type DIM client write/read(regname) DIM server Stores list of registers with the different settings Writes/reads to various types I2CLBUSJTAGetc. Ccpc/reg/cmnd Ccpc/reg/srvc recipes sets of predefined register settings (recipes) can be stored in a local cache or database at start up these recipes can be loaded to configure the experiment recipe types can vary according to run type DU_N DU_2 IT_DAQ_ST1 IT_DAQ_ST2 DCSHVDAQIDAQIT_V1 IT_DAQ_ST3 LHCb Control Units IT_HV_ST1 IT_HV_ST2 IT_HV_ST3 IT_HV_ST2_BOXC IT_HV_ST2_BOXDIT_HV_ST2_BOXU IT_HV_ST2_BOXA DU_1 IT_V2 IT_ST1IT_ST2IT_ST3 IT_DCS_ST2_LV IT_DCS_ST2_COOLINGIT_DCS_ST2_TEMP IT_DCS_ST2_GAS IT_DCS_ST1 IT_DCS_ST2 IT_DCS_ST2_HUM IT_DCS_ST3 IT_DCSIT_HV IT_DAQIIT_DAQ IT_DAQ_ST2_TELL1 IT_DAQ_ST2_FE Implementation each domain has a defined transition from one state into the other with the possibility to autorecover (in case of error) change in hardware (datapoint) can trigger a state transition of a device unit rules defined for control unit if its children make a state transition commands can cause state transitions and act on datapoint of device unit, triggering itself a callback function User Interface each device unit offers a set of panels to configure and interact with the hardware (datapoints) the control unit panel from where commands can be launched shows its own and the children’s state Partitioning to allow for more flexibility subparts of the tree can be excluded (either its state is ignored or commands are not allowed) helps to commission subdetectors or to ignore faulty hardware