1. Control and operation of detector systems and their readout electronics in a complex experiment control system
Stefan Koestner (CERN) on behalf of the LHCb Online Group
Multi-Event-Packages (MEPs) sent to PC farm
(Gigabit Ethernet)
Hardware Layer & design issues:
Tell1
Tell1
Tell1
Readout Supervisor
Throttle Or
The LHCb experiment:
Single-arm forward spectrometer
dedicated to B-physics
TeV, luminosity=2·1032 cm-2s-1
acceptance: (250)mrad
1012 bb/year, full B spectrum
Trigger & Fast Control:
distributes clock
sends trigger and broadcasts
assigns farmnode IP
sends throttle and resets
L1 electronics:
1 MHz readout (~35 GB/s)
collection and preprocessing
push-protocol supposes enough buffer in next stage
Dose outside detector up to 10 Gy in 10 yrs
Cavern-side:
Counting house-side:
No radiation hard electronics required
PC-Farm:
High Level Triggers
Reconstruction
Storage
Embedded Microcontrollers (CCPC):
readout boards in the radiation save area host embedded microcontrollers (credit card PC)
access to I2C, JTAG and general purpose bus
isolated access path  robustness
controls network (10/100 Ethernet) separated from data network
generic server to communicate with ECS (DIM) client
L0 trigger
~ 1 MHz, 4 µs fixed latency
Time synchronized to FE-chips
DIM protocol
Service box
Inside concrete bunker close to detectors – requires radiation tolerant electronics:
frontend configuration (SPECS mezzanine) and analog to digital conversion
Configuration DB:
stores register settings
uploaded at start up
Optical fibres ~100m
SPECS mezzanine card:
Hosted on controller cards in radiation area
Contains the SPECS slave – a radiation tolerant ACTEL ProASIC FPGA which translates the SPECS protocol into the required I2C, JTAG or local bus
SPECS is a custom protocol made for LHCb to work in radiation and B-field environment
SPECS is used for control and readout of ASICs (e.g. preamps) and sensors (e.g. temperature)
For infrastructure the ELMB (CAN bus) is used
e.g.
Inner Tracker
Experiment Control System
based on Industrial SCADA (PVSSII)
LHC common JCOP framework
distributed and scalable system
coherent interface
platform independend (CONTROL interpreter)
SPECS Master:
PCI card which can talk with up to 32 SPECS slaves
generic server running on same PC to communicate with ECS (DIM) client
Oracle/SQL
Analog Data (twisted pair ~5m)
Long distance I2C
SPECS ~100m (4 different unidirectional lines)
DIM protocol
Communication Layer & Protocols:
CCPC – Credit Card sized PC
15 different types of DAQ and trigger boards (in total some 400)
a few hundred registers each (monitor) and memory blocks (upload)
common readout board (Tell1) based on FPGA technology to adopt specific needs
DIM –
Distributed Information Management System
portable lightweight communication layer to interface hardware from ECS
if server crashes it can easily republish on DNS node (Robustness)
clients can be installed on any machine just specifying DNS node (Portability) – no need to take care of connectivity
SPECS – Serial Protocol for the Experiment Control System
10 Mbit/s serial link for configuration of remote electronics
single master multi-slave (up to 32) bus – simple, fast and cheap
serial bus with two signals in each direction (2 times clk & data)
BLVDS on shielded twisted pair (8-bin RJ45, cat 5 Ethernet)
FPGA-based readout board
ADC conversion or optical conv.
synchronisation, reordering, pedestal subtraction, common mode sub., zero suppression (PP)
MEP packing, IP framing (SL)
Gigabit Ethernet Link
DIM server running on CCPC or SPECS master PC publishes services to DIM Name Server (DNS) from where the client (ECS) can subscribe to it.
Data exchange peer to peer from server to client
Client sends commands (write/read) – server updates services (data/status)
I2C like protocol without acknowledgements
slave can send interrupt and master repeats
clock only active during transfer (10 MHz)
variable number of words (<256) with 10 bits
Separated ECS interface (CCPC)
4” – SM520PCX from Digitallogic
i486 compatible microcontroller (AMD ELAN 520)
Linux Kernel at 133 MHz
reading from local bus 20 MB/s
filesystem shared over server
PVSS Event
manager
Database
PVSS
Control
API
GUIs
Distribution
Driver
DIM client
PVSS00dim
User interface layer
Processing layer
Communication and memory layer
Driver layer
Distributed System
SPECS slave designed as portable VERILOG code
implemented in ACTEL APA150 FPGA (tested up to 40 krad)
SEU and SEL immune due to triple voting and one-hot state used
SPECS mezzanine board houses the SPECS slave
provides 16 long distance JTAG chains (4 signals each)
provides 15 long distance I2C busses with selectable frequency
local i2C bus and 16 bit parallel bus (8 bit address range)
DCU chip with 6 ADC channels (12 bit) and 1 temp sensor
32 configurable I/O lines
The PVSS software architecture comprises a number of managers which may run on independent machines and communicate with the event manager via a PVSS internal protocol on top of TCP/IP
a DIM driver is implemented which allows to communicate with hardware over ethernet
access to board via gluecard over a PLXPCI9030 bridge:
Parallel bus (8/16/32), 3 fast JTAG chains (2 MHz) to load firmware, 4 I2C lines and 9 GPIO lines.
low level libraries under SLC4
SPECS multi-master board – 4 different busses
standard 32-bit 33 MHz PCI board
inserted on the same PC where SPECS server is running
SPECS master PC
running SPECS server or
embedded controller
running CCPC server
DNS
node
server
Support PC
running DNS node
Ethernet
Abstraction Layer & Finite State Machines:
Modelling of the entire experiment with Finite State Machines
in order to model the experiment at an expert system level states and actions have to be defined
From the top node (run control) commands can be propagated downwards a hierarchical tree
Status and alarms can be propagated upwards the hierarchical tree to the run control
The final branch of a tree is always a device unit acting on the hardware (connected to datapoint!)
Additional control units can be implemented to give a better logical structure (domains)
Representation of the hardware inside the Experiment Control System
PVSS datapoints
each type of hardware has to be represented as a PVSS datapoint type, from which the various instances can be derived (datapoints)
a datapoint is a complex structure connected to the internal PVSS database
if the data inside a datapoint changes a callback function or alert can be launched
registers can be organized as substructures to mimic the organization of a hardware
each register consists of various datapoints holding the settings of each register or allowing to launch DIM commands or receive DIM services
Implementation
each domain has a defined transition from one state into the other with the possibility to autorecover (in case of error)
change in hardware (datapoint) can trigger a state transition of a device unit
rules defined for control unit if its children make a state transition
commands can cause state transitions and act on datapoint of device unit, triggering itself a callback function
Partitioning
to allow for more flexibility subparts of the tree can be excluded (either its state is ignored or commands are not allowed)
helps to commission subdetectors or to ignore faulty hardware
FwHw tool
a tool (GUI) was developed which eases the development of hardware types
scripts using the framework functions can be automatically generated
the tool takes care of the connection of the registers to the DIM driver
DU_N
DU_2
IT_DAQ_ST1
IT_DAQ_ST2
DCS HV
DAQI
DAQ
IT_V1
IT_DAQ_ST3
LHCb Control Units
IT_HV_ST1
IT_HV_ST2
IT_HV_ST3
IT_HV_ST2_BOXC
IT_HV_ST2_BOXD
IT_HV_ST2_BOXU
IT_HV_ST2_BOXA
DU_1
IT_V2
IT_ST1
IT_ST2
IT_ST3
IT_DCS_ST2_LV
IT_DCS_ST2_COOLING
IT_DCS_ST2_TEMP
IT_DCS_ST2_GAS
IT_DCS_ST1
IT_DCS_ST2
IT_DCS_ST2_HUM
IT_DCS_ST3
IT_DCS
IT_HV
IT_DAQI
IT_DAQ
IT_DAQ_ST2_TELL1
IT_DAQ_ST2_FE
User Interface
each device unit offers a set of panels to configure and interact with the hardware (datapoints)
the control unit panel from where commands can be launched shows its own and the children’s state
DIM client
write/read(regname)
abstraction
hides diversity and complexity of the various hardware/bus types
registers can be subscribed  register settings (address, etc.) sent to server to be stored in list  DIM services and commands created with unique register names
write/read functions by calling just register names
server takes care to treat it in the appropriate way according to type
Ccpc/reg/cmnd
Ccpc/reg/srvc
DIM server
Stores list of registers with
the different settings
Writes/reads to various types
recipes
sets of predefined register settings (recipes) can be stored in a local cache or database
at start up these recipes can be loaded to configure the experiment
recipe types can vary according to run type
I2C
LBUS
JTAG
etc.