1. M. Sozzi NA62 TDAQ WG meeting CERN – 20/10/2010
TDAQ news and issues
M. Sozzi NA62 TDAQ WG meeting CERN – 20/10/2010

2. Outline
TEL62
TDCB
Clock
L0 processor
Interfacing to PCs
Simulation

3. TEL62 - overview Newer FPGAs and memories
Doubled bus bandwidth with SL-FPGA
Fewer I/O pins in new chosen PP-FPGA + constraints for compatibility with LHCb cards = After price confirmation for 500 FPGAs: increased size of PP-FPGA.
All 5 FPGAs on the TEL62 of the same type: EP3SL110F1152C4N (4x the original)
Additional pins used for communication bus between PP-FPGAs (useful for LKr/L0)
Internal note NA distributed Comments from Lausanne group received

4. TELL1 PP PP SL
GbE PP PP

5. TEL62 PP PP SL
GbE PP PP

6. TEL62 - status
Some trouble with power distribution Difficult to estimate power consumption before firmware is ready
Design finished and start layout work at CERN at beginning of November
Components for 2 prototypes being procured
Production tests, firmware development, etc. ?

7. TDCB – status and plans
Some delays in mounting 2 prototypes of V4 expected next week
Firmware debugging ongoing
Detailed documentation being written
Need to prepare test-bench system
Preliminary tests in Pisa
Collect orders for first batch
First batch production (<16): [these might be used in a synchronization run]
Tests and validation by sub-detector groups

8. TDCB – validation # boards (first batch) Test where? Test who? CEDAR
CHANTI
LAV
RICH
Perugia
M. Piccini
CHOD
Perugia ?
M. Piccini ?
STRAWS (?)
IRC/SAC
MUV

9. Please check and update!
Components
Please check and update!
NA62 TDAQ
Fallback
CEDAR
GTK
CHANTI
LAV
STRAWS
RICH
CHOD
IRC/SAC
LKR
LKR/L0
MUV
TDAQ
TOTAL
Spares
Birmingham
Ferrara
Napoli
Pisa
CERN
Perugia
Perugia ?
Bulgaria
Roma2
Mainz
Channels
128
200
2496x2
7360
2000
196
Channels/station
260x2
1840
1000 64
Particle rate
10 MHz
Overall total rate
256 MHz
250 MHz
20 MHz
1 MHz
FE-DAQ distance 5m
<5m
<6m
TEL62 3 1 15 6 2 46 4 85 20
102
Of which: spares 8
0.24 23
0.23
Prototypes
First batch 7
Custom crates 10 9
0.00
Full TEL62 crates 14 17 5 22
0.12
0.14
VME crates
TDC boards 50
104 80
184
0.21 16 38
(first batch) 18
LTU
0.05
0.10
TTCex 21
# of lasers/module 73
# of lasers 24
117
TTCoc
Optical attenuators
0.04
TTCrq 35

10. Clock - modules
2 NA62 LTU prototypes available, ready for testing at CERN (Birmingham) Then: full production (19+ modules), paid
19 new TTCex modules for NA62 in production
Need to finalize number of lasers: Birmingham will distribute a final poll
Payment to NA62 account
5 old TTCex available in E-POOL

11. Clock - infrastructure
NA48 clock generator seems OK (spare?)
Organize central purchase of splitters, attenuators, fibres (Birmingham)
Finalize “officially supported” VME CPU
Install central crate(s) + CPU
Sub-detectors: define TTC distribution network
Lay down fibres in ECN3

12. Burst sequence
Sub-detectors only see 2 hardware signals: START OF BURST and END OF BURST
Start of burst signal (via TTC special command): reset timestamp counters
Start of burst trigger (via TTC): can send data
Normal triggers… send data
End of burst signal (via TTC special command): store timestamp counter value
End of burst trigger (via TTC): send end-of-burst data, stop data-taking

13. Calibration triggers
Sub-detector drives sub-detector calibration system and sends signal pulse to L0TP
L0TP delivers (if possible, if allowed) calibration trigger (after fixed, small delay)
In-burst vs. Out-of-burst calibrations: L0TP knows when the beam really ends (EE), and can: - Deliver special control trigger (e.g. to instruct sub-detectors to change pulser parameters) - Modify acceptance rules for calibration triggers
This scheme avoids need for additional trigger and synchronization lines

14. Central (L0) Trigger System (logical)
Pulser
40MHz
Gen
CHOD
Time-match L0 primitives, decide L0 and trigger type, check latency, downscale, monitor
MUV
Record L0 data
Data (GbE)
Farm
LKR/L0
L0 primitives (GbE)
Burst
SPS
LAV
(RICH)
Burst (GbE)
Farm
Generate special triggers
Monitor & record choke/error, send special triggers
Re-sync, encode
Choke/Error (LVDS)
Clock, L0, Burst (TTC)
All sub-detectors
All sub-detectors

15. Custom FPGA-based board
Central (L0) Trigger System (functional)
Pulser
40MHz
Gen
CHOD
MUV
Data (GbE)
Farm
LKR/L0
L0 primitives (GbE)
Custom FPGA-based board OR
Real-time capable PC
PC interface
Burst
SPS
LAV
(RICH)
PC (DIM)
Burst (GbE)
Farm
Spare
Choke/Error (LVDS)
Re-synchronizer
Clock, L0, Burst (TTC)
TTC modules
All sub-detectors
All sub-detectors

16. Central (L0) Trigger System (implementation)
Pulser
40MHz
Fan-out
Gen
CHOD
MUV
Data (GbE)
Farm
GbE TX
LKR/L0
L0 primitives (GbE)
GbE RX
Burst
SPS
LAV
Burst Fan-out
(RICH)
PC (DIM)
Burst (GbE)
Farm
Spare
PCIe
Core logic
PCIe
L0 Fan-out
LTU
TTCex
Choke/Error (LVDS)
Clock, L0, Burst (TTC)
All sub-detectors
All sub-detectors
Custom board or PC
Re-sync memory
PCIe custom board

17. L0TP: Real-time tests (Ferrara)
CPU
Involvement of very experienced group working on APE.
Using a custom board with 2 Intel CPUs (standard Linux, so far) and FPGAs to test real-time response of a basic L0TP algorithm
PCI-express
FPGA

18. L0TP: Real-time tests (Ferrara)
First naïve results: latency FPGA → CPU → MEM → FPGA with 0 computation time and standard Linux (depending on output buffering)
Now making more tests with more realistic time-matching algorithm
11μs
18μs
25μs
Further tests using standard PC and Pisa Altera Stratix IV PCIe development board (can such system be a L0TP?)
Expect report at next meeting

19. L0 Trigger Processor Rather limited progress
Detailed descriptive note in preparation
Temporary solution might be quite different from final system

20. Interfacing to PCs
Started development in Pisa of buffer manager code for readout
User-space independent processes
Fixed-format shared memory
Shared memory
GbE
CONS
PROD
GbE
PROD
CONS
CTRL
GbE
PROD
CONS
GbE
PROD
CONS
Procs

21. To be used for: L1 PCs
Multiple processes either running L1 (single sub-detector) algorithm or reading sub-detector data to L2 farm (on different event sets)
Sub-detector data
L1 algo
L1 primitives
L1 algo
GbE
L1 Trigger Processor PC
L1 algo
Sub-detector data
GbE
Readout
L2 farm

22. Condensed Sub-detector data
Maybe also: software L0 ?
Multiple GPUs running L0 primitive generation (for some sub-detector).
Algorithm time and latency not an issue.
Raw data-transfer time not an issue.
Control time? From CPU thread without operating system? From hardware PCIe sequencer?
Condensed Sub-detector data
GPU
L0 primitives
GPU
GbE
L0 Trigger Processor
GPU
GPU

23. Online software / run control
Existing solutions (DIM, SMI++) seem adequate
Possible help from CERN DCS group to port some of the above solutions to NA62
Need coordination and expertise within NA62
Write note describing NA62 requirements (volunteers willing to help?)

24. Towards a run control Common interface to run control:
Single machine per sub-system
Respond to the following commands: - Enable/disable global control - Cold-start initialization - Start run <number> <global_conf> <local_conf> - End run - Query status - Reset
No need of start/end burst (special triggers)

25. Simulation - Tools Some work started (see talks).
- Do we have everything we need for trigger simulation in the official MC? Can it be ran in a fast enough mode?
- Do we need a fast parameterized MC?

26. Simulation - Issues
- Verification of L0 rates - How much below 1 MHz can we stay? - Effect of LAV?
- Which control triggers can be allowed?
- CHOD/RICH vs. MUV acceptance matching
- Can CHOD be used in L0? Efficiency (accidentals)? - How much can RICH add? (At L1?)
- How much can STRAWS contribute at L1?
- What can be done realistically with LKr tiles at L0? Can something more be done with tiles at L1? Is some (summary) single-cell information useful at L1?
- Most significant correlation cut at L2? Which sub-detectors are required at L2? With which resolution?

27. Synchronization run? TEL62 test and firmware
No help available within NA62 - Roma Tor Vergata cannot help in common TEL62 firmware development - Still missing sub-detector involvement in firmware
Possibility of help from a PhD student in the STRAWS group
L0 Trigger Processor
Need commitments and tests