1. The CMS HCAL Data Concentrator: A Modular, Standards-based Design
A Review of the CMS HCAL DAQ
Gueorgui Antchev, Eric Hazen, Jim Rohlf, Shouxiang Wu
Boston University
Drew Baden, Rob Bard, Rich Baum, Hans Breden, John Giganti, Tullio Grassi, Aaron McQueen, Jack Touart
University of Maryland
Mark Adams, Kyle Burchesky, Weiming Qian
University of Illinois, Chicago
Wade Fisher, Jeremy Mans, Chris Tully
Princeton University
J. Elias, T. Shaw, J. Whitmore…
Fermi National Laboratory

2. Outline HCAL Trigger DAQ Overview HCAL Trigger Readout Card (HRC)
Requirements
Overview Diagram
DAQ Crate
HCAL Trigger Readout Card (HRC)
Block Diagram
Photo
HCAL Data Concentrator (DCC)
Motherboard
Link Receiver
Logic Board
HCAL Readout Controller (HRC)
Data Format
System Tests
Summary
13 September 2001
LEB - Stockholm

3. HCAL Trigger/DAQ Requirements
Digitize scintillator pulses every BX
Pulses occupy ~3 BX periods (25ns each)
Dynamic range required is about 104 (14-15 bits)
QIE (custom floating point ADC) used
Reliably assign energy measurement to a single BX for Level 1
F.I.R. filter implemented in FPGA (ala FERMI)
Synchronized across HCAL for input to Level 1
Transmit data to DAQ on L1A
13 September 2001
LEB - Stockholm

4. HCAL FE/DAQ Overview DAQ Crate (in UXA) Readout Box (RBX)
(On detector)
Raw Front-End Data
(One Charge sample per Bunch Crossing)
QIE
CCA
Readout
Module
GOL Tx D C H R C H
T R H T R H T R H T R
HPD T C
Triggered
L2 Data
Trigger
Primitives
DAQ
Level 1 Trigger
13 September 2001
LEB - Stockholm

5. HCAL Channel Counts Region Towers GOL Fibers Trigger Towers Barrel
Crates
Barrel
2,160
720
2160 6
Outer
Endcap
1,728 2
Forward
2016
672
144
Overlap
864
288
Total
9,360
3,120
4,176 26
13 September 2001
LEB - Stockholm

6. HCAL DAQ Crate 9U x 400 VME64x Crate Controller (HRC)
Commercial CPU/Bridge
TTC Fanout
Readout Cards (HTR)
Front-end data in (GOL)
Level 1 Out (Vitesse Cu)
Level 2 Out (LVDS)
Data Concentrator (DCC)
Level 2 in (LVDS) x 18
S-Link Out x 2
Front End: QIE ADC
GOL Optical Link Tx
Secondary DAQ
For Trigger
3 channels/fiber
@ 1.6 Gbps D C H R C H
T R H T R H T R H T R
LVDS
80MB/s
S-Link 64
Generic Links T C
1 Gb/s
Vitesse Cu
Primary
DAQ
Level 1 Trigger
13 September 2001
LEB - Stockholm

7. HCAL Trigger Readout Card (HTR)
Level 1 (Trigger) Path
Energy Sum and Bunch Crossing Determination
Output Trigger Primitives to Level 1 synch’d to BX
Level 2 (DAQ) Path
Zero-supress Level 2 data (energy filter)
Package raw data plus trigger primitives
Send to Data Concentrator every L1A
Implementation
FIR filter for bunch crossing (ala FERMI)
Single large FPGA for core logic
13 September 2001
LEB - Stockholm

8. HCAL Trigger Readout Card
Fanout BC0, Clock
TTCRx
Local BC0, Clock FE
Data
Level 1 Trigger Primitive Generator
Serial
Link
Board
Trigger
Primitives To
Level 1
Level 1 Pipeline
8 or 16 Fibers
24 or 48 Channels
Level 1 Pipeline
Energy Sum, BX Identification
Level 2 Pipeline
Level 1+2
Data To
DCC
Zero-Suppression, Data formatting
13 September 2001
LEB - Stockholm

9. HCAL Trigger Readout Card
Timing Inputs (TTC)
VME
Interface
&
Config.
(FPGA)
FE data Inputs:
8-16 digital serial fibers
16-72 Trigger Towers
800 Mb/s Gb/s
DESER
OPTICAL RX P1
DESER
...
DESER
LVDS TX
Level 1 and Level 2 data to DCC
FPGA
Trigger Path
DAQ Path P2
16 bit x 40MHz
SLB-PMC
(ECAL design)
Trigger Primitive outputs
1Gb/s copper link …
SLB-PMC
(ECAL design)
13 September 2001
LEB - Stockholm

10. HTR Demonstrator Board
4 G-Link Fiber Receivers
TTCRx Board
Altera APEX
Logic FPGA
Trigger Output
Mezzanine Card
Status:
Prototypes Built/Tested
New Design using GOL
Links underway
13 September 2001
LEB - Stockholm

11. HCAL Data Concentrator
DAQ Out
Slink 64
From HTRs
Trigger
Data Out
LVDS Serial
80MB/s
Slink
Spy Out
VME
Fast Status
13 September 2001
LEB - Stockholm

12. HCAL DCC Prototype Architecture
Overflow Warning
Fast Busy
To TTS
13 September 2001
LEB - Stockholm

13. HCAL DCC Motherboard 3-Channel LVDS Receivers DCC Logic Board
PC-MIP Card
PMC Card (oversize)
3-Channel
LVDS
Receivers T
33MHz
PCI bus 1 M
PC-MIP Card
3.3V T 32
DCC
Logic
Board
PC-MIP Card
Universe II
VME T
VME-PCI
PCI
64x
Bridge Y
PC-MIP Card
3.3V T M
33MHz
PCI bus 2
PC-MIP Card 32 64 T
PCI
Bridge X T
33MHz
PC-MIP Card 5V
PCI bus 3 T 5V
PCI busses give access to all devices for monitoring
PMC Card
M/T T
(Standard)
JTAG
Local
Spare PMC Site
Test/Config
Control
13 September 2001
LEB - Stockholm

14. HCAL DCC Motherboard PC-MIP sites Universe II VME64x PMC Connectors
VME-PCI Bridge
VME64x
(Geographical
Addressing)
PMC Connectors
PCI Bridges
T.I. PCI2031
Status:
10 Produced and Tested.
No known problems
Local control
Flash memory, JTAG
13 September 2001
LEB - Stockholm

15. PC-MIP 3 Channel Link Receiver
3 “Channel Link” LVDS receivers
PCI target interface
On-board logic:
ECC (Hamming) plus parity
Correct 1-bit, detect multi-bit errors
On-the-fly Event Building
Event number checking
Overflow warning (discard data payload on overflow)
Missing header/trailer detection & repair
Monitoring:
Count of words, events, errors
Status update on “marked” event for synchronization of monitoring
Status: 20 second-generation prototypes build (design is done)
LVDS Serial Receivers
512k byte buffer
Per channel
(i.e. National Semi "Channel Link)
SSRAM 512kx32
DS90CR286
Link A
PCI Bus
ALTERA
DS90CR286
(32 bit
33 MHz)
Link B
(ACEX 1K130)
DS90CR286
Link C
FPGA
(PCI Interface and
readout/monitor logic)
Link Receiver Board(s)
Logic Board 3 2 1 3 2 1 1 3 2 1
Event Fragments
combined during
Event Fragments
PCI bus transfer
13 September 2001
LEB - Stockholm

16. PC-MIP 3-Channel Link Receiver
Altera ACEX FPGA
EP1K100FC484-1
PC-MIP Interface
(PMC 33MHz 32 bit)
Programming Connector
For flash memory
MT55L512V ZBT
512Kx32 SSRAM
DS90CR256A
Channel-Link RX
Cypress Clock
Multipliers
13 September 2001
LEB - Stockholm

17. Monitoring (via VME to CPU)
DCC Logic Board
TTCRx
L1A FIFO
2 Events max
Level 2 FIFO
200 Mbytes/s
Average
Data
Decoder
~100MHz
Processing
PCI 1
Level 1 FIFO
DAQ FIFO
Event
Builder
Summary FIFO
Trigger FIFO
33MHz x 32 bits
Two busses
Level 2 FIFO
Data
Decoder
Spy FIFO
PCI 2
Level 1 FIFO
Summary FIFO
Monitoring (via VME to CPU)
13 September 2001
LEB - Stockholm

18. DCC Prototype Logic Board
Features:
On-board TTCrx controls operation
3 Altera FPGAs for PCI interfaces (use Altera PCI-MT/32 core)
Xilinx Virtex-2 contains all other logic:
Event Builder
Monitoring
Buffering (DDR SDRAM interface at 800Mbytes/s)
S-Link output (32 in demo; 64 final)
On-board flash memory for FPGA initialization
JTAG Interface
Status:
Prototype PCB Working
Continuous DAQ transfer at 80MB/s demonstrated (one PCI bus only)
2nd Prototype layout done
JTAG
TTCrx
S-LINK 64
XC2V1000
S-LINK
PCI 1
PCI 3
PCI 2
2Mx32
SDRAM
1Mx8
FLASH
13 September 2001
LEB - Stockholm

19. DCC Prototype Logic Board
PCI-1
32 bit
33 MHz
HTR inputs
1-9
PCI-3
64 bit
33 MHz
TTCRx
PCB (back)
DDR SDRAM
Xilinx
XC2V1000
PCI-2
32 bit
33 MHz
HTR inputs
1-9
13 September 2001
LEB - Stockholm

20. DCC Prototype Logic Board
TTCRx
PCB
S-Link 32
LSC
13 September 2001
LEB - Stockholm

21. Data Concentrator Demonstrator
LVDS Serial
Link Receiver
TTCRx
S-Link LSC
(Transmitter)
DCC Logic Board
Spare PMC Site for Testing
13 September 2001
LEB - Stockholm

22. HCAL Readout Controller (HRC)
Run Control
Initialization, shutdown
“Slow” monitoring via VME
Error recovery:
Monitor status registers of modules via VME
Report serious errors via DCS
Reset/Restart on command
TTC Fanout
Fanout encoded TTC to all modules
Fanout Locally Decoded CLK, BC0 to HTR modules
I2C Control of local TTCrx
Custom 6U/9U Adapter
6U VME
CPU
Or Bus Bridge
LVDS
(3 pairs)
To HTRs
And DCC
LVDS Fanouts
I2C
TTCrx
From TTC
Optical F/O
13 September 2001
LEB - Stockholm

23. HCAL DAQ Data Format Data format follows TriDAS Guidelines   
HCAL payload: (details t.b.d)
Raw QIE (ADC) samples
Level 2 Filter output
Trigger Primitives
Zero-suppression mask
Error summary:
Front-end errors
Uncorrected Link errors
Synchronization errors
We will stay tuned for updates to the data format   
13 September 2001
LEB - Stockholm

24. Radioactive Source Calibration Test
System Tests
Radioactive Source Calibration Test
NOW at Fermilab
Record data at 80Mbyte/s for detector calibration
Uses demonstrator hardware for all system components and verifies basic functionality
Test Beam – Summer ’02 at CERN
Record data at realistic LHC rates
A few hundred channels
Uses (2nd) prototype hardware for all components
Verify high-rate operation under realistic conditions
13 September 2001
LEB - Stockholm

25. Summary Front-End: HTR: DCC: HRC: Major Concerns:
HCAL RBX
Front-End:
RBX Mechanics/Cooling Designed+Prototyped
Readout Card prototypes under test
HTR:
6U Demonstrator prototypes working
New prototype with GOL under design
DCC:
9U Demonstrator prototypes working
No major changes anticipated for production version (significant FPGA coding remains)
HRC:
Use commercial CPU for now
TTC Fanout Design Done (U.I.C.)
Major Concerns:
QIE Performance (pending prototype tests now underway)
Performance of 1.6 Gbit GOL link
HTR/DCC FPGA Design – Quite complex
HTR Demonstrator
DCC Demonstrator
VME Pentium CPU
13 September 2001
LEB - Stockholm

26. Backup Slides
13 September 2001
LEB - Stockholm

27. To Trigger Throttling System
HCAL DAQ Buffering
HTR
DCC
L1A
40MHz
128kb FIFO 1000 events
(2) 33MHz 32 bit
PCI busses
Derand.
Buffer
Link Tx
Link Rx
~ 100MHz
Processing
8Mb Buffer
> 4000 events
PCI
Derand.
Buffer
Link Tx
Derand.
Buffer
Link Tx
Link Rx
Event
Builder
PCI
32/33
PCI
Derand.
Buffer
Link Tx
Derand.
Buffer
Link Tx
Link Rx
S-Link64
Interface
Derandomizer Buffer Protected against overflow by trigger rules
Link RX logic discards data when FIFO “almost full” (block structure maintained)
S-Link
LSC
LVDS link speed same as HTR output logic  no bottleneck
[18 Links per DCC]
Busy/
Ready
Overflow
Warning
Level 2 Data
To RUI
To Trigger Throttling System
13 September 2001
LEB - Stockholm

28. HCAL Timing / L1A Distribution
Fanout Both:
Encoded TTC signal
For synchronization with incoming FE data (individual skew control)
Decoded BC0, CLK
For synchronization across all HCAL and ECAL of TPG to Level 1
Similar to ECAL solution (J. Carlos Dasilva)
TTC Fanout (1 per crate)
PIN Diode
TTCrx
Decode
FPGA
LVDS F/O
LVDS F/O
HTR Cards
BC0, CLK (TPG)
TTCrx
Decode
FPGA
BC0, CLK (FE)
BC0, CLK (TPG)
TTCrx
Decode
FPGA
BC0, CLK (FE)
13 September 2001
LEB - Stockholm

29. HCAL Controls and Monitoring
Fast Controls (via TTC):
L1A, Start Run, Stop Run
Reset (complete and partial – need to define!)
Fast Monitoring (dedicated signals to TTS)
Overflow Warning (buffer full above preset limit)
Busy/Ready (reset, start/stop completed)
Slow Monitoring (link errors, loss of sync, etc)
Counters in FPGAs collect information in real time
Reported via CPU
13 September 2001
LEB - Stockholm

30. HTR Details Rx Input: Lookup table (LUT) Pipeline (“Level 1 Path”)
20 bit frames
40 Mframe/sec 1
PLD
Ch1A data
Ch1B data 7
LUT
Ch1A address
Ch1B address
7/9 16
Level 1 Path
(Trigger Primitives)
Level 2 Path
(Pipeline)
Level 1
Frame Bit
“Cap” Bits 2
“Cap” Error Bit
DCC BC
Input:
QIE 7 bit floating plus 2 bits “cap”
Lookup table (LUT)
Convert to 16 bit linear energy
Pipeline (“Level 1 Path”)
Transmit to Level 1 trigger, buffer for Level 1 Accept, 3 ms latency
Level 2 Buffer (“Level 2 Path”)
Asynchronous buffer, sized based on physics requirements
13 September 2001
LEB - Stockholm

31. PCI Development PC-MIP Cards VME Motherboard Integration
Use adapter in standard PC motherboard
LRB Prototype was completely developed before motherboard
VME Motherboard
Test all sites with standard PC-MIP and PMC cards
Integration
Integration of motherboard and mezzanine cards was quite smooth
PCI Interface logic
Use Altera Core (motherboard and logic boards)
Own design simple slave for LRB
PCI Test Setup
PC-MIP to PCI Adapter
(Commercial)
PC-MIP card
Under test
Standard PC Motherboard
13 September 2001
LEB - Stockholm

32. Flash ADC Quantization
13 September 2001
LEB - Stockholm