1. 28 February 2003Paul Dauncey - HCAL Readout1 HCAL Readout and DAQ using the ECAL Readout Boards Paul Dauncey Imperial College London, UK

2. 28 February 2003Paul Dauncey - HCAL Readout2 CALICE ECAL has 30 layers, 18  18 channels/layer, 9720 total Each gives analogue signal, 14-bit dynamic range Very-front-end (VFE) ASIC (Orsay) multiplexes 18 channels to one output line VFE-PCB handles up to 12 VFEs (216 channels) Cables from VFE-PCBs go directly to UK VME readout boards ECAL readout electronics overview

3. 28 February 2003Paul Dauncey - HCAL Readout3 VFE-PCB needs several control and timing signals VFE-PCB control and timing Analogue: 12 differential input, 2 differential output from readout board Digital: 4 LVDS input, 9 LVDS output from readout board, plus 7 uncommitted LVDS spares (could be in or out; assume out here)

4. 28 February 2003Paul Dauncey - HCAL Readout4 Readout board structure Front End (FE) FPGA controls all signals on VFE-PCB cable Back End (BE) FPGA gathers and buffers all event data from FE and provides interface to VME Trigger logic in BE for timing and backplane distribution; only active in one board First readout board prototypes this summer, final boards end 2003; cost per board ~ €10k Readout board overview

5. 28 February 2003Paul Dauncey - HCAL Readout5 Board clock ~40 MHz Almost all signals edges timed to 25ns periods Sample-and-hold timing needs to be better than 10ns Effectively the VFE-PCB trigger; VFE shaping time of 180ns sets maximum trigger latency also Fine-tune rising edge of signal on  8 clock, ~3ns steps Each cable can be timed independently Dual 16-bit ADCs and 16-bit DAC DAC fed back for internal as well as VFE calibration No data reduction planned in hardware Read out all ECAL channels for every event 2 Bytes  1728 channels/board = 3.5 kBytes/board 2 Bytes  9720 channels = 19 kBytes total On-board buffer memory; 8 Mbytes Allows up to ~2k events for ECAL during beam spill Readout board features

6. 28 February 2003Paul Dauncey - HCAL Readout6 Trigger/DAQ logic Idea is to implement trigger/DAQ interface for all subsystems

7. 28 February 2003Paul Dauncey - HCAL Readout7 Multiple (4) trigger and “activity” (background) inputs Can be enabled and disabled through VME Need clean events with no pile-up Activity prevents subsequent triggers within configurable time period Trigger records following activity; read out with event Trigger latches off logic, preventing further triggers Reset through VME; single reset for system so no synchronisation issues Trigger is fanned out and sent to backplane connector Distribution to other boards in crate using custom cable Point-to-point LVDS (probably) Trigger also configurably delayed and output to connector For distribution to other systems (beam monitoring, HCAL?) Assuming 16 outputs, LVDS; we have a LVDS  NIM converter available Beam on signal allows buffering during spill, readout after spill Trigger handling

8. 28 February 2003Paul Dauncey - HCAL Readout8 We have tried to keep design flexible enough to allow readout boards to be used for HCALs also Assuming the on-detector electronics is compatible Aim for single crate (or at least a single DAQ PC) if possible No need for inter-PC communication Single VME interface reduces amount of code needed Makes offline analysis more uniform for different HCALs also Trigger use and distribution is more uniform Particularly if single VME crate Common buffering method if not reading every event before next trigger One downside may be less flexibility during testing before beam (Semi-)custom 9U crate (CMS tracker standard);would need to buy specific crate for tests ~ €6k Is rate high enough with single VME crate? See later… Motivations for reuse of readout boards

9. 28 February 2003Paul Dauncey - HCAL Readout9 Tile AHCAL has ~1500 analogue channels (AFAIK) Each ECAL readout board has 96 ADCs; 16 boards, need 2 crates Multiplex AHCAL to fit into smaller number of boards? E.g. 8-fold multiplexing could fit into two readout boards Are systems compatible? Can something like twelve (multiplexed) signals be fed into each cable? Is multiplex control compatible with number of LVDS signals available? What is maximum allowed trigger latency for AHCAL? Are ADC input, DAC output ranges compatible? Trivial if using (slightly modified) VFE chip But who is designing equivalent of VFE-PCB? Otherwise would probably need (non-UK) firmware development FE FPGA only; rewrite cable signal handling, keep BE interface BE and VME would be identical; retains common VME interface Use for analogue HCAL

10. 28 February 2003Paul Dauncey - HCAL Readout10 Digital HCAL has ~350k channels (AFAIK) Assume zero suppression, as data volume otherwise is high (45 kBytes/event), and data concentrator on detector Use LVDS lines for clock, trigger, configuration, control and data readout of concentrator What is maximum allowed trigger latency for DHCAL? Clearly requires FE firmware development, as for AHCAL But BE and VME are still uniform across system Are the number of lines compatible with concentrator interface? Minimum would be clock, trigger, serial in, serial out per concentrator Each cable could handle 4 concentrators, i.e. 12 outputs, 4 inputs per readout board, so each readout board handles 32 concentrators E.g. with 1k channels per concentrator, see talk by Gary Drake Feb 6, this requires 11 readout boards (What sets number of channels per concentrator?) Leaves three free VME slots in the crate for beam monitoring readout Use for digital HCAL

11. 28 February 2003Paul Dauncey - HCAL Readout11 ECAL 18 channels multiplexed per line; all lines read in parallel ADC maximum rate is 500kHz; actually needs around 3  s per channel Total time around 50  s per event Analogue HCAL With multiplexing at or below 18 channels per line, will be less than or equal to the above (assuming able to multiplex output at 500kHz) Digital HCAL Time set by maximum data volume of all concentrators for each event For 4 Bytes/channel, assume average maximum data volume per concentrator is 1 kByte = 250 channels (pessimistic?) With single serial line at 40 MHz, rate is 40Mbits/s = 5 MBytes/s, so time required is 200  s Transfer rates to readout board

12. 28 February 2003Paul Dauncey - HCAL Readout12 Readout board can run in any of three modes Single trigger readout Read all data between each trigger; slow but sure Semi-buffered readout Read minimal information from each board per event; e.g. trigger number and buffer occupancy Must read any unbuffered electronics for each event Buffer rest of data and read later (after beam spill) Fully-buffered readout Nothing read from readout boards per trigger; only unbuffered electronics All readout board data buffered until end of spill A missed trigger corrupts whole spill of data Semi-buffered is to avoid the need for a more sophisticated fast control and timing system Modes of operation

13. 28 February 2003Paul Dauncey - HCAL Readout13 Assumptions: Average event data: ECAL 19 kBytes, DHCAL 5 kBytes (?) (AHCAL 3 kBytes), other (beam monitoring, etc) 2 kBytes (??) Average time for clear/trigger/interrupt from end of one event to start of read of next: 0.1ms. Implies beam rate >10kHz VME data speeds: non-block transfer 4 MBytes/s, block transfer 16 MBytes/s Semi-buffered readout of ECAL and HCAL is 400 Bytes per event total, read out with non-block transfer so takes 0.1ms All beam monitoring, etc, data is always unbuffered and is read out with non-block transfer so takes 0.5ms per event Significantly bigger than data transfer time to readout board of 0.2ms Transfer time is not a limiting factor Disk write speed: ~ 20 MBytes/s; always outperforms VME DAQ readout rates

14. 28 February 2003Paul Dauncey - HCAL Readout14 Rates for the different modes: Single trigger: 24kBytes block transfer 1.5ms, total 2.3ms per event; rate limited to 430Hz Semi-buffered: 400 Bytes takes 0.1ms so 0.9ms per event during spill; rate limited to 1.1kHz. 1.5ms per event outside of spill; rate limited to 670Hz Fully-buffered: Only time saved is readout of 100 Bytes, so 0.8ms per event during spill; rate limited to 1.2kHz. 1.6ms per event outside of spill; rate limited to 630Hz No significant gain from using fully-buffered mode over semi- buffered mode Simpler trigger, better error detection and event verification in semi-buffered mode DAQ readout rates (cont)

15. 28 February 2003Paul Dauncey - HCAL Readout15 Want to save unsuppressed data for noise and pedestal studies but also want processed data for analysis Raw event size ~ 26 kBytes (assuming no significant “gzip” compression) Processed event size ~ 4 kBytes (assuming DHCAL can be reduced?) Total disk space needed per event ~ 30 kBytes Want to run in many configurations Particle type, energy, HCAL, preshower, angle, etc; ~ 100 configurations Need ~ 10 6 events per configuration Total event sample ~ 10 8 events ~ 3 TBytes Just brought 3.2 TBytes for BaBar for ~ €6k We have budgeted ~ €12k for DAQ disk Processed data is ~ 400 GBytes DESY have said they will host data long term Total data sample sizes

16. 28 February 2003Paul Dauncey - HCAL Readout16 How would these numbers change if DHCAL raw bits are read out? Event data size is now fixed at 45 kBytes per event Transfer time to readout board reduced to 0.03ms as now always transfering 1kbit = 128 bytes Buffer memories will contain 45 kBytes spread over 11 boards, i.e. ~ 4 kBytes per board, so can still buffer ~ 2k events VME readout modes: single trigger becomes 210Hz, semi- buffered and fully buffered become 250Hz out of spill but are still above 1kHz during the spill The total event size is 66 kBytes per event, giving a total data sample size of 7 Tbytes; processed data size is unaffected Could always do zero suppression in DAQ software so disk space as before Worth considering if this simplifies (and reduces cost) of on- detector electronics significantly What if no zero suppression in DHCAL?

17. 28 February 2003Paul Dauncey - HCAL Readout17 With some firmware effort, the ECAL readout boards may be able to be used for both the AHCAL and DHCAL options Savings in software effort, code uniformity, trigger distribution, etc. Rates of around 1kHz during a spill, 600Hz outside of a spill would be possible Buffering up to 2k events during the spill on-board Data volumes would be large but affordable Even if DHCAL is read out unsuppressed BUT… important to cross check the assumptions made here are realistic Compatibility of different HCAL readouts Amounts of HCAL data Beam spill timings Summary