RICH Data Flow Jianchun Wang

2 HPD Readout Electronics 944 HPDs 163 channels / HPD 1 FE Hybrid / HPD ~160 FEMs 6 FE Hybrids / FEM 1-13 Cables / FEM DCBs ~8 FEMs / DCB Event Data Timing and Control Data

3 Serial Line Cables  Timing and control link(tcdat) at 151 MHz (20  refclk), 16-bit T/C word per BCO. Reference: BTeV-doc-1920 by Walter Stuermer  Moving information between HPD multiplexers (MUX, FEM) and data combiner boards (DCB).  One or more cables per multiplexer board.  Each cable has 4 point-to-point serial lines, each with a differentially driven twisted pair (e.g. Category 6 cable).  All data is encoded in 8B10B format.  Reference clock link(refclk) at 7.6 MHz (132 ns BCO).  Event data links(evdat0, evdat1) at 640 MHz (7  12  refclk), 2  67.2 bits data per BCO, or 2  63.6 Mbyte/s.

4 The HPD Occupancy  BTeV GEANT simulation with 2 interactions per bunch crossing.  Total about 14,000 events simulated (thanks to Raja).  Columns with odd ID are shifted down by a half HPD size.  Simplified row ID.  Electronic noise is not included in this plot.

5 Occupancy of The Hottest HPD  2 interactions per bunch crossing ( 132 ns ) are simulated.  Electronic noise is not included in the plot.  The hottest HPD has 19.9 hits per event, or 17.2 per bunch crossing.  There are 1.26 hits/event averaged over 944 HPDs.

6 Occupancy of The Hottest HPD  6 interactions per bunch crossing ( 396 ns ) are simulated.  Electronic noise is not included in the plot.  The hottest HPD has 52.6 hits per event, or 52.5 per bunch crossing.  There are ~3.7 hits/event averaged over 944 HPDs.

7 HPD Grouping  HPDs are grouped into 6-HPD modules. Each module is connected to 1 FEM board.  Some modules at the edge of HPD arrays contain less number of HPDs.  Only one of two HPD arrays is shown in the plot.  The last columns in both arrays are not grouped, and probable will be skipped.  There are 150 groups in total.

8 Occupancy of HPD Groups 2 Interaction per bunch crossing, no electronic noise. The hottest group: 57 hits / bunch crossing.

9 Occupancy of HPD Groups 6 Interactions per bunch crossing, no electronic noise. The hottest group: 180 hits / bunch crossing.

10 How Many Cables Needed?  The hottest HPD group with 2 interactions per bunch crossing:  Occupancy: 57.0 hits / bunch crossing  Noise level:  1%  Total occupancy:  hits / bunch crossing  Address length: 18 bits/hit (10 bits for 944 HPD, 8 bit for 163 channels)  Data moving: bits / bunch crossing (132 ns)  Cables needed: (  cables  + 20% overhead:  (  cables   Total number of cables for 2 interactions per bunch crossing, 1% noise and 20% overhead: 458 cables.  With 6 interactions per bunch crossing, there are more physics hits (~ factor of 3) per bunch crossing, save level of noise, but more time to move data (factor of 3). In total 324 cables are needed.

11 Effect of Noise 2 interactions/bunch crossing (132 ns)6 interactions/bunch crossing (396 ns) No overhead, with 1% noise or no noise

12 Proposed Number of Cables 2 int 2 interactions/bunch crossing Number of cables: interactions/bunch crossing Number of cables: 324 Including1% noise, 20% overhead

13 Data Flow Structure  This is just a guess. I am not quite sure how several lines will share the load, rotating to serve events or splitting an event. Currently I assume there is no time wasted.  128K FIFO is quite small. For the simulation it is big enough, and can save more than 100 bunch crossings for the hottest group. FPGA 128K FIFO 8B10B encoder buffer

14 Data Flow Simulation  For each bunch crossing, the elastic and inelastic collisions are fully simulated with BTeV GEANT with average 2 interactions per bunch crossing. The channels with no physics signal are simulated individually to have 1% change of noise hits. Note that 13.5% of the bunch crossing, there is no interaction but noise only.  For each bunch crossing a block of event data is generated, with 18-bit word for each hit. And two 18-bit words as event header: an event tag (3FFFF) and a BCO.  For each bunch crossing, the event data block is saved to FIFO first. Then part or all of the data saved in FIFO is transferred to outside via 8B10B encoder and cables within the bunch crossing time (132 ns).  If the buffer overflows, then only three 18-bit words are stored and transferred, with the extra word indicating the overflow status.  It is impossible to generate millions of events using BTeV GEANT. Instead, 14,000 events with 2 interactions per bunch crossing are fully simulated. The simulation then randomly pick events from this pool.

15 The FIFO Occupancy  The highest occupied FIFO happens to be of the hottest group.  With 4 million bunch crossings simulated, the maximum occupancy, ~30 Kbits is much smaller than FIFO capacity, 128 Kbits picked for this simulation. 2 interactions/bunch crossing6 interactions/bunch crossing

16 Summary  This study is to roughly estimate what we need to move data out from front end electronics.  With noise level of 1% and 20% overhead, we need 458 cables to move the data from FEM to DCB at 2 interactions per bunch crossing and BCO time of 132 ns.  We need 324 cables at 6 interactions per bunch crossing and bunch crossing time of 396 ns.  The simulation shows that the FIFO occupancy reaches 26Kbits. A FIFO size of 128 Kbits is big enough.

17 Occupancy of The Hottest HPD Group  2 interactions per bunch crossing ( 132 ns ) are simulated.  Electronic noise is not included in the plot.  The hottest HPD group has 66.2 hits per event, or 57.0 per bunch crossing.  There are 7.9 hits/event averaged over all HPD groups.

18 Occupancy of The Hottest HPD Group  6 interactions per bunch crossing ( 396 ns ) are simulated.  Electronic noise is not included in the plot.  The hottest HPD group has180.2 hits per event, or per bunch crossing.  There are 23.1 hits/event averaged over all HPD groups.

19 HPD Occupancy 2 interactions / bunch crossing, no electronic noise The hottest HPD: 19.9 hits/event, 17.2 hits/bunch crossing

20 HPD Occupancy 6 interactions / bunch crossing, no electronic noise The hottest HPD: 52.6 hits/event, 52.5 hits/bunch crossing

21 Row ID by Tomasz

22 The FIFO Occupancy  With 4 million bunch crossings simulated, the maximum occupancy, ~30 Kbits is much smaller than FIFO capacity, 128 Kbits picked for this simulation. 2 interactions/bunch crossing6 interactions/bunch crossing

23 Summary of Numbers 43DCBs 58 (100)52 (92)Cables (data lines) 24 FE Modules PMT 3633DCBs 682 (1135)659 (1056)Cables (data lines) 332 FE Modules MAPMT 30 DCBs 517 (951)498 (940)Cables (data lines) 200 FE Modules HPD 16-bit format (4 extra words) 18-bit format (2 extra words)