1. Physics 335 project update Anton Kapliy April 2007

2. Experiment - JPARC Process: FCNC sd → dd Final state: Two photons Major BG: Four photons, two missed Critical to correctly detect all photons that hit CsI, and nail down the timing

3. Data flow PMT – 1 channel (total = hundreds) 14-bit DIGI 125 MHz FPGA Trigger Data packing Pipelining Interface to PC Amp/shaper VME, GLink Sum of 16 Trigger board … x16 Global trigger My work

4. Why need a trigger? >300 RO channels, digitized at 125 MhZ 300*(14 bits)*(125,000,000) = 60 TB/sec No computer can keep up! BUT: mostly junk Physics simulations predict event rate <100KHz for entire detector! (14 bit)*100k = 2 GB/sec Can save and apply HLT on PC Note: in reality, the rates must be multiplied by 30

5. Data gets into FPGA 2 µs data pipeline data Local trig (threshold) yes/no 2 µs trigger pipeline Data waits here for global trigger decision data x 16 Sum of 16 These are slightly out of time To trigger board Used to make global trigger decision Described on next page

6. ADC data AND Global trigger Local trigger Trigger pipeline To global trigger board Board sum Readout initiator Raw pipeline data Data packer Temp buffer 1 Temp buffer 2 Temp buffer 3 Temp buffer 4 prio 1 prio 2 prio 3 prio 4 Channel hypervisor MUX Data (ch 1) (to board hypervisor) Board hypervisor Data (ch 1) Data (ch 2) Data (ch 16)...... Per-channel logic: Per-board logic (serves 16 channels): VME buffer for writing VME buffer for reading To VME crate

7. Local Triggering Subtract pedestalAverage over 4 cycles data Trigger We massage the data before applying a trigger threshold: Pedestal: 2 seconds idle 1 second collisions When idle: collect 16 clocks of noise, save the average (“pedestal”) Subtract pedestal from real data in the subsequent 1-sec interval Average over 4 cycles: Smoothes out small spikes

8. Local triggering: illustration Threshold Subtract pedestal Avg over 4 clocks Note: input signal has correct shape, but baseline and noise have been over-emphasized to better illustrate the algorithm.

9. Global Trigger comes 2 µs data pipeline 2 µs trigger pipeline Global trigger AND Coincidence between LOCAL and GLOBAL triggers Start readout of the data from the pipeline Start readout Local trigger

10. Data packer 2 µs data pipeline read out 32 clocks around trigger point Data packer: timing and IDs 01010110100 … timing ch/b id Data (29 words) 16 bit 0 32 words Timing: an integer 0 - 1.25M identifies an event within a 1/100 sec spill Id: channel ID (0-15) within a board, and board ID (1 to ~400) Data: remaining 29 words DFF trig offset + pulse offset (~19) Can we now save it to a computer? NOT YET!

11. 0010111000 … timing ch/b id Data (29 words) 16 bit 00 32 words Control bits

12. PC readout basics PC – VME master Buffer for WRITING Buffer for READING VME-addressable Readout board serves 16 channels. VME slave Saves ~100KhZ of data for the duration of spill cycle New data from any of 16 channels Can process 1 at a time Switches in 1/100 s

13. Intermediate buffers 32x16 bit packed dataprio1 32x16 bit packed dataprio2 32x16 bit packed dataprio3 32x16 bit packed dataprio4 2 µs data pipeline For *each* channel Uses priorities to choose next available buffer Channel hypervisor SUM reports channel priority to board hypervisor To VME slave buffer Buffer for Writing Chooses highest-priority buffer to read out Board hypervisor One per *board* VME/GLink Buffer for Reading Switch every 1/100 s

14. EARLIERLATER | …… 012313029 0 | Already saved! 16 | Readout! 2116 …… 012 Overlap 0 Trigger sampling time Data time No downtime, little overlap

15. 1/100 sec switching caveat PC – VME master Buffer for WRITING Buffer for READING VME-addressable New data from any of 16 channels Can process 1 at a time Every 1/100 sec write and read buffers switch roles. New spill is started, and timing stamp is reset. What if there are old intermediate buffers that are not read-out yet? Solution: delay the read-write switch until all those are saved.

16. Project status Implemented entire trigger/readout logic Tested portions, trying to get the whole system up Most work done in dedicated Altera simulation Programmed actual board (16 channels) Observed scope traces of the signals Altera Stratix II appears to meet timing requirements

17. Plans and issues Very technical stuff, hard to present Currently devising more visual tests Still unable to run the whole system Must be ready for FNAL test in December But I am confident I’ll get it to work!

18. SUPPLEMENTAL SLIDES Intermediate buffer data flow Detailed description

19. 1 0 0 0 Raw pipeline 1 2 1 0 0 3 3 2 1 0 6 0 2 1 0 3 0 7 6 0 13 1 7 6 0 Raw pipeline 14 (a) (d) (b) (e) (c) (f)

20. 1 0 0 0 Raw data pipeline 1 Suppose a trigger comes (i.e. coincidence between local and global). Choose 1 st buffer with zero priority. Start writing and increment prio to 1. address 0 address 31

21. 0 0 0 0 Raw data pipeline 0 While we are writing, suppose a board hypervisor selects this channel to read. We can start reading this buffer, even though not all 32 words are written yet! to VME RAM read out NOTE: even though prio1=0, the channel hypervisor knows not to write to this buffer!

22. 0 0 0 0 Raw data pipeline 0 Buffer 1 if completely read out. The channel is now empty.

23. 1 0 0 0 Raw data pipeline 1 A new trigger comes. It is routed to buffer 1, which is now empty. It’s priority is immediately increased. address 0 address 31

24. 2 1 0 0 Raw data pipeline 3 Another trigger comes when 1 st one is more than 16 clocks done. Start writing (initially with the same data) into next 0-prio buffer. Increment all non-zero priorities by 1 - i.e. those that are older Note: we require that 1 st buffer is >16 cycles. See next slide!

25. | …… 012313029 0 | Already saved! 15 2115 … … 012 Overlap Trigger sampling Second data readout (A) (B) First data readout

26. 132 1 16 132 Overlap 132 (A)(B) No trigger Can trigger 12 21 2 nd readout 1 st readout trigger readout trigger readout

27. 2 1 0 0 Raw data pipeline 3 Another trigger comes when the two are still writing. Since we are already saving everything, ignore it.

28. 3 2 1 0 Raw data pipeline 6 Another trigger comes after 1 st buffer is fully uploaded. Start its write (again, duplicating some of the data from buffer 2). Increase prio of old buffers, because we want to read them ASAP. Note: maximum priority value is 4!

29. 0 2 1 0 Raw data pipeline 3 Now, the channel has a high enough prio that the board hypervisor selects it. Then the channel hypervisor selects the highest-prio buffer *within* the channel. In our case, we read out buffer 1. It’s prio is immediately nulled to VME RAM

30. 1 3 2 0 Raw data pipeline 6 Suppose at this moment another trigger comes. The data is routed to buffer 1 – it’s already available! Its prio is increased. Note that we again increase the priorities of old buffers! to VME RAM new eventold event

31. 1 0 2 0 Raw data pipeline 6 After this read is finished, another read comes. Buffer 2 – having the highest priority – is selected. to VME RAM

32. 1 0 0 0 Raw data pipeline 6 After the two reads are finished, another read comes. Buffer 3 is selected. to VME RAM

33. 6 0 0 0 Raw data pipeline 6 Suppose at this moment a VME spill comes (every 1/100 s). We’d like to read out the large VME-addressable buffer, but we cannot: The 32-deep buffers contain data from a previous spill! Solution: promote up to two priorities to “critical”: prio+5. Don’t let VME read the data until all critical buffers are emptied! to VME RAM CRIT Reported to board hypervisor

34. 0 1 0 0 Raw data pipeline 1 After 3 rd buffer is read out, 1 st one is selected. VME will be able to read out when: none critical, none reading. If another trigger comes at this moment (belonging to a new spill), it will be saved as it normally would. No downtime! to VME RAM Last critical buffer!

35. 0 1 0 0 Raw data pipeline 1 When the last critical buffer is read-out, the VME addressable buffers switch. VME is informed that it can start the readout from the current buffer. Data from the new spill can now be saved to a fresh VME buffer! VME can RO!

36. VME-addressable pipe 8K x 32 bit VME-addressable pipe 8K x 32 bit Data from buffers reports “vme ready” Must be finished within 1/100 sec! VME readout reports max address Board hypervisor VME block MUX VME can RO!