1. Physics 335 project update Anton Kapliy April 2007

2. Experiment - JPARC Process: FCNC sd → dd Final state: Two photons Major BG: Four photons, two missed Critical to correctly detect all photons that hit CsI, and nail down the timing

3. PMT (1 channel) 14-bit DIGI 125 MHz FPGA Trigger Data packing Pipelining Interface to PC Amp/shaper VME, GLink Sum of 16 Trigger board … x16 Global trigger My work … (x16)

4. Why need a trigger? >300 RO channels, digitized at 125 MhZ 300*(14 bits)*(125,000,000) = 60 TB/sec No computer can keep up! BUT: mostly junk Physics simulations predict event rate <100KHz for entire detector! (14 bit)*100k = 2 GB/sec Can save and apply HLT on PC Note: in reality, the rates must be multiplied by 30

5. Data gets into FPGA 2 µs data pipeline data Local trig (threshold) yes/no 2 µs trigger pipeline Data waits here for global trigger decision data x 16 Sum of 16 These are slightly out of time To trigger board Used to make global trigger decision Described on next page

6. Local Triggering Subtract pedestalAverage over 4 cycles data Trigger We massage the data before applying a trigger threshold: Pedestal: 2 seconds idle 1 second collisions When idle: collect 16 clocks of noise, save the average (“pedestal”) Subtract pedestal from real data in the subsequent 1-sec interval Average over 4 cycles: Smoothes out small spikes

7. Local triggering: illustration Threshold Subtract pedestal Avg over 4 clocks Note: input signal has correct shape, but baseline and noise have been over-emphasized to better illustrate the algorithm.

8. Global Trigger comes 2 µs data pipeline 2 µs trigger pipeline Global trigger AND Coincidence between LOCAL and GLOBAL triggers Start readout of the data from the pipeline Start readout Local trigger

9. Data packer 2 µs data pipeline read out 32 clocks around trigger point Data packer: timing and IDs 010101110100 … timing ch/b id Data (29 words) 16 bit 0 32 words Timing: an integer 0 - 1.25M identifies an event within a 1/100 sec spill Id: channel ID (0-15) within a board, and board ID (1 to ~400) Data: remaining 29 words DFF trig offset + pulse offset (~19) Can we now save it to a computer? NOT YET!

10. PC readout basics PC – VME master Buffer for WRITING Buffer for READING VME-addressable Readout board serves 16 channels. VME slave Saves ~100KhZ of data for the duration of spill cycle New data from any of 16 channels Can process 1 at a time Switches in 1/100 s

11. Intermediate buffers 32x16 bit packed dataprio1 32x16 bit packed dataprio2 32x16 bit packed dataprio3 32x16 bit packed dataprio4 2 µs data pipeline For *each* channel Uses priorities to choose next available buffer Channel hypervisor SUM reports channel priority to board hypervisor To VME slave buffer Buffer for Writing Chooses highest-priority buffer to read out Board hypervisor One per *board* VME/GLink Buffer for Reading Switch every 1/100 s

12. EARLIERLATER | …… 012313029 0 | Already saved! 16 | Readout! 2116 …… 012 Overlap 0 Trigger sampling time Data time No downtime, little overlap

13. 1/100 sec switching caveat PC – VME master Buffer for WRITING Buffer for READING VME-addressable New data from any of 16 channels Can process 1 at a time Every 1/100 sec write and read buffers switch roles. New spill is started, and timing stamp is reset. What if there are old intermediate buffers that are not read-out yet? Solution: delay the read-write switch until all those are saved.

14. Project status Implemented entire trigger/readout logic Tested portions, trying to get the whole system up Most work done in dedicated Altera simulation Programmed actual board (16 channels) Observed scope traces of the signals Altera Stratix II appears to meet timing requirements

15. Plans and issues Very technical stuff, hard to present Currently devising more visual tests Still unable to run the whole system Must be ready for FNAL test in December But I am confident I’ll get it to work!

16. SUPPLEMENTAL SLIDES Intermediate buffer data flow Detailed description

17. 1 0 0 0 Raw data pipeline 1 Suppose a trigger comes (i.e. coincidence between local and global). Choose 1 st buffer with zero priority. Start writing and increment prio to 1. address 0 address 31

18. 0 0 0 0 Raw data pipeline 0 While we are writing, suppose a board hypervisor selects this channel to read. We can start reading this buffer, even though not all 32 words are written yet! to VME RAM read out NOTE: even though prio1=0, the channel hypervisor knows not to write to this buffer!

19. 0 0 0 0 Raw data pipeline 0 Buffer 1 if completely read out. The channel is now empty.

20. 1 0 0 0 Raw data pipeline 1 A new trigger comes. It is routed to buffer 1, which is now empty. It’s priority is immediately increased. address 0 address 31

21. 2 1 0 0 Raw data pipeline 3 Another trigger comes when 1 st one is more than 16 clocks done. Start writing (initially with the same data) into next 0-prio buffer. Increment all non-zero priorities by 1 - i.e. those that are older Note: we require that 1 st buffer is >16 cycles. See next slide!

22. EARLIERLATER | …… 012313029 0 | Already saved! 16 | Readout! 2116 …… 012 Overlap 0 Trigger sampling time Data time

23. 2 1 0 0 Raw data pipeline 3 Another trigger comes when the two are still writing. Since we are already saving everything, ignore it.

24. 3 2 1 0 Raw data pipeline 6 Another trigger comes after 1 st buffer is fully uploaded. Start its write (again, duplicating some of the data from buffer 2). Increase prio of old buffers, because we want to read them ASAP. Note: maximum priority value is 4!

25. 0 2 1 0 Raw data pipeline 3 Now, the channel has a high enough prio that the board hypervisor selects it. Then the channel hypervisor selects the highest-prio buffer *within* the channel. In our case, we read out buffer 1. It’s prio is immediately nulled to VME RAM

26. 1 3 2 0 Raw data pipeline 6 Suppose at this moment another trigger comes. The data is routed to buffer 1 – it’s already available! Its prio is increased. Note that we again increase the priorities of old buffers! to VME RAM new eventold event

27. 1 0 2 0 Raw data pipeline 6 After this read is finished, another read comes. Buffer 2 – having the highest priority – is selected. to VME RAM

28. 1 0 0 0 Raw data pipeline 6 After the two reads are finished, another read comes. Buffer 3 is selected. to VME RAM

29. 6 0 0 0 Raw data pipeline 6 Suppose at this moment a VME spill comes (every 1/100 s). We’d like to read out the large VME-addressable buffer, but we cannot: The 32-deep buffers contain data from a previous spill! Solution: promote up to two priorities to “critical”: prio+5. Don’t let VME read the data until all critical buffers are emptied! to VME RAM CRIT Reported to board hypervisor

30. 0 1 0 0 Raw data pipeline 1 After 3 rd buffer is read out, 1 st one is selected. VME will be able to read out when: none critical, none reading. If another trigger comes at this moment (belonging to a new spill), it will be saved as it normally would. No downtime! to VME RAM Last critical buffer!

31. 0 1 0 0 Raw data pipeline 1 When the last critical buffer is read-out, the VME addressable buffers switch. VME is informed that it can start the readout from the current buffer. Data from the new spill can now be saved to a fresh VME buffer! VME can RO!

32. VME-addressable pipe 8K x 32 bit VME-addressable pipe 8K x 32 bit Data from buffers reports “vme ready” Must be finished within 1/100 sec! VME readout reports max address Board hypervisor VME block MUX VME can RO!