1. Mu3e Data Acquisition Ideas Dirk Wiedner July 2012 7/5/20121Dirk Wiedner Mu3e meeting Zurich

2. Outline Readout Requirements Occupancies Pixel Readout Scheme Data Link Scheme Summary 7/5/2012Dirk Wiedner Mu3e meeting Zurich2

3. Silicon Pixel Detector 7/5/2012Dirk Wiedner Mu3e meeting Zurich3 Inner double layer o 12 and 18 stripes of 1 x 12 cm Outer double layer o 24 and 28 stripes of 2 x 36 cm Re-curl layers o 24 and 28 stripes of 2 x 72 cm o Both sides (x2) 180 inner sensors 4680 outer sensors

4. Timing Detectors 7/5/2012Dirk Wiedner Mu3e meeting Zurich4 Scintillating fiber hodoscope o 250 μm fibers o 5 layers  O(7500) fibers Timing tiles o 1 x 1 cm tiles o O(7000) tiles DRS5 readout

5. Readout Requirements 7/5/2012Dirk Wiedner Mu3e meeting Zurich5 2.5 GHz muon decays 50 ns readout frames O(5000) pixel chips o 800 Mb/s readout links O(7500) fibers O(7000) timing tiles o DRS readout Online filtering

6. Maximum Occupancies 7/5/2012Dirk Wiedner Mu3e meeting Zurich6 All numbers per frame of 50 ns Vertex detector o 5 hits per sensor Central silicon tracker o 2 hits per sensor Recurl stations o 1 hit per sensor Fiber hodoscope o 0.24 hits per fiber Timing tiles o 0.14 hits per tile φ z Vertex occupancies

7. Average Occupancies 7/5/2012Dirk Wiedner Mu3e meeting Zurich7 All numbers per frame of 50 ns Vertex detector o 2 hits per sensor Central silicon tracker o 0.6 hits per sensor Recurl stations o 0.13 hit per sensor Fiber hodoscope o 0.16 hits per fiber Timing tiles o 0.09 hits per tile Tile occupancies

8. Pixel Readout Scheme 7/5/2012Dirk Wiedner Mu3e meeting Zurich8

9. Pixel Readout Scheme 7/5/2012Dirk Wiedner Mu3e meeting Zurich9 Pixel logic o Pixel address (8 bit) o Fine time (4 bit) o 50 ns frames Column logic o Pixel data o Column address o Coarse time o Error bits Frame logic o Super Frame o Contains 16 x 50 ns readout frames o + Sensor header Readout buffer Serializer and fast link(s) Pixel address Pixel Logic Column Logic Frame logic Readout buffer Serializer Time stamp Time stamp Coarse time Coarse time Error and overflow

10. Pixel Readout Scheme 7/5/2012Dirk Wiedner Mu3e meeting Zurich10 Pixel logic o Pixel address (8 bit) o Fine time (4 bit) o 50 ns frames Column logic o Pixel data o Column address o Coarse time o Error bits Frame logic o Super Frame o Contains 16 x 50 ns readout frames o + Sensor header Readout buffer Serializer and fast link(s) Pixel address Pixel Logic Column Logic Frame logic Readout buffer Serializer Frame logic Readout buffer Serializer 8 bit Time stamp 4 bit 12 bit Coarse time Coarse time Error and overflow

11. Pixel Readout Scheme 7/5/2012Dirk Wiedner Mu3e meeting Zurich11 Pixel logic o Pixel address (8 bit) o Fine time (4 bit) o 50 ns frames Column logic o Pixel data o Column address o Coarse time o Error bits Frame logic o Super Frame o Contains 16 x 50 ns readout frames o + Sensor header Readout buffer Serializer and fast link(s) Pixel address Pixel Logic Column Logic Frame logic Readout buffer Serializer Frame logic Readout buffer Serializer 8 bit Time stamp 4 bit 12 bit Coarse time Error and overflow 16 bit 4 bit 32 bit

12. Pixel Readout Scheme 7/5/2012Dirk Wiedner Mu3e meeting Zurich12 Pixel logic o Pixel address (8 bit) o Fine time (4 bit) o 50 ns frames Column logic o Pixel data o Column address o Coarse time o Error bits Frame logic o Super Frame o Contains 16 x 50 ns readout frames o + Sensor header Readout buffer Serializer and fast link(s) Pixel address Pixel Logic Column Logic Frame logic Readout buffer Serializer 8 bit Time stamp 4 bit 12 bit Coarse time Error and overflow 32 bit 4 x 1 bit @ 800 Mb/s 16 bit 4 bit

13. Super Frame Header 7/5/2012Dirk Wiedner Mu3e meeting Zurich13 Coarse time o 16 bit Sensor address o 16 bit Number of hits o 10 bits Sub-frame overflow o 16 bit o If more than 63 hits per frame Error propagation o 6 bits 64 bits in total Coarse timeOverflow Sensor address # HitsError 64 bit

14. Super Frame Header 7/5/2012Dirk Wiedner Mu3e meeting Zurich14 Coarse time o 16 bit Sensor address o 16 bit Number of hits o 10 bits Sub-frame overflow o 16 bit o If more than 63 hits per frame Error propagation o 6 bits 64 bits in total Coarse timeOverflow Sensor address # HitsError 64 bit

15. Super Frame Data Dirk Wiedner Mu3e meeting Zurich15 Data from 16 frames o 0.8 μs interval Fine time 4 bit Pixel address 16 bit o 8 rows o 8 columns Maximum of 1008 hits o Maximum of 20160 bits Header Fine timerowcolumn… Fine timerowcolumn… ………… Fine timerowcolumn… 16 x 20 bit 7/5/2012

16. Number of Links Vertex 7/5/2012Dirk Wiedner Mu3e meeting Zurich16 Hottest region (average) o 5 hits per sensor / frame  Super frame size: o 80 hits per super frame  1664 bits average  2.6 Gb/s required i.e 4 x 800 Mb/s links per sensor

17. Number of Links Central Silicon Tracker 7/5/2012Dirk Wiedner Mu3e meeting Zurich17 Hottest region (average) o 2 hits per sensor / frame  Super frame size: o 32 hits per super frame  704 bits average  1.1 Gb/s required i.e 2 x 800 Mb/s links per sensor

18. Number of Links Recurl Stations 7/5/2012Dirk Wiedner Mu3e meeting Zurich18 Hottest region (average) o <0.25 hits per sensor / frame  Super frame size: o 4 hits per super frame  144 bits average  0.225 Gb/s required i.e 1 x 400 Mb/s links per sensor

19. Data Link Scheme 7/5/2012Dirk Wiedner Mu3e meeting Zurich19

20. Link Overview 7/5/2012Dirk Wiedner Mu3e meeting Zurich20 Front end links o Pixel sensor to on-detector FPGA 400 – 800 Mbit/s LVDS o Timing readout See Stefan Ritt Optical links from detector o On-detector FPGAs o … to off-detector readout boards o 5 Gbit/s Optical links in counting room o Off-detector read out boards o …to PC Farm

21. Link Overview 7/5/2012Dirk Wiedner Mu3e meeting Zurich21 Front end links o Pixel sensor to on-detector FPGA 400 – 800 Mbit/s LVDS o Timing readout See Stefan Ritt Optical links from detector o On-detector FPGAs o … to off-detector readout boards o 5 Gbit/s Optical links in counting room o Off-detector read out boards o …to PC Farm Pixel Sensor Silicon FPGAs x86 Readout board x12 Readout board x12 PC x48 PC x48

22. Link Overview 7/5/2012Dirk Wiedner Mu3e meeting Zurich22 Front end links o Pixel sensor to on-detector FPGA 400 – 800 Mbit/s LVDS o Timing readout See Stefan Ritt Optical links from detector o On-detector FPGAs o … to off-detector readout boards o 5 Gbit/s Optical links in counting room o Off-detector read out boards o …to PC Farm Pixel Sensor Fiber DRS Tile DRS Pixel Sensor Fiber DRS Tile DRS Pixel Sensor Fiber DRS Tile DRS Pixel Sensor Fiber DRS Tile DRS Silicon FPGAs x86 Fiber FPGAs x48 Tile FPGAs x48 Readout board x16 Readout board x16 Readout board x8 Readout board x8 Readout board x8 Readout board x8 x6336x7540?x6912? PC x48 PC x48

23. Link Overview 7/5/2012Dirk Wiedner Mu3e meeting Zurich23 Front end links o Pixel sensor to on-detector FPGA 400 – 800 Mbit/s LVDS o Timing readout See Stefan Ritt Optical links from detector o On-detector FPGAs o … to off-detector readout boards o 5 Gbit/s Optical links in counting room o Off-detector read out boards o …to PC Farm Pixel Sensor Fiber DRS Tile DRS Tile DRS Pixel Sensor Fiber DRS Tile DRS Tile DRS Pixel Sensor Fiber DRS Tile DRS Tile DRS Pixel Sensor Fiber DRS Tile DRS Tile DRS Silicon FPGAs x86 Fiber FPGAs x48 Fiber FPGAs x48 Tile FPGAs x48 Tile FPGAs x48 Readout board x16 Readout board x16 Readout board x8 Readout board x8 Readout board x8 Readout board x8 x6336 PC x48 PC x48 x376

24. Link Overview 7/5/2012Dirk Wiedner Mu3e meeting Zurich24 Front end links o Pixel sensor to on-detector FPGA 400 – 800 Mbit/s LVDS o Timing readout See Stefan Ritt Optical links from detector o On-detector FPGAs o … to off-detector readout boards o 5 Gbit/s Optical links in counting room o Off-detector read out boards o …to PC Farm Pixel Sensor Fiber DRS Tile DRS Pixel Sensor Fiber DRS Tile DRS Pixel Sensor Fiber DRS Tile DRS Pixel Sensor Fiber DRS Tile DRS Silicon FPGAs x86 Fiber FPGAs x48 Tile FPGAs x48 Readout board x16 Readout board x16 Readout board x8 Readout board x8 Readout board x8 Readout board x8 x6336x7540?x6912? PC x48 PC x48 x376x192

25. Link Overview 7/5/2012Dirk Wiedner Mu3e meeting Zurich25 Front end links o Pixel sensor to on-detector FPGA 400 – 800 Mbit/s LVDS o Timing readout See Stefan Ritt Optical links from detector o On-detector FPGAs o … to off-detector readout boards o 5 Gbit/s Optical links in counting room o Off-detector read out boards o …to PC Farm Pixel Sensor Fiber DRS Tile DRS Pixel Sensor Fiber DRS Tile DRS Pixel Sensor Fiber DRS Tile DRS Pixel Sensor Fiber DRS Tile DRS Silicon FPGAs x86 Fiber FPGAs x48 Tile FPGAs x48 Readout board x16 Readout board x16 Readout board x8 Readout board x8 Readout board x8 Readout board x8 x6336x7540?x6912? PC x48 x376x192 x96

26. Pixel Sensor Links 7/5/2012Dirk Wiedner Mu3e meeting Zurich26 Vertex Sensor chips o 180 chips o 4 LVDS links o 800 Mbit/s per link Central Silicon Tracker o 936 chips o 2 LVDS links o 800 Mbit/s Recurl stations o 3744 chips o 1 LVDS link o 400 Mbit/s

27. Pixel Sensor Links 7/5/2012Dirk Wiedner Mu3e meeting Zurich27 Vertex Sensor chips o 180 chips o 4 LVDS links o 800 Mbit/s per link Central Silicon Tracker o 936 chips o 2 LVDS links o 800 Mbit/s Recurl stations o 3744 chips o 1 LVDS link o 400 Mbit/s Pixel Sensor 800 Mbits/s Front end FPGA

28. Front End FPGAs 7/5/2012Dirk Wiedner Mu3e meeting Zurich28 FPGAs on detector o 86 (+96) pieces Receive sensor data o 108 LVDS inputs 5 Gbit/s outputs o 8 optical links o … to counting house Switching between readout boards A-D Optical transceiver FE board

29. Front End FPGAs 7/5/2012Dirk Wiedner Mu3e meeting Zurich29 FPGAs on detector o 86 (+96) pieces Receive sensor data o 108 LVDS inputs 5 Gbit/s outputs o 8 optical links o … to counting house Switching between readout boards A-D Front end FPGA 800 Mbit/s LVDS in x 108 5 Gbit/s optical Readout board A Readout board A Pixel Sensor Readout board B Readout board B Readout board C Readout board C Readout board D Readout board D

30. Readout Board 7/5/2012Dirk Wiedner Mu3e meeting Zurich30 FPGAs off detector o 32 pieces 5 Gbit/s optical input o 16-28 inputs o … from on detector FPGAs 10 Gbit/s optical output o 12 outputs o … to PCs Switching network o A-D sub-nets o One output per PC Readout board Front end FPGA 5 Gbit/s Optical x28 PC x12 10 Gbit/s Optical x12 Readout board Front end FPGA Front end FPGA 5 Gbit/s Optical x28 PC x12 PC x12 10 Gbit/s Optical x24 AB

31. GPU-PC 7/5/2012Dirk Wiedner Mu3e meeting Zurich31 PC with GPU o 48 pieces 10 Gbit/s Fiber input o 8 inputs o … from readout boards Switching network o One input per readout board o A-D subnets Data filtering o Timing trigger on FPGA o Data to tape < 100 MB/s FPGA PCIe board GPU computer Optical mezzanine connectors

32. GPU-PC 7/5/2012Dirk Wiedner Mu3e meeting Zurich32 PC with GPU o 48 pieces 10 Gbit/s Fiber input o 8 inputs o … from readout boards Switching network o One input per readout board o A-D subnets Data filtering o Timing Filter on FPGA o Data to tape < 100 MB/s Readout board PC 10 Gbit/s Optical x8 Readout board PC 10 Gbit/s Optical x8 AB

33. Timing Filter 7/5/2012Dirk Wiedner Mu3e meeting Zurich33 Entire event on PCIe FPGA Tile and Fiber data o Easy to match o Look for three hits Reject data without three hits o … inside time interval 1 3 2

34. Summary Occupancies simulated (Nik) Serial data from pixel sensors 800 Mbit/s LVDS links On detector FPGA boards Optical data transfer Read out board in counting room Switched optical network GPU PCs with optical inputs 7/5/2012Dirk Wiedner Mu3e meeting Zurich34