1. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO KM3NeT CLBv2 1

2. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO General Thanks Diego for the “FPGA Selection Considerations” document. 1.FPGA 2.CPU 3.Softcore 4.Reliability 5.Cost 6.Power Lets try to address these items and update the document 2

3. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO CLB should support 31 TDC’s 1 ns resolution “Knowledge” of absolute time (1 ns resolution) Data pushed from PMTs to Shore Station I2C: PMT-HV, Threshold, Compass, Tilt Other IO: Temp, Nano beacon, Aucoustics Firmware must be reconfigurable Low Power Low Cost Part of a scalable system (with respect to the the complete detector) Highly reliable 3

4. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO Planning October 1, 2012 FPGA choice ◦ First we need to know what system we are going to build! ◦ This needs study… And time… December 1, 2012 TDC and Timing Verified in FPGA ◦ Planning absolutely not feasible! ◦ Again, First we need to know what system we are going to build! Q1 2014 functional CLB’s operational? ◦ Far from realistic! 4

5. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO How to divide into system components Shore Station GPS DOM CLB CLB is not a separate entity! Take lower OSI layers of CLB and Shore Station both into account! Optical Network 5

6. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO Ethernet based Current implementation “DOM in Antares” Switch DOM SFP Buffer SFP Broadcast Optical Network Timing Start Rx Tx Stop1  : DDMTD Reference Clock Stop2 Stop3 Stop4 MAC does not tolerate discontinuities in transmission Timing calibration corrupts Ethernet Packets (=> need TCP/IP) Shore Station interface 6 Rx  : DDMTD Rx  : DDMTD Rx  : DDMTD Note: Current implementation supports one DOM only! (broadcast not yet implemented!)

7. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO System (= CLB + Shore Station) choices: Ethernet Based (Like “DOM in Antares”) ProsCons Extremely flexiblePower Board space Cost Timing (Clock/Command) implemented on Physical Layer => Timing corrupts Ethernet packets If this is a problem (probably not in the case of data, but yes in the case of Slow Control) then need TCP/IP needs understanding of: DDMTD Extremely flexible… … but way too much for what is needed … flexibility adds complexity that may degrade system performance 7

8. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO System (= CLB + Shore Station) choices: Downscaled Ethernet Based ProsCons No Operating system! Less power Less board space (no external large memory) Lower cost Only a small CPU needed Small CPU adds complexity UDP/IP in firmware (ultra-light) TCP/IP, ARP and PING in software Timing (Clock/Command) implemented on Physical Layer => Timing corrupts Ethernet packets If this is a problem (probably not in the case of data, but yes in the case of Slow Control) then need TCP/IP needs understanding of: DDMTD 8

9. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO White Rabbit (PTP, IEEE1588) Switch DOM SFP Buffer SFP Broadcast Optical Network Start Tx t 4 Stop1 Reference Clock PTP Time Stamp t 1 Time Stamp t 4 t 4 Stop2 Time Stamp t 4 t 4 Stop3 Time Stamp t 4 t 4 Stop4 Time Stamp t 4 Timing implemented on transport Layer. PTP Ethernet Packets No corruption of packets Shore Station interface 9 Rx  : DDMTD Rx  : DDMTD Rx  : DDMTD Rx  : DDMTD

10. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO PTP, IEEE1588 10 t offset  Ref Clk Cnt Master SFP Slave Rx Tx SFP Tx Rx  Cnt t1t1 t 2 (= Cnt + SlaveBitSlide) Sync message Follow_Up Message (t 1 ) t 1 t 2 t 3 t 1 t 2 Delay_Req Message Delay_Response Message (t 4 ) t 4 t 3 t 1 t 2 t1t1 t2t2 Master Clock Time Slave Clock Time t3t3 t4t4 t 4 (= Cnt + MasterBitSlide +  ) =0 Data (Packets) Clock Time Stamp t2t2 t3t3 t4t4 t4t4 t1t1 t1t1 1.t 2 –t 1 =offset + MSdelay 2.t 4 –t 3 =-offset + SMdelay 3.MSdelay=SMdelay

11. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO System (= CLB + Shore Station) choices: Ethernet + PTP (White Rabbit) ProsCons Based on standardsAdd more complexity There is a White Rabbit PTP Core available http://www.ohwr.org/projects/wr- cores/wiki/Wrpc_core (~50 man years firmware & software) White Rabbit PTP Core is a “black box”, however one needs understanding of: DDMTD LM32 WR-MAC Mini-NIC Pipelined Wishbone interface Timing and Ethernet combined Flexible and scalable while this can still be implemented low power and low cost This is the way to go to build KM3NeT!But absolutely not implemented in a CLBv2 by Q1 2014! 11

12. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO Home brew Switch DOM SFP Buffer SFP Broadcast Optical Network Shore Station interface Timing Start Tx Reference Clock Timing is hidden in “special” 8B/10B Characters and home made “simple” MAC can be made resistant to them. 12 Rx Stop1  : DDMTD Stop2 Stop3 Stop4 Rx  : DDMTD Rx  : DDMTD Rx  : DDMTD

13. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO System (= CLB + Shore Station) choices: Home brew ProsCons “Easy” (fits the impossible planning better) Need private protocol Pay attention to Slow Control which needs to be guaranteed data delivery. watch out for scalability! Configure KM3NeT “by hand” (ARP and easy packet routing only after reception in Shore Station interface) Add timing without disturbing data transfers Either: need study on MAC that is “timing marker” insensitive. OR buffer management to idle the data stream when a timing marker must be propagated. Note that a MAC study might solve the issue of the limited number of TEMACs per device (Shore Station) A test PC needs a receiver card. Note that the Shore station needs special hardware anyhow due to broadcast. Small CPU comes in very handy (= a must for various Slow Control) Do I overlook something?I may overlook something very trivial! 13

14. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO What we need anyhow… Knowledge about DDMTD => look and learn code (this is not an easy task) ◦ from Yassir Mouden Saclay ◦ White Rabbit (http://www.ohwr.org/projects/white- rabbit/wiki)http://www.ohwr.org/projects/white- rabbit/wiki CLB Timing Loopback External PLL ◦ Study loop filter optimization TDC (in Xilinx?; ISERDES) ◦ port Albert Zwart / Yassir Mouden design 14

15. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO What we need anyhow-2… MAC ◦ Amount of Xilinx TEMACs per device is limited (Issue for Shore Station) ◦ Open Cores MAC? ◦ Most simple implementation can be made ourselves (which creates the possibility to add timing on the data link layer without corrupting (raw) Ethernet packets.  Simplifications:  no real need for Host Interface  no real need for MDIO interface (MAC PHY) 15

16. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO What we need anyhow-3… Implement interfaces: ◦ I2C and other IO (ADCs for Acoustics) Implement re-configurability Software needed! (nowhere on the current planning…) ◦ Embedded (in CLB) ◦ Test software on a “Test Shore Station” PC 16

17. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO FPGA choice All of us have Xilinx experience so this seems a natural choice (although the above statement is still valid). Still… design should preferably be Vendor independent as much as possible. SerDes > 1.25 Gbps Timing: SerDes must be BitSlide trick capable TDC: IO capable of handling 1 ns resolution Enough Resources ◦ memory! (Especially important when on chip CPU is used). Power ◦ Unused resources add up in “leakage” current (which is a significant amount compared to resources used). 17

18. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO CPU Needed at all? ◦ I think it makes life and testability much easier and it is very much worth the investment (even with the home brew solution) No operating system ◦ Needs too much memory and resources while there is no (or small) advantage. Software needs to be revised anyway… External or internal in FPGA ◦ Board space / Reliability ◦ Price ◦ Power ◦ Experience so far… If internal to the FPGA then Vendor independent ◦ LEON, LM32 (Note: MicroBlaze, NIOS, Zynq and Virtex5-PPC are vendor dependent!) ◦ Select FPGA with enough:  1) Memory (RAM Blocks)  2) Logic Resources 18

19. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO Conclusions FPGA choice… ◦ Choose between Vendor => I guess we choose Xilinx ◦ Avoid “Vendor lock in”! ◦ Family (Virtex-6, Spartan-6, Kintex-7, Artix-7) will all do the job => buy a Kintex-7 evaluation kit and find out if we can downscale to Artix-7 eventually More important, decide between: ◦ “Ethernet based” ◦ “White Rabbit” ◦ “Home Brew” Please don’t use an operating system! ◦ Way to much for the job and it consumes lots of power and money Lets start as soon as possible to try and port/implement various code (see “What we need anyhow”) on the evaluation kit. Planning is tight, lets see how far we will come! 19

20. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO Questions? Comments? 20

21. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO Backup Slides 21

22. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO Atrix, Kintex, Virtex 22

23. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO 23

24. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO Timing: Preliminary BitSlide 1) capable study 24 Artix-7Kintex-7Virtex-7 Transceiver Count163296 Transceiver typeGTPGTX GTH GTZ 1) As described in http://www.nikhef.nl/pub/services/biblio/technicalreports/ETR2010-01.pdf Speed [Gb/s] BitSlide 1) capable Described in Xilinx documentation: GTP6.6YesGTP Transceivers User Guide UG482 (v1.1.1) “Manual Alignment” page 123 and Table 4-21 GTX12.5YesGTX/GTH Transceivers User Guide UG476 (v1.5) “Manual Alignment” page 178 and Table 4-25 GTH13.1 GTZ28.05?No documentation (yet?) Preliminary means: Described in Xilinx documentation but not proven in hardware!

25. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO Resources For comparison SPIDR (virtex-6) 1 Gbps, 4-UDP port, 1 CPU port IP engine (jumbo packet) ◦ See https://wiki.nikhef.nl/detector/pub/Main/SpiDr/UDP_PacketBufferOverview.pdfhttps://wiki.nikhef.nl/detector/pub/Main/SpiDr/UDP_PacketBufferOverview.pdf ◦ Tx 32 KB buffer (3 jumbo’s) ◦ Rx 64 KB buffer (7 jumbo’s) LEON3 CPU (@ 62,5 MHz) ◦ 64K on chip ROM, 128K on chip RAM ◦ JTAG Debug link ◦ UART ◦ Timer ◦ Interrupt ◦ I2C ◦ GPIO (interfacing to the rest of the SPIDR design (i.e. registers/settings etc.) Hard TEMAC (@125 MHz) Resource count Includes 4 Front-End interfaces to Medipix-3 chips! Resources (ML605; Virtex-6 xc6vlx240t, package ff1156, speed -1) ◦ Number used as Flip Flops: 7,828 ◦ Number of Slice LUTs: 10,099 out of 150,720 6% ◦ Number of RAMB36E1/FIFO36E1s: 44 out of 416 10% ◦ Number of RAMB18E1/FIFO18E1s: 85 out of 832 10% ◦ Number used as BUFGs: 4 ◦ Number of GTXE1s (= SerDes): 1 out of 20 5% ◦ Number of TEMAC_SINGLEs (= Hard MAC): 1 out of 4 25% Power (Xilinx XPower analyzer) see next slide… 25

26. Peter Jansweijer Nikhef Amsterdam Electronics- Technology September 14, 2012KM3NeT, CLBv2 Meeting via EVO Power For comparison SPIDR (virtex-6) Power (Xilinx XPower analyzer) 4,83 Watt 26