1. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology 1 Implementing White Rabbit in your design Peter Jansweijer

2. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology Outline: 2 My background: KM3NeT / FPGA / Serial Communication White Rabbit introduction Technology / Implementations WR PTP Core in detail How to implement WR in your design

3. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology KM3NeT Artist impression & detection method 3 100m ~ 860m 640 strings 18 DOM/string 11.520 DOMs 357.120 PMTs Volume: ~5 km 3 11.520 DOMs in the deep sea at 3-5 km depth  Cherenkov

4. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology The very beginning… “Measuring propagation delay over a coded serial communication channel using FPGA.” 4 Presented at the VLVnT09, October 15, 2009 in Athens Elsevier Nuclear Instruments and Methods: http://dx.doi.org/10.1016/j.nima.2010.04.126 FPGA transceivers propagation delay study: http://www.nikhef.nl/pub/services/biblio/technicalreports/ETR2010-01.pdf 0 0001 = 10 0000 = 00 0011 = 3 RxRecClk BitSlide(4:0) 011101011000001010110111010110000010101101110101 0 Start/Stop delay 31

5. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology White Rabbit Introduction 5

6. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology Layer 1 Syntonization 6 Syntonization = “The adjustment of two electronic circuits or devices in terms of frequency” All network devices use the same physical layer clock. Clock is encoded in the Ethernet carrier and recovered by the receiver chip. Phase detection allows sub-ns delay measurement

7. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology Precision Time Protocol (IEEE 1588) 7 Frame-based synchronization protocol. Synchronizes local clock with master clock. Link delay evaluated by measuring and exchanging frames with tx/rx timestamps

8. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology Digital Dual Mixer Time Difference (DDMTD) 8

9. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology Using the DDMTD as phase detector and phase shifter 9

10. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology White Rabbit: an extension of Ethernet 10 Two separate services (enhancements to Ethernet) provided by WR: Synchronization: accuracy better than 1 ns precision (tens of ps sdev skew max) Deterministic, reliable and low-latency Control Data delivery

11. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology Standardization 11 Ongoing standardization effort to include WR into PTPv3 Extension to PTP (IEEE1588) – defined as PTP profile Compatible with “standard” PTP According to ISPCS Plug Fest results… “… White Rabbit is the most accurate PTP implementation in the world!”

12. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology 12 White Rabbit Switch http://www.ohwr.org/projects/white-rabbit/wiki/Switch Central element of WR network 18 1000BASE-BX10 ports Open design (H/W and S/W) Commercially available

13. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology 13 White Rabbit Node example 1: SPEC http://www.ohwr.org/projects/spec/wiki FMC-based Hardware Kit All carrier cards are equipped with a White Rabbit port. Mezzanines can use the accurate clock signal and “TAI” (synchronous sampling clock, trigger, time tag, …). Starting kit: http://www.ohwr.org/projects/wr-starting-kit/wiki http://www.ohwr.org/projects/wr-starting-kit/wiki

14. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology 14 Other White Rabbit Node examples: Central Logic Board (KM3NeT) CUTE-WR (LHAASO) CRIO-WR (CERN) SVEC (CERN) SPEXI (CERN)

15. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology T-Sense / Unique ID 15 White Rabbit PTP Core http://www.ohwr.org/projects/wr-cores/wiki/Wrpc_core

16. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology White Rabbit PTP Core in detail 16

17. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology T-Sense / Unique ID 17 PHY: Interface to the physical network, with deterministic phase relationship between gigabit clock and system clock White Rabbit PTP Core in detail

18. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology T-Sense / Unique ID 18 Endpoint: Ethernet MAC, with precision time stamping capabilities of in- and out- bound Ethernet frames White Rabbit PTP Core in detail

19. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology T-Sense / Unique ID 19 Redirector: Redirects all PTP Ethernet packets to the Mini-NIC Receives the payload of the PTP packets and stores payload data in memory “Normal” packets White Rabbit PTP Core in detail

20. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology T-Sense / Unique ID 20 Embedded Processor system (LatticeMico32 + Bus interface + Memory) running a PTP-daemon White Rabbit PTP Core in detail

21. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology T-Sense / Unique ID 21 Now the CPU has “knowledge” of time. It can adjust the local time and update the proper TAI time and control the Pulse Per Second (PPS) generation White Rabbit PTP Core in detail

22. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology T-Sense / Unique ID 22 White Rabbit PTP Core in detail The local oscillator that is used for TX is locked onto the RX receive clock. A PLL is made using Digital Dual Mixer Time Difference (DDMTD) technique and software on the LatticeMico32 (hence Soft-PLL) to tune the oscillator. DDMTD allows for a result of calculations of the round trip delay (Timing Synchro)

23. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology T-Sense / Unique ID 23 1-Wire is used to fetch a unique MAC address UART is used for debug only White Rabbit PTP Core in detail

24. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology How to implement WR in your design 24

25. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology 25 How to implement White Rabbit in your design http://www.ohwr.org/projects/white-rabbit/wiki/WRReferenceDesign PHYSFP DAC REF clock generator DMTD clock generator GTREFCLK CLK125M_PLLREF 125 MHz CLK20M_VCXO 20+ MHz DAC PLL CLK_DMTD VCXO SPI Unique-ID (MAC-addr) (optional) FPGA Your own stuff DATA (1 Gbps) Timing (TAI[ns],REFCLK)

26. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology White Rabbit & CLBv2 (KM3NeT) 26 Rx_mac2buf I2C Fifo 31 TDCs TDC 0 Management & Control Data Control Wishbone bus RxPacket Buffer 64KB IP/UDP Packet Buffer Stream Selector (IPMUX) Rx_buf2data RxPort 1 RxPort 2 RxPort_m Management & Config. Tx_pkt2macTx_data2buf TxPort 1 TxPort 2 TxPort_m Flags Rx Stream Select TxPacket Buffer 32KB Flags Tx Stream Select 31 PMTs UTC time & Clock (PPS, 125 MHz) Pause Frame ADC Management & Control Hydrophone Fifo TDC 30 Fifo Nano Beacon GPIO Debug LEDs I2C Debug RS232 Temp Compass Tilt Point to Point interconnection Xilinx Kintex-7 Start Time Slice UTC & Offset counter since Time Slice Start MEM S 2 nd CPU LM32 M M WB Crossbar (1x7) WB Crossbar (3x2) S M S M M S S M M M SS S UART S M M S S M M State Machine SPI S M Flash FPGA

27. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology 27 WR Users http://www.ohwr.org/projects/white-rabbit/wiki/WRUsers http://www.ohwr.org/projects/white-rabbit/wiki/WRUsers Picture taken @ 4th WR workshop (2011), GSI Darmstadt Many slides in this presentation orginate from the WR group: Thanks! White Rabbit Group: http://www.ohwr.org/projects/white-rabbit/wiki

28. 10 - 13 November 2014, Groningen/Dwingeloo, the Netherlands 3rd International VLBI Technology Workshop Peter Jansweijer Nikhef Amsterdam Electronics- Technology 28 Thank you