1. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 1 Measuring time offset over a bidirectional data link using FPGAs P.P.M. Jansweijer, H.Z. Peek

2. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting KM3NeT 2 OM Why measure time offset? Shore Station GPS OM 1 km OM Distributed: 1 cubic kilometer Synchronize system timing High precision: ~ 1 ns = “Measurement and control application”

3. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 3 Measurement and control application example - II (Super) LHC P4 P5P5 P8 P2CCR TTC backbone TTC off-detector TTC on-detector From a presentation given at the ATLAS Upgrade "ROD" Workshop Sophie Baron – CERN June 18, 2009 Synchronize system timing High precision: aim < 100 ps Distributed: LHC diameter 8,6 Km

4. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 4 measure time offset? How measure time offset? Clock & Data coded into one stream DC-Balance Special code-groups / Word Alignment 1 2 3 Could we use existing serial communication channels to measure propagation delay? Serial Communication Coding Properties:

5. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 5 1 3 Regenerate clock at the receiver Using a barrel shifter for word alignment delay known with bit clock resolution 8B/10B coded channel example Bit Clk Data 0 K28.5 0111110101 D16.2 0010001011 K23.7 110101000x Dx.Y xxxxxyyyy00111110 IDLETiming marker Details :VLVnT09, October 15, 2009 in Athens Presentation:http://www.nestor.noa.gr/vlvnt09/pres/Jansweijer_MeauringPropagationDelay.ppthttp://www.nestor.noa.gr/vlvnt09/pres/Jansweijer_MeauringPropagationDelay.ppt Paper:http://dx.doi.org/10.1016/j.nima.2010.04.126http://dx.doi.org/10.1016/j.nima.2010.04.126 K28.5 RxRec Clk IDLE

6. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 6 Start Test-Bed Transmitter Tx FPGA SerDes PCI Expressx1 Evaluation Board Rx FPGA SerDes ML507 Board LEDs Stop 100 Km fiber 156.25 MHz 156.25 MHz Test-Bed Receiver Measure propagation delay using FPGA SerDes Word Alignment information one direction from Transmitter to Receiver @ 3.125 Gbps History of WHAT has been done First test setup

7. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 7 Rx FPGA (Stop) Tx FPGA (Start) Start Stop 50 Km fiber 50 Km fiber Test-bed Receiver Test-bed Transmitter First test setup

8. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 8 First test setup conclusions It is feasible to measure propagation delay over an 8B/10B coded link over 100 Km of fibre. A 3.125 Gbps serial link provides a resolution of 320 ps. This can be implemented in FPGA Presented at the VLVnT09, October 15, 2009 in Athens Presentation: http://www.nestor.noa.gr/vlvnt09/pres/Jansweijer_MeauringPropagationDelay.ppthttp://www.nestor.noa.gr/vlvnt09/pres/Jansweijer_MeauringPropagationDelay.ppt Paper:http://dx.doi.org/10.1016/j.nima.2010.04.126http://dx.doi.org/10.1016/j.nima.2010.04.126

9. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 9 Master Tx Rx SFP Slave Rx Tx SFP RxUsrClk TxUsrClk RxUsrClk Start Stop History of WHAT has been done Second test setup Measure time offset between a master and slave Bidirectional Loopback the recovered clock @ 1.25 Gbps conform standard IEEE 802.3 (1000BASE-X, Gigabit Ethernet)

10. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 10 Second test setup MasterSlave Start Stop 10 Km fiber Stop Clock Loopback (DPLL) VCXO DAC

11. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 11  Reference Clk Fine time Count Coarse time Start Stop Master SFP Slave Rx Tx SFP RxUsrClk TxUsrClk RxUsrClk 1.# of system clocks 2.# bit clocks (i.e. barrel shifts) 3.Phase between Master node Tx and Rx clock t offset Tx Rx Second test setup conclusions It is feasible to measure time offset between a master and a slave node Time offset is determined by: For details please see Technical Report “ETR2010-01”: http://www.nikhef.nl/pub/services/biblio/technicalreports/ETR2010-01.pdf 1.25 Gpbs 10.. 100 ps 20 x #bit clocks 800 ps

12. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 12 The ultimate goal: End-Node Timing aware Ethernet switches and End-Nodes i.e. normal switches / nodes with some extra hardware to facilitate synchronous Ethernet and Precision Timing Protocol (PTP) Primary Reference Clock (GPS) CPU Farm

13. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 13 White Rabbit Project at CERN Open Hardware http://www.ohwr.org/projects “White Rabbit is a fully deterministic Ethernet-based field bus for general purpose data transfer and synchronization. The aim is to be able to synchronize ~1000 nodes with sub-ns accuracy over fiber and copper lengths of up to 10 km. The key technologies used are Synchronous Ethernet and PTP (IEEE 1588).” Share knowledge on switches, Synchronous Ethernet and PTP

14. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 14 The perfect symbiosis! Next goal: create a “White Rabbit IP-Core” Software Switches Timing ? Top Down Bottom up Software? Switches? Timing “We:” White Rabbit: Cooperation

15. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 15 White Rabbit IP-Core things to be done… Second test setup hardware facilitates fine time measurement, however it does not include actual time stamping and the software for PTP (IEEE 1588). Time stamping functionality must be added. A media access control (MAC) service interface must be added and PTP time stamps must be synchronized to the data packets (IEEE P802.3bf) All functionality should be included in an IP-core such that it is easy for users to implement a timing aware Ethernet interface.

16. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 16 Thank you

17. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 17 Future research - II EPDSPDPayload OSI model Physical and Data Link layer Use standards  IEEE 802.3 “CSMA/CD “ (Ethernet)  IEEE 1588 “Precision Clock Synchronisation Protocol (PTP)”  IEEE P802.3bf “Ethernet Support for Time Synchronization Protocol”; In preparation… 001111101010010001010010010111xxxxx K28.5D16.2K27.7Dx.Y /I//S//D/ Dx.Y /D/

18. P. Jansweijer Nikhef Amsterdam Electronics- Technology Amsterdam July 5-6, 2010KM3NeT: General WPF/L meeting 18 Master time t pd 1 Slave time t offset t disp  mTx  sRx  mRx  sTx 1 2 Time offset and fibre dispersion (t disp = 15150 ps over 10.7 km => D( ) = l416 ps/km)