1. WR & KM3NeT Peter Jansweijer

2. Artist impression & detection method
KM3NeT
Artist impression & detection method
100m
~ 860m
640 strings
18 DOM/string
DOMs
PMTs
Volume: ~5 km3
Cherenkov m n
DOMs in the deep sea at 3-5 km depth

3. Digital Optical Module (DOM)
Lower
Hemisphere
19 PMTs
Upper
Hemisphere
12 PMTs
Central
Logic
Board
PMT Base:
High Voltage Supply
Analog Front-End

4. Central Logic Board (CLB)
PMTs
Lower hemisphere
Xilinx KINTEX-7
(containing WRPC)
HPC FMC
Debug/Expansion
PMTs
Upper hemisphere
SFP
Readout 31 TDC’s with 1 ns bin size
Other IO: Compass/Tilt, Acoustics, Nano beacon, Temperature
White Rabbit enabled
deterministic PHY for WR available: “wr_gtx_phy_kintex7.vhd”

5. Shore Station Demonstrator setup See presentation of 7-Solutions
SPEC cards used for demonstrator.
Later replaced by CLBs
SPEC cards used for demonstrator.
Later replaced by CLBs

6. Optical Network History: first idea
Shore
Subsea
Digital Optical Modules
Shore Station
Optical network
up to 100 Km, 80 wavelengths / fiber
4x18
DOMs
Continuous Wave (CW) laser at the shore station.
Reflective Electrical Absorption Modulator (REAM) in the sea.
Pros:
No lasers sub sea => reliable!
tuning, temperature stabilization, repair, replacement at shore station
Simple timing:
Shore Station does time stamping (all electronics on shore)
only need to add time delay “DOM to Shore Station”
Easy measurement: Put the REAM in reflective mode, send a pulse back and forth and measure the transit time (OTDR)
No complexity off shore => no need for Slow Control
Sea cable is very expensive! => use many optical channels per fiber

7. 1 Rayleigh Backscattering
Bidirectional light transmission in the same fiber limited (~2 Km) due to Rayleigh Backscattering
use two fibers for the long stretch
Add extra path to “accurately” measure the time delay between DOM and shore station (OTDR method).
After 100 Km, need a bit of amplification
Shore
Sub-sea
< 1 Km
up to 100 Km
Only one
DOM drawn
for simplicity
Idea:
Pre calibrate time difference between forward and backward path…

8. 2 Slow Control low extinction ratio
Slow Control needed
Solution: add “low extinction ration” modulation on the CW laser
Slow Control Modulation
(broadcast) CW
Shore
Sub-sea
MOD
Only one
DOM drawn
for simplicity

9. 2a Slow Control
Low Extinction Ration modulation was proven to work only in laboratory environment.
More rigid to use a separate Slow Control wavelength!
Shore
Sub-sea
Slow Control
(broadcast)
Only one
DOM drawn
for simplicity

10. 3 Single pulse delay measurement?
Measuring time with a single pulse is not so accurate…
Solution: White Rabbit
“Every 125 MHz tick is put to good use” [1]
Shore
Sub-sea
Slow Control
(broadcast)
Only one
DOM drawn
for simplicity
The idea:
Network asymmetry can be modelled in a revised “WR Link Model”
[1] The White Rabbit Project

11. 4 REAM to SFP The REAM is expensive and single supplier => SFP
Slow Control
(broadcast)
Shore
Sub-sea

12. Are we there? Need a dedicated WR implementation for the Shore Station
See presentation this afternoon by 7-Solutions
Non-standard Ethernet: No Auto Negotiation, packet routing, pause frame handling
Calibration is not simple…
The idea to model network asymmetry in the “WR Link Model” needs knowledge of fiber length of either the broadcast channel, or the individual DOM-> Shore Station links.
SFP = laser sub-sea
We may loose the DOM-> Shore Station communication due to wavelength shift over time. Need possibility for laser tuning over a unidirectional communication channel.
Diverting from WR main development
Need to apply KM3NeT adjustments to new firmware/software releases.
Connecting to standard Ethernet equipment (for tests) needs constant attention.

13. Timing Calibration Using WR as a ps accurate measurement tool
SoftPLL
t(l)
D(l)
KC705
Reference: IEC 86A/1419/CDV OPTICAL FIBRES
Chromatic dispersion measurement methods
Usual WR calibration => Measure PPS differences and tweak a
KM3NeT challenge => Measure a and put it into the system and rely on accurate PPS alignment (we cannot probe it!)

14. Using WR as a ps accurate measurement tool
Measure RRT for 8 different wavelengths and calculate propagation delay of fiber for 8 wavelengths as all other delays are measured as a reference on forehand and known.
Loop back at the far end
fiber in under-sea cable (length ~95 km)
EDFA
(for MEOC Italy) f2
SFP f1
SFP
Att. 15dB f3
SFP in WR Master:
transmitter: 0 till 5dBm
receiver: -24dBm (pindiode)
SFP in WR Slave:
transmitter: 2 till 5dBm
receiver: -35dBm (APD)

15. Using WR as measurement tool courtesy to Mar v. d
Using WR as measurement tool courtesy to Mar v.d. Hoek, Gerard Kieft and Antonio d’Amico, Nikhef Amsterdam
Gerard
Sellmeier equation: “an empirical relationship between refractive index and wavelength”
Group delay
t(l)
Antonio

16. WR synchronization after restart courtesy to Diego Real, David Calvo IFIC Valencia and 7-Solutions
Once in a while “wrong” synchronization
(11/1200)
Version 3.3 switch software (test with V4 ongoing)
SPEC f_eval_sync_detect_threshold = 8192
CLB f_eval_sync_detect_threshold =

17. Future? (KM3NeXT)?
1 fiber =
80 wavelengths =
40 bidirectional 1Gbps links =
40 WR switches ~ 9 x 40 DOMs = 360 DOMs
(note: 4 x 18 = 72 DOMs per fiber in the current topology)
1x18 DOMs
(Or ~100 Mbps/DOM)
Shore
Sub-sea
Actual rates based on pessimistic PPM-DU data (10 KHz / PMT)
(see slide 9 of Marco Circella, Steering Committee July 29,
10 KHz rate * 31 PMTs / DOM = 310 KHz / DOM
6 bytes / event => 6 * 310 KHz = 1,81 MB/s => ~ 20 Mbps / DOM

18. KM3NeT Future? (KM3NeXT)? Pros: Calibration issues solved
Standard WR (keep up with software/firmware updates)
Standard Ethernet:
Standard Pause frame Flow control
No complicated packet routing
Auto negotiation
plug and play with industry network cards => easy testing.
Optical network with industry standard DWDM may be more relaxed when data is accumulated.
No broadcast link bandwidth limit or single point failure
Cons:
Need development for a 15 year reliable switch in a new form factor.
Need subsea switch maintenance.
General remark:
A wish for 10 Gbps (data) uplink ports at first level Shore Station switches.
KM3NeT

19. Thank you

20. Backup Slides

21. Stream Selector (IPMUX)
CLB block diagram
IP/UDP Packet Buffer
Stream Selector (IPMUX)
31 TDCs
Start Time Slice UTC &
Offset counter since
Fifo
TDC0
Time Slice Start
RxPacket
Buffer
64KB
RxPort 1
31 PMTs
RxPort 2
Fifo
TDC 30
Rx_mac2buf
Rx_mac2buf
Rx_buf2data
Rx Stream Select
Flags
RxPort_m
Management
& Control
aux_master S 6 1 7
Pause Frame
Management
& Config.
State Machine 5
ADC
Fifo 4
Hydrophone 2 3
TxPacket
Buffer
32KB
TxPort 1
Management
& Control
WB Crossbar
(3x3) S0 M1 S1 S2 M0 M2
TxPort 2 S
Tx_pkt2mac
Tx_pkt2mac
Tx_data2buf
Tx Stream Select
Multiboot
Flags
TxPort_m S
Management
& Control S
WB Crossbar
(1x8) M M M S M
Debug LEDs M M M M M M S
GPIO
2nd CPU
LM32 M S S S
Xilinx
Kintex-7
MEM S
WB Crossbar
(3x3) S0 M2 S1 M0 M1 S2
SPI
UART
I2C3
I2C1 S
I2C2 S
Data
UTC time & Clock (PPS, 125 MHz)
Control
Point to Point interconnection
SPI
Flash
Debug RS232
Temp
Tilt & Compass
Nano
Beacon
Wishbone bus

22. Optical network