1. IRFU The ANTARES Data Acquisition System S. Anvar, F. Druillole, H. Le Provost, F. Louis, B. Vallage (CEA) ACTAR Workshop, 2008 June 10

2. IRFU The ANTARES detector http://antares.in2p3.fr

3. IRFU Acquisition nodes Line Control Junction box Electromechanical cable Electro-optical cable Slow-control Data Clock Energy Processing nodes OffshoreOnshore  12 detection lines (400 m)  300 acquisition nodes (25 / line)  900 photomultipliers (3 / node)  1800 data sources @ 20 Mb/s max  System spread over 30000000 m3 @2500 m depth  Onshore Farm of 80 processing nodes The "0.1 km2« project photomultiplier

4. IRFU ANTARES Sector Line

5. IRFU ANTARES Line deployment (1)

6. IRFU ANTARES Line deployment (2)

7. IRFU Photomultipliers counting rates bioluminescence burst (April 2003)* seconds kHz days kHz Base line (April 2003)* * data published on the ANTARES site http://antares.in2p3.fr High fluctuations in counting rates

8. IRFU The Photomultiplier signal processing 1 Single Photon Electron (SPE) : Charge/Time stamp = 6 bytes 1 full waveform (Anode): Anode + clock samples = 263 bytes Waveform used for detector calibration / PM signal analysis (very useful) On line trigger (Neutrino track finder) only uses SPE events. Detector operates in SPE mode  ~ 10 Mb/s per PM in average  *900 PM = ~9 Gb/s for the full detector ARS ASIC : Analog Ring Sampler

9. IRFU Offshore: on-board system Thermal dissipation (titanium container) Limited room (~15 boards of 12 cm  ) Limited electrical power (~35 W / storey) Very limited access (1  / 3 years) Numerous modules: MTBF problem (  400 spatial « satellites »)

10. IRFU LCM Detector readout : an Ethernet network Slow control LCM MLCM LCM SCM Data Responsibilities : IN2P3 : Slow-control/data base CEA : Sub-marines data NIKHEF : Data wavelength multiplexing For each sector (DWDM) NIKHEF : Run control/on shore acquisition Ethernet 100 Mb/s (1 fibre WDM/45 m max) Ethernet 1 Gb/s (2 fibres DWDM/330 m max) Junction Box 5x Ethernet 1 Gb/s 1x Ethernet 100 Mb/s (2 fibres DWDM/100 m) 60x Ethernet 1 Gb/s 12x Ethernet 100 Mb/s (24 fibres DWDM/40 Km) 5 Sectors 12 Lines Processor board Switch board Shore station LCM ; Local Control Module MLCM ; Master LCM SCM : String control Module A separate proprietary Network to time stamp the PM signals (1 ns precision)

11. IRFU RTOS SDRAM Memory(64 MB) Flash Memory (4 MB) Boot / Local File System Offshore Acquisition node A dedicated processor board Processor (Motorola MPC860P@80MHz) Slow Control Slow Control Task Slow control for the storey Ethernet Link 100Mb/s To Shore station Data Task Data Data from the storey (ASIC ARS) Programmable Logic Programmable Logic FPGA

12. IRFU LCM data/slow control processing ARS data Logic Device (FPGA) Multiplexor Buffers Dynamic Memory 1/Status 2/RAZ Time 3/Counters 4/SPE 5/Anode Waveform 6/Anode Waveform +Dynode Counters SPE Anode Waveform Counters ARS 0 ARS 1 ARS 0 ARS 5 ARS 1 ARS 2 ARS 3 ARS 4 Processor 100 Mb/s Ethernet Port 104 ms Data To shore PC 1 To shore PC 2 104 ms Periodic Slow control/on demand/configuration 104 ms Data Clock Experiment Slow Control object

13. IRFU Offshore Processor board Power 4W full charge TCP/IP Network throughput –Running Linux/MontaVista : 25 Mb/s –Running vxWorks/windRiver : 30 Mb/s, 50 Mb/s With « Zero Copy Buffer » option Operational Configuration –Selected RTOS vxWorks –No DAQ performances drop due to slow- control –Measured data rate : 50 Mb/s

14. IRFU Global DAQ Architecture Off-shore : 300 Detection nodes 300 Detection nodes No trigger / All data to shore No trigger / All data to shore Nodes synchronised by a global clock (Physics event time stamp) Nodes synchronised by a global clock (Physics event time stamp) Each node send 104 ms of data to the same On-shore processing node Each node send 104 ms of data to the same On-shore processing node On-shore : 80 Processing nodes 80 Processing nodes Each node treat a full 104 ms detector view. Neutrino Track finding. Each node treat a full 104 ms detector view. Neutrino Track finding. Full software trigger Full software trigger Ethernet switch routing Off-shore : robustness/hardware frozen On-shore : Processing power/upgradable

15. IRFU On-line Trigger principle No dead time “Self triggered” SPE signals send to shore All data send to shore. No hardware trigger (local storey level1 trigger implemented but not used) High fluctuations (bioluminescence) absorbed in the front end processor board  Max storey rates: 120 Mb/s High Rate Veto applied to a PM, typically 400 kHz

16. IRFU Network topology Used as data concentrator multiples input ports  one single output port PC ON-SHORE SWITCH: (in: 60x1000, out: 100x1000) SECTOR SWITCH (in: 5x100, out: 1x1000) LCM  congestion risk  intelligent control ~21 Mb/s

17. IRFU Acquisition Data flow Offshore processor PM High rate Veto @400 kHz x3 x5 Off shore Ethernet Switch x60 Ethernet Backbone x1 50 M bit/s max Data Filter x80 Offshore Onshore 250 M bit/s max 10 Hz Neutrino Candidates (2,6 G Bytes/day) Offline : ~3 ascending Neutrino/day Counting rate (12 G Bytes/day)

18. IRFU Conclusion No hardware trigger. All Data to shore concept (pushed by NIKHEF/DWDM) Offshore system fully configurable from shore (firmware, software, RTOS image) Onshore trigger fully upgradable Concept is working fine. 12 lines currently operating by 2500 m depth. Development time not reduced because software development and debugging may be as time consuming as hardware.