1. Addressing the Software and Hardware Trade-Offs of an Embedded Distributed System The Case of the ANTARES Data Acquisition and Detector Control S. Anvar, CEA Saclay Dapnia On behalf of the Antares collaboration

2. 2 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control http://antares.in2p3.fr A Neutrino Telescope

3. 3 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control The Antares Collaboration Institutes Particle physics Astronomy Sea science

4. 4 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Constraints Environmental constraints Baseline data rate of ~100 kHz / PMT Data rate fluctuations ~2MHz / PMT due to bioluminescence Strong uncertainties on environment parameters Reduced space and thermal conductivity … Reliability (MTBF) Maximize simplicity Minimize consumption Minimize number of components … Deployment constraints High cost of submarine cables Limits on string cable rigidity or sea operations …

5. 5 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Design principle String control Junction box Electro-mechanical cable Electro-optical cable (50 km) Detectors Slow control Data Clock Energy Processing nodes Acquisition nodes OffshoreOnshore 12 strings 25 detecting nodes per string (every 12.5m) 3 optical modules (PMTs) per acquisition node 2 "ARS" digitizing chips per optical module Real time processing: Offshore: readout 1800 ARS chips spread out over 30 000 000 m 3 @ 2500m underwater Onshore: perform real time trigger computations on incoming data.

6. 6 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Each offshore acquisition node: one DAQ/SC Board ARS readout/SC Local trigger 10 6 gates Xilinx FPGA Real-time embedded CPU PPC MPC860 VxWorks SDRAM Memory 64 MB SDRAM Memory 64 MB DAQ/SC Board Achieved power consumption of ~5W including optical Ethernet tranceiver laser Ethernet 100 Serial links (RS485, RS232)

7. 7 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control DAQ/SC Board ARS chip The ARS chip and the DAQ/SC board

8. 8 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Network Topology (1) Cable rigidity Cable diameter Connection complexity Each LCM has its own I/O data flow (Slow Control & Physics Data) SCM Sectors M SCM M To Shore (DWDM) 100 Mb/s 1 Gb/s

9. 9 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Network Topology (2)

10. 10 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Offshore switch Board Allayer chip The offshore switch board

11. 11 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control The electronic crate

12. 12 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Used as data concentrator many input ports one output port Network Topology (3) Solution barrel shifting synchronized by global clock PC ON-SHORE SWITCH: (in: 60x1000, out: 20x1000) SECTOR SWITCH (in: 5x100, out: 1x1000) NNNNN NNNNN risks of congestion intelligent flow control in node ~15 Mb/s Instant ~6 Gb/s

13. 13 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Massively Distributed and Heterogeneous System Final System: A Network of ~320 Nodes ~300 Offshore Nodes (MPC860 + VxWorks) ~20 Onshore Nodes (PC + Linux) A number of distributed applications running in parallel and communicating DAQ L2 Trigger Slow Control / Acoustic positioning acquisition L1 trigger ? Run control

14. 14 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Software Technologies Programming Languages C/C++ for offshore software components C++ / Java for onshore processing Java / C++ for onshore display Distribution Multiple messaging and data servers (ControlHost) TCP/IP based VxWorks Zero-copy TCP/IP (~25Mb/s ~38 Mb/s per node) Offshore network traffic control (barrel shifting) Data and configuration storage Oracle database Root files Design & Development Object Paradigm UML (Unified Modeling Language) + Extensions Model-driven engineering

15. 15 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Interrupt – shared memory communication ARS readout (FPGA) intrHandler (RTOS) frameSender task (RTOS) Semaphore (RTOS) Data Memory (SDRAM) write read and write Actual frame formatting and sending through TCP/IP socket

16. 16 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Interrupt – shared memory communication seen as asynchronous method call ARS_Read +fillBuffers(id:int) frameSender1 6 FrameSender +acq(in frameID:int) acq(in frameID:int) :ARS_Readout:FrameSender 6 > AR_Node readout Interrupt - SharedMem 1 > FS_Node framer FrameSenderARS_Readout > asynchronous method call reference to design pattern

17. 17 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Applying the design pattern: model transformation ARS_Read +fillBuffers(id:int) frame Sender 1 6 FrameSender +acq(in x:Frame) Implemented in VHDL IntrSharedMemProxy +doIntr(in x:PrmRefType) FrameSender Skeleton +accept(in x:Frame) IntrSharedMemSkeleton #run() #installInterrupt() +accept(in pr:PrmRefType) PrmRefType {bind(Frame)} RealTime::Thread #run() reusable software 1

18. 18 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Widespread belief Performance goes against abstraction & flexibility Wrong if the issue is explicitly addressed Actually Abstraction modularity specific optimization perf. Wrong if the issue is explicitly addressed Actually Abstraction modularity specific optimization perf.

19. 19 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Evolutions (1) ARS readout/SC Local trigger 10 6 gates Xilinx FPGA Real-time embedded CPU PPC MPC860 VxWorks SDRAM Memory 64 MB SDRAM Memory 64 MB DAQ/SC Board Ethernet 100 Serial links (RS485, RS232) ARS readout/SC Local trigger Real-time embedded CPU System On Chip Xilinx Virtex II Pro? Xilinx Virtex IV ARS readout/SC Local trigger Real-time embedded CPU System On Chip Xilinx Virtex II Pro? Xilinx Virtex IV SDRAM Memory 64 MB SDRAM Memory 64 MB DAQ/SC Board Ethernet 100 Serial links (RS485, RS232) Lower consumption Reduced number of components increased reliability

20. 20 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Evolutions (2) Present scheme: « all data to shore » Unexpected variable baseline rates: ~100 400 kHz instead of ~70

21. 21 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Evolutions (2) Present scheme: « all data to shore » Unexpected variable baseline rates: ~100 400 kHz instead of ~100 Software L1 trigger Reduced L1 partial data sent from offshore to central L1 workstation Central L1 computes trigger with global vision of detector Trigger decision broadcasted through software Expected gain in data rate: x1000 From 5 to 30 cm/s (!)

22. 22 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control Conclusions Slow control Data Clock Energy HEP DAQ and trigger techniques High data rates Very weak signal Highly distributed system Soft real time constraints … Challenges unusual for HEP No known beam (calibration) Very embedded system Very hostile environment Uncontrolled factors (biolum) … Both Robust and Flexible Design Allows adaptation Allows for remote evolution …