1. The ANTARES Offshore Data Acquisition A Highly Distributed, Embedded and COTS-based System S. Anvar, H. Le Provost, F. Louis – CEA Saclay DAPNIA

2. A Neutrino Telescope http://antares.in2p3.fr

3. The "0.1 km 2 " Project 13 Strings 30 Detecting Nodes Per String (every 10m) 3 Optical Modules (PMs) Per Detecting Node 2 "ARS" Digitizing Chips Per Optical Module Offshore Data Acquisition System: Readout in Real Time 2340 ARS Chips Spread Out Over 30000000m 3 @ 2000m Underwater Onshore: Perform Real Time Trigger Computations On Incoming Data Farm of 100 PC-Linux Workstations

4. Detector Topology To Shore String Control Module ( SCM ) Local Control Module ( LCM ) Junction box ( JB ) Electro Mechanical cable ( EMC ) Electro Optical Cable ( EOC ) Optical Module ( OM ) Slow-Control Data Clock Energy

5. In Each LCM: One DAQ Board ARS 1 Séquenceur Arbitrage Bus MPC860 DAQ Memory Controller Interface Bus MPC860 Données Adresses COTS Processor MPC860 max 80MHz Max P: ~700 mW Integrated: 4 Serial 1 Ethernet 101 DAQ Memory Event Type 1 Event type 2 Event type 3 System Communications Conversion serial // Process Unit Data reduction Time reordering CRC Mémoire ARS Contrôle Mémoire Lecture ARS 2 ARS 3 ARS 4 ARS 5 ARS 6 RARS 110 Ethernet ARS Memory Supervisor Control & Status Register Write Memory controller ARS Memory ReadOut Memory controller MPC860 Bus Manager MPC860 Bus Interface Addresses Data DAQ board LCM Slow-Control System Memory Status ARS Memory Status Self-test

6. Network Topology (1) Cable rigidity Cable diameter Connection complexity SCM Each LCM has its own I/O data flow (Slow Control & Physics Data) Sectors S S S S M S S SCM S S M To Shore (DWDM)

7. Used as data concentrator many input ports one output port Network Topology (2) PC ON-SHORE SWITCH: (in: 78x1000, out: 100x1000) SECTOR SWITCH (in: 5x100, out: 1x1000) LCM risks of congestion intelligent flow control in LCM ~15 Mb/s

8. Massively Distributed System Final System: A Network of ~500 Nodes –~400 Offshore Nodes (MPC860 + RTOS) –~100 Onshore Nodes (PC + Linux) 3 Communicating Parallel Applications –DAQ –Trigger –Slow Control

9. Software Technologies Programming Languages –C++ For Fast Software Components –Java for Non Real-Time (not yet mature for RT) Design & Development –Object Paradigm –UML (Unified Modeling Language) + Extensions –Trigger/DAQ Design Patterns (Functional/Distribution Separation of Concerns) –Specific Development Methodology for Distributed Systems (MORDICUS)

10. Distribution Design Patterns LCM PC CASCADED SWITCHES Performance Parallelism (Scalability) Intrinsic Parallelism (Progressive deployment)

11. acq (in frameID:int) accept (in f:Frame) :ARS_Read :Storage [TRIG = OK] Sequence Diagram (Functional) :Frame Formatter [TRIG = OK] dump() :Trigger:EventBuilder store() [TRIG OK] trig (e:Event)

12. Class Diagram (Functional) Trigger +trig(in d:Event) : bool 1 FrameEvent +rawParts 0..n ARS_Read +fillBuffers(id:int) 0..1 +processor +builder 1 n +storage Data 1 +formatter1 6 +accept(in x: Frame) #dump(in x:Frame) EventBuilder FrameFormatter +acq(in frameID:int) Storage +store(in d:Event)

13. > ALTERA20K readout Interrupt - SharedMem 12 PowerPC-RTOS framer > PC-Linux processor > CORBA ** TriggerFrameFormatterStorageARS_Read Specialized Deployment Diagram EventBuilder >

14. 1 +builder +acq(in frameID:int) FrameFormatter > EventBuilderInterface +accept(in x:Frame) Adapts FrameFormatter EventBuilder (Remote + Farm) Communication on Caller Side Implements Farm Management POLICY Automatic Model Transformation 1 > EventBuilderProxy +accept(in x:Frame) > EventBuilder +accept(in x:Frame) 11

15. New Challenge in HEP Real-Time Computing Massively Distributed –400 Offshore Processors –100 Onshore Processors Deeply Embedded –Heat Dissipation –Limited Space –Limited Power Numerous Modules: MTBF Problem –400 "Satellites" With 1/10 th of Budget –Robustness is Critical DESIGN & DEVELOPMENT METHODOLOGY