Pierre VANDE VYVRE ALICE Online upgrade October 03, 2012 Offline Meeting, CERN

03-Oct-2012 P. Vande Vyvre 2 Detector Event Size (MByte) After Zero After Data Suppression Compression Input to Online System (GByte/s) Peak Output to local data storage (GByte/s) Avrg Output Computing Center (GByte/s) ITS TPC TRD Others (1) Total Requirements: Event Size and Rate Expected peak Pb-Pb Minimum Bias rate after LS2: 50 kHz System must be able to scale with a safety factor of 2 (readout part from the start) Expected average over a fill Pb-Pb Minimum Bias (MB) rate after LS2: 20 kHz Global data compression strategy:  Optimize the overall cost by doing zero suppression on the detectors  Data throughput reduced in the online farm by data compression (no selection) Combined efficiency of ALICE and LHC: ~10 6 s/month of data taking 1 month of HI: ~2.0 x events (1) Under study

03-Oct-2012 P. Vande Vyvre 3 Online System Requirements Common DAQ and HLT farm for cost reason. Should also be usable as offline Tier 1. Detector readout  Detector throughput at 50 kHz: 9 Tbit/s + safety factor 2 for 100 kHz  Capacity : 25 Tbit/s (~2500 detector links 10 Gb/s) First-Level Processors (FLPs)  Input : 12 inputs at 10 Gb/s  ~250 FLPs needed Event building  Input to global reconstruction: 50 kHz * ~ 4.5 MB/event: ~ 225 GB/s  Output to data storage: ~ 82.5 GB/s  Total network throughput : ~310 GB/s or 2.5 Tb/s  250 x 2 links at 10 Gb/s (or 1 link at 40 Gb/s) to event building and HLT Event-Building and Processing nodes (EPNs)  Current HLT: ~200 nodes with GPUs, ~2500 cores  Computing power requirements increase by ~100 in 6 years  Technology evolution (“Moore’s law” extension to computers): factor ~16  → ~1250 EPNs with GPUs needed  ~1250 links at 10 Gb/s or 40 Gb/s to the network

Upgrade Online 2x10 or 40 Gb/s FLP DAQ and HLT 10 or 40 Gb/s FLP EPN FLP ITS TRD Muon FTP L0 L1 FLP EMCal EPN FLP TPC Data Storage FLP TOF FLP Farm Network Farm Network PHOS Trigger Detectors ~ 2500 DDL3s 10 Gb/s ~ 250 FLPs~ 1250 EPNs L0 Data Storage EPN Storage Network Storage Network RORC3 03-Oct-2012 P. Vande Vyvre 4

Dataflow Combination of continuous and triggered readout Fast Trigger Processor to complement/replace the present CTP L0: Minimum Bias trigger for every interaction L1: selective trigger LHC clock and L0 trigger distributed for data tagging and test purpose Detector triggering and electronics Continuous readout for TPC and ITS when average inter-events arrival time < TPC drift time At 50 kHz, ~5 events in TPC during the TPC drift time of 92 µs TRD: MB L0 trigger. Max: 50 kHz. TOF: MB L0 trigger. Max: 400 kHz. EMC, PHOS, Muon: L1 rare trigger Detector readout RORC3: DDL 3 interface and cluster finder in the same FPGA shared by DAQ and HLT Event building and processing FLPs: clusters of sub-events for triggered detectors and time-windows for continuous readout detectors EPNs: Tracking of time windows Association of clusters to events only possible after the tracking Final step of event building after the online reconstruction 03-Oct-2012 P. Vande Vyvre 5

03-Oct-2012 P. Vande Vyvre 6 DDL and RORC evolution Detector End (DDL)Online System End (RORC) Run DDL1: 2 Gb/s Operation phase Daughter card RORC1: 2 DDLs/board (1 to DAQ and 1 to HLT) PCI-X and PCIe Gen1 x4 (1 GB/s) Run 2~ DDL2: 6 Gb/s Compatible with DDL1 Prototyping phase VHDL + optical transceiver RORC2: 12 DDLs/board (6 to DAQ and 6 to HLT) PCIe Gen2 x8 (4 GB/s) Used by HLT for transition to PCIe Run 3~ DDL3: 10 Gb/s Technology investigation RORC3: DDLs/board (all to common DAQ/HLT) PCIe Gen3 x16 (16 GB/s) or PCIe Gen

03-Oct-2012 P. Vande Vyvre 7 First-Level Processors (FLP): powerful I/O machine Most recent architecture of Intel dual sockets servers (Sandy Bridge):  Dual socket  40 PCIe Gen lanes directly connected to the processor (40 GB/s)  Memory bandwidth: 6-17 GB/s per memory bank FLP I/O Throughput CPU 1 QPI DDR3DDR3 DDR3DDR3 DDR3DDR3 DDR3DDR3 DDR3DDR3 DDR3DDR3 DDR3DDR3 DDR3DDR3 PCIe Gen3 40 lanes CPU 1 DDR3DDR3 DDR3DDR3 DDR3DDR3 DDR3DDR3 DDR3DDR3 DDR3DDR3 DDR3DDR3 DDR3DDR3 PCIe Gen3 40 lanes

03-Oct-2012 P. Vande Vyvre 8 C-RORC aka RORC-2 prototype Performance test of PCIe Gen 2 interface on 2 generations of Intel CPU New CPU (Sandy Bridge) provide a significantly better performance than the previous one (Nehalem) in particular for small event size Fulfils the needs: 1 PCIe Gen 2 x8 provides GB/s C-RORC Prototyping Plot of H. Engel

2 different topologies are considered for the computing farm network Fat-tree network with one central director switch or router and several edge switches  + one single box to configure  - one central point of failure Spine and leaf network with several spine switches interconnecting leaf switches  + cheaper cost per port, lower power consumption, more ports per Rack Unit (RU)  + graceful degradation in case of failure  - more cabling 03-Oct-2012 P. Vande Vyvre 9 Network Topology Uplink m m … Director Switch Edge Switch 1 n P. Vande Vyvre 9 p p … 1 Spine Switch m m … Leaf Switch 1 n Uplinks

03-Oct-2012 P. Vande Vyvre 10 2 network technologies are being considered: Infiniband and Ethernet Both can be used with both topologies Network Layout Infiniband Infiniband network using a fat tree topology Edge switches: SX6025 (36 ports, 4.03 Tb/s) 32 ports: data traffic to/from the nodes 2 ports: data traffic to/from the director switch Director switch IS5200 (216 ports, 17.3 Tb/s) Total throughput: 2 x 48 x 40 Gb/s = 3.8 Tb/s Total ports: 48 x 32 = 1536 ports at 40 Gb/s 2 x 40 Gb/s … Network requirements Ports: ~250 FLPs and ~1250 EPN Total ~1500 ports at 40 Gb/s Total throughput: EPN input: 50 kHz * ~ 4.5 MB/event: ~ 225 GB/s EPN output: ~ 82.5 GB/s Total: 310 GB/s or 2.5 Tb/s

03-Oct-2012 P. Vande Vyvre 11 Network Layout Ethernet Ethernet network using a fat tree topology Leaf switch: Z9000 (128 ports 10 GbE, ) 75 ports 10 GbE: data traffic to/from the nodes 4 ports 40 GbE: data traffic to/from the other switches Spine switch: Z9000 (32 ports 40 GbE) Total throughput: 4 x 24 x 40 =3.8 Tb/s Total ports: 24 x 75 = 1800 ports at 10 Gb/s x 40 Gb/s … Network requirements Ports: ~2*250 FLPs and ~1250 EPN Total ~1750 ports at 10 Gb/s Total throughput: EPN input: 50 kHz * ~ 4.5 MB/event: ~ 225 GB/s EPN output: ~ 82.5 GB/s Total: 310 GB/s or 2.5 Tb/s

03-Oct-2012 P. Vande Vyvre 12 The ALICE online and offline sw frameworks are to be redesigned and rewritten:  New requirements (common farm, higher rates and throughput, large fraction of “offline” processing performed online etc)  New computing platforms requiring much more parallelism  DAQ-HLT-Offline Framework Panel Common Firmware framework to be defined  Detector readout  Cluster finder  Accommodate for HLT modes Software and Firmware

03-Oct-2012 P. Vande Vyvre Strategy definition and R&D.  Kick-off meeting during October ALICE week: Wednesday 10 th October 14:00-18:00  Definition of the strategy to achieve a common DAQ, HLT and offline software framework.  R&D on key hardware, firmware and software technologies Simulation, demonstrators and prototypes. Choice of technologies.  Development and exploitation of a program simulating the trigger and dataflow architecture.  Development of demonstrators and prototypes for the key hardware, firmware technologies.  Development of the new common software framework.  Selection of technologies used in production.  Technical Design Report Production and procurement. Staged deployment.  Production of the hardware developed by the projects.  Market surveys, tendering and procurement of commercial equipement.  Staged deployment with a profile compatible with the detector installation for the readout part (DDL3 and FLPs) and the accelerator luminosity for the processing (EPNs, network and data storage). Next steps (Project)