1. Analysis efficiency Andrei Gheata ALICE offline week 03 October 2012

2. Sources for improving analysis efficiency The analysis flow involves mixed processing phases per event – Reading event data from disk – sequential (!) – De-serializing the event object hierarchy – sequential (!) – Processing the event - parallelizable – Cleaning the event structures - sequential – Writing the output – sequential but parallelizable – Merging the outputs – sequential but parallelizable The efficiency of the analysis job: – job_eff = (t ds +t proc +t cl )/t total – analysis_eff = t proc / t total Time/event for different phases depending on many factors – T read ~ IOPS*event_size/read_throughput – to be minimized Minimize event size, keep under control read throughput – T ds +T cl ~ event_size*n_branches – to be minimized Minimize event size and complexity – T proc = ∑ wagons T i – to be maximized Maximize number of wagons and useful processing – T write = output_size/write_throughput – to be minimized t read t ds t proc t cl t write Event #0 Event #1 Event #2 Event #m Event #0 Event #1 Event #2 Event #n Event #0 Event #1 Event #2 Event #p t merge

3. Monitoring analysis efficiency Instrumentation at the level of TAlienFile and AliAnalysisManager Collecting timing, data size transfers, efficiency for different stages – Correlated with site, SE, LFN, PFN Collection of data per subjob, remote or local – mgr->SetFileInfoLog(“fileinfo.log”); – Already in action for LEGO trains

4. Monitored analysis info ################################################################# pfn /11/60343/578c4420-6178-11e1-9cd1-00266cfd8b68#AliAOD.root url root://xrootd3.farm.particle.cz:1094//11/60343/578c4420-6178-11e1-9cd1-00266cfd8b68#AliAOD.root se ALICE::Prague::SE image 1 nreplicas 0 openstamp 1348559810 opentime 0.701 runtime 2836.503 filesize 668.547 readsize 671.568 throughput 0.237 ################################################################# pfn /13/34934/2ed51c74-618b-11e1-a1cc-63e6dd7c661e#AliAOD.root url root://xrootd3.farm.particle.cz:1094//13/34934/2ed51c74-618b-11e1-a1cc-63e6dd7c661e#AliAOD.root se ALICE::Prague::SE image 1 nreplicas 0 openstamp 1348562662 opentime 1.484 runtime 3890.802 filesize 642.818 readsize 640.459 throughput 0.165 #summary######################################################### train_name train root_time 36865.630 root_cpu 2564.810 init_time 74.647 io_mng_time 34510.137 exec_time 2280.846 alien_site CERN host_name lxbse13c04.cern.ch Processed input files Analysis info

5. Throughput plots A simple and intuitive way to present the results Will allow diagnosing both the infrastructure & the analysis Throughput [MB/sec] Time [sec] PFN1PFN2PFN3PFN4PFN5 Initialization I/O execution

6. Few numbers for an empty analysis Spinning 50 MB/sAccess time 13 msRead size 270 MB AOD PbPb Job time 45.5 secThroughput 5.93 MB/sJob efficiency 86.5 % SSD 266 MB/sAccess time 0.2 msRead size 270 MB AOD PbPb Job time 39.5 secThroughput 6.83 MB/sJob efficiency 94.1 % Inter site 7.4 MB/s (JINR)Access time = RTT 63 ms + local disk access time (?) Read size 21.53 MB AOD PbPb L=200, Job time 258 secThroughput 0.083 MB/sJob efficiency 2.5 % L= number of concurrent processes running on the disk storage server I/O latency is a killer for events with many branches De-serialization is determinant for locally available data – it depends on the size, but ALSO on the complexity (number of branches) L=5, Job time 46.8 secThroughput 0.46 MB/sJob efficiency 13.4 %

7. The source of problems Highly fragmented buffer queries over high latency network – Big number of buffers retrieved sequentially No asynchronous reading or prefetching enabled in xrootd or elsewhere ROOT provides the mechanism to compact buffers and read them async: TTreeCache – Not used until now – Now added in AliAnalysisManager

8. Reading improvement AOD AOD PbPb, JINR::SE (RTT=65ms to CERN) Cache sizeAsync. readSpeedup 0 (current status)-1 50 MBNo Yes 4.05 4.20 100 MBNo Yes 4.91 5.45 200 MBNo Yes 4.91 5.53 300 MBNo Yes 4.59 5.72

9. Reading improvement AOD AOD pp, LBL::SE (RTT=173ms to CERN) Cache sizeAsync. readSpeedup 0 (current status)-1 50 MBNo Yes 20.50 20.53 100 MBYes20.12

10. Reading improvement MC ESD pp, CNAF::SE (RTT=20 ms to CERN) ESD pp, CERN::EOS (RTT=0.3 ms) Cache sizeAsync. readSpeedup 0 (current status)-1 50 MB, ESD cache onlyYes1.02 50 MB, ESD, TK, TR cachesYes4.22 Cache sizeAsync. readSpeedup 0 (current status)-1 50 MB, ESD, TK, TR cachesYes1.32

11. What to do to get it For AOD or ESD data, nothing – Cache set by default to 100 MB, async read enabled – Size of cache can be tuned via: mgr->SetCacheSize(bytes) For MC, the cache sizes for kinematics and TR will follow the manager setting – Don’t forget to use: mcHandler->SetPreReadMode(AliMCEventHandler::kLmPreRead)

12. To do’s Feed analysis info to alimonitor DB – Provide info in real time about analysis efficiency and status of data flows – Point out site configuration and dispatching problems TTreePerfStats-based analysis – Check how our data structures perform and pin down eventual problems