1. Filesytems and file access Wahid Bhimji University of Edinburgh, Sam Skipsey, Chris Walker …. Apr-101Wahid Bhimji – Files access

2. This session: 2 Themes: What are the demands made on storage by LHC data analysis – how might they change What solutions (filesystems or other) help us to meet such demands Apr-10Wahid Bhimji – Files access2

3. Demands - Data access LHC data analysis at Tier2s involves large files read from servers over the network. Significant load and bottlenecks on storage. Tuning is affected by : File access pattern File size (e.g. whether can be cached in RAM) Access method (direct local access, copy to WN etc.) And more… Problems can be “fixed” by the VO or site. Apr-10Wahid Bhimji – Files access3

4. Examples – rfio access A reminder of STEP last year (Sam/ Glasgow) “Rfio” access Large data files. “random” file access Large rfio buffers –> Disk server Memory exhausted, network saturated Solution Employed: Smaller RFIO buffers and copy to WN But ….. Apr-10Wahid Bhimji – Files access4

5. copy to WN – moves the problem.. More recent tests using FILESTAGER Disk seek count hits max for drive Solution: SSDs (see Sams talk)? More WN disks? Or …: Smaller files/ Sequential access ?.... Apr-10Wahid Bhimji – Files access5

6. Heart of the matter - ROOT I/O Fundamental to most LHC analysis… Need not be the bad guy. Root Data structure – “Tree” has multiple “Entries” – (physics “events”) and “Branches” (Objects – e.g. a Track). Apr-10Wahid Bhimji – Files access6

7. Problems Order that elements of Tree written into file Ordering by branches –compresses well - good for extracting a single branch BUT: in experiments data, there are lots of branches, branches are uneven sized, buffers are fixed (and small) so Branches for the same event are scattered in the file Apr-10Wahid Bhimji – Files access7

8. Aside on testing 1.“ROOT Test” - Read through the Tree with Tree->GetEntry, observe pattern with TTreePerfStats 2.Athena / CMSSW - Real analysis jobs 3.Hammercloud tests - Multiple continuous athena jobs Apr-10Wahid Bhimji – Files access8

9. “Random” access – ROOT test on Atlas AOD file RFIO access (128k buffer) Scatter in read Poor cpu Efficiency (~ 15 % ) Apr-10Wahid Bhimji – Files access9

10. Tuning / Filesystem can make a difference GPFS: 3 x faster than rfio access (in this test for this file) Buffering Apr-10Wahid Bhimji – Files access10

11. CMS – similar picture Brian Bockelman https://twiki.cern.c h/twiki/bin/view/ Sandbox/CmsIOW ork Apr-10Wahid Bhimji – Files access11

12. Reordering helps Reordered ATLAS AOD Tree using CloneTree(- 1,"SortBaskets ByEntry"); Factor 5 quicker in reading through Entries (or rfio) Apr-10Wahid Bhimji – Files access12

13. “new” ATLAS files Recently both CMS and ATLAS have been optimising no. of branches and writing baskets in event order. Files are clearly reordered - CPU eff = ~90% (for this ROOT test ) Apr-1013Wahid Bhimji – Files access

14. Hammercloud Local Access CPU eff / event rate SiteReordered files "Old" files CPU effEvent RateCpu EffEvent Rate QMUL (Lustre file://)85%40 Hz65 %27.2 Hz Glasgow (dpm rfio)62%40 Hz54%25.5 Hz Apr-10Wahid Bhimji – Files access14 http://gangarobot.cern.ch/hc/1243/test/ http://gangarobot.cern.ch/hc/1246/test/ Indicative results – many things can effect these – further testing needed

15. Copy to WN with reordered files HDD < 120 Seeks/s Much Happier! Apr-10Wahid Bhimji – Files access15

16. TTree Caching Group into a buffer all blocks from used branches, sort in ascending order and merge so read sequentially. Learn from the first few events what branches are of interest. Claims to reduce “typically by a factor 10000 the number of transactions with the disk and in particular the network with servers like xrootd or dCache.” (vector reads) Configuration still under investigation – Size of cache / training – Behavior with rfio – inc. possible segfaults. – Not yet used “by default” in ATLAS – config likely to be site specific Apr-10Wahid Bhimji – Files access16

17. Shows further improvements e.g. Dcache - Guenter Duckeck Apr-10Wahid Bhimji – Files access17

18. Conclusions I/O pattern of software can cost money for site Much of this is closely coupled to storage choices and configuration Close feedback from sites to VOs can help access method (and site tuning) to become (and stay) optimal Hotch-Potch of evidence – could do with systematic study Apr-10Wahid Bhimji – Files access18

19. Apr-10Wahid Bhimji – Files access19 Memory Tree Each Node is a branch in the Tree tr 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 T.Fill() T.GetEntry(6) T Memory Rene Brun - Sinaia

20. Wahid Bhimji – Files access20 ROOT I/O -- Split/Cluster Tree version Streamer File Branches Tree in memory Tree entries Apr-10

21. Andrew Lahiff – RAL - CMS Reconstruction (4000 events)Skimming (4000 events) Apr-1021Wahid Bhimji – Files access

22. Apr-10Wahid Bhimji – Files access22