1. EMI is partially funded by the European Commission under Grant Agreement RI-261611 Future Proof Storage with DPM Oliver Keeble (on behalf of the CERN IT-GT-DMS section)

2. EMI INFSO-RI-261611 Disk storage for the grid 36PB – 10 sites with > 1PB Over 200 sites in 50 regions Over 300 VOs have access to DPMs DPM today Future Proof Storage with DPM, EGI TF Prague2

3. EMI INFSO-RI-261611 Current production version is 1.8.3 – https://svnweb.cern.ch/trac/lcgdm/blog/official-release- lcgdm-183 https://svnweb.cern.ch/trac/lcgdm/blog/official-release- lcgdm-183 – In EPEL, EMI1 and EMI2 – No gLite – see tutorial on Wed on how to upgrade What’s new – EPEL compliance – HTTP/DAV frontend (old dpm httpd is gone) – NFSv4.1 frontend (read only) – Thread safe clients – Synchronous get requests – … DPM today Future Proof Storage with DPM, EGI TF Prague3

4. EMI INFSO-RI-261611 Next core release: 1.8.4 – First DMLITE release (see later…) DAV frontend will start using the new libraries Other components not using dmlite for now – Improved replication mechanism – 32bit client support – Lots of other small fixes – Feature complete… finishing certification Change in the release process – Independent component releases – Faster, lighter releases 1.8.4 Future Proof Storage with DPM, EGI TF Prague4

5. EMI INFSO-RI-261611 DMLITE is the result of a significant refactoring effort to make DPM modular Better separation between frontends and backend – Cleaner, more open, much improved performance Improved integration with standard building blocks – Hadoop, Memcache, S3, Lustre, … Refactoring & DMLITE Future Proof Storage with DPM, EGI TF Prague5

6. EMI INFSO-RI-261611 Standard protocols, standard clients – HTTP/DAV + extras, WAN transfers, 3 rd party copy… – NFS 4.1/pNFS (r/o) – Xrootd (rewritten) Forthcoming – GridFTP Legacy interfaces remain untouched – No large scale revalidation required Improved Frontends Future Proof Storage with DPM, EGI TF Prague6

7. EMI INFSO-RI-261611 Standard protocols, standard clients – HTTP/DAV + extras, WAN transfers, 3 rd party copy… – NFS 4.1/pNFS (r/o) – Xrootd (rewritten) Forthcoming – GridFTP – Ubiquitous access to grid storage Legacy interfaces remain untouched – No large scale revalidation required Improved Frontends Future Proof Storage with DPM, EGI TF Prague7

8. EMI INFSO-RI-261611 This is where DMLITE shines – Plugin based, open for constant evolution Improved nameserver performance – Connection pooling, improved SQL, memcache layer, … Support for multiple pool types – Legacy DPM, Hadoop/HDFS, S3, … – Sharing a single namespace if desired – Possibility for opportunistic pools Federation – See the following presentation And this is the beginning, much more coming – Python bindings, Lustre, VFS, … Improved Backends Future Proof Storage with DPM, EGI TF Prague8

9. EMI INFSO-RI-261611 This is where DMLITE shines – Plugin based, open for constant evolution Improved nameserver performance – Connection pooling, improved SQL, memcache layer, … Support for multiple pool types – Legacy DPM, Hadoop/HDFS, S3, … – Sharing a single namespace if desired – Possibility for opportunistic pools Federation – See the following presentation And this is the beginning, much more coming – Python bindings, Lustre, VFS, … Improved Backends Future Proof Storage with DPM, EGI TF Prague9

10. EMI INFSO-RI-261611 I/O performance Future Proof Storage with DPM, EGI TF Prague10 LAN / Chunk Size: 10k-20k / File Size: 2G ProtocolN. ReadsRead SizeRead Time HTTP50022,773,1120.43 HTTP100046,027,1430.87 XROOT50022,773,1120.39 XROOT100046,027,1430.78 RFIO50022,773,112136.35 RFIO100046,027,143274.89 PRELIMINARY

11. EMI INFSO-RI-261611 Performance, performance, performance Future Proof Storage with DPM, EGI TF Prague11 https://cdsweb.cern.ch/record/1458022?ln=en

12. EMI INFSO-RI-261611 Puppet for configuration – Popular among large data centers – Lots of modules for popular tools (which we now rely on – apache, memcache, nagios, …) Nagios for monitoring – We reuse as much as possible And there’s a lot already available – Added specific plugins for detailed status and performance monitoring Easy administration Future Proof Storage with DPM, EGI TF Prague12

13. EMI INFSO-RI-261611 Puppet for configuration – Popular among large data centers – Lots of modules for popular tools (which we now rely on – apache, memcache, nagios, …) Nagios for monitoring – We reuse as much as possible And there’s a lot already available – Added specific plugins for detailed status and performance monitoring Easy administration Future Proof Storage with DPM, EGI TF Prague13

14. EMI INFSO-RI-261611 Standards – Leverage existing components & clients – Mature ecosystem – Less maintenance work Community – Guidance by the stakeholders – Independence from funding cycles – Now in talks for the creation of a “DPM Collaboration” to drive the project post-EMI Why “future proof”? Future Proof Storage with DPM, EGI TF Prague14

15. EMI INFSO-RI-261611 DPM is the most numerous SE on the WLCG infrastructure The project has never been in better shape We are soliciting statements of support for a collaboration In discussions with – GridPP (UK) – WLCG France – Taipei (WLCG Tier 1 using DPM) The DPM Collaboration Future Proof Storage with DPM, EGI TF Prague15

16. EMI INFSO-RI-261611 DPM around the world Future Proof Storage with DPM, EGI TF Prague16

17. EMI INFSO-RI-261611 DPM has received a lot of investment thanks to EMI – These developments are now being released This investment has gone into – Making it modular – Supporting standards (HTTP, NFSv4.1,…) – Profiting from existing technology – Performance – Manageability Summary Future Proof Storage with DPM, EGI TF Prague17

18. EMI INFSO-RI-261611 Issue with LFC API / Py26 (#84716) – Tracked down to issue with EPEL5 Swig version – Currently considering the best solution Cleaning up invalid LFC replicas (#83335) – Solved… bulk requests using API/CLI tools DAV EMI2 install failure (#85141) – Related to dependencies on gridsite, solved with the EPEL update EMI2 LFC dies regularly (#85161) – Happens to any of lfc/dpm/dpns daemons – Documented as a known issue with 1.8.3 https://svnweb.cern.ch/trac/lcgdm/blog/official-release-lcgdm-183 32 bit support (#81508) – Available in EMI2 and EPEL repositories Recent Issues Future Proof Storage with DPM, EGI TF Prague18