2. Storage for Science at CERN
Giuseppe Lo Presti on behalf of the CERN IT Storage Group
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

3. G. Lo Presti, Storage for Science at CERN
Outline
Storage for Physics Data:
CASTOR, EOS
Ceph and CephFS for HPC
Storage for the Scientific Community: CERNBox
Apps ecosystem
Access methods
The Broader Sync&Share Community
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

4. G. Lo Presti, Storage for Science at CERN
CASTOR
Numbers
205 PB data (221 stored), ~800 PB capacity
Rates remarkably lower than 2016
Pure tape archive
Tape developments: see Michael’s talk
“Only” 17 PB of disk cache, 50% on CentOS 7
But still with a heavy duty-cycle
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

5. G. Lo Presti, Storage for Science at CERN
EOS
The project
New docs & web site
New Puppet module
Numbers
80 PB data (160 stored), 250 PB capacity
1.8B files
50K readers (10:1 ratio vs. writers)
Operations
Half of the LHC instances run Citrine
IPv6 picking up, 9% of users’ traffic
Migration to CentOS 7
Coming next:
Use spare CPU cycles for batch jobs, a.k.a. BEER
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

6. G. Lo Presti, Storage for Science at CERN
EOS
Traffic dominated by Physics
xrootd protocol
~Constant through the year
Peak traffic on Aug 2017
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

7. G. Lo Presti, Storage for Science at CERN
Ceph
Started “just” to support the OpenStack infrastructure
3 years operations, 1 PB, 31K IOPS (1.2 GB/s)
Ceph’s “killer application”
Growing role for Physics
Disk buffer for CASTOR (ALICE), in prod for 2 years
NFS over RBD, CephFS
High Performance Computing coming up
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

8. G. Lo Presti, Storage for Science at CERN
CephFS for HPC
Surely HEP is embarrassingly parallel …
Storage not targeting HPC requirements
But several HPC corners do exist:
Beam simulations, accelerator physics, plasma simulations, computation fluid dynamics, QCD …
Require full POSIX, read-after-write consistency, and parallel I/O
~100 HPC nodes accessing ~1 PB of CephFS since mid-2016:
Few bugs found, quite stable
Locking issues requiring development – in touch with the Ceph community
About to migrate our NFS-based services to CephFS
Expect to get more practical experience during 2018
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

9. Ceph Scalability Experiment
“Bigbang” scale tests mutually benefitting CERN & Ceph project
Bigbang I: 30PB, 7200 OSDs, Ceph hammer. Found several osdmap limitations.
Bigbang II: Similar size, Ceph jewel. Scalability limited by OSD-MON traffic.
Bigbang III: 65PB, OSDs, Ceph luminous.
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

10. G. Lo Presti, Storage for Science at CERN
CERNBox
Fast growing service
Add-ons to integrate new ways to interact with your data
Large and growing user base
11K users
1.2 PB and counting, 390M+ files
Broader scientific/research community
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

11. The CERNBox apps ecosystem
From HEP to Scientific and Cloud Computing
Microsoft Office integration
Galleries (pictures as well as ROOT histograms)
Notebooks (SWAN)
…and more to come
Collaborating with ownCloud and the sync&share community
Custom version of the server, contributing to upstream
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

12. Microsoft Office Online at CERN
As part of our site-license with Microsoft, we have deployed an on premise Office Online Service
Similar to Microsoft Office 365
Any CERN user can use & share Office docs
Based on the WOPI REST API
Web-application Open Platform Interface
The API is publicly available
Good candidate for integration in CERNBox
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

13. G. Lo Presti, Storage for Science at CERN
WOPI and CERNBox
A WOPI server has been developed to connect Microsoft Office Online to CERNBox
Leveraging the python ecosystem:
Flask
xrootd-python
JSON Web Toolkit
Open sourced at Github
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

14. Design and Architecture
Stateless server to ease scale-out and operations
An access_token argument holds the request context
Decoupling the WOPI business logic from the storage access
Storage access via xrootd, the native protocol of EOS
Feasible to plug in a different storage, provided it exposes some POSIX-like calls, even via http (e.g. through ownCloud itself)
All other interactions happen via http(s)
Following the REST approach
/wopi/… is the WOPI implementation
/wopi/cbox/… includes some CERNBox hooks, used by the wopiviewer ownCloud app
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

15. G. Lo Presti, Storage for Science at CERN
Experience so far
Development started in late December 2016, for a first pre-prod in March 2017
Rapidly discovered a number of limitations of the Web Apps vs. the Desktop versions…
Beta-testing started at AARNet, Australia, in July 2017
Forked project, now integrated in the main repo
Meanwhile, collected feedback and improved locking and concurrent editing
WOPI incorporates a locking mechanism, but it requires some compensations…
Conflicts handling in place when files concurrently edited from Office Online vs. FUSE/SMB vs. sync client
Full production since September 2017
In the order of daily users so far
No public announcement yet
in production
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

16. G. Lo Presti, Storage for Science at CERN
Future integrations
Another interesting Office platform:
Started prototyping the integration with CERNBox
Possibly run it on top of the FUSE EOS mount
And… what about ?
A commercial solution is being negotiated
We expect it to be integrated in the future
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

17. G. Lo Presti, Storage for Science at CERN
Not only Office…
CERNBox is dominated by Physics data
Service for Web-based ANalysis (SWAN)
Powered by CERNBox
In production for two years
Based on Jupiter and ROOT, and being extended to support analytics (see Andrei’s talk)
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

18. G. Lo Presti, Storage for Science at CERN
Access methods
xrootd: the EOS native protocol
Most stable/performant
GridFTP
HTTP (ownCloud)
FUSE
Still in active development, picking up more activity over the last months
SMB, on top of FUSE
Beta-testing
The Video recording service started to use it
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

19. G. Lo Presti, Storage for Science at CERN
The Broader Community
A few examples of ongoing collaborations with other non-HEP institutes
The CERNBox applications ecosystem makes its adoption appealing also to sites not normally operating xrootd & friends
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

20. G. Lo Presti, Storage for Science at CERN
Joint Research Centre (JRC), Ispra (Italy)
Science Service of the European Commission
A. Burger and P.Soille (JRC)
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

21. EOS at AARNet, Australia
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

22. G. Lo Presti, Storage for Science at CERN
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

23. G. Lo Presti, Storage for Science at CERN
CERNBox in a Box
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

24. G. Lo Presti, Storage for Science at CERN
Education
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

25. 3rd Cloud Services for Synchronisation and Sharing
Novel applications, cloud storage technology, collaborations
Amsterdam, January *** participants *** companies

26. CS3 Workshop, 4th edition http://cs3.cyfronet.pl/
Krakow, January 2018
Submit an abstract!
Register to the Conference!
CS3 group on
HEPiX | Oct 2017
G. Lo Presti, Storage for Science at CERN

27. Thank you for your attention