Systems and Communications1 CERN Openlab – 10/11 December 2013 ESRF: IT Challenges

Systems and Communications2 ESRF Beamlines X-ray standing waves anomalous scattering surface diffraction gamma spectroscopy dultra-fast scattering/ high pressure materials science circular polarisation microbeam protein crystallography inelastic scattering medical nuclear spectroscopy high energy absorption spectroscopy powder diffraction spectroscopy on ultra diluted samplessi protein crystallographye

Systems and Communications3 What is a Beamline?

Systems and Communications4 IT General architecture Central storage Infrastructure servers Network Tape Backup Compute Clusters Detectors

Systems and Communications5 How much do we store each day?

Systems and Communications6 Challenges ESRF Phase II (higher photon flux) starts in 2015 Already, two Eiger detectors arrive at ESRF in Fall 2014  4 Mpixel x 3 bytes = 12MB/image  750Hz = 750*12MB = 9GB/s  Local compression => 1GB/s minimum  10% duty cycle => 9TB/day Limiting factors:  Human resources – data sorting and exploitation  Data Analysis – online  Data Analysis – offline  Speed of the Central storage

Systems and Communications7 Ongoing works LBS only for Linux-based detectors DSS time consuming but MB/s per unit HSN High-Speed Network 40Gbps 10Gbps Central storage Infrastructure servers Network Tape Backup Compute Clusters Detectors Local Buffer Storage (LBS) Dedicated Storage Server (DSS)

Systems and Communications8 Requirements for one Beamline ONLINE DATA ANALYSIS (CPU+GPU) EXPORT STORAGE BACKUP ARCHIVE BUFFER STORAGE BUFFER STORAGE LONG TERM STORAGE LONG TERM STORAGE OFFLINE DATA ANALYSIS (CPU+GPU) Etc.

Systems and Communications9 Overall requirements Typically 6 – 10 Beamlines simultaneously Typically 6 high-speed detectors per Beamline Typically 2 detectors operating simultaneously per Beamline Peak bandwidth per Beamline = 1 – 10 Gbytes/sec

Systems and Communications10 Overall architecture, with one Beamline BUFFER STORAGE Windows DETECTOR PC > 1 GB/ REALTIME ONLINE DATA ANALYSIS (CPU+GPU) VISUALISATION LONG TERM STORAGE + OFFLINE DATA ANALYSIS (CPU+GPU) BACKUP ARCHIVE USER EXPORT GREEN = DEDICATED TO A BEAMLINE Buffer Storage boxes may be either dedicated to a detector or shared by several detectors Linux DETECTOR PC Linux DETECTOR PC BUFFER STORAGE BUFFER STORAGE >1 GB/s BLUE = SHARED BETWEEN BEAMLINES

Systems and Communications11 Other challenges Data Confidentiality

Systems and Communications12 Questions?