1. HDF5 at the European XFEL
Presenter:
Dr. Gero Flucke
Control and Analysis Software Group, European XFEL GmbH
Main author:
Djelloul Boukhelef
IT and Data Management Group, European XFEL GmbH
HDF5 ICALEPCS 2017
Barcelona, October 8th, 2017

2. Outline Introduction: The European XFEL
Data acquisition - integrated into Karabo control system
Data organisation: file naming and data structure
HDF5 hardware compression studies

3. European XFEL: the most brilliant X-ray Free Electron Laser
Karabo
DOOCS Tine EPICS
European XFEL: the most brilliant X-ray Free Electron Laser
*) “User assisted commissioning” since Sept. 14, 2017
Linear electron accelerator:
2.1 km, starts at DESY
Magnetic structures (undulators):
Stimulate coherent photon emission
Photon optics
Transport photon beam to six instruments
XFEL Instruments
FXE Femtosecond X-ray Experiments *)
SCS Spectroscopy & Coherent Scattering
SPB/SFX Single particles, Clusters and Biomolecules *)
SQS Small Quantum Systems
HED High Energy Density Science
MID Materials Imaging and Dynamics

4. European XFEL: Photons come in trains with short pulses
Pattern:
Trains with 10 Hz
“Storage unit” for data acquisition
Up to 2700 pulses per train
pulse duration < 100 fs
220 ns spacing (4.5 MHz)
=> train length 600 µs
On average: up to 27 kHz of pulses

5. Karabo: Experiment Control, Data Acquisition and (Fast-Feedback) Analysis
drive hardware and complex experiments
monitor variables & trigger alarms
Karabo Framework:
Tight integration of applications for
Experiment control
Data Acquisition
Data Management
Data Analysis
DAQ
data readout
online processing
quality monitoring (vetoing)
Sample Injector
allow some control & show hardware status
Accelerator
Undulator
Beam Transport
show online data whilst running DM
storage of experiment & control data
data access, authentication authorization etc.
setup computation & show scientific results DM DA
Control
DAQ DA
processing pipelines
distributed and GPU computing
specific algorithms (e.g. reconstruction)

6. Karabo Communication and Data Flow
Equipment
Control
e.g. motor, pump, valve, sensor
e.g. 2D-detectors (AGIPD, LPD, DSSC)
DAQ
Equipment
Gui interface(s)
DAQ
Equipment
e.g. commercial camera DM DA
Control
DAQ
Message Broker
Gui
Server
and other services, e.g. calibration manager, run configurator
Karabo Framework:
Broker based communication
Point-to-point shortcut possible
Data Pipelines
Fast, not too big data
Analysis Pipelines
Data Storage
Node
e.g. storage of data from runs
Composite
(Middle Layer)
Device
e.g. complex detector motion control
Analysis
Node
e.g. image proessing

7. Data acquisition and management
Data acquisition, handling and processing
Collect and store data
Consolidating fast/slow, small/large data streams
Data, tagged with train-id, is received by software devices at 10Hz (train rate)
Data curation
Creation, storage, archiving
Reliable retrieval of data in future
Data policy
Data retention/custody
Access by users to scientific data
Devices = datasources
Shared memory
Aggregators (events)
Data curation is the management of data throughout its lifecycle, from creation and initial storage to the time when it is archived for posterity or becomes obsolete and is deleted. The main purpose of data curation is to ensure that data is reliably retrievable for future research purposes or reuse.
Metadata catalogue
User

8. Karabo: Near-Realtime Data Analysis using non-Karabo tool
See presentation: „Data Analysis Support in Karabo at European XFEL“ by Hans Fangohr
in parallel session on “Data Analytics”, Tuesday, Oct. 10, h

9. Data Sources and DAQ system
Data Aggregators
Storage
Devices continuously push data to the PC layer
Image data: 2D detectors  big & fast  require multi channel transfer from single detector (non-Karabo protocol)
2D or Pulse data: e.g. cameras, digitizers  fast  can aggregate several data sources (Karabo pipeline protocol)
Control data: e.g. sensors, motors  slow  can aggregate many data sources (Karabo protocol)

10. How data arrives at the PC layer
Control data (“slow”)
These are parameters of Karabo devices (e.g. motor positions)
PC layer aggregator uses the standard Karabo mechanism to register for any update of the parameter value
Available through the Karabo broker service
Instrument data
Data acquired and processed by electronic devices (“big and fast”)
Available as a binary object described by XFEL Train Data Format (XTDF)
sent out to PC layer using application level Train Transfer Protocol (TTP) over UDP network protocol
Data acquired and sent out to PC layer by software device controlling the hardware via dedicated DAQ network (“fast”)
Available as Hash structure
Sent out to PC layer using Karabo pipeline
FEM Ctrl device
header
images
PC layer device
descriptor
detector
Digitizer device
raw
processed

11. File naming convention
Raw data repository structure follows Metadata Catalogue model
File naming convention
type-rXXXX.h or type-rXXXX-infix-sXXXXX.h5
r – run, s – sequence number, X – digit, h5 – hdf5 extension
type – can be raw, cal, proc (to be defined)
Infix – based on the aggregator/PC layer node
“-” – separator
General concept exists for splitting data across many files
Certain data groups can be stored in separate files
Train data from different files can be correlated without additional catalogue
File name is “computed” based on several configuration parameters (T0, Nc, N, infix) and is function of T (train id)
Facility
Proposal
Type
Run
FacilityCycle
XFEL
p001234
raw
r0002
201701
Instrument
FXE
Stub file :
raw-r0001.h5
LPD detector data
raw-r0001-lpd-s00000.h5
raw-r0001-lpd-s00001.h5
raw-r0001-lpd-s00002.h5 …

12. Run data files Data files store datasets Stub file Stub file
Each file may contain data from one or multiple data sources or properties
XFEL component naming convention is used
Stub file
Unique per run
Point to actual files that store datasets
Indicate formula parameters (e.g. HDF5 attributes) to calculate the actual set of files that store a given dataset
Dataset in different file groups might be indexed using different parameters
/detlab_det_lpd/2d/lpd1
/detlab_det_lpd/fem/1/
/detlab_det_alas1/vac/1
/detlab_det_alas1/motor/3
0001.h5
Stub file
r0001-agg1-s0000.h5
AGG1
r0001-agg1-s0001.h5
r0001-agg1-s0002.h5
r0001-agg1-s0003.h5
r0001-agg0-s0000.h5
AGG0
r0001-agg0-s0001.h5
r0001-lpd-s0000.h5
r0001-lpd-s0001.h5
r0001-lpd-s0002.h5
r0001-lpd-s0003.h5
r0001-lpd-s0004.h5
r0001-lpd-s0005.h5
PCL0
PCL1
PCL2
PCL3

13. Control data sources Only selection of control data is stored
One value stored for each train
Timestamp is preferably assigned at hardware level, otherwise as soon as control system sees the data
If data does not change, previous value is duplicated, but timestamp kept
Errors can be identified by special timestamp value i.e. zero
/control/index/trainId
/control/detlab_det_lpd/fem/1/femVoltage0/timestamp
/control/detlab_det_lpd/fem/1/femVoltage0/value
/control/detlab_det_lpd/fem/1/powerCardTemp0/timestamp
/control/detlab_det_lpd/fem/1/powerCardTemp0/value
150234
21.022
150235
150236
150237
21.023
150238
21.024

14. Instrument data characteristics
Train data
exists for each train
Variable train data
like train data but may not exist for all trains
Pulse data
received per train
pulses give extra dimension
Variable pulse data
like pulse data but some values can be omitted (e.g. vetoed)
Run data
data valid for entire run

15. Key to high-performance data writing: Column oriented HDF5 Structure
Record oriented
Data comes as hierarchical key-value container
(“Hash” in Karabo)
One write / data update / data source
Column oriented Hash vectorization
Internal buffering of N updates
One write / multiple data update / data source

16. FPGA-Accelerated Data Compression
Storage space getting an issue:
Current 2D detectors: 4 Mpixel, up to 500 frames per train => 80 GB/s (4 bytes per pixel)
Compression CPU intense
=> Study FPGA-acceleration
IBM Power8 with FPGA-based accelerator
GenWQE/PCIe GZIP Accelerator
@p8.desy.de
© Copyright IBM 2016

17. Test data files eXtended Tagged Container - (.xtc) format
Raw data files from LCLS detectors:
eXtended Tagged Container - (.xtc) format
“Data files” / diffraction pattern - (.cxi) format
HDF5, NeXus-inspired and ~compatible
Data available from
Sequential crystallography: idb22
beamline
Not-so-weakly scattering: idb30
beamline
Weakly scattering – TODO
LCLS: SLAC Linac Coherent Light Source
CXIDB: Coherent X-ray Imaging Data Bank

18. Compression with FPGA
Same executable run seamlessly with software or hardware-accelerated compression (no recompile is needed)
Enable FPGA compression by setting environment variables:
ZLIB_ACCELERATOR=GENWQE LD_PRELOAD=/usr/lib64/genwqe/libz.so.1 /path_to/prog
Comparison criteria between SW and HW compression:
space saving = 1 – 1/comp_ratio
comp_ratio = uncompressed_size / compressed_size
data compression rate (speed)
I/O from/to 3 alternatives: disk / memory / null
Disk:

19. Compression rates with FPGA
Sequential crystallography, idb22, ~175GB of 16 TB
Space saving [41-51%], depends on run #
Data rate (single thread):
Disk: 0.95 GB/s RAM: 1.05 GB/s RAM/null O: 1.12 GB/s
Not-so-weakly scattering, idb30, ~210 GB of 4 TB
Space saving [32-42%], depends on run #
Disk: 0.85 GB/s RAM: 0.89 GB/s RAM/null O: 1.00 GB/s

20. Comparison against software
Speed / data rate
FPGA: ~1GB/s. ~100x faster than software:
id22: [8.6, 8.8] MB/s
id30: [8.7, 9.9] MB/s
Storage saved:
FPGA: [32, 51]% raw data storage save
Software could save more, [41, 57]%:
id22: [48, 57]% => ~[12,17]% (relative) higher space saving
id30: [41,50]% => ~[17,27]% (relative) higher space saving
Other software implementations (gzip, custom ‘gzip’) can be 6-7% faster
+ on different machines (exflpclXXnY), up to [19, 21]% faster.

21. Conclusions and future work
European XFEL tightly integrates data acquisition into Karabo control system
Different data sources: “fast & big”, “fast”, and “slow/control” data
Column-oriented HDF5 file structure
Database-like tables correspond to datasets
Data buffering/bulk writing is faster than single writes
FPGA-accelerated compression on IBM power8
API is well integrated with Karabo and HDF5 libraries
Storage saving: Raw data: ~32-51%
FPGA compression rates “close to” 1 GB/s
FPGA speed is faster than software by x[93, 128]
Next step
Use real data from last user operation
Disk I/O setup using GPFS
Test new generation GenWQE - CAPI

22. Acknowledgements Federico Montesino Pouzols (ex-member of ITDM)
Control and Analysis Software group (CAS) for Karabo
IBM for providing test hardware and support

23. Shared memory DAQ and DM components
PC layer: data acquisition layer
Set of data aggregator devices running on cluster of high-performance computers
Collect, monitor and store experiment data
Disseminate data to online analysis pipeline (fast-feedback)
Run management service
Coordinates PC layer activities related to data
Metadata catalogue (MDC)
Organize and keep track of experiment data in a homogeneous and consistent way
Data retrieval service
Common data interface with analysis algorithms (online, offline)
Devices = datasources
Shared memory
Aggregators (events)

24. Data series types Pulse data Variable pulse data Deterministic push
Digitizer
2D pixel detector
Pulse data
e.g. Pulse digitizer, BPM, …
Variable pulse data
eg. 2D pixel detector images
Deterministic push
Train data
eg. 2D pixel detector train info
Variable train data
eg. Slow camera
Train control data
Event driven
eg. Sensors, actuators, motors, …
Run data
eg. 2D detector configuration
Data integrated over multiple trains
Train Event data/time series

25. Correlation of instrument and control data
Sequence number can be calculated assuming statically configured distribution of train data over multiple channels
r0001-lpd-s0000.h5 T
Train Id T0
First train Id within the run Nc
Number of data acquisition channels. N
Number of train blocks per file c
Channel id. Channels are ordered. i
Record id for train based data within a table {i=0,…,N-1} m
File sequence number within the channel s
File sequence number within the run
r0001-lpd-s0001.h5
r0001-lpd-s0002.h5
r0001-lpd-s0003.h5
r0001-lpd-s0004.h5
For the consecutive train ids:
𝑐 𝑇 = 𝑇− 𝑇 0 𝑚𝑜𝑑 𝑁 𝑐
𝑚 𝑇 = 𝑇− 𝑇 0 𝑁 𝑐 ×𝑁
𝑖 𝑇 = 𝑇− 𝑇 0 𝑚𝑜𝑑 𝑁 𝑐 ×𝑁 𝑁 𝑐
 𝑠 𝑇 =𝑚 𝑇, 𝑁 𝑐 ,𝑁 × 𝑁 𝑐 + 𝑐 𝑇, 𝑁 𝑐
Can be generalized for other train patterns
Example:
T0 = 0, Nc=3, N=2
Where is train T = 10?
c = 1, m = 1, i = 1, s = 4
Train T=10 is stored in file:
r0001-lpd-s0004.h5
at record i=1
r0001-lpd-s0005.h5
r0001-lpd-s0006.h5
T0 = 0, Nc=3, N=2, infix=lpd
These parameters are stored
as group attribute in the stub file

26. Instrument data Optimize for size when necessary
train_data (header, trailer, det. spec.)
Optimize for size when necessary
Create indexes to navigate between different groups of data
Record Idx
Train Id
linkId
pulseCount
detectorDataBlock
TrainDataBlock …
234 10 3
Q;slwwd% 1
250 4
A%%ad8*8__
@#$8uuq
pulse_data (descriptors)
Record Idx
Idx
Variable pulse data
Idx train data
Train Id
Pulse Id
cellId
status -1
234 1
150 2
843 34 3
250 14 4 45 5 55 6 91 32
var_pulse_data (Images)
Record Idx
Idx pulse data
Train Id
Pulse Id
Image 1
234
150 2
843 4
250 45 3 6 91

27. Chapter break

28. How to edit the title slide
Upper area (title): Title of your talk, max. 2 rows of the defined size (28 pt) Lower area (subtitle): your name and affiliation, location, date, max. 4 rows of the defined size (22 pt) 1 1 2 2

29. How to use slide layouts1
New slide: Click on the text “New Slide” (not on the icon) to select one of the templates. New layout: Click on the button “Layout” for changing the layout of an existing slide. Reset: Use the button “Reset” to re-apply the selected layout. 2 3 1 2 3

30. How to edit text 1
Bullets: New text slides are showing orange bullet points. For writing copy text: go to the beginning of the text slide and press backspace to delete the bullet point. Indent: The first level shows an orange bullet point. Select “Increase list level” to go to the next levels. “Decrease list level” will bring you back to the first level. 1 2 2

31. How to edit the header 1
Info: The header shows the title of the presentation, your name/function and the date. The text can be edited in the slide master. 1