1. WP18, High-speed data recording Krzysztof Wrona, European XFEL
CRISP
WP18, High-speed data recording
Krzysztof Wrona, European XFEL
CERN, 23 Sep 2013

2. High-speed Data Recording
Objectives:
“High-speed recording of data to permanent storage and archive”
“Optimized and secure access to data using standard protocols”
Partners:
DESY, ESRF, ESS, European XFEL, GANIL, ILL, Univ. Cambridge

3. General status Milestone MS21 originally planned for month 24
Architecture document as the base for implementation of the prototype system
Preparation for the document ongoing.
Current estimation: delayed by 1-2 months

4. Proposed architecture
Proposed architecture consists of multiple layers
Actual implementations may vary between facilities due to specific requirements and restrictions
Additional layers can be added
Some layers may be skipped

5. Proposed architecture
1 Detectors or dedicated electronic devices
2 Real time data processing, data aggregation from different sources and formatting
3 Local buffer as a temporary data storage
4 Central storage system
5 Data archive
6 Data and experiment monitoring
7 Data pre-processing
8 Data analysis, data export services
9 Online shared scratch disk space
10 Offline shared scratch disk space
Proposed architecture
5. Archive
1. Detectors/Electronics
2. PC layer
3. Online data cache
4. Central storage
6.Monitoring cluster
7.Online Computing cluster
8. Offline Computing cluster A B C D H G F E J I
9. Shared disk space
10. Shared disk space K
Requires further discussion between WP18 partners

6. Proposed architecture
1 Detectors or dedicated electronic devices
2 Real time data processing, data aggregation from different sources and formatting
3 Local buffer as a temporary data storage
4 Central storage system
5 Data archive
6 Data and experiment monitoring
7 Data pre-processing
8 Data analysis
9 Online shared scratch disk space
10 Offline shared scratch disk space
Proposed architecture
5. Archive
1. Detectors/Electronics
2. PC layer
3. Online data cache
4. Central storage
6.Monitoring cluster
7.Online Computing cluster
8. Offline Computing cluster A B C D H G F E J I
9. Shared disk space
10. Shared disk space K

7. Proposed architecture
Data is sent from detectors (1) and received on the PC layer (2)
Data is sent over high speed network, i.e. using multiple 10GE links.
Received data are then aggregated, processed and formatted on the PC layer
At this stage data processing may alter the data content before it becomes data is persistent
Proposed architecture
5. Archive
1. Detectors/Electronics
2. PC layer
3. Online data cache
4. Central storage
6.Monitoring cluster
7.Online Computing cluster
8. Offline Computing cluster A B C D H G F E J I
9. Shared disk space
10. Shared disk space K

8. 10GE data transfer
10GE network transfer for data acquired by detectors
UDP and TCP protocols have been successfully implemented and tested for high throughput parallel data streams
http protocol has been investigated for commercial devices dumping data to files and where interoperability between different operating system is needed

9. Online data processing
Design of data processing layer at European XFEL
Data receivers
Read train data and store it in local memory buffer.
UDP for big train data (2D detectors)
TCP for slow and small data
Processing pipeline
Users’ algorithms
Perform data monitoring, rejection, and analysis.
Aggregator, formatter, writer
Filter data and merge results,
Format data into HDF5 files,
Send files to data cache.
Data receivers
Data aggregator & formatter
Scheduler
Shared Memory
Synchronization
Analysis
1:N
Check
1:10
Rejection
1:1
Monitoring
Slow data
Fast data
Network writer
Multicast
HDF5 files
Primary process
Secondary process
Online data Cache & Scientific Computing

10. Online data processing
PC layer node software is divided into a primary process and one or more secondary processes.
Primary process
Performs critical tasks such as data receiving, storing, and scheduling.
Requires super-user mode
Secondary processes
Run users’ algorithms (pipeline)
Run at normal user-mode
Data exchange is done through inter-process shared memory
Scheduler
Monitors tasks and data status
Coordinates threads activities
Data receivers
Data aggregator & formatter
Scheduler
Shared Memory
Synchronization
Analysis
1:N
Check
1:10
Rejection
1:1
Monitoring
Slow data
Fast data
Network writer
Multicast?
HDF5 files
Primary process
Secondary process
Online Data Cache & Scientific Computing

11. Proposed architecture
B. Store data in the local buffer
From the PC layer (2) data is sent to the online data cache (3).
If the PC layer (2) is not implemented data is sent directly from detectors (1) to the data cache (3)
Capacity of the online cache should be sufficient to allow for data storage up to several days
Proposed architecture
5. Archive
1. Detectors/Electronics
2. PC layer
3. Online data cache
4. Central storage
6.Monitoring cluster
7.Online Computing cluster
8. Offline Computing cluster A B C D H G F E J I
9. Shared disk space
10. Shared disk space K

12. Online data cache
Sending single channel formatted data through 10GE interface using TCP
Storage performance: cached vs. direct IO
Tests with two types of storage systems:
14 x 900GB 10Krpm , SAS, 2 x 6Gbps, RAID6
12 x 3TB Krpm, NL SAS, 2 x 6Gbps, RAID6
Results
Achieved data rate per channel:
1.1GB/s and 0.97GB/s, resp.
Direct IO improves performance and stability

13. Proposed architecture
C. Send data required for online data monitoring
Results of real time data processing are sent to the monitoring cluster
Monitoring system prepares data for visualization (i.e. histograms) and provides slow feedback for experimentalists
Proposed architecture
5. Archive
1. Detectors/Electronics
2. PC layer
3. Online data cache
4. Central storage
6.Monitoring cluster
7.Online Computing cluster
8. Offline Computing cluster A B C D H G F E J I
9. Shared disk space
10. Shared disk space K

14. Proposed architecture
D. Send data for online processing
Multicast can be used to send data from the PC layer to the online data cache and processing cluster
Subset of data is sent to the online computing cluster where additional algorithms can be run without strict control on execution time.
Data received on the online cluster can be stored on the shared disk space and inspected with standard tools used by the experimentalists
Proposed architecture
5. Archive
1. Detectors/Electronics
2. PC layer
3. Online data cache
4. Central storage
6.Monitoring cluster
7.Online Computing cluster
8. Offline Computing cluster A B C D H G F E J I
9. Shared disk space
10. Shared disk space K

15. Proposed architecture
E. Data are pre-processed before storing in the central storage system (4).
Additional filtering, data reduction and merging may be performed before registering dataset and storing in the central storage system
Results of the pre-processing determines if data is useful for further analysis or it should be discarded
Proposed architecture
5. Archive
1. Detectors/Electronics
2. PC layer
3. Online data cache
4. Central storage
6.Monitoring cluster
7.Online Computing cluster
8. Offline Computing cluster A B C D H G F E J I
9. Shared disk space
10. Shared disk space K

16. Proposed architecture
F. Output data from the pre-processing are stored in the online data cache
This data can be stored in addition or instead of the original raw data
Proposed architecture
5. Archive
1. Detectors/Electronics
2. PC layer
3. Online data cache
4. Central storage
6.Monitoring cluster
7.Online Computing cluster
8. Offline Computing cluster A B C D H G F E J I
9. Shared disk space
10. Shared disk space K

17. Proposed architecture
G. Send data to central data storage
Entire dataset or selected good quality data are stored in the central system
At this point all datasets are registered in the metadata catalogue
Proposed architecture
5. Archive
1. Detectors/Electronics
2. PC layer
3. Online data cache
4. Central storage
6.Monitoring cluster
7.Online Computing cluster
8. Offline Computing cluster A B C D H G F E J I
9. Shared disk space
10. Shared disk space K

18. Data storage systems At DESY and XFEL At SKA
Testing dCache system as the candidate for the central data storage
dCache presents data distributed over multiple servers as a single namespace
Data is accessible using pNFS4.1 protocol
dCache implements access control list according to the NFS4 specification
dCache manages tape data archiving/restoring
At SKA
Initial tests of the Lustre distributed filesystem

19. Proposed architecture
H. Archive data
Data received in the central storage system needs to be secured for long term storage
Implementation of the long term data archive is recommended
Proposed architecture
5. Archive
1. Detectors/Electronics
2. PC layer
3. Online data cache
4. Central storage
6.Monitoring cluster
7.Online Computing cluster
8. Offline Computing cluster A B C D H G F E J I
9. Shared disk space
10. Shared disk space K

20. Proposed architecture
I. Read data required for offline analysis
Access to data needs to be protected according to the adopted data access policy
Data should be readable using standard protocols i.e. pNFS4.1
Offline analysis performed on cluster of computing nodes
Proposed architecture
5. Archive
1. Detectors/Electronics
2. PC layer
3. Online data cache
4. Central storage
6.Monitoring cluster
7.Online Computing cluster
8. Offline Computing cluster A B C D H G F E J I
9. Shared disk space
10. Shared disk space K

21. Proposed architecture
J. Write processed data to the central data storage
Results of user data analysis are stored in the central storage system, archived and kept according to the data policy adopted at facility
Proposed architecture
5. Archive
1. Detectors/Electronics
2. PC layer
3. Online data cache
4. Central storage
6.Monitoring cluster
7.Online Computing cluster
8. Offline Computing cluster A B C D H G F E J I
9. Shared disk space
10. Shared disk space K

22. Proposed architecture
K. Scratch shared disk space for data processing
Fast, short term data storage
May be required for accessing intermediate data within application or between execution runs
Proposed architecture
5. Archive
1. Detectors/Electronics
2. PC layer
3. Online data cache
4. Central storage
6.Monitoring cluster
7.Online Computing cluster
8. Offline Computing cluster A B C D H G F E J I
9. Shared disk space
10. Shared disk space K

23. Data storage systems
Performed initial tests of the Fraunhofer filesystem.
Plan to use it as a scratch space for fast access required by demanding analysis applications
Fhgfs setup at DESY

24. Summary Preliminary architecture design exists
Require further feedback from WP18 participants
Architecture document needs to be prepared soon
Implementation of the prototype system will follow at XFEL, DESY and possibly at SKA.