1. WP18, High-speed data recording
CRISP
WP18, High-speed data recording
Krzysztof Wrona
Hamburg, 8th December 2011

2. K.Wrona, Harmonzation meeting
WP18 objective
Provide solutions for:
high-speed recording of data to permanent storage and archive
optimised and secured access to data using standard protocols
08/12/2011
K.Wrona, Harmonzation meeting

3. K.Wrona, Harmonzation meeting
Participants
RIs
ESRFUP, EuroFEL, European XFEL, ESS, ILL2020, SKA, SPIRAL2
Institutes
ESRF, DESY, European XFEL, ESS, ILL,UOXF.DB, GANIL
08/12/2011
K.Wrona, Harmonzation meeting

4. K.Wrona, Harmonzation meeting
Motivations
Extremely high data rates
Exceeding tens of GB/s per data source
Especially when triggers cannot be applied
Cost effective recording of data to storage systems and archives becomes an increasingly complex and challenging task
experiment setup is changed frequently
Aggregation of data from multiple sources
Ensuring integrity of data through the full acquisition chain
How to store, archive and access data
What are the implications from data protection models
08/12/2011
K.Wrona, Harmonzation meeting

5. K.Wrona, Harmonzation meeting
Task 1
Assembling requirements and use cases for high-speed data recording to storage systems and data archives.
Description of use cases
Identification of common critical issues
Requirement document by June 2012
Reviewing available technologies, selecting tools, and investigating their usability for defined use cases
Survey on hardware and software technologies
Exchange of experience
Mapping concrete solutions to identified use cases
08/12/2011
K.Wrona, Harmonzation meeting

6. K.Wrona, Harmonzation meeting
Task 2
Collecting requirements for data protection and understanding their implications for high-speed data recording and data access.
Description of use cases
Definition of data protection model
Integration with authentication system (WP16)
Requirement document by Sep 2012
Evaluating existing data-protection schemes including simple and advanced access-control models (i.e. NFSv4 and POSIX ACLs) and mapping them according to the defined requirements
08/12/2011
K.Wrona, Harmonzation meeting

7. K.Wrona, Harmonzation meeting
Task 3
Defining and selecting use case applications requiring high throughput data access.
Selection of several flagship applications
Definition of requirements and benchmark criteria
Evaluating the usability of standard access protocols
i.e. NFS4.1 (pNFS)
integration issues with storage, network and computing infrastructures
Defining data-access architecture
Parallel access, caching strategies, etc
Architecture document by Sep 2013
08/12/2011
K.Wrona, Harmonzation meeting

8. K.Wrona, Harmonzation meeting
Task 4
Implementing the prototype system for the selected data-protection and data-access models according to the architecture design.
Complete test bed system will be implemented at selected facility(ies)
Other institutes may implement system partially
Re-evaluating and refining the system architecture.
Improving implementation.
Deploying the prototype system and demonstrating its functionality.
08/12/2011
K.Wrona, Harmonzation meeting

9. Relations to other projects/WPs
Data archiving
Compliance with PanData Scientific data policy
Data access requires strong authentication service
CRISP/WP16, PanDataODI/WP3
proposed AAI needs to provide appropriate level of security
Data protection scheme should be consistent with equivalent solution for metadata
CRISP/WP17
Data access
Is moonshot project helpful here?
08/12/2011
K.Wrona, Harmonzation meeting