Research Data Archive - technology Infrastructure and services for reliable long term storage of scientific and cultural data

Long time storage versus Archive positioning of the archive service A. Project storage – Long term storage Personal access Primary and cold experiment data Sharing within the local group or community Simple upload with accepted protocols B Good scientific practice store Like A. above and additionally. Guaranteed storage for at least 10 years Reliable storage with integrity assurance Minimal set of descriptive metadata and PID assignment Open Access C Archive conforming - Repository Like B. above and additionally Forever or as long as funding sustains Data curation Extended metadata search capabilities, OAI-PMH Interface Conforming with standards i.e. OAIS, DSA, ISO16363 06/10/2016 BWDA - Archive infrastructure for scientific data

Portfolio Further details and information at https://www.rda.kit.edu  Data protection hardware based checksums between server memory and all storage media (T10 PI)  trusted encrypted network transfers per file checksum comparison when data is read (MD5) data copies at distinct physical locations user verifiable checksum (of published data) - EUDAT - Data storage support for large volumes ( > 10 PetaByte ) and large file sizes ( > 1 TB ) on-line directory meta-data and recently used content (through disk cache) storage classes: single, dual (on divergent technologies), triple/shelf Data access easy access with common protocols and clients (sftp, gridftp) using federated authentication and local user db management account chaining, access to data remains after change of home organisation supporting rules for good scientific practice (retention of data for at least 10 year) simple publication of data - in development (bwDIM) - Miscellaneous sharing of data - in development - scheduled deletion - in development - features for distinct user groups, communities and projects that require long time storage for specific applications Further details and information at https://www.rda.kit.edu 06/10/2016 BWDA - Archive infrastructure for scientific data

HPSS Hardware test systems Core/Mover DB/Cache MD storage 1 core DB servers MD storage 2 cache storage 1 admin nodes GridKa data movers data movers cache storage 2 cache storage Test Nodes 06/10/2016 BWDA - Archive infrastructure for scientific data

HPSS as archive platform scalable to billions of objects It’s the DataBase stupid! HPC users like the throughput scalability end to end data protection (IEEE T10 dif) user definable checksums tape optimised reads POSIX access (incl. acls) OpenStack Swift tape container detailed API core servers standby production front-end nodes (sftp, gridftp) FUSE mount hpss mover nodes SAN 2 x disk cache Data Centre A IBM TS1140 drives Data Centre B Oracle 10kD drives 06/10/2016 BWDA - Archive infrastructure for scientific data

scalability of FUSE interface varying number of parallel streams 1,2,4,6,8,12,16 parallel streams per file (chunks of a single file by opening multiple network sockets) 16 concurrent connections (i.e. 16 files are transferred at the same time varying number of concurrent connections 2,4,8,16,32 concurrent connections (2,4,8,16,32 files are transferred at the same time by opening multiple control connections) 8 parallel streams 2 4 6 8 12 16 parallel streams 2 4 8 16 32 concurrent connections 06/10/2016 BWDA - Archive infrastructure for scientific data

Archive Usage  Pilot from KIT and HLRS Data protection hardware based checksums between server memory and all storage media (T10 PI)  trusted encrypted network transfers per file checksum comparison when data is read (MD5) data copies at distinct physical locations user verifiable checksum (of published data) - in development (EUDAT) - Data storage support for large volumes ( > 10 PetaByte ) and large file sizes ( > 100 GigaBytes ) on-line directory meta-data and recently used content (through disk cache) storage classes: single, dual (on different technology) Data access easy access with common protocols and clients using federated authentication of bwIDM in BW supporting rules for good scientific practice (retention of data for at least 10 year) simple publication of data - in development (bwDIM) - published data is automatically set read-only (immutable data) - in development - Miscellaneous sharing of data - in development - scheduled deletion - in development - features for distinct user groups, communities and projects that require long time storage for specific applications *) Pilot from KIT and HLRS Support for enduser and projects (e.g. RADAR, EUDAT) As of 1.10.2016, over 2 PB data stored, ca. 50 users i.c. projects 06/10/2016 BWDA - Archive infrastructure for scientific data

Go Live 24.10 Production test run (Version 1) 22.11 Production start

Next steps IDPs must set “Archive” entitlement others must contact us per person per institute per group per ... it all comes down to money Formalities are here to stay Responsibilities and costs must be cleared beforehand Storage service contracts (with KIT procurement, financial, legal dept.) Security concept descriptions Planning Version 2 features with ‘bwDIM’ publication light AAI features Migration of other existing data: GridKa, LSDF, ForHLR 06/10/2016 BWDA - Archive infrastructure for scientific data