22 September 2017, ESA/ESRIN - Frascati Technology for Long-Term Digital Preservation - First workshop Gilbert Barrot, ACRI-ST Gaston Briot, adwaïsEO 22 September 2017, ESA/ESRIN - Frascati

One significant example: EODAS ESA project ACRI-ST and its subsidiary adwaisEO are involved in several projects involving data storage One significant example: EODAS ESA project

EO Data Archiving Service (EODAS) Implements the ESA Master Archive through a dedicated service: Contract Kicked-off in August 2016 Two copies of the data held > 200 Km apart Connection to the ESA EO WAN Checks of the data at all stages from ingestion to delivery Scalability required for growth of contents Management of the data information (MetaData) in fully redundant Databases

EODAS Consortium The industrial consortium is made of ACRI-ST (FR), adwäisEO (LU) and KSAT (NO), with the following distribution of roles: ACRI-ST: Data Archiving (and delivery) Service provider – Prime contractor adwäisEO: data archiving (and delivery) operation provider with its leading edge IT and connected secured infrastructure in Luxembourg KSAT: data knowledge provider and overall ingestion validation

EODAS drivers Strong drivers from H/Div: Requirements limited to “Archival and bulk extraction” of non Copernicus data No operational use of the data Data extracted as it was ingested – no transformation / conversion No tight integration to the PDGS of the missions; not for end-user dissemination Added requirements for Data Ingestion / Evolution / Alignment / Management Added requirements for Service Management and Reporting Out of scope: transcription from historical media, consolidation, transformation, conversion

Tape transfer to Backup Archive Data retrieval 8 LTO7 drives 500 TB cache 11 PB capacity (LTO7) Data ingestion Data evolution Data alignment DMS Tape transfer to Backup Archive Data retrieval

NAS HDD tapes ftp access listings Data ingestion Data evolution Data alignment DMS EODAS central archive Tiering solution (Quantum) Disk cache (XCellis) Tape library (i6000 - ANTF) Weekly tapes transfer or real-time FTP transfer (LM) Data retrieval (Quantum) Tiering solution (Quantum) Disk cache (M440) Tape library (i6000 - ANTF) Tape storage (vault) Archive backup

Product status evolution during the ingestion process Data ingestion / archiving process Ingestion of historical + Live missions (Cryosat-2 and SMOS already operational) data Products/files from source media to scratch storage Packaging (AIP) Copy to archive (disk  tape) Transfer to backup center (tape or FTP) Copy in backup archive + DMS update Dire que l’EODAS est une archive bande Product status evolution during the ingestion process aaaaa

Data packaging process (AIP) One single internal package format to simplify the internal process Distributed data = same format as original

Data integrity Data integrity verified after each process: copy/packaging/transfer zip integrity or MD5 checksum

Ingestion chain Ingestion chain is based on ACRI-ST DPMC system Data and Processing Management Core system  DMS + PMS + orchestrator Most of EODAS operations are performed in parallel on several processing nodes (up to 136 simultaneous processing threads)

DELL (servers) + Quantum (storage/library) + Cisco (network) Infrastructure DELL (servers) + Quantum (storage/library) + Cisco (network)

Quantum tapes tower + Synology NAS (from ESA) Infrastructure Quantum tapes tower + Synology NAS (from ESA)

http://www.eodas.info/

On-line access archives Disk or tapes? Volume Full-tape Hybrid (e.g. HSM) On-line access archives Cold archives EODAS Full-disk Performance

Disk vs tapes: the criteria Disk or tapes? Disk vs tapes: the criteria Bulk or random-access ? Volume ? Short or long-term data archive ? Cost ? Versioning ?

Disk or tapes?

Many thanks for listening. Q&A ?