RSS Data-farm: from local storage to the cloud Julio Carreira ESA-ESRIN, Frascati , Roma (Italy) Storage Solutions for Digital Preservation APARSEN Webinars , 14th April 2014 EOP-GTR Systems

What is RSS RSS architecture and environments have been design and made available by the Research/Service Support and Ground Segment Technology (EOP-GSR) section of ESA Earth Observation Ground Segment Department. Its primary mission is to Provide support to the EO community to exploit EO data and to the researchers and service provider to ease the development of applications to generate value added information Support and promote ground segment harmonization activities through the identification and collection of ground segment technology needs EOP-GSR

Standardised IFs following HMA Guidelines RSS systems Data Farm EO Products, Image Collection UK PAC ESA RSS Storage Standardised IFs following HMA Guidelines MEA Storage OGC GIS Web Service Online Archives Metadata, Auxiliary data G-POD Catalogue OGC Web GIS Services Reference Data Sets DAIL SSE EOP-GSR

RSS storage infrastructure GPOD have 20 storages (NAS/SAN) , the full space is ~ 600 TB ~7 000 000 products storage ~ 350 TB of data Our storages are from 2005 till 2013 Increase of > 50 TB year Use of RAID 5/6 with/without spare disk Monitoring the storage system using OPSView EOP-GSR

The use of data at RSS EOP-GSR

Ancient times (around 2010) The problem All the Storages had different paths to the data sctructure A catalogue was necessary to know were was the data No direct access to a flow of data (ex. A full year of a dataset) EOP-GSR

The answer! The solution was to create a datafarm(distributed file system): POSIX based file system Can be deploy on top of an pre-existent file system Mirroring and replication Load balancing Storage quotas ACLs for user access to data EOP-GSR

RSS datafarm The RSS DataFarm allows much more flexibility than before in accessing data. For example, it is now possible to ingest data directly from the former G-POD dedicated storages into the RSS WebMap Server, with no need to copy data on a local storage. The same applies to SSE, KEO and the other RSS environments. Other benefits brought by the DataFarm are:   Optimized storage space utilization   Easy access control    Easy scalability     At the moment the RSS archive, composed by ENVISAT, ERS and third party missions data has a total volume around 350TB growing by some 50TB/yr. RSS DataFarm move a step towards the Cloud as well. Indeed, this novel RSS infrastructure model is been naturally extended to the Cloud, therefore constituting a robust and scalable basis for providing more and more efficient and flexible support to EO data users in the coming years.  EOP-GSR

Today! All the Storages have a single point of access

In the far future (Dec 2014) EOP-GSR

Main tecnical Problems Main Problems Network Polices Network velocity Security EOP-GSR

Different Elements of Risk Availability Performance Portability Responsiveness Confidentiality Legal Problems Backup Data integrity Disaster recovery Access control Reporting and monitoring Client expectations Responsiveness to users Maintaining corporate values of quality of service High High Medium Medium High Medium High Medium High EOP-GSR

At this moment! At this moment on the development environment, we have 15 Tb on the cloud, and we are: Testing the geo replication of “critical” data, Making stress testing with massive data requests, Monitoring the availability and the performance of the file system, Planning the security and the encryption of the communications, Creating a software for a easy portability between cloud and virtualization providers, Making a cost model of the storage on the cloud comparing to the local storage. EOP-GSR

Conclusions We are happy with our solution of unifed storage using a distributed file system We are testing the portability to the cloud with sucess We are creating policies/rules to mitigate the main risks of having the storage on the cloud EOP-GTR

Thank you EOP-GTR