1. RSS Data-farm: from local storage to the cloud
Julio Carreira
ESA-ESRIN, Frascati , Roma (Italy)
Storage Solutions for Digital Preservation
APARSEN Webinars , 14th April 2014
EOP-GTR Systems

2. What is RSS
RSS architecture and environments have been design and made available by the Research/Service Support and Ground Segment Technology (EOP-GSR) section of ESA Earth Observation Ground Segment Department.
Its primary mission is to
Provide support to the EO community to exploit EO data and to the researchers and service provider to ease the development of applications to generate value added information
Support and promote ground segment harmonization activities through the identification and collection of ground segment technology needs
EOP-GSR

3. Standardised IFs following HMA Guidelines
RSS systems
Data Farm
EO Products, Image Collection
UK PAC
ESA RSS Storage
Standardised IFs following HMA Guidelines
MEA Storage
OGC GIS Web Service
Online Archives
Metadata, Auxiliary data
G-POD Catalogue
OGC Web GIS Services
Reference Data Sets
DAIL SSE
EOP-GSR

4. RSS storage infrastructure
GPOD have 20 storages (NAS/SAN) , the full space is ~ 600 TB
~ products storage
~ 350 TB of data
Our storages are from 2005 till 2013
Increase of > 50 TB year
Use of RAID 5/6 with/without spare disk
Monitoring the storage system using OPSView
EOP-GSR

5. The use of data at RSS
EOP-GSR

6. Ancient times (around 2010) The problem
All the Storages had different paths to the data sctructure
A catalogue was necessary to know were was the data
No direct access to a flow of data (ex. A full year of a dataset)
EOP-GSR

7. The answer!
The solution was to create a datafarm(distributed file system):
POSIX based file system
Can be deploy on top of an pre-existent file system
Mirroring and replication
Load balancing
Storage quotas
ACLs for user access to data
EOP-GSR

8. RSS datafarm
The RSS DataFarm allows much more flexibility than before in accessing data. For example, it is now possible to ingest data directly from the former G-POD dedicated storages into the RSS WebMap Server, with no need to copy data on a local storage. The same applies to SSE, KEO and the other RSS environments.
Other benefits brought by the DataFarm are:
  Optimized storage space utilization
  Easy access control
   Easy scalability    
At the moment the RSS archive, composed by ENVISAT, ERS and third party missions data has a total volume around 350TB growing by some 50TB/yr.
RSS DataFarm move a step towards the Cloud as well. Indeed, this novel RSS infrastructure model is been naturally extended to the Cloud, therefore constituting a robust and scalable basis for providing more and more efficient and flexible support to EO data users in the coming years. 
EOP-GSR

9. Today!
All the Storages have a single point of access

10. In the far future (Dec 2014)
EOP-GSR

11. Main tecnical Problems
Main Problems
Network Polices
Network velocity
Security
EOP-GSR

12. Different Elements of Risk
Availability
Performance
Portability
Responsiveness
Confidentiality
Legal Problems
Backup
Data integrity
Disaster recovery
Access control
Reporting and monitoring
Client expectations
Responsiveness to users
Maintaining corporate values of quality of service
High
High
Medium
Medium
High
Medium
High
Medium
High
EOP-GSR

13. At this moment!
At this moment on the development environment, we have 15 Tb on the cloud, and we are:
Testing the geo replication of “critical” data,
Making stress testing with massive data requests,
Monitoring the availability and the performance of the file system,
Planning the security and the encryption of the communications,
Creating a software for a easy portability between cloud and virtualization providers,
Making a cost model of the storage on the cloud comparing to the local storage.
EOP-GSR

14. Conclusions
We are happy with our solution of unifed storage using a distributed file system
We are testing the portability to the cloud with sucess
We are creating policies/rules to mitigate the main risks of having the storage on the cloud
EOP-GTR

15. Thank you
EOP-GTR