1 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp SIPAD New Generation: Dominique Heulet CNES 18, Avenue E.Belin Toulouse Cedex 9 France A multi-field system for accessing scientific data and added-value services

2 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp Introduction  SIPAD-NG –What it is: »a complete operational system for preserving and accessing data, »implements proven off-the-shelf products and a sophisticated data model, »meets the requirements of a data center with a long-term mission, »a centralized system with its own catalogue –Why it has been developed: »CNES wants to offer a complete generic software to be used as a basis for building data centers

3 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp SIPAD-NG: follow-on of the SIPAD software  The SIPAD system –Developed in the 1990s for the CDPP »CDPP: Space Plasma Physics Data Centre »SIPAD enables users to retrieve data and metadata through a web server –SIPAD software designed to be used by any scientific discipline: »Plasma Physics (CDPP & CASSINI projects), »Oceanography (MERCATOR project), »Atmospheric Chemistry & Astronomy –Advantages of SIPAD: »SIPAD implements all the basic functions of a data access system »Open-ended Data Model based on a Data Dictionary allowing the control of delivered metadata –Disadvantages: »Limitations to implement new functions (e.g. interoperability), »Web user interface not flexible enough (not easily customizable)

4 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp The Data Model: graphs  For each project, definition of the Data Model  First goal of the Data Model: organize the catalogue in order to make data search & selection by navigation easy –Definition of a data graph, a browse graph and a document graph »Data graph:storage objects (archived data – e.g. a ‘tar’ file –) data objects (data files seen by end users) data sets (groups of data objects – e.g. instrument level 1 data –) collections (groups of data sets or collections) »Browse graph: browse products, browse sets, browse collections »Document graph: documents, document collections –Definition of links between these graphs  Web server pages are automatically generated from the Data Model

5 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp The Data Model: graphs Project Document collection Data collection Browse collection Data set Data Object Browse set Browse product Document Data set Data object Browse product Browse product Data collection Data set Browse collection Browse set Document collection Storage object Storage object Storage object

6 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp The Data Model: attributes  Elements of the graphs can be described by attributes  Generic attributes »Size of a data object »Number of data objects in a data set  Specific attributes according to scientific discipline »Plasma Physics: start time & stop time of an observation »Earth Observation: geographical coordinates  Use of attributes –To present information associated to data –To select data by criteria »The web server provides selection pages for entering criteria values

7 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp The Data Model: attributes Project Mission documents Mission DocumentData set Data object Experiment documents Data Object Id Start time Stop time Version etc … Title Author Name Description Measured parameters etc … Selection criteria Dataset Id Level etc … Browse collection Browse product Browse set

8 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp SIPAD & SIPAD-NG common functions  Data search & selection »Navigation in the Data Model, »Search on criteria, »Selection of product browse  User management »Access rights, »Quotas  Order management »Data extraction from an archive, »Media production, »Network transfer, »User workspace

9 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp New functions implemented in SIPAD-NG  Integration of Added Value Services (AVSs) –SIPAD-NG allows to: »associate AVSs to Data Model elements in order to propose them to users, »call AVSs by giving them parameters, »manage access rights and user quotas –SIPAD-NG implements a specific interface for: »Batch AVSs: process data which have already been ordered and deliver the result to the user (e.g. coordinate transformation), »Interactive AVSs: process data available in the user workspace (e.g. graphic representation of data from several data sets)  Interoperability –SIPAD-NG provides interfaces to send requests to or to receive requests from another Data Centre »Web Services & SOAP protocol

10 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp Adaptability  Adaptation of the web server to the project needs –SIPAD-NG web server easily customizable to project needs –SIPAD-NG web server can be replaced by a project specific web server  Implementation in Data Centres located inside CNES –SIPAD-NG compliant with CNES Computer Centre services: »SUN SOLARIS operating system, »interfaces with CNES archiving and media production facilities –SIPAD-NG compliant with CNES security rules  Implementation in Data Centres located outside CNES –SIPAD-NG can run on a single machine or its components can be distributed over several machines –SIPAD-NG can run on both SOLARIS and LINUX operating systems

11 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp Architectural design SIPAD-NG kernel Catalogue Basic services Administration & Supervision Ingestion User management Catalogue consultation Order Management User space Administration Web Server Administrator Local Data Centre services Client applications On line archiveNear line archiveMedia production Web Server Added Value Service Remote Data Centre

12 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp Client applications provided by SIPAD-NG  The SIPAD-NG software system is composed of: –a kernel: »the six basic services, »the Administration Web Server, »the data base –several client applications: »complementary to the basic services (e.g. automatic ingestion), »main client application: User Web Server  Characteristics of the SIPAD-NG User Web Server: »compliant with the six basic services interfaces, »easily customizable, »A project can replace it by its own web server and only reuse the communication layer with the basic services

13 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp Time selection Implementation examples Internet machine Geographic selection Other SIPAD-NG services & Administration Web Server STAF SEM Oracle server CDPP catalogue Solaris server SIPAD-NG User Web Server & Catalogue Consultation service CDPP at CNES Computer Centre ICARE Data Centre File server On-line data SIPAD-NG basic services SIPAD-NG user web server ICARE catalogue Linux server

14 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp

15 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp

16 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp Technology User Web Server & Administration Web Server Basic services - JAVA - Apache Tomcat - Struts → MVC2 - Axis → Web Services - JAVA - RMI → Inter-machines communication - XML → Ingestion & Basic services interfaces ORACLE 9i ‘Object-Relational Mapping’ layer ‘Business’ layer ‘Application Logic’ layer ‘Client’ layer ‘Database’ layer

17 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp Conclusion  Schedule »2003: requirements, specification »February 2004: beginning of development »July 2005: first release »December 2005:final release  SIPAD-NG contractor: CS SI  SIPAD-NG users –those already using SIPAD »CDPP, MERCATOR, CASSINI »Migration of the catalogues from SIPAD to SIPAD-NG –Future potential users »ICARE: Data Centre for Clouds & Aerosols »SSALTO: Altimetry & Orbitography