DataGrid France 12 Feb – WP9 – n° 1 WP9 Earth Observation Applications

DataGrid France 12 Feb – WP9 – n° 2 Global Ozone (GOME) Data Processing and Validation  The DataGrid testbed provides a collaborative processing environment for 3 geographically distributed EO sites (Holland, France, Italy)  Level-1 data (raw satellite measurements) are analysed to retrieve the physical quantities : Ozone Profiles (Level-2 data)  Ozone Profiles contain measurements of OZONE within a vertical column of atmosphere at a given lat/lon location above the Earth’s surface  Two different Grid processing techniques are being investigated  OPERA (Holland) - Tightly coupled - based on MPI  NOPREGO (Italy) - Loosely coupled - based on Neural Networks  Selected Profiles are validated by comparing them with ground-based LIDAR measurements in a coincident area and time

DataGrid France 12 Feb – WP9 – n° 3 ESA(IT) – KNMI(NL) Processing of raw GOME data to ozone profiles. 2 alternative algorithms ~28000 profiles/day IPSL(FR) Validate some of the GOME ozone profiles (~10 6 /y) Coincident in space and time with Ground-Based measurements Visualization & Analyze EDG EO challenge: Processing / validation of 1y of GOME data LIDAR data (7 stations, 2.5MB per month) DataGrid environment Level 2 (example of 1 day total O 3 ) Level 1 Raw satellite data from the GOME instrument (~75 GB - ~5000 orbits/y)

DataGrid France 12 Feb – WP9 – n° 4 EO Use Case File Numbers DatasetNumber of filesSize Level 14,72415 Mb Level 2 NNO (ESA) 9,448,00010 kb Level 2 Opera (KNMI) 9,448,00012 kb Validation Lidar (IPSL) Mb Total: 18,900,736 files 267 Gb (just one part of a 5-year dataset) 1 Year of GOME data :

DataGrid France 12 Feb – WP9 – n° 5 GOME Ozone Profile Processing  Select geographic area and time to be investigated  Retrieve list of available Level-1 data files from the Product Catalogue  Transfer files to the GRID  Perform Level-1 Processing, retrieve Ozone Profiles  Visualize results

DataGrid France 12 Feb – WP9 – n° 6 GOME Ozone Profile Validation Goals of the DataGrid application to point out : the capability to validate satellite data with all ground based data available in an easy way:  Comparison of ozone profiles provided by satellite with lidar data in differents locations and times (see the web portal)  Statistical comparison and analysis in order to improve algorithms. OZONE LAYER 50 km 10 km ERS/GOME satellite Lidar at the Haute Provence Observatory (5/02/2003)

DataGrid France 12 Feb – WP9 – n° 7 Validation Processing Sequence Level 2 Catalogue Lidar data catalogue Queries and data information retrieval from the Lidar metadata catalogue GRID Computing Element Storage Elements with Lidar data Queries and data information retrieval from the Gome Level 2 orbit or pixel metadata catalogues When completed comparison between lidar and satellite ozone profiles Satellite data validation Lidar site Level 2 Catalogue (5/02/2003) GRID Portal Storage Elements with Gome L2 data Submission of the Job in the GRID

DataGrid France 12 Feb – WP9 – n° 8

DataGrid France 12 Feb – WP9 – n° 9

DataGrid France 12 Feb – WP9 – n° 10 Web Portal EO Product Catalogue EDG Storage Element EDG User Interface EDG Resource Broker EDG Computing Element EO Replica Catalogue Processing Sequence EO Grid Engine EO Product Archive 1. Search Level-1 catalogue 2. Retrieve Level-2 products 3. Level-2 Products already registered in RC? 8. Submit jobs to process Level-1 data 7. Register Level-1 data 11. Register level-2 data 9. Process Level-1 data 10. Transfer Level-2 data to SE 12. Return new Level-2 products Yes? 4. Return available Level-2 products No? 5. Perform GRID processing on-the-fly 6. Transfer Level-1 data from Archive to the Grid

DataGrid France 12 Feb – WP9 – n° 11 The EO WebMap Portal  The scientist interacts with the GOME information system using the WebMap portal  This allows the user to choose a geographic area and time interval to be investigated  The WebMap portal retrieves and displays the available GOME products on the map  It does this by communicating with several background services :  Web Map Server – provides map images  Web Service Broker – provides GRID web services  EO Grid Engine – integrates the Data Grid services  Products which were recently requested will be already available  Other products will be processed “on the fly” using the GRID resources

DataGrid France 12 Feb – WP9 – n° 12 Summary  The Web Portal Integrates several components of the EO Infrastructure  The EO GRID Engine provides a standard way for Web Clients to request GRID operations  The Web Portal and the EO Grid Engine allow the complexity of the underlying components to be hidden from the user  This Framework can be adapted to provide new tools for the satellite end-users