AVHRR 1 km Data Set Consolidation Consolidating 30 years of EO data hosted by multiple data holders 28 th LTDP WG Meeting Frascati (ESA/ESRIN) 04–05 November 2014

Feasibility Study Inter-agency activity - Consolidate and make available AVHRR 1 km (LAC) data held by data centers in Europe and Canada Feasibility Study:

User Requirements Capture Survey distributed to selected users to capture user scenarios and user requirements for Data and metadata – format, structure, content Preservation and usability - data records, information & tools Discovery and retrieval Exploitation tools & infreastructure

User Requirements Capture Respondents Brockmann Consult, Germany Norwegian Meteorological Institute, Norway University of Berne, Switzerland Eidgenössische Technische Hochschule Zurich, Switzerland Helmholtz-Zentrum Geesthacht, Germany Remote Sensing Centre of Institute of Geodesy and Cartography, Warsaw, Poland DLR, Germany ESA CCI

User Requirements – Summary 1 Data Level L0 to L2 for user access Standard formats: NetCDF, HRPT, HDF, ASCII, ESA Beam compatible Consistent across satellite generations, quality flags Metadata Product metadata: OGC-EOP Following CF (Climate and Forecast) standard Keep inside in the hdf5/netcdf4 Collection metadata: ISO and INSPIRE compliant Controlled volcabularies

User Requirements – Summary 2 Preservation and usability Data records Preserve HRPT raw & auxiliary data Reprocess as calibration data or method changes Keep some (tbd.) generations of the previous data sets Complete datasets Consistent versioning / numbering scheme Referenced by DOI Associated knowledge Information: NOAA docs, provenance, quality - available to user Tools: processing chain, visualization, and analysis HW & OS independent, maintain libraries, open source preferred Test suite

User Requirements – Summary 3 Discovery Centralized online portal with advanced browse capability Searchable by: satellite name, time frame, spatial domain, orbit number, product version Standardized interfaces, machine-harvestable catalog (OAI-PMH, OGC CSW) Retrieval Same data format and access interfaces/protocols if data held in distributed achives Automated download using data retrieval script Request for data streams instead of download - OpENDAP preferred Time series retrieval of stacked products for a single pixel (e.g. ASCII) Direct download of bulk data Data ordering not favored Distribution option physical media

User Requirements – Summary 4 Exploitation tools & infrastructure Tools: processing chain, visualization, and analysis Exploration and exploitation capabilities provided by ESA desirable Possibility to integrate other data sources beyond EO May not be necessary if OPeNDAP access is provided

Summary There is a demand for a complete, consistent, calibrated time series Good agreement on data format: NetCDF, CF Diverse requirements for processing level Central discovery & access Procedures, provenance, etc. must be documented and available

Questions? Tyler Christensen DLR - German Remote Sensing Data Center Information Technology

Backup Slides

Analysis DLR and Dundee Inventories Number of NOAA AVHRR 1 km scenes (passes) available at DLR and at Dundee Satellite Receiving Station. Duplicate scenes have been removed for the analysis. The purple line indicates the approximate number of scenes required to ensure an adequate spatial and temporal coverage for the AOI of the TIMELINE project (6 passes x 24 hrs x 365 days). Year Number of scenes (orbits) AOI of the DLR TIMELINE project

Details of User Requirements

User Requirements - 1 User Scenarios Users will process raw data into their own interpreted products: land cover, snow extent, water body extent, NDVI, fires and burned area, etc. Some users want higher-level AVHRR products (e.g. snow, surface temperature) that they can compare to their own model outputs Consolidated data set would be used to extend existing analyses in space and/or time, sometimes using existing processing chains—so data format will be very important

User Requirements - 2 General Requirements for Consolidated Dataset All levels requested, from L0 all the way to L2 interpreted products o Some users need data as raw as possible—no atmospheric correction, individual swaths not stitched, not reprojected, etc. o Some users need fully processed data—mosaics, georeferenced, time-aggregated (daily) Must include quality flags & pixel-center location grids Must be internally consistent (i.e. corrections applied between different satellites for a smooth time series), but should be fully documented Ancillary data: ozone, water vapor, aerosol properties if available

User Requirements - 3 Data - format, structure, content Format Most data users requested NetCDF, one said HDF would also be OK. A few also required the raw and L1b data in original HRPT format. Perhaps also ASCII, especially for a time-series of an individual pixel. Readable with ESA BEAM toolbox. Structure Descriptive, searchable file names—sensor, dates, mission, etc. NetCDF files should have metadata headers—some requested that metadata follow the Climate & Forecast (CF) convention. Content Separate scientific data sets (SDS) for reflectances, brightness temperatures, acquisition angles, scanline times, and NWP data used for atmospheric correction. Requested in metadata header: long_names, grid information, standard_name

User Requirements - 4 Metadata (collection, product) - format, structure, content Format CF (Climate and Forecast standard) metadata inside NetCDF file Collection metadata: ISO and INSPIRE compliant Product metadata: OGC-EOP Keep inside in the hdf5/netcdf4 Structure NetCDF Discovery Attribute Convention Content Follow CF-conventions for netCDF data Utilize controlled vocabularies (C&F standard names, GCMD Science Keywords) Sensor, platform, temporal and spatial coverage, calibration information, applied processing chain, ancillary data and parameters used

User Requirements - 5 Preservation and Usability Needs Reprocess dataset whenever the calibration data or the calibration method changes Complete datasets Any changes to the dataset need to be carefully documented and the information needs to be made available to the user – via the metadata or an easily and permanently accessible document. Datasets should be referenced by DOI A consistent versioning / numbering scheme is required Some (tbd.) generations of the previous data sets should be kept

User Requirements - 6 Data Record Needs Original HRPT raw & auxiliary data must be kept internally, to allow for future reprocessing as improved algorithms, new data formats, or different intercalibrations emerge. Metadata and browse images should also be kept. L1b data & L2 products– possibly several generations of re-processed data Must be kept in standardized data formats

User Requirements - 7 Additional Information and Tools to be Preserved NOAA documentation (e.g. POES user guide) should be available and linked to data. Data processing documentation should be kept: procedures for aggregation, temporal averaging, inter-calibration, etc. Information on provenance should be archived: platforms, sensor characteristics of each platform, calibration updates etc. If possible all software required to generate each version of the product should be maintained: calibration coefficients, NWP data, tle information, etc. Quality assessments, validation and inter-comparison reports Data use policies Data format descriptions

User Requirements - 8 Needs for Software and Tools Software content to preserve: processing chain software, visualization tools, data readers for HRPT and other formats, and BEAM tools (i.e. data readers, visualization, processing, analysis) Also keep copies of libraries that the software uses, and/or compile the software against static libraries so it would be usable even if newer library versions are not compatible. Try not to depend on specific hardware or OS; use standard software platforms and compilers. Need a test suite. Open source preferred.

User Requirements - 9 Discovery Needs Centralized online portal with quick looks and advanced browse capability, searchable by: satellite name, time frame, spatial domain, orbit number, product version A catalogue interface with machine-harvestable metadata would be useful. One user states that OAI-PMH is preferred over OGC CSW due to the simplicity in the interface. Should be no access difference between historic and current missions Data policies should be aligned

User Requirements - 9 Data Retrieval Needs 1 Automated download using a data retrieval script should be possible Data sets can be hosted in separate archives, however, the same data format and access interfaces/protocols (FTP, WCS, other?) should be used by all providers Request for data streams which users could attach a processing chain to, instead of downloading a local copy of the dataset OpENDAP access is preferred. Direct download of bulk data using shell scripts, query specifying file name pattern, etc. Preference for selectable times/locations and 'mission-at-once'.

User Requirements - 10 Data Retrieval Needs 2 Standardized formats and interfaces/transfer protocols in case data need to be transferred to third party processing infrastructure Retrieve a time series of stacked products for a single pixel. Data ordering is acceptable but not preferred. Distribution on physical media (as opposed to online download) is also requested.

User Requirements - 11 Exploitation Infrastructure Needs (hosted processing) Most users don’t think this is necessary, especially if OPeNDAP access is provided. One user requested exploration and exploitation capabilities provided by ESA, with a possibility to integrate with other data sources beyond EO. Would require a collaborative effort to build virtual data exploitation platforms. Exploration and exploitation capabilities provided by ESA desirable

User Requirements - 12 Needs for Current (vs. historic) AVHRR Data New data are as important as the historical archive. Most users update their products regularly, and would like to have daily updates as new data become available. Some users do not need updated data, if they are doing retrospective analysis. New data should have exactly the same format as archived data.

User Requirements - 13 Time Series Needs Most users will download the data to a local copy, and then perform their own time series analysis. Request to receive a time series for a single pixel as a text file. Remote data processing and/or OpENDAP access would reduce the need for users to download large amounts of data for the whole time series.