Presentation is loading. Please wait.

Presentation is loading. Please wait.

HDF and HDF-EOS Workshop November 30, 2006 The Homogenization and Reporting of Groundbased Atmospheric Datasets for the Validation.

Published byRandolf Cross Modified over 8 years ago

Similar presentations

Presentation on theme: "HDF and HDF-EOS Workshop November 30, 2006 The Homogenization and Reporting of Groundbased Atmospheric Datasets for the Validation."— Presentation transcript:

1 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 The Homogenization and Reporting of Groundbased Atmospheric Datasets for the Validation of Earth Observing Satellite Instruments - the Aura and Envisat validation approach using HDF B. R. Bojkov, NASA-GSFC (USA) R. M. Koopman, ESA-ESRIN (I) M. De Maziere, BIRA-IASB (B) I. Boyd, NIWA (NZ)

2 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 Outline Context (and some history) Implementation for Envisat and Aura Future activities

3 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 Why did we do this for atmospheric sciences? (from the perspective of validation)

4 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 Ensuing from European Arctic campaigns in the 1990’s ~35 stations reporting » operational NRT and scientific All stations uses NASA-Ames » “simple” ASCII More than 18 file variations » nomenclature and formulations Clear need of homogenization Near-real-time ozonesondes

5 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 The problem is accentuated: Different measurement networks » LIDAR, MWR, FTIRs, UV-Vis instruments, etc. Different platforms » Groundbased, aircraft, balloon, ship, in-situ, etc. Different agencies » “Dialog” across timezones, reporting facilities, etc.

6 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 So, in 1996/1997… Near impossible to use effectively for research or tools/RDB/web implementations Need to resolve this “file reporting ” problem by investigating/understanding the state of affairs in the field First objective: satellite validation of atmospheric chemistry instruments » up and coming ESA and NASA missions

7 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 COSE COSE - Compilation of atmospheric Observations in support. of Satellite measurements over Europe European Commission (EC) funded project Consortium of 25 groundbased and satellite teams Timeline: 1998-2000

8 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 COSE (2) Approach: » Investigated existing file formats, usage, community needs in consultation with the different stake-holders Recommendations: » Defined fixed file formulation: HDF4 (SDS) » Rigid metadata, including strict guidelines for attributes requirements, variable naming, etc. » Make files truly homogeneous and self explanatory » Final document by Bojkov et al., 2002 (available through AVDC)

9 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 Basic HDF4 SDS file layout Global attributes: 31 attributes » Data source / contact information » Dataset contents / location / … » File generation date / version / caveats / … Attributes for each SDS: 19 attributes » Variable description / notes / caveats » Dependencies / dimensions » Units / SI conversion factor / Valid min. & max. / Fill values » Display attributes (label, axis, …)

10 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 Global attributes Variable attributes

11 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 Variable naming convention Variable name construction: 3 part construction VARIABLE_NAME + VARIABLE_MODE (+ VARIABLE_DESCRIPTOR) Example: » The variable name for O 3 measured by an O 3 sonde and a Brewer spectrometer are: O3_CONCENTRATIONfor the ozonesonde O3_VERTICAL.SOLARfor the Brewer in direct sun mode O3_VERTICAL.ZENITHfor the Brewer in Umkehr mode » The RMS for a FTIR ozone measurement can be expressed as: O3_VERTICAL.ZENITH_UNCERTAINTY.RMS

12 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 Similar conventions for Attribute entries Instrument naming File names …

13 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 First Implementation: ESA Spring 2000: ESA Envisat mission was is it’s final validation planning: » 10 instruments, focus on land, atmosphere and oceans. ESA required: » Fully relational database for calibration and validation » File “QA/QC” on all incoming files » Specifically COSE concept, including HDF 4, to be implemented for atmosphere and ocean instrument validation

14 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 Envisat Cal/Val Implemented in October 2000 » Meets all ESA requirements, including QA and the RDB » 200+ users, intercontinental participation » Backbone of the coordinated validation efforts of 3 atmospheric and 2 ocean instruments » Data sources collected from ship-based to satellites » Extended to other Envisat/ESA validation activities (2004) One issue: heterogeneous reporting -the ozonesonde problem was back!

15 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 2004-present EOS-Aura validation activities The Network for Detection of Atmospheric Composition Change (NDACC - former NDSC)

16 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 Aura Validation Data Center Support the NASA EOS-Aura mission » 4 atmospheric instruments » troposphere to mesosphere measurements » Share validation communities with Envisat AVDC uses the same concept/approach as Envisat » Maintain Envisat files compatibility Operational since October 2004 » ~300 users - many new participants » ~2TB validation data

17 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 AVDC modifications Refined file and metadata requirements through thorough analysis of Envisat Cal/Val in early 2004 » Eliminate common user errors and misconceptions » Added HDF5 implementation (user request due to HDF5 satellite data) Implemented rigid measurement reporting templates » Collaborative effort with NDACC » Measurement specific and defined by expert community » Results in truly homogeneous files Document describing changes by Bojkov et al., 2006 available through AVDC web site

18 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 Future of the AVDC/Envisat implementation AVDC refinements are being implemented into Envisat Cal/Val AVDC/Envisat format being phased into NDACC network » LIDAR, MWR, FTIR completed AVDC concept to be extended to NASA A-train (and most probably NPP wrt validation data) Format is ESA requirement for ESA/EC GMES program » To be extended to land and radar altimetry NASA and ESA to remain synchronized

19 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 Closing remarks This type of work takes time » Began in 1998, and will be ongoing Requires proactive involvement by all parties » Validation teams, instrument PIs, satellite teams, … Requires rigid guidelines - down to the measurement level Once implemented, it is extremely useful for all involved as experienced for the Envisat and Aura missions when validation relies on very different data sources

20 http://avdc.gsfc.nasa.gov/ HDF and HDF-EOS Workshop November 30, 2006 bojan.bojkov@gsfc.nasa.gov http://avdc.gsfc.nasa.gov/

Download ppt "HDF and HDF-EOS Workshop November 30, 2006 The Homogenization and Reporting of Groundbased Atmospheric Datasets for the Validation."

Similar presentations

BADC Workshop 2: BADC Services to Data Suppliers Royal Met. Soc. Conference – 14 September 2005 Ag Stephens et al.

BADC Workshop 2: BADC Services to Data Suppliers Royal Met. Soc. Conference – 14 September 2005 Ag Stephens et al.
6-7 May 2002 CEOS GRID Workshop 1 “CEOS GRID Proposals” roundtable: Introduction ESRIN, 6-7 May 2002 CEOS Workshop on GRID.

6-7 May 2002 CEOS GRID Workshop 1 “CEOS GRID Proposals” roundtable: Introduction ESRIN, 6-7 May 2002 CEOS Workshop on GRID.
M. De Mazière et al., HYMN, May 2007 Status of WP4: FTIR measurements M. De Mazière (BIRA-IASB) K. Petersen, T. Warneke, J. Notholt (U. Bremen)

M. De Mazière et al., HYMN, May 2007 Status of WP4: FTIR measurements M. De Mazière (BIRA-IASB) K. Petersen, T. Warneke, J. Notholt (U. Bremen)
COST ES0602: Towards a European Network on Chemical Weather Forecasting and Information Systems.

COST ES0602: Towards a European Network on Chemical Weather Forecasting and Information Systems.
SMOS SAG, Villafranca November 2-3, 2006 Development of a Global In-Situ Soil Moisture Network: A SMOS Project Contribution P.J. van Oevelen.

SMOS SAG, Villafranca November 2-3, 2006 Development of a Global In-Situ Soil Moisture Network: A SMOS Project Contribution P.J. van Oevelen.
ORM 9, Geneva, May 14 -16, 2014 Evolution of Measurement Capabilities + cooperating networks.

ORM 9, Geneva, May , 2014 Evolution of Measurement Capabilities + cooperating networks.
A. Minchella (RSAC c/o ESA-ESRIN) on behalf of ESA EOPI Team ESA Advanced Training Course in Land Remote Sensing, 2 July 2013, Athens, Greece Access to.

A. Minchella (RSAC c/o ESA-ESRIN) on behalf of ESA EOPI Team ESA Advanced Training Course in Land Remote Sensing, 2 July 2013, Athens, Greece Access to.
CAPACITY Final Presentation, 2 June 2005, ESTEC Operational Atmospheric Chemistry Monitoring Missions (“CAPACITY”) Final Presentation ESA-ESTEC, Noordwijk.

CAPACITY Final Presentation, 2 June 2005, ESTEC Operational Atmospheric Chemistry Monitoring Missions (“CAPACITY”) Final Presentation ESA-ESTEC, Noordwijk.
M. De Mazière et al. NDACC Steering Committee, 15-19/10/2012 Progress in the first year of NORS De Mazière Martine, Hocke Klemens, Richter Andreas, Godin-Beekmann.

M. De Mazière et al. NDACC Steering Committee, 15-19/10/2012 Progress in the first year of NORS De Mazière Martine, Hocke Klemens, Richter Andreas, Godin-Beekmann.
The HDF Group July 8, 2014HDF 2014 ESIP Summer Meeting HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann The.

The HDF Group July 8, 2014HDF 2014 ESIP Summer Meeting HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann The.
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Mike Folks, The HDF Group Ruth Duerr, NSIDC 1.

Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Mike Folks, The HDF Group Ruth Duerr, NSIDC 1.
05 December, 2002HDF & HDF-EOS Workshop VI1 SEEDS Standards Process Richard Ullman SEEDS Standards Formulation Team Lead

05 December, 2002HDF & HDF-EOS Workshop VI1 SEEDS Standards Process Richard Ullman SEEDS Standards Formulation Team Lead
NASA/ESA Interoperability Efforts CEOS Subgroup - CINTEX Alexandria, Sept 12, 2002 Ananth Rao Yonsook Enloe SGT, Inc.

NASA/ESA Interoperability Efforts CEOS Subgroup - CINTEX Alexandria, Sept 12, 2002 Ananth Rao Yonsook Enloe SGT, Inc.
Coordination Group for Meteorological Satellites - CGMS Climate Monitoring Architecture: Status and way forward Presented to CGMS-41 plenary session.

Coordination Group for Meteorological Satellites - CGMS Climate Monitoring Architecture: Status and way forward Presented to CGMS-41 plenary session.
DM_PPT_NP_v01 SESIP_0715_AJ HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann Gerd Heber, John Readey, Joel Plutchak The HDF Group HDF.

DM_PPT_NP_v01 SESIP_0715_AJ HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann Gerd Heber, John Readey, Joel Plutchak The HDF Group HDF.
M. De Mazière NDACC SC 2011, Nov. 11, 2011 NORS Demonstration Network Of ground-based Remote Sensing observations in support of the GMES Atmospheric Service.

M. De Mazière NDACC SC 2011, Nov. 11, 2011 NORS Demonstration Network Of ground-based Remote Sensing observations in support of the GMES Atmospheric Service.
Sentinel-3 Validation Team (S3VT) Meeting ESA/ESRIN, Frascati, Italy, 26-29 th November 2013 General Presentation Template for S3VT meeting Project name.

Sentinel-3 Validation Team (S3VT) Meeting ESA/ESRIN, Frascati, Italy, th November 2013 General Presentation Template for S3VT meeting Project name.
MODIS Land and HDF-EOS HDF-EOS Workshop Presentation September 20, 2000 Robert Wolfe NASA GSFC Code 922, Raytheon ITSS MODIS Land Science Team Support.

MODIS Land and HDF-EOS HDF-EOS Workshop Presentation September 20, 2000 Robert Wolfe NASA GSFC Code 922, Raytheon ITSS MODIS Land Science Team Support.
A Metadata Based Approach For Supporting Subsetting Queries Over Parallel HDF5 Datasets Vignesh Santhanagopalan Graduate Student Department Of CSE.

A Metadata Based Approach For Supporting Subsetting Queries Over Parallel HDF5 Datasets Vignesh Santhanagopalan Graduate Student Department Of CSE.
Establishing a UK OCO/GOSAT expert group Paul Palmer, Hartmut Bösch, Paul Monks, Peter Bernath et al White paper Expert Group: Why? What? Who? OCO Project.

Establishing a UK OCO/GOSAT expert group Paul Palmer, Hartmut Bösch, Paul Monks, Peter Bernath et al White paper Expert Group: Why? What? Who? OCO Project.

Similar presentations

Ads by Google