1. An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format Ken Stone, Center for Lower Atmospheric Studies - University of Colorado, Boulder with help from...

2. Acknowledgements Larry L. Gordley - GATS, Inc., Newport News, Virginia The HALogen Occultation Experiment (HALOE) Team Cheryl Craig, NCAR Joe McInerney, University of Colorado

3. Overview Overview of the toolkit –History –Motivation –Some examples HDF-EOS extensions –Problems & Limitations –Strengths –Lessons learned Summary

4. Motivation Problem: It takes considerable time and effort to make the many multiple comparisons, internal and external, needed during validation phase. Goal: A simple set of tools that allow validation staff to select, line-up, compare and plot profiles and cross sections. Latest twist: HDF-EOS files are now the archive format of choice.

5. History LIMS - Early 80’s –Fortran routine for comparisons and plotting –Fast, low memory usage –Input: multiple files –Output: text summaries and plots HALOE - Early 90’s –Perl Scripts and Standard I/O file format (bsel format) –Fortran routine ported to IDL –New functionality and visualization tools. HIRDLS –HDF-EOS Swath capability –Additional Tools (dot-plots, map differences)

6. Toolkit Overview IDL procedures with Perl scripts for control –Complex scenarios developed both with interactive use (storing results in “journal” file) and stringing scripts together with Unix shell. Analytical Visualization tools –Mapping, cross sections, line plots. –Focus on quality and simplicity rather than complex visualizations. Analysis tools –Coincidence finder, statistical computations. Standardized file format(s) for I/O “Profile” based format hinges on vectors of geo-referenced quantities.

7. Toolkit Detail Prior to HDF-EOS extension the Toolkit used a single file type for input (BSEL format) –Similar in some ways to HDF 4 SDS but limited to sets of 1-d arrays stored as records. User input for any given IDL scenario can be recorded in a journal (text) file for later use. Perl “wrapper” scripts create this file as output. –Keyword substitutions in a “mask” file. –Depending on the mode (mood?) the user is in, they could modify the journal file directly, or change the inputs on the script.

8. Toolkit Script Methodology Perl Script (e.g., 2dlat) IDL procedure or C, or Fortran... Mask File Input File User Input: -f filename -I id, -o outputfile -g -h, etc Postscript BSEL File Perl helper lib Journal file creation from IDL scripts Analysis routines

9. Toolkit Data Flow: Archive and Working Formats SelectionAnalysisDisplay DataSourceDataSource HDF-EOS Bsel Toolkit Extension “Standard” Data Format Archive “working” data format

14. Toolkit Data Model (BSEL Format) BSEL File Event (1)Event (i) Plot Info Range, axis Label, legend Label, Scale, Logflag Event Header Real(8) Long(8) char*60(6) Data (j,k)... Data Header Data order (=2) J (Profile count) K (levels/event)

15. HIRDLS HDF-EOS Swath Metadata Data Fields HDF-EOS Swath File Inst. Name, Ranges, Labels, Scales, Missing Values, Profile count Time Global Attributes Geolocation Fields PressureOzone Mixing Ratio nLevels nTimes Dimensions: nTimes, nLevels etc.

16. HDF-EOS Extension Strengths Fast! Ten times faster than native (Bsel) format Easier to identify field names (and more flexible) Standard way of identifying missing data IDL/Fortran/C I/O interfaces Potentially more tools available

17. HDF-EOS Extension Difficulties Lots of different ways to store data and still meet the standard. –Aura standard format guidelines helps here. HDF-EOS Swath isn’t “packet-like” –You can’t just “cat” files together! IDL/Fortran/C I/O interfaces –Some trouble with stability of certain I/Fs. Point Interface (and raw Vdata) failed from the start due to lack of “id” resources.

18. Lessons Learned Developed general purpose swath reader in IDL –usage: data = get_sw_data(filename, swathname, varname) –Generally reads any variable or attribute. –http://www.eos.ucar.edu/hirdls/get_sw_data.pro Working out the best way to “easily select” variables from swaths is a work-in-progress. To map HDF to the internal data structure a small external data table was needed...

19. Mapping Table HDF-EOS - Internal SwathName VarNameLabelID Min Max Log Scale Unit HIRDLSH2OH2O Mixing Ratio20.010.00 1.0e-06 ppmv HIRDLSNO2 NO2 Mixing Ratio30.010.00 1.0e-09 ppbv HIRDLSO3 O3 Mixing Ratio40.010.00 1.0e-06 ppmv HIRDLSCH4 CH4 Mixing Ratio60.02.00 1.0e-06 ppmv HIRDLSCFCl3 CFCl3 Mixing Ratio1510.00.50 1.0e-09 ppbv CH4L2gpValue CH4 Mixing Ratio60.02.00 1.0e-06 ppmv O3L2gpValue O3 Mixing Ratio40.010.00 1.0e-06 ppmv Internal to toolkit Internal to HDF-EOS Designations for MLS Files

20. Summary HDF-EOS Swath format provides a good storage facility for atmospheric profile data. –Current swath implementation (a la Aura) provides solid groundwork for future sophisticated software development. Interface instability can contribute to major headaches. Ancillary tools to “view and edit” or simply compute additional fields from existing ones could prove quite useful.

21. Future Extend data base table for mapping fields. Develop interface for Grid format. –Already tested in limited cases. Testing, testing, testing… Perhaps incorporate new IDL features –Improved color contouring Develop an easier to use interface for the scripting tools.