HDF & HDF-EOS Workshop VIII 2004 October Aurora, CO Bruce Beaumont, Matt Smith, Helen Conover, Sara Graves Subsetting at UAH

Why Subset? Goal: to provide a science data user with only the data they need as quickly as possible. Benefits science data users and data centers: - Reduces analysis time by reducing amount of data - Reduces time for data delivery - Reduces resources (network, personnel, media, etc.) Steps: - Locate spatial / temporal / spectral area of interest - Extract - Re-assemble for distribution/use

What is HSE? HEW Subsetting Engine A new packaging option for the HDF- EOS Subsetter designed for users that want subsetting, but do not want to host the full HEW installation or even the standalone HEW back-end.

What is HEW? HDF-EOS Web-based Subsetter –Prototype software designed to be dataset- independent (HDF-EOS) –Funded by NASA/ESDIS for EOS-DIS –To be used within ECS (EOS-DIS Core System) –Original Front-end/GUI (optional) Uses HTML forms and JavaScript –Original Back-end Needed subset criteria and HDF-EOS data Performed subsetting as a “batch” job

What are HSE’s capabilities? Versions available for HDF-EOS 2 and HDF-EOS 5. Subsets multiple files in one call. Subsets properly-formatted HDF-EOS grid and swath objects. Subsets spatially by latitude/longitude or row/column. Subsets swaths temporally by date/time range. Subsets swaths by full or partial (subscan) lines. Subsets spectrally by HDF-EOS field. Subsamples along any dimension by stride (repeating interval) or by discrete index list. (see chart) Copies all file and field attributes to the output file and updates HDF-EOS “core” metadata when possible.

Stride Subsampling on a Dimension STRIDE = 2 Indexed Subsampling on a Dimension INDEXES = (1, 3, 4, 5, 7) Selected

Existing HEW Back-End Architecture Front end supplies subsetting criteria file Subsetter writes messages to status file Subsetter writes messages to log file Subsetter sends to end-user Many sites do not want to create ODL files or the hassle of dealing with message or log files or .

New HSE Architecture User application code builds subsetting criteria structure Subsetting engine function calls user’s functions for status and log messages No is sent All subsetting functionality is contained within a callable function

SPOT Subsettability “checker” –Displays content/structure of HDF-EOS4 (or HE5) files –Examines files for subsettability by HSE –Simple command-line interface –Stand-alone operation –Available at subset.org

HEW integration with ECS

Product Availability Matrix Product HDF-EOS 2 (HDF4) HDF-EOS 5 (HDF5) HSE SGI  Sun  Linux N/A HSE-5N/A SGI  Sun  Linux HEWBE SGI  Sun  Linux Planned HEW SGI  Sun  Linux N/A SPOT SGI  Sun  Linux

Currently Available/Planned Subsetting Applications HDF-EOS Subsetting Engine (HDF-EOS, HE5)Status Deployments –Complete Systemavailable GHRC –Subsetting Engine Onlyavailable GSFC –SPOT - Subsettability Checkeravailable many –HSE Integrated with ECS Data Order Systemavailable NSIDC, EDC –HSE Integrated with AMSR-E Processingavailable AMSR-E SIPS –Subsetting as a Web Service planned (ECHO) Customized Subsetting –MODIS toolsavailable Science teams –Coarse-grain SSM/I Subsetteravailable GHRC General Purpose Customizable Subsetting –Subsetting Tool using ESMLin work (various)