NDE XML Tailoring Service HDF and HDF-EOS Workshop XIII November 4, 2009 Tom Feroli, Peter MacHarrie

Goal Reduce the labor effort to create user-specific tailored products (I.e., image, gridded, map reprojected, and other specific file formats, etc.) –Objective Develop (InputFormat + Output Format) software applications instead of (Input Format * Output Format) software applications. –Approach Develop a service that provides routine data manipulation (i.e., sub-setting, sub-sampling, filtering, aggregation) and places output into a well-defined intermediate format. Create tailoring applications that accept a well-defined intermediate format and configurable output options to create a specific end-user product.

Data Selection Driver Uses Binary Format (Subsample, Subset, Aggregate, Filter, Scale/Offset) netCDF4 netCDF3 HDF5+ netCDF4 ReadsWrites Modular design, accepts drivers to accommodate additional input and output file formats (e.g., HDF-EOS5, netCDF3, etc.) XML

Enterprise “Measure” Enterprise Measure is a logical construct –It is the concatenation of all related arrays in all files in an “enterprise” File 1 – Array1 (Scan = 12, Sample=96, Channel=22) File 2 – Array1 (Scan =12, Sample = 96, Channel=22) Enterprise Measure1 (Scan=24, Sample=96, Channel=22) Enterprise Measure reference “Enterprise Dimensions” A file, is a storage unit, for a partition, of the Enterprise Measure –The partition is a range of values of one or more Enterprise Dimensions File 1 – Scan Dimension File 2 – Scan Dimensions 13-24

Enterprise “Dimension” Enterprise Dimension is a unique logical mapping to dimensions within all data files. –In VIIRS, Image and Moderate product profiles, the dimension AcrossTrack is defined as 6400 and 3200 –Enterprise dimension are named uniquely for each AcrossTrack-I 6400 AcrossTrack-M 3200

Enterprise Data Cube Data Cube is a structure of one or more Enterprise Measures. Within a data cube: –Measures share the same dimensions –Measure names are unique –Measures may have different data types In VIIRS SDRs all arrays are named Radiance –Enterprise measure uniquely name each array Example –Radiance_I1 -> Product I1->Radiance Array –Radiance_I2 -> Product I2->Radiance Array

Enterprise Data Structures Tailoring Service XML References Enterprise Data Structures

Measures (additional information) Named Subset –Array: AntennaTemperature(12, 96, 22) –Measure: AntennaTemperature, references entire AntennaTemperature array –Measure: AntennaTemp-Channel_1, references array AntennaTemperature, with sub-sample specification of Start=(1,1,1), Stride=(1,1,1), Count=(12,96,1) Named BitMask –Array: QF1_VIIRSCMIP(768, 3200) –Measure: CloudConfidencePixel Has Integer DataType References array QF1_VIIRSCMIP starting at bit 2 for 2 bits Scale and Offset –Measure defined as float, references an integer array and its scale and offset are applied Measures can be referenced in the or elements of the tailoring XML. – –(i.e., CloudConfidencePixel < 2)