HDF5 Tools Update Peter Cao - The HDF Group November 6, 2007 This report is based upon work supported in part by a Cooperative Agreement with NASA under NASA NNX06AC83A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration.

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 1 Outline Overview of current tools New tools in HDF5 HDF-Java 2.4 release

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 2 Readers h5dump, h5diff, h5ls new tools: h5check, h5stat HDF5 Command Line Tools Writers h5repack, h5repart, h5import, h5jam/h5unjam new tool: h5copy, h5mkgrp Converters h4toh5, h5toh4, gif2h5, h52gif

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 3 Where to start Tools located at bin/ directory 1. Use “-h” option for help 2. Online help to THG help desk

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 4 4 h5dump Dumps file content to stdout in ASCII text XML Binary

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 5 h5dump -H SDS.h5 HDF5 "SDS.h5" { GROUP "/" { GROUP "Floats" { DATASET "FloatArray" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } } DATASET "IntArray" { DATATYPE H5T_STD_I32LE DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) } }

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 6 h5dump - d /Floats/FloatArray SDS.h5 HDF5 "SDS.h5" { DATASET "/Floats/FloatArray" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } DATA { (0,0): 0.01, 0.02, 0.03, (1,0): 0.1, 0.2, 0.3, (2,0): 1, 2, 3, (3,0): 10, 20, 30 }

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 7 h5dump -x SDS.h5

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 8 h5dump Binary Output -b F, --binary=[MEMORY|FILE|LE|BE] a b c d e f e f a b c d h5dump -d /IntArray -o out_le.bin -b LE SDS.h5

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 9 h5diff Between Show Differences two objects or two files

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 10 h5diff SDS.h5 SDS2.h5 Dataset: and 5 differences found

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 11 Report differences Dataset: and positionIntArrayIntArraydifference [ 0 0 ]01010 [ 1 0 ] [ 2 0 ] [ 3 0 ] [ 4 0 ] differences found h5diff SDS.h5 SDS2.h5 -r /IntArray

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 12 h5repack Copies a file to a new file with Compression filter Data layout Other user options

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 13 h5repack: new filters -f { GZIP|SZIP|SHUF|FLET|NBIT|SOFF|NONE} For example h5repack -i SDS.h5 -o out.h5 -f GZIP=6

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 14 h5repack: data layout -l {CHUNK| COMPA| CONTI} For example h5repack -i SDS.h5 -o out.h5 -l CHUNK=2x3

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 15 h5repack: latest version of file format h5repack optionsFile size in bytes Original file3,167,264 --latest2,267,346 --latest –compact=92,162,254 --latest –compact=9 –ssize=50:dtype299,882 The testing file was created with HDF The root group of the file contains 100 groups. Each group at the root contains 9 sub groups. Each sub group contains one compound dataset. The compound dataset has 40 members of 32 bit integer

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 16 h5repart Repartitions a file into a family of files For example h5repart -m 200m int16kx16k.h5 part200m%d.h5 977 MB 200 MB part200m0.h5 200 MB part200m1.h5 200 MB part200m2.h5 200 MB part200m3.h5 177 MB part200m4.h5

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 17 h5import PATH work/First-set INPUT-CLASS TEXTFP RANK 3 DIMENSION-SIZES OUTPUT-CLASS FP OUTPUT-SIZE 64 OUTPUT-ARCHITECTURE IEEE OUTPUT-BYTE-ORDER LE CHUNKED-DIMENSION-SIZES MAXIMUM-DIMENSIONS GROUP "/" { GROUP "work" { DATASET "First-set" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H5S_UNLIMITED ) } DATA { (0,0,0): 1.01, 1.02, 1.03, 1.04, (0,1,0): 1.11, 1.12, 1.13, 1.14, (1,0,0): 1.21, 1.22, 1.23, 1.24, (1,1,0): 1.31, 1.32, 1.33, 1.34, (2,0,0): 1.41, 1.42, 1.43, 1.44, … } }} Imports binary/ASCII data into an HDF5 file h5import infile -c config_file [ infile -c config_file2...] -outfile outfile For example h5import float5x4x2.txt -c First_set.conf -o First_set.h5

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 18 h5jam/h5unjam h5jam h5jam -u test_ub.txt -i test_ub.h5 h5unjam h5unjam -i test_ub.h5 -o out_ub.txt -o out_ub.h5 HDF5User Block HDF5User Block unjam Jam

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 19 h5ls /Floats Group /Floats/DoubleArray Dataset {10, 5} /Floats/FloatArray Dataset {4, 3} /Floats/subs Group /IntArray Dataset {5, 6} Lists selected information about file objects in the specified format For Example, h5ls -r SDS2.h5

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 20 gif2h5 / h52gif gif2h5 – converts a GIF file into HDF5 gif2h5 apollo17_earth.gif apollo17_earth.h5 h52gif – converts an HDF5 file into GIF h52gif apollo17_earth.h5 apollo17_earth2.gif -i /apollo17_earth.gif/Image0 -p "/apollo17_earth.gif/Global Palette"

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 21 h5toh4 / h4toh5 h5toh4 -- Converts an HDF5 file to an HDF4 file h4toh5 -- Converts an HDF4 file to an HDF5 file

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 22 New tools h5stath5mkgrp h5copy h5check

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 23 h5copy Copies an object within a file/cross files / FloatArray Floats IntArray / FloatArray

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 24 h5copy -f shallow / i1 floats integers 64-bit i2 f32 f2f1 / floats 64-bit f32 f2f1 / floats 64-bit f32 -f shallow

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 25 h5copy -f soft / -f soft dset_SL/f1 f1 / dset_SL/f1 f1 / dset_SL/f1

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 26 h5copy -f ref / -f ref d1 dset_ref d / d1 dset_ref d / dset_ref 0 0

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 27 h5mkgrp Creates groups / infodata test1test2 h5mkgrp test.h5 /info /data /data/test1 /data/test2

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 28 h5stat Prints statistics about HDF5 file Header size Raw data size Attributes, datatypes, and etc. Helps To troubleshoot overhead in HDF5 files To choose specific object’s properties and storage strategies

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 29 h5check A validation tool that verifies if an HDF5 file is encoded according to the HDF5 File Format Specification

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 30 Why h5check? Verify if the file is compliant with the File Format As a verification tool required by the application of HDF5 File Format to be an ANSI standard Serves as a watch dog that the HDF5 library implementation is compliant with the File Format

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 31 Example: a compliant file % h5check example1.h5 VALIDATING example1.h5 FOUND super block signature VALIDATING the super block at 0... VALIDATING the object header at VALIDATING the btree at FOUND btree signature. VALIDATING the local heap at FOUND local heap signature. … Result: File is in compliance.

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 32 Example: a non-compliant file h5check invalid2.h5 FOUND super block signature VALIDATING the super block at 0... VALIDATING the object header at VALIDATING the btree at FOUND btree signature. VALIDATING the SNOD at FOUND SNOD signature. VALIDATING the object header at check_sym(at 1248): Errors from check_obj_header() decode_validate_messages(): Failure in type->decode(). H5O_sdspace_decode(): Bad version number in simple dataspace message. VALIDATING the local heap at FOUND local heap signature. Main(): Errors from check_obj_header(). decode_validate_messages(): Failure in type->decode(). H5O_attr_decode(): Can't decode attribute dataspace. H5O_sdspace_decode(): Bad version number in simple dataspace message. … Result: File is not in compliance.

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 33 Implementation Status Basic file format components DONE IN PROGRESS

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 34 HDF-Java ftp://ftp.hdfgroup.org/HDF5/hdf-java HDF-Java 2.4 Released !!

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 35 HDF-Java 2.4 Release Build with HDF4 2r2 and HDF New platforms Mac intel Linux 64-bit AMD Solaris 64-bit 36 new enhancements and 44 bugs fixed Fixed memory leak Test suite Enhanced documentation

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD 36 Memory leak in old version while (true) { H5File f = new H5File (fname, H5File.READ); f.open(); f.close(); }