HDF and HDF-EOS Workshop XII HDF5 Tools Peter Cao The HDF Group October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII Outline A short demo of HDFView HDF5 command line tools October 15-18, 2008 HDF and HDF-EOS Workshop XII 2

HDF and HDF-EOS Workshop XII What is HDFView? HDFView HDF Application Browser/Editor Written in Java For more information and download, visit www.hdfgroup.org/hdf-java-html October 15-18, 2008 HDF and HDF-EOS Workshop XII 3

HDF and HDF-EOS Workshop XII HDFView Demo Browse objects in an HDF5 file Show data content in table, image Make a subset View attributes and metadata Create a new HDF5 file and edit data content October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII HDFView 2.5 Release planned in December, 2008 Build with HDF4 2r4 and HDF5 1.8.2 Fix bugs and add new features Plug memory leaks Support iRODS (Windows and linux only) Use jdk 1.6.0_06 or above (for performance) Drop mac PowerPC October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII HDF5 Command Line Tools Readers h5dump, h5diff, h5ls, h5check, h5stat Writers h5repack, h5repart, h5import, h5jam, h5unjam, h5copy, h5mkgrp Converters h4toh5, h5toh4, gif2h5, h52gif October 15-18, 2008 HDF and HDF-EOS Workshop XII 6

HDF and HDF-EOS Workshop XII Where to Start? Tools located in bin/ directory Use “-h” option for help Consult online help http://www.hdfgroup.org/hdf5tools.html Email help desk help@hdfgroup.org October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5dump Dumps contents of HDF5 file to stdout ASCII text XML Binary October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5dump: Object Headers > h5dump -H SDS.h5 HDF5 "SDS.h5" { GROUP "/" { GROUP "Floats" { DATASET "FloatArray" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } } DATASET "IntArray" { DATATYPE H5T_STD_I32LE DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) } October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5dump: Dataset > h5dump –d /Floats/FloatArray SDS.h5 HDF5 "SDS.h5" { DATASET "/Floats/FloatArray" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } DATA { (0,0): 0.01, 0.02, 0.03, (1,0): 0.1, 0.2, 0.3, (2,0): 1, 2, 3, (3,0): 10, 20, 30 } October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5dump: Binary Output -b F, --binary=[MEMORY|FILE|LE|BE] > h5dump -d /IntArray -o out_le.bin -b LE SDS.h5 0000000 00000000 00000001 00000002 00000003 00000004 00000005 0000030 0000000a 0000000b 0000000c 0000000d 0000000e 0000000f 0000060 00000014 00000015 00000016 00000017 00000018 00000019 0000110 0000001e 0000001f 00000020 00000021 00000022 00000023 0000140 00000028 00000029 0000002a 0000002b 0000002c 0000002d October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5diff Show differences between… two objects or two files October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5diff > h5diff SDS.h5 SDS2.h5 Dataset: </IntArray> 5 differences found October 15-18, 2008 HDF and HDF-EOS Workshop XII

h5diff: Report Differences > h5diff SDS.h5 SDS2.h5 -r /IntArray Dataset: </IntArray> and </IntArray> position IntArray IntArray difference ------------------------------------------------------ [ 0 0 ] 0 10 10 [ 1 0 ] 10 100 90 [ 2 0 ] 20 200 180 [ 3 0 ] 30 300 270 [ 4 0 ] 40 400 360 5 differences found October 15-18, 2008 HDF and HDF-EOS Workshop XII

h5diff: Unsolved Issues How to compare two files with different Structures Storage layouts Properties October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5repack Copies a file to a new file with Compression filter Data layout Other user options October 15-18, 2008 HDF and HDF-EOS Workshop XII

h5repack: Compression Filter -f {GZIP|SZIP|SHUF|FLET|NBIT|SOFF|NONE} For example > h5repack -i SDS.h5 -o out.h5 -f GZIP=6 October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5repack: Data Layout -l {CHUNK| COMPA| CONTI} For example > h5repack -i SDS.h5 -o out.h5 -l CHUNK=1000x1000 October 15-18, 2008 HDF and HDF-EOS Workshop XII

h5repack: File Format Version h5repack options File size in bytes Original file 3,167,264 --latest 2,267,346 --latest –compact=9 2,162,254 --latest –compact=9 –ssize=50:dtype 299,882 The test file was created with HDF5 1.6.5. The root group of the file contains 100 groups. Each group at the root contains 9 sub groups. Each sub group contains one compound dataset. The compound dataset has 40 members of type 32 bit integer October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5import Imports binary/ASCII data into an HDF5 file h5import infile -c config_file [infile -c config_file2 ...] -outfile outfile For example > h5import float5x4x2.txt -c First_set.conf -o First_set.h5 PATH work/First-set INPUT-CLASS TEXTFP RANK 3 DIMENSION-SIZES 5 2 4 OUTPUT-CLASS FP OUTPUT-SIZE 64 OUTPUT-ARCHITECTURE IEEE OUTPUT-BYTE-ORDER LE CHUNKED-DIMENSION-SIZES 2 2 2 MAXIMUM-DIMENSIONS 8 8 -1 GROUP "/" { GROUP "work" { DATASET "First-set" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H5S_UNLIMITED ) } DATA { (0,0,0): 1.01, 1.02, 1.03, 1.04, (0,1,0): 1.11, 1.12, 1.13, 1.14, (1,0,0): 1.21, 1.22, 1.23, 1.24, (1,1,0): 1.31, 1.32, 1.33, 1.34, (2,0,0): 1.41, 1.42, 1.43, 1.44, … } }} October 15-18, 2008 HDF and HDF-EOS Workshop XII

h5import: Requested Features Make it interchangeable with h5dump Excel spreadsheet October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII New for 1.6.6+ and 1.8.0+ h5copy h5mkgrp h5stat h5check October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5copy Copies an object within a file or across files / / Floats IntArray FloatArray FloatArray October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5copy –f shallow DST1 floats -f shallow SRC f32 64-bit floats integers DST2 i1 i2 f32 64-bit floats f1 f2 f32 64-bit f1 f2 October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5mkgrp Creates a group or group structure > h5mkgrp test.h5 /info /data /data/test1 /data/test2 / info data test1 test2 October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5stat Prints statistics about HDF5 file Header size Raw data size Attributes, datatypes, etc. Helps Troubleshoot overhead in HDF5 files Choose specific object’s properties and storage strategies October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5stat: Example >h5stat hdf5_test.h5 Filename: hdf5_test.h5 File information # of unique groups: 4 # of unique datasets: 13 … Object header size: (total/unused) Groups: 288/32 Datasets: 6824/864 Dataset storage information: Total raw data size: 1769009 October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5check Verifies whether an HDF5 file is compliant with a specific HDF5 File Format. By default, the file is checked against 1.8.x. Use -f16 option to check against 1.6.x. October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII h5check > h5check --verbose=2 example.h5 VALIDATING example.h5 according to library version 1.8.0 FOUND super block signature VALIDATING the super block at physical address 0... Validating version 2 superblock... VALIDATING the object header at logical address 48... VALIDATING version 2 object header ... FOUND Version 2 object header signature VALIDATING version 2 btree at logical address 1864... VALIDATING version 2 btree header at logical address 1864… : No non-compliance errors found October 15-18, 2008 HDF and HDF-EOS Workshop XII

> h5check --verbose=2 –f16 example.h5 VALIDATING example.h5 according to library version 1.6.6 VALIDATING the super block at physical address 0... Validating version 0/1 superblock... ***Error*** Errors found when checking superblock. Validation stopped. Disk Format Level 0-File Metadata-->0A-File Signature and Super Block Superblock v.0/1:Unable to read root symbol table entry ***End of Error messages*** Non-compliance errors found October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII Thank You! This presentation is based upon work supported in part by a Cooperative Agreement with NASA under NASA NNX06AC83A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration October 15-18, 2008 HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII Questions/comments? October 15-18, 2008 HDF and HDF-EOS Workshop XII