1. HDF and HDF-EOS Workshop XII
HDF5 Tools
Peter Cao
The HDF Group
October 15-18, 2008
HDF and HDF-EOS Workshop XII

2. HDF and HDF-EOS Workshop XII
Outline
A short demo of HDFView
HDF5 command line tools
October 15-18, 2008
HDF and HDF-EOS Workshop XII 2

3. HDF and HDF-EOS Workshop XII
What is HDFView?
HDFView
HDF Application
Browser/Editor
Written in Java
For more information and download, visit
October 15-18, 2008
HDF and HDF-EOS Workshop XII 3

4. HDF and HDF-EOS Workshop XII
HDFView Demo
Browse objects in an HDF5 file
Show data content in table, image
Make a subset
View attributes and metadata
Create a new HDF5 file and edit data content
October 15-18, 2008
HDF and HDF-EOS Workshop XII

5. HDF and HDF-EOS Workshop XII
HDFView 2.5
Release planned in December, 2008
Build with HDF4 2r4 and HDF
Fix bugs and add new features
Plug memory leaks
Support iRODS (Windows and linux only)
Use jdk 1.6.0_06 or above (for performance)
Drop mac PowerPC
October 15-18, 2008
HDF and HDF-EOS Workshop XII

6. HDF and HDF-EOS Workshop XII
HDF5 Command Line Tools
Readers
h5dump, h5diff, h5ls, h5check, h5stat
Writers
h5repack, h5repart, h5import, h5jam, h5unjam, h5copy, h5mkgrp
Converters
h4toh5, h5toh4, gif2h5, h52gif
October 15-18, 2008
HDF and HDF-EOS Workshop XII 6

7. HDF and HDF-EOS Workshop XII
Where to Start?
Tools located in bin/ directory
Use “-h” option for help
Consult online help
help desk
October 15-18, 2008
HDF and HDF-EOS Workshop XII

8. HDF and HDF-EOS Workshop XII
h5dump
Dumps contents of HDF5 file to stdout
ASCII text
XML
Binary
October 15-18, 2008
HDF and HDF-EOS Workshop XII

9. HDF and HDF-EOS Workshop XII
h5dump: Object Headers
> h5dump -H SDS.h5
HDF5 "SDS.h5" {
GROUP "/" {
GROUP "Floats" {
DATASET "FloatArray" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } }
DATASET "IntArray" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) }
October 15-18, 2008
HDF and HDF-EOS Workshop XII

10. HDF and HDF-EOS Workshop XII
h5dump: Dataset
> h5dump –d /Floats/FloatArray SDS.h5
HDF5 "SDS.h5" {
DATASET "/Floats/FloatArray" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) }
DATA {
(0,0): 0.01, 0.02, 0.03,
(1,0): 0.1, 0.2, 0.3,
(2,0): 1, 2, 3,
(3,0): 10, 20, 30 }
October 15-18, 2008
HDF and HDF-EOS Workshop XII

11. HDF and HDF-EOS Workshop XII
h5dump: Binary Output
-b F, --binary=[MEMORY|FILE|LE|BE]
> h5dump -d /IntArray -o out_le.bin -b LE SDS.h5
a b c d e f
e f
a b c d
October 15-18, 2008
HDF and HDF-EOS Workshop XII

12. HDF and HDF-EOS Workshop XII
h5diff
Show
differences
between…
two objects or
two files
October 15-18, 2008
HDF and HDF-EOS Workshop XII

13. HDF and HDF-EOS Workshop XII
h5diff
> h5diff SDS.h5 SDS2.h5
Dataset: </IntArray>
5 differences found
October 15-18, 2008
HDF and HDF-EOS Workshop XII

14. h5diff: Report Differences
> h5diff SDS.h5 SDS2.h5 -r /IntArray
Dataset: </IntArray> and </IntArray>
position IntArray IntArray difference
[ 0 0 ]
[ 1 0 ]
[ 2 0 ]
[ 3 0 ]
[ 4 0 ]
5 differences found
October 15-18, 2008
HDF and HDF-EOS Workshop XII

15. h5diff: Unsolved Issues
How to compare two files with different
Structures
Storage layouts
Properties
October 15-18, 2008
HDF and HDF-EOS Workshop XII

16. HDF and HDF-EOS Workshop XII
h5repack
Copies a file to a new file with
Compression filter
Data layout
Other user options
October 15-18, 2008
HDF and HDF-EOS Workshop XII

17. h5repack: Compression Filter
-f {GZIP|SZIP|SHUF|FLET|NBIT|SOFF|NONE}
For example
> h5repack -i SDS.h5 -o out.h5 -f GZIP=6
October 15-18, 2008
HDF and HDF-EOS Workshop XII

18. HDF and HDF-EOS Workshop XII
h5repack: Data Layout
-l {CHUNK| COMPA| CONTI}
For example > h5repack -i SDS.h5 -o out.h5 -l CHUNK=1000x1000
October 15-18, 2008
HDF and HDF-EOS Workshop XII

19. h5repack: File Format Version
h5repack options
File size in bytes
Original file
3,167,264
--latest
2,267,346
--latest –compact=9
2,162,254
--latest –compact=9 –ssize=50:dtype
299,882
The test file was created with HDF The root group of the file contains 100 groups. Each group at the root contains 9 sub groups. Each sub group contains one compound dataset. The compound dataset has 40 members of type 32 bit integer
October 15-18, 2008
HDF and HDF-EOS Workshop XII

20. HDF and HDF-EOS Workshop XII
h5import
Imports binary/ASCII data into an HDF5 file
h5import infile -c config_file [infile -c config_file2 ...] -outfile outfile
For example
> h5import float5x4x2.txt -c First_set.conf -o First_set.h5
PATH work/First-set
INPUT-CLASS TEXTFP
RANK 3
DIMENSION-SIZES 5 2 4
OUTPUT-CLASS FP
OUTPUT-SIZE 64
OUTPUT-ARCHITECTURE IEEE
OUTPUT-BYTE-ORDER LE
CHUNKED-DIMENSION-SIZES 2 2 2
MAXIMUM-DIMENSIONS
GROUP "/" {
GROUP "work" {
DATASET "First-set" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H5S_UNLIMITED ) }
DATA {
(0,0,0): 1.01, 1.02, 1.03, 1.04,
(0,1,0): 1.11, 1.12, 1.13, 1.14,
(1,0,0): 1.21, 1.22, 1.23, 1.24,
(1,1,0): 1.31, 1.32, 1.33, 1.34,
(2,0,0): 1.41, 1.42, 1.43, 1.44, … } }}
October 15-18, 2008
HDF and HDF-EOS Workshop XII

21. h5import: Requested Features
Make it interchangeable with
h5dump
Excel spreadsheet
October 15-18, 2008
HDF and HDF-EOS Workshop XII

22. HDF and HDF-EOS Workshop XII
New for and
h5copy
h5mkgrp
h5stat
h5check
October 15-18, 2008
HDF and HDF-EOS Workshop XII

23. HDF and HDF-EOS Workshop XII
h5copy
Copies an object within a file or across files / /
Floats
IntArray
FloatArray
FloatArray
October 15-18, 2008
HDF and HDF-EOS Workshop XII

24. HDF and HDF-EOS Workshop XII
h5copy –f shallow
DST1
floats
-f shallow
SRC
f32
64-bit
floats
integers
DST2 i1 i2
f32
64-bit
floats f1 f2
f32
64-bit f1 f2
October 15-18, 2008
HDF and HDF-EOS Workshop XII

25. HDF and HDF-EOS Workshop XII
h5mkgrp
Creates a group or group structure
> h5mkgrp test.h5 /info /data /data/test1 /data/test2 /
info
data
test1
test2
October 15-18, 2008
HDF and HDF-EOS Workshop XII

26. HDF and HDF-EOS Workshop XII
h5stat
Prints statistics about HDF5 file
Header size
Raw data size
Attributes, datatypes, etc.
Helps
Troubleshoot overhead in HDF5 files
Choose specific object’s properties and storage strategies
October 15-18, 2008
HDF and HDF-EOS Workshop XII

27. HDF and HDF-EOS Workshop XII
h5stat: Example
>h5stat hdf5_test.h5
Filename: hdf5_test.h5
File information
# of unique groups: 4
# of unique datasets: 13 …
Object header size: (total/unused)
Groups: 288/32
Datasets: 6824/864
Dataset storage information:
Total raw data size:
October 15-18, 2008
HDF and HDF-EOS Workshop XII

28. HDF and HDF-EOS Workshop XII
h5check
Verifies whether an HDF5 file is compliant with a specific HDF5 File Format.
By default, the file is checked against 1.8.x.
Use -f16 option to check against 1.6.x.
October 15-18, 2008
HDF and HDF-EOS Workshop XII

29. HDF and HDF-EOS Workshop XII
h5check
> h5check --verbose=2 example.h5
VALIDATING example.h5 according to library version 1.8.0
FOUND super block signature
VALIDATING the super block at physical address 0...
Validating version 2 superblock...
VALIDATING the object header at logical address 48...
VALIDATING version 2 object header ...
FOUND Version 2 object header signature
VALIDATING version 2 btree at logical address
VALIDATING version 2 btree header at logical address 1864… :
No non-compliance errors found
October 15-18, 2008
HDF and HDF-EOS Workshop XII

30. > h5check --verbose=2 –f16 example.h5
VALIDATING example.h5 according to library version 1.6.6
VALIDATING the super block at physical address 0...
Validating version 0/1 superblock...
***Error***
Errors found when checking superblock. Validation stopped.
Disk Format Level 0-File Metadata-->0A-File Signature and Super Block
Superblock v.0/1:Unable to read root symbol table entry
***End of Error messages***
Non-compliance errors found
October 15-18, 2008
HDF and HDF-EOS Workshop XII

31. HDF and HDF-EOS Workshop XII
Thank You!
This presentation is based upon work supported in part by a Cooperative Agreement with NASA under NASA NNX06AC83A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration
October 15-18, 2008
HDF and HDF-EOS Workshop XII

32. HDF and HDF-EOS Workshop XII
Questions/comments?
October 15-18, 2008
HDF and HDF-EOS Workshop XII