1. Introduction to HDF5 Session Five Reading & Writing Raw Data Values
Keys to the HDF Secret Handshake
Copyright © 2010 The HDF Group. All Rights Reserved

2. Raw Data Values Data Values Data Values User Application HDF5 Software
mental model of
data
User Application
Data Values
Data Values
HDF5 Software
HDF5 File
Copyright © 2010 The HDF Group. All Rights Reserved

3. Write – Memory to Disk memory disk
Copyright © The HDF Group. All Rights Reserved

4. Remember HDF5 Dataspaces
Dim_1 = 5
Dim_2 = 7
Dim_0 = 4
HDF5 Dataspace 3
Rank
Dim_2 = 7
Dimensions
Dim_0 = 4
Dim_1 = 5
Specifications for array dimensions
Multi-dimensional array of
identically typed data elements
HDF5 datasets organize and contain “raw data values”.
HDF5 dataspaces describe the logical layout of the data elements.
Copyright © 2010 The HDF Group. All Rights Reserved

5. HDF5 Dataspaces – Multiple Roles
Describe the logical layout of data elements…
… in defining a Dataset
rank and dimensions are a permanent part of the Dataset in the File
… in an existing Dataset as the basis for selecting which elements will be read or written
… in an application’s data buffer as the basis for selecting which elements will be read or written
HDF5 File
Rank = 3
Dimensions = 4x5x7
Rank = 3
Dimensions = 4x5x7
Rank = Dimensions = 20
Copyright © The HDF Group. All Rights Reserved

6. Partial I/O Hyperslab: A portion of a dataset
Hyberslab selection: A logically contiguous collection of points or a regular pattern of points or blocks
Move part of a dataset
memory
disk
(a) Selection from a 2D array to the corner of a smaller 2D array
(b) Regular series of blocks from a 2D array to a contiguous sequence at a certain offset in a 1D array
memory
disk
Copyright © The HDF Group. All Rights Reserved

7. Partial I/O
Data values are copied in “row-major” order. First dimension varies the slowest.
Move part of a dataset
memory
disk
(c) A sequence of points from a 2D array to a sequence of points in a 3D array.
memory
disk
(d) Union of hyperslabs in file to union of hyperslabs in memory.
Copyright © The HDF Group. All Rights Reserved

8. HDF5 Filters
The HDF5 Library can apply filters that act on raw data as it is written and read.
improves storage efficiency and transmission speed
compression
Copyright © 2010 The HDF Group. All Rights Reserved

9. Chunked Storage Layout
Dataset is stored as fixed-size N-dimensional “blocks”
N == rank of the Dataset, specified by its Dataspace
Since N can be > 3, we call the blocks “chunks”
Better access time to subsets of the dataset
chunked storage
Datasets that are extensible and/or have filters must use the chunked storage layout
Copyright © The HDF Group. All Rights Reserved

10. Hyperslab, Compression Filter, Chunked Storage
representation of dataset
representation of
region and chunks
in dataset
representation of chunks and region elements on disk
Copyright © 2010 The HDF Group. All Rights Reserved

11. Data ReadPipeline
Copyright © 2010 The HDF Group. All Rights Reserved

12. Session Summary
HDF5 has a rich set of features to support complex data access patterns and handle large datasets.
Hyperslab selection for raw data value reads and writes
Filters for compression, encryption, …
Chunked storage for efficient transfers, extensible datasets, …
Key features of the HDF5 Library
More details later, as they can dramatically affect your performance
Copyright © 2010 The HDF Group. All Rights Reserved

13. Stretch Break
Copyright © 2010 The HDF Group. All Rights Reserved