Introduction to HDF5 Session Five Reading & Writing Raw Data Values Keys to the HDF Secret Handshake Copyright © 2010 The HDF Group. All Rights Reserved

Raw Data Values Data Values Data Values User Application HDF5 Software mental model of data User Application Data Values Data Values HDF5 Software HDF5 File Copyright © 2010 The HDF Group. All Rights Reserved

Write – Memory to Disk memory disk Copyright © The HDF Group. All Rights Reserved

Remember HDF5 Dataspaces Dim_1 = 5 Dim_2 = 7 Dim_0 = 4 HDF5 Dataspace 3 Rank Dim_2 = 7 Dimensions Dim_0 = 4 Dim_1 = 5 Specifications for array dimensions Multi-dimensional array of identically typed data elements HDF5 datasets organize and contain “raw data values”. HDF5 dataspaces describe the logical layout of the data elements. Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 Dataspaces – Multiple Roles Describe the logical layout of data elements… … in defining a Dataset rank and dimensions are a permanent part of the Dataset in the File … in an existing Dataset as the basis for selecting which elements will be read or written … in an application’s data buffer as the basis for selecting which elements will be read or written HDF5 File Rank = 3 Dimensions = 4x5x7 Rank = 3 Dimensions = 4x5x7 Rank = 1 Dimensions = 20 Copyright © The HDF Group. All Rights Reserved

Partial I/O Hyperslab: A portion of a dataset Hyberslab selection: A logically contiguous collection of points or a regular pattern of points or blocks Move part of a dataset memory disk (a) Selection from a 2D array to the corner of a smaller 2D array (b) Regular series of blocks from a 2D array to a contiguous sequence at a certain offset in a 1D array memory disk Copyright © The HDF Group. All Rights Reserved

Partial I/O Data values are copied in “row-major” order. First dimension varies the slowest. Move part of a dataset memory disk (c) A sequence of points from a 2D array to a sequence of points in a 3D array. memory disk (d) Union of hyperslabs in file to union of hyperslabs in memory. Copyright © The HDF Group. All Rights Reserved

HDF5 Filters The HDF5 Library can apply filters that act on raw data as it is written and read. improves storage efficiency and transmission speed compression Copyright © 2010 The HDF Group. All Rights Reserved

Chunked Storage Layout Dataset is stored as fixed-size N-dimensional “blocks” N == rank of the Dataset, specified by its Dataspace Since N can be > 3, we call the blocks “chunks” Better access time to subsets of the dataset chunked storage Datasets that are extensible and/or have filters must use the chunked storage layout Copyright © The HDF Group. All Rights Reserved

Hyperslab, Compression Filter, Chunked Storage representation of dataset representation of region and chunks in dataset representation of chunks and region elements on disk Copyright © 2010 The HDF Group. All Rights Reserved

Data ReadPipeline Copyright © 2010 The HDF Group. All Rights Reserved

Session Summary HDF5 has a rich set of features to support complex data access patterns and handle large datasets. Hyperslab selection for raw data value reads and writes Filters for compression, encryption, … Chunked storage for efficient transfers, extensible datasets, … Key features of the HDF5 Library More details later, as they can dramatically affect your performance Copyright © 2010 The HDF Group. All Rights Reserved

Stretch Break Copyright © 2010 The HDF Group. All Rights Reserved