Presentation on theme: "HDF5-HL Packet Tables. The Situation: A Stream of Data An instrument takes measurements at regular intervals – Data arrives in "packets"--one value at."— Presentation transcript:
HDF5-HL Packet Tables
The Situation: A Stream of Data An instrument takes measurements at regular intervals – Data arrives in "packets"--one value at a time – Data is arriving in real-time Or multiple instruments are being used – Packets consist of one or more measurements – Packets vary in size and content
The Solution: Packet Tables A high-level API for HDF5 – Designed to support streams of data – High-performance for real-time data – Supports both fixed-length packets and variable-length packets Available in C and C++
Packet Tables vs. H5TB Tables The "Packet Table" and "Table" interfaces both create tables in HDF5. H5TB Tables are flexible. H5TB Tables support insertions. Packet Tables are high-performance and support variable-length entries. A table is one or the other, but not both!
Example – Boeing flight test HDF5 Packet Some other HDF5 Table package
Using Packet Tables A Packet Table contains either fixed-length or variable-length packets. – Use H5PTcreate_fl or H5PTcreate_vl – Once set, a Packet Table's type never changes Packet Tables need to be opened and closed like HDF5 datasets. – Use H5PTopen and H5PTclose
Using Packet Tables Write packets from the data stream – Use H5PTappend Read packets back in order – Set the starting point with H5PTset_index – Use H5PTget_next to move through the data …Or, out of order – Use H5PTread_packets
Fixed-length vs. Variable-length Time Data Time a. Fixed length packets.b. Variable length packets.
Fixed-Length vs. Variable Length Both types of Packet Table use the same API calls – Fixed-length tables use HDF5 datatypes Variable-Length Packet Tables use hvl_t structs – HDF5's natural support for variable-size data – During reads, a buffer is allocated and must be freed -- use H5PTfree_vlen_readbuff
Packet Tables in Action An overview of Packet Tables http://hdf.ncsa.uiuc.edu/HDF5/hdf5_hl/doc/RM_hdf5pt_intro.html See the Packet Table use cases: http://hdf.ncsa.uiuc.edu/HDF5/hdf5_hl/doc/RM_hdf5pt_usecases.html – Simple examples of Packet Tables in use
SQL, Science data and HDF5 While the commercial world has standardized on the relational data model and SQL, no single standard or tool has critical mass in the scientific community. There are many parallel and competing efforts to build these tool suites – at least one per discipline. Data interchange outside each group is problematic. In the next decade, as data interchange among scientific disciplines becomes increasingly important, a common HDF-like format and package for all the sciences will likely emerge. Scientific Data Management in the Coming Decade, Jim Gray, et al. Cyberinfrastructure Technology Watch Quarterly, Volume 1, Number 2, February 2005. http://www.ctwatch.org/quarterly/articles/2005/02/scientific-data-management/ Jim Gray, Distinguished Engineer at Microsoft, 1998 Turing Award winner