What is FITS? FITS = Flexible Image Transport System Invented ~1980 to transfer Radio Astronomy datasets from one observatory to another Format now used by most astronomical software Widely used as data archive format FITSIO and other libraries developed to read/write FITS efficiently using C, Fortran, Perl, Python, Java… Always big-endian (so portable) Format defined to the last bit in primary literature. Images and binary tables fairly space-efficient.

FITS Structure in outline Any number of Header-Data units (HDUs) Header has metadata: lines of 80 characters NAME = value /[units] other commentary A few reserved names, e.g. DATE-OBS, no rules otherwise Data section can be Image or Table Image: 8, 16, 32-bit integers, 32 or 64-bit floats Up to 999 dimensions (mostly 2 or 3 dimensions) Table up to 999 columns, up to 2 billion rows 8/16/32-bit integers, 32/64-bit floats, strings, logical, 64/128-bit complex. Vector columns (optional variable length vectors) Null values can be defined

What’s wrong with FITS Flat structure for metadata, single name-space, only 8-character names Blocked in units of 2880 bytes Length of data unit must be declared in header unit, so cannot stream output No structure to HDUs, just a simple list No direct access to HDUs, have to decode each header in turn to find size of data unit. Always big-endian, requires byte-swap on most popular platforms. Not quite fully 64-bit clean

What’s wrong with VOTable Three alternative ways of including binary: Pure XML but very verbose, 4-byte real may turn into ~24 bytes, e.g. <TD>-0.12345678E-04</TD> Blob encoded in Base64 Pointer to separate FITS file Possible loss of linkage Duplication of header data Tabular datasets only, no format yet for images, spectra, etc. Cannot use direct offset to required row or field

What we want Flexible header to contain metadata XML very suitable Data section to contain heterogeneous data (table) or homogeneous (image etc) but to be efficient, evenly spaced Binary formats as in FITS very suitable Endian-neutral Suitable for streaming data

VOTable++ or FITS++? Use MIME/DIME to have separate sections of file, as proposed for BinX, and have: XML-based header, very similar to VOTable Binary data units to follow, similar to FITS Use BinX/DFDL in header to describe data unit. Can choose big or little-endian when file created Allow streaming format for tables, no pre-defined length. Allow FITS HDU as special case of data unit for compatibility. Make fully 64-bit clean for tables >2Grows.