Other Projects Relevant (and Not So Relevant) to the SODA Ideal: NetCDF, HDF, OLE/COM/DCOM, OpenDoc, Zope Sheila Denn INLS April 16, 2001

NetCDF (Network Common Data Form) Consists of an interface, a library, and a format. Designed for the creation, access, and sharing of scientific data. Developed at the Unidata Program Center by Glenn Davis, Russ Rew, Steve Emmerson, John Caron, and Harvey Davies, starting around It is still under active development (the most recent beta version was released this February). Freely distributed libraries for C, C++, Fortran, Java, and Perl.

NetCDF data is: Self-Describing. Architecture-independent. Direct-access. Appendable. Sharable.

NetCDF is designed to handle data that is stored in arrays by giving the user access to the members of the array without having to know the details of how it is stored. A wide array of third party client software has been created for the NetCDF format; see for a comprehensive list. Unidata claims usage of NetCDF on some 2000 separate hosts in 55 countries.

NetCDF File Structure Header – contains all information about dimensions, attributes, and variables Data – includes fixed-size data and variable-size data

General form of CDL (Common Data Form Language) description of NetCDF file: netCDF name { dimensions:... variables:... data:... }

Dimensions are used to describe physical dimensions like time, latitude, longitude, height, etc. Each dimension has a name and a length. Variables are the basic data units. Each dimension has a name, data type, and shape, which is a list of dimensions and determines whether the variable is a scalar (0 dimensions), a vector (1 dimension), a matrix (2 dimensions), etc.

Attributes are used to store metadata. Most attributes provide information about a specific variable. Global attributes can be used to provide metadata about the dataset as a whole. Attributes have an associated variable, a name, data type, length and value. All attributes are treated as vectors.

netcdf example_1 { // example of CDL notation for a netCDF dataset dimensions: // dimension names and lengths are declared first lat = 5, lon = 10, level = 4, time = unlimited; variables: // variable types, names, shapes, attributes float temp(time,level,lat,lon); temp:long_name = "temperature"; temp:units = "celsius"; float rh(time,lat,lon); rh:long_name = "relative humidity"; rh:valid_range = 0.0, 1.0; // min and max int lat(lat), lon(lon), level(level); lat:units = "degrees_north"; lon:units = "degrees_east"; level:units = "millibars"; short time(time); time:units = "hours since "; // global attributes :source = "Fictional Model Output"; data: // optional data assignments level = 1000, 850, 700, 500; lat = 20, 30, 40, 50, 60; lon = -160,-140,-118,-96,-84,-52,-45,-35,-25,-15; time = 12; rh =.5,.2,.4,.2,.3,.2,.4,.5,.6,.7,.1,.3,.1,.1,.1,.1,.5,.7,.8,.8,.1,.2,.2,.2,.2,.5,.7,.8,.9,.9,.1,.2,.3,.3,.3,.3,.7,.8,.9,.9, 0,.1,.2,.4,.4,.4,.4,.7,.9,.9; }

HDF Developed and supported by NCSA as a container for scientific data. First version released in 1988, still under active development.

HDF Basic structure: –Primary Objects Groups Datasets –Secondary Objects Datatypes Dataspaces –Additional organizational structures Attributes Sharable objects Storage and access properties

HDF Data Model A group is a directory structure that contains datasets, groups, and other objects. A dataset is a multidimensional array of elements along with metadata. Datasets contain: –Array –Dataspace – information about size and shape of dataset –User-defined attributes

From “HDF Overview” at

HDF Dataset Structure Group –Header –Symbol table Dataset –Header Name Datatype Dataspace Storage layout –Data Array Attribute –Name –Value

Creating an HDF Dataset Create a dataset identifier Define dataset characteristics (datatype, dataspace, properties) Create dataset Close datatype, dataspace, dataset, etc.

OLE/COM/DCOM These are elements of Microsoft’s “component object” approach to software design. This is a completely different approach to the problems that SODA-based systems are trying to solve: here, the data is still created in a proprietary system, and these component objects act as middleware to pull data out of their proprietary formats when needed.

COM Architecture Client Component Inprocess COM run time provider Security RPC RPC COM run time Security provider RPC RPC Local LPC Protocol stack DCOM network- protocol Remote From “An Overview of ActiveX” at

OpenDoc Architecture for compound documents. Developed by IBM starting around Was implemented on the Mac and then quickly abandoned.

OpenDoc Features Documents –Parts (data + code) –Frames Support Features –Editors –Viewers –Stationery –Translators –Services

OpenDoc Structure Every document has a single root part. Parts are displayed in frames. Container parts can contain other parts. Parts have both a part kind, which is basically its file format (TIFF, GIF, etc.) and a part category, which is more general (image) Parts are responsible for displaying themselves.

OpenDoc Display A canvas describes the drawing environment for a part. Frames are represented on the canvas as facets. Each facet is associated with a canvas, frames, and document parts. Frames and facets are associated with shapes and transforms. The shape describes the geometric shape, while the transform gives information about how and where that shape should appear.

OpenDoc Display (cont.) A part can be shown in different frames using a different presentation. –Ex. A spreadsheet part could be displayed both as a table and as a graph.

Interacting with OpenDoc Each collection of document parts has a document shell. The shell receives user events and hands them off to individual editors through the OpenDoc Dispatcher. The structure is designed to allow for in- place editing of disparate document parts.

OpenDoc Storage A container holds one or more document objects. A document contains one or more draft objects. Each draft contains storage units. The storage units contain the data that forms each part. The draft records the state of the document at different points in time. Only the most recent draft is editable.

Zope “Zope consists of a number components which work together to provide a complete yet flexible application server package. Zope includes an internet server, a transactional object database, a search engine, a web page templating system, a through the web development and management tool, and comprehensive extension support.” (from