1. Recent Work in Progress
John Caron, June 3, 2003
THREDDS development
Dynamic Catalogs: DQC, Resolvers
IDD Data Server
ADDE Cataloger
NetCDF development
NetCDF Markup Language (NcML)
More efficient Java I/O (NIO)
NetCDF/DODS/HDF5 Data Models
90% done – last 10% take another 90%.

2. THREDDS Catalogs Catalog Generator Data Server Client Application
HTTP Server
CatalogRef.xml
Catalog.xml
Catalog Generator
Data Server
DODS, ADDE, FTP, HTTP
Client
Application
Catalogs are lists of dataset URLs. Static case.
Catgen talks to server or file system.
Catalog might be static or dynamically generated.
CatalogRef allows dynamic partial generation (benno)
Datasets
hostname.edu

3. Dynamic Catalogs = Services
HTTP Tomcat Server
Catalog.xml
Catalog Service
Catalog Generator
CatalogRef.xml
Query Resolver Service
DQC.xml
Client
Application
Resolver Service
URI
URL
Need more dynamic system for real time and very large datasets.
Catalog is a file, but these are services, that is, code.
Show IDD Server catalog – show sattellite DQC, then show radar DQC
Data Server
DODS, ADDE, FTP, HTTP
Datasets
hostname.edu

4. Dataset Query Capability (DQC)
XML document.
Describes what the user can ask for as a set of orthogonal “selections”.
On the client, a “query URL” is formed based on the user’s choices, and sent to the server.
The “query resolver” server finds which datasets satisfy the query and returns a list of real dataset URLs.
The DQC describes the queries that the server is capable of responding to.
In response to Level 3 radar data
Generality of it was Ethan’s idea.
Robb implemented the radar server.
Note its an indirection: a dataset which you use to find other datasets.

5. Resolver Services Logical Dataset, eg “latest ETA model run”
Dataset with Service type “Resolver”
On the client, the URI of the logical dataset is sent to the server
The server finds what is available and returns a list of real dataset URLs.
DQC as a fancy kind of Resolver Service.
URI to URL
Credit Jeff and Don.

6. ADDE Cataloger Catalog.xml Client Application ADDE Data Server
HTTP Tomcat Server
Catalog.xml
Catalog Service
ADDE
Cataloger
CatalogRef.xml
Query Resolver Service
DQC.xml
Xxxxx
Xxxx
Client
Application
Try to improve response to ADDE Servers.
Poll / cache results
Add backdoor to ADDE servers. Will try on gozer.
If time, show ADDE Cataloger – client.
ADDE Data Server
hostname.edu
Datasets
IDD

7. Summary IDD Data Server
Get as much of the IDD Data feeds available via THREDDS as possible.
NCEP model data (catgen) (DODS)
Level 3 NEXRAD (custom server/DQC) (ADDE)
SSEC/Unidata Satellite data (ADDE Cataloger) (ADDE)
Text Data: Metars, Surface Obs, etc (DQC/custom server), returns text or XML.
Profiler Data (custom server/DQC) (ADDE)
Credit Robb

8. NetCDF 3 NetCDF-3 library API Client Application NetCDF OpenDAP File
Dataset
protocol
NcML Dataset XML
Virtual dataset
Local file
HTTP protocol
NetCDF-3 library
API
HTTP, Dataset Java-only
Client
Application

9. NetCDF Markup Language
XML representation of netCDF metadata, uses XML Schema
Core: existing netCDF data model
Coordinate System: general and georeferencing coordinate system
Dataset: redefine, aggregate, subset
Luca Cinquini (NCAR/SCD/ESG), John Caron, Ethan Davis, Bob Drach (LLNL), Stefano Nativi (Florence), Russ Rew

10. NcML Coordinate Systems
Convention Parser
ATDRadar
AWIPS
COARDS CF
CSM
GDV
NUWG
WRF
Zebra
NetCDF File
OpenDAP Dataset
Netcdf Dataset
NcML Dataset XML

11. GeoGrids, GeoTiffs, Geowhiz!
NetCDF File
OpenDAP Dataset
Convention Parser
Netcdf Dataset
GeoGrid factory
VisAD / IDV
GeoGrid
Dataset
WCS Server
Have sent GeoTIFFS to ESRI using this (credit Roland) using ccm2 files.
Limited to lat/lon, want to add projections.
Yuan will continue work on this.
Stefano work on WCS.
GeoTiff Writer
Strange land
of GIS
OpenGIS
WCS
GeoTiff File

12. NcML Dataset : “virtual view”
NetCDF File
OpenDAP Dataset
NcML Dataset XML
Dataset XML Parser
Java-netCDF 2.1
CS output format of Convention parsing
Dataset is an input format.
Client Application
NetCDF Dataset

13. NcML Dataset
Use NcML like CDL, to declare the contents of a netCDF file.
Add, delete or rename Variables, Attributes, and Dimensions
Subset Variables
Reorder a Variable’s dimensions
Aggregate multiple netCDF files, a la DODS Aggregation Server
NcML Dataset is a “virtual view” or can make copy to a local netCDF file.
Declarative vs procedural
Subset AWIPS
Investigate Integrate with thredds catalogs?
Demo copy dods file from Ken Keiser’s UAH “passive microwave” data.

14. 2: NcML Datasets on a Server
Catalog.xml
DODS Agg/Netcdf Server
DODS, ADDE, FTP, HTTP
Dataset XML Parser
Client
Application
NcML Dataset XML
Datasets
hostname.edu

15. 3: NcML Datasets via Catalogs
Catalog.xml
NcML Dataset XML
NetCDF File
OpenDAP Dataset
Catalog/Dataset XML Parser
Java-netCDF v 2.1.1
Third party metadata.
Client
Application

16. NIO
Rewrite ucar.nc2 I/O layer using java.nio package (currently using ucar.netcdf)
Uses memory mapping, bulk I/O transfer
Prototype has 7x speedup on large files.
Requires JDK 1.4+
HTTP access must be rewritten

17. NIO vs current Java NIO Current old/new First access
small (3.9 Mb)
large (240 Mb)
Average next 5 accesses
small
large
Time in millisecs to sequentially read entire file
Wintel 2GHz, 1 GB main memory
Java client

18. NIO vs optimized C NIO C C/NIO First access small (3.9 Mb) 281 370 1.3
large (240 Mb)
Average next 5 accesses
small
large
Java –client vs. VC 6.0 /O2
Probably OS / JDK dependent.

19. NetCDF Data Model NetcdfFile Dimension Variable Attribute Attribute
DataType
byte
char
short
int
float
double

20. OpenDAP Data Model Dataset BaseType primitive (8) string array
grid
structure
sequence
Dataset
array
Dimension
BaseType
BaseType
Attribute
Attribute
structure /
sequence
Allow DB like queries on sequences, eg all records with x < 3.0
Attribute
BaseType
Attribute

21. HDF5 Data Model Datatype Dataset Groups Datatype DataSpace Attribute
Fixed point
floating point
date/time
string
bit field
Opaque
Compound
Reference
Enumeration
Variable length
Array
Dataset
DataSpace
Attribute
Datatype
Groups
File directory structure inside HDF file.
Data storage
Compact
External
Layout
Indexed
Striped
Neither HDF5 or OpenDAP has shared global dimensions.

22. Possible Extensions to netCDF data model
Add new data types:
Strings: variable length arrays of bytes, plus an encoding attribute.
Structures: collections of any other element types, allow nested structures.
Vector: a variable length 1D array of any type.
Allow reusable structure definition = user defined data type.
Allow unnamed, undeclared dimensions = anonymous dimensions.
Allow multiple unlimited dimensions (outer dimension only)
Compression. Push scale/offset into library, allow variable bit sizes.
Explicit support for coordinate variables/axes.
All probably feasible on top of HDF5.

23. New NetCDF Data Model NetcdfFile Variable Dimension Attribute
Structure
DataType
byte
short
int
long
float
double
String
Structure
Vector
Dimension
DataType
DataType
DataType
Deprecate char, replace with String.
Vector
Length
Attribute
DataType
Attribute

24. NetCDF 4 NetCDF 4 library API Client Application NetCDF V.1 and 2 File
OpenDAP
Dataset
HDF5
File
OpenDAP
4.0
protocol
Local file or
HTTP protocol
NcML Dataset XML
NetCDF 4 library
API
Influence both OpenDAP and HDF5
Ambitious, but useful for THREDDS clients
Unidata Priorities ??
The most compelling reason….
Virtual dataset
Client
Application