Presentation is loading. Please wait.

Presentation is loading. Please wait.

Recent Work in Progress

Similar presentations

Presentation on theme: "Recent Work in Progress"— Presentation transcript:

1 Recent Work in Progress
John Caron, June 3, 2003 THREDDS development Dynamic Catalogs: DQC, Resolvers IDD Data Server ADDE Cataloger NetCDF development NetCDF Markup Language (NcML) More efficient Java I/O (NIO) NetCDF/DODS/HDF5 Data Models 90% done – last 10% take another 90%.

2 THREDDS Catalogs Catalog Generator Data Server Client Application
HTTP Server CatalogRef.xml Catalog.xml Catalog Generator Data Server DODS, ADDE, FTP, HTTP Client Application Catalogs are lists of dataset URLs. Static case. Catgen talks to server or file system. Catalog might be static or dynamically generated. CatalogRef allows dynamic partial generation (benno) Datasets

3 Dynamic Catalogs = Services
HTTP Tomcat Server Catalog.xml Catalog Service Catalog Generator CatalogRef.xml Query Resolver Service DQC.xml Client Application Resolver Service URI URL Need more dynamic system for real time and very large datasets. Catalog is a file, but these are services, that is, code. Show IDD Server catalog – show sattellite DQC, then show radar DQC Data Server DODS, ADDE, FTP, HTTP Datasets

4 Dataset Query Capability (DQC)
XML document. Describes what the user can ask for as a set of orthogonal “selections”. On the client, a “query URL” is formed based on the user’s choices, and sent to the server. The “query resolver” server finds which datasets satisfy the query and returns a list of real dataset URLs. The DQC describes the queries that the server is capable of responding to. In response to Level 3 radar data Generality of it was Ethan’s idea. Robb implemented the radar server. Note its an indirection: a dataset which you use to find other datasets.

5 Resolver Services Logical Dataset, eg “latest ETA model run”
Dataset with Service type “Resolver” On the client, the URI of the logical dataset is sent to the server The server finds what is available and returns a list of real dataset URLs. DQC as a fancy kind of Resolver Service. URI to URL Credit Jeff and Don.

6 ADDE Cataloger Catalog.xml Client Application ADDE Data Server
HTTP Tomcat Server Catalog.xml Catalog Service ADDE Cataloger CatalogRef.xml Query Resolver Service DQC.xml Xxxxx Xxxx Client Application Try to improve response to ADDE Servers. Poll / cache results Add backdoor to ADDE servers. Will try on gozer. If time, show ADDE Cataloger – client. ADDE Data Server Datasets IDD

7 Summary IDD Data Server
Get as much of the IDD Data feeds available via THREDDS as possible. NCEP model data (catgen) (DODS) Level 3 NEXRAD (custom server/DQC) (ADDE) SSEC/Unidata Satellite data (ADDE Cataloger) (ADDE) Text Data: Metars, Surface Obs, etc (DQC/custom server), returns text or XML. Profiler Data (custom server/DQC) (ADDE) Credit Robb

8 NetCDF 3 NetCDF-3 library API Client Application NetCDF OpenDAP File
Dataset protocol NcML Dataset XML Virtual dataset Local file HTTP protocol NetCDF-3 library API HTTP, Dataset Java-only Client Application

9 NetCDF Markup Language
XML representation of netCDF metadata, uses XML Schema Core: existing netCDF data model Coordinate System: general and georeferencing coordinate system Dataset: redefine, aggregate, subset Luca Cinquini (NCAR/SCD/ESG), John Caron, Ethan Davis, Bob Drach (LLNL), Stefano Nativi (Florence), Russ Rew

10 NcML Coordinate Systems
Convention Parser ATDRadar AWIPS COARDS CF CSM GDV NUWG WRF Zebra NetCDF File OpenDAP Dataset Netcdf Dataset NcML Dataset XML

11 GeoGrids, GeoTiffs, Geowhiz!
NetCDF File OpenDAP Dataset Convention Parser Netcdf Dataset GeoGrid factory VisAD / IDV GeoGrid Dataset WCS Server Have sent GeoTIFFS to ESRI using this (credit Roland) using ccm2 files. Limited to lat/lon, want to add projections. Yuan will continue work on this. Stefano work on WCS. GeoTiff Writer Strange land of GIS OpenGIS WCS GeoTiff File

12 NcML Dataset : “virtual view”
NetCDF File OpenDAP Dataset NcML Dataset XML Dataset XML Parser Java-netCDF 2.1 CS output format of Convention parsing Dataset is an input format. Client Application NetCDF Dataset

13 NcML Dataset Use NcML like CDL, to declare the contents of a netCDF file. Add, delete or rename Variables, Attributes, and Dimensions Subset Variables Reorder a Variable’s dimensions Aggregate multiple netCDF files, a la DODS Aggregation Server NcML Dataset is a “virtual view” or can make copy to a local netCDF file. Declarative vs procedural Subset AWIPS Investigate Integrate with thredds catalogs? Demo copy dods file from Ken Keiser’s UAH “passive microwave” data.

14 2: NcML Datasets on a Server
Catalog.xml DODS Agg/Netcdf Server DODS, ADDE, FTP, HTTP Dataset XML Parser Client Application NcML Dataset XML Datasets

15 3: NcML Datasets via Catalogs
Catalog.xml NcML Dataset XML NetCDF File OpenDAP Dataset Catalog/Dataset XML Parser Java-netCDF v 2.1.1 Third party metadata. Client Application

16 NIO Rewrite ucar.nc2 I/O layer using java.nio package (currently using ucar.netcdf) Uses memory mapping, bulk I/O transfer Prototype has 7x speedup on large files. Requires JDK 1.4+ HTTP access must be rewritten

17 NIO vs current Java NIO Current old/new First access
small (3.9 Mb) large (240 Mb) Average next 5 accesses small large Time in millisecs to sequentially read entire file Wintel 2GHz, 1 GB main memory Java client

18 NIO vs optimized C NIO C C/NIO First access small (3.9 Mb) 281 370 1.3
large (240 Mb) Average next 5 accesses small large Java –client vs. VC 6.0 /O2 Probably OS / JDK dependent.

19 NetCDF Data Model NetcdfFile Dimension Variable Attribute Attribute
DataType byte char short int float double

20 OpenDAP Data Model Dataset BaseType primitive (8) string array
grid structure sequence Dataset array Dimension BaseType BaseType Attribute Attribute structure / sequence Allow DB like queries on sequences, eg all records with x < 3.0 Attribute BaseType Attribute

21 HDF5 Data Model Datatype Dataset Groups Datatype DataSpace Attribute
Fixed point floating point date/time string bit field Opaque Compound Reference Enumeration Variable length Array Dataset DataSpace Attribute Datatype Groups File directory structure inside HDF file. Data storage Compact External Layout Indexed Striped Neither HDF5 or OpenDAP has shared global dimensions.

22 Possible Extensions to netCDF data model
Add new data types: Strings: variable length arrays of bytes, plus an encoding attribute. Structures: collections of any other element types, allow nested structures. Vector: a variable length 1D array of any type. Allow reusable structure definition = user defined data type. Allow unnamed, undeclared dimensions = anonymous dimensions. Allow multiple unlimited dimensions (outer dimension only) Compression. Push scale/offset into library, allow variable bit sizes. Explicit support for coordinate variables/axes. All probably feasible on top of HDF5.

23 New NetCDF Data Model NetcdfFile Variable Dimension Attribute
Structure DataType byte short int long float double String Structure Vector Dimension DataType DataType DataType Deprecate char, replace with String. Vector Length Attribute DataType Attribute

24 NetCDF 4 NetCDF 4 library API Client Application NetCDF V.1 and 2 File
OpenDAP Dataset HDF5 File OpenDAP 4.0 protocol Local file or HTTP protocol NcML Dataset XML NetCDF 4 library API Influence both OpenDAP and HDF5 Ambitious, but useful for THREDDS clients Unidata Priorities ?? The most compelling reason…. Virtual dataset Client Application


Download ppt "Recent Work in Progress"

Similar presentations

Ads by Google