Presentation is loading. Please wait.

Presentation is loading. Please wait.

THREDDS development –Dynamic Catalogs: DQC, Resolvers –IDD Data Server –ADDE Cataloger NetCDF development –NetCDF Markup Language (NcML) –More efficient.

Similar presentations


Presentation on theme: "THREDDS development –Dynamic Catalogs: DQC, Resolvers –IDD Data Server –ADDE Cataloger NetCDF development –NetCDF Markup Language (NcML) –More efficient."— Presentation transcript:

1 THREDDS development –Dynamic Catalogs: DQC, Resolvers –IDD Data Server –ADDE Cataloger NetCDF development –NetCDF Markup Language (NcML) –More efficient Java I/O (NIO) –NetCDF/DODS/HDF5 Data Models Recent Work in Progress John Caron, June 3, 2003

2 HTTP Server THREDDS Catalogs Client Application Datasets Catalog.xml hostname.edu Catalog Generator Data Server DODS, ADDE, FTP, HTTP CatalogRef.xml

3 HTTP Tomcat Server Dynamic Catalogs = Services Client Application Datasets Catalog.xml hostname.edu Data Server DODS, ADDE, FTP, HTTP Query Resolver Service DQC.xml Catalog Service Catalog Generator CatalogRef.xml Resolver Service URI URL

4 Dataset Query Capability (DQC) XML document. Describes what the user can ask for as a set of orthogonal “selections”. On the client, a “query URL” is formed based on the user’s choices, and sent to the server. The “query resolver” server finds which datasets satisfy the query and returns a list of real dataset URLs. The DQC describes the queries that the server is capable of responding to.

5 Resolver Services Logical Dataset, eg “latest ETA model run” Dataset with Service type “Resolver” On the client, the URI of the logical dataset is sent to the server The server finds what is available and returns a list of real dataset URLs.

6 HTTP Tomcat Server ADDE Cataloger Client Application Datasets Catalog.xml hostname.edu ADDE Data Server Catalog Service ADDE Cataloger CatalogRef.xml Query Resolver Service DQC.xml IDD Xxxxx Xxxx

7 Summary IDD Data Server Get as much of the IDD Data feeds available via THREDDS as possible. –NCEP model data (catgen) (DODS) –Level 3 NEXRAD ( custom server/ DQC) (ADDE) –SSEC/Unidata Satellite data (ADDE Cataloger) (ADDE) –Text Data: Metars, Surface Obs, etc (DQC/custom server), returns text or XML. –Profiler Data ( custom server/ DQC) (ADDE)

8 NetCDF 3 NetCDF File NetCDF-3 library API Local file HTTP protocol Client Application OpenDAP Dataset OpenDAP protocol NcML Dataset XML Virtual dataset

9 NetCDF Markup Language XML representation of netCDF metadata, uses XML Schema Core: existing netCDF data model Coordinate System: general and georeferencing coordinate system Dataset: redefine, aggregate, subset Luca Cinquini (NCAR/SCD/ESG), John Caron, Ethan Davis, Bob Drach (LLNL), Stefano Nativi (Florence), Russ Rew

10 NcML Coordinate Systems NcML Dataset XML NetCDF File OpenDAP Dataset Convention Parser ATDRadar AWIPS COARDS CF CSM GDV NUWG WRF Zebra Netcdf Dataset

11 GeoGrids, GeoTiffs, Geowhiz! NetCDF File OpenDAP Dataset Convention Parser GeoGrid factory GeoGrid Dataset Netcdf Dataset GeoTiff Writer Strange land of GIS GeoTiff File OpenGIS WCS WCS Server VisAD / IDV

12 NcML Dataset : “virtual view” NetCDF File OpenDAP Dataset NetCDF Dataset NcML Dataset XML Dataset XML Parser Java-netCDF 2.1 Client Application

13 NcML Dataset Use NcML like CDL, to declare the contents of a netCDF file. Add, delete or rename Variables, Attributes, and Dimensions Subset Variables Reorder a Variable’s dimensions Aggregate multiple netCDF files, a la DODS Aggregation Server NcML Dataset is a “virtual view” or can make copy to a local netCDF file.

14 2: NcML Datasets on a Server Client Application Datasets Catalog.xml hostname.edu DODS Agg/Netcdf Server DODS, ADDE, FTP, HTTP NcML Dataset XML Dataset XML Parser

15 3: NcML Datasets via Catalogs Client Application Catalog.xml NcML Dataset XML Catalog/Dataset XML Parser Java-netCDF v NetCDF File OpenDAP Dataset

16 NIO Rewrite ucar.nc2 I/O layer using java.nio package (currently using ucar.netcdf) Uses memory mapping, bulk I/O transfer Prototype has 7x speedup on large files. Requires JDK 1.4+ HTTP access must be rewritten

17 NIO Current old/new First access small (3.9 Mb) large (240 Mb) Average next 5 accesses small large Time in millisecs to sequentially read entire file Wintel 2GHz, 1 GB main memory Java client NIO vs current Java

18 NIO C C/NIO First access small (3.9 Mb) large (240 Mb) Average next 5 accesses small large Java –client vs. VC 6.0 /O2 NIO vs optimized C

19 NetcdfFile NetCDF Data Model VariableDimension Attribute DataType byte char short int float double

20 BaseType primitive (8) string array grid structure sequence Dataset OpenDAP Data Model BaseType Attribute array Dimension BaseType structure / sequence BaseType Attribute

21 Datatype Fixed point floating point date/time string bit field Opaque Compound Reference Enumeration Variable length Array Groups File directory structure inside HDF file. HDF5 Data Model Data storage Compact External Layout Indexed Striped Dataset DataSpace Attribute Datatype

22 Possible Extensions to netCDF data model Add new data types: –Strings: variable length arrays of bytes, plus an encoding attribute. –Structures: collections of any other element types, allow nested structures. –Vector: a variable length 1D array of any type. Allow reusable structure definition = user defined data type. Allow unnamed, undeclared dimensions = anonymous dimensions. Allow multiple unlimited dimensions (outer dimension only) Compression. Push scale/offset into library, allow variable bit sizes. Explicit support for coordinate variables/axes.

23 NetcdfFile New NetCDF Data Model Variable Dimension Attribute DataType byte short int long float double String Structure Vector Structure Vector Length DataType

24 NetCDF 4 OpenDAP Dataset HDF5 File NetCDF 4 library API OpenDAP 4.0 protocol Local file or HTTP protocol Client Application NcML Dataset XML NetCDF V.1 and 2 File Virtual dataset

25


Download ppt "THREDDS development –Dynamic Catalogs: DQC, Resolvers –IDD Data Server –ADDE Cataloger NetCDF development –NetCDF Markup Language (NcML) –More efficient."

Similar presentations


Ads by Google