1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA

2 What is Dapper? Web server that provides distributed access to in-situ data via OPeNDAP protocol Clients include GrADS, Matlab Available data: –PMEL EPIC database –NODC World Ocean Database 2001

3 Problems with in-situ data Irregular (x,y,z,t) Not all variables available in all profiles Data from a given profile may be revised as QC is applied Each profile usually stored as individual file No standard APIs

4 Problems with in-situ data Irregular (x,y,z,t) Not all variables available in all profiles Data from a given profile may be revised as QC is applied Each profile usually stored as individual file Use Climate Data Portal for aggregation

5 Problems with in-situ data No standard APIs Use OPeNDAP Sequence datatype

6 Dapper architecture CORBA (IIOP) Climate Data Portal MySQL Java netCDF library netCDF profile netCDF profile Dapper CDP service netCDF service OPeNDAP client OPeNDAP Protocol (HTTP)

7 Climate Data Portal (CDP) CORBA based Metadata in database (MySQL) Variables (measurements) in individual netCDF files Developed by Don Denbo and Willa Zhu

8 OPeNDAP Sequence example (header): Dataset { Sequence { Sequence { Float32 longitude; Float32 longitude; UInt32 time[tsize = 2]; UInt32 time[tsize = 2]; Float32 latitude; Float32 latitude; Int32 _id; Int32 _id; Sequence { Sequence { Float32 depth; Float32 depth; Float32 temperature; Float32 temperature; } profile; } profile; } location = { { , {77, }, , 3645, { { 3, }, { 5, }, … } location = { { , {77, }, , 3645, { { 3, }, { 5, }, …

9 OPeNDAP (cont.) Sequences accessible to clients via Java or C++ libraries _id field in “outer” sequence allows unique identification of an individual profile

10 Dapper design Consists of configurable services –Climate Data Portal service –netCDF service Services route OPeNDAP request Implemented as Java servlet

11 Dapper architecture revisited CORBA (IIOP) Climate Data Portal MySQL Java netCDF library netCDF profile netCDF profile Dapper CDP service netCDF service OPeNDAP client OPeNDAP Protocol (HTTP)

12 CDP service example 1. HTTP request: depth,_id,temp&lat>0&lat<20 depth,_id,temp&lat>0&lat<20 2. Dapper contacts CDP via CORBA IIOP 3. CDP accesses MySQL for metadata, appropriate profile files for data 4. CDP returns data to Dapper via CDP API 5. Dapper returns to client as OPeNDAP sequence

13 CDP service performance < 500 us/profile for coordinate info 5 ms/profile to read one variable with z axis containing 1000 pts netCDF library enhancements: –Parsing of netCDF header/attributes is optional –Use String(char[]) constructor instead of String(byte[]) constructor 2X speed up

14 netCDF service Individual profile files available as OPeNDAP Grid datatype netCDF library enhancements: –True streaming server

15 Future work Stream data from CDP Distribute netCDF file processing using scatter/gather patterns

16 Availability