1. Weathertop Consulting, LLC Wednesday, January 14, 2009 IIPS 11A.2 1 A General Purpose System for Server-side Analysis of Earth Science Data Roland Schweitzer Weathertop Consulting, LLC Steve Hankin and Ansley Manke NOAA/PMEL Kevin O’Brien and Jeremy Malczyk JISAO

2. Weathertop Consulting, LLC January 14, 2009 2 Highlights The OPeNDAP Story The OPeNDAP Story Server-side analysis Server-side analysis Motivation Motivation Implementation Implementation Implementation of the server (and why you might care) ‏ Implementation of the server (and why you might care) ‏ Community participation and “call to action” Community participation and “call to action” Summary Summary

3. Weathertop Consulting, LLC January 14, 2009 3 The OPeNDAP Story Local access to remote data... Local access to remote data... Clients which use well-known API’s like netCDF are re-linked with the OPeNDAP client library Clients which use well-known API’s like netCDF are re-linked with the OPeNDAP client library Data providers install OPeNDAP server software (several are available) and data files Data providers install OPeNDAP server software (several are available) and data files Clients can access remote data via the API as if the data files were local Clients can access remote data via the API as if the data files were local

4. Weathertop Consulting, LLC January 14, 2009 4 Server-side Analysis In general server-side analysis is a computation made by an OPeNDAP server at the request of a client. In general server-side analysis is a computation made by an OPeNDAP server at the request of a client. The specification of the computation is transmitted to the server via the OPeNDAP URL. The specification of the computation is transmitted to the server via the OPeNDAP URL. The data returned are the results (and only the results) of the computation The data returned are the results (and only the results) of the computation

5. Weathertop Consulting, LLC January 14, 2009 5 F-TDS The Ferret-THREDDS Data Server (F-TDS) is our implementation a general purpose server-side analysis engine which plugs into new or existing TDS installations. The Ferret-THREDDS Data Server (F-TDS) is our implementation a general purpose server-side analysis engine which plugs into new or existing TDS installations. Where TDS is an OPeNDAP server implemented in Java and available from Unidata. Where TDS is an OPeNDAP server implemented in Java and available from Unidata.

6. Weathertop Consulting, LLC January 14, 2009 6 Our Motivation We are interested in server-side analysis for use with the Live Access Server (LAS). We are interested in server-side analysis for use with the Live Access Server (LAS). Reduces comparisons between data defined on different grids to a problem already solved Reduces comparisons between data defined on different grids to a problem already solved Reduces user-defined analysis requests to a problem already solved Reduces user-defined analysis requests to a problem already solved

7. Weathertop Consulting, LLC January 14, 2009 7 A highly configurable Web server designed to provide flexible access to geo-referenced scientific data The Live Access Server (LAS) ‏

8. Weathertop Consulting, LLC January 14, 2009 8 The Why You Should Care If you already serve data via TDS you can add server-side analysis to your TDS for the cost of installing Ferret, and a few Java classes and configuration files. If you already serve data via TDS you can add server-side analysis to your TDS for the cost of installing Ferret, and a few Java classes and configuration files. Gives your remote collaborators an efficient to access and perform analysis on your data. Gives your remote collaborators an efficient to access and perform analysis on your data.

9. Weathertop Consulting, LLC January 14, 2009 9 A GDS Digression The GrADS Data (DODS) Server was an early implementation of this concept. The GrADS Data (DODS) Server was an early implementation of this concept. Both GDS and F-TDS use the Java Runtime environment to invoke the associated legacy app (GrADS or Ferret) to do the heavy lifting. Both GDS and F-TDS use the Java Runtime environment to invoke the associated legacy app (GrADS or Ferret) to do the heavy lifting.

10. Weathertop Consulting, LLC January 14, 2009 10 F-TDS Capabilities F-TDS takes advantage of several characteristics of Ferret. F-TDS takes advantage of several characteristics of Ferret. New "virtual" data variables can be defined New "virtual" data variables can be defined Can build the metadata (netCDF header described by dimensions, coordinate variables and the structure of data variables) without performing any heavy calculations for both data read from files and “virtual” data variables Can build the metadata (netCDF header described by dimensions, coordinate variables and the structure of data variables) without performing any heavy calculations for both data read from files and “virtual” data variables Only performs calculations when the data are requested Only performs calculations when the data are requested Only calculates the minimal set needed to fulfil the current request Only calculates the minimal set needed to fulfil the current request

11. Weathertop Consulting, LLC January 14, 2009 11 Our Implementation The Java netCDF library allows new data container formats to be plugged-in by implementing the I/O Service Provider interface. The Java netCDF library allows new data container formats to be plugged-in by implementing the I/O Service Provider interface. Once “plugged-in” clients using nj22 have access to the data from this container. Once “plugged-in” clients using nj22 have access to the data from this container. We implemented a Ferret I/O Service provider which can read Ferret command scripts and direct Ferret to perform the calculations as needed to satisfy data requests. We implemented a Ferret I/O Service provider which can read Ferret command scripts and direct Ferret to perform the calculations as needed to satisfy data requests.

12. Weathertop Consulting, LLC January 14, 2009 12 The THREDDS Data Server TDS is an OPeNDAP server based on netCDF Java. TDS is an OPeNDAP server based on netCDF Java. nj22 I/O Service Providers can be plugged in to TDS. nj22 I/O Service Providers can be plugged in to TDS. The combination of the Ferret I/O Service Provider and TDS (aka F-TDS) serves via OPeNDAP data which are represented by Ferret command scripts (both data read from disk by Ferret and virtual data computed on-the-fly by Ferret). The combination of the Ferret I/O Service Provider and TDS (aka F-TDS) serves via OPeNDAP data which are represented by Ferret command scripts (both data read from disk by Ferret and virtual data computed on-the-fly by Ferret).

13. Weathertop Consulting, LLC January 14, 2009 13 Community Action Can we define a general syntax for server-side analysis requests? Can we define a general syntax for server-side analysis requests? Some common operations (ave, diff, linear interp.) with standard syntax Some common operations (ave, diff, linear interp.) with standard syntax Still allow server-specific (native) operations Still allow server-specific (native) operations Can we define a general syntax for geo- referenced projections? Can we define a general syntax for geo- referenced projections? ERDDAP’s (180),10,(360) syntax ERDDAP’s (180),10,(360) syntax TDS’s north=17.3&south=12.088&west=180&east=360.0 netCDF subset syntax TDS’s north=17.3&south=12.088&west=180&east=360.0 netCDF subset syntax

14. Weathertop Consulting, LLC January 14, 2009 14 OPeNDAP “Working” Group Working is in quotes because we have not been doing any work recently Working is in quotes because we have not been doing any work recently See: See:http://docs.opendap.org/index.php/Server-side_Functions

15. Weathertop Consulting, LLC January 14, 2009 15 Summary Server-side analysis is critical for LAS. Server-side analysis is critical for LAS. NetCDF Java and the THREDDS Data Server is a great platform for implementing this type of analysis with a legacy analysis application (like Ferret). NetCDF Java and the THREDDS Data Server is a great platform for implementing this type of analysis with a legacy analysis application (like Ferret). A community-developed server-side analysis framework would make it easier to get the advantages of server-side analysis from other servers. A community-developed server-side analysis framework would make it easier to get the advantages of server-side analysis from other servers.

16. Weathertop Consulting, LLC January 14, 2009 16 More Information http://ferret.pmel.noaa.gov/LAS/ http://ferret.pmel.noaa.gov/LAS/Documentation/the-ferret-thredds-data-server-f-tds Roland.Schweitzer@noaa.gov Roland.Schweitzer@noaa.gov