A Data Access Framework for ESMF Model Outputs Roland Schweitzer Steve Hankin Jonathan Callahan Kevin O’Brien Ansley Manke

20 July ESMF Workshop 2 Some Motivation Model Components produce …  Model Runs which produce …  Data and Metadata which should/could produce … should/could produce …  A data portal which allows user to: Search, browse, compare, validate and analyze results … Search, browse, compare, validate and analyze results …  which in turn helps produce insight

20 July ESMF Workshop 3 Some More Motivation ESMF and the ESCurator manage components, runs and output ESMF and the ESCurator manage components, runs and output LAS is a candidate to provide a data portal LAS is a candidate to provide a data portal

20 July ESMF Workshop 4 The Live Access Server (LAS) A highly configurable Web server designed to provide flexible access to geo-referenced scientific data

20 July ESMF Workshop 5 LAS LAS is an information broker that: LAS is an information broker that: ingests descriptions of dataingests descriptions of data data sets data sets variables variables (optional reshuffling of above into new categories) (optional reshuffling of above into new categories) domains: 1d-4d domains: 1d-4d special constraints (ensemble index) special constraints (ensemble index) rendering hints (smoothing, palettes) rendering hints (smoothing, palettes) presents the descriptions to the user in a coherent fashionpresents the descriptions to the user in a coherent fashion asks other software to perform tasks for the userasks other software to perform tasks for the user and gives the results to the userand gives the results to the user

20 July ESMF Workshop 6 Architecture Components User Interface Server (UIS) Product Server (PS) Backend Server (BES) Ferret Data (OPeNDAP) Server (FDS) Backend Server (BES) Data

20 July ESMF Workshop 7 LAS Offers Independence from underlying data Independence from underlying data Uniform interface for browse (visualize), compare, validate and analyze results for many data types Uniform interface for browse (visualize), compare, validate and analyze results for many data types Easy way to make connections to metadata and documentation Easy way to make connections to metadata and documentation Customizable user interface Customizable user interface

20 July ESMF Workshop 8 Example (Multiple Hierarchies)

20 July ESMF Workshop 9 Example (Collapse Hierarchies) Experimental Development Server

20 July ESMF Workshop 10 Example (Custom Headers) The NVODS Server (

20 July ESMF Workshop 11 Example (Adopt a Drifter)

20 July ESMF Workshop 12 Return to the Motivation Model Components produce …  Model Runs which produce …  Data and Metadata which should/could produce … should/could produce …  A data portal which allows user to: Search, browse, compare, validate and analyze results … Search, browse, compare, validate and analyze results …  which in turn helps produce insight

20 July ESMF Workshop 13 Data Browsing

20 July ESMF Workshop 14 Analysis and Comparison Compute differences Compute differences Can define new variables that are the result of some analysis operation (done in delayed mode so only data that is needed is computed) Can define new variables that are the result of some analysis operation (done in delayed mode so only data that is needed is computed)

20 July ESMF Workshop 15 Analysis and Comparison Can use FDS to define regriddings (or other analysis) to allow more complex comparisons Can use FDS to define regriddings (or other analysis) to allow more complex comparisons

20 July ESMF Workshop 16 Validation Use FDS to create samplings of in situ observations which can be compared to model data Use FDS to create samplings of in situ observations which can be compared to model data (This is a planned for a future release.) (This is a planned for a future release.)

20 July ESMF Workshop 17 Detailed Analysis Can jump to local tool (Matlab, IDL, Ferret, etc.) by requesting scripts for that application Can jump to local tool (Matlab, IDL, Ferret, etc.) by requesting scripts for that application Scripts encode access to remote LAS data via FDS Scripts encode access to remote LAS data via FDS Can take advantage of FDS analysis to compute some data reductions and transfer only the result to the local application Can take advantage of FDS analysis to compute some data reductions and transfer only the result to the local application

20 July ESMF Workshop 18 Additional Flexibility beyond Configuration Use LAS framework with different backend engine (new Product Server is designed to make this as easy as possible) Use LAS framework with different backend engine (new Product Server is designed to make this as easy as possible) Web-services interfaces Web-services interfaces To Product Server from other clientsTo Product Server from other clients Will allow implementation of “well- known” service like WMSWill allow implementation of “well- known” service like WMS

20 July ESMF Workshop 19 Architecture Components User Interface Server (UIS) Product Server (PS) Backend Server (BES) Ferret Data (OPeNDAP) Server (FDS) Backend Server (BES) Data

20 July ESMF Workshop 20 Access Control and the GRID The new architecture makes participating as a GRID partner easier The new architecture makes participating as a GRID partner easier Web services-based authentication schemes can be used by the Product Server and Backend Server to get authentication information (if not supplied by a browser certificate). Web services-based authentication schemes can be used by the Product Server and Backend Server to get authentication information (if not supplied by a browser certificate). Implemented as another “filter” in a already- existing mechanism for filtering on IP-address, number of accesses, etc. Implemented as another “filter” in a already- existing mechanism for filtering on IP-address, number of accesses, etc. Configuration metadata can be augmented to identify allowed access “roles” by dataset and/or variable (and maybe by output product). Configuration metadata can be augmented to identify allowed access “roles” by dataset and/or variable (and maybe by output product).

20 July ESMF Workshop 21 Return to the Motivation Model Components produce …  Model Runs which produce …  Data and Metadata which should/could produce … should/could produce …  A data portal which allows user to: Search, browse, compare, validate and analyze results … Search, browse, compare, validate and analyze results …  which in turn helps produce insight

20 July ESMF Workshop 22 More Information Subscribe to the LAS User’s mailing list Subscribe to the LAS User’s mailing list

20 July ESMF Workshop 23 The End

20 July ESMF Workshop 24 browser UI (java/script) gif,netCDF,etc. servers UIS PS LAS XML configuration metadata browser interface BES HTTP Request (later SOAP) scripts &/or commands products HTTP Response data LAS Architecture with OPeNDAP Legacy OPeNDAP Server browser OPeNDAP Server BES Legacy OPeNDAP UIS = User Interface Server PS = Product Server BES = Backend Server FDS = Ferret Data Server SOAP addXML FDS

20 July ESMF Workshop 25 Architecture Components User Interface Server (UIS) – keeps browser up-to-date with relevant interface widgets User Interface Server (UIS) – keeps browser up-to-date with relevant interface widgets Product Server (PS) – takes users requests and returns results Product Server (PS) – takes users requests and returns results Backend Server (BES) – runs legacy application to fulfill request Backend Server (BES) – runs legacy application to fulfill request Ferret Data Server (FDS) – General OPeNDAP server for LAS data Ferret Data Server (FDS) – General OPeNDAP server for LAS data

20 July ESMF Workshop 26 The UI Server Architecture User Interface Server User Interface Server Installation process translates LAS configuration into User Interface Widgets need to present datasets, variables, geo-spatial and time extents in HTMLInstallation process translates LAS configuration into User Interface Widgets need to present datasets, variables, geo-spatial and time extents in HTML Allows dataset collection to be easily organized into any hierarchies the installer choosesAllows dataset collection to be easily organized into any hierarchies the installer chooses Communicates requests to the Product Server and returns responses (images, files, data tables, etc.) to the Web BrowserCommunicates requests to the Product Server and returns responses (images, files, data tables, etc.) to the Web Browser Many other customization featuresMany other customization features

20 July ESMF Workshop 27 The Product Server Product Server Product Server Has knowledge of the LAS configurationHas knowledge of the LAS configuration Makes requests from one or more Back-end ServersMakes requests from one or more Back-end Servers Fully implemented Web-services interface will be available to client writersFully implemented Web-services interface will be available to client writers Can interact with a series of Backend-servers to fulfill a single request from the user clientCan interact with a series of Backend-servers to fulfill a single request from the user client Back-end server Back-end server Can produce product itselfCan produce product itselfor Interact with a single legacy application (like CDAT or Ferret) to produce productInteract with a single legacy application (like CDAT or Ferret) to produce product Requests are communicated via Request and Response XML that is exchanged via SOAP. Requests are communicated via Request and Response XML that is exchanged via SOAP.

20 July ESMF Workshop 28 FDS (Ferret Data Server) FDS is an OPeNDAP server FDS is an OPeNDAP server Automatically installed with LAS Automatically installed with LAS Therefore every gridded LAS dataset is served via OPeNDAP Therefore every gridded LAS dataset is served via OPeNDAP Has extended analysis capabilities beyond basic OPeNDAP servers Has extended analysis capabilities beyond basic OPeNDAP servers