OOI CyberInfrastructure: Technology Overview - Hyrax January 2009 Claudiu Farcas OOI CI Architecture & Design Team UCSD/Calit2

OOI-CYBERINFRASTRUCTURE 2 Hyrax Overview OPeNDAP Hyrax is a server for scientific data provisioning (using DAP2 standard) Uses a modular architecture to support different application- level protocols – Data access using DAP –Catalogs using THREDDS –Browsing using HTML and ASCII Modules for data access –Different file types –Potential for database and scripting Modules for commands –Commands provide varying operations for different protocols

OOI-CYBERINFRASTRUCTURE 3 Hyrax High-level Architecture Two cooperating components: –OpenDAP Lightweight Front-end Server (OLFS) provides DAP, plain ASCII & HTML interfaces –Back-end server (BES) reads data from repositories Both parts can be customized –Front-end: different network protocols –Back-end: different data formats/systems OLFSBES Data

OOI-CYBERINFRASTRUCTURE 4 Hyrax Architecture Front-end (OLFS): –Runs inside Tomcat servlet container –Authentication & Authorization performed via Tomcat –Provides DAP2 compliant interface –SOAP interface for programmatic access –Extensible via pluggable dispatch handlers Back-end (BES): –Data resource access. –Extensible (see Patrick’s slides) Single/Multiple Machine Installations Possible Communication between OLFS & BES via TCP/IP or UNIX Sockets

OOI-CYBERINFRASTRUCTURE OLFS Java Servlet Engine BES Unix Daemon BES Commands XML- encapsulated object Data Store File system with data files, SQL Database, … DAP2 THREDDS HTML Optional THREDDS catalogs Hyrax Architecture

OOI-CYBERINFRASTRUCTURE 6 OLFS DAP Request Processing DAP requests are made using URLs –URL:. [? ] Pathname: names a unique data source Request: what the client wants Constraint: optional modification to the request OLFS –parses the URL into the pathname, request and constraint –uses knowledge of the BES commands & capabilities to instruct the BES how to build the response –packages the resulting response from the BES (e.g., builds a HTTP ‘text/plain’ response document and inserts the correct content).

OOI-CYBERINFRASTRUCTURE 7 Extending the OLFS Extension ‘modules’ written in Java Added to a directory within Tomcat The new modules have complete access to the request information Both HTTP GET and POST requests There is some significant processing done before the handler is called: –Conditional GET Requests –Authorization & Authentication

OOI-CYBERINFRASTRUCTURE 8 BES Architecture Network Protocol and Process start/stop activities Data Store Interfaces BES Framework PPT* Initialization/ Termination DAP2 Access NetCDF3HDF4FreeForm … Data Catalogs Commands** BES Commands/ XML Documents *PPT: point to point transport **Some commands are built in, most load at run-time

OOI-CYBERINFRASTRUCTURE 9 BES Components Modular architecture in C++ Modules interact with the BES and can interact with other modules Modules can add/modify functionality of BES New modules are loaded at runtime when needed The command handler can be replaced to replace the BES’s default command syntax Handlers are not limited to DAP

OOI-CYBERINFRASTRUCTURE 10 BES Handler Interaction Response handlers: –Respond to commands from the ‘command handler’ –Know how to create the response object –Do not necessarily fill in the response object, but know how the response should be filled in (by asking appropriate request handlers) –Build & transmit new responses Request/Format handlers –Register interests and capabilities (i.e., know-how to fill a response) –Read data and provide object content Handlers can: –Build responses directly from data –Use output from other handlers –Query one or more handlers for fill-in information Reporters: Record usage information (e.g., logging, statistics) Aggregators: Build aggregate responses

OOI-CYBERINFRASTRUCTURE 11 BES Extensibility New request handlers (data handlers like netcdf, hdf4) –Request/data handlers know how to fill in a response object. (e.g., how to open a netcdf file, access attributes within that file, and add those attributes to the DAS response object) New response handlers (flat, Tab, Info) –Response handlers know how to create the response object and what needs to happen to get that response object filled in. (e.g., for a DAP request, the DAP response handler knows to create a DAS object and contact the request handler for each container to fill in the DAS object) New commands (e.g., hello world example) Container and Definition storage Aggregation engines (multiple can be installed) Methods of returning data (return as nc) Initialization/Termination callbacks Exception Handlers (registered with the Exception Manager) Reporters (data access statistics, metrics, logging, etc)

OOI-CYBERINFRASTRUCTURE 12 BES Example Definitions def as c1, c2 get das for def Containers c1,file1,csv c2,file2,csv ce,file3,nc c4,file4,ff def command Response Handlers das, BESDASResponseHandler help, BESHelpResponseHandler Request Handlers csv, CSVRequestHandler das, fill_das_func help, fill_help_func das csv das c1 das c1 c2 das show help help csv

OOI-CYBERINFRASTRUCTURE 13 BES Built-in Commands String commands sent from client to server –show help; (version, process, status, keys) –set container … –show containers; –delete container ; –delete containers; –define … –show definitions; –delete definition ; –delete definitions; –set context to ; –show context;

OOI-CYBERINFRASTRUCTURE 14 BES DAP Commands DAPCommandModule adds: –get das for [return as type]; –get dds … –get ddx … –get dods … –show catalog [for “node”]; (info) dap-server modules (www,usage,ascii) add: –get info_page … –get ascii … –get html_form … Data handlers (nc, ff, csv) don’t add any new commands.

OOI-CYBERINFRASTRUCTURE 15 Thank you