OPeNDAP Hyrax Back-End Server (BES) Original version presented at APAC 2007 OPeNDAP Workshop Patrick West

OPeNDAP’s Hyrax Architecture OLFS Java Servlet Engine BES Unix Daemon BES Commands XML- encapsulated object File system with data files, SQL Database, … DAP2 THREDDS HTML Optional THREDDS catalogs Module

In what ways can you extend the BES? Extensions are dynamically loaded from a shared object module Containers and Definitions, and ways of storing them Dataset Catalogs Interface Initialization/Termination New commands (like our hello world example) New response objects New response handlers New request handlers (data handlers like netcdf, freeform, csv) Aggregation engines Methods of returning your data (return as netcdf) Reporters Exception Handlers Debugging

Modules are dynamically loaded into the BES Within the BES configuration you specify the modules that you want to load into the BES. The module is derived from the C++ abstract class BESAbstractModule. The class impelements a c-style function called maker that instantiates the module class The class implements an initialization and termination method to set up and clean up the module. The module can add named handlers to the different pieces of the BES. The module can extend already existing modules, for example extending the DAP2 module. The module can modify already existing modules, such as changing the way certain responses are built.

BES Configuration File /usr/local/etc/bes/bes.conf

Running what we have Let’s see what we have already installed besctl start (if it’s not already running) bescmdln [-h localhost -p 10022] show help; show version; show keys; set container in catalog values c, /data/nc/fnoc1.nc; define d as c; get dds for d; define d as c with c.constraint=“u”; get dds for d; get ascii for d;

Or use the OLFS ${CATALINA_HOME}/bin/startup.sh Start your browser Localhost:8080/opendap/ Select data, then nc, then fnoc1.nc Click the check box next to the variable u Click the button ‘Get ASCII’

The CSV Data Handler Module $HOME/src/hyrax-1.9/src/modules/csv_handler "Station ","latitude ","longitude ","temperature_K ","Notes ” "CMWM",-34.7,23.7,264.3, "BWWJ",-34.2,21.5,262.1,"Foo” "CWQK",-32.7,22.3,268.4, "CRLM",-33.8,22.1,270.2,"Blah” "FOOB",-32.9,23.4,269.69,"FOOBAR"

Setting up the example cd $HOME/src/hyrax-1.9/src/modules/csv_handler Autoreconf -fiv./configure make; make check; make install –This will build and install the module library and the sample data –But, remember on the VM, this has all been done already…

BES Configuration File /usr/local/etc/bes/bes.conf

Restart it all [${CATALINA_HOME}/bin/shutdown.sh] besctl stop besctl start [${CATALINA_HOME}/bin/startup.sh] Back in the browser –localhost:8080/opendap/ –Select data, then csv, then temperature.csv –Click the button ‘Get ASCII’

CSVModule class Example Module class $HOME/src/hyrax- 1.9/src/modules/csv_handler Implements initialize and terminate methods (and dump) Adds: (typical for data handlers) –new request handler –new catalog –new container storage using the catalog

Containers A container represents data –Typically a data file –Can represent a request to a remote system, such as WCS Has three main parts: 1.Symbolic name (cool_data) 2.Real name - e.g. filename 3.Data Type (nc, h4, h5, ff, csv) - matches the name of the data/request handler Derive from the abstraction BESContainer BESFileContainer built in WCSContainer

Accessing the container Implement the access method: virtual string access() ; Returns the data container to be accessed, typically full path to a file. –BESFileContainer returns the stored file name –WCSContainer makes the WCS request to the remote server, stores the resulting data file, and returns the path to that resulting, cached, data file.

Storing Containers The default storage of containers is volatile for the duration of the session Can create new persistent stores: –Store in database per user or group or project Create class inherited from BESContainerStorage Give the new class a unique name (user_store) In the Module class add this new storage class to the container storage list BESContainerStorageList::TheList()->add_persistence( storage ) ; Create containers in that storage set container in user_store values s,r,t;

Storing a container virtual void add_container( const string &sym_name, const string &real_name, const string &type ) ; BESContainerStorageCatalog uses the catalog to create the full path to the file and uses the file extension to determine the data type. WCSContainerStorage creates a WCSContainer whose access method makes the WCS request on the remote machine, caches the resulting file, and returns that as the file to access. BESContainerStorageVolatile simply stores the container in a volatile stl map for the duration of the session.

Definitions Like a view of data Can encompass many containers (available soon) Can specify constraints for each container Can specify data attributes from each container Specify aggregation engine and command define d as [,…, ] [with.constraint=“…”,.attributes=“…”,.constraint=“…”,.attributes=“…” aggregate using by ];

Storing Definitions Default storage is volatile for the duration of the session Can create new persistent stores: –Store in database per user or group or project Create class inherited from BESDefinitionStorage Give the new class a unique name (user_store) In the Module class add this new storage to the definition storage list DefinitionStorageList::TheList()->add_persistence( storage ) ; Create definitions in that storage define d in user_store as …

Storing a Definition virtual void add_definition( const string &def_name, BESDefine *def ) ; BESDefinitionStorageVolatile simply stores the definition in a volatile stl map for the duration of the session.

Dataset Catalogs OLFS uses BES and THREDDS catalogs for dataset catalog traversal Can have multiple catalogs in a single BES representing any type of catalog (coming soon) –Filesystem –Database catalog Each Catalog is configurable Catalogs inherit from the abstraction BESCatalog Add to the list of catalogs in the module class: BESCatalogList::TheCatalogList()->add_catalog( catalog ) ;

Typical Catalog Configuration Data handlers, e.g. nc_module, add a BESCatalogDirectory representing a file system. In the BES configuration file: –BES.Catalog.catalog.RootDirectory - data root directory of the catalog named “catalog” –BES.Catalog.catalog.TypeMatch - data types matched by file extensions –BES.Catalog.catalog.Include - what nodes to include in the catalog list (usually everything) –BES.Catalog.catalog.Exclude - what nodes to exclude (e.g. dot files) Let’s look at $prefix/etc/bes/bes.conf

Catalogs and Containers Can create containers within the context of catalogs using BESContainerStorageCatalog Uses BES.Catalog.catalog.TypeMatch to determine data type. set container in catalog values c, ;

Putting it together so far

The Interface There is a single interface into the BES. Inherits from the abstraction BESInterface –Installed version uses BESCmdInterface –CEDAR uses BESApacheInterface –Creating BESXMLInterface Interface runs a set of steps: –initialize the BES environment –build the execution plan –execute the request plan building the response object –transmit the response object –log the status of the request –report on the request –end the request

BESDataHandlerInterface BESDataHandlerInterface structure is carried throughout the execution of the commands –Holds on to the response handler –Stores the list of containers –Holds the action being taken –Holds map of name/value pairs to be used during execution –Holds an error object if an error/exception occurs

Initialize/Termination callbacks Can register callbacks during the initialization and end steps static void BESInterface::add_init_callback( p_bes_init init ) ; static void BESInterface::add_end_callback( p_bes_end end ) ; typedef bool (*p_bes_init)( BESDataHandlerInterface &dhi ) ; typedef void (*p_bes_end)( BESDataHandlerInterface &dhi ) ; Examples: –Authentication/authorization –Initialize database connections –Clean up database connections and files

Building the request plan Derived classes of BESInterface implement: virtual void build_data_request_plan() ; –BESCmdInterface parses the incoming command string –BESApacheInterface translates information from apache and parses the incoming command string –BESXMLInterface parses the XML document Sets up a response handler to execute the command

Commands String commands sent from client to server Built in commands: –show help; (version, process, status, keys) –set container … –show containers; –delete container ; –delete containers; –define … –show definitions; –delete definition ; –delete definitions; –set context to ; –show context;

DAP Commands DAPCommandModule adds: –get das for [return as type]; –get dds … –get ddx … –get dods … –show catalog [for “node”]; (info) dap-server modules (www,usage,ascii) add: –get info_page … –get ascii … –get html_form … Data handlers (nc, ff, csv) don’t add any new commands.

Response Objects Derived from BESResponseObject. –BESDASResponseObject –BESInfo No methods required to be implemented Created by a BESResponseHandler Filled in by BESResponseHandler or delegated to BESRequestHandler

Informational Response Objects Built in informational response objects derived from BESInfo: 1.BESTextInfo 2.BESHTMLInfo 3.BESXMLInfo 4.BESSilentInfo Each one formats an informational response according to its type Can add new informational response objects. BESInfoList::TheList()->add_info_builder( name, function ) ; Function instantiates the derived BESInfo object. Created in BESResponseHandler instances by calling: BESInfo *info = BESInfoList::TheList()->build_info() ; Set in BES configuration file: BES.Info.Type= e.g. txt, html, xml

Response Handlers: represent a specific response, such as a DAS response, a DDS response, a help response... know how to create the response object (DAS, DDS, BESInfo) do not necessarily fill in the response object, but know how the response should be filled in BESDASResponseHandler knows to create a DAS object and that it needs to go to each request handler for each container. A version response handler knows to create an informational response object and that it needs to go to all registered request handlers. A status response handler knows to create a text response object and fill it in with the status of the server. know how to transmit the response object

Implementing a Response Handler Inherits from the C++ abstract class BESResponseHandler (e.g. BESDASResponseHandler, BESHelpResponseHandler) Implement the methods: virtual void execute( BESDataHandlerInterface &dhi ) ; –Creates the response object –Does the work to fill in the response object, or delegate to a request handler to fill in the response object virtual void transmit( BESTransmitter *transmitter, BESDataHandlerInterface &dhi ) ; –Transmits the resulting response object using the appropriate method on the transmitter (more later)

Request Handlers Derived from BESRequestHandler Fills in response objects Register functions that say “I know how to fill in some response”

Registering a Request Handler Inside the Module class the request handler is registered with the BES BESRequestHandlerList::TheList()->add_handler( name, handler ) CSVModule::initialize() BESRequestHandlerList::TheList()-> add_handler( modname, new CSVRequestHandler( modname ) ) ;

I know how Inside the constructor for the request handler you register functions that know how to fill in responses add_handler( “name”, function ) CSVRequestHandler::CSVRequestHandler add_handler( DAS_RESPONSE, CSVRequestHandler::csv_build_das ) ; add_handler( DDS_RESPONSE, CSVRequestHandler::csv_build_dds ) ; add_handler( DATA_RESPONSE, CSVRequestHandler::csv_build_data ) ; add_handler( VERS_RESPONSE, CSVRequestHandler::csv_build_vers ) ; add_handler( HELP_RESPONSE, CSVRequestHandler::csv_build_help ) ;

How does it work Definitions def as c1, c2 get das for def Containers c1,file1,csv c2,file2,csv ce,file3,nc c4,file4,ff def command Response Handlers das, BESDASResponseHandler help, BESHelpResponseHandler Request Handlers csv, CSVRequestHandler das, fill_das_func help, fill_help_func das csv das c1 das c1 c2 das csv

The CSV Data Handler Module $HOME/src/hyrax-1.9/src/modules/csv_handler Look at CSVModule –Adds CSVRequestHandler –Adds BESCatalogDirectory –Adds BESContainerStorageCatalog Look at CSVRequestHandler –Adds functions to fill in DAS, DDS, DataDDS, Version and Help responses These functions fill in the respective response objects

Transmit Sends the response object back to the client Two built in transmitters –BESBasicTransmitter –BESBasicHTTPTransmitter (adds header info) –These implement methods to transmit informational responses (BESInfo) BESDapTransmit implements static methods to send DAS, DDS, and DataDDS responses –Adds named static functions to the two built in transmitters BESResponseHandler instances know what functions to call on the transmitter to transmit their response objects

Adding Transmit Functions BESDapTransmit BESTransmitter *t = BESReturnManager::TheManager()-> find_transmitter( BASIC_TRANSMITTER ) ; if( t ) { t->add_method( DAS_TRANSMITTER, BESDapTransmit::send_basic_das ) ; t->add_method( DDS_TRANSMITTER, BESDapTransmit::send_basic_dds ) ; t->add_method( DDX_TRANSMITTER, BESDapTransmit::send_basic_ddx ) ; t->add_method( DATA_TRANSMITTER, BESDapTransmit::send_basic_data ) ; }

Adding Transmit Functions BESTransmitter *t = BESReturnManager::TheManager()-> find_transmitter( HTTP_TRANSMITTER ) ; if( t ) { t->add_method( DAS_TRANSMITTER, BESDapTransmit::send_http_das ) ; t->add_method( DDS_TRANSMITTER, BESDapTransmit::send_http_dds ) ; t->add_method( DDX_TRANSMITTER, BESDapTransmit::send_http_ddx ) ; t->add_method( DATA_TRANSMITTER, BESDapTransmit::send_http_data ) ; }

Adding new Transmitter Inherit from BESTransmitter Implement send_text, send_html Add static functions to transmit other response objects in the constructor E.G. ESG project, transmitter to return data object as a netcdf file get dods for return as netcdf;

Aggregation You can have multiple aggregation engines installed in the BES Inherit from BESAggregationServer Implement aggregate method: virtual void aggregate( BESDataHandlerInterface &dhi ) ; In Module class add aggregation function that instantiates your instance BESAggFactory::TheFactory()->add_handler( name, function ) ; typedef BESAggregationServer *(*p_agg_handler)(string name);

Reporters After command has been executed and response object transmitted Inherit from BESReporter Implement method: virtual void report( const BESDataHandlerInterface &dhi ) ; Register with reporter list: BESReporterList::add_reporter( name, reporter_instance ) ;

Exception Handling Register exception handler functions with the Exception Manager virtual void add_ehm_callback( p_bes_ehm ehm ) ; typedef int (*p_bes_ehm)( BESException &e, BESDataHandlerInterface &dhi ) ; BESExceptionManager::TheEHM->add_ehm_callback( func ) ; When exception caught, passed to Exception Manager 1.Iterates through registered functions 2.If not handled, default handler Set error_info in BESDataHandlerInterface – This is transmitted in place of response object

Informational and Debug Logging in the BES The BES has two forms of logging. 1.Event logging in a bes.log file. This is configured in the bes.conf file: BES.LogName=/path/to/bes.log BES.LogVerbose=no 2.Debug logging for the BES besctl start -d “/path/to/bes.debug,nc,bes,ppt” Logging and Debugging

BES Debugging 1.Format of the command line option -d “cerr|,,…, ” E.g. -d “/tmp/bes.debug,nc,bes” Specify a context of “all” to turn on debugging for everything 2.Writing debug statements in your code Macro BESDEBUG( “ ”, ) E.g. BESDEBUG( “nc”, “reading DAS for “ << file << endl ) Block of debugging If( BESDebug::IsSet( “nc” ) ) { list ::const_iterator i = _somelist.begin() ; while( ; i != _somelist.end(); i++ ) BESDEBUG( “nc”, “ “ << (*i) ) }

BES Debugging Most BES classes derived from BESObj, and some libdap class derived from DapObj You can dump BESObj instances and DapObj instances to ostream virtual void dump( ostream &strm ) ; E.g. DAS &das ; BESDEBUG( “nc”, “DAS = “ << endl << das ) Allows for indentation as well Register context within your module: BESDebug::Register( “nc” ) ;

BES Debugging - dump method example

Hello World cd src/bes/hello_world Autoreconf -fiv./configure –prefix=$prefix make; make check; make install –NB: For this software, you need to do this besctl start bescmdln [-h localhost -p 10022] say hello to world;

Writing the code: names SampleResponseNames.h –#define SAY_WHAT “say_what” –#define SAY_TO “say_to”

Writing the code: command SamplesayCommand.cc dhi.data[SAY_WHAT] = my_token ; // Next token should be the token "to” my_token = tokenizer.get_next_token() ; if( my_token != "to" ) { tokenizer.parse_error( my_token + " not expected\n" ) ; } my_token = tokenizer.get_next_token() ; if( my_token == ";" ) { tokenizer.parse_error( my_token + " not expected\n" ) ; } dhi.data[SAY_TO] = my_token ;

…Reporter Keep track of what is being said to whom cp../hello_world/SayReporter.*. Edit SampleModule.cc –Add includes #include “BESReporterList.h” #include “SayReporter.h” –Add Reporter at bottom of initialize BESDEBUG( “ adding Say reporter” << endl ) BESReporterList::TheList()-> add_reporter( modname, new SayReporter ) ;

Build, restart, say hello Add to Makefile.am BES_SRCS = SayReporter.cc \ BES_HDRS = SayReporter.h \ Edit bes.conf at bottom add Say.LogName=./say.log make besctl stop besctl start bescmdln -h localhost -p –say hello to world; –exit cat say.log