Near Real Time Ocean Observations Online the Escape of SEACOOS (Southeastern Atlantic Coastal Ocean Observing System) Data Management and Visualization Secrets

Near Real Time Ocean Observations Online : SEACOOS The (tiny) big picture

Near Real Time Ocean Observations Online : SEACOOS

The nuts & bolts Data is aggregated and stored Data is aggregated and stored Data is normalized Data is normalized Data is visualized Data is visualized Data is disseminated Data is disseminated

Near Real Time Ocean Observations Online : SEACOOS Data aggregation & storage Format flavors Format flavors netCDF: coverage may vary; format may not vary netCDF: coverage may vary; format may not vary PNG: coverage may not vary; color scale may not vary PNG: coverage may not vary; color scale may not vary Technology Technology Perl Perl

Near Real Time Ocean Observations Online : SEACOOS Data aggregation & storage Relational database Relational database One table per in-situ variable type One table per in-situ variable type E.g. a wind_table, sst_table E.g. a wind_table, sst_table Contains the real data Contains the real data One table per raster variable One table per raster variable E.g. a sst_raster_table E.g. a sst_raster_table Contains pointers to PNG files + boundaries Contains pointers to PNG files + boundaries One table per model variable per hour One table per model variable per hour E.g. a 12_00_00_current_forecast_table E.g. a 12_00_00_current_forecast_table

Near Real Time Ocean Observations Online : SEACOOS Normalization Reporting time varies Reporting time varies In-situ data In-situ data E.g. daily, hourly, half-hourly, every 10 minutes E.g. daily, hourly, half-hourly, every 10 minutes Remotely-sensed data Remotely-sensed data E.g. twice daily E.g. twice daily Reporting area varies Reporting area varies Remotely-sensed data passes Remotely-sensed data passes

Near Real Time Ocean Observations Online : SEACOOS Visualization Main engines Main engines PHP, PHP-MapScript PHP, PHP-MapScript Perl Perl MapServer, of course MapServer, of course Icing on the cake Icing on the cake Dynamic shapefile creation for colorbars Dynamic shapefile creation for colorbars AnIS and GIFSicle for animations AnIS and GIFSicle for animations ImageMagick for image manipulation and raster queries ImageMagick for image manipulation and raster queries Mouseovers Mouseovers Time-series graphs using Gnuplot Time-series graphs using Gnuplot

Near Real Time Ocean Observations Online : SEACOOS Icing on the cake : dynamic shapefiles

Near Real Time Ocean Observations Online : SEACOOS Icing on the cake : animations AnIS and GIFsicle AnIS and GIFsicle Produce several images and glue them together Produce several images and glue them together

Near Real Time Ocean Observations Online : SEACOOS Icing on the cake : raster queries Let MapServer take care of projection issues Let MapServer take care of projection issues How to get an SST, for example, from a raster PNG How to get an SST, for example, from a raster PNG 1) Dump the current extent view of just the raster to PNG 2) Get (x,y) clicked pixel RGB of the dumped image with Perl::ImageMagick 3) Lookup pixel RGB in database to find out SST 4) Return SST in the normal query data train

Near Real Time Ocean Observations Online : SEACOOS Icing on the cake : mouseovers Use MapServer 4.x searchmap Use MapServer 4.x searchmap Produce two images for an HTML page Produce two images for an HTML page One is static; with the querymap One is static; with the querymap The other is an input image The other is an input image Use javascript to swap the two images based on user activity The querymap is produced by making a simple LWP::simple request for a querymap, and then it is merged with the HTML page

Near Real Time Ocean Observations Online : SEACOOS Icing on the cake : time series graphs Pass a query result set to Gnuplot for a time-series graph Pass a query result set to Gnuplot for a time-series graph

Near Real Time Ocean Observations Online : SEACOOS Dissemination Maps and charts, of course Maps and charts, of course OPeNDAP (DODS) data access OPeNDAP (DODS) data access OGC-friendly: WMS, WFS OGC-friendly: WMS, WFS SEACOOS pushed a lot of data to the IOOS interoperability project SEACOOS pushed a lot of data to the IOOS interoperability project Down the pipeline Down the pipeline Ad-hoc CSV and/or netCDF data extraction Ad-hoc CSV and/or netCDF data extraction

Near Real Time Ocean Observations Online : SEACOOS Magic 8 ball Everyone wants speed Everyone wants speed ‘Near real time’ implies constantly updated databases ‘Near real time’ implies constantly updated databases Recently introduced round-robin-ing of database requests Recently introduced round-robin-ing of database requests As machine A is populated with new data, send all DB requests to machine B As machine A is populated with new data, send all DB requests to machine B When machine A is done populating, have it resume DB handling, and go update machine B When machine A is done populating, have it resume DB handling, and go update machine B Want to round-robin MapServer requests, too, as well as pull application server to its own box Want to round-robin MapServer requests, too, as well as pull application server to its own box

Near Real Time Ocean Observations Online : SEACOOS A friend in the business Development site: Development site: Production site (new interface to be released soon): Production site (new interface to be released soon): Very active listserv dealing with mainly remote-sensing issues: Very active listserv dealing with mainly remote-sensing issues: mailto: mailto: message text: subscribe remotesensing message text: subscribe remotesensing Who am I? Who am I? Charlton Purvis, University of South Carolina, SEACOOS Charlton Purvis, University of South Carolina, SEACOOS Happy to help and share. Happy to help and share.