Near Real Time Ocean Observations Online the Detailed Escape of SEACOOS (Southeastern Atlantic Coastal Ocean Observing System) Data Management and Visualization Secrets

Near Real Time Ocean Observations Online : SEACOOS Software : engines Mapserver Mapserver Open source from University of Minnesota Open source from University of Minnesota USC runs a mixture of versions from 3.6 to 4.2 USC runs a mixture of versions from 3.6 to 4.2 PostgreSQL PostgreSQL Open source from postgresql.org Open source from postgresql.org USC runs version USC runs version PostGIS PostGIS Open source from Refractions.net Open source from Refractions.net USC runs version USC runs version 0.8.1

Near Real Time Ocean Observations Online : SEACOOS Software : additional PHP PHP Open source from php.org Open source from php.org USC runs version USC runs version Perl Perl Open source from perl.org Open source from perl.org USC runs version USC runs version Miscellaneous Miscellaneous ANiS and gifsicle ANiS and gifsicle Imagemagick Imagemagick

Near Real Time Ocean Observations Online : SEACOOS Hardware application server application server Apache 2.x Apache 2.x MapServer MapServer Perl, PHP, misc. Perl, PHP, misc. database A database A In-situ In-situ RS RS database B database B In-situ In-situ model output model output data scout data scout In-situ In-situ RS RS

Near Real Time Ocean Observations Online : SEACOOS Directory structure Data that sits on USC filesystems Data that sits on USC filesystems RS images RS images cached images cached images All files have strict naming convention that includes timestamp All files have strict naming convention that includes timestamp

Near Real Time Ocean Observations Online : SEACOOS System administration Databases backed up nightly Databases backed up nightly requires some downtime (~ 2 hours) requires some downtime (~ 2 hours) Databases cleaned up and optimized nightly Databases cleaned up and optimized nightly significant overhead ~ 4 hours significant overhead ~ 4 hours Some backend data massaging (mainly for model output aggregation) Some backend data massaging (mainly for model output aggregation)

Near Real Time Ocean Observations Online : SEACOOS Database structure One table category per data type One table category per data type e.g. in-situ winds e.g. in-situ winds wind_prod contains all wind data wind_prod contains all wind data wind_map contains wind data appropriate for maps wind_map contains wind data appropriate for maps e.g. QuikSCAT winds e.g. QuikSCAT winds more complicated since requires aggregation and normalization more complicated since requires aggregation and normalization e.g. OI SST e.g. OI SST table containing pointers to data files on disk table containing pointers to data files on disk Ancillary tables for specific data types Ancillary tables for specific data types e.g. OI SST e.g. OI SST table containing RGB to SST lookup values (for querying purposes) table containing RGB to SST lookup values (for querying purposes)

Near Real Time Ocean Observations Online : SEACOOS Data processing overview Data scout (netCDF) Data scout (netCDF) Perl code flattens incoming netCDF into arrays which are turned into SQL INSERT statemtents. Perl code flattens incoming netCDF into arrays which are turned into SQL INSERT statemtents. Triggers update the new records suitable for normalization as well as Mapserver display elements. Triggers update the new records suitable for normalization as well as Mapserver display elements. Procedures run to optimize tables for display. Procedures run to optimize tables for display. RS images (HDF to PNG) RS images (HDF to PNG) begin as HDF but are availed to USC as PNG’s begin as HDF but are availed to USC as PNG’s Images have standard naming convention and agreed upon extents as well as predefined RGB to value, e.g. RGB to SST, pairs. Images have standard naming convention and agreed upon extents as well as predefined RGB to value, e.g. RGB to SST, pairs. Incoming files cause postgreSQL table to be updated with the new file and timestamp. Incoming files cause postgreSQL table to be updated with the new file and timestamp.