EPIC Tools for in-situ data collections Donald W. Denbo, NOAA/PMEL Willa H. Zhu, NOAA/PMEL/JISAO John Osborne, OceanAtlas Software Christopher Moore, NOAA/PMEL/JISAO NOAA Coastal Storms Initiative Meeting May 6-8, 2003

EPIC Components  Data Archive and Retrieval  Multiple file formats supported with “epslib”  Metadata database supports subsetting of large data collections  “Classic EPIC”  Unix programs for desktop display, analysis  Gateways to Matlab, PPLUS, Ferret, IDL, etc.  EPIC Web Browser  Search, graph, list, download from the Web  Toolkit for building custom web applications  Java applications  Java OceanAtlas - Sophisticated browse and data exploration  Climate Data Portal - distributed data access (CORBA/OPeNDAP)  OceanShare - Network data collaboration tool  ncBrowse - Generalized browse tool for data/attributes (local files, network files, OPeNDAP files)

EPIC Archive and Retreival for Classic EPIC, Web, and Java Applications  Data collections may include thousands or even millions of profiles or time series  Typically the researcher will use EPIC tools to:  Select data by lat, lon, time, keyword, etc.  Refine the initial data selection  Display and/or analyze the selected data

Pacific NODC Alaska Meta Data Data File Collections Data Loader Utility Selected datasets pointers to data files Data Search Lat, long, time keyword, etc creates Data Search and Selection “Classic EPIC” programs Matlab Ferret IDL PPLUS EPIC Web Java OceanAtlas Climate Data Portal OceanShare ncBrowse MySQL database EPIC Data Management Strategy EPIC data usage

EPIC Data File Input/Output Applications programs API File Independent Layer FDL ExistingPlanned  Data file format  Multiple data file formats are supported  New formats are readily added  Applications software  Independent of file format  API for most programming languages  Gateways  to Matlab and other graphics/analysis packages EPS Library

Classic EPIC Suite of Unix programs for the desktop  Over 100 EPIC programs are available for:  Data selection, metadata and file editing...  Graphics  Time series, multivariable plots, section plots...  Parameter calculation  Dynamic height, geostrophic velocity...  Statistical, spectral, tidal harmonic analyses...  EPIC’s modularity allows a researcher to easily add new analysis techniques or use a favorite graphics or analysis program.

EPIC Web Browser

EPIC Web Browser Data selection, graphics, download

EPIC Web Browser Capabilities  Plots  User customized plots for profiles and time series  (T, S, Sigma-t, Ox) vs Pressure/Depth  Temperature vs Salinity  One or two user-selected Properties vs Pressure/Depth  Contoured section plots  List/download data, metadata  New plots and functions readily included

EPIC tools for the Web for building customized web access to data EPIC Web Toolkit EPIC Web Toolkit   Interface design   Backend graphics   Map data selector Example applications Example applications EPIC Web Browser (general) TAO (El Nino buoys) Unaami (Arctic time series) OSCAR (Satellite ocean currents) Multibeam data (hydrography)

EPIC Java Tools and Applications Distributed data access and Advanced data browsing

Java Tools EPIC Java Applications for locating and selecting from distributed data collections

Challenge  Locating and selecting individual observations from large, distributed collections of in situ data observations   Examples: Individual oceanic or atmospheric profiles, time series, buoy tracks, etc. EPIC Java Applications

Some solutions  Tools and Applications   LDAP Directory Services   NdEdit for subsetting large in-situ collections   Interactive Java graphics   Intensive exploration of multiple data types   OPeNDAP (DODS) clients and servers for in-situ data   Access from Desktop and Web  Assembled from OpenSource components   EPIC Java Applications

LDAP Directory Service Finding distributed data collections  Search on geographic region, observed quantities or keyword  Automatically populated from data servers  Metadata instantly known to LDAP directories  Access control for experimental data products  Open Source  Standards-based  Platform independent  Scalable, extensible  Supports desktop or web client EPIC Java Applications

X-Y map view Y-T Y-T time range NdEdit Investigate space-time distribution Select data in any View Explore Data Locations and Subset Data Collections Interactive, linked windows View any 2D sliice xy, xt, xz, yt, yz, yt Select time range See data in range X-Y map view Y-T Y-T time range EPIC Java Applications

Section Tool - select stations along a depth contour NdEdit EPIC Java Applications

Selected stations are in a narrow time band NdEdit EPIC Java Applications

Polygon tool – select an oblique section line See time range of stations NdEdit EPIC Java Applications

NdEdit  NdEdit is an interactive Java tool  NdEdit is being re-engineered for the Web EPIC Java Applications

SGT: A Scientific Graphics Toolkit  Java toolkit for scientific graphics  On-line documentation, examples, tutorials, streaming video  JavaBeans in next release  Features  Contour, line and vector plots,  Axes user settable or automatically scaled  multiple Time Axis formats  mouse-movable, customizable Labels, line keys, icons  Legend generation automatic  Widely used, freely available  scientific and commercial applications use SGT  14,782 downloads by 5,151 unique sites from 70 countries  EPIC Java Applications

Java Applications for the Desktop and the Network EPIC Java Applications

ncBrowse Java tool for data/metadata browse  NetCDF and full OPeNDAP support   Desktop Java application   Axis mapping, parametric plots, flexible mapping of independent variables to axes   Line, vector and contour plots   Animations, 3D graphics via VisAD  Popular, freely available   8,187 downloads by 3,415 unique sites in 59 countries   OPeNDAP (DODS) client, Unidata netCDF client, WOCE data CD, etc.   EPIC Java Applications

Java OceanAtlas for browsing ocean data  Intensive Java desktop browsing of ocean data  Property-property plots with up to 7 x axes, waterfall diagrams, Contour plots/residual contour plots, Map plots with optional "bubble" isosurface plots, Linked browsing of all plot types, Built-in and custom calculations, NdEdit for data location/selection  Reads WOCE, EPIC, OPeNDAP netCDF data, some ARGO formats  Extended for wide range of data types  Profile, time series, drifting buoy, gridded analyses, model outputs  Widely used, freely available   EPIC Java Applications

Climate Data Portal Distributed ocean observations  Major in situ data collections:   TAO El Nino buoy data (realtime/PMEL), Global sea level data (realtime/U of Hi), GTSPP (NODC), PMEL hydrographic data and time series collection, WOCE ADCP (U of Hi), IPRC/APDRC in situ data, US Coastal Ocean Time Series database (NODC) OPeNDAP: in-situ data server and client OPeNDAP: in-situ data server and client LDAP directory services NdEdit for data selection/subsetting Data Servers, Clients (Java and Web) – focus on in situ data EPIC Java Applications

OceanShare Network collaboration tool  Collaboration features  Local, remote data file access and Java graphics  Standard collaboration tool features (annotation, whiteboard, etc)  Secure LDAP Document Repository  Proposals, papers in progress, data, images  Access control by document or by folder  Accessible from web and desktop  OPeNDAP compatible soon  EPIC Java Applications

EPIC Java Tools and Applications  Modular Web and Open Source components  Java interactivity  Low cost development/maintenance  OpenSource, standards-based  Scalable, extensible  OPeNDAP supported or planned  