1. A Distributed Approach to Ocean, Atmosphere & Climate Model Data Interoperability Rich Signell USGS Woods Hole, MA COAWST Training Workshop Woods Hole, MA: Aug 25-28, 2014 Rich Signell USGS Woods Hole, MA COAWST Training Workshop Woods Hole, MA: Aug 25-28, 2014

2. 2

3. US Integrated Ocean Observing System (IOOS ® ) IOOS ® Plan defines: IOOS ® Plan defines: Global ComponentGlobal Component Coastal ComponentCoastal Component  17 Federal Agencies  11 Regional Associations

4. The problem: lots of unique (stovepipe) web sites The GoMOOS Nowcast/Forecast Circulation Model (University of Maine)

5. IOOS Core Principles Adopt open standards & practices Avoid customer-specific stovepipes Standardized access services implemented at data providers Adopt open standards & practices Avoid customer-specific stovepipes Standardized access services implemented at data providers 5 Customer Web access service DataProvider Observations Models

6. Issue: Ocean grids are not regularly spaced! Stretched surface and terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates

7. NetCDF Climate and Forecast (CF) Conventions provide a solution Groups using CF: GO-ESSP: Global Organization for Earth System Science Portal IOOS: Integrated Ocean Observing System ESMF: Earth System Modeling Framework OGC: Open Geospatial Consortium (GALEON: WCS profile)

8. CF-Compliant ROMS file double s_rho(s_rho) ; s_rho:long_name = "S-coordinate at RHO-points" ; s_rho:valid_min = -1. ; s_rho:valid_max = 0. ; s_rho:positive = "up" ; s_rho:standard_name = "ocean_s_coordinate_g1" ; s_rho:formula_terms = "s: s_rho C: Cs_r eta: zeta depth: h depth_c: hc" ; s_rho:field = "s_rho, scalar" ; float temp(ocean_time, s_rho, eta_rho, xi_rho) ; temp:_FillValue = 1.e+37f ; temp:long_name = "potential temperature" ; temp:units = "Celsius" ; temp:time = "ocean_time" ; temp:coordinates = "lon_rho lat_rho s_rho ocean_time" ;

9. IOOS Model Data Interoperability Design ROMS ADCIRC HYCOM SELFE NCOM NcML Common Data Model OPeNDAP+CF WCS NetCDF Subset THREDDS Data Server Standardized (CF-1.6, UGRID-0.9) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python ERDDAP NetCDF-Java SOS Geoportal Server GeoNetwork GI-CAT Observed data (buoy, gauge, ADCP, glider) Godiva2 CKAN NcML Grid Ugrid TimeSeries Profile Trajectory TimeSeriesProfile Nonstandard Data Files Catalog Services

10. WMS Browsing with THREDDS/ncWMS

11. Matlab NCTOOLBOX https://github.com/nctoolbox Objective: Make it simple to access CF data Example function: [t, geo]=nj_tslice(URL,’temp’,1); t = 22x120x180 single geo = –lat: [120x180 single] –lon: [120x180 single] – z: [22x120x180 double] –time: 733582 (matlab datenum) nj_tslice works identically for ROMS, POM, ECOM, WRF, Wavewatch3 URL can be: local NetCDF, remote NetCDF, NcML, OpenDAP Data URL Objective: Make it simple to access CF data Example function: [t, geo]=nj_tslice(URL,’temp’,1); t = 22x120x180 single geo = –lat: [120x180 single] –lon: [120x180 single] – z: [22x120x180 double] –time: 733582 (matlab datenum) nj_tslice works identically for ROMS, POM, ECOM, WRF, Wavewatch3 URL can be: local NetCDF, remote NetCDF, NcML, OpenDAP Data URL

12. Comparing Models with Data in Matlab Model 1: UMASS-ECOM Model 2: UMAINE-POM Data: SST 2008-Sep-08 07:32

13. Skidaway modena glider (Sep 11-23)

14. compare_secoora_model_sections.m (using nc_genslice.m)

15. Buoy_comp.m

16. 3D visualization of data with IDV

17. NetCDF Subset Service to CSV

19. CSV Time Series from Subset Service CSV request is a “RESTful” URL: http://geoport.whoi.edu/thredds/ncss/grid/coawst_4/use/fmrc/coawst_4_use_best.ncd?var =Hwave&latitude=39.5&longitude=-69.5&time_start=2014-08- 27T01:00:00Z&time_end=2014-08-28T00:00:00Z&vertCoord=&accept=csv

20. Accessing the CSV data in Excel

21. Access the CSV data in Python, R, etc

22. Iris Python tools from the UK Met Office

23. ESPreSSO model (Rutgers)

24. Global RTOFS (NOAA)

25. IOOS Models Notebook on Wakari

26. 60 minutes to serving your data 1.Install Sun/Oracle Java (10 min) 2.Install/configure Tomcat (15 min) 3.Install/configure the Thredds Data Server (15 min) 4.Put NetCDF (or HDF4, Grib1, Grib2, HDF5 files) in a directory (5 min) 5.Add NcML (XML) files for CF compliance and aggregation (15 min)

27. Adding data to THREDDS rsignell@gam:/usgs/data0/bbleh/tidal$ ls *wide* … his_bbleh_wide_0048.nc his_bbleh_wide_0100.nc his_bbleh_wide_0152.nc his_bbleh_wide_0049.nc his_bbleh_wide_0101.nc his_bbleh_wide_0153.nc his_bbleh_wide_0050.nc his_bbleh_wide_0102.nc his_bbleh_wide_0154.nc his_bbleh_wide_0051.nc his_bbleh_wide_0103.nc wide.ncml $more wide.ncml

29. Browsing WMS with Godiva2

30. GI-CAT catalog broker service architecture Searching for Data

32. USGS COAWST East Coast Forecast

35. Automated model comparison

36. Summary (1 of 2) Common data models for “feature types” (structured and unstructured grids, time series, profiles, swaths) (Unidata CDM) Standard web services for delivering these data and metadata (OGC, Unidata) Tools to access and process these services in common analysis environments: R, Matlab, Python, ArcGIS, JavaScript

37. Summary (2 of 2) It’s easy (1 hour) to deploy free, supported systems that allow for standards-based delivery of aggregated data from native model grids that put little effort on the data provider What do you get? –Lots of choices for data access (Browser, Matlab, Python, Excel, IDV, R, IDL) –More usage of model results by more people –Faster feedback to modelers, leading to improved models –Shared code base in the community –Increased community support for standards-based access –Less time wasted messing with data, more time spent on ecosystem based management What should you do? Encourage providers to use these standards, and develop tools that use standardized access It’s easy (1 hour) to deploy free, supported systems that allow for standards-based delivery of aggregated data from native model grids that put little effort on the data provider What do you get? –Lots of choices for data access (Browser, Matlab, Python, Excel, IDV, R, IDL) –More usage of model results by more people –Faster feedback to modelers, leading to improved models –Shared code base in the community –Increased community support for standards-based access –Less time wasted messing with data, more time spent on ecosystem based management What should you do? Encourage providers to use these standards, and develop tools that use standardized access

38. It’s all on GitHub + rsignell

40. Glossary (1/6) CF Conventions: NetCDF Climate and Forecast Metadata Conventions http://cf-pcmdi.llnl.gov/ (Standards for identifying geospatial coordinates in NetCDF files and grid relationships) DMAC: Data Management And Communications subsystem http://www.ioos.gov/data/dmac/welcome.html (One of three subsystems of IOOS, providing infrastructure for the Observing and the Modeling and Applications subsystems) ECOM: Estuarine, Coastal, and Ocean Model. http://www.hydroqual.com/ehst_env_hyd.html (3D ocean circulation model developed principally by Alan Blumberg) ERDDAP: Environmental Research Division's Data Access Program http://coastwatch.pfeg.noaa.gov/erddap/index.html (A web service that aggregates data from diverse remote sources and offers a simple, consistent way to access the data) CF Conventions: NetCDF Climate and Forecast Metadata Conventions http://cf-pcmdi.llnl.gov/ (Standards for identifying geospatial coordinates in NetCDF files and grid relationships) DMAC: Data Management And Communications subsystem http://www.ioos.gov/data/dmac/welcome.html (One of three subsystems of IOOS, providing infrastructure for the Observing and the Modeling and Applications subsystems) ECOM: Estuarine, Coastal, and Ocean Model. http://www.hydroqual.com/ehst_env_hyd.html (3D ocean circulation model developed principally by Alan Blumberg) ERDDAP: Environmental Research Division's Data Access Program http://coastwatch.pfeg.noaa.gov/erddap/index.html (A web service that aggregates data from diverse remote sources and offers a simple, consistent way to access the data)

41. Glossary (2/6) ESMF: Earth System Modeling Framework http://www.esmf.ucar.edu/ (Software for building and coupling weather, climate, and related models) GO-ESSP: Global Organization for Earth System Science Portal http://go-essp.gfdl.noaa.gov/ (Collaboration designed to develop a new generation of software infrastructure that will provide distributed access to observed and simulated data from the climate and weather communities) IDV: Integrated Data Viewer http://www.unidata.ucar.edu/software/idv/ (Java-based software for analyzing and visualizing geoscience data) IOOS: Integrated Ocean Observing System http://ioos.noaa.gov/ (Multidisciplinary system designed to collect, deliver, and use ocean information) IOOC: Interagency Ocean Observing Committee http://www.iooc.us/ ESMF: Earth System Modeling Framework http://www.esmf.ucar.edu/ (Software for building and coupling weather, climate, and related models) GO-ESSP: Global Organization for Earth System Science Portal http://go-essp.gfdl.noaa.gov/ (Collaboration designed to develop a new generation of software infrastructure that will provide distributed access to observed and simulated data from the climate and weather communities) IDV: Integrated Data Viewer http://www.unidata.ucar.edu/software/idv/ (Java-based software for analyzing and visualizing geoscience data) IOOS: Integrated Ocean Observing System http://ioos.noaa.gov/ (Multidisciplinary system designed to collect, deliver, and use ocean information) IOOC: Interagency Ocean Observing Committee http://www.iooc.us/

42. Glossary (3/6) NcWMS: Web Map Service http://www.resc.rdg.ac.uk/trac/ncWMS/ (ncWMS is a Web Map Service for geospatial data that are stored in CF-compliant NetCDF files. ) NcML: NetCDF Markup Language http://www.unidata.ucar.edu/software/netcdf/ncml/ (Used to standardize and aggregate non-standard NetCDF files) NetCDF-Java: NetCDF-Java Library http://www.unidata.ucar.edu/software/netcdf-java/ (Library that takes advantage of CF conventions, NcML to facilitate building standards-based clients) NcWMS: Web Map Service http://www.resc.rdg.ac.uk/trac/ncWMS/ (ncWMS is a Web Map Service for geospatial data that are stored in CF-compliant NetCDF files. ) NcML: NetCDF Markup Language http://www.unidata.ucar.edu/software/netcdf/ncml/ (Used to standardize and aggregate non-standard NetCDF files) NetCDF-Java: NetCDF-Java Library http://www.unidata.ucar.edu/software/netcdf-java/ (Library that takes advantage of CF conventions, NcML to facilitate building standards-based clients)

43. Glossary (4/6) OPeNDAP: Open-source Project for a Network Data Access Protocol http://opendap.org (allows efficient hyperslabbing of geospatial data) POM: Princeton Ocean Model http://www.aos.princeton.edu/WWWPUBLIC/htdocs.pom/ (Sigma coordinate, free surface ocean model with embedded turbulence and wave sub-models, and wet-dry capability) RCOOS: Regional Coastal Ocean Observing System http://www.ioosassociation.org/ (Generic term for one of the 11 regional observing systems in IOOS) OPeNDAP: Open-source Project for a Network Data Access Protocol http://opendap.org (allows efficient hyperslabbing of geospatial data) POM: Princeton Ocean Model http://www.aos.princeton.edu/WWWPUBLIC/htdocs.pom/ (Sigma coordinate, free surface ocean model with embedded turbulence and wave sub-models, and wet-dry capability) RCOOS: Regional Coastal Ocean Observing System http://www.ioosassociation.org/ (Generic term for one of the 11 regional observing systems in IOOS)

44. Glossary (5/6) ROMS: Regional Ocean Modeling System www.myroms.org (A free-surface, terrain-following, primitive equations ocean model widely used by the scientific community for a diverse range of applications) SCCOOS: Southern California Coastal Ocean Observing System http://www.sccoos.org/ (Brings together coastal observations in the Southern California Bight to provide information necessary to address issues in climate change, ecosystem preservation and management, coastal water quality, maritime operations, coastal hazards and national security. One of the 11 IOOS regions.) THREDDS (Thematic Realtime Environmental Distributed Data Services) http://www.unidata.ucar.edu/projects/THREDDS (Services for delivering standards-based geospatial data via OpenDAP, WCS and more, works with NcML) ROMS: Regional Ocean Modeling System www.myroms.org (A free-surface, terrain-following, primitive equations ocean model widely used by the scientific community for a diverse range of applications) SCCOOS: Southern California Coastal Ocean Observing System http://www.sccoos.org/ (Brings together coastal observations in the Southern California Bight to provide information necessary to address issues in climate change, ecosystem preservation and management, coastal water quality, maritime operations, coastal hazards and national security. One of the 11 IOOS regions.) THREDDS (Thematic Realtime Environmental Distributed Data Services) http://www.unidata.ucar.edu/projects/THREDDS (Services for delivering standards-based geospatial data via OpenDAP, WCS and more, works with NcML)

45. Glossary (6/6) WCS: Web Coverage Service http://www.opengeospatial.org/standards/wcs (OGC standard for serving gridded information: time series, profiles, hyperslabs of remote sensing data, bathymetry, model output) WW3: WaveWatch III http://polar.ncep.noaa.gov/waves/wavewatch/wavewatch.html (Ocean wave model developed by Hendrik Tolman) WRF: Weather Research and Forecasting Model http://www.wrf-model.org/index.php (Next-generation mesocale numerical weather prediction system designed to serve both operational forecasting and atmospheric research needs) WCS: Web Coverage Service http://www.opengeospatial.org/standards/wcs (OGC standard for serving gridded information: time series, profiles, hyperslabs of remote sensing data, bathymetry, model output) WW3: WaveWatch III http://polar.ncep.noaa.gov/waves/wavewatch/wavewatch.html (Ocean wave model developed by Hendrik Tolman) WRF: Weather Research and Forecasting Model http://www.wrf-model.org/index.php (Next-generation mesocale numerical weather prediction system designed to serve both operational forecasting and atmospheric research needs)