The Model Output Interoperability Experiment in the Gulf of Maine: A Success Story Made Possible By CF, NcML, NetCDF-Java and THREDDS Rich Signell (USGS, Woods Hole) Sachin Kumar Bhate (Mississippi State) Made possible by: The GOMODP Modeling Committee & The UNIDATA team! Low-hanging fruit for IOOS! (The Integrated Ocean Observing System)

The Problem The GoMOOS Nowcast/Forecast Circulation Model (University of Maine)

We want the data!

Starting Point Each group had: Each group had: A different oceanographic model: POM, ECOM, FVCOM, ROMS, WaveWatch 3 A different oceanographic model: POM, ECOM, FVCOM, ROMS, WaveWatch 3 A different Matlab toolkit for analysis A different Matlab toolkit for analysis Yet, each group also had: Yet, each group also had: NetCDF output files NetCDF output files A public web server A public web server

Typical Ocean Model Native Grid Stretched surface and terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates

Climate and Forecast (CF) Metadata Conventions Groups using CF: GO-ESSP: Global Organization for Earth System Science Portal IOOS: Integrated Ocean Observing System ESMF: Earth System Modeling Framework OGC: Open Geospatial Consortium (via Galeon WCS)

CF Conventions for curvilinear orthogonal grids

CF conventions for stretched (dimensionless) vertical coordinates

NcML to the rescue XML markup language for NetCDF

CF-Compliant Model Access in Matlab

THREDDS

GOMODP THREDDS Catalog

CF Toolkit for Matlab Objective: make it simple to access CF-compliant data Objective: make it simple to access CF-compliant data Example function: Example function: [t, geo]=cf_tslice(uri,’temp’,itime); [t, geo]=cf_tslice(uri,’temp’,itime); t = 22x120x180 single t = 22x120x180 single geo = geo = lat: [120x180 single] lat: [120x180 single] lon: [120x180 single] lon: [120x180 single] z: [22x120x180 double] z: [22x120x180 double] jdmat: jdmat: cf_tslice works identically for FVCOM, POM, ROMS, ECOM, WRF, Wavewatch3 cf_tslice works identically for FVCOM, POM, ROMS, ECOM, WRF, Wavewatch3

Access via Amazon S3 My Server: My Server: [t,g]=cf_tslice(' /models/share/ecom_test.nc','temp',1); [t,g]=cf_tslice(' /models/share/ecom_test.nc','temp',1); Elapsed time is seconds. Elapsed time is seconds. Amazon S3: Amazon S3: [t,g]=cf_tslice(' /ecom_test.nc','temp',1); [t,g]=cf_tslice(' /ecom_test.nc','temp',1); Elapsed time is seconds. Elapsed time is seconds.

Comparing models with data

Visualization with IDV

ncWMS + godiva2

Community Needs For IOOS: Need to create a THREDDS catalog for each of the 11 regions (and a Master IOOS catalog) For IOOS: Need to create a THREDDS catalog for each of the 11 regions (and a Master IOOS catalog) Continued CF toolkit development for Matlab Continued CF toolkit development for Matlab CF toolkit development for Python, IDL, etc CF toolkit development for Python, IDL, etc Adapt existing clients to use CF data (IOOS Yi’s/Schwab GoogleMaps tools, GNOME, LTRANS, etc) Adapt existing clients to use CF data (IOOS Yi’s/Schwab GoogleMaps tools, GNOME, LTRANS, etc) Build new clients to utilize CF Build new clients to utilize CF

CF Convention Needs CF conventions for staggered grids, masking (simple and/or gridspec) CF conventions for staggered grids, masking (simple and/or gridspec) CF conventions for mosaics of grids (gridspec) CF conventions for mosaics of grids (gridspec) CF conventions for unstructured grid models (likely based on cell_types and connectivity arrays) CF conventions for unstructured grid models (likely based on cell_types and connectivity arrays)

Conclusions (1 of 2) NetCDF + NcML + THREDDS Data Server => standards-based (OpenDAP+CF) system for delivering geospatial data from native structured and unstructured grid models. THERE IS NO OTHER STANDARD OUT THERE FOR THIS TYPE OF DATA! NetCDF + NcML + THREDDS Data Server => standards-based (OpenDAP+CF) system for delivering geospatial data from native structured and unstructured grid models. THERE IS NO OTHER STANDARD OUT THERE FOR THIS TYPE OF DATA! Simple for both providers and users Simple for both providers and users THREDDS Catalogs for regions can provide a single point of access for a region THREDDS Catalogs for regions can provide a single point of access for a region

Conclusions (2 of 2) Remote HTTP access is a great toehold Remote HTTP access is a great toehold To get more eyeballs on the model products, need tools that work with software scientists already know how to use To get more eyeballs on the model products, need tools that work with software scientists already know how to use More eyeballs on the model products does result in more rapid model improvement More eyeballs on the model products does result in more rapid model improvement

Try the Matlab & IDV demos yourself! Gulf of Maine Modeling Pilot Project Page: committee/gulf-of-maine-model- interoperability-pilot-project Gulf of Maine Modeling Pilot Project Page: committee/gulf-of-maine-model- interoperability-pilot-project committee/gulf-of-maine-model- interoperability-pilot-project committee/gulf-of-maine-model- interoperability-pilot-project Or just google “GOMODP” Or just google “GOMODP”

Additional Slides

A modest proposal Each modeling center places NetCDF files on their web site, or installs a THREDDS Data Server (more efficient serving NetCDF, HDF and GRIB) Each modeling center places NetCDF files on their web site, or installs a THREDDS Data Server (more efficient serving NetCDF, HDF and GRIB) Use NcML to create virtual datasets in a standard form (CF) Use NcML to create virtual datasets in a standard form (CF) Each RA creates a central THREDDS catalog Each RA creates a central THREDDS catalog Create central catalog for all 11 RAs! Create central catalog for all 11 RAs!

Our THREDDS Server: Google “coast-enviro”

Structured grid data

Unstructured Grid Data

OPeNDAP Open-source Project for a Network Data Access Protocol: delivery of multidimensional scientific data via http Open-source Project for a Network Data Access Protocol: delivery of multidimensional scientific data via http allows efficient slicing from large 4D data via the web, just as NetCDF works for local files. allows efficient slicing from large 4D data via the web, just as NetCDF works for local files. OpenDAP + CF allows standardized delivery of lon,lat,time,vertical coordinates OpenDAP + CF allows standardized delivery of lon,lat,time,vertical coordinates

NetCDF Machine independent, self-describing, binary format for multidimensional scientific data Machine independent, self-describing, binary format for multidimensional scientific data Efficient hyperslab extraction of data Efficient hyperslab extraction of data nc_varget(fid,var,corner,edge,stride) nc_varget(fid,var,corner,edge,stride) nc_varget(‘file’,’temp’,[0 0],[20 30],[1 2]) nc_varget(‘file’,’temp’,[0 0],[20 30],[1 2]) Interfaces: Fortran, C, C++, Java, Perl, IDL, Python, Matlab Interfaces: Fortran, C, C++, Java, Perl, IDL, Python, Matlab MIT-style license, supported by NSF at Unidata since 1987 MIT-style license, supported by NSF at Unidata since 1987

A Simple NetCDF File

CF-Compliant ROMS File

GoMODP Modeling Group Tasks: Authority, Discovery, Accessibility, Interoperability, Community Catalog models and contacts (GCMD) Catalog models and contacts (GCMD) Standardize the model output Standardize the model output Deliver standardized output via web services Deliver standardized output via web services Develop a toolkit for Matlab that brings the standardized data to the scientists desktop Develop a toolkit for Matlab that brings the standardized data to the scientists desktop

Vorosmarty, UNH=>CUNY Or IOOS/OOI/NFRA Modeling Framework? +CF ConventionsSWE/SOS

Sample THREDDS Catalog #1

Sample THREDDS Catalog #2

ncWMS + godiva2