1. The Model Output Interoperability Experiment in the Gulf of Maine: A Success Story Made Possible By CF, NcML, NetCDF-Java and THREDDS Rich Signell (USGS, Woods Hole) Sachin Kumar Bhate (Mississippi State) Made possible by: The GOMODP Modeling Committee & The UNIDATA team! Low-hanging fruit for IOOS! (The Integrated Ocean Observing System)

2. The Problem The GoMOOS Nowcast/Forecast Circulation Model (University of Maine)

3. We want the data!

4. Starting Point Each group had: Each group had: A different oceanographic model: POM, ECOM, FVCOM, ROMS, WaveWatch 3 A different oceanographic model: POM, ECOM, FVCOM, ROMS, WaveWatch 3 A different Matlab toolkit for analysis A different Matlab toolkit for analysis Yet, each group also had: Yet, each group also had: NetCDF output files NetCDF output files A public web server A public web server

5. Typical Ocean Model Native Grid Stretched surface and terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates

6. Climate and Forecast (CF) Metadata Conventions Groups using CF: GO-ESSP: Global Organization for Earth System Science Portal IOOS: Integrated Ocean Observing System ESMF: Earth System Modeling Framework OGC: Open Geospatial Consortium (via Galeon WCS)

7. CF Conventions for curvilinear orthogonal grids

8. CF conventions for stretched (dimensionless) vertical coordinates

9. NcML to the rescue XML markup language for NetCDF

11. CF-Compliant Model Access in Matlab

12. THREDDS

13. GOMODP THREDDS Catalog

14. CF Toolkit for Matlab Objective: make it simple to access CF-compliant data Objective: make it simple to access CF-compliant data Example function: Example function: [t, geo]=cf_tslice(uri,’temp’,itime); [t, geo]=cf_tslice(uri,’temp’,itime); t = 22x120x180 single t = 22x120x180 single geo = geo = lat: [120x180 single] lat: [120x180 single] lon: [120x180 single] lon: [120x180 single] z: [22x120x180 double] z: [22x120x180 double] jdmat: 733582 jdmat: 733582 cf_tslice works identically for FVCOM, POM, ROMS, ECOM, WRF, Wavewatch3 cf_tslice works identically for FVCOM, POM, ROMS, ECOM, WRF, Wavewatch3

15. Access via Amazon S3 My Server: My Server: [t,g]=cf_tslice('http://coast-enviro.er.usgs.gov /models/share/ecom_test.nc','temp',1); [t,g]=cf_tslice('http://coast-enviro.er.usgs.gov /models/share/ecom_test.nc','temp',1); Elapsed time is 12.613305 seconds. Elapsed time is 12.613305 seconds. Amazon S3: Amazon S3: [t,g]=cf_tslice('http://ricsigltxp2.s3.amazonaws.com /ecom_test.nc','temp',1); [t,g]=cf_tslice('http://ricsigltxp2.s3.amazonaws.com /ecom_test.nc','temp',1); Elapsed time is 2.801449 seconds. Elapsed time is 2.801449 seconds.

16. Comparing models with data

17. Visualization with IDV

18. ncWMS + godiva2

19. Community Needs For IOOS: Need to create a THREDDS catalog for each of the 11 regions (and a Master IOOS catalog) For IOOS: Need to create a THREDDS catalog for each of the 11 regions (and a Master IOOS catalog) Continued CF toolkit development for Matlab Continued CF toolkit development for Matlab CF toolkit development for Python, IDL, etc CF toolkit development for Python, IDL, etc Adapt existing clients to use CF data (IOOS Yi’s/Schwab GoogleMaps tools, GNOME, LTRANS, etc) Adapt existing clients to use CF data (IOOS Yi’s/Schwab GoogleMaps tools, GNOME, LTRANS, etc) Build new clients to utilize CF Build new clients to utilize CF

20. CF Convention Needs CF conventions for staggered grids, masking (simple and/or gridspec) CF conventions for staggered grids, masking (simple and/or gridspec) CF conventions for mosaics of grids (gridspec) CF conventions for mosaics of grids (gridspec) CF conventions for unstructured grid models (likely based on cell_types and connectivity arrays) CF conventions for unstructured grid models (likely based on cell_types and connectivity arrays)

21. Conclusions (1 of 2) NetCDF + NcML + THREDDS Data Server => standards-based (OpenDAP+CF) system for delivering geospatial data from native structured and unstructured grid models. THERE IS NO OTHER STANDARD OUT THERE FOR THIS TYPE OF DATA! NetCDF + NcML + THREDDS Data Server => standards-based (OpenDAP+CF) system for delivering geospatial data from native structured and unstructured grid models. THERE IS NO OTHER STANDARD OUT THERE FOR THIS TYPE OF DATA! Simple for both providers and users Simple for both providers and users THREDDS Catalogs for regions can provide a single point of access for a region THREDDS Catalogs for regions can provide a single point of access for a region

22. Conclusions (2 of 2) Remote HTTP access is a great toehold Remote HTTP access is a great toehold To get more eyeballs on the model products, need tools that work with software scientists already know how to use To get more eyeballs on the model products, need tools that work with software scientists already know how to use More eyeballs on the model products does result in more rapid model improvement More eyeballs on the model products does result in more rapid model improvement

23. Try the Matlab & IDV demos yourself! Gulf of Maine Modeling Pilot Project Page: http://www.gomodp.org/modeling- committee/gulf-of-maine-model- interoperability-pilot-project Gulf of Maine Modeling Pilot Project Page: http://www.gomodp.org/modeling- committee/gulf-of-maine-model- interoperability-pilot-project http://www.gomodp.org/modeling- committee/gulf-of-maine-model- interoperability-pilot-project http://www.gomodp.org/modeling- committee/gulf-of-maine-model- interoperability-pilot-project Or just google “GOMODP” Or just google “GOMODP”

24. Additional Slides

25. A modest proposal Each modeling center places NetCDF files on their web site, or installs a THREDDS Data Server (more efficient serving NetCDF, HDF and GRIB) Each modeling center places NetCDF files on their web site, or installs a THREDDS Data Server (more efficient serving NetCDF, HDF and GRIB) Use NcML to create virtual datasets in a standard form (CF) Use NcML to create virtual datasets in a standard form (CF) Each RA creates a central THREDDS catalog Each RA creates a central THREDDS catalog Create central catalog for all 11 RAs! Create central catalog for all 11 RAs!

26. Our THREDDS Server: Google “coast-enviro”

27. Structured grid data

28. Unstructured Grid Data

29. OPeNDAP http://www.opendap.org Open-source Project for a Network Data Access Protocol: delivery of multidimensional scientific data via http Open-source Project for a Network Data Access Protocol: delivery of multidimensional scientific data via http allows efficient slicing from large 4D data via the web, just as NetCDF works for local files. allows efficient slicing from large 4D data via the web, just as NetCDF works for local files. OpenDAP + CF allows standardized delivery of lon,lat,time,vertical coordinates OpenDAP + CF allows standardized delivery of lon,lat,time,vertical coordinates

30. NetCDF Machine independent, self-describing, binary format for multidimensional scientific data Machine independent, self-describing, binary format for multidimensional scientific data Efficient hyperslab extraction of data Efficient hyperslab extraction of data nc_varget(fid,var,corner,edge,stride) nc_varget(fid,var,corner,edge,stride) nc_varget(‘file’,’temp’,[0 0],[20 30],[1 2]) nc_varget(‘file’,’temp’,[0 0],[20 30],[1 2]) Interfaces: Fortran, C, C++, Java, Perl, IDL, Python, Matlab Interfaces: Fortran, C, C++, Java, Perl, IDL, Python, Matlab MIT-style license, supported by NSF at Unidata since 1987 MIT-style license, supported by NSF at Unidata since 1987

31. A Simple NetCDF File

32. CF-Compliant ROMS File

33. GoMODP Modeling Group Tasks: Authority, Discovery, Accessibility, Interoperability, Community Catalog models and contacts (GCMD) Catalog models and contacts (GCMD) Standardize the model output Standardize the model output Deliver standardized output via web services Deliver standardized output via web services Develop a toolkit for Matlab that brings the standardized data to the scientists desktop Develop a toolkit for Matlab that brings the standardized data to the scientists desktop

34. Vorosmarty, UNH=>CUNY Or IOOS/OOI/NFRA Modeling Framework? +CF ConventionsSWE/SOS

35. Sample THREDDS Catalog #1

36. Sample THREDDS Catalog #2

37. ncWMS + godiva2