HYCOM Data Service New Datasets, Functionality and Future Development Ashwanth Srinivasan, (FSU) Steve Hankin (NOAA/PMEL) Major contributors: Jon Callahan (Mazama Consulting) Roland Schweitzer (Weathertop Consulting), Ansley Manke (NOAA/PMEL) Jeremy Malczyk (UW/JISAO) Peter Cornillon (URI) HYCOM NATIONAL MEETING – APRIL 2007

April '07 HYCOM Nat’l Meeting 2 Three data mgmt themes 1. help you with day-to-day operations 2. promote collaboration across the Consortium 3. help Consortium to reach broader classes of users

April '07 HYCOM Nat’l Meeting 3 Three data mgmt themes 1. help you with day-to-day operations 2. promote collaboration across the Consortium 3. help Consortium to reach broader classes of users Home Page (soon to be replaced)

April '07 HYCOM Nat’l Meeting 4  Launched in 2002 in Miami. Served HYCOM Outputs via FTP, LAS & OPeNDAP  Started Serving Near Real Time Atlantic data in 2003  Hosted ~3 TB of data until Dec, 2006  Jan, 2007 – moved to FSU. Expanded to a 100 TB SAN.  Ready to serve global near real time outputs. HYCOM Data Service: History and Current Status

April '07 HYCOM Nat’l Meeting 5 Now: 1/12 Global Free Run output Now: 1/12 Global Assimilative Run 2003/ /05 Soon: Near real-time 1/12 global prediction system output Soon: Several 1/12 Gulf of Mexico Simulations for inter-comparing data assimilation schemes (HYDAE) Near real-time 1/12 Atlantic Ocean prediction system output (June 2003 – Present) Monthly mean 1/12 Pacific Ocean Simulation output ( ) HYCOM outputs for MERSEA/GODAE (sub-sampled outputs interpolated to depth levels) Recent Additions and Available Datasets

April '07 HYCOM Nat’l Meeting 6 TOP 5 Data Management Priorities 1. The HYCOM data services should include a reliable capability to deliver custom netCDF subsets (user specifies region and variables) – done 2. Procedures to better inform the HYCOM Consortium members of new data management capabilities, new datasets, etc. – Data shopper catalog. Mailing list. Updated website soon. 3. The HYCOM data services should provide OPeNDAP, LAS, and FTP access to all data. Outputs should be available on native grids as well as engines for format transfer and regriddin – done (native grid and regridded to standard depth levels) 4. In order to make "nesting" from HYCOM to HYCOM models simpler provide “packaged access” (ability to request a tar file) of all files needed to set up the nested run - tested. Will be implemented shortly 5. Provide access to detailed model run metadata – model domain; source code configuration; forcing fields, BCs, ICs; PI name - blkdat.input, regional.grid,regional.depth and info available

April '07 HYCOM Nat’l Meeting 7 Moving the Data Service: Timeline Dec-2006 Dec-2006 Installed and tested the new hardware/OS etc. at FSUInstalled and tested the new hardware/OS etc. at FSU Jan-April, 2007 Jan-April, 2007 Downloaded 13 TB of data from NAVO and Miami -- Global, Atlantic and Pacific (Thanks to Skinman and Joe Metzger for suggesting ways to speed up the transfer from NAVO! )Downloaded 13 TB of data from NAVO and Miami -- Global, Atlantic and Pacific (Thanks to Skinman and Joe Metzger for suggesting ways to speed up the transfer from NAVO! ) Software installationSoftware installation Rewrite of the programs that update the server weekly due to changes in the THREDDS server configurationRewrite of the programs that update the server weekly due to changes in the THREDDS server configuration April-20-22, 2007 April-20-22, 2007 FSU service completely operational. Global data added. Website updated.FSU service completely operational. Global data added. Website updated.

April '07 HYCOM Nat’l Meeting 8 Hardware Software Software Red Hat Linux Red Hat Linux Red Hat cluster suite and Global File System Red Hat cluster suite and Global File System Apache Web Server Apache Web Server THREDDS THREDDS LAS Server LAS Server Vsftpd server Vsftpd server  100 TB Fibre Channel SAN  Three 8 CPU machines  32 GB RAM/machine  Network throughput: 50 GB/hr

April '07 HYCOM Nat’l Meeting 9 Dataset Listing Updated HYCOM Data Service “site map”

April '07 HYCOM Nat’l Meeting 10 Review of LAS …

April '07 HYCOM Nat’l Meeting 11 Desktop:Matlab, IDL, IDV, Ferret, GrADS, … Information Products netCDF, HYCOM binary, ASCII, GIS layers

April '07 HYCOM Nat’l Meeting 12 Access to native coordinates and regridded fields

April '07 HYCOM Nat’l Meeting 13 Hybrid-Z regridding native-Z fixed-Z

April '07 HYCOM Nat’l Meeting 14 Three data mgmt themes 1. help you with day-to-day operations 2. promote collaboration across the Consortium 3. help Consortium to reach broader classes of users

HYDAE model intercomparison On-line Demo

April '07 HYCOM Nat’l Meeting 16

April '07 HYCOM Nat’l Meeting 17

April '07 HYCOM Nat’l Meeting 18

April '07 HYCOM Nat’l Meeting 19

April '07 HYCOM Nat’l Meeting 20

April '07 HYCOM Nat’l Meeting 21

HYDAE model intercomparison

April '07 HYCOM Nat’l Meeting 23 Three data mgmt themes 1. help you with day-to-day operations 2. promote collaboration across the Consortium 3. help Consortium to reach broader classes of users

New LAS Architecture All-Java – more flexible and much faster - User Interface Server Product Server Remote Data Backend Service Metadata (XML) Local Data LAS user product metadata product request XML (REST) back end request (SOAP)

April '07 HYCOM Nat’l Meeting 25 How does LAS work? (cont’d) Back-end services can also perform special functions, such as accessing data from a database. Back-end services can also perform special functions, such as accessing data from a database. Services can be chained together into work-flows. Services can be chained together into work-flows.

April '07 HYCOM Nat’l Meeting 26 Workflow orchestration GIS service Backend Service Metadata (XML) SQL database LAS metadata product request XML (REST) back end request (SOAP) Product Server Backend Service OPeNDAP netCDF files Legacy CDAT JDBC Legacy Ferret Service proxy A more detailed look at back end services

April '07 HYCOM Nat’l Meeting 27

April '07 HYCOM Nat’l Meeting 28 HYCOM data at the desktop (for Matlab, IDL, Ferret, GrADS, …) F-TDS is an OPeNDAP server Based on Unidata’s “THREDDS Data Server” Based on Unidata’s “THREDDS Data Server” Analyses and regridding on the server Analyses and regridding on the server Custom server-side analysis expressed as part of the “filename” (actually, a URL) Custom server-side analysis expressed as part of the “filename” (actually, a URL) E.g. Vertical average of variable “TEMP”

April '07 HYCOM Nat’l Meeting 29 F-TDS Greatly reduces network data volumes Greatly reduces network data volumes “Delayed evaluation” “Delayed evaluation” Entire domain appears to be transformedEntire domain appears to be transformed Calculations are on-demand, “surgical”Calculations are on-demand, “surgical” Regridding on-the-fly Regridding on-the-fly

April '07 HYCOM Nat’l Meeting 30 NVODS: an example of a distributed community made interoperable thru OPeNDAP and LAS. HYCOM … along with GODAE, NOAA/NOMADS, NAVO/NCDDC/NOMADS, IOOS/DMAC(MAST), … All are also interoperable thru OPeNDAP and LAS Now add server-side regridding and analysis … Proposal: we implement standard metrics (à la GODAE) interpolations to standard coordinates Class 1 (3D grids) Class 2 (tracks and profiles) comparisons with observations Class 4 (on-going )

April '07 HYCOM Nat’l Meeting 31 Handling Forecast Data Forecast aggregation capabilities of TDS … Soon HYCOM will offer forecast views along 3 types of time axes netcdf/ncml/v2.2/FmrcAggregation.html 1 2 3

April '07 HYCOM Nat’l Meeting 32 Three data mgmt themes 1. help you with day-to-day operations 2. promote collaboration across the Consortium 3. help Consortium to reach broader classes of users

New LAS user interface (currently “alpha” level) Interact with the graphics

April '07 HYCOM Nat’l Meeting 34 HYCOM output will reach broader community through standard GIS protocols (“WMS” and “WCS”)

April '07 HYCOM Nat’l Meeting 35 HYCOM output via Google Earth

April '07 HYCOM Nat’l Meeting 36 HYCOM plot on Google Earth ( Sea Water Potential Temperature ) Fine structure materializes as we zoom in HYCOM plot on Google Earth ( Sea Water Potential Temperature )

April '07 HYCOM Nat’l Meeting 37 Can mark grid points on the globe (COADS climatology example)

April '07 HYCOM Nat’l Meeting 38 … to obtain time series and profiles

April '07 HYCOM Nat’l Meeting 39 View plot/data on browser

April '07 HYCOM Nat’l Meeting 40 Future: FTP access can be integrated in. A single, uniform interface for many services.

April '07 HYCOM Nat’l Meeting 41 Priorities for the next year Serve 1/12 global HYCOM as available Serve 1/12 global HYCOM as available Add SlideSorter to HCOM LAS Add SlideSorter to HCOM LAS Add access via Google Earth, WMS, WCS Add access via Google Earth, WMS, WCS Implement metrics (details tbd) Implement metrics (details tbd) incl. reference fields (e.g. GHRSST, Reynolds) incl. reference fields (e.g. GHRSST, Reynolds) Modernize user interface Modernize user interface Other HYCOM Consortium suggestions … Other HYCOM Consortium suggestions …

April '07 HYCOM Nat’l Meeting 42 Thank you

April '07 HYCOM Nat’l Meeting 43 New LAS UI Features and Concepts Tree and tab based navigation on a single page. Products can be created and modified without returning to the beginning of the process (steps taken to create the product are always there in a compact “tree”). Products can still be generated in a separate window (as in the current UI). No page reloading, all client - server interactions are asynchronous resulting in fast interaction and product generation. UI elements can be turned off and skipped -- a user can enter the interface at any stage of the process if desired. Portable javascript widgets and JSON services can be used for custom UI development (the LAS User Interface Toolkit). Dynamic “Google Maps” style panning and zooming on plots using OpenLayers and new LAS WMS output capabilities. All user interface elements are created by client-side code, and can be modified by the installer without rebuilding Armstrong. Style changes and debugging are easy using new DOM tools like Firebug.