GDS – The GrADS/DODS Server Jim Kinter Center for Ocean-Land-Atmosphere Studies (COLA) NVODS Workshop 10 September 2003

GrADS-DODS Server (GDS)* Joe Wielgosz, Brian Doty, Jennifer Adams James Gallagher, Daniel Halloway *(server-side integration of GrADS and DODS - now OpeNDAP) GrADS Jennifer Adams, Reinhard Budich, Luigi Calori, Brian Doty, Wesley Ebisuzaki, Mike Fiorino, Tom Holt, Don Hooper, Jim Kinter, Steve Lord, Gary Love, Karin Meier, Matt Munnich, Uwe Schulzweida, Arlindo da Silva, Michael Timlin, Pedro Tsai, Brian Wilkinson, Katja Winger (and others)

Ultimate Goal integrated analytical quantitative intuitive (for domain scientists) format-independent data type-independent e.g. grids, stations etc. supporting rapid, domain-relevant subsetting Data Interoperability Data Distribution Distributed Analysis that is

Grid Analysis and Display System INTEGRATED USER INTERFACE Maps, Charts, Animations Expressions, Functions of Original Variables General slices of { 4D Grids In Situ Obs Images User Definable, Extensible Arbitrary Domains Optimized for Typical Geophysical Queries Accessing, SubsettingAnalyzing Visualizing InteractiveQuantitative

GrADS – A Tool for Geophysics “Natural” user interface for scientific computations, and graphical production –Used at O(10 2 ) laboratories worldwide –Used by over O(10 3 ) scientists worldwide –E.g., 2002 J. Climate - Over ½ of all figures (and computations?) produced using GrADS Handles many geophysical data formats in “native” mode –Widely used for analysis and display of data from the National Weather Service, other WMO sources

GrADS Analysis Model ENABLES VERY SOPHISTICATED ANALYSIS TASKS IN A HIGHLY ENCAPSULATED WAY Scientists only need to specify: dimension constraint list of data sets analysis expression This approach to geophysical data analysis, despite its apparent simplicity, is extremely powerful.

Clients (User Programs) Local/Remote Data Analysis Decision-Support Models Web-Based Visualization internet Server Data Subsetting Distributed Data Analysis Many Data Formats Observational data Model simulations and forecasts Result cache Data Distributed Data Access and Analysis Model

Make GrADS-readable datasets - both gridded and in-situ - accessible across the network, to a diverse range of clients What the GDS can do Perform server-side analysis and comparisons against other distributed datasets serversclients and more... web browser LAS GDS OPeNDAP server GDS comparison data GRIB HDF4 netCDF BUFR Unidata IDV ncBrowse GrADS Ferret Matlab IDL OPeNDAP binary GrADS Java servlet

sdfopen set gxout shaded set time jul1980 d p sdfopen set gxout contour d prate.2*86400*31 Data Interoperability Example: Data from two Servers

Example: Analysis at the Server sdfopen {tmave(maskout(aave(... }{-180:0,0:90,500:500, jan1950,dec1990} set gxout shaded display result

GDS in production, well-received Positive response from: COLA scientists GrADS user community - research, corporate, hobbyists NOAA/CIRES CDC (earliest adopters outside COLA) Some public GDS servers: (google on "grads dods server") COLA Public Data Server: cola8.iges.org:9191 COLA Monsoon Data Server: monsoondata.org NOAA/CIRES CDC: FNMOC / GODAE: usgodae.org NCEP (NOMADS): nomad2.ncep.noaa.gov GFDL (NOMADS):data1.gfdl.noaa.gov NASA / GSWP: voda.gsfc.nasa.gov:9090 NASA / LIS: lis1.sci.gsfc.nasa.gov:9090 NASA / NSIPP:beta.gsfc.nasa.gov:9090 IPRC:aprdc.soest.hawaii.edu:9090 CSAG (South Africa) plus activity at centers in France, Britain (BADC), Italy (CINECA) and Japan...

COLA GDS User Categories Report from Jennifer Adams (2 years of statistics): 1.Users who have automated their access 2.Users who come regularly, but not automatically 3.Project-oriented users who access intensely but only in the short-term 4. Users who download data and treat GDS as a subsetting FTP server (this may be a subset of category 3) 5. Casual/curious users who are just looking, not downloading 6. Robots (e.g., google; these don’t really count, even though they are "unique IPs") incentive? how? need more robot-friendly content on top page?

Hits on All COLA GDS Bytes on All COLA GDS 100 GB 10 GB 1 GB Jan2002Jan2003 Jul2002 Jan2002 Jul2002 Jan2003 Jul2003

Desktop Weather Forecasting NCEP Global Weather Forecasts NCEP Global Weather Forecasts Global Weather Forecasts COLA GrADS-DODS Server GrADS-DODS Server Region-Specific Lateral BCs WWW PC-Based Regional NWP PILOT PROJECT WITH US NWS IN SOUTH AFRICA, VIETNAM

What's new in 1.2 New data type support Station data - GrADS format and BUFR Remote OPeNDAP data Subsampling ("striding") for gridded data Core code refactored Anagram - generic data server framework Swappable, reusable modules Designed for efficiency - streaming I/O XML-based configuration, with more flexibility in: Dataset loading Logging Security Resource management Improved web interface Custom links to help, home, dataset info URL-based administration interface Scales better to 1000's of datasets Organizes data catalog into directories Faster startup and smarter caching

GDS – What’s new and interesting for NVODS GrADS 1.9 beta release by 10/31/2003; production by 12/31/2003 much more robust support for in situ data DTYPE station via OpeNDAP DTYPE BUFR (WMO station data format) Handling GRIB-2 (WMO gridded data format) new interface for netCDF/HDF for non-COARDS-compliant data sets (via GrADS descriptor file) GDS 1.2 – Anagram framework for building servers set of reusable classes documented on white paper in preparation (Joe Wielgosz)

Administrator-friendly Complete online documentation: Stable and fast COLA public GDS currently handling > 1.5 million hits/month Install in minutes (really!) No root privileges needed Cross-platform Java and ANSI C Easy to configure Edit one (simple) XML file, and make updates on-the-fly Secure Restrict dataset access & resource usage by IP address And more... Automatic scans for new datasets Detailed logging Graceful handling of heavy loads Easily integrated with Apache... <dataset name="test" file="testdata/big_endian.ctl" format="ctl" /> <log mode="rotate" frequency="monthly" file="log/gds.log" level="info" />

GDS Enables … Sharing data: Enterprise-wide; Internet-wide --- data-format independent Data interoperability: Consistent metadata for many data types Distributed analysis: Saves scientists’ time*, reduces network load; improves interactivity Automation of analysis techniques: Analysis techniques can be captured in the form of scripts and provided on server and/or client “Data-portal changed my life.” – Ben Kirtman (COLA)