1. Community Analysis and Visualization Tools for the Geosciences Sylvia Murphy Don Middleton Mary Haley National Center for Atmospheric Research Computational and Information Systems Laboratory Boulder, Colorado SC2008  Austin, Texas  November 15-21, 2008

2. Community and open source tools and environments Earth System Grid Enables analysis of and knowledge development from global Earth System models NCAR Command Language Scripting language designed for scientific data analysis and visualization PyNGL Python interface to NCL’s graphics library PyNIO Python interface to NCL’s file I/O library

3. Challenges in climate, weather, and environmental research Datasets are globally distributed and growing in size and complexity petabytes  exabytes How to provide effective access and analyses across geoscientific tools and environments? Growing international community presents unique requirements Need to train users worldwide to get the most out of their research in these complex environments

4. The ESKE Science Gateway Framework The Science Gateway Framework is aimed at providing common infrastructure for a range of distributed, federated data management efforts ESG: will deploy the SGF for an early testbed for IPCC AR5/CMIP5 this Fall ASP/DyCore Workshop: A Curator, ESG, and SGF collaboration to provide a system that spans models, data, and tools CADIS: A new prototype Gateway for polar research is undergoing review

5. NCAR Command Language (NCL) Reads many data formats popular in geosciences Remote access to data Hundreds of analysis functions Publication-quality visualizations Several workshops yearly Knowledgeable consulting Supported on UNIX systems Binaries provided Open source NCL graphic by Dennis Shea, NCAR Wheeler-Kiladis Space-Time Spectra

6. NCL Visualizations High-quality and customizable visualizations Contours, XY, vectors, streamlines, maps Specialized scripts for skew-T, wind roses, histograms, taylor diagrams, bar charts, meteograms Over 1,400 graphical options available Meteogram: John Ertl, FNMOC Terrain rasters: Mark Stevens

7. PyNGL (“pingle”) Python interface similar to NCL graphical interfaces Same publication-quality graphics as NCL Supports NumPy MaskedArrays Some geoscientific data analysis functions Online tutorial PyNGL graphic showing nested grids Ufuk Turuncoglu Istanbul Technical University Turkey Climate Change Scenarios Open source August 2008

8. PyNIO (“pie-nee-oh”) Reads/writes same formats as NCL Excellent GRIB 1 & 2 reader Easy-to-use interface Unified NetCDF-like view of all data formats Supports NumPy MaskedArrays Extensive subscripting capability Open source, August 2008 import Nio ncdf = Nio.open_file(“file.nc”,”r”) ngrb = Nio.open_file(“RUC.grb”,”r”)

9. Why a Python interface? Shares similarities with NCL easy to develop in parallel Popular with growing scientific community Widely used across many disciplines Mainstream language Demand for high-quality 2D visualizations Demand for specialized analysis functions Collaboration opportunities Contribution to Python community Open source

10. What’s coming in NCL V5.0.1 – December 2008 Major map database overhaul Much more accurate outlines New state/province outlines for China, India, and Brazil, ice shelves of Antarctica New projections Many updates to GRIB 1 & 2 readers (one of the best GRIB readers) Several new analysis functions Wheeler-Kiladis space-time spectra MJO [Madden-Julian Oscillation] diagnostics Special focus on analyzing WRF-ARW data

11. What’s new in PyNGL/PyNIO August 2008 First open source version PyNIO and PyNGL released as separate packages Support for NumPy masked arrays Extensive subscripting added to PyNIO Several new PyNGL and PyNIO examples Map database updated (same as for NCL)

12. Future plans for NCL, PyNIO, PyNGL More I/O formats and functionality: –Handle large (> 2Gb) arrays –NetCDF 4, HDF5, HDFEOS 5 –Datasets aggregated from multiple files Supplement display model –larger color maps, transparency, anti-aliasing, image formats, more font support More analysis functions Streamlines/vectors on triangular meshes Collaborative projects –VAPOR, WRF, CCMval, CCSM, ESG

13. import Ngl, Nio # Open the NetCDF file. nf = Nio.open_file("mtemp.cdf","r") # Get lat/lon/temperature variables. lat = nf.variables["lat"][:] lon = nf.variables["lon"][:] T = nf.variables["t"][0,:,:] # Open a PS workstation. wks = Ngl.open_wks("ps","mecca") # Contour & scalar field resources. res = Ngl.Resources() res.sfXArray = lon res.sfYArray = lat res.cnFillOn = True # Draw contour plot. contour = Ngl.contour(wks,T,res) Ngl.end() PyNGL/PyNIO load "gsn_code.ncl" begin ; Open the NetCDF file. nf = addfile("mtemp.cdf","r") ; Get lat/lon/temperature variables. lat = nf->lat lon = nf->lon T = nf->t(0,:,:) ; Open a PS workstation. wks = gsn_open_wks("ps","mecca") ; Contour & scalar field resources. res = True res@sfXArray = lon res@sfYArray = lat res@cnFillOn = True res@lbPerimOn = False ; Draw contour plot. contour = gsn_contour(wks,T,res) end NCL/GSUN Sample PyNGL/PyNIO and NCL/GSUN scripts

14. Sample WRF-ARW visualizations Scripts maintained by Cindy Bruyere, NCAR/MMM http://www.mmm.ucar.edu/wrf/OnLineTutorial/Graphics/NCL/

15. New map outlines in V5.0.1

16. New and improved projections

17. Taylor diagram Courtesy of Dennis Shea and Adam Phillips, CGD

18. A T-S diagram is a graph showing the relationship between temperature and salinity as observed together at, for example, specified depths in a water column. Isopleths of constant density are often also drawn on the same diagram as a useful additional interpretation aid. In the ocean, certain T-S combinations are preferred, leading to the procedure of identification via the definition of water types and water masses and their distributions. Image contributed by Christine Shields, NCAR/CGD.

19. Madden Julian Oscillation Climate Variability Image courtesy of Dennis Shea, NCAR

20. Evans plot: (Jason Evans, Yale University) A way to visualize spatially, two variables of interest, one of which provides some measure of "importance".

21. NCL image courtesy of Christine Shields, CGD Paleogeography data courtesy David Rowley, PGAP

22. Image courtesy Dave Brown POP Grid

23. Connectivity graphs from a biological neural network model. Trevor Law, Univ of California @ Irvine

24. Will be able to import PNG images and overlay on other projections

25. Types of special grids that NCL and PyNGL can contour (using triangular meshes) Coming soon: vectors and streamlines on non-uniform grids

26. http://isccp.giss.nasa.gov/

27. ARPEGE GRID (Meteo France) Christophe Cassou (CNRS/CERFACS) Similar to ISCCP grid, but with somewhat finer resolution. Grid is rotated with respect to the globe so as to put its poles somewhere other than at the North and South Pole.

28. Data from Dave Randall, Todd Ringler, Ross Heikes of CSU Most geodesic grids appear to be formed by elaborating an icosahedron; each of the 20 faces of the icosahedron is subdivided into smaller triangles in a more or less obvious way.

29. Christophe Cassou (CNRS/CERFACS) This grid could be described as a tripole grid that is further modified by the arbitrary displacement of some portions of the grid to achieve finer resolution over areas of interest (typically, ocean areas).

30. Triangular mesh Tom Gross NOAA/NOS/CSDL/MMAP

31. Curly vectors on a triangular mesh to be added to NCL/PyNGL FY2009

32. Earth System Grid http://www.earthsystemgrid.org NCAR Command Language http://www.ncl.ucar.edu PyNGL and PyNIO http://www.pyngl.ucar.edu Sylvia Murphy (murphys@ucar.edu) Don Middleton (don@ucar.edu) Mary Haley (haley@ucar.edu) Questions?