1. Tutorial on the Manipulation of Netcdf Climate Data: Application to Statistically Downscaled Climate Projections Michael Notaro Nelson Institute Center for Climatic Research University of Wisconsin-Madison mnotaro@wisc.edu 608-261-1503 20th Annual Conference of The Wildlife Society

2. NCO Observational Data The source of observed data was roughly 4100 weather stations measuring daily maximum/minimum temperature and precipitation during 1950-2006 in ten of the central-eastern North American Landscape Conservation Cooperatives (LCCs), from the NWS Cooperative Observer Program. Model Projection Data The climate output that was analyzed was produced by 9 GCMs from the Coupled Model Intercomparison Project Phase 3, according to 3 emission scenarios (A2, A1B, B1). The coarse climate projections were downscaled to a 0.1° lat x 0.1° lon grid over 10 LCCs and debiased against observed temperature/precipitation from station observations within the NWS Cooperative Observer Program. By interpolating and debiasing probability distribution functions and their attributes, a realistic representation of the variance and extremes of temperature/precipitation was achieved, in addition to a realistic representation of the means of both variables. In computing the projected changes in a variable, the difference in the mean of that variable was computed from 1961-2000 to either 2046-2065 or 2081-2100. CMIP3 Models CCCMA_CGCM3_1 = Canadian Centre for Climate Modelling and Analysis CNRM_CM3 = Meteo-France / Centre National de Recherches Meteorologiques CSIRO_MK3_0 = CSIRO Atmospheric Research, Australia CSIRO_MK3_5 = CSIRO Atmospheric Research, Australia GFDL_CM2_0 = US Department of Commerce / NOAA / Geophysical Fluid Dynamics Laboratory GISS_MODEL_E_R = NASA / Goddard Institute for Space Studies, US MIUB_ECHO_G = Meteorological Institute University of Bonn, Germany MPI ECHAM5 = Max Planck Institute for Meteorology, Germany MRI_CGCM2_3_2a = Meteorological Research Institute, Japan Project Funding The statistical downscaling project was funded by: (1) Michigan DNR through an EPA grant with the Great Lakes Restoration Initiative, (2) Upper Midwest and Great Lakes LCC, and (3) Wisconsin Focus on Energy. The downscaling was performed by Dr. David Lorenz. STATISTICAL DOWNSCALING

3. NCO Interactive mapping website for examining the downscaled data http://ccr.aos.wisc.edu/resources/data_scripts/LCC

4. NCDUMP NCO NCVIEW NCL TOOLS Additional tools: Text editor: Emacs or vi Ghostview

5. NCO BASIC UNIX COMMANDS cd = change directory Example: cd /data/Share/workshop/20c3m ls = list files in a directory pwd = list the name of directory you are in mv = move (or rename) file Example: mv oldfile newfile Example: mv oldfile /data/Share/workshop/ cp = copy file Example: cp oldfile newfile mkdir = create a directory Example: mkdir dirname rm = delete file Example: rm filename rmdir = delete directory Example: rmdir dirname emacs = text editor Example: emacs = opens emacs Example: emacs & = opens emacs in background vi = text editor Example: vi = opens vi Example: vi filename = opens old or new file UNIX Tutorials: http://kb.iu.edu/data/afsk.html http://www.math.utah.edu/lab/unix/unix-commands.html http://people.ischool.berkeley.edu/~kevin/unix-tutorial/toc.html

6. NCO Change directory cd /data/Share/workshop/20c3m Open ncview and look at a temperature file for 1990. Ampersand puts ncview in background, so you can still type in terminal. ncview temp_01_1990.nc & Click on “tmax” button

7. NCO Variable: Maximum daily temperature 365 Days, beginning Jan 1, 1990 Range of values - 18.06°C to 40.825°C Move your pointer across map & notice the x,y coordinates and associated temperature values change

8. NCO Click on Opts to set options Choose USA states Click OK State boundaries should appear

9. NCO Quit = Exit ncview 3gauss = Toggle among color schemes Inv P = flip the map vertically Inv C = Invert color map Range = Modify the max/min displayed values Bi-Lin = Either show raster boxes or interpolated values Print = Save the plot as a postscript file or send to a printer

10. NCO Click on ? The variable tmax is “maximum daily temperature” of type short with units of °C and a missing value of -32768

11. NCO Click on, and >> to move backwards/forwards in time, just like on a DVD player.

12. NCO Click on any grid cell on map. A time series will appear of daily maximum temperatures during 1990. Move your pointer over time series to see data values. Click on Print button to save the plot as a postscript file. Click on Dump button to write actual data values to a text file.

13. NCO Click on 4 different grid cells on the map to compare their time series.

14. NCO NCDUMP ncdump –h prcp_01_1982.nc

15. NCO NCDUMP ncdump –c prcp_01_1982.nc List the actual latitudes, longitudes, times lon = -115.3, -115.2, -115.1, ……, -67 lat = 24.6, 24.7, 24.8, ……, 54.7 time = 0, 1, 2, ……, 364 The resolution is 0.1 degree. ncdump –k prcp_01_1982.nc Tells you “netCDF-4” Possible outcomes: Classic, 64-bit offset, netCDF-4, netCDF-4 classic model

16. NCO The temperature files, such as temp_01_1982.nc, contain multiple variables (tmax, tmin). Let’s say we want to create a new file, only containing tmin. ncks: “kitchen sink” ncks –v tmin temp_01_1982.nc newfile.nc (-v stands for variable) Now, to verify that it worked, use ncdump. ncdump –h newfile.nc The file only contains tmin, not tmax. So it worked.

17. NCO The downscaling data covers a large region, 24.6-54.7N, 115.3-67W. What if we wanted to extract precipitation data in Wisconsin only? ncks –d lat,42.,47.5 –d lon,-93.5,-86.5 prcp_01_1996.nc small.nc (remember to use decimal points) To check the data, ncview small.nc Our dataset is now much smaller, only containing data around Wisconsin.

18. NCO How can we compute the mean precipitation for July 1985? On the Julian calendar, starting with 0, July runs from day 181 to 211. First, extract only the July days from the file. ncks –d time,181,211 prcp_01_1985.nc july.nc Now, use the record averager, ncra. ncra july.nc julymean.nc We can view the results with ncview. ncview julymean.nc

19. NCO How can we compute the difference between the mean annual daily maximum temperature in 1990 and 1989? First, compute the annual mean for 1989. ncra temp_01_1989.nc 89.nc Second, compute the annual mean for 1990. ncra temp_01_1990.nc 90.nc Third, compute the difference between the two files using the operator, ncdiff. ncdiff 90.nc 89.nc diff.nc Finally, view the file with ncview. ncview diff.nc

20. NCO How can we extract daily precipitation (mm/d) from 1995 for Madison, Wisconsin and dump the values to a text file? Use ncdump to check the content of file prcp_01_1995.nc. ncdump –h prcp_01_1995.nc Notice that variable prcp is type short, with scale factor of 0.025 & offset of 819.175. If we print out values, they will not be easily used. They need to be converted to float. ncap2 –s ‘prcp=float(prcp)’ prcp_01_1995.nc float95.nc This created file float95.nc, containing precipitation in easily readable values. Now, let’s extract data values for Madison only. ncks –d lat,43.07 –d lon,-89.40 float95.nc madison.nc Finally, let’s pipe those values to a text file. ncdump madison.nc > madison.txt Open madison.txt with a text editor (emacs, vi) to see the result.

21. NCO NCL: NCAR Command Language Every script should begin with load commands, to access key libraries. I’ll only show them once here, but always include them at the top. In a text editor, create a file called hello.ncl. Insert the following text and save. load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl" load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl" load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl" load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl” print(“hello”) * To save time, cut and paste these load commands out of file loadcommands. Now, run the script by: ncl script.ncl The word hello should be printed to the screen. You have run your first NCL script.

22. NCO NCL: NCAR Command Language Given that the downscaled data is so large, it was written in a compact format, specifically netcdf4. We need to specify that we’re working with this unique type of data. setfileoption = specifies the type of data we’ll be working with setfileoption(“nc”,”Format”,”NetCDF4Classic”) Then, we can open the netcdf4 file. addfile = opens a data file “r” means readable, “w” is writable This command allows us to open these types of data files: netcdf, grib, hdf4, hdfeos, hdf5, hdfeoss, ccm, shapefile a=addfile(“temp_01_1991.nc”,”r”)

23. NCO NCL: NCAR Command Language We can retrieve the latitude and longitude values from the file. lat=a->lat lon=a->lon We can print the latitude values. print(lat)

24. NCO NCL: NCAR Command Language Our current script: script.ncl load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl" load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl" load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl" load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl” setfileoption(“nc”,”Format”,”NetCDF4Classic”) a=addfile(“temp_01_1991.nc”,”r”) lat=a->lat lon=a->lon print(lat)

25. NCO NCL: NCAR Command Language Let’s also retrieve the variable tmax from the file. Remember that tmax was in a short format and so we want to convert it to float. short2flt = Converts short format into float, using the offset and scale factor We can check the attributes/dimensions of our new variable using the command printVarSummary. setfileoption(“nc”,”Format”,”NetCDF4Classic”) a=addfile(“temp_01_1991.nc”,”r”) lat=a->lat lon=a->lon tmax=short2flt(a->tmax) printVarSummary(tmax) Now run the script. Notice the dimensions (time,lat,lon) and type float.

26. NCO NCL: NCAR Command Language Let’s try to print out the daily maximum temperature values for Madison from 1991. Recall that tmax had dimensions of time,lat,lon. When specifying the closest grid cell to Madison, provide the latitude and longitude inside { }. To acquire all time values, specify : for the time dimension. setfileoption(“nc”,”Format”,”NetCDF4Classic”) a=addfile(“temp_01_1991.nc”,”r”) tmax=short2flt(a->tmax) print(tmax(:,{43.07},{-89.40}))

27. NCO NCL: NCAR Command Language Let’s collect some basic statistics on a data file, specifically 1985’s precipitation. We’ll use the functions: avg, min, and max. setfileoption("nc","Format","NetCDF4Classic") a=addfile("prcp_01_1985.nc","r") prcp=short2flt(a->prcp) print(avg(prcp)) print(min(prcp)) print(max(prcp))

28. NCO NCL: NCAR Command Language Let’s say we want to plot a time series of Madison’s precipitation during 1985. First, let’s retrieve Madison’s values and print them to the screen as a check. setfileoption("nc","Format","NetCDF4Classic") a=addfile("prcp_01_1985.nc","r") prcp=short2flt(a->prcp) madison=prcp(:,{43.07},{-89.40}) print(madison)

29. NCO NCL: NCAR Command Language We want to plot time on x-axis and precipitation on y-axis. First, create a “workstation”, wks. We will be creating a postscript (ps) plot, named script.ps. setfileoption("nc","Format","NetCDF4Classic") a=addfile("prcp_01_1985.nc","r") prcp=short2flt(a->prcp) madison=prcp(:,{43.07},{-89.40}) wks=gsn_open_wks(“ps”,”script”) Time Temperature

30. NCO NCL: NCAR Command Language We will use the function gsn_csm_xy to make an XY plot. gsn_csm_xy(workstation, x-axis variable, y-axis variable, resource) For the x-axis, we will use “ispan(1,365,1)” which means a range of integers spanning from days 1 to 365 at a 1-day interval. For simplicity, assign “False” to resource options in gsn_csm_xy. setfileoption("nc","Format","NetCDF4Classic") a=addfile("prcp_01_1985.nc","r") prcp=short2flt(a->prcp) madison=prcp(:,{43.07},{-89.40}) wks=gsn_open_wks(“ps”,”script”) plot=gsn_csm_xy(wks,ispan(1,365,1),madison,False) Run the script. To view the plot, use ghostview in command line: gv script.ps

31. NCO NCL: NCAR Command Language Let’s now try plotting a spatial map of the mean climatological maximum temperatures for March (averaged across 1961-2000). This information is contained in file temp_mean_1961_2000.nc. Let’s use ncdump to check the file’s content. ncdump –h temp_mean_1961_2000.nc Notice the dimensions of tmax: time 12 (one value per month) latitude 302 longitude 484

32. NCO NCL: NCAR Command Language Dimensions of tmax are time, lat, lon. To retrieve March only and all latitudes/longitudes, specify we want tmax(2,:,:). Remember the counting starts at zero, so March=2. setfileoption("nc","Format","NetCDF4Classic") a=addfile("temp_mean_1961_2000.nc","r") lat=a->lat lon=a->lon tmax=short2flt(a->tmax) march=tmax(2,:,:) J0 F1 M2 A3 M4 J5 J6 A7 S8 O9 N10 D11

33. NCO NCL: NCAR Command Language setfileoption("nc","Format","NetCDF4Classic") a=addfile("temp_mean_1961_2000.nc","r") lat=a->lat lon=a->lon tmax=short2flt(a->tmax) march=tmax(2,:,:) wks = gsn_open_wks ("ps", "script") res=True res@cnFillOn=True res@gsnAddCyclic=False res@mpMinLatF=min(lat) res@mpMaxLatF=max(lat) res@mpMinLonF=min(lon) res@mpMaxLonF=max(lon) res@mpOutlineBoundarySets = "AllBoundaries” plot = gsn_csm_contour_map_ce(wks,march, res) Opens a new workstation Plot will be called script.ps

34. NCO NCL: NCAR Command Language setfileoption("nc","Format","NetCDF4Classic") a=addfile("temp_mean_1961_2000.nc","r") lat=a->lat lon=a->lon tmax=short2flt(a->tmax) march=tmax(2,:,:) wks = gsn_open_wks ("ps", "script") res=True res@cnFillOn=True res@gsnAddCyclic=False res@mpMinLatF=min(lat) res@mpMaxLatF=max(lat) res@mpMinLonF=min(lon) res@mpMaxLonF=max(lon) res@mpOutlineBoundarySets = "AllBoundaries” plot = gsn_csm_contour_map_ce(wks,march, res) Creates a new set of resources named res

35. NCO NCL: NCAR Command Language setfileoption("nc","Format","NetCDF4Classic") a=addfile("temp_mean_1961_2000.nc","r") lat=a->lat lon=a->lon tmax=short2flt(a->tmax) march=tmax(2,:,:) wks = gsn_open_wks ("ps", "script") res=True res@cnFillOn=True res@gsnAddCyclic=False res@mpMinLatF=min(lat) res@mpMaxLatF=max(lat) res@mpMinLonF=min(lon) res@mpMaxLonF=max(lon) res@mpOutlineBoundarySets = "AllBoundaries” plot = gsn_csm_contour_map_ce(wks,march, res) Specify to turn color fill on

36. NCO NCL: NCAR Command Language setfileoption("nc","Format","NetCDF4Classic") a=addfile("temp_mean_1961_2000.nc","r") lat=a->lat lon=a->lon tmax=short2flt(a->tmax) march=tmax(2,:,:) wks = gsn_open_wks ("ps", "script") res=True res@cnFillOn=True res@gsnAddCyclic=False res@mpMinLatF=min(lat) res@mpMaxLatF=max(lat) res@mpMinLonF=min(lon) res@mpMaxLonF=max(lon) res@mpOutlineBoundarySets = "AllBoundaries” plot = gsn_csm_contour_map_ce(wks,march, res) Our data is not global, or “cyclic”

37. NCO NCL: NCAR Command Language setfileoption("nc","Format","NetCDF4Classic") a=addfile("temp_mean_1961_2000.nc","r") lat=a->lat lon=a->lon tmax=short2flt(a->tmax) march=tmax(2,:,:) wks = gsn_open_wks ("ps", "script") res=True res@cnFillOn=True res@gsnAddCyclic=False res@mpMinLatF=min(lat) res@mpMaxLatF=max(lat) res@mpMinLonF=min(lon) res@mpMaxLonF=max(lon) res@mpOutlineBoundarySets = "AllBoundaries” plot = gsn_csm_contour_map_ce(wks,march, res) Specify the area to plot

38. NCO NCL: NCAR Command Language setfileoption("nc","Format","NetCDF4Classic") a=addfile("temp_mean_1961_2000.nc","r") lat=a->lat lon=a->lon tmax=short2flt(a->tmax) march=tmax(2,:,:) wks = gsn_open_wks ("ps", "script") res=True res@cnFillOn=True res@gsnAddCyclic=False res@mpMinLatF=min(lat) res@mpMaxLatF=max(lat) res@mpMinLonF=min(lon) res@mpMaxLonF=max(lon) res@mpOutlineBoundarySets = "AllBoundaries” plot = gsn_csm_contour_map_ce(wks,march, res) Show state and country boundaries

39. NCO NCL: NCAR Command Language setfileoption("nc","Format","NetCDF4Classic") a=addfile("temp_mean_1961_2000.nc","r") lat=a->lat lon=a->lon tmax=short2flt(a->tmax) march=tmax(2,:,:) wks = gsn_open_wks ("ps", "script") res=True res@cnFillOn=True res@gsnAddCyclic=False res@mpMinLatF=min(lat) res@mpMaxLatF=max(lat) res@mpMinLonF=min(lon) res@mpMaxLonF=max(lon) res@mpOutlineBoundarySets = "AllBoundaries” plot = gsn_csm_contour_map_ce(wks,march, res) Plot a contour map of march temperatures, using resources “res”

40. NCO NCL: NCAR Command Language setfileoption("nc","Format","NetCDF4Classic") a=addfile("temp_mean_1961_2000.nc","r") lat=a->lat lon=a->lon tmax=short2flt(a->tmax) march=tmax(2,:,:) wks = gsn_open_wks ("ps", "script") res=True res@cnFillOn=True res@gsnAddCyclic=False res@mpMinLatF=min(lat) res@mpMaxLatF=max(lat) res@mpMinLonF=min(lon) res@mpMaxLonF=max(lon) res@mpOutlineBoundarySets = "AllBoundaries” plot = gsn_csm_contour_map_ce(wks,march, res) Now, run the script and use ghostview to display the plot. ncl script.ncl gv script.ps

41. NCO NCL: NCAR Command Language