1. Preparing input data for models Jon Goodall Hydroinformatics Fall 2014 This work was funded by National Science Foundation Grants EPS 1135482 and EPS 1208732

2. Data Required by Hydrologic Models Terrain Land use/land cover Soils Precipitaiton Streamflow Etc. Each of these data have to be gathered, transformed, and loaded into models. Many of these data require GIS (terrain, soils, land use), others we have already focused on in this class (point observations). Thus we will focus on this task on gridded precipitation data.

3. Preparing Precipitation Data Required by a Model Task: Create one (or more) time series of 10+ years of daily (or subdaily) precipitation for a specific region. Challenges: – Getting the raw data – Reading and extracting the subset of data required for your model – Writing data files in the format required by the model Result: Data prep can be a significant portion of the time spent on modeling Goal: use scripts to automate these steps: reproducible, reusable, sharable!

4. We will focus on a gridded precipitation data product from NWS US National Weather Service Advanced Hydrologic Prediction Service (AHPS) 4x4km spatial resolution 24 hour accumulated precipitation “Multisensor” dataset using NEXRAD and gauged data http://water.weather.gov/pre cip/ http://water.weather.gov/pre cip/

5. Why focus on this data? You have already seen ways of dealing with point observational data, so I am confident you can work with weather data collected at gauging locations. Gridded data present new challenges including different file formats (NetCDF, GRIB, HDF, etc.) There are many gridded weather data products that are helpful to hydrologic studies including: – Weather reanalysis products (e.g., http://www.esrl.noaa.gov/psd/data/gridded/reanalysis/) http://www.esrl.noaa.gov/psd/data/gridded/reanalysis/ – Satellite-derived precipitation (e.g., TRMM http://disc.sci.gsfc.nasa.gov/daac- bin/DataHoldingsPDISC.pl)http://disc.sci.gsfc.nasa.gov/daac- bin/DataHoldingsPDISC.pl – IPCC GCM output data (e.g., http://www.ipcc-data.org/sim/gcm_monthly/ )http://www.ipcc-data.org/sim/gcm_monthly/ The NWS AHPS product serves as an example and my goal is to provide background information to help you access and use any gridded weather/climate dataset.

6. Back to the AHPS Product - Data Access Data is available in shapefile or NetCDF format Data is available for 2004- present on an FTP server http://water.weather.gov/precip/download.php

7. FTP Server with daily data 2004 to present http://water.weather.gov/precip/p_download_new/

8. Gridded data is often stored as a NetCDF file What is NetCDF? – Network Common Data Form – “NetCDF is a set of software libraries and self-describing, machine- independent data formats that support the creation, access, and sharing of array-oriented scientific data.” Created by UCAR’s Unidata: http://www.unidata.ucar.eduhttp://www.unidata.ucar.edu There are interfaces for working with NetCDF in Matlab, R, Python, etc. In Python alone, there are multiple options available: https://www.unidata.ucar.edu/software/netcdf/software.html#Pyth on https://www.unidata.ucar.edu/software/netcdf/software.html#Pyth on

9. NetCDF files are binary (not ASCII) Why use binary? – Smaller file size for a given amount of data. Problem: – You can’t see the contents unless you covert a NetCDF file into an ASCII representation called a CDL (Common Data form Language). This is like the ESRI GRID format used for raster data in GIS. GRID is a binary format, but you can convert the file to an ASCII GRID and then view its contents with a text editor. For a given raster, the ASCII GRID will be much larger than the binary GRID dataset. (see http://en.wikipedia.org/wiki/Esri_grid for a nice explanation.) http://en.wikipedia.org/wiki/Esri_grid

10. Example NetCDF file represented in ASCII text as CDL netcdf foo { // example netCDF specification in CDL dimensions: lat = 10, lon = 5, time = unlimited; variables: int lat(lat), lon(lon), time(time); float z(time,lat,lon), t(time,lat,lon); double p(time,lat,lon); int rh(time,lat,lon); lat:units = "degrees_north"; lon:units = "degrees_east"; time:units = "seconds"; z:units = "meters"; z:valid_range = 0., 5000.; p:_FillValue = -9999.;_FillValue = -9999.; rh:_FillValue = -1;_FillValue = -1; data: lat = 0, 10, 20, 30, 40, 50, 60, 70, 80, 90; lon = -140, -118, -96, -84, -52; } Source: http://www.unidata.ucar.edu/software/netcdf/docs/CDL.html#CDLhttp://www.unidata.ucar.edu/software/netcdf/docs/CDL.html#CDL File name Dimensions of the arrays Stored within the file. Variables stored within the file. Each variables can be a multi-dimensional array and each variable stores data of a single data type. For example, lat is a one dimensional array with 10 elements all of which are integers. These are attributes of the file. They can describe attributes of a variable in the file or about the file itself. The data itself, written as a list of values separated by commas. Semicolons end the values for a variable.

11. The Classic NetCDF Data Model Source: http://www.unidata.ucar.edu/software/netcdf/docs/html_tutorial/netcdf_data_model.html#classic_model http://www.unidata.ucar.edu/software/netcdf/docs/html_tutorial/netcdf_data_model.html#classic_model

12. Converting between NetCDF and CDL A command line tools called “ncdump” and “ncgen” allow for the conversion between NetCDF and CDL format There is also an XML-based way of representing the “metadata” in a NetCDF file (NcML): http://www.unidata.ucar.edu/software/thred ds/current/netcdf-java/ncml/ http://www.unidata.ucar.edu/software/thred ds/current/netcdf-java/ncml/

13. Working with NetCDF file in Python using the netCDF 4 package We will use the netCDF4 package to work with netCDF files: http://unidata.github.io/netcdf4-python/http://unidata.github.io/netcdf4-python/ This package should be preinstalled with Canopy. To test if it is installed, type the following at the command prompt in the Canopy Editor: – import netCDF4 If it is not installed (i.e., if you get an error when you try to import netCDF4), then open the Canopy Package Manager, search for netCDF4, and install both lib_netcdf4 and netCDF4 (see next slide for details).

14. Installing NetCDF Packages These two packages must be installed.

15. Reading a NetCDF File using Python Download the NetCDF (.nc) and CDL (.cdl) files I put on Canvas. As an FYI: – I obtained the NetCDF file from the NWS FTP site (http://water.weather.gov/precip/p_download_new/ )http://water.weather.gov/precip/p_download_new/ – YOU DON’T NEED TO DO THIS FOR THIS CLASS, BUT FOR REFERENCE I created the CDL file from the NetCDF file with the following command: ncdump nws_precip_conus_20141117.nc > nws_precip_conus_20141117.cdl NOTE: – I installed the netcdf tools following this approach: http://mazamascience.com/WorkingWithData/?p=1474 http://mazamascience.com/WorkingWithData/?p=1474 – On Windows you could do this: http://www.unidata.ucar.edu/software/netcdf/docs/winbin.html http://www.unidata.ucar.edu/software/netcdf/docs/winbin.html – On other operating systems: http://www.unidata.ucar.edu/packages/netcdf/INSTALL.htmlhttp://www.unidata.ucar.edu/packages/netcdf/INSTALL.html

16. Portion of the NWS AHPS file viewed as CDL

17. Alternative: Download file directly from FTP server The file I put on Canvas is also available here:http://water.weather.gov/precip/p_download_ne w/2014/11/17/nws_precip_conus_20141117.nchttp://water.weather.gov/precip/p_download_ne w/2014/11/17/nws_precip_conus_20141117.nc You can download the file using Python (the code below downloads the file to your current working directory) import urllib urllib.urlretrieve("http://water.weather.gov/precip/p_download_new/2014/1 1/17/nws_precip_conus_20141117.nc", "nws_precip_conus_20141117.nc")

18. In Class Example Open Canopy Editor and follow along as I show how to read the netCDF file using Python.

19. Opening the NetCDF file for Reading #open a NetCDF file for reading import netCDF4 ds = netCDF4.Dataset("nws_precip_conus_20141117.nc", "r")

20. Print Structure of Dataset #print out the file’s metadata print ds RESPONSE: root group (NETCDF3_CLASSIC data model, file format UNDEFINED): dimensions(sizes): hrapy(813), hrapx(1051), latlong(4), dates(11) variables(dimensions): int32 amountofprecip(hrapy,hrapx), float32 lat(latlong), float32 lon(latlong), float32 true_lat(), float32 true_lon(), |S1 timeofdata(dates), |S1 timeofcreation(dates), float32 hrap_xor(), float32 hrap_yor() groups:

21. Examples of Variables int32 amountofprecip(hrapy,hrapx) This means there is a 2D array named stored as the variable ‘amountofprecip” with dimensions ‘hrapy’ and ‘hrapx.’ hrapy(813), hrapx(1051) This means that the dimension hrapy has 813 elements and hrapx has 1051 elements.

22. Attributes of amountofprecip Variable print ds.variables['amountofprecip'] RESPONSE: int32 amountofprecip(hrapy, hrapx) long_name: 24-Hour Rainfall units: 1/100 mm grid: hrap_grid=polar stereograph projection resolution: 4km*4km dateofdata: 2014111712Z dateofcreation: 2014111721Z location: continental united states author: national weather service rfcs genereated by program ROUT comments: preliminary data...subject to change unlimited dimensions: current shape = (813, 1051) filling off When observation was made

23. Get a single attribute of the amountofprecip variable print ds.variables['amountofprecip'].long_name Response: 24-Hour Rainfall print ds.variables['amountofprecip'].units Response: 1/100 mm

24. Get values of amountofprecip variable precip = ds.variables['amountofprecip'] print precip[0,0] REPONSE: -1 #means no data (e.g., over the ocean) precip[450,850] #this gets the precip value for cell y=450, x=850 RESPONSE: 3322 # units are 1/100 mm, so this is about 1.3 inches of rain for this cell over the 24-hr period

25. Mapping between HRAP coordinates and lat/lon coordinates Shapefile is available here: http://water.weather.gov/precip/p_download_new/nws_precip_allpoint.tar.gz http://water.weather.gov/precip/p_download_new/nws_precip_allpoint.tar.gz

26. Rainfall in Charlottesville Precip[512, 944] RESPONSE: 218 (1/100 mm) = 0.09 inches Over what duration? Recall that the dateofdata attribute = 2014111712Z – or November 17, 2014 12:00 UTC – or November 17, 2014 07:00 EST

27. NWS AHPS Documentation From http://water.weather.gov/precip/about.php: http://water.weather.gov/precip/about.php "Observed" data is expressed as a 24-hour total ending at 1200 Z (same as Greenwich Mean Time, or GMT), with longer periods simply being a summation of multiple 24-hour periods. 1200 GMT is used as the ending time for a 24-hour total, because it is the end of the "hydrologic day", a standard used in river modeling.

28. Conclusion According to the NWS AHPS product, 0.08 inches of precip fell at the Charlottesville, VA Airport during the period November 16, 2014 07:00 EST to November 17, 2014 07:00 EST Checking our answer: Observed data at Charlottesville Airport is available here:http://w1.weather.gov/obhistory/KCHO.htm lhttp://w1.weather.gov/obhistory/KCHO.htm l – According to this site, the airport received about 0.03 inches between November 16, 2014 07:00 EST to November 17, 2014 06:53 EST.

29. Challenge Problem Create a Python script that: 1.Downloads a precip *.nc file from NWS 2.Extracts the precip for a given cell 3.Prints the precip value along with the date of observation I put my solution on the GitHub site: https://github.com/goodalljl/hydroinformatics_class/blo b/master/Class23_GetPrecipForCville.py https://github.com/goodalljl/hydroinformatics_class/blo b/master/Class23_GetPrecipForCville.py

30. Next Class We will work in class to extend your Python script to repeat for multiple days to derive a precipitation time series for a location I will briefly introduce OPeNDAP (http://en.wikipedia.org/wiki/OPeNDAP) as a means for remote (server-side) subsetting of NetCDF fileshttp://en.wikipedia.org/wiki/OPeNDAP NOTE: There will not be a HW assignment for this material. We want you to be spending your time on your class projects instead.