Presentation is loading. Please wait.

Presentation is loading. Please wait.

Use of OpenEarthTools to make netCDF 1 3 D:\... 3 tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP.

Similar presentations


Presentation on theme: "Use of OpenEarthTools to make netCDF 1 3 D:\... 3 tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP."— Presentation transcript:

1 Use of OpenEarthTools to make netCDF 1 3 D:\ tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP server data data providers = data users Extract Transform LoadProvide Subversion repository 2

2 Contents 1 3 D:\ tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP server data data providers = data users 2a use tools > Extract Transform LoadProvide 2b read data 2c > save as netCDF

3 OpenEarthTools Download and install TortoiseSVN Subversion client Make a checkout of OpenEarthTools (full or just Matlab subset) Start Matlab and go to OpenEarthTools >> cd F:\checkouts\OpenEarthTools\matlab\ Add OpenEarth paths to Matlab: >> oetsettings Inspect OpenEarthTools When many people work on oen thing, we need a few basic rules Document your functions and toolboxes Exploit the clickable links to files + directories in new Matlab version >> help functionname % in path >> help directoryname % in path Displays: >all H1 lines (first comment line in file) >or displays help in contents.m Add crosslinks: with 3 crosslinks, 6 clicks connect 729 files ≈ size OpenEarthTools See also

4 OpenEarthTools documentation other directories in path other m function www links

5 >> help applications OpenEarth applications rijkswaterstaat - data: rijkswaterstaat data types (donar) delft3d - delftd file formats knmi - read knmi wind and meteo timeseries swan - swan toolbox >> help rijkswaterstaat donar_read - read ASCII text file from getwaterbase - get data from waterbase.nl getwaterbasestation - get stations names for parameter getwaterbaseparameter - get parameter from waterbase.nl hmcz_read - read meteo file from rws >> help donar_read DONAR_READ read ASCII text file from DAT = donar_read(fname, ) See also: getwaterbase, % DONAR_READ read ASCII text file from % % DAT = donar_read(fname, ) % % See also: getwaterbase, OpenEarthTools documentation H1 line contents.m donar_read.m

6 Contents 1 3 D:\ tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP server data data providers = data users Extract Transform LoadProvide 2a use tools > 2b read data 2c > save as netCDF

7 Read a raw dataset Always read data into one struct Separate columns (Excel) into fieldnames Include as much meta-info as possible in the struct Use English names Use SI units Use documentation: cells are block starting with % (~ chapters) Align

8 Read a raw dataset Dedicated matlab tools >> xlsread >> textscan >> csvread Dedicated OpenEarthTools >> donar_read >> KNMI knmi_potwind >> knmi_etmgeg / SeaDataNet profile data (CTD):www.NODC.nl >> odvread Dedicate yourselves by peeling the file & add to OpenEarthTools: >> oetnewfun, load_template >> fgetl, fscanf % ascii >> fread % binary

9 Read a raw data waterbase

10 Read a raw data KNMI wind

11 Read a raw data KNMI meteo

12 Read a raw data NODC ODV

13 Test with some raw data Test for yourselve some raw data with dedicated scripts, choose one. Find the associated matlab function using the help documentation. Rijkswaterstaat KNMI NODC / SeaDataNet profile data (CTD):

14 Inspect a raw dataset Different types of raw data Different structs for time series. Hmm. >Waterbase >KNMI wind >KNMI meteo How to unify? No guidelines for structs International guidelines for netCDF! Idea: collaborate! Each BwN participant transforms raw data they use anyway into netCDF Everyone can readily access previously used data in a uniform manner raw data struct raw data file netCDF No standards available International standards available

15 Contents 1 3 D:\ tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP server data data providers = data users Extract Transform LoadProvide 2a use tools > 2b read data 2c > save as netCDF

16 netCDF What is netCDF How to make netCDF Matlab netCDF interface Conventions (vocabularies): >Standard meta-info: CF convention + GNU/CC/? >standard quantities: CF convention >Standard units: Unidata UD units >Standard coordinate codes: EPSG Save script in repository OpenEarthRawData: for dedicated script OpenEarthTools: for more general script Inspect and plot netCDF

17 netCDF = linked dimensions, variables, attributes units standard_name comment fillvalue t =f(t) z = f(z) # = f(#) chlorophyll = f(t,z,#) disclaimer institute copyright long_name standard_name units long_name standard_name units long_name units long_name standard_name units conventions title website t # z Self-describing file: meta-info never lost no manual needed open file format guarantees future access

18 Let’s look at a basic netCDF file nc_cf_example2nc.nc (in OpenEarthTools checkout) >> nc_dump(‘nc_cf_example2nc.nc’) ncBrowse HDFview Hierarchical structure as file system Attributes Dimensions Variables

19 netCDF (1) Any amount of meta-data (Attributes) can be appended. Handles large datasets (file > 4GB allowed with in 64-bit offset mode). Scaleable for future applications (divide data over multiple files). Open source specifications and interfaces. Fast data storage and retrieval (direct access binary). Small data storage (binary is small + internal zip option). Simple yet robust. “ Following examination by 20 reviewers, NASA's Earth Science Data Systems Standards Process Group has concluded that netCDF classic should be adopted as a recommended standard. A major strength of netCDF classic, according to the reviewers, was that it has fostered data interoperability and exchange through its self-describing file format, platform independent architecture, and robust access methods. Additionally, its overall file format and metadata attributes were simple enough to be easily understood and applied yet robust enough to describe and store multidimensional data of different types in the same file” (source: Unidata news)Unidata news

20 netCDF (2) 2 independent implementations of NetCDF interface (C & Java): reliable redundant Generally used, so there’s always someone who can offer help available. netCDF is de facto standard for coastal ocean circulation models (GOTM, ROMS, ECOM-SED and in the near future also Delft3D-FLOW). Lots of simple and free tools on the web that operate on NetCDF files: ncBrowse (NOAA)ncBrowse NCO Panoply (NASA)Panoply ncview (linux)ncview Implemented in numerous commercial tools too Quickplot (Delft3D)

21 netCDF (3) Interfaces to sophisticated scientific data manipulation languages: C-interface Java-interface Fortran Python IDL Matlab >Through java-interface (in OET) >mexcdf for C-interface (in OET) >Native matlab interface (since release 2008b) Merged with OPeNDAP-interface Handle netCDF file on web as if it on your own PC. Additional, widely accepted conventions for NetCDF implemented in main NetCDF viewers CF meta data convention

22 Creating a netCDF file See : nc_cf_example2nc.m Now step by step walking through Use nc_cf_example2nc.nc as template when doing it yourselves

23 >> nc_create_empty % 1 units standard_name comment fillvalue disclaimer institute copyright long_name standard_name units long_name standard_name units long_name units long_name standard_name units conventions title website t # z t =f(t) z = f(z) # = f(#) chlorophyll = f(t,z,#)

24 >> nc_create_empty % 0 Web-sources (waterbase, knmi) Read directly from web Or download to local cache Your own data: Save raw data in repository OpenEarthRawData with INSPIRE XML meta-data Then read data in Matlab and create empty netCDF file

25 >> nc_attput % 1 units standard_name comment fillvalue disclaimer institute copyright long_name standard_name units long_name standard_name units long_name units long_name standard_name units conventions title website t # z t =f(t) z = f(z) # = f(#) chlorophyll = f(t,z,#)

26 >> nc_attput % 1 See CF meta-data conventions: OpenEarth conventions: disclaimer + terms_for_use $Headurl$

27 >> nc_add_dimension % 2 units standard_name comment fillvalue disclaimer institute copyright long_name standard_name units long_name standard_name units long_name units long_name standard_name units conventions title website t # z chlorophyll = f(t,z,#) t =f(t) z = f(z) # = f(#)

28 >> nc_add_dimension % 2

29 >> nc_addvar % 3 units standard_name comment fillvalue disclaimer institute copyright long_name standard_name units long_name standard_name units long_name units long_name standard_name units conventions title website t # z chlorophyll = f(t,z,#) t =f(t) z = f(z) # = f(#)

30 >> nc_addvar % 3

31 CF convention vocabulary of quantities >> nc_varput % 4

32 UNIDATA UD units package vocabulary of units

33 >> nc_varput % 4 units standard_name comment fillvalue disclaimer institute copyright long_name standard_name units long_name standard_name units long_name units long_name standard_name units conventions title website t # z chlorophyll = f(t,z,#) t =f(t) z = f(z) # = f(#)

34 >> nc_varput % 4

35 >> nc_dump % 5 units standard_name comment fillvalue disclaimer institute copyright conventions title website t # long_name standard_name units z long_name standard_name units long_name standard_name units long_name units chlorophyll = f(t,z,#) t =f(t) z = f(z) # = f(#)

36 >> nc_dump % 5

37

38 Plot netCDF in free ncBrowse

39

40 Plot netCDF in Matlab

41 Plot netCDF in ArcGis ArcGis also reads and writes netcdf files. ArcGIS

42 Check netCDF in your own text editor (xml) xml representation of a netcdf file XML

43 Check netCDF in … NCO #diff ncdiff -v time file1.nc file2.nc #compression & packingncpdq -4 -L 9 in.nc out.nc # Deflated packing (~80% lossy compression) #selecting variables by regex ncks -v '^Q..' in.nc # Q01--Q99, QAA--QZZ, etc. IDV Very useful Web hyperslabs, cool! Not so stable. JAVA netCDF tools Useful

44 Summary current session 1 3 D:\ tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP server data data providers = data users 2a use tools > Extract Transform LoadProvide 2b read data 2c > save as netCDF

45 Next session: using netCDF from others 1 3 D:\ tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP server data data providers = data users Extract Transform LoadProvide 2

46 Read a raw data very large example WATERBASE returns 400 Mb ASCII files for water levels Has incomplete meta-info (no location code) Mixture of Amersfoort, (lat,lon), UTM31 coordinates

47 Read a raw data example Get ASCII wind file from KNMI website Read ASCII wind files as if they were one Transform to Delft3D standards (time in minutes): W2 = f(W) Write as Delft3D *.wnd file


Download ppt "Use of OpenEarthTools to make netCDF 1 3 D:\... 3 tools models add meta information netCDF on web server transform to netCDF netCDF on OPeNDAP."

Similar presentations


Ads by Google