1. Open source visualisatie met ADAGUC Ernst de Vreede Maarten Plieger adaguc@knmi.nl http://adaguc.knmi.nl NMDC workshop “Visualisatie” 9 oktober 2014 TNO, Utrecht

2. NMDC workshop “Visualisatie” 2014 Outline Atmospheric data ADAGUC software – Overview Application examples Application contexts Open Source Conclusions

3. Atmospheric data – what’s so special? Observations in space and time Point data for example observing stations Gridded data for example satellite or precipitation radar Model data: forecasts in time and space, usually starting from a certain “reference time” Usually gridded data (“fields”) Multidimensional data: Atmosphere is a 3D structure Time makes 4D Reference time makes 5D Ensembles make 6D NMDC workshop “Visualisatie” 2014 3

4. Atmospheric data – data formats Standard meteorological data formats: GRIB/GRIB2: gridded, compressed data, WMO standard BUFR: binary format for observations etc., WMO NetCDF: binary format, with metadata to be “self-describing”, origin: Unidata (becoming OGC standard) NetCDF3: simple binary format NetCDF4: stores data in HDF5 format; more complex binary structures/groups/hierarchical data CF-conventions: set of conventions for description of Climate and Forecast data (becoming OGC standard) Non-standard formats: A whole universe of formats, binary and text-based, for satellite data, radar data, observations etc. (XML, HDF4, HDF5,….) NMDC workshop “Visualisatie” 2014 4

5. Atmospheric data – MetOcean DWG Coincidentally: ocean looks like upside-down atmosphere Meteorological and oceanographical communities started a MetOcean DWG in OGC (supported by WMO) First concrete result: Best Practices document for WMS, for handling time and vertical dimensions. Currently work is being done on developing MetOcean aspects of WCS 2.0 and on creating Best Practices for handling ensembles of forecast data in WMS NMDC workshop “Visualisatie” 2014 5

6. ADAGUC software - Overview Open source software for visualisation of meteorological data Client application in browser Server components in C++ Adhering to open standards: WMS: 1.1.1 and 1.3.0 WCS: 1.0.0 (for now) and NetCDF and CF-Conventions NMDC workshop “Visualisatie” 2014 6

7. ADAGUC software - Overview Open source software for visualisation of atmospheric data Server: -OGC WMS server (1.1.1 and 1.3.0) with WCS 1.0.0 server -Server can access -CF-NetCDF data for grids, point data, swath data from satellites -RGB images in NetCDF -HDF5 datasets (KNMI HDF5 for sat/radar) -Server can also access remote OpenDAP datasets (e.g. climate datasets) Client: -Generic web portal for (any) WMS services -Embeddable viewer component for re-use in web applications NMDC workshop “Visualisatie” 2014 7

8. 8 Web Map Service - WMS Generates visualizations of geospatial data in the form of 2D images, suitable for transfer over the internet (JPG/PNG/GIF) MSG-CPP - Precipitation

9. NMDC workshop “Visualisatie” 2014 ADAGUC WMS: Detail NetCDF CF datafiles: grids/RGB images/point data Multidimensional (time, elevation, ensemble members, etc): 4D, 5D, 6D Implements MetOcean BP reference_time and elevation Preconfigurable styling (based on standard_name attribute) Autoconfigurable for example for visualisation of WPS output Extensions: GetReferenceTimes request GetFeatureInfo/GetPointValue »Can also return data in JSON/JSONP »Multiple dimension values (e.g. elevation=* returns data for all elevations) The GetMap extensions found in ncWMS (from UoR)

10. FOSS4G-E 2014 Bremen ADAGUC WMS: GetFeatureInfo extensions get value for one point in JSON: http://[...]&REQUEST=GETPOINTVALUE &QUERY_LAYERS=air_temperature__at_2m &[...]&TIME=2014-06-01T00:00:00Z &INFO_FORMAT=application/json get time series a certain location in JSON: http://....TIME=2014-06-01T00:00:00Z/2014-06-03T00:00:00Z &INFO_FORMAT=application/json get time series for all ensemble members in JSON: http://....TIME=2014-06-01T00:00:00Z/2014-06-03T00:00:00Z &DIM_member=* &INFO_FORMAT=application/json

11. NMDC workshop “Visualisatie” 2014 How ADAGUC WMS works GRIB(2) NetCDF e.g. Fimex ADAGUC ingest WMS GetMap WMS GetFeatureInfo ADAGUC service (meta)data HDF5 e.g. PyTroll

12. NMDC workshop “Visualisatie” 2014 ADAGUC viewer: Detail Javascript browser application Uses ExtJS, Jquery for GUI elements Bespoke mapping subcomponents Viewer component is embeddable in other web pages

13. NMDC workshop “Visualisatie” 2014 Applications – GLAMEPS portal

14. NMDC workshop “Visualisatie” 2014 Applications – MSGCPP

15. NMDC workshop “Visualisatie” 2014 Applications – Polar satellite viewer (Suomi NPP)

16. 16 Applications: KNMI Data Centre Portal for KNMI data: http://data.knmi.nl Storage, catalogisation and publication of data and metadata All sorts of data Focus on metadata and searchability NetCDF4 (and HDF5) data can be (pre)viewed with ADAGUC NMDC workshop “Visualisatie” 2014

17. Application contexts of visualisation at KNMI Operational (weather room) General research (models, satellites) Dedicated research (climate data) External parties (LVNL, VCNL) General public (www.knmi.nl) NMDC workshop “Visualisatie” 2014 17

18. Application contexts: operational Operational (weather room) 4/5 positions, each with +/- 10 screens very much information Tasks: Weather analysis & forecast Security Aviation services Fairly homogeneous audience NMDC workshop “Visualisatie” 2014 18

19. Application contexts: operational (2) Visualisation tools in operational context: dedicated: radar, satellite, observations “generic”: Meteorological Work Station for model data, observations, radar and satellite combinations (not all data!) intranet: very inhomogeneous collection of graphics Intranet: More and more “precooked products”: driven by efficiency, work pressure, costs, slowness of change of dedicated systems. Very ad-hoc Web services can really help here to create and combine all these products in a flexible way. NMDC workshop “Visualisatie” 2014 19

20. Application contexts: operational (3) NMDC workshop “Visualisatie” 2014 20

21. Application contexts: research General research: Many positions Tasks: research Very inhomogeneous audience Visualisation tools in research context: Mostly more or less generic tools: Fortran libraries (ECMWF) Metview (ECMWF) Python libraries, like matplotlib IDL Matlab And many others. Web services can also help here: unification of access and style, flexibility NMDC workshop “Visualisatie” 2014 21

22. Application contexts: dedicated research Research portals dedicated to some subject: Climate data: ecad.knmi.nl Climate models: climate4impact.eu NMDC workshop “Visualisatie” 2014 22

23. Application contexts: External parties LVNL air traffic control: Data, visualised products, embedded forecaster VCNL road traffic control: Data, visualised products, embedded forecaster Embedded forecaster : “light visualisations”, web-based Web-services also directly deployable in external party’s systems NMDC workshop “Visualisatie” 2014 23

24. NMDC workshop “Visualisatie” 2014 What determines success (quality) of web service? Usability Flexibility Configurability Scalability Performance

25. FOSS4G-E 2014 Bremen Success factors – Usability/Flexibility/Configurability Usability: Service and/or viewer should be re-usable in different apps Standards compliance Too strong dependance on extensions can affect usability Flexibility: Service should easily adapt to other data sources (CF Conventions) Configurability: Service should be easy to configure (many options).

26. FOSS4G-E 2014 Bremen Scalability Scalability is possibility to apply service on a large scale Depends on: Caching Access type: tiles from a restricted single projection set (“Google Maps”) or free rectangles

27. FOSS4G-E 2014 Bremen Performance Used algorithms Hardware: Real machines vs. virtual machines NetCDF data can use compression: trade off between I/O and CPU NetCDF compression can use chunking: compression of subsets of a dataset Which sort of chunking is optimal depends on use case.

28. FOSS4G-E 2014 Bremen Performance: use cases

29. EGOWS 2014 - Oslo Applications – climate4impact.eu WMS WCS WPS OPENDAP FOSS4G-E 2014 Bremen

30. NMDC workshop “Visualisatie” 2014 Open Source In 2013: KNMI made ADAGUC open source License: MIT for server and Web Mapping component GPL V3 for ADAGUC Viewer http://dev.knmi.nl: Code repository Wiki ADAGUC workshop at KNMI in june 2013

31. NMDC workshop “Visualisatie” 2014 Conclusions ADAGUC is a success for us: we’re using and re-using it. It could play a big role in operations, but resources needed for development. It can be used for batch production of maps etc. It can be used for building web applications for different purposes. By applying ADAGUC we hope to build it out further.

32. NMDC workshop “Visualisatie” 2014 Links ADAGUC information and demo site: http://adaguc.knmi.nl Redmine repository/wiki: http://dev.knmi.nl ADAGUC applications: http://msgcpp.knmi.nl http://climate4impact.eu