1. Data Are from Mars, Tools Are from Venus
H. Joe Lee
The HDF Group
All images used in this presentation are from autodraw.com for public use.
This work was supported by NASA/GSFC under Raytheon Co. contract number NNG15HZ39C

2. No “Earth” in title?
“Men Are from Mars, Women are from Venus” - John Gray

3. Are data from Mars? Why can’t I use Earth tools?
Correct geo-referencing

4. Are tools from Venus?
Why can’t I open Earth data?

5. Data Producers from Mars
I want my data slim and efficient.
How can I save money in managing data?
Tool Developers from Venus
I make my tool work for popular data first.
Can I make money by supporting your data?

6. Result: Frustrated Users

7. We (HDFEOS.org) can help.
Our website plays the role of match maker. We’d like to know which product is hot and which tool is cool.
We anticipate the user’s demand.

8. We identify gaps in File Formats
Hierarchical Data Format (HDF)
Network Common Data Format (netCDF)
Geospatial Tagged Image File Format (GeoTIFF)
Keyhole Markup Language (KML) / zipped KML (KMZ)
Comma-separated values (CSV)
etc.

9. We identify gaps in Libraries
hdf
netcdf-C
netcdf-Java
HDF – Earth Observing System (hdf-eos)
Climate and Forecast Metadata (CF) conventions
Geospatial Data Abstraction Library (GDAL)
etc.

10. We identify gaps in Tools
Microsoft Excel
Esri ArcGIS
Google Earth
MATLAB
Python
Interactive Data Language (IDL)
Panoply
Integrated Data Viewer (IDV)
HDFView
h5dump
Etc.
Tools may add additional features that require certain conventions on top of libraries.

11. We identify gaps in Services
Open-source Project for a Network Data Access Protocol (OPeNDAP)
Web Map Service (WMS)
Web Map Tile Service (WMTS)
Web Coverage Service (WCS)
etc.

12. AND we provide Solutions…
File conversion
Libraries and tools usage
NASA HDF product specific examples
Demo services (e.g., Hyrax*, THREDDS**)
*Hyrax is the data server from OPeNDAP.
**Thematic Real-time Environmental Distributed Data Services

13. Suggestions for data producers
Make HDF5 data work with
GDAL
netCDF
Hyrax/THREDDS
Don’t forget a few key CF conventions.
Follow DIWG* recommendations.
*for ASCII
*Data Interoperability Working Group

14. Suggestions for tool developers
Download and test NASA HDF products.
Support them natively.
Support augmentation.
VRT* in GDAL
NcML** in netCDF
Support 3D visualization for data in the air.
*for ASCII
*Virtual Dataset in XML format
**netCDF Markup Language

15. 3D Visualization
3D map is now well supported by Excel, ArcGIS Online, and Cesium, not just Google Earth.

16. Suggestions for end-users
Try OPeNDAP first. CSV may be enough.
Try netCDF conversion / augmentation.
Correct metadata with NcML / VRT.
Try GEE* instead of GDAL.
Use CMR** wisely.
*for ASCII
*GDAL Enhancement for ESDIS project
** Core Metadata Repository

17. How about Big (fast) data?
Hadoop / Spark (streaming) / Dask
Parquet / Arrow
Elastic Search / Kibana

18. Streaming Swath Hadoop + Spark Streaming + Elasticsearch & Kibana
S3 + AWS Lambda + Elasticsearch & Kibana

19. Future: (Deep) Machine Learning?
scikit-learn / keras / h2o.ai
Please contact us at if you’d like to see examples on machine learning.

20. Deep Learning: Amazon Rekognition

21. Alexa, what does MODIS see now?
If tools and services are put together correctly, users will be happy.
Alexa: I can recognize “Alps.”

22. This work was supported by NASA/GSFC under Raytheon Co
This work was supported by NASA/GSFC under Raytheon Co. contract number NNG15HZ39C