1. OLYMPEX DATA: GHRC Archival and Services for a Variety of Data Users
9/20/2016
OLYMPEX DATA: GHRC Archival and Services for a Variety of Data Users
Deborah Smith
Staff Scientist
GHRC
University of Alabama in Huntsville, NASA MSFC
9/20/2016

2. OLYMPEX Meeting, Seattle, WA
What is a DAAC?
SEDAC
LaRC ASDC DAAC
ORNL DAAC
LP DAAC
NSIDC DAAC
PO DAAC
ASF DAAC
MODAPS/LAADS
CDDIS
GES DISC
GHRC DAAC
OB DAAC
A Distributed Active Archive Center is a NASA funded entity charged with the processing, archiving, documenting and distributing of data from Earth Observing System (EOS) satellites and field measurement programs
OLYMPEX Meeting, Seattle, WA

3. OLYMPEX Meeting, Seattle, WA
9/20/2016
About GHRC DAAC
Mission Statement
To provide a comprehensive and active archive with a focus on hazardous weather, its governing dynamical and physical processes, and associated applications
GHRC focuses on lightning, tropical cyclones and storm-induced hazards through integrated collections of satellite, airborne, and in-situ data sets
We abide by the NASA Earth Science Data and Information System (ESDIS) vision and standards to provide free and open Earth science data that is interactive, interoperable, and accessible for research and societal benefit.
OLYMPEX Meeting, Seattle, WA
9/20/2016

4. GHRC User Tools and Services
9/20/2016
GHRC works to provide users with tools and services to make data more discoverable
Documentation – Ensure independently understandable
Standards/ Best Practices
Adhere to policies for Archival and Preservation
Provenance – Keep the history
Serves as NASA’s Earth science data stewards for scientific, educational, commercial and governmental communities, with a focus on event/episodic data
Data, Science, and Tool Expertise
Curated data or information collections
Increase use of data
Provide tools to visualize and analyze data
Increase discovery and distribution of data
Provide customized portals for field campaigns
Serve the needs of multiple communities
What it means to be an active archive… not only do we ingest, manage, preserve, store, catalog and provide access to data, but we also work to add value and increase data use among a broader audience. We are Earth Science data stewards that focus on event/episodic data.
OLYMPEX Meeting, Seattle, WA
9/20/2016

5. Passive Microwave Data
What Data We Provide
Lightning Data
Passive Microwave Data
Field Campaign Data
GHRC is experienced at managing a large number of episodic, heterogeneous datasets and can handle the “long tail” of science data
Ingest, archive, produce, generate, reprocess and distribute data from the TRMM and ISS Lightning Imaging Sensor (LIS), plus ancillary lightning data sets utilized by the LIS scientists
Historic Satellite data –
Optical Transient Detector OrbView-1 (1995 to 2000)
Ancillary data –
National Lightning Detection Network
Lightning Climatology Products
GHRC and its predecessor programs have been ingesting, processing, archiving and distributing passive microwave data for over 35 years
MSU, SSMI, AMSU,
AMPR, TMI, AMSR-E
This climate sensitive data record extends back to 1978 providing an unbroken inventory of climate information that continues today
Hurricane Science
Data from successive field campaigns since 1990 with common procedures, consistent metadata, and archival and discovery tools for easy access
GPM Ground Validation (GV)
Ground and airborne precipitation datasets supporting validation of satellite-based precipitation retrieval algorithms
We are known as the “Lightning DAAC” because we provide a wide variety of lightning data, including TRMM LIS and the newly operating ISS LIS. However, we also have Passive microwave satellite data from SSM/I, SSMIS, TMI and MSU/AMSU as well as Field Campaign data for both GPM-GV and Hurricane Science.
OLYMPEX Meeting, Seattle, WA

6. The Science and Outreach Team
Some of Our Responsibilities:
Assist data producers with data archival process
Develop instructional guides to use data effectively
Create introductory articles aimed at increasing broad interest in the data
Improve information content on the GHRC DAAC web site
Curate data collections around a topic, theme, or dataset
Advise tool and service creation using science expertise
Try using the data to see where future users may need extra help and develop code/tools as needed to help them
Present GHRC tools, services and archived data at conferences, NASA webinars, ESIP, and scientific meetings.
Participate in NASA DAAC working groups that develop standards for data management, archival and distribution.
All aimed at increasing usability of the data for broad audiences
OLYMPEX Meeting, Seattle, WA

7. GPM-GV Archives at GHRC
Services provided:
Campaign portals
Data archival
Data Landing Pages / DOIs
Consistent and complete dataset user guides
Micro Articles
Data Recipes
Curated Virtual Collections
New ways to look at data
GCPEx
Jan/Feb 2012
MC3
Apr – June 2011
IFloodS
May/June 2013
IPHEx
May/June 2014
LPVEx
Sept/Oct 2010
OLYMPEX
Nov/Dec 2015
OLYMPEX Meeting, Seattle, WA

8. OLYMPEX Meeting, Seattle, WA
GRHC OLYMPEX Portal
Data Portal
Upload Instructions
File Naming Conventions
Metadata Survey
OLYMPEX Meeting, Seattle, WA

9. OLYMPEX Meeting, Seattle, WA
products
OLYMPEX Data Archival
What do we need to archive your data?
Completed question form
An accurate citation
Files named according to standard instructions
Data uploaded with PI documents that will help future users understand your data
Summary of data quality and known data problems/caveats
Assess data processing level see processing-levels-for-eosdis-data-products
Suggestions:
Citations – Keep author list to a reasonable number (no more than 5, if possible)
If we’ve archived your data before - Identify pertinent instrument changes and differences between new and previous datasets
Let us know if your data are preliminary, intermediary or final. Final datasets be published first, preliminary datasets will be delayed until intermediary or final versions are available. Let us know if you will be reprocessing again.
Identify and provide contact info for your best point-of-contact (no necessarily the PI). Let us know who filled out the question form so we can reach them if questions. This is very important!
OLYMPEX Meeting, Seattle, WA

10. What is the data publication process?
1) We carefully move your data from the ingest folder to a private server location
2) We determine metadata, develop a user guide, and enter all details into the database
Metadata: Data about data! Information needed for sharing data details across systems and used to make data more accessible
Accurate metadata required at dataset, granule (file) and sub-granule (contents of the file) levels
3) We rename files if standards not followed. We tar or gzip data if needed
These are required to improve access and dataset download time for users
4) We obtain a DOI - digital object identifier for the dataset
5) When all is right, we
publish the metadata in database
place data on the public server
provide information and links to the data on the web page
provide metadata/DOI to the Global Change Master Directory and the CMR metadata repository
OLYMPEX Meeting, Seattle, WA

11. NEW Data Publication Tool
Data Publication will be online from initiation to publication
Provider interface for question form completion
Providers can check on their publication status
Tool speeds publication of datasets
Improves accurate metadata generation
Interfaces with new Postgres database
We are in the midst of making changes at GHRC to our database and how we publish data and search for data.
OLYMPEX Meeting, Seattle, WA

12. Field Campaign Explorer Tool
A data visualization tool that can be used for a variety of airborne data
Watch the NASA webinars:
Feb 8, 2017
March 8, 2016
Or the NASA Lightning Talk at AGU :
Tool designed for the HS3 – Hurricane and Severe Storm Sentinel Campaign with intent to apply to other field campaigns
Can visualize and preview data (on-the-fly)
Can select a specific feature during flight and subset the data
Can re-enact mission flights and display instrument data
Can explore mission reports containing photos and information
Web-based, no software download/installation required
We have a field campaign explorer tool that displays airborne data collected during field campaigns. Check out the demonstrations.
OLYMPEX Meeting, Seattle, WA

13. Micro Articles/Data Recipes
Curated resources for students and non-traditional users that are concise and understandable and point users to the data or help them to utilize data.
Types of Micro Articles:
Instruments
Phenomena
Events or Case Studies
Publications
Types of Data Recipes:
Using NetCDF data in ArcGIS
Python Notebooks and scripts
Dataset plotting routines
Leveraging PyCMR
GHRC tool tutorials
OLYMPEX Meeting, Seattle, WA

14. OLYMPEX Meeting, Seattle, WA
Science as Art
Poster made for a “Research as Art” session at the Summer 2016 Earth Science Information Partners (ESIP) meeting.
Now posted on the walls of NASA Headquarters
Consists of a mosaic of browse images of the UND Citation cloud microphysics data from the MC3E field campaign flights
OLYMPEX Meeting, Seattle, WA

15. OLYMPEX Meeting, Seattle, WA
OLYMPEX Data Users
OLYMPEX scientists
Modelers
Students
Graduate
Undergraduate
High school
Applications
Visualization and tool developers
Scientists from other fields
OLYMPEX Meeting, Seattle, WA

16. OLYMPEX Meeting, Seattle, WA
Going Forward
Bring us your ideas for new tools, micro articles or data recipes. We would love to work with you!
We want to produce a curated collection around the best cases from OLYMPEX to highlight the science. Share your ideas or non- published data with us.
We would like to reuse (with credit of course) some resources you may have – figures, plots, or browse imagery.
How can we better support you?
OLYMPEX Meeting, Seattle, WA

17. OLYMPEX Meeting, Seattle, WA
Current Status of Data
Some issues with data in files. Identified so far:
erroneous year values in APU data
2DVD file format and format description disagrees (# fields differ)
MASC data dropped onto server, but no information or contact provided
AVAPS – were all soundings from DC8? Or were some from ER2?
HIRWAP data with 0 latitude and 0 longitude values
No data yet submitted for:
AirMSPI
EXRAD
eMAS
Need question forms or metadata confirmed by some investigators
OLYMPEX Meeting, Seattle, WA

18. Contact us with any questions
Database Management Team
Marilyn Database manager
Sherry
Mary
Lamar Data handling
Science Outreach Team
Deborah
Leigh
Kaylin
Amanda
OLYMPEX Meeting, Seattle, WA

19. OLYMPEX Meeting, Seattle, WA
Thank you!
OLYMPEX Meeting, Seattle, WA

20. Why Publish Your Data at GHRC?
You get DOI and Data Citation on a designated dataset description page forever
Publication credit for the data
Traceability for where and how your data are used
Links from the scientific articles to dataset information and download page
Your data are ingested, cataloged and archived using standards-based engineering processes used across NASA
Your data will be accessible via NASA data systems and tools
Added automatically to GCMD
Also indexed by Google and other search engines
Because Journals now require it for publication
With the increase in proof of data archival
OLYMPEX Meeting, Seattle, WA