1. Earth Science Datacasting Informed Pull of GHRSST data using Really Simple Syndication (RSS) http://datacasting.jpl.nasa.gov http://datacasting.jpl.nasa.gov Earth Science Datacasting Informed Pull of GHRSST data using Really Simple Syndication (RSS) http://datacasting.jpl.nasa.gov http://datacasting.jpl.nasa.gov Andrew Bingham Bob Deen, Kevin Hussey Tim Stough, Sean McCleese, Nick Toole Andrew Bingham Bob Deen, Kevin Hussey Tim Stough, Sean McCleese, Nick Toole Funded through NASA ROSES/ACCESS Program

2. Overview  Overview of RSS and podcasting  Datacasting approach  Demo  GDAC use and request to data providers to consider setting up Datacasting Feeds  Overview of RSS and podcasting  Datacasting approach  Demo  GDAC use and request to data providers to consider setting up Datacasting Feeds

3. The Datacasting Concept  Enable discover and/or automatic download of only the data files that meet a predefined need  Need all data for a specific event (flood, fire, storm, oil spill etc)  Need only cloud free data  Need only data that meet (or fail) a quality threshold  Datacasting allows users to easily and simply get at the data they need to complete a specific task:  Eliminates costly manual data searches  Automatically download data that is needed  Provides automated data thinning and consequently reduces data volume  Enables subscriptions  Builds on common web-based standards (RSS)  A possible solution for integrating and correlating disparate information sources  Enable discover and/or automatic download of only the data files that meet a predefined need  Need all data for a specific event (flood, fire, storm, oil spill etc)  Need only cloud free data  Need only data that meet (or fail) a quality threshold  Datacasting allows users to easily and simply get at the data they need to complete a specific task:  Eliminates costly manual data searches  Automatically download data that is needed  Provides automated data thinning and consequently reduces data volume  Enables subscriptions  Builds on common web-based standards (RSS)  A possible solution for integrating and correlating disparate information sources

4. Really Simple Syndication (RSS) NYT > Home Page http://www.nytimes.com/index.html New York Times > Breaking News, World News en-us Copyright 2007 The New York Times Company Sun, 21 Oct 2007 03:05:01 GMT NYT > Home Page http://graphics.nytimes.com/images/section/NytSectionHeader.gif http://www.nytimes.com/index.html Tighter Border Delays Re-entry by U.S. Citizens http://www.nytimes.com/2007/10/21/us/21border.html After decades of being waved through, Americans returning from Mexico are increasingly being checked and questioned like foreigners, leading to wait times of two hours or more. JULIA PRESTON http://www.nytimes.com/2007/10/21/us/21border.html Sun, 21 Oct 2007 02:55:12 GMT In Myanmar, Fear Is a Constant Companion http://www.nytimes.com/2007/10/21/world/asia/21myanmar.html Beneath the ominous calm that has settled since the recent uprising is anger, uncertainty and hopelessness. CHOE SANG-HUN http://www.nytimes.com/2007/10/21/world/asia/21myanmar.html Sun, 21 Oct 2007 02:57:29 GMT `  RSS is a family of web feeds for publishing frequently updated content such as blog entries, news headlines or podcasts

5. Aggregation Services

6. Podcasting All About Everything http://www.example.com/podcasts/everything/index.html en-us A show about everything John Doe A show about everything Shake Shake Shake Your Spices John Doe A short primer on table spices This week we talk about salt and pepper shakers http://example.com/podcasts/archive/aae20050615.m4a Wed, 15 Jun 2005 19:00:00 GMT 7:04

7. Datacasting Approach  Datacasting contributes to the RSS family through publishing of updated Earth Science data streams  Scenario  Data providers create RSS feeds for a data stream - update each time a new data file is available  Feed uses the RSS standard, with extensions that describe  The web location of the file for downloading  The description & structure of the files for data extraction  The content & meaning of the data for filtering  Users subscribe to RSS feeds and are notified when new data files are available  Users create “filters” based on a need, which in turn selects and download the files  After the file is on the users hard drive, the user has the option to visually inspect the data  Datacasting contributes to the RSS family through publishing of updated Earth Science data streams  Scenario  Data providers create RSS feeds for a data stream - update each time a new data file is available  Feed uses the RSS standard, with extensions that describe  The web location of the file for downloading  The description & structure of the files for data extraction  The content & meaning of the data for filtering  Users subscribe to RSS feeds and are notified when new data files are available  Users create “filters” based on a need, which in turn selects and download the files  After the file is on the users hard drive, the user has the option to visually inspect the data

8. RSS Feeds for Datacasting GHRSST-L4 Hurricane Sea Surface Temp http://podaac.jpl.nasa.gov/hurricanes Daily Sea Surface Temperatures from GHRSST for Hurricanes GHRSST: UKMO's OSTIA, NCDC's AVHRR and AMSR+AVHRR Wed, 17 Oct 2007 22:52:12 GMT Datacasting Feed Publishing Tools http://datacasting.jpl.nasa.gov/datacasting.html NARI at 2007-09-17 12:00 GMT http://podaac.jpl.nasa.gov/cgi-bin/hurr/storm.pl?id=12W&date=20070917&inst=GI Mon, 17 Sep 2007 00:00:00 GMT Mon, 17 Sep 2007 23:59:00 GMT 15.000000 120.000000 40.000000 145.000000 <enclosure url="http://podaac.jpl.nasa.gov/hurricanes/dataGI/OSTIA/2007/20070917-UKMO-L4HRfnd-GLOB-v01-fv02-OSTIA- 12W.nc.gz" length="299728" type="application/x-gzip”/ http://podaac.jpl.nasa.gov/hurricanes/OSTIA/2007/20070908-UKMO-L4HRfnd-GLOB-v01-fv02- OSTIA12W.jpg Wed, 19 Sep 2007 00:55:00 GMT PO.DAAC Hurricanes GHRSST-L4 Hurricane Sea Surface Temp http://podaac.jpl.nasa.gov/hurricanes Daily Sea Surface Temperatures from GHRSST for Hurricanes GHRSST: UKMO's OSTIA, NCDC's AVHRR and AMSR+AVHRR Wed, 17 Oct 2007 22:52:12 GMT Datacasting Feed Publishing Tools http://datacasting.jpl.nasa.gov/datacasting.html NARI at 2007-09-17 12:00 GMT http://podaac.jpl.nasa.gov/cgi-bin/hurr/storm.pl?id=12W&date=20070917&inst=GI Mon, 17 Sep 2007 00:00:00 GMT Mon, 17 Sep 2007 23:59:00 GMT 15.000000 120.000000 40.000000 145.000000 <enclosure url="http://podaac.jpl.nasa.gov/hurricanes/dataGI/OSTIA/2007/20070917-UKMO-L4HRfnd-GLOB-v01-fv02-OSTIA- 12W.nc.gz" length="299728" type="application/x-gzip”/ http://podaac.jpl.nasa.gov/hurricanes/OSTIA/2007/20070908-UKMO-L4HRfnd-GLOB-v01-fv02- OSTIA12W.jpg Wed, 19 Sep 2007 00:55:00 GMT PO.DAAC Hurricanes Dataset info & definition of custom metadata Granule info (including custom metadata values)

9. System Configuration  Data provider creates XML feeds and provides access to files using the Datacasting Feed Publishing software  Users subscribe and download relevant files using the Datacasting Feed Reader software  Data provider creates XML feeds and provides access to files using the Datacasting Feed Publishing software  Users subscribe and download relevant files using the Datacasting Feed Reader software

10. Datacasting Tools Publishing Software  A set of easy to use, portable, Python based tools for publishing your data  Text based configuration file specifies location of repository, data format, and other information for the feed Publishing Software  A set of easy to use, portable, Python based tools for publishing your data  Text based configuration file specifies location of repository, data format, and other information for the feed Feed Reader  Uses an email reader type interface  Subscribes to feeds  Filters available data files and downloads only what the user needs  Stores and manages files for later use  Previews data for interactive selection Feed Reader  Uses an email reader type interface  Subscribes to feeds  Filters available data files and downloads only what the user needs  Stores and manages files for later use  Previews data for interactive selection

11. Rich Metadata  Filtering is a function of the metadata  Some metadata are part of the core Datacasting XML specification, e.g.  Location or Extent  Acquisition or Start time  Other metadata can be defined by the data provider and included in the XML feed, for example:  % of cloud free pixels  Min, mean & max of data parameter  Event information capture in the data (algae blooms, hurricanes, fires……)  Quality control parameters  Filtering is a function of the metadata  Some metadata are part of the core Datacasting XML specification, e.g.  Location or Extent  Acquisition or Start time  Other metadata can be defined by the data provider and included in the XML feed, for example:  % of cloud free pixels  Min, mean & max of data parameter  Event information capture in the data (algae blooms, hurricanes, fires……)  Quality control parameters

12. Demo

13. Feed Reader start up

14. Acquire Datcasting Feed URL

15. Add Datacasting Feed

16. Interrogate feed

18. Add RSS feed

19. Create a filter

20. Reduced list based on need – configure to download file

21. Combine other Datacasting Feeds

22. Mashup - Combining data and information from more than one source into a single integrated tool - Creating a new and distinct web service that was not originally envisaged by either source

23. Now What?  Need data providers to create Datacasting Feeds http://datacasting.jpl.nasa.gov  Need suggestions on what metadata should be included in the GDAC feeds Current:  Time, space extents  % of valid pixels  Max & Min SST, AOD, SSI, Wind Speed, Deviation from previous day SST Anything else?  Quality parameters? E.g. GMPE or Pixie metrics  Statistical parameters that summarize the granule content  Need data providers to create Datacasting Feeds http://datacasting.jpl.nasa.gov  Need suggestions on what metadata should be included in the GDAC feeds Current:  Time, space extents  % of valid pixels  Max & Min SST, AOD, SSI, Wind Speed, Deviation from previous day SST Anything else?  Quality parameters? E.g. GMPE or Pixie metrics  Statistical parameters that summarize the granule content