1. U.S. Department of the Interior U.S. Geological Survey LP DAAC Big Earth Data Initiative (BEDI) Developed Web Services 1 Jason Werpy LP DAACEnterprise Architect SGT, Inc., Contractor to the USGS EROS Center *Work performed under USGS contract G15PD00766

2. NASA’s LP DAAC  Land Processes (LP) Distributed Active Archive Center (DAAC)  https://lpdaac.usgs.gov  Located in Sioux Falls, SD at the USGS Earth Resources Observation Science (EROS) Center 2

3. LP DAAC Surface Reflectance, Land Cover, Vegetation Indices LP DAAC Surface Reflectance, Land Cover, Vegetation Indices

4. Big Earth Data Initiative  NASA’s BEDI efforts  Increase accessibility of Earth Science Data  Increase availability of Earth Science Data  Increase usability of Earth Science Data 4

5. BEDI – LP DAAC Response  The LP DAAC decided that to address the BEDI goals, we would work on services designed and architected to give users improved data visibility, access, and tools. 5

6. BEDI – LP DAAC Response  Tiered Concept  LP DAAC created a tiered approach to data access 6 Data Services Middleware API UI

7. A Quick Aside – Getting data  Why HTTP services?  Simple  Standard bases protocol  Securable  Dynamic interaction methods (get/post, etc.)  Stream based data delivery  It’s even… 7

8. A Quick Aside – Getting data  Shown as a good way to get data in the ‘future’… 8

9. BEDI – LP DAAC Response  Data Access methods  OPeNDAP - Open-source Project for a Network Data Access Protocol. Provides a direct way of subsetting LP DAAC data from our Data Pool. 9 Data Services Middleware API UI

10. BEDI – LP DAAC Response  Middleware Web Services  Tilemap – Coverts Latitude and Longitude values into tile and pixel values for datasets located on a common grid structure.  Fromjulian – Converts dates from a Julian date format to a mm- dd-yyyy format.  Tojulian – Converts dates from a mm-dd-yyyy format to a Julian date format of the number of days since 01-01-2000.  Product (coming soon!) – Service that provides information about the different products available from the LP DAAC.  Inventory – Service that returns information about the granules in the LP DAAC inventory. Powered by NASA Common Metadata Repository (CMR). 10 Data Services Middleware API UI

11. BEDI – LP DAAC Response  Interface  AppEEARS – Interface for selecting point based data from the LP DAAC archive, analyzing that data, and downloading the specific values desired. 11 Data Services Middleware API UI

12. OPeNDAP Data Access  OPeNDAP was a key component of NASA’s BEDI efforts  OPeNDAP access to the LP DAAC’s Data Pool provides information about singular data granules.  Because MODIS tiles (and WELD tiles) are stored in a uniform grid of tiled data, the LP DAAC was able to create Aggregations by date. This allows an extra parameter that selects data from the date desired.  LP DAAC OPeNDAP Services  http://opendap.cr.usgs.gov/opendap/hyrax 12 Data Services Middleware API UI

13. OPeNDAP Data Access 13 Data Services Middleware API UI

14. OPeNDAP Data Access 14 Data Services Middleware API UI

15. OPeNDAP Data Access 15 Data Services Middleware API UI

16. OPeNDAP Data Access  Access and extract discrete subsets Spatial Band/Layers Temporal (from aggregation)  Request data via URL http://opendap.cr.usgs.gov/opendap/hyrax/MOD09Q1.005/h11v04.ncm l.ascii?sur_refl_b01[683:1:683][2000:1:2010][2000:1:2010]  Returns data that the user intends to use!  Reduces time spent downloading and processing  Minimizes the amount of data downloaded 16 Data Services Middleware API UI

17. OPeNDAP Data Access OPeNDAP request: http://opendap.cr.usgs.gov/opendap/hyrax/MOD09Q1.005/h11v04.ncml.ascii?sur_refl_b0 1[683:1:683][2000:1:2010][2000:1:2010] Return: Dataset: h11v04.ncml sur_refl_b01[0][0], 0.5359, 0.5657, 0.5258, 0.4739, 0.4648, 0.5087, 0.5234, 0.5234, 0.6214, 0.5793, 0.5374 sur_refl_b01[0][1], 0.5759, 0.5414, 0.5338, 0.4489, 0.432, 0.5036, 0.6051, 0.6214, 0.5793, 0.455, 0.446 sur_refl_b01[0][2], 0.4594, 0.5409, 0.6224, 0.6289, 0.6289, 0.6051, 0.5005, 0.5005, 0.3325, 0.3092, 0.3826 sur_refl_b01[0][3], 0.5831, 0.5831, 0.6126, 0.6307, 0.6307, 0.5277, 0.612, 0.3325, 0.3092, 0.332, 0.4036 sur_refl_b01[0][4], 0.6329, 0.6126, 0.6061, 0.5691, 0.5691, 0.612, 0.5841, 0.5841, 0.4301, 0.4782, 0.6271 sur_refl_b01[0][5], 0.6092, 0.6092, 0.6152, 0.5691, 0.6297, 0.6035, 0.5052, 0.5052, 0.4782, 0.6518, 0.6913 sur_refl_b01[0][6], 0.5712, 0.6152, 0.6248, 0.5686, 0.5686, 0.3522, 0.5088, 0.5912, 0.6316, 0.4729, 0.5756 sur_refl_b01[0][7], 0.5737, 0.5737, 0.524, 0.5368, 0.3522, 0.5691, 0.5944, 0.4085, 0.4729, 0.5375, 0.5375 sur_refl_b01[0][8], 0.5606, 0.5211, 0.5368, 0.5874, 0.5874, 0.343, 0.26, 0.26, 0.3825, 0.5794, 0.5863 sur_refl_b01[0][9], 0.5883, 0.5591, 0.5263, 0.4325, 0.343, 0.4485, 0.626, 0.626, 0.5462, 0.3682, 0.3463 sur_refl_b01[0][10], 0.5435, 0.5263, 0.4325, 0.4887, 0.4887, 0.5175, 0.626, 0.5475, 0.4101, 0.4592, 0.4203 17 Data Services Middleware API UI

18. OPeNDAP Data Access  OPeNDAP URLs are complex  Need to know: Syntax Product name Layer name Time Geolocation  Tile locations (H & V) – Data is aggregated over time by tile  Pixel location (Ydim & Xdim) http://opendap.cr.usgs.gov/opendap/hyrax/MOD09Q1.005/h11v04.ncml. ascii?sur_refl_b01[683:1:683][2000:1:2010][2000:1:2010] 18 Data Services Middleware API UI

19. Utility / Metadata Services  These Services can help fill in the “complicated” parts of the OPeNDAP URLs  Geolocation  The Tilemap service was created to convert Latitude and Longitude information into tile H and V coordinates and x and y pixel locations within the tile.  Time Information  Time data in OPeNDAP is returned as “number of days since 01- 01-2000 which is hard to convert “in your head” 19 Data Services Middleware API UI

20. Utility / Metadata Services  Product Service  The LP DAAC’s (coming soon!) Product Service can be used to get information about LP DAAC products and the bands contained within those products.  Inventory Service  The inventory service provides a view of data granules in the LP DAAC archive, it presents an alternate way to OPeNDAP for data access if a user wants to find entire granules of data. 20 Data Services Middleware API UI

21. Utility / Metadata Services and Data Services  Demo  Tilemap (also - how do I use this; or – Help!)  An OPeNDAP request (through the LP DAAC aggregated view)  Fromjulian  Tojulian  Inventory Service 21 Data Services Middleware API UI

22. Demo 22 Data Services Middleware API UI Wait…. we don’t know how this service works! But if you don’t know how to call it the service will tell you…. https://lpdaacsvc.cr.usgs.gov/services/tilemap Service Call

23. Demo 23 Data Services Middleware API UI Service Help Response

24. Demo 24 Data Services Middleware API UI https://lpdaacsvc.cr.usgs.gov/services/tilemap?product=MOD09A 1&latitude=44.306&longitude=-96.7881 Service Call Service Response 11 04 1366.06 176.27

25. Demo 25 Data Services Middleware API UI Located!

26. Demo  Location is now known  Create OPeNDAP call for that location http://opendap.cr.usgs.gov/opendap/hyrax//MOD09A1.005/h11v04.ncml.ascii?Latitud e[1366:1:1366][176:1:176],Longitude[1366:1:1366][176:1:176],sur_refl_b01[730:1:7 33][1366:1:1366][176:1:176],time[730:1:733]  Output Dataset: h11v04.ncml Latitude[0], 44.3062499960216 Longitude[0], -96.7900810678634 sur_refl_b01[0][0], 0.5054 sur_refl_b01[1][0], 0.3101 sur_refl_b01[2][0], 0.3967 sur_refl_b01[3][0], 0.4866 time, 5860, 5868, 5876, 5884 26 Data Services Middleware API UI

27. Demo  Output Dataset: h11v04.ncml Latitude[0], 44.3062499960216 Longitude[0], -96.7900810678634 sur_refl_b01[0][0], 0.5054 sur_refl_b01[1][0], 0.3101 sur_refl_b01[2][0], 0.3967 sur_refl_b01[3][0], 0.4866 time, 5860, 5868, 5876, 5884  The [730:1:733] is getting the last four acquisition dates, but what are they? Fromjulian can convert the values to dates. https://lpdaacsvc.cr.usgs.gov/services/fromjulian?day= 5860, 5868, 5876, 5884 Jan 17, 2016;Jan 25, 2016;Feb 2, 2016;Feb 10, 2016  Tojulian can convert calendar dates to the from 01-01-2000 dates to calculate these values in reverse. 27 Data Services Middleware API UI

28. Demo  Other LP DAAC Middleware services provide higher level metadata interactions  Inventory Service – Lists LP DAAC granules and select granule information (powered by NASA CMR)  Lets take a look at this service…. 28 Data Services Middleware API UI

29. Demo  Inventory Service  The LP DAAC Inventory service returns select information about granules in the LP DAAC inventory. It utilizes the information in the NASA Common Metadata Repository (CMR)  URL for the Service  https://lpdaacsvc.cr.usgs.gov/servcies/inventory 29 Data Services Middleware API UI

30. Demo 30 Data Services Middleware API UI Service Help Response

31. Demo 31 Data Services Middleware API UI Service Form Response

32. Demo 32 Data Services Middleware API UI Service Form Response

33. Demo 33 Data Services Middleware API UI Service REST GET Call https://lpdaacsvc.cr.usgs.gov/services/inventory? product=MOD11A1.006& latitude=43.6& longitude=-96.7& date=2010-01-01,2016-03-01

34. Demo 34 Data Services Middleware API UI Service Data Response http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2010.01.01/MOD11A1.A2010001.h10v04.006.2016025150337.hdf http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2010.01.02/MOD11A1.A2010002.h10v04.006.2016025151521.hdf http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2010.01.03/MOD11A1.A2010003.h10v04.006.2016025163542.hdf http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2010.01.04/MOD11A1.A2010004.h10v04.006.2016025165101.hdf. http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2011.06.13/MOD11A1.A2011164.h10v04.006.2016057141329.hdf http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2011.06.14/MOD11A1.A2011165.h10v04.006.2016057152744.hdf

35. Demo 35 Data Services Middleware API UI Service REST GET Call https://lpdaacsvc.cr.usgs.gov/services/inventory? product=MOD11A1.006& bbox=-98.7,42.8,-96.7,43.6& julianrange=90-132& years=2006-2011& response=metadataurl&output=html

36. Demo 36 Data Services Middleware API UI Service Data Response http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.01/MOD11A1.A2006091.h10v04.006.2015283114357.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.02/MOD11A1.A2006092.h10v04.006.2015283133023.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.03/MOD11A1.A2006093.h10v04.006.2015283140035.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.04/MOD11A1.A2006094.h10v04.006.2015283153952.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.05/MOD11A1.A2006095.h10v04.006.2015283162912.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.06/MOD11A1.A2006096.h10v04.006.2015283174328.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.07/MOD11A1.A2006097.h10v04.006.2015283184609.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.08/MOD11A1.A2006098.h10v04.006.2015283200712.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.09/MOD11A1.A2006099.h10v04.006.2015283212242.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.10/MOD11A1.A2006100.h10v04.006.2015287200907.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.11/MOD11A1.A2006101.h10v04.006.2015287213340.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.12/MOD11A1.A2006102.h10v04.006.2015287233224.hdf.xml. http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2011.05.10/MOD11A1.A2011130.h10v04.006.2016055053025.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2011.05.11/MOD11A1.A2011131.h10v04.006.2016055055057.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2011.05.12/MOD11A1.A2011132.h10v04.006.2016055074200.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2011.05.13/MOD11A1.A2011133.h10v04.006.2016055080155.hdf.xml

37. Demo 37 Data Services Middleware API UI Click on the URL, get the metadata file 1.0 EDC SC:MOD11A1.006:2195949529 2195949529 2015-10-10 07:03:24.858 MOD11A1 6 MOD11A1.A2006091.h10v04.006.2015283114357.hdf 4309570 CKSUM 4191800068 DPLIngst. Hint: This would work the same for the browse URL or the data granule URL!

38. With the Building Blocks in Place What Next?  Using these building blocks the LP DAAC (or anyone who wants to really) can build new interfaces and systems like…. 38

39. AppEEARS  Application for Extracting and Exploring Analysis Ready Samples  Web application interface for accessing and extracting LP DAAC’s tiled MODIS and WELD data 39 Data Services Middleware API UI

40. AppEEARS  Built on top of services and tools  Capabilities  Easy access and selection of MODIS and WELD land data  Extracts pixel values for sample point locations  Data exploration and interaction  Dynamic Graphs  Tables  Download Bundle  Analysis-ready data 40 Data Services Middleware API UI

41. DEMO https://lpdaacsvc.cr.usgs.gov/appeears/ 41 Data Services Middleware API UI

42. Future Activities  The LP DAAC is planning on expanding its service offerings in the future  AppEEARS Area Services API and UI  Quality Values Services  Data Analysis Services  Addition of more tiled datasets into the API and UI 42 Data Services Middleware API UI

43. Future Activities  Area Services will allow for the selection of data over more than just the pixel space.  Data harmonization  Align different datasets  Sample datasets into comparable samples. 43 Data Services Middleware API UI

44. For More Information about our services 44 Subscribe to our listserv: https://lists.nasa.gov/mailman/listinfo/lpdaac

45. AppEEARS has an API too!  Even AppEEARS is built on top of public services, the AppEEARS API is directly usable, too (its not just an interface!)  AppEEARS API  Product Service – list of products available through AppEEARS (note the same help context)  Quality Service – list the quality layers for products  Sample Extraction Service – extracts a sample of data from OPeNDAP  DEMO 45 Data Services Middleware API UI

46. Future Activities 46 Data Services Middleware API UI

47. Demo 47 Data Services Middleware API UI Service Help Response - json

48. Demo 48 Data Services Middleware API UI Service Help Response - xml

49. Demo  If we know the dates we want are 10/08/2015, 10/16/2015, and 10/24/2015, we can find the number of days since 01-01-2000 for those dates https://lpdaacsvc.cr.usgs.gov/services/tojulian?day=10-08-2015,10-16-2015,10-24,2015 &output=f2k Output: 5759,5767,5775 Match those dates with the output form the time parameter to get that data. http://opendap.cr.usgs.gov/opendap/hyrax//MOD09A1.005/h11v04.ncml.ascii?time[717:1: 719] Dataset: h11v04.ncml time, 5759, 5767, 5775 49 Data Services Middleware API UI

50. Demo  Now we can get that data http://opendap.cr.usgs.gov/opendap/hyrax/GroupedAggregates/MOLT/MOD09A1.005/h11v04.nc ml.ascii?Latitude[1365:1:1365][176:1:176],Longitude[1365:1:1365][176:1:176],sur_refl_b01[717:1: 719][1365:1:1365][176:1:176],time[717:1:719]  Output Dataset: h11v04.ncml Latitude[0], 44.3062499960216 Longitude[0], -96.7900810678634 sur_refl_b01[0][0], 0.0656 sur_refl_b01[1][0], 0.0772 sur_refl_b01[2][0], 0.0654 time, 5759, 5767, 5775 50 Data Services Middleware API UI

51. Demo  Product Service  The LP DAAC Product service returns select information about products in the LP DAAC archive.  Coming soon to URL  https://lpdaacsvc.cr.usgs.gov/servcies/product 51 Data Services Middleware API UI

52. Demo 52 Data Services Middleware API UI Service Help Response

53. Demo 53 Data Services Middleware API UI Service Help - Form Response

54. Demo 54 Data Services Middleware API UI Service Form Entry

55. Demo 55 Data Services Middleware API UI Service Data Response [ { "collection_data": { "title": "Surface Reflectance 8-Day L3 Global 500m", "version_id": "006", "collection_name": "MOD09A1.006", "short_name": "MOD09A1", "data_center": "LP DAAC" } } ]

56. Demo 56 Data Services Middleware API UI Service REST GET Call https://lpdaacsvc.cr.usgs.gov/services/product?product=MOD09A1.006&response=layers

57. Demo 57 Data Services Middleware API UI Service Data Response [ { "collection_data": { "layers": [ { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 1", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b01", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 2", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b02", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 3", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b03", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 4", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b04", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 5", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b05", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 6", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b06", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 7", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b07", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "32-bit unsigned integer", "scale_factor": "NA", "description": "Surface reflectance 500m band quality control flags", "units": "Bit Field", "offset": "\n", "valid_range": "0 to 4294966275", "pixel_size": "\n", "sds_name": "Sur_refl_qc_500m", "rows_cols": "\n", "fill_value": "4294967295" }, { "data_type": "16-bit signed integer", "scale_factor": "NA", "description": "Surface eflectance 500m state flags", "units": "Bit Field", "offset": "\n", "valid_range": "0 to 57343", "pixel_size": "\n", "sds_name": "Sur_refl_state_500m", "rows_cols": "\n", "fill_value": "65535" }, { "data_type": "16-bit signed integer", "scale_factor": "0.01", "description": "MODIS solar zenith angle", "units": "Degrees", "offset": "\n", "valid_range": "0 to 180", "pixel_size": "\n", "sds_name": "Sur_refl_szen", "rows_cols": "\n", "fill_value": "0" }, { "data_type": "16-bit signed integer", "scale_factor": "0.01", "description": "MODIS view zenith angle", "units": "Degrees", "offset": "\n", "valid_range": "0 to 180", "pixel_size": "\n", "sds_name": "Sur_refl_vzen", "rows_cols": "\n", "fill_value": "0" }, { "data_type": "16-bit signed integer", "scale_factor": "0.01", "description": "MODIS relative azimuth angle", "units": "Degrees", "offset": "\n", "valid_range": "-8000 to 18000", "pixel_size": "\n", "sds_name": "Sur_refl_raz", "rows_cols": "\n", "fill_value": "0" }, { "data_type": "16-bit signed integer", "scale_factor": "NA", "description": "Day of the year for the pixel", "units": "Julian day", "offset": "\n", "valid_range": "0 to 366", "pixel_size": "\n", "sds_name": "Sur_refl_day_of_year", "rows_cols": "\n", "fill_value": "-1" } ] } } ]