Presentation is loading. Please wait.

Presentation is loading. Please wait.

BULK DATA RETRIEVAL ECHO Technical Interchange Meeting April 30 & May 1, 2013 Raytheon EED Program | ECHO Technical Interchange 2013.

Similar presentations


Presentation on theme: "BULK DATA RETRIEVAL ECHO Technical Interchange Meeting April 30 & May 1, 2013 Raytheon EED Program | ECHO Technical Interchange 2013."— Presentation transcript:

1 BULK DATA RETRIEVAL ECHO Technical Interchange Meeting April 30 & May 1, 2013 Raytheon EED Program | ECHO Technical Interchange 2013

2 What is it For? Quick access to Publicly Available Data via URLs No processing options User Driven Pull Near-instant Raytheon EED Program | ECHO Technical Interchange 2013

3 State of the Union (Reverb) Put items in your cart, click “Download” URL Options: Data Metadata Browse Download Options Text File FTP Batch Script Raytheon EED Program | ECHO Technical Interchange 2013

4 How Does Reverb Do It Catalog-REST! Granule Searches “atom” format results Scan for “links” to URLs Download a file containing those links. Raytheon EED Program | ECHO Technical Interchange 2013

5 How do we do it? Catalog-REST! Granule Searches “atom” format results Scan for “links” to URLs Create a file containing those links. Get them. Raytheon EED Program | ECHO Technical Interchange 2013

6 Example (cURL)* curl -gG “https://testbed.echo.nasa.gov/catalog- rest/echo_catalog/granules.atom?echo_collection_id=C 3878-LPDAAC_ECS&bounding_box=10.488%2C %2C53.331%2C68.906&temporal[]= T10%3A00%3A00Z%2C T12%3A00%3A00Z” This gets all granules with: echo_collection_id of: C3878-LPDAAC_ECS Spatial bounding box: , , , (W, S, E, N) Time constraint: T10:00:00Z T12:00:00Z ~80 hits! Use -I as options to curl, and look for: “Echo-Hits” * also from perl/bulk/get_bulk.pl Raytheon EED Program | ECHO Technical Interchange 2013

7 What do the Results Look Like? G10607-LPDAAC_ECS SC:MCD43A4.005: T14:01:49.076Z MODIS/Terra+Aqua Nadir BRDF-Adjusted Reflectance 16-Day L3 Global 500m SIN Grid V005 MCD43A4.A h21v hdf LPDAAC_ECS T00:00:00.000Z T23:59:59.999Z e e true DAY YIKES! Raytheon EED Program | ECHO Technical Interchange 2013

8 “link” is your friend! Raytheon EED Program | ECHO Technical Interchange 2013

9 Only link…/data# Please! curl -gG “https://testbed.echo.nasa.gov/catalog- rest/echo_catalog/granules.atom?echo_collection_id=C 3878-LPDAAC_ECS&bounding_box=10.488%2C %2C53.331%2C68.906&temporal[]= T10%3A00%3A00Z%2C T12%3A00%3A00Z” | perl –ne “printf if m/link.*\/data#/;” “Just show me the results that have ‘data’ type in the links” Slightly more clever perl: … | perl -anF/\”/ -e “printf qq(\$F[1]\n) if m/link.*\/data#/;” But watch out for Windows vs. Mac/Linux quoting! Raytheon EED Program | ECHO Technical Interchange 2013

10 Loop it Over your Echo-Hits append &page_size=500 to URL_String end_pages = (Echo-Hits DIV page_size) + 1 for page 1.. end_pages curl (string+&page_num=$page) | clever.perl >> output.URLs Now you have an output.URLs with lots of URLs in them… Raytheon EED Program | ECHO Technical Interchange 2013

11 So What if I Have Some URLs? Scripting curl to the rescue! Linux/Mac/Unix: for url in $(

12 Questions? Raytheon EED Program | ECHO Technical Interchange 2013


Download ppt "BULK DATA RETRIEVAL ECHO Technical Interchange Meeting April 30 & May 1, 2013 Raytheon EED Program | ECHO Technical Interchange 2013."

Similar presentations


Ads by Google