1. UAF-grid project status Steve Hankin 12 Jan., 2010

2. UAF-grid approach (recap) A unified access framework for NOAA gridded data Initial focus: gridded data. Why? –relatively mature data standards, network protocols, and clients 1.Assemble a well-maintained collection linked by a THREDDS catalog 2.Attach popular clients (both desktop and Web) 3.Pull together the best metadata we reasonably can (but don’t stop the show!) 4.Advertise the capabilities 5.Build on success: add profiles and time series; add non- NOAA data; improved metadata; …

3. What’s been accomplished so far Created a secure, version-controlled mechanism for UAF tech members collaboratively to add elements to THREDDS catalog (a hassle) Collaborative adding of sites, editing, improving … Serve the resultant catalog at a public site http://ferret.pmel.noaa.gov/thredds/geoideCatalog.html

4. The current catalog GEO-IDE OARNMFSNOSNESDISNWS PMELGFDL …NODCNGDC … We refer to the present version as the “raw catalog”

5. Site 1 DAP URL Site 2 DAP URL GEO-IDE DAP URL the “raw” catalog Non-netCDF-like, or non-gridded In need of aggregation

6. Site 1 DAP Aggregated URL DAP URL Site 2 DAP URL GEO-IDE the “clean” catalog

7. The “raw” catalog is an XML highway map that links many datasets Challenge we discovered: 1.a high proportion of files are not netCDF-like (e.g. CSV or image files) or non-gridded (e.g. 250K Argo profile files) 2.gridded files in need of aggregation dominate (numerically) the ready-to-use CF datasets Aggregation will consolidate by x10-1000 the number of apparent datasets We need to peer through a forest to see the trees.

8. how to create a clean catalog Short term (weeks from now) Make the Web crawler smarter –ignore extraneous stuff –detect need for aggregations –auto-aggregate Goal (say, 5 months from now) Crawler  feeds database  analysis inside database  clean catalog

9. next steps - in progress 1.Continuing to add sites to the “raw catalog” 2.NGDC – creating data base (just got PMEL’s crawler) 3.Make the crawler smarter: a)by this Friday -- able to exclude 1D datasets b)by next Friday -- outputs a clean catalog by dropping aggregatable datasets c)by end of the month – clean catalog with auto- aggregations 4.(starting at 3b – 1 st “clean catalog) connect applications

10. Thank you Questions?