1. VSO Status Update HDMC Meeting 9 June 2008

3. Vision To allow solar physicists to identify and search for data even if they don't know it exists. Make it easier for them to get to the data whether they knew where it was before or not.

4. Why VSO? Why not CDAWeb? Will they take > 2TB/day of data? Why not FTP and VSPO: Do you want to scan > 70k files per day to determine which ones meet your needs? Every time you perform a search? (these numbers are just for SDO)

5. How to accomplish: Standardized API for querying & requesting data using a consistent model at each provider System to allow UIs to query a single API and have the queries distributed & responses aggregated.

6. Current Status Data Providers : 14 Observatories : 26 Instruments : 55 Data Sets : 180?... dating back to 1915

7. Current Status (SPASE terms) Repositories : 19 Observatories : 26 Observatory Groups : 3 Instruments : 66+ Instrument Groups : ~10 Numerical Data : ~180 products Display Data : (not tracked directly) Granules : ? millions

8. Data Sets Planned: In progress: Hinode/SOT Huairou NRL EIT & LASCO TIMED/SEE Planned: Hinode/EIS SMART SDO... and catalogs

9. SPASE use: um... no?

10. Current Interfaces / APIs VSO SOAP API (finding & ordering): Web-based interface 3 custom VSO-derived interfaces (NSO, SAO, SSC) IDL-based interface Dominic Zarro's IDL commands in SSW Perl command line interface AstroGrid Registry API Catalog 'Display' API for HelioViewer

11. Interfaces / APIs In progress: IDL client for Registry API More complex Catalogs API(s) Planned: Cart API and UI in IDL Catalog APIs for HELM access via IDL SPASE compliant registry API or at least SPASE descriptions of holdings

12. Data Coverage: Incomplete. Measurement method: People are still asking us to add data sets.

13. Goals: Prepare for the flood of SDO data (and other recent missions: STEREO, Hinode )

14. Plans for the Future: Catalogs Summary Rows Looking to the work of others

15. Empowering new searches! We’re doing some work under the hood!

16. Moving beyond time An event or feature driven approach Science catalogs LASCO CME, GOES X-ray Flare, NOAA Active Region Number, Coronal Holes, Bright points etc. Science queries One of the drivers for this SDO So much data need to move to feature / event driven approach, But useful now …

17. Give me all of these! Actually I really want all XRT jets observed on the limb with coincident SOT and EIS data. Would have made things much easier if I could have performed this query!

18. Science Queries How do we know what queries to support? We get asked! “It would be nice if … / I’d really like to be able to …” Coincident observations Observations of same features in multiple instruments within specified time window. Cadence Find an hours worth of observations where there are images every minute. (Helioseismology) DEM Datasets Find XRT datasets suitable for DEM analysis. (Closely spaced observations, number of different filters/wavelengths). EIS data would be a bonus!

19. Science Queries Investigating EIT dimming regions as proxies for CMEs Combine GOES Flare List + LASCO CME list to generate list of on disk flares without associated CMEs. Compare list to catalog of EIT dimming regions. Do spatial comparison. Generate list of EIT dimming regions whose properties are observed to be associated with CME list. Currently waiting on EIT dimming region catalog! Can do the first part to at least generate list of EIT images that should be analyzed. Support driven by science questions!

20. Conclusions We are moving from just data discovery in a certain time range, to finding datasets of interest that fit a number of scientific criteria. Searches for features and events and in the process blurring the distinction between data and catalogs. SDO will require these new approaches. But it’s going to return a LOT of data.

21. Plans : Summary Rows VSO data providers can return a record summarizing multiple data granules Data Providers can define their own groupings, UIs can decide to expand them. Summary products still work like regular products Data providers serve a tarball or data cube in response to a data transfer request UIs that don’t understand summary products can still retrieve the collection Changing over to a new JavaScript toolkit to provide AJAX summary row expansion

22. Plans : Look to Others Catalogs Bobby Candey's HELM project (VxO funded) Peter Fox's SESDI project (NASA funded) Related IVOA & astronomy standards Presentation / UI Use existing JavaScript toolkits ExtJS and GWT... still requires customization to serve our needs, but gives head start

23. Plans : Look to Others (2) Massive amounts of data: How libraries handle interlibrary loan How libraries handle 'de-duplicating' of records Functional Requirements of Bibliographic Records … is going to require significant changes at the data providers.

24. The Future of VSO We are moving from just data discovery in a certain time range, to finding datasets of interest that fit a number of scientific criteria. Searches for features and events and in the process blurring the distinction between data and catalogs. SDO will require these new approaches.

25. http://virtualsolar.org/