1. Heliophysics Integrated Observatory (HELIO) HELIO The Heliophysics Integrated Observatory Bob Bentley (UCL-MSSL) Andre Csillaghy (FHNW) Jean Abourdarham (Obs. Paris) 20 November 2008 European Space Weather Week, Brussels

2. Heliophysics Integrated Observatory (HELIO) 2 Heliophysics Heliophysics explores the Sun-Solar System Connection Space weather is a subset of Heliophysics – SWx++ A virtual observatory that supports Heliophysics must facilitate access to data from a number of communities Solar, heliospheric, magnetospheric and ionospheric physics, aeronomy… Throughout the solar system – not just Sun-Earth Globally – beyond national or European level Heliophysics sits in the boundary between communities Astrophysics and Planetary sciences (including Earth sciences) A VO for heliophysics must be aware of the need to support the interests of these communities Heliophysics Integrated Observatory (HELIO) Proposal submitted under EC’s FP7

3. Heliophysics Integrated Observatory (HELIO) 3 Driven by science Desire to solve science problems that span disciplinary boundaries is driving the need to provide integrated access to data across the communities The communities have evolved independently over decades Each has very different ways of describing, storing and exploiting the data from their observations, varying use of standards To facilitate access, we need to find ways to: Tie the data together through searches across all the domains Present any results in a form that does not require a detailed understanding of each discipline U.Alaska

4. Heliophysics Integrated Observatory (HELIO) 4 HELIO Concept HELIO concept implemented with a service oriented architecture Service to curate and access search metadata Search engine implemented as a service Services to access data repositories Services to extract and process required observations Workflow tool used to bind the services together User interface closely liked with this tool Domain interoperability facilitated by semantic-driven approach Single data model for all domains not practical Service oriented architecture allows a researcher to use components of HELIO as they wish – as a chain or as individual services Chaining of services will provide maximum support for new users and allows a diverse community to study complex science problems in heliophysics Users may only require access to metadata and employ their own search tools or may just wish to use the dataset location capabilities Format of data products can be tailored to meet community needs

5. Heliophysics Integrated Observatory (HELIO) 5 Issues related to Metadata Metadata are the key to accessing observations Rapidly increasing volumes make good metadata essential There are many issues related to existing metadata Poor quality or missing, lack of interoperability… This affects ability of all users to do science – not just HELIO To resolve the issues requires the re-evaluation of capabilities provided within each community and some corrective action HELIO plans to work closely with the community on this Metadata can be grouped in several ways, one is: Search metadata o Metadata used to identify interesting time intervals and locations Observational metadata o Metadata used to describes the observations, e.g. FITS headers Storage metadata o Metadata that describes how the data are stored and accessed Administrative metadata o Metadata that allows the system to exploit the available resources

6. Heliophysics Integrated Observatory (HELIO) 6 Data Storage Access to data should be a matter of mechanics and generic Cheap storage and the Internet have greatly enhanced access Well established protocols for access – http, ftp, … How data are stored within a data source can make a lot of difference to their accessibility A variety of file formats should be accommodated EGSO has the concept of resource-rich and resource-poor providers Resource-rich – should provide what is needed in response to simple query Resource-poor – may only be able to make data accessible over Internet Guidelines/standards on ways that data should be organized could improve capabilities of all providers Providing data following simple naming conventions in an ordered directory structure would make them simpler to access Simple catalogue (textual?) might provide additional information Discussed at VOiG in 2007; IAU WG discussion

7. Heliophysics Integrated Observatory (HELIO) 7 Observational metadata Provides information about how the observation was made Important for exploiting the data; key for HELIO Often quality issues related to the metadata that is provided Parameters sometimes missing, or wrong Inconsistent use of information, “synonyms” for keywords In solar data, space-based observations much better described than their ground-based counterparts o Ground-based observations are only source in some wavelengths and need access to as many observatories as possible Researchers often used to deficiencies in their own domain Difficult for machines to handle if it is not quantified properly Consequence can be that the data are “unusable” Need to encourage all organizations that generate data to adopt and comply with agreed standards Where possible the standards should have generic parts to facilitate interoperability

8. Heliophysics Integrated Observatory (HELIO) 8 Searches In heliophysics, we are interested in how an event on the solar surface can propagate through the heliosphere and affect planetary environments May also want to work backwards and look for the cause of an effect – what solar event caused this ionospheric activity… Searches should identify interesting time intervals based on a combination of event, features, etc. metadata Light curves and images my also be used to augment the search Location of observer affects whether phenomena seen Each community of some combination of these metadata Differences in how some quantities are expressed, what are included There are concerns about the quality and integrity of these metadata and whether they are adequate to support the searches HELIO would like to undertake

9. Heliophysics Integrated Observatory (HELIO) 9 Solar search metadata Searches in solar physics are mainly event driven Phenomena occur on or near the solar surface Event data gives time and location of phenomena Feature data provides details of location and size of structures that may be relevant Time information can be expressed in many ways Essentially these are the same, with simple transformations Spatial information can be expressed in terms of: Coordinates in the observing frame – e.g. arcsecs from disc centre Coordinates on the rotating body of the Sun – Carrington coordinates The location of the observer largely ignored o Helio-seismology is an exception In the bigger picture of Heliophysics, also need to include the viewing perspective (c.f. STEREO)

10. Heliophysics Integrated Observatory (HELIO) 10 Other search data Observation of phenomena in the heliosphere and near/on planets are more complex For in-situ observations in the heliosphere Time is when a phenomena affected (passed) the observer Position of the observer relative to the Sun is key to understanding When the in-situ observations are made on/near a planet Position of the observer relative to the planet is also important Relating events that are defined from in-situ data to those on/near the Sun requires an understanding of how events propagate Details of the velocity structure of CMEs and the solar wind are not easy to determine… HELIO plans to develop a tool that will use even/feature data to refine a model to trace effects forwards from causes, etc.

11. Heliophysics Integrated Observatory (HELIO) 11 Simple, but not so simple In principle this all seems fairly obvious, but lets look in detail at some common solar event data On 20 January 2005 there was an X7.1 flare that was intensely geo- effective. The flare was associated with particle event and a CME; it was also observed by ground-level neutron monitors – a GLE. Many superlatives were used to describe the event o "The solar energetic particle event of January 20 2005 has been called, by some measures, the most intense in 15 years..." (Mewaldt et al., 2005) o ”The fastest rising SEP event of current cycle [cycle 23]" (Rawat et al., 2006) o ”The most spectacular [solar event] of the Space Age" (Tylka et al., 2006) o ”The largest GLE [GLE 69] in half a century" (Bartol Research Institute) But event is absent from the NOAA SEC list of "Solar Proton Events Affecting the Earth Environment" When you look at the data and how lists are created, you realize that the lists are deficient in several ways Humans and SmFCACs can understand what happened, but It is harder for machines...

12. Heliophysics Integrated Observatory (HELIO) 12 X7.1 of 20 Jan 2005 The event was one of several from AR 10720 Two other X class flares and several M class flares occurred in previous 3 days; others before this

13. Heliophysics Integrated Observatory (HELIO) 13 X7.1 of 20 Jan 2005

14. Heliophysics Integrated Observatory (HELIO) 14 X7.1 of 20 Jan 2005 At the time of the event, the proton levels had not returned to normal after previous events The criteria fails to recognize a new event o NOAA lists event on 16 Jan The X-ray data also suffers from problems The end of an event is defined by when the counts drop to 50% o New events can “interrupt” existing events The shape and true duration of the decay phase are lost o NOAA gives start 0636; end 0726 Not all locations are tagged!! Significant brightenings seen on images not declared as flares

15. Heliophysics Integrated Observatory (HELIO) 15 Some of the problems Automated searches are difficult when major events can be “missed” A search for long duration events would yield spurious results If the locations of all flares are not known (in a timely fashion), it is impossible to know whether they will be geo-effective Instrument flare lists have gaps – nights, off times, etc. – but the reason for a null result is not included

16. Heliophysics Integrated Observatory (HELIO) 16 Some of the problems Automated searches are difficult when major events can be “missed” A search for long duration events would yield spurious results If the locations of all flares are not known (in a timely fashion), it is impossible to know whether they will be geo-effective Instrument flare lists have gaps – nights, off times, etc. – but the reason for a null result is not included

17. Heliophysics Integrated Observatory (HELIO) 17 Improving Metadata Existing event lists can give a distorted picture what has occurred Such deficiencies make it difficult for non-experts to use them The community “knowledge” is not written down Need to re-evaluate/regenerate the event data in all domains with the idea that they will be used in a joint search across the domains Ensure events are described more accurately Include information that might explain null results Metadata should comply with agreed standards that have generic components to ensure interoperability The situation has changed with enhancements to technology o Providers need to ensure they are more compliant Virtual observatories should try to handle problems with old metadata o Limits to what can be achieved if metadata is poorly formed Standards need to be developed in collaboration with the community and funding agencies

18. Heliophysics Integrated Observatory (HELIO) 18 Conclusions We have the technology, but developing a virtual observatory to support heliophysics will not be simple Cooperation of the community is essential if we are to succeed Some of the possible problems have been highlighted The quality of observational metadata needs to be improved We must improve the quality and content of the search metadata These problems affect the Space Weather community in a similar way to the Heliophysics community To address these issues we need to engage the communities in all the domains that constitute heliophysics and develop standards that will facilitate the process HELIO has a large Networking component that is targeted at addressing these issues and fully involving the community

19. Heliophysics Integrated Observatory (HELIO) 19