Birmingham/Warwick AstroGrid Science Workshop Meeting Goals Silvia Dalla Oct 10-11, 2006

Workshop goals Aim to work with you, the early users of AstroGrid, to demonstrate its capabilities and the potential of the Virtual Observatory (VO) Introduce you to the elements of the system and how to do science with it Look at a science problem in your research area and use VO tools to tackle it

Workshop goals (contd) Collect your feedback  What you like/dislike  What needs to be improved  Which data you need  Which applications you want to see running in AG

Workshop Plan Today: background talks on AstroGrid and how the system works (am). Initial workgroup set up (pm) Tomorrow: workgroups work on science cases. Report and feedback Session (pm) Don’t miss the dinner (Today – 7:00 pm The Chamon)

Logistics Rooms for workgroups:  A Extra-Galactic: Astro Terminal Room (227)  B Stellar: Astro Meeting Room (229)  C Solar: HiROS Room (G38) Enjoy…

What you can do with AstroGrid AstroGrid Science Team Silvia Dalla, Eduardo Gonzalez-Solares, Jonathan Tedds, Anita Richards and Nicholas Walton

Outline 1.Background: Virtual Observatories (VOs) and AstroGrid 2.Example of non-VO solution to a data retrieval and processing task: making a movie of solar images 3.The AstroGrid solution 4.Introduction to AstroGrid components

What is the VO? VO = collection of integrated astronomical data archives and software tools that utilize computer networks to create an environment in which research can be conducted Thanks to Ohishi Masatoshi (NAOJ) for the pointer…

VOs worldwide Similar efforts now in 15 countries: –UK, USA, Canada, France, Germany, Italy, Holland, Japan, Australia, India, China, Russia, Hungary, South Korea, ESO, Spain Active collaboration among projects –Standards, common demos –International VO roadmap being developed –Regular telecons over 10 timezones Formal collaboration: International Virtual Observatory Alliance (IVOA)

AstroGrid: the UK’s VO Consortium of several UK Universities, funded by PPARC to build a VO for the UK. Part of the Euro-VO Astrogrid release : Data discovery, access to data and applications.

AstroGrid Goals Enable science by: –Improving the quality, easy, speed of on-line astronomy –Making integration and comparison of data from diverse sources transparent –Removing data access barriers to multiwavelength analysis –Enabling access and manipulation of large datasets

The need for a VO Data volume doubles every year Increase of size and multiplex capabilities of new instruments –WFCAM: 100Gb/night (100Tb/yr) –VISTA: 300Gb/night (300Tb/yr) –e-MERLIN/ALMA: 1Tb/day –SDO: 1 Tb/day We will have Petabytes of data by 2010

The need for a VO Hubble UDF Million second exposure 6000x6000 pix 11.5 sq. arcmin 10,000 galaxies

The need for a VO SWIRE ELAIS N1 9 sq. degrees (~3000 UDF) (moon ~0.2 sq. deg.) ~ 600,000 objects

The need for a VO Solar Dynamics Observer Launch 2008 >1 TB/day Over 5 years: 1.8 PB

Making a solar movie – non VO Web interface to database of SOHO/EIT observations

Making a solar movie – non VO Download images to local machine

Making a solar movie – non VO IDL SolarSoft required to calibrate images and make a movie. Routines available, however each user rewrites code calling them to produce a movie.

Making a solar movie – non VO: weaknesses If user wishes to make a movie for a different time period, the above steps need to be repeated – by hand Similarly for astronomy data if one is interested in many objects The entire archive is not ‘visible’ to the user – only the downloaded subset can be processed Scripting eg with Python, Perl etc is possible though code is different for different archives – not easy for the ‘general’ user / for many datasets. This limits feasibility of multi-wavelength, multi-instrument work

Making a solar movie – with AG Solar Movie Maker science workflow. Other science workflows available: Redshift Maker, Colour Cutter, Cone Search, SWIRE images

Making a solar movie – with AG: under the hood Access to database of observations – via AG DSA (DataSet Access) software Requests are sent using ADQL (Astronomy Data Query Language), similar to SQL Database of observations (at archive) DSA AG Workbench/ workflow engine Input: ADQL query Output: table of observations satisfying the query, in VOTable format

Making a solar movie – with AG: under the hood: CEA CEA AG workbench/ workflow engine Input: CEA application input parameters Output: whatever the output of the application is, delivered to user’s Myspace Capability to call a set of processing routines that do operations on data, or a model (apps in any programming language can be wrapped) – CEA Application Application. Eg software that processes data, model etc (on apps server)

Making a solar movie – with AG: under the hood: Workflow Workflow capability so that queries to datasets and calls to applications can be managed Jobs are run remotely and asynchronously Queries and workflows can be re-used and shared AstroGrid is currently the only VO project with a workflow capability

AstroGrid: Workbench Workbench gives access to AG services Data discovery: AstroScope and HelioScope Task Launcher – send a query to a database or launch a application

AstroGrid: Workbench- contd Run a ready made science workflow Build your own workflow View your files in MySpace

AstroGrid: MySpace Virtual disk space where you can store results, temporary files, and new things like query files and workflow files, so you can adjust and re- run jobs on a later day. Visible from any computer.

Finding Information : the Registry How do you find the data you require? How do you decide which resource (data, application, information, disk, …) to use? The registries are the yellow pages for astronomical resources All VO registries harvesting each other: thus querying any one returns full list of globally held resources.

VOTable: an interchange format XML standard for the interchange of data represented as a set of tables A table is an unordered set of rows, as specified in the table metadata Each row is a sequence of table cells, each of them containing a primitive data type or an array of such primitives It bridges two ways to express structured data: XML and FITS Use of Universal Column Descriptors (UCDs) to express the content of each parameter.

<VOTABLE version="1.1" xmlns:xsi=" xsi:noNamespaceSchemaLocation=" Velocities and Distance estimations <PARAM name="Telescope" datatype="float" ucd="phys.size;instr.tel" unit="m" value="3.6"/> <FIELD name="RA" ID="col1" ucd="pos.eq.ra;meta.main" ref="J2000" datatype="float" width="6" precision="2" unit="deg"/> <FIELD name="Dec" ID="col2" "pos.eq.dec;meta.main" ref="J2000" datatype="float" width="6" precision="2" unit="deg"/> <FIELD name="Name" ID="col3" ucd="meta.id;meta.main" datatype="char" arraysize="8*"/> <FIELD name="RVel" ID="col4" ucd="src.veloc.hc" datatype="int" width="5" unit="km/s"/> <FIELD name="e_RVel" ID="col5" ucd="stat.error;src.veloc.hc" datatype="int" width="3" unit="km/s"/> <FIELD name="R" ID="col6" ucd="phys.distance" datatype="float" width="4" precision="1" unit="Mpc"> Distance of Galaxy, assuming H=75km/s/Mpc N N N Table Metadata Table data (XML, FITS) VOTable Example

Visualising results: Topcat

Astrogrid Help Pages Linked from workshop pages

Summary Workshop is an opportunity to see AG in action Current capability can be of use to a diverse range of science problems New cases – perhaps emerging from this workshop – may demand new science services: input is welcome from you!