Aus-VO: Progress in the Australian Virtual Observatory Tara Murphy Australia Telescope National Facility

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: who we are Science infrastructure effort, funded by ARC LIEF grants Science infrastructure effort, funded by ARC LIEF grants The Aus-VO Partner Institutes are: The Aus-VO Partner Institutes are: –CSIRO Australian Telescope National Facility –The University of Melbourne –The University of Sydney –Anglo-Australian Observatory –Swinburne University of Technology –The University of Queensland –Monash University –The University of New South Wales –Victorian Partnership for Advanced Computing –ANU Mount Stromlo Observatory & Supercomputing Facility Plus collaborators: Plus collaborators: –CSIRO Mathematical and Information Sciences –Australian Partnership for Advanced Computing

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 APAC Astronomy Grid Grid infrastructure effort Grid infrastructure effort Funded by the Australian Partnership for Advanced Computing (APAC) Funded by the Australian Partnership for Advanced Computing (APAC) Active members & partners are: Active members & partners are: –Melbourne and Swinburne –UNSW and Sydney –ANU Project Leader: David Barnes Project Leader: David Barnes Research Leader: Rachel Webster Research Leader: Rachel Webster

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: our focus Bring legacy astronomy archives on-line Ensure future project compliance Provide access to archived realizations of simulations Describe all data fully, and support a small and well-chosen set of interoperability protocols Develop tools and interfaces to find, acquire, process and visualize data Build national grid infrastructure to host the data, tools and interfaces

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: what we do Science content Science content –providing data –providing software –metadata –registry content Science services –source finding –coordinate transformations –catalogue matching –visualization –interfaces to legacy environments

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 APAC Grid: what we do Data grids – –data store – –data access protocols – –authentication & authorisation – –network facilities Compute grids – –workflow – –job configuration, submission & monitoring – –code-to-data paradigms

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: current research Data mining – –catalogue matching – –searches for new empirical relationships & rare objects Legacy software integration – –using legacy code in VO services – –using VO services in legacy software environments – –transparent access to Web and Grid Services from the desktop Visualisation – –remote visualisation service (legacy s/w drawing to remote “canvases” via Corba, SOAP, …) – –VOTable to VRML service – –volume rendering

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: projects 2004 Common format on-line archive projects: – –HIPASS catalog: HI Parkes All Sky Survey: neutral Hydrogen spectral line survey, 4,300 sources with ~30 parameters and 1024-channel radio spectra – –SUMSS catalog: Sydney University Molonglo Sky Survey: radio continuum survey at 843 MHz, 107,000 sources with ~15 parameters – –2dFGRS QSO catalog: 2-degree Field Galaxy Redshift Survey: optical spectra of >20,000 southern quasi-stellar objects – –ATCA archive: Australia Telescope Compact Array archive: all observations since 1988, circa 1.5 TB of more than 1,000 separate observing projects! Substantial exercise in describing data with metadata. – –MACHO, RAVE, Pulsar timing archives, Gemini archive and more in 2004!

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: projects 2004 Server- and client-based visualisation tools: – –client canvas for legacy software package AIPS++ to draw upon from a remote server. – –3d visualisation of catalogues Data reduction pipelines for: – –Molonglo Observatory Synthesis Telescope – –AT Compact Array archive – –Gemini, for Australian-used instruments Portal for theoretical astrophysics jobs: – –configuration and execution – –monitoring – –result retrieval and initial analysis and display

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 The Australia Telescope Online Archive Consists of: Consists of: 1.an archive of all data observed with the Australia Telescope Compact Array 2.a data reduction pipeline to process the data for ‘quick and dirty’ images 3.a remote visualisation system to inspect the images on the client machine People: People: –Peter Lamb, Tara Murphy, Anil Chandra, Praveena Tokachichu, Chris Owen

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune The archive All ATCA data since 1990 All ATCA data since 1990 Total of 1.9 Tb covering over 1000 projects Total of 1.9 Tb covering over 1000 projects Hosted on a machine with 2GB memory and 6TB of disk Hosted on a machine with 2GB memory and 6TB of disk Back end is a 20 node Beowulf cluster Back end is a 20 node Beowulf cluster

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune The archive Challenges: Challenges: –Unifying metadata to fully describe data –Capturing the intentions of the observers determining the calibration strategy determining the calibration strategy –Dealing with the significant minority of non-standard observations

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune The pipeline Automatically runs complete data reduction process for ATCA data Automatically runs complete data reduction process for ATCA data Underlying work done by AIPS++ Underlying work done by AIPS++ Methods exposed as web services using Glish/Java Methods exposed as web services using Glish/Java Uses metadata from archive to make decisions about reduction Uses metadata from archive to make decisions about reduction

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune The pipeline Challenges: Challenges: –Dealing with legacy code –Dealing with legacy data (via the archive) –Getting the pipeline to make sensible decisions about how to process the data

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune Visualization system A software system that allows scientists to view and analyse astronomical data that are stored in remote locations. A software system that allows scientists to view and analyse astronomical data that are stored in remote locations. RVS Server User Data Centre User connects to view data Visualisation Data (Image) sent to user RVS response Dataset sent to RVS Browse data

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune Visualization system Challenges: Challenges: –Performance RVS Image data format RVS Image data format Connection speeds between services Connection speeds between services –Session Management How to coordinate data centres and clients? How to coordinate data centres and clients? Caching of images for future sessions Caching of images for future sessions

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 ATOA Walkthrough 1. Query the archive Query the archive Query the archive 2. Query results Query results Query results 3. Automatic data reduction Automatic data reduction Automatic data reduction 4. Visualise data Visualise data Visualise data

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 2004 Aus-VO: links RVS will be public in ~2 months RVS will be public in ~2 months The ATOA will be public in ~2 months The ATOA will be public in ~2 months The prototype pipeline will be public in ~3 months The prototype pipeline will be public in ~3 months