1. Future of Astronomy: enormous datasets, massive computing, innovative instrumentation Rachel Webster & David Barnes (Project Leader & Project Scientist, Australian Virtual Observatory) School of Physics, The University of Melbourne

2. Topics 1.Astronomy is a theoretical and observational science with massive heterogeneous datasets. 2.The Virtual Observatory (VO) 3.Aus-VO: the Australian project 4.A new local opportunity: MWA low frequency array Read here for a summary of each slide…..

3. 2. What is a Virtual Observatory? A Virtual Observatory (VO) is a distributed, uniform interface to the data archives of the world’s major astronomical facilities.A Virtual Observatory (VO) is a distributed, uniform interface to the data archives of the world’s major astronomical facilities. A VO is realised with advanced data mining and visualisation tools which exploit the unified interface to enable cross-correlation and combined processing of distributed and diverse datasets.A VO is realised with advanced data mining and visualisation tools which exploit the unified interface to enable cross-correlation and combined processing of distributed and diverse datasets. VOs will rely on, and provide motivation for, the development of national and international computational and data grids.VOs will rely on, and provide motivation for, the development of national and international computational and data grids. Virtual observatories will effect a “sea change” in the way astronomy is done.

4. International Data deluge! Dozens of new surveys 2003 to 2008Dozens of new surveys 2003 to 2008 Many (10 – 100) terabytes per surveyMany (10 – 100) terabytes per survey 10 – 100 researchers per survey10 – 100 researchers per survey International collaborations (almost always)International collaborations (almost always) Data is non-proprietary (usually)Data is non-proprietary (usually) Surveys are no longer within the scope of the solo researcher, and also cannot be accommodated by isolated computing and storage facilities. Enter Grid Computing and the Virtual Observatory New surveys of the whole sky need a new paradigm: enter the Virtual Observatory.

5. 3. Aus-VO and APAC Grid Project 10 institutions; 4 large grants over 2-4 years (LIEF & APAC)10 institutions; 4 large grants over 2-4 years (LIEF & APAC) VO Data Warehousing (10 major datasets)VO Data Warehousing (10 major datasets) Gravity Wave Research GridGravity Wave Research Grid VO Theory PortalVO Theory Portal Registry, storage service, hpc, query languages, visualisation, data miningRegistry, storage service, hpc, query languages, visualisation, data mining Melbourne-led (at present)Melbourne-led (at present) The Australian Astronomy Grid will be developed to handle data storage and access needs.

6. IVOA and the International Context More than 15 active national VO programs;More than 15 active national VO programs; Multi-million $$ investments in UK, USA and EuropeMulti-million $$ investments in UK, USA and Europe Loose but collegial collaborationLoose but collegial collaboration Responsible for international standardsResponsible for international standards Active meeting program (we have no funds to participate)Active meeting program (we have no funds to participate) The Australian Astronomy Grid will be developed to handle data storage and access needs.

7. Australian data storage and access Australian astronomy data holdings presently exceed ~40 TB in size, and are growing rapidly.Australian astronomy data holdings presently exceed ~40 TB in size, and are growing rapidly. Typical high-end workstations can store only ~100 GB or so.Typical high-end workstations can store only ~100 GB or so. Providing access to the data – raw and processed – requires a distributed, high- bandwidth network of data servers.Providing access to the data – raw and processed – requires a distributed, high- bandwidth network of data servers. The Australian Virtual Observatory project is developing the Australian Astronomy Grid to handle future demand.The Australian Virtual Observatory project is developing the Australian Astronomy Grid to handle future demand. The Australian Astronomy Grid will be developed to handle data storage and access needs.

8. The Australian Astronomy Grid 2004

9. The HI Parkes All Sky Survey Parkes 64m radio telescope in NSW.Parkes 64m radio telescope in NSW. Hyperfine transition of atomic Hydrogen, =21cm.Hyperfine transition of atomic Hydrogen, =21cm. 280 days over 4 years; 40 observers; 1000GB raw data.280 days over 4 years; 40 observers; 1000GB raw data. 400 image “cubes” searched by computer for significant signals.400 image “cubes” searched by computer for significant signals. The Parkes telescope has surveyed the entire southern sky for emission from Hydrogen.

10. The Two Micron All Sky Survey 4M images4M images 470M point sources470M point sources 1.6M extended sources1.6M extended sources ~500 parameters per source!~500 parameters per source! 25 TB of data!25 TB of data! All-sky map of 1.6 million 2MASS extended sources. Another example is 2MASS which has catalogued nearly half a billion objects in the sky. Jarrett et al., 2000 Less than 10% of the catalogue fits in memory on a typical workstation

11. Other major surveys... Sloan Digital Sky Survey (SDSS)Sloan Digital Sky Survey (SDSS) –position and brightness of 100M objects –distance to more than 100K quasars –15 Terabytes of data! Radial Velocity Experiment (RAVE)Radial Velocity Experiment (RAVE) –50M stars: velocities, metallicities, and abundance ratios –10 TB of data! Faint Images of the Radio Sky (FIRST)Faint Images of the Radio Sky (FIRST) –811,000 sources with radio continuum flux densities at 20cm wavelength Dozens of major, terabyte-scale survey projects are underway or planned.

12. Theoretical Astronomy Theory provides models of the phenomena discovered by observations.Theory provides models of the phenomena discovered by observations. Theory makes predictions of what will be seen by future facilities.Theory makes predictions of what will be seen by future facilities. Many theories are non- analytic, and sophisticated numerical simulations are run on supercomputers to produce realisations of synthetic universes.Many theories are non- analytic, and sophisticated numerical simulations are run on supercomputers to produce realisations of synthetic universes. Simulations can produce realisations of synthetic universes from fundamental physics.

13. Linking theory to observations Simulations are not expected to produce our particular Universe.Simulations are not expected to produce our particular Universe. Instead, they generate systems which can be compared statistically to our Universe.Instead, they generate systems which can be compared statistically to our Universe. Realisations of a good model should be statistically indistinguishable from the observed Universe.Realisations of a good model should be statistically indistinguishable from the observed Universe. Useful statistical comparisons demand high quality data and large numbers of objects independent of how you bin the data.Useful statistical comparisons demand high quality data and large numbers of objects independent of how you bin the data. Deeper, faster and more sophisticated surveys are called for...Deeper, faster and more sophisticated surveys are called for... Bigger and better simulations demand super surveys for statistical comprehension.

14. 4. MWA: Mileura Widefield Array Low frequency radio domain (110-240 MHz) largely unexploredLow frequency radio domain (110-240 MHz) largely unexplored Not easy: ionosphere, FM band, etcNot easy: ionosphere, FM band, etc BUT: aim to detect the first sourcesBUT: aim to detect the first sources and map Epoch of Reionisation One of 3 international experimentsOne of 3 international experiments (strongest project ) (strongest project ) New US/Australian low frequency array

15. Remote WA, for first light in 2007Remote WA, for first light in 2007 6TB fibre link to Geraldton6TB fibre link to Geraldton Storage: 100’s TBsStorage: 100’s TBs CPU: 50TflopsCPU: 50Tflops Melbourne, in collaboration with MIT, ATNF, Harvard and othersMelbourne, in collaboration with MIT, ATNF, Harvard and others Industry partnersIndustry partners New low frequency array will use innovative data-handling algorithms

16. MWA: Basic Approach ‘Desert Australia’ is probably the best site in the world for low frequency astronomy

17. MWA: Mileura Widefield Array Construction in Melbourne Haystack Observatory Mitcham Multi-skilled theorist

18. MWA: attennaes New design for antennas

19. MWA: Early Deployment 3 tiles to be commissioned by Christmas (Melbourne @Molonglo)3 tiles to be commissioned by Christmas (Melbourne @Molonglo) SKA site to be selectedSKA site to be selected NSF grant submitted and awarded!NSF grant submitted and awarded! Receiver and correlator design to be determinedReceiver and correlator design to be determined Negotiations with industry partnersNegotiations with industry partners Collaboration and timescales ‘solidifying’ this week

20. MWA: Signal Processing 500 tiles (x16 dipoles)500 tiles (x16 dipoles) 125,000 baselines, 4 polarization products125,000 baselines, 4 polarization products FPGA based hardwareFPGA based hardware Receiver: analog and mixed-signal front end; digital back endReceiver: analog and mixed-signal front end; digital back end Data stream: ~2 billion visibilities/0.5 sec Data stream: ~2 billion visibilities/0.5 sec Technical requirements and directions

21. Melbourne Astrophysics Requirements High Bandwidth Communications (Access Grid): scientific collaboration & conferencesHigh Bandwidth Communications (Access Grid): scientific collaboration & conferences Functional Grid: storage & processingFunctional Grid: storage & processing Institutional Commitment: planning, resourcing, r&d,Institutional Commitment: planning, resourcing, r&d, Institutional Leadership: NEWInstitutional Leadership: NEW