Alex Szalay Department of Physics and Astronomy The Johns Hopkins University and the SDSS Project The Sloan Digital Sky Survey

Alex Szalay, JHU A project run by the Astrophysical Research Consortium (ARC) Goal: To create a detailed multicolor map of the Northern Sky over 5 years, with a budget of approximately $80M Data Size: 40 TB raw, 1 TB processed Goal: To create a detailed multicolor map of the Northern Sky over 5 years, with a budget of approximately $80M Data Size: 40 TB raw, 1 TB processed The University of Chicago Princeton University The Johns Hopkins University The University of Washington Fermi National Accelerator Laboratory US Naval Observatory The Japanese Participation Group The Institute for Advanced Study Max Planck Inst, Heidelberg SLOAN Foundation, NSF, DOE, NASA The University of Chicago Princeton University The Johns Hopkins University The University of Washington Fermi National Accelerator Laboratory US Naval Observatory The Japanese Participation Group The Institute for Advanced Study Max Planck Inst, Heidelberg SLOAN Foundation, NSF, DOE, NASA The Sloan Digital Sky Survey

Alex Szalay, JHU Scientific Motivation Create the ultimate map of the Universe:  The Cosmic Genome Project! Study the distribution of galaxies:  What is the origin of fluctuations?  What is the topology of the distribution? Measure the global properties of the Universe:  How much dark matter is there? Local census of the galaxy population:  How did galaxies form? Find the most distant objects in the Universe:  What are the highest quasar redshifts?

Alex Szalay, JHU Cosmology Primer The spatial distribution of galaxies is correlated, due to small ripples in the early Universe P(k): P(k): power spectrum v = H o r Hubble’s law v = H o r Hubble’s law The Universe is expanding: the galaxies move away from us spectral lines are redshifted  = density/critical  = density/critical if  <1, expand forever The fate of the universe depends on the balance between gravity and the expansion velocity  d >  * Most of the mass in the Universe is dark matter, and it may be cold (CDM)

Alex Szalay, JHU The ‘Naught’ Problem What are the global parameters of the Universe? H 0 the Hubble constant55-75 km/s/Mpc  0 the density parameter  0 the cosmological constant Their values are still quite uncertain today... Goal: measure these parameters with an accuracy of a few percent What are the global parameters of the Universe? H 0 the Hubble constant55-75 km/s/Mpc  0 the density parameter  0 the cosmological constant Their values are still quite uncertain today... Goal: measure these parameters with an accuracy of a few percent High Precision Cosmology!

Alex Szalay, JHU The Cosmic Genome Project The SDSS will create the ultimate map of the Universe, with much more detail than any other measurement before Gregory and Thompson 1978 deLapparent, Geller and Huchra 1986 daCosta etal 1995 SDSS Collaboration 2002

Alex Szalay, JHU Area and Size of Redshift Surveys

Alex Szalay, JHU Clustering of Galaxies We will measure the spectrum of the density fluctuations to high precision even on very large scales The error in the amplitude of the fluctuation spectrum 1970x x ± ± ± ±0.02 The error in the amplitude of the fluctuation spectrum 1970x x ± ± ± ±0.02

Alex Szalay, JHU Finding the Most Distant Objects Intermediate and high redshift QSOs Multicolor selection function. Luminosity functions and spatial clustering. High redshift QSO’s (z>5). Intermediate and high redshift QSOs Multicolor selection function. Luminosity functions and spatial clustering. High redshift QSO’s (z>5).

Alex Szalay, JHU Special 2.5m telescope, located at Apache Point, NM 3 degree field of view. Zero distortion focal plane. Two surveys in one: Photometric survey in 5 bands. Spectroscopic redshift survey. Huge CCD Mosaic 30 CCDs 2K x 2K(imaging) 22 CCDs 2K x 400(astrometry) Two high resolution spectrographs 2 x 320 fibers, with 3 arcsec diameter. R=2000 resolution with 4096 pixels. Spectral coverage from 3900Å to 9200Å. Automated data reduction Over 100 man-years of development effort. (Fermilab + collaboration scientists) Very high data volume Expect over 40 TB of raw data. About 1 TB processed catalogs. Data made available to the public. Features of the SDSS

Alex Szalay, JHU Apache Point Observatory Located in New Mexico, near White Sands National Monument Located in New Mexico, near White Sands National Monument

Alex Szalay, JHU The Telescope Special 2.5m telescope 3 degree field of view Zero distortion focal plane Wind screen moved separately Special 2.5m telescope 3 degree field of view Zero distortion focal plane Wind screen moved separately

Alex Szalay, JHU Northern Galactic Cap 5 broad-band filters ( u', g', r', i', z’ ) limiting magnitudes (22.3, 23.3, 23.1, 22.3, 20.8) drift scan of 10,000 square degrees 55 sec exposure time 40 TB raw imaging data -> pipeline -> 100,000,000 galaxies 50,000,000 stars calibration to 2% at r'=19.8 only done in the best seeing (20 nights/yr) pixel size is 0.4 arcsec, astrometric precision is 60 milliarcsec Southern Galactic Cap multiple scans (> 30 times) of the same stripe Continuous data rate of 8 Mbytes/sec The Photometric Survey

Alex Szalay, JHU Survey Strategy Overlapping 2.5 degree wide stripes Avoiding the Galactic Plane (dust) Multiple exposures on the three Southern stripes Overlapping 2.5 degree wide stripes Avoiding the Galactic Plane (dust) Multiple exposures on the three Southern stripes

Alex Szalay, JHU Measure redshifts of objects  distance SDSS Redshift Survey: 1 million galaxies 100,000 quasars 100,000 stars Two high throughput spectrographs spectral range Å. 640 spectra simultaneously. R=2000 resolution. Automated reduction of spectra Very high sampling density and completeness Objects in other catalogs also targeted The Spectroscopic Survey

Alex Szalay, JHU The Mosaic Camera

Alex Szalay, JHU The Fiber Feed System Galaxy images are captured by optical fibers lined up on the spectrograph slit Manually plugged during the day into Al plugboards 640 fibers in each bundle The largest fiber system today Galaxy images are captured by optical fibers lined up on the spectrograph slit Manually plugged during the day into Al plugboards 640 fibers in each bundle The largest fiber system today

Alex Szalay, JHU First Light Images Telescope: First light May 9th 1998 Equatorial scans Telescope: First light May 9th 1998 Equatorial scans

Alex Szalay, JHU The First Stripes Camera: 5 color imaging of >100 square degrees Multiple scans across the same fields Photometric limits as expected Camera: 5 color imaging of >100 square degrees Multiple scans across the same fields Photometric limits as expected

Alex Szalay, JHU NGC 2068

Alex Szalay, JHU UGC 3214

Alex Szalay, JHU NGC 6070

Alex Szalay, JHU The First Quasars The four highest redshift quasars have been found in the first SDSS test data !

Alex Szalay, JHU Methane/T Dwarf Discovery of several new objects by SDSS & 2MASS SDSS T-dwarf (June 1999)

Alex Szalay, JHU Detection of Gravitational Lensing 28,000 foreground galaxies and 2,045,000 background galaxies in test data (McKay etal 1999)

Alex Szalay, JHU SDSS Data Flow

Alex Szalay, JHU Data Processing Pipelines

Alex Szalay, JHU Concept of the SDSS Archive Operational Archive (raw + processed data) Science Archive (products accessible to users) Other Archives

Alex Szalay, JHU Distributed Collaboration Japan Fermilab U.Washington U.Chicago USNO JHU VBNS NMSU Apache Point Observatory I. Advanced Study Princeton U. ESNET

Alex Szalay, JHU All raw data saved in a tape vault at Fermilab Object catalog400 GB parameters of >10 8 objects Redshift Catalog 1 GB parameters of 10 6 objects Atlas Images 1.5 TB 5 color cutouts of >10 8 objects Spectra 60 GB in a one-dimensional form Derived Catalogs 20 GB - clusters - QSO absorption lines 4x4 Pixel All-Sky Map 60 GB heavily compressed Object catalog400 GB parameters of >10 8 objects Redshift Catalog 1 GB parameters of 10 6 objects Atlas Images 1.5 TB 5 color cutouts of >10 8 objects Spectra 60 GB in a one-dimensional form Derived Catalogs 20 GB - clusters - QSO absorption lines 4x4 Pixel All-Sky Map 60 GB heavily compressed SDSS Data Products

Alex Szalay, JHU Geometric Indexing “Divide and Conquer” Partitioning 3  N  M3  N  M 3  N  M3  N  M Hierarchical Triangular Mesh Split as k-d tree Stored as r-tree of bounding boxes Using regular indexing techniques AttributesNumber Sky Position 3 Multiband FluxesN = 5+ Other M= 100+ AttributesNumber Sky Position 3 Multiband FluxesN = 5+ Other M= 100+

Alex Szalay, JHU User Interface Analysis Engine Master Objectivity RAID Slave Objectivity RAID Slave Objectivity RAID Slave Objectivity RAID Slave SX Engine Objectivity Federation Distributed Implementation

Alex Szalay, JHU Collaboration with Particle Physics Collaboration with the Analysis Data Grid: proposal to the NSF KDI program by JHU, Fermilab and Caltech (H. Newman, J. Bunn) + Objectivity, Intel and Microsoft (Jim Gray) Involves computer scientists, astronomers and particle physicists Accessing Large Distributed Archives in Astronomy and Particle Physics experiment with scalable server architectures, create middleware of intelligent query agents, apply to both particle physics and astrophysics data sets Status: 3 year proposal just funded

Alex Szalay, JHU The next generation of astronomical archives with Terabyte catalogs will dramatically change astronomy top-down design large sky coverage built on sound statistical plans uniform, homogeneous, well calibrated well controlled and documented systematics The technology to acquire, store and index the data is here we are riding Moore’s Law Data mining in such vast archives will be a challenge, but possibilities are quite unimaginable Integrating these archives into a single entity is a project for the whole community => Virtual National Observatory The Age of Mega-Surveys

Alex Szalay, JHU New Astronomy – Different! Systematic Data Exploration will have a central role in the New Astronomy Digital Archives of the Sky will be the main access to data Data “Avalanche” the flood of Terabytes of data is already happening, whether we like it or not! Transition to the new may be organized or chaotic

Alex Szalay, JHU NVO: The Challenges Size of the archived data 40,000 square degrees is 2 trillion pixels One band: 4 Terabytes Multi-wavelength: Terabytes Time dimension: few Petabytes The development of new archival methods new analysis tools new standards (metadata, interchange formats) Hardware/networking requirements Training the next generation!

Alex Szalay, JHU SummarySummary The SDSS project combines astronomy, physics, and computer science It promises to fundamentally change our view of the universe It will determine how the largest structures in the universe were formed Its ‘virtual universe’ can be explored by both scientists and the public It will serve as the standard astronomy reference for several decades Through its archive it will create a new paradigm in astronomy

Alex Szalay, JHU