Data-Intensive Statistical Challenges in Astrophysics Alex Szalay The Johns Hopkins University Collaborators: T. Budavari, C-W Yip (JHU), M. Mahoney (Stanford), I. Csabai, L. Dobos (Hungary)

The Age of Surveys CMB Surveys (pixels) 1990 COBE Boomerang 10, CBI 50, WMAP 1 Million 2008 Planck10 Million Galaxy Redshift Surveys (obj) 1986 CfA LCRS dF SDSS BOSS LAMOST Angular Galaxy Surveys (obj) 1970 Lick 1M 1990 APM 2M 2005 SDSS200M 2011 PS1 1000M 2020 LSST 30000M Time Domain QUEST SDSS Extension survey Dark Energy Camera Pan-STARRS LSST… Petabytes/year …

Sloan Digital Sky Survey “The Cosmic Genome Project” Two surveys in one –Photometric survey in 5 bands –Spectroscopic redshift survey Data is public –2.5 Terapixels of images => 5 Tpx –10 TB of raw data => 120TB processed –0.5 TB catalogs => 35TB in the end Started in 1992, finished in 2008 Extra data volume enabled by –Moore’s Law –Kryder’s Law

Analysis of Galaxy Spectra Sparse signal in large dimensions Much noise, and very rare events 4Kx1M SVD problem, perfect for randomized algorithms Motivated our work on robust incremental PCA

Galaxy Properties from Galaxy Spectra Continuum EmissionsSpectral Lines

Galaxy Diversity from PCA [Average Spectrum] [Stellar Continuum] [Finer Continuum Features + Age] [Age] Balmer series hydrogen lines [Metallicity] Mg b, Na D, Ca II Triplet 1st 2nd 3rd 4th 5th PC

Streaming PCA Initialization –Eigensystem of a small, random subset –Truncate at p largest eigenvalues Incremental updates –Mean and the low-rank A matrix –SVD of A yields new eigensystem Randomized algorithm! T. Budavari, D. Mishin 2011

Robust PCA PCA minimizes σ RMS of the residuals r = y – Py –Quadratic formula:  r 2 extremely sensitive to outliers We optimize a robust M-scale σ 2 (Maronna 2005) –Implicitly given by Fits in with the iterative method! Outliers can be processed separately

Eigenvalues in Streaming PCA Classic Robust 9

Examples with SDSS Spectra Built on top of the Incremental Robust PCA Principal Component Pursuit (I. Csabai et al) Importance sampling (C-W Yip et al)

Principal component pursuit Low rank approximation of data matrix: X Standard PCA: –works well if the noise distribution is Gaussian –outliers can cause bias Principal component pursuit –“sparse” spiky noise/outliers: try to minimize the number of outliers while keeping the rank low –NP-hard problem The L1 trick: –numerically feasible convex problem (Augmented Lagrange Multiplier) * E. Candes, et al. “Robust Principal Component Analysis”. preprint, Abdelkefi et al. ACM CoNEXT Workshop (traffic anomaly detection)

Slowly varying continuum + absorption lines Highly variable “sparse” emission lines This is the simple version of PCP: the position of the lines are known but there are many of them, automatic detection can be useful spiky noise can bias standard PCA DATA: Streaming robust PCA implementation for galaxy spectrum catalog (L. Dobos et al.) SDSS 1M galaxy spectra Morphological subclasses Robust averages + first few PCA directions Testing on Galaxy Spectra

PCA PCA reconstruction Residual

Principal component pursuit Low rank Sparse Residual λ=0.6/sqrt(n), ε=0.03

Not Every Data Direction is Equal A = C X Galaxy ID Wavelength Galaxy ID Selected WavelengthsWavelength Procedure: 1. Perform SVD of A = U  V T 2. Pick number of eigenvectors = K 3. Calculate Leverage Score =  i ||V T ij || 2 / K Selected Wavelengths Mahoney and Drineas 2009

Wavelength Sampling Probability k = 2 c = 7 k = 4 c = 16 k = 6 c = 25 k = 8 c = 29

Ranking Astronomical Line Indices (Yip et al in prep.)(Worthey et al. 94; Trager et al. 98) Subspace Analysis of Spectra Cutouts: -Othogonality -Divergence -Commonality

Identify Informative Regions “NewMethod” 1.Pick the λ with largest P λ 2.Define its region of influence using  λ P λ convergence. Mask λ’ s from future selection. 3.Go back to Step 1, or quit. “MahoneySecond” 1.Over-select λ’s from the targeted number. 2.Merge selected λ if two pixels lie within a certain distance 3.Quit.

Identifying New Line Indices, Objectively (Yip et al in prep.)

New Spectral Regions (MahoneySecond; k = 5; Overselecting 10 X; Combining if < 30 Å)

NewMethod vs MahoneySecond NM M2

Gunawan & Neswan 2000)

Angle between Subspaces JHULick

 λ P λ JHU Lick

Line Indices for Galaxy Parameter Estimations

Importance Sampling and Galaxies Lick indices are ad hoc The new indices are objective –Recover atomic lines –Recover molecular bands –Recover Lick indices –Informative regions are orthogonal to each other, in contrast to Lick Future –Emission line indices –More accurate parameter estimation of galaxies

Summary Non-Incremental changes on the way Science is moving increasingly from hypothesis- driven to data-driven discoveries Need randomized, incremental algorithms –Best result in 1 min, 1 hour, 1 day, 1 week New computational tools and strategies … not just statistics, not just computer science, not just astronomy, not just genomics… Astronomy has always been data-driven…. now becoming more generally accepted