1. Numerical Methods for Empirical Covariance Matrix Analysis Miriam Huntley SEAS, Harvard University May 15, 2013 18.338 Course Project

2. RMTReal World Data “When it comes to RMT in the real world, we know close to nothing.” -Prof. Alan Edelman, last week

3. Who Cares about Covariance Matrices? Basic assumption in many areas of data analysis: multivariate data You get, want to find can be a very bad estimator if finite Current standard using PCA (=SVD): distinguish from null model In RMT language: any eigenvalues which lie very far away from the distribution expected for a white Wishart matrix should be considered signal

4. Who Cares about Covariance Matrices? Data from: Alizadeh A, et al. (2000) Distinct types of diffuse large B-cell lymphoma identifed by gene expression profiling. Nature 403:503-511.

5. Why adventure beyond white Wishart? Null model not particularly sophisticated. Can we do better? Noise with structure  Example: Financial data  What if there is no right edge? Known, how many samples do we need before we recover it from empirical data?

6. Approach: General MP Law Data matrix where and define Y entries are iid (real or complex) and Let H p be the spectral distribution of and assume H p converges weakly to H ∞ Let F P be the spectral distribution of (empirical) and its Stieltjes transform Then: nxp pxp See: Silverstein, J. W. and Bai, Z. D. (1995). On the empirical distribution of eigenvalues of a class of large-dimensional random matrices. J. Multivariate Anal. 54, 2,175–192. El Karoui, N., Spectrum estimation for large dimensional covariance matrices using random matrix theory, Ann. Statist. 36 (2008), 2757–2790

7. Numerical Solutions of General MP Discretize in z Numerically Solve Single, True Covariance Matrix True Covariance Matrix Spectral Distribution Empirical Spectral Distribution Live Demos…

8. Inverse Solutions of General MP? Single, True Covariance Matrix True Covariance Matrix Spectral Distribution Empirical Spectral Distribution Discretize in z Numerically Solve

9. Toy Example: Block Covariance Matrix ? Warning: Don’t try this at home

10. Toy Example: Block Covariance Matrix

11. Thanks! This was fun. Colwell LJ, Qin Y, Manta A and Brenner MP (2013). Signal identification from Sample Covariance Matrices with Correlated Noise. Under Review El Karoui, N., Spectrum estimation for large dimensional covariance matrices using random matrix theory, Ann. Statist. 36 (2008), 2757–2790 MARCENKO, V. A. and PASTUR, L. A. (1967). Distribution of eigenvalues in certain sets of random matrices. Mat. Sb. (N.S.) 72 507–536. Silverstein, J. W. and Bai, Z. D. (1995). On the empirical distribution of eigenvalues of a class of large-dimensional random matrices. J. Multivariate Anal. 54, 2,175–192.