1. Szapudi´s talk – False Detection Rate Simultaneous hypothesis testing - Setting the statistical significance Detections of Non-Gaussianity in CMB observations Brief review of detections/methods used to set the statistical significance. Application of FDR to individual statistical methods. Application of FDR to a combination of statistical methods?. What is the appropriate method for assessing the statistical significance of localized detections?.

2. CMB: Non-Gaussianity tests Non- Gaussianity tests Real space Spherical Harmonic space Wavelet space Blind tests – The alternative to the null hypothesis is not specified.

3. Non-Gaussianity tests: Real Space Courtesy of WMAP Science Team Based on the temperature fluctuation observed at each pixel i, ΔT(i) Tests (detections): Genus (Park 2003). N-point correlation function (Eriksen et al. 2003, Eriksen et al. 2004). Minkowski functionals and length of skeleton (Eriksen et al. 2004). Extrema (Larson & Wandelt 2004) 2-point correlation function of maxima and minima (Tojeiro et al. 2005, Larson & Wandelt 2005)

4. Non-Gaussianity Tests: Spherical Harmonic Space Based on the complex coefficients Tests (detections): Power spectrum distribution (Eriksen et al. 2003, Hansen et al. 2004). Correlations between adjacent multipoles (Prunet et al. 2004).

5. Non-Gaussianity Tests: Wavelet Space Based on wavelet coefficients calculated at each pixel i, at a given scale R, wv(i,R). Tests (detections): Kurtosis – Spherical Mexican Hat Wavelet (Vielva et al. 2003, Mukherjee & Wang 2004, McEwen et al. 2004, Liu & Zhang 2005, Cruz et al. 2006). Skewness – Real Morlet Wavelet (McEwen et al. 2004, Liu & Zhang 2005). Number and area and volume of spots – SMHW (Cruz et al. 2004, Cruz et al. 2006). Higher Criticism – SMHW (Cayon et al. 2005, Cruz et al. 2006).

6. Testing the null hypothesis through Monte Carlo simulations Simulating a CMB map: At each pixel (i – θ,ψ) 1) B l – Antenna C l - Cosmology 2)Add Noise - dispersion given by simulated experiment. Mask – Galaxy plus point sources (zeros to the pixel in the mask). Monopole and dipole removal.

7. Testing the null hypothesis through Monte Carlo simulations Map ΔT(i) ΔT(i) Test statistic: Over whole map or Different combinations of pixels (N-point corr.) alm Test Statistic: Several multipoles or Several combinations of multipoles (bispectrum) Wavelet coeff. wv(i,R) at scale R Test Statistic: Over whole map at each scale and/or above/below threshold (spots)

8. Statistical Hypothesis Testing Testing the null hypothesis with a single configuration (confidence level). Simultaneous hypotheses testing : - Χ 2 Example (McEwen et al.2004) SMHW Kurtosis- scale 9 above 99% confidence level. SMHW Skewness and kurtosis – 24 statistics. Detection at the 99.9% significance level. - Hypothesis Test Larson & Wandelt, astro-ph/0505046 (maximum risk of false detection at the same level as the claimed significance) - Conservative significance level Based on marginal distribution of all statistics. Ex. Above 95.3% significance level.

9. Statistical Hypothesis Testing (FDR) Simultaneous Hypotheses testing – False Detection Rate - Control of the fraction of false discoveries (detections) over the total number of discoveries. - No assumption on the Gaussianity of the error distribution. - Correlations between statistics can be taken into account (?). αx100 % of discoveries may be mistakes.

10. False Discovery Rate – Ex. Wavelet Space No correlations, α=0.05, detection scales=9,8,7 Correlations, α=0.1, detection scales=9,8,7 16 tests / statistical test No correlations, α=0.1, detection scales=9,8 Correlations, α=0.2, detection scales=9 Figs from Cruz et al. 2006

11. False Discovery Rate – Ex. Wavelet Space Statistical TestNo Correlations C N =1 α |scales ( detect.) Correlations C N =3.38 (16 scales) α|scales (detect.) Kurtosisα=0.059,8,7α=0.19,8,7 Maxα=0.059,10α=0.29,10 Higher Criticismα=0.19,8α=0.29 Area above 3σα=0.058,9α=0.28,9 Area above 3.5σα=0.059,8α=0.059,8 Area above 4σα=0.059,8,10α=0.29,8,10 Area above 4.5σα=0.19,8α=0.29 Acknowledgement – M. Cruz

12. What is the appropriate method for setting the statistical significance? Only considering detections based on SMHW: 112 statistical tests. - Different statistical methods. - Several scales. Is it possible to assess the statistical significance of all detections all together? The pixels behind some of the detections are localized.