Basics of fMRI Inference Douglas N. Greve

Overview Inference False Positives and False Negatives Problem of Multiple Comparisons Bonferroni Correction Cluster Correction (voxel-wise threshold) False Discovery Rate Selection Bias

3 Statistical Inference Group Population (All members) Hundreds? Thousands? Billions? Sample 18 Subjects Can your conclusions be extended to data you have not seen? –Subjects, Time Points, Groups, Scanners Or are your results the product of a chance occurrence that is unlikely to be repeated? Generalizability, Repeatability, Reproducibility, Predictability Uncertainty Beyond good Experimental Design

4 Truth Table Effect Is Not Present (Neg) Effect Is Present (Pos) Effect Is Not Present (Neg) True NegativeFalse Positive Effect Is Present (Pos) False NegativeTrue Positive Conclusion Reality

5 Error Rate Effect Is Not Present (Neg) Effect Is Present (Pos) Effect Is Not Present (Neg) True Negative TNR=1-  False Positive FPR  Effect Is Present (Pos) False Negative FNR =  True Positive TPR = 1-  (Power) Conclusion Reality False Positive Rate (FPR) – probability that you declare an effect to be present when there is no effect False Negative Rate (FNR) - probability that you declare no effect to be present when there is an effect

6 How Do You Draw Conclusions? Protocol: reduce all your data to one number (the “test statistic” T). If T is greater than some threshold (  ) then conclude that an effect is present (ie, a positive) Otherwise conclude that an effect is not present (ie, a negative). Every protocol has some FPR and some FNR, though it is not always easy to figure out!

7 Noise Causes Uncertainty Voxel 1 Voxel 2

8 GLM Inference T=8 T=1

9 Example Protocol Collect data Motion Correct Smooth by 5mm FWHM Extract Voxel 1 (throw away rest of data) Compute Mean and StdDev of ON time points Compute Mean and StdDev of OFF time points Compute test statistic T If T > 3.41, Conclude that the voxel is active Test Statistic (T) is the t-ratio Threshold (  ) is 3.41 What is the FPR (  ) and FNR (  ) for this protocol?

10 Example Protocol: False Positive Rate “NULL” Distribution Student’s t-Distribution p-value is area under curve to the right of T For T = 3.4, FPR = p =.01 For T=8, FPR = p = For T=1, FPR = p = 0.1 Assumptions: Gaussian noise Independent noise Homoskedastic (equal variances) Violation of assumptions change FPR Student’s t Distribution FPR=area under curve to the right of line (p-value)

11 Example Protocol: False Negative Rate Need to know what the effect size is Previous data Guess Power Analysis Grants require a power analysis!

12 Trade Off of Error Rates Inverse relationship between error rates As False Positives (  ) are reduced, the False Negatives (  ) increase Increase sample size decreases , does not affect  Which Error is more important? Depends.. Science? FPR=.05ish, TNR<0.2 Pre-operative surgery? FPR=.10 FPR=.01 FPR=10 -7

13 What conclusions to draw from this? Brain is activated? Visual Cortex? Auditory Cortex? False Positive Rate? Need a protocol!

14 Possible Protocol First Level Analysis Compute t-ratio for each voxel Compute p-value for each voxel If any brain voxel has p <.01, declare a positive Same as Test Statistic: T = max(T i ) Threshold:  =3.4 What is the False Positive Rate for this protocol?

15 What does p<.01 mean? p<.01 means one expects 1% of voxels will be active purely by chance Protocol gives a False Positive any time even a single voxel has p<.01 What is the probability that at least one voxel has p<.01? Rand(0,1) 100x100 10,000 vox p < vox p < vox p < vox

16 The “Problem of Multiple Comparisons”  Vox = voxel-wise threshold (p<  Vox )  FWE = Protocol False Positive Rate (FWE = Family-wise Error) N = Number of voxels (“Search Space”)  Vox  FWE  Vox =.10  Vox =.01  Vox =10 -7 N = 10,000

17 Bonferroni Correction Compute Voxel-wise threshold needed to achieve a desired Family-wise FPR. To achieve  FWE = 0.01 with N = 10,000 voxels Need  Vox = (10 -6 )

18 Search Space Set of voxels over which positives are searched Severity of correction increases with size of search space (regardless of method) Reduce Search Space Reduce the area to a ROI (eg, superior temp gyrus) Increase voxel size (cover same volume with fewer voxels) Spatial Smoothing

Full-Width/Half-max Spatially convolve image with Gaussian kernel. Kernel sums to 1 Full-Width/Half-max: FWHM =  /sqrt(log(256))  = standard deviation of the Gaussian 0 FWHM5 FWHM10 FWHM 2mm FWHM 10mm FWHM 5mm FWHM Full Max Half Max Smoothing causes irreversible loss of information (resolution)

Spatial Smoothing Smoothing causes irreversible loss of information (resolution), similar to increasing voxel size. 5mm Smoothing Increased Voxel Size 0mm10mm 4mm1mm8mm

Resel Pixel = picture element Voxel = volume element Resel = resolution element (depends on smoothing level) Resel = (FWHM) 3 for volumes Resel = (FWHM) 2 for surfaces If FWHM>Voxel Size, fewer Resels than Voxels. Correct based on the number of Resels instead of number of voxels (math is more complicated, need Random Field Theory) Bonferroni

Clusters True signal tends to be clustered False Positives tend to be randomly distributed in space Cluster – set of spatially contiguous voxels that are above a given threshold.  Vox =.10  Vox =.01  Vox =10 -7

Cluster-wise Correction Cluster – set of spatially contiguous voxels that are above a given threshold. Protocol Perform 1 st level analysis. Threshold volume at  Vox Find clusters. If Cluster Size > Threshold (  ), Declare a Positive Test Statistic: Cluster Size What is the FPR (  FWE ) for this protocol?

Random Field Theory  FWE = f(  Vox,N,FWHM,ClusterSize) p=.05

Smoothing increases size of random clusters FWHM 0FWHM 2FWHM 4FWHM 6 Z Z>2.3 p<.01

Cluster Images Sig Map p Vox <.001 Cluster Map p Cluster <.05 Some small clusters do not “survive”

Cluster Table RL Radiological Orientation Cluster MNI305 X Y Z Size (mm 3 ) Cluster p-value Atlas Location ~0 Right Lateral Occipital ~0 Left Lateral Occipital Left Superior Frontal Left Precentral ROI Atlas

Cluster Data Extraction Spatial average over cluster of each subject’s contrast Can correlate with other measures (age, test score, etc) Be careful of “Selection Bias” (“Voodoo Correlations”)

Cluster Correction Summary Cluster – set of supra-threshold voxels (size) Critical Size Threshold given by Random Field Theory Search Space Voxel-wise threshold (arbitrary) FWHM (smoothing level) Assumptions on each Loose small clusters (False Negatives)

False Discovery Rate (FDR) Given the voxel-wise threshold, know expected number of False Positives If there are more Positives than this, then some of them must be True Positives p < vox p < vox p < vox

False Discovery Rate (FDR) Number of False Positives = N*  Vox Total Number of Positives = Count from image  Vox = f(FDR,N,Data)

False Discovery Rate (FDR) FDR =.05 means that 5% of Positives are False Positives Which 5%, no one knows How to interpret? FDR =.05  Vox =.0070 FDR =.01  Vox =.0070

False Discovery Rate (FDR) FDR =.05 means that 5% of Positives are False Positives Which 5%, no one knows How to interpret? FDR =.05  Vox =.0070 FDR =.01  Vox =.0070 Would you change your opinion of this blob if 50 of the voxels were False Positives?

False Discovery Rate (FDR) FDR =.05 means that 5% of Positives are False Positives Which 5%, no one knows How to interpret? FDR =.05  Vox =.0070 FDR =.01  Vox =.0070 Would you change your opinion of this blob if 50 of the voxels were False Positives?

False Discovery Rate Summary False Discoveries FDR does not control FPR (False Positive Rate) Careful when interpreting Voxel-wise threshold is Data Dependent

Summary Can your conclusions be extended to data you have not seen? Truth Table: False Positives (  ) and False Negatives (  ) Protocol – describes how you will draw conclusions Problem of Multiple Comparisons (Family-wise Error) Search Space, Search Space reduction Larger voxels (less resolution) Smoothing (Resels) Bonferroni Correction Cluster Correction (voxel-wise threshold) False Discovery Rate Selection Bias

37