1. 1 Statistics and Image Quality Evaluation III Oleh Tretiak Medical Imaging Systems Fall, 2002

2. 2 This Lecture Mean square error as quality measure –Shortcomings How to do ROC by hand Visual ROC experiment This and other files for todays lecture at –http://www.ece.drexel.edu/courses/ECE-S684/Notes/IQ3/

3. 3 Analytic Image Fidelity Mean square difference between images u(m, n) - perfect image, u’(m, n) - defective image

4. 4 ‘White Noise’ Pattern

5. 5 Noise Patterns White, low frequency, and high frequency noises. All have same standard deviation

6. 6 Effect of noise on image quaity: UL ~ original 8-bit image; UR ~ white noise; LL ~ low pass noise; LR ~ high pass noise. Noise standard deviation is equal to 8, so that PTP signal to rms noise ratio is 32.

7. 7 Effect of noise on image quaity: UL ~ original 8-bit image; UR ~ white noise; LL ~ low pass noise; LR ~ high pass noise. Noise standard deviation is equal to 32, so that PTP signal to rms noise ratio is 8.

8. 8 Conclusions Same SNR does not produce the same image degradation when noises are different ‘Low frequency’ noise is most visible ‘High frequency’ noise is most damaging

9. 9 “The mean square error criterion is not without limitations, especially when used as a global measure of image fidelity. The prime justification for its common use is the relative ease with which it can be handled mathematically…” Anil Jain, in Fundamentals of Digital Image Processing.

10. 10 Method Rating on a 1 to 5 scale (5 is best) Rating performed by 21 subject Statistics: –Average, maximum, minimum, standard deviation, standard error for Case A, Case B –Difference per viewer between Case A and Case B, and above statistics on difference

11. 11 Two Kinds of Errors In a decision task with two alternatives, there are two kinds of errors Suppose the alternatives are ‘healthy’ and ‘sick’ –Type I error: say healthy if sick –Type II error: say sick if healthy

12. 12 X - observation, t - threshold  = Pr[X > t | H 0 ] (Type I error)  = Pr[X < t | H 1 ] (Type II error) Choosing t, we can trade off between the two types of errors

13. 13 ROC Terminology ROC — receiver operating characteristic H0 — friend, negative; H1 — enemy, positive Pr[X > t | H0] = probability of false alarm = probability of false positive = P FP =  Pr[X > t | H 1 ] = probability of detection = probability of true positive = P TP = 

14. 14 The ROC The ROC shows the tradeoff between P FP and P TP as the threshold is varied

15. 15 Binormal Model Negative: Normal, mean = 0, st. dev. = 1 Negative: Normal, mean = a, st. dev. = b

16. 16

17. 17 ROC from Experimental Data Strarting data: 9 samples of N(0, 1) pseudorandom variable, simulating negative population. Middle: data sorted, with cumulative distribution. Right: plot of middle.

18. 18 ROC from Experimental Data 2 Strarting data: 7 samples of N(2, 1.5) pseudorandom variable, simulating positive population. Middle: data sorted, with cumulative distribution. Right: plot of middle.

19. 19 ROC from Experimental Data 3 Formula in FPF column =IF(N4="N",O4,P3) Formula in TPF column =IF(N4="P",O4,Q3) Merged sorted data and ‘Empirical ROC’ plot.

20. 20 Comparison of Finite and Infinite Sample Empirial ROC 9 negative and 7 positive cases Theoretical ROC What would have been obtained from a very large sample

21. 21 Metz University of Chicago ROC project: http://www- radiology.uchicago.edu/krl/toppage11.htm Software for estimating A z, also sample st. dev. And confidence intervals. Versatile

22. 22 Ordinal Dominance Theory Donald Bamber, Area above the Ordinal Dominance Graph and the Area below the Receiver Operating Characteristic Graph, J. of Math. Psych. 12: 387-415 (1975). Sample A z is an unbiased, efficient, asymptotically normal estimate of population A z Bamber gives a formula for variance of A z Worst-case estimate (overbound) of variance

23. 23 Visual ROC Target Target is 9x16 pixels, Gaussian noise plus 4x4 dark rectangle on one side Protocol: Answer L, R if you think target is on left or right. In this example, target is on left

24. 24 Experiment Goal Test human (your) visual system processing power Target is easy to see in absence of noise This sort of stimulus can be analyzed, and theoretical best ROC found Use paper form for your answers, enter your results on computer & mail to me. Alternate method: Download file ROC-test- form.xls, enter data on form, and send to me by e- mail.

25. 25 Group 1

26. 26 Group 2