1. Psychophysical methods Lavanya Sharan January 26th, 2011

2. Announcements Check class website: http://graphics.cs.cmu.edu/courses/P2P / http://graphics.cs.cmu.edu/courses/P2P / Pick presentations slots for Feb 9th and Feb 14th. One CVG and one P slot per person (not on same day) Email slot preference to sharan@cs.cmu.edu sharan@cs.cmu.edu Have to meet instructors 2x before presentation!

3. Overview Basics of designing a perceptual experiment Types of experiments Analysis of experiments IRB

4. Basics Independent variable This is what varies Can have several levels (i.e. values) An experiment may have many indep. vars. Dependent variable This is what is measured

5. An example How well do people recognize objects (as compared to a favorite CV algorithm)? Images from the Visual Object Classes Challenge 2010 Independent variable? Dependent variable? Class participation!

6. An example How well do people recognize objects? Images from the Visual Object Classes Challenge 2010 Independent variable Object class (20 levels) Dependent variable Accuracy, Reaction time (RT)

7. Understanding your variables Discrete or continuous? (e.g., object class is discrete) Measurement scale - Nominal: there is a difference (e.g., car vs. bus) - Ordinal: make sense (e.g., rank in VOC2010 challenge) - Interval: size and sign of difference makes sense (e.g., Fahrenheit scale) - Ratio: all the above and having a 0 makes sense (e.g., percentage correct)

8. Confounds Ideally, stimuli only differ in the levels of the independent variable. Images from the Visual Object Classes Challenge 2010 Confounding variables Color, shape, size of object, contrast of images, background of objects, location in image, lighting, etc.

9. Confounds Ideally, stimuli only differ in the levels of the independent variable. Images from the Visual Object Classes Challenge 2010 In practice, impossible to eliminate confounding variables. A well-designed study minimizes as many confounds as possible.

10. Dealing with confounds Control for confounding factors by: Removing confounding variables in stimuli Balancing the presence of confounding variables by adding control conditions* Separating participants into groups and comparing their responses i.e., control group Condition = any manipulation of independent variables.

11. Controls Images from the Visual Object Classes Challenge 2010 Hypothesis: Size of object in image is enough to distinguish the 20 object classes. Confounding (or control) variable = Visual size of objects Option 1: Resize and re-crop all the images, never allow size to be an issue.

12. Hypothesis: Size of object in image is enough to distinguish the 20 object classes. Confounding (or control) variable = Visual size of objects Option II: Create second set of images with resized objects, test if size is an issue. Question: Control condition (same subjects) or control group (different subjects)? Controls Images from the Visual Object Classes Challenge 2010

13. Experimental Designs Between-subjects: Different subjects in different conditions. Minimize learning effects, more subjects Within-subjects: Same subjects in all conditions. More power, fewer subjects Mixed design: Some conditions within- subjects and some between-subjects.

14. Balanced designs Even after adding control conditions and/or groups, we have to worry about order effects Seeing bikes before buses Seeing dogs before buses

15. Balanced designs Even after adding control conditions and/or groups, we have to worry about order effects Seeing bikes before buses Seeing dogs before buses Does it matter? Priming can occur.

16. Balanced designs Even after adding control conditions and/or groups, we have to worry about order effects Seeing bikes before buses Seeing dogs before buses Solution: Randomization Counterbalancing

17. Balanced designs Use randomization when lots of subjects or each subject sees the same conditions many times. Use counterbalancing otherwise. E.g., 3 conditions (Dog, Bike, Bus) Slide content: Aude Oliva, MIT OCW Order 1 Order 2 Order 3 S 1,4,7,10... S 2,5,8,11... S 3,6,9,12... DogBikeBus DogBike BusDog

18. Overview Basics of designing a perceptual experiment Types of experiments Analysis of experiments IRB

19. Thresholds Absolute: At which visual size can you detect that there is a bike? Difference: How much detail is needed to recreate the original shape?

20. Measuring Thresholds Method of Constant Stimuli: Present stimuli at several levels, some below and some above threshold, measure proportion detected. Method of Limits: Start at one level, if detected decrease else increase level. Converge to an estimate of threshold. Method of Adjustment: Let subjects adjust the level until they can just detect the stimulus.

21. Method of Constant Stimuli + Easy to conduct and analyze - Need to have an idea of threshold beforehand - Takes longer, more trials Psychometric function Image source: http://www.ncbi.nlm.nih.gov/books/NBK11513/http://www.ncbi.nlm.nih.gov/books/NBK11513/ Slide content: Lynee Werner, University of Washington

22. Method of Limits + Fewer trials + Don’t need to estimate threshold beforehand - Noise prone Staircase procedure Image source: http://www.ncbi.nlm.nih.gov/books/NBK11513/http://www.ncbi.nlm.nih.gov/books/NBK11513/ Slide content: Lynee Werner, University of Washington

23. Method of Adjustment + Intuitive for subject - Can be unreliable Image source: http://psychology.wikia.com/wiki/File:Method_of_Adjustment.pnghttp://psychology.wikia.com/wiki/File:Method_of_Adjustment.png Slide content: Lynee Werner, University of Washington

24. Overview Basics of designing a perceptual experiment Types of experiments Analysis of experiments IRB

25. Back to detection What about response bias? Image source: http://www.walker.co.uk/walkerdam/getimage.aspx?id=9781406314403-1&size=webusehttp://www.walker.co.uk/walkerdam/getimage.aspx?id=9781406314403-1&size=webuse Slide content: Lynee Werner, University of Washington Subjects might be more prone to saying yes (or no).

26. Signal detection theory Slide content: Lynee Werner, University of Washington Subject said yesSubject said no Stimulus present HitMiss Stimulus absent False Alarm Correct Rejection ‣ d’ is computed from Hits and False Alarms ‣ Measure of sensitivity ‣ Bias-free

27. 2-AFC design Image source: http://www.ratracetrap.com/wp-content/uploads/2009/09/Fork-in-the-road-300x237.pnghttp://www.ratracetrap.com/wp-content/uploads/2009/09/Fork-in-the-road-300x237.png Slide content: Lynee Werner, University of Washington ‣ Instead of asking for yes/no response on each trial, force subject to choose from two options ‣ Another way of removing response bias (interval bias can remain)

28. How do you know subjects aren’t guessing? (Bad subjects.) Image source: http://academic.kellogg.edu/mckayg/buad112/web/pres/coin%20flip.jpghttp://academic.kellogg.edu/mckayg/buad112/web/pres/coin%20flip.jpg Need to calculate chance performance for every task Subjects might have been sleeping, distracted, perverse, or simply unable to do your task because it is humanly impossible.

29. Is performance better than chance? Statistics to the rescue. VOC2010 Challenge example: 20 object classes implies chance = 1/20 = 5% Between-subjects design, Group ‘No Size Control’ (Performance = 80%, N=10) Group ‘Size Control’ (Performance = 75%, N=10) ✓ Use independent one-sample t-tests to compare performance in both groups to chance ✓ Choose significance threshold (usually p = 0.05) ✓ Divide significance threshold by number of tests (Bonferroni correction), here 0.05/2 = 0.025 ✓ Quote the t-statistic and p-value if less than corrected threshold as showing statistical significance.

30. Is performance better in one condition than another? Again, statistics to the rescue. VOC2010 Challenge example: 20 object classes implies chance = 1/20 = 5% Between-subjects design, Group ‘No Size Control’ (Performance = 80%, N=10) Group ‘Size Control’ (Performance = 75%, N=10) ✓ Use independent two-sample t-test to compare performance in two groups to each other ✓ Choose significance threshold (usually p = 0.05, if combining with two previous tests then 0.05/3 = 0.0167) ✓ Quote the t-statistic and p-value if less than threshold as showing statistical significance.

31. Choosing the right tests is VERY important Otherwise you don’t know if you are measuring noise or a real effect. For example, for a within-subjects design, you would use a paired samples t-test. Get hold of a good statistics book and package and understand precisely what you are doing. I recommend using SPSS and material from: http://www.statisticshell.com

32. Overview Basics of designing a perceptual experiment Types of experiments Analysis of experiments IRB

33. IRB and other legalities You are dealing with human subjects, this means you need to be very ethical and careful. CMU has a Regulatory Compliance Office: http://www.cmu.edu/osp/regulatory-compliance/human- subjects.html You are encouraged to take the online human subjects training: https://www.citiprogram.org/Default.asp? Image source: http://www.icts.uiowa.edu/drupal/sites/all/themes/icts/custom/images/news/warning.jpghttp://www.icts.uiowa.edu/drupal/sites/all/themes/icts/custom/images/news/warning.jpg

34. IRB and other legalities For this class: If you will not use any perceptual data you gather ever again, you don’t need to write an IRB protocol and get approval. If there is even a tiniest chance, the perceptual data you gather will be show up anywhere, you NEED to write an IRB protocol and get it approved in time. Come talk to me about this. Image source: http://www.icts.uiowa.edu/drupal/sites/all/themes/icts/custom/images/news/warning.jpghttp://www.icts.uiowa.edu/drupal/sites/all/themes/icts/custom/images/news/warning.jpg

35. Summary Figure out your independent and dependent variables. Think of all possible confounds. Control for confounds, balance your designs. Get IRB to run study. Analyze data using standard statistical procedures. Don’t do all this a day before your deadline!