1. Facial identity and expression perception in the human visual system Human Vision and Eye Movement Laboratory Medicine (Neurology) Ophthalmology and visual sciences Psychology University of British Columbia NANOS 2011 Vancouver

2. DISCLOSURES: NO COMMERCIAL/PROPRIETARY INTERESTS FUNDING SUPPORT: NIMH 1R01 MH069898 CIHR MOP-77615, MOP-85004, MOP-102567, MOP-106511 NSERC RGPIN 355879-08 Michael Smith Foundation for Health Research CI-SSH-035(03-1) Canada Research Chairs 950-202111 Student support from MSFHR, CIHR, Alzheimer’s Foundation Canada Research Chairs

3. Bruce and Young (1986) cognitive model COMMON CONCEPT: perception of identity and expression emerge from divergent streams of face processing

4. EXPRESSION Encode facial dynamics Invariant for static structure allow generalization across people IDENTITY Encode static structure Invariant for dynamic changes allow stable recognition despite changes in expression and age COMMON CONCEPT: perception of identity and expression emerge from divergent streams of face processing Are expression and identity functionally and structurally independent? If so, this would imply that the human brain has evolved highly specific systems for extracting different structural properties of the same object.

5. 1. Behavioural studies 2. Anatomic fMRI studies 3. Patient studies

7. 1. BEHAVIOURAL INVESTIGATION use FACE-ADAPTATION to probe the functional relationship between the perception of facial expression and perception of facial identity

8. 1. Behavioural adaptation 90/1070/3050/5030/7010/90 fear anger a. Expression adaptation : are these invariant for the identity of the face? Method - PERCEPTUAL BIAS TECHNIQUE - create a morph continuum between two expressions: - stare at one ‘end-face’ for 5 sec.... then flash a 300ms ambiguous morph face.... and ask a question: “Did you see fear or anger?” “EXPRESSION AFTER-EFFECT”: Subjects are more likely to perceive the other expression ambiguous stimuli

9. 1. Behavioural adaptation 5-second Adapting stimulus 300 ms Probe stimulus (*three expression pairs: Angry/afraid; happy/sad; disgusted/surprised)

10. Same person Different person, Same gender Different person, Different gender Changing identity reduces but does not eliminate the expression after- effect. Expression representations may have two components: 1. Behavioural adaptation } identity-dependent component } identity-invariant component Aftereffect magnitude (%)

12. 90/1070/3050/5030/7010/90 “Bob” “Joe” b. Identity adaptation : is it invariant for the expression of the face? Method - create a morph continuum between two different people’s faces: - after adaptation to BOB, subjects are more likely to respond that a following ambiguous face is more like JOE. 1. Behavioural adaptation

13. Is the identity aftereffect partially dependent on expression? 1. Behavioural adaptation Same expression Different expression Answer: NO. In contrast to expression aftereffects, there is no expression-dependent component of identity aftereffects. Aftereffect magnitude

14. Would we find expression-dependency in the identity after-effect for more familiar faces? not familiar minimally familiar culturally familiar very familiar Answer: NO 1. Behavioural adaptation Aftereffect magnitude

15. Both contribute to expression adaptation identity adaptation There is an interesting asymmetric relationship between identity and expression! One possible (speculative) interpretation, influenced by network models…. 1. Behavioural adaptation

17. 1. STRUCTURAL INVESTIGATION Use fMRI-ADAPTATION to probe the structural relationship between perception of facial expression vs. identity Fusiform face area Superior temporal sulcus

18. Expression morphing Identity morphing 2-D Morph Matrix with “categorical boundaries”: EXPRESSION BOUNDARY 2. fMRI adaptation Ultimately, what mattered most in this experiment was not the stimulus but the perceptual decision - whether the subject said that they saw the face as similar or different. IDENTITY BOUNDARY

19. 2. Rapid event-related fMRI adaptation study identical establishes baseline of full adaptation In all of these, face 2 is 33% morph different from face 1. Subject sees pairs of faces: Do FFA and STS differ in adaptation effects induced by: physical changes in expression vs identity? perception of difference in expression vs identity? Subjects perform 2 runs with the same stimulus set: Expression task Identity task similar expression different expression identity is identical (held constant): similar identity different identity expression is identical (held constant):

20. same diff identity expression FUSIFORM FACE AREA (FFA) * FFA is sensitive to any structural change 2. fMRI adaptation Baseline = adaptation for repetition of identical face Release from adaptation (area ‘detects’ a difference): # FFA adaptation release is more when subject perceives a difference in EITHER identity or expression (as reported by subject)

21. SUPERIOR TEMPORAL SULCUS (STS) identity task expression task same diff identity expression same diff identity expression same STS is modulated by TASK - differences only seen when subject is actively processing expression 2. fMRI adaptation DURING EXPRESSION TASK: * STS is sensitive to any stimulus change, # STS adaptation release is more when subject perceives a difference in EITHER identity or expression

22. Conclusions of fMRI experiment FFA and STS activity is sensitive to physical properties of face, Correlate with perceptual report of subject, for both identity AND expression STS is recruited during expression task 2. fMRI adaptation Identity and expression specific signals not seen until later in cortical hierarchy: Identity – precuneus Expression – middle STS

24. 1. NEUROPSYCHOLOGY Differential impact of LESIONS on the perception of facial expression and perception of facial identity

25. 3. Lesion study Region of interest Subject prosopagnosic fMRI localization data in patients

26. expression identity Using 2-D morph matrix, create 4 oddity paradigm tests: Which face is the different expression? Identity varies Expression variesExpression fixed Identity fixed Which face is the different person? We vary the level of difficulty by choosing morph faces of varying distance in the matrix 3. Lesion study: are expression and identity differentially impaired?

27. 3. Lesion study Identity taskExpression task expression varying expression fixed identity varying identity fixed Identity but not expression impaired by lesions of: OFA/FFA Ant temporal lobe Medial fusiform Expression but not identity impaired by lesion of: STS Expression constancy of identity judgments impaired by STS lesion! Other dimension fixed Impaired expression only Impaired expression AND Impaired identity Impaired Identity only normal

29. Summary Behavioural adaptation results: identity representations are expression-invariant expression representations have identity-dependent and identity- invariant components fMRI-adaptation results: effects of perceptual decision for both expression and identity are REFLECTED in the activity of both FFA and pSTS (does NOT mean that they are encoded there). Lesion results: “fusiform - anterior temporal stream” lesions impair identity perception but NOT expression perception STS lesion impaired expression perception and impedes identity judgments when expression must be discounted.

30. 1. Behavioural adaptation Middle STS Posterior STS Insula Amygdala FFA Anterior temporal Precuneus? Other fusiform regions

31. Bicycle tour of Vancouver! WEDNESDAY meet at registration desk 1200h

32. Behavioural adaptation: Fox CJ, Barton JJS. What is adapted in face adaptation? The neural representations of expression in the human visual system. Brain Res 2007; 1127: 80- 9. Butler A, Oruc I, Fox CJ, Barton JJS. Factors contributing to the adaptation aftereffects of facial expression. Brain Res 2008; 1191: 116-26. Fox CJ, Oruc I, Barton JJS. It doesn’t matter how you feel. The facial identity aftereffect is invariant to changes in facial expression. J Vision 2008; 8(3): 11.1-13. fMRI-adaptation: Fox CJ, Moon S-Y, Iaria G, Barton JJS. The correlates of subjective perception of identity and expression in the face network: an fMRI adaptation study. Neuroimage 2009; 44: 569-80. Lesions: Fox CJ, Iaria G, Duchaine BC, Barton JJS. Behavioral and fMRI studies of identity and expression perception in acquired prosopagnosia. Vision Sciences Society, Naples 2008 PERSONNEL Andrea Butler Brad Duchaine Chris Fox Giuseppe Iaria So Young Moon Ipek Oruç

33. 3. Lesion study Desired test properties: equivalent perceptual difficulty, controls perform well but not at ceiling low variance CONTROL DATA: