1. PSY 369: Psycholinguistics Language Comprehension: Perception of language

2. Different signals Visual word recognitionSpeech Perception Some parallel input Orthography Letters Clear delineation Difficult to learn Serial input Phonetics/Phonology Acoustic features Usually no delineation “Easy” to learn Where are you going

3. Different signals Visual word recognitionSpeech Perception Some parallel input Orthography Letters Clear delineation Difficult to learn Serial input Phonetics/Phonology Acoustic features Usually no delineation “Easy” to learn Where are you going

4. Speech perception Brief review of last time Acoustic features of speech: formants, transitions, bursts, VOT ‘Hard problems’ in speech perception Linearity, invariance, co-articulation, trading relations Link between acoustics and articulation Categorical perception, motor theory of speech perception The focus was on bottom-up processing, today let’s look at some top-down effects

5. Top-down effects on Speech Perception Speech sounds are not typically used in isolation Sentence context effects Phoneme restoration effect Segmentation effects

6. Phoneme restoration effect Participants listened to a sentence which contained a word from which a phoneme was deleted and replaced with another noise (e.g., a cough) The state governors met with their respective legi*latures convening in the capital city. * /s/ deleted and replaced with a cough Click here for a demo and additional information Warren (1970)

7. Phoneme restoration effect Typical results: Participants heard the word normally, despite the missing phoneme Usually failed to identify which phoneme was missing Interpretation We can use top-down knowledge to “fill in” the missing information Warren (1970)

8. Phoneme restoration effect What if the missing phoneme was ambiguous? The *eel was on the axle. Results: Participants heard the contextually appropriate word normally, despite the missing phoneme The *eel was on the shoe. The *eel was on the orange. The *eel was on the table. Warren & Warren (1970)

9. Phoneme restoration effect Possible loci of phoneme restoration effects Perceptual loci of effect: Lexical or sentential context influences the way in which the word is initially perceived. Post-perceptual loci of effect: Lexical or sentential context influences decisions about the nature of the missing phoneme information.

10. Beyond the segment The scientist made a new discovery last year. Hear: NUDIST Shillcock (1990): Participants hear a sentence, make a lexical decision to a word that pops up on computer screen (cross-modal priming)

11. Cross-modal priming The scientist made a novel discovery last year. Hear: NUDIST Shillcock (1990): Participants hear a sentence, make a lexical decision to a word that pops up on computer screen (cross-modal priming)

12. Cross-modal priming The scientist made a novel discovery last year. Hear: The scientist made a new discovery last year.faster Shillcock (1990): Participants hear a sentence, make a lexical decision to a word that pops up on computer screen (cross-modal priming) NUDIST

13. Cross-modal priming The scientist made a novel discovery last year. Hear: NUDIST gets primed by segmentation error faster Although no conscious report of hearing “nudist” The scientist made a new discovery last year. Shillcock (1990): Participants hear a sentence, make a lexical decision to a word that pops up on computer screen (cross-modal priming) NUDIST

14. Speech recognition Models of spoken word recognition Cohort Model Items may be eliminated from the cohort if inconsistent with the context but only after the initial cohort has been activated by bottom up information. Activation levels among cohorts varies and this is the mechanism which allows for frequency effects and lexical similarity. TRACE Model Unlike the cohort model this model allows for top down effects at all levels of processing Thus this model can account for effects like the phonemic restoration effect at the earliest levels of processing (in the cohort model this could only occur after processing was complete).

15. Speech recognition Speech perception is an active, constructive process Listeners do not simply attend to spoken information Categorical perception Also attend to visual information McGurk effect Also use lexical (word) and contextual knowledge to generate hypotheses about the likely form of the spoken information. Phoneme restoration, segmentation studies

16. Different signals Visual word recognitionSpeech Perception Some parallel input Orthography Letters Clear delineation Difficult to learn Serial input Phonetics/Phonology Acoustic features Usually no delineation “Easy” to learn Where are you going

17. Why so much research using visual language? We do use it Easy to use in research The parts Letters Words Eye movements Visual perception of language

18. Same object category (‘G’) may have different shapes, sizes, and orientations G G G G G G G G G G G G G G G G G G G G G G Perhaps the brain is able to represent these objects in a way that is “translationally invariant” and “size invariant”. Invariance a problem in vision too?

19. Letter Recognition How do we recognize a group of lines and curves as letters? Two common explanations: Template matching Feature detection Okay, I’m going to show you some stimuli really fast and you need to tell me what they are

27. Template matching Store in brain a copy of what every possible input will look like. Match observed object to the proper image in memory

28. Perceptual Representation Memory Representations Template matching

29. Problems with Template matching Costly: Massive numbers of templates are required (remember all those G’s?).. Predicts no transfer to novel views of the same object Normalization before matching - ”mentally cleaning it up” before matching to templates

30. Problems with Template matching Costly: Massive numbers of templates are required (remember all those G’s?).. Predicts no transfer to novel views of the same object Normalization before matching - ”mentally cleaning it up” before matching to templates Objects are often obstructed/occluded E FROG

31. Prolblems with Template matching E FROG Objects are often disamiguated by context Costly: Massive numbers of templates are required (remember all those E’s?).. Normalization before matching - ”mentally cleaning it up” before matching to templates Predicts no transfer to novel views of the same object Objects are often obstructed/occluded

32. Read aloud the following word CT

33. TE

34. So what is the middle letter? TE CT Clearly, top-down influences. However it is unclear how this works with template matching

35. Feature detection Analysis-by-synthesis 1. Letter broken down to its constituent parts 2. List of parts compared to patterns in memory 3. Best matching pattern chosen

36. A fixed set of elementary properties are analyzed Independently and in parallel across visual field. Possible examples Line Orientations: Different Sizes: Curvature: +45deg. -10deg. Free line endings: Colors: Feature detection

37. Perceptual Representation 3 Horizontal lines 1 Vertical line 4 Right angles Memory Representation 3 Horizontal lines 1 Vertical line 4 Right angles E F 2 Horizontal lines 1 Vertical line 3 Right angles A simple theory of Feature detection

38. Evidence for Features: The visual search task is straightforward, you are given some target to look for, and asked to simply decide, as quickly as possible, whether the target is present or absent in a set of objects. For example, let’s try a few searches to give you a feel for this. Search 1 - Is there an O present in the following displays?

39. Is an O present? T T O T T T

40. T T T T TTT T T T O TTT T TT TT T TT T T TT T T T TT T TTT T TT Is an O present?

41. Q Q Q O Q Q Q Q Is an O present?

42. Q QQ Q Q QQQ QQQ Q QQQ Q O Q Q QQ Q Q QQ Q Q Q QQQ Q QQQQ Q Is an O present?

43. T T O T T T T T T T TTT T T T O TTT T TT TT T TT T T TT T T T TT T TTT T TT Q Q Q O Q Q Q Q Q QQ Q Q QQQ QQQ Q QQQ Q O Q Q QQ Q Q QQ Q Q Q QQQ Q QQQQ Q

44. A theory of Feature detection Selfridge’s Pandemonium system, 1959Pandemonium

45. Another theory of Feature detection

46. Interactive Activation Model (AIM) McClelland and Rumelhart, (1981) Nodes: (visual) feature (positional) letter word detectors Inhibitory and excitatory connections between them. Previous models posed a bottom-up flow of information (from features to letters to words). IAM also poses a top- down flows of information

47. Inhibitory connections within levels If the first letter of a word is “a”, it isn’t “b” or “c” or … Inhibitory and excitatory connections between levels (bottom-up and top-down) If the first letter is “a” the word could be “apple” or “ant” or …., but not “book” or “church” or…… If there is growing evidence that the word is “apple” that evidence confirms that the first letter is “a”, and not “b”….. Interactive Activation Model (AIM)

48. + Until the participant hits some start key The Word-Superiority Effect (Reicher, 1969)

49. COURSE Presented briefly … say 25 ms The Word-Superiority Effect (Reicher, 1969)

50. U &&&&& A Mask presented with alternatives above and below the target letter … participants must pick one as the letter they believe was presented in that position. The Word-Superiority Effect (Reicher, 1969)

51. + E E & T + PLANE E &&&&& T + KLANE E &&&&& T Letter only Say 60% Letter in Nonword Say 65% Letter in Word Say 80% Why is identification better when a letter is presented in a word?

52. IAM & the word superiority effect We are processing at the word and letter levels simultaneously Letters in words benefit from bottom-up and top- down activation But letters alone receive only bottom-up activation.

53. Other Relevant Findings? Bias towards “well-formed” stimuli Misidentify words with uncommon spelling patterns BOUT as BOAT misidentify non-words (e.g., SALID) as words that are like it (SALAD). Difficulty identifying non-words with irregular spelling patterns (e.g., ITPR) more than those with regular spelling patterns (e.g., PIRT).

54. Sublexical units bigger than phonemes and graphemes? onsets and rimes onset: initial consonant or consonant cluster in a word or syllable rime: following vowel and consonants if words broken at onset-rime boundary, resulting letter clusters more easily recognized as belonging together than if broken at other points example: FL OST ANK TR vs. FLA ST NK TRO Sublexical units

55. Adding a bigram level By adding a frequency-sensitive bigram level, we can account for the findings of well-formedness along with the others.

56. Summing up Evidence supports the view that our word recognition processes are based on a feature-detector system Biased to perceive common or recently occurring features

57. Studying Word Identification Generally people ask: what makes word identification easy or difficult? The assumption: Time spent identifying a word can be a measure of difficulty Measures of identification time are usually indirect

58. Some Identification Time Measures Measure how long people take to say a string of letters is (or is not) a word (lexical decision) Measure how long people take to categorise a word (“apple” is a fruit) Measure how long people take to start saying a word (naming or pronunciation time) Measure how long people actually spend looking at a word when READING Line by line reading Word by word reading using eye movement monitoring techniques

59. Line-by-line A banker is a fellow

60. Line-by-line who lends you his umbrella

61. Line-by-line when the sun is shining

62. Line-by-line but wants it back

63. Line-by-line the minute it begins to rain.

64. Line-by-line Problem: Overall reading time for entire sentence or phrase need for more “on-line” measurements Timing on a smaller scope See effects at level of word

65. Word-by-word RSVP (rapid serial visual presentation)

66. Word-by-word A

67. lie

68. Word-by-word can

69. Word-by-word travel

70. Word-by-word halfway

71. Word-by-word around

72. Word-by-word the

73. Word-by-word world

74. Word-by-word while

75. Word-by-word the

76. Word-by-word truth

77. Word-by-word is

78. Word-by-word putting

79. Word-by-word on

80. Word-by-word its

81. Word-by-word shoes.

82. Word-by-word Moving window

83. Word-by-word I xxxx xxxxx xxx xx xxxxxxxxx xxxxxxxxx xxxx xx xxxxxxxxx.

84. Word-by-word x have xxxx xxx xx xxxxxxxxx xxxxxxxxx xxxx xx xxxxxxxxx.

85. Word-by-word x xxxx never xxx xx xxxxxxxxx xxxxxxxxx xxxx xx xxxxxxxxx.

86. Word-by-word x xxxx xxxxx let xx xxxxxxxxx xxxxxxxxx xxxx xx xxxxxxxxx.

87. Word-by-word x xxxx xxxxx xxx my xxxxxxxxx xxxxxxxxx xxxx xx xxxxxxxxx.

88. Word-by-word x xxxx xxxxx xxx xx schooling xxxxxxxxx xxxx xx xxxxxxxxx.

89. Word-by-word x xxxx xxxxx xxx xx xxxxxxxxx interfere xxxx xx xxxxxxxxx.

90. Word-by-word x xxxx xxxxx xxx xx xxxxxxxxx xxxxxxxxx with xx xxxxxxxxx.

91. Word-by-word x xxxx xxxxx xxx xx xxxxxxxxx xxxxxxxxx xxxx my xxxxxxxxx.

92. Word-by-word x xxxx xxxxx xxx xx xxxxxxxxx xxxxxxxxx xxxx xx education.

93. Word-by-word A couple of methods RSVP (rapid serial visual presentation) Moving window Better, more “on-line” But, these measures are also a little bit unnatural (especially RSVP) e.g., Don’t allow regressions (looking back)

94. Eye-movements The kite fell on the dog Eyemovement studies:

95. Eye-movements The kite fell on the dog Eyemovement studies:

96. Eye-movements The kite fell on the dog Eyemovement studies:

97. The Human Eye At its center is the fovea, a pit that is most sensitive to light and is responsible for our sharp central vision. The central retina is cone- dominated and the peripheral retina is rod- dominated.

98. Retinal Sampling

100. Eye Movements Within the visual field, eye movements serve two major functions Saccades to Fixations – Position target objects of interest on the fovea Tracking – Keep fixated objects on the fovea despite movements of the object or head

101. Fixations The eye is (almost) still – perceptions are gathered during fixations The most important of eye “movements” 90% of the time the eye is fixated duration: 150ms - 600ms

102. Saccades Saccades are used to move the fovea to the next object/region of interest. Connect fixations Duration 10ms - 120ms Very fast (up to 700 degrees/second) No visual perception during saccades Vision is suppressed Evidence that some cognitive processing may also be suppressed during eye-movements (Irwin, 1998)

103. Saccades Move to here

104. Saccade w/o suppression

105. Saccades Move to here

106. Saccades

107. Saccades are used to move the fovea to the next object/region of interest. Connect fixations Duration 10ms - 120ms Very fast (up to 700 degrees/second) No visual perception during saccades Vision is suppressed Ballistic movements (pre-programmed) About 150,000 saccades per day

108. Smooth Pursuit Smooth movement of the eyes for visually tracking a moving object Cannot be performed in static scenes (fixation/saccade behavior instead)

109. Smooth Pursuit versus Saccades Saccades Jerky No correction Up to 700 degrees/sec Background is not blurred (saccadic suppression) Smooth pursuit Smooth and continuous Constantly corrected by visual feedback Up to 100 degrees/sec Background is blurred

110. Eye-movements in reading Clothes make the man. Naked people have little or no influence on society. Eye-movements in reading are saccadic rather than smooth

111. Eye-movements in reading Clothes make the man. Naked people have little or no influence on society. Eye-movements in reading are saccadic rather than smooth

112. Eye-movements in reading Clothes make the man. Naked people have little or no influence on society. Eye-movements in reading are saccadic rather than smooth

113. Eye-movements in reading Clothes make the man. Naked people have little or no influence on society. Eye-movements in reading are saccadic rather than smooth

114. Eye-movements in reading Clothes make the man. Naked people have little or no influence on society. Eye-movements in reading are saccadic rather than smooth

115. Eye-movements in reading Clothes make the man. Naked people have little or no influence on society. Eye-movements in reading are saccadic rather than smooth

116. Eye-movements in reading Clothes make the man. Naked people have little or no influence on society. Eye-movements in reading are saccadic rather than smooth

117. Eye-movements in reading Clothes make the man. Naked people have little or no influence on society. Eye-movements in reading are saccadic rather than smooth

118. Eye-movements in reading Limitations of the visual field 130 degrees vertically, 180 degrees horizontally (including peripheral vision Perceptual span for reading: 7-12 spaces Clothes make the man. Naked people have little or no influence on society.

119. Purkinje Eye Tracker Laser is aimed at the eye. Laser light is reflected by cornea and lens Pattern of reflected light is received by an array of light- sensitive elements. Very precise Also measures pupil accomodation No head movements Measuring Eye Movements

120. Video-Based Systems Infrared camera directed at eye Image processing hardware determines pupil position and size (and possibly corneal reflection) Good spatial precision (0.5 degrees) for head-mounted systems Good temporal resolution (up to 500 Hz) possible