1. Speech Perception in Infant and Adult Brains
Colin Phillips Cognitive Neuroscience of Language Laboratory Department of Linguistics University of Maryland

2. Overview of Talks 1. The Unification Problem
2. Building Syntactic Relations
3. Abstraction: Sounds to Symbols
4. Linguistics and Learning
どの生徒に…

3. University of Delaware Evniki Edgar Bowen Hui
with help from ...
University of Maryland Shani Abada Sachiko Aoshima Daniel Garcia-Pedrosa Ana Gouvea Nina Kazanina Moti Lieberman Leticia Pablos David Poeppel Beth Rabbin Silke Urban Carol Whitney
University of Delaware Evniki Edgar Bowen Hui
Baris Kabak Tom Pellathy Dave Schneider Kaia Wong Alec Marantz, MIT Elron Yellin, MIT
National Science Foundation James S. McDonnell Foundation Human Frontiers Science Program Japan Science & Technology Program Kanazawa Institute of Technology

4. Sensory Maps
Internal representations of the outside world. Cellular neuroscience has discovered a great deal in this area.

5. Encoding of Symbols: Abstraction
But most areas of linguistics (phonology, morphology, syntax, semantics) are concerned with symbolic, abstract representations, ...which do not involve internal representations of dimensions of the outside world. …hence, the notion of sensory maps does not get us very far into language

6. Outline Categories & Abstraction Speech Perception in Infancy
Electrophysiology: Mismatch Paradigm
Speech Discrimination in Adult Brains
Speech Categorization in Adult Brains
Conclusion

7. A Category

8. Another Category 3
III

9. Categories for Computation
Membership in a category like “bird” is a graded property
Membership in a category like “three” is an all-or-nothing property
“Three” can be part of a symbolic computation
“Bird” cannot

10. Abstraction Benefits of abstraction Costs of abstraction
representational economy
representational freedom
allow combinatorial operations
Costs of abstraction
distant from experience - impedes learning
distant from experience - impedes recognition

11. Phonetic vs. Phonological Categories
Phonetic category membership is graded
Phonological category membership is an all-or-nothing property: all members are equal
Phonological categories are the basis of storage of lexical forms
Phonological categories participate in a wide variety of combinatorial computations

12. Timing - Voicing

13. Voice Onset Time (VOT)
60 msec

14. Perceiving VOT
‘Categorical Perception’

15. Discrimination
Same/Different

16. Discrimination
Same/Different
0ms ms

17. Discrimination
Same/Different
0ms ms
Same/Different

18. Discrimination
Same/Different
0ms ms
Same/Different
0ms ms

19. Discrimination Same/Different 0ms 60ms Same/Different 0ms 10ms

20. Discrimination Same/Different 0ms 60ms Same/Different 0ms 10ms

21. Discrimination Same/Different 0ms 60ms Same/Different
Why is this pair difficult?
0ms ms
Same/Different
40ms 40ms

22. Discrimination Same/Different 0ms 60ms Same/Different
Why is this pair difficult?
0ms ms
(i) Acoustically similar?
(ii) Same Category?
Same/Different
40ms 40ms

23. Discrimination A More Systematic Test Same/Different 0ms 60ms
Why is this pair difficult?
0ms ms
(i) Acoustically similar?
(ii) Same Category?
Same/Different
40ms 40ms

24. Discrimination A More Systematic Test Same/Different 0ms 60ms 0ms 20ms

25. Discrimination A More Systematic Test D D D T T T Same/Different
0ms ms
0ms
20ms D
20ms
40ms T
Same/Different
0ms ms T T
40ms
60ms
Same/Different
Within-Category Discrimination is Hard
40ms 40ms

26. Cross-language Differences

27. Cross-language Differences

28. Cross-Language Differences
English vs. Japanese R-L

29. Cross-Language Differences
English vs. Hindi
alveolar [d]
retroflex [D] ?

30. Outline Categories & Abstraction Speech Perception in Infancy
Electrophysiology: Mismatch Paradigm
Speech Discrimination in Adult Brains
Speech Categorization in Adult Brains
Conclusion

31. Development of Speech Perception
Unusually well described in past 30 years
Learning theories exist, and can be tested…
Jakobson’s suggestion: children add feature contrasts to their phonological inventory during development
Roman Jakobson, Kindersprache, Aphasie und allgemeine Lautgesetze, 1941

32. Developmental Differentiation
Universal Phonetics
Native Lg. Phonetics
Native Lg. Phonology
0 months
6 months
12 months
18 months

33. #1 - Infant Categorical Perception
Eimas, Siqueland, Jusczyk & Vigorito, 1971

34. Discrimination A More Systematic Test D D D T T T Same/Different
0ms ms
0ms
20ms D
20ms
40ms T
Same/Different
0ms ms T T
40ms
60ms
Same/Different
Within-Category Discrimination is Hard
40ms 40ms

35. English VOT Perception
To Test 2-month olds
Not so easy!
High Amplitude Sucking
Eimas et al. 1971

36. General Infant Abilities
Infants’ show Categorical Perception of speech sounds - at 2 months and earlier
Discriminate a wide range of speech contrasts (voicing, place, manner, etc.)
Discriminate Non-Native speech contrasts e.g., Japanese babies discriminate r-l e.g., Canadian babies discriminate d-D

37. Universal Listeners
Infants may be able to discriminate all speech contrasts from the languages of the world!

38. How can they do this? Innate speech-processing capacity?
General properties of auditory system?

39. What About Non-Humans?
Chinchillas show categorical perception of voicing contrasts!

40. #2 - Becoming a Native Listener
Werker & Tees, 1984

41. When does Change Occur? About 10 months Janet Werker
U. of British Columbia
Conditioned Headturn Procedure

42. When does Change Occur?
Hindi and Salish contrasts tested on English kids
Janet Werker
U. of British Columbia
Conditioned Headturn Procedure

43. What has Werker found?
Is this the beginning of efficient memory representations (phonological categories)?
Are the infants learning words?
Or something else?

44. 6-12 Months: What Changes?

45. Structure Changing
Patricia Kuhl U. of Washington

46. #3 - What, no minimal pairs?
Stager & Werker, 1997

47. A Learning Theory…
How do we find out the contrastive phonemes of a language?
Minimal Pairs

48. Word Learning
Stager & Werker ‘bih’ vs. ‘dih’ and ‘lif’ vs. ‘neem’

49. Word learning results
Exp 2 vs 4

50. Why Yearlings Fail on Minimal Pairs
They fail specifically when the task requires word-learning
They do know the sounds
But they fail to use the detail needed for minimal pairs to store words in memory
!!??

51. One-Year Olds Again
One-year olds know the surface sound patterns of the language
One-year olds do not yet know which sounds are used contrastively in the language…
…and which sounds simply reflect allophonic variation
One-year olds need to learn contrasts

52. Maybe not so bad after all...
Children learn the feature contrasts of their language
Children may learn gradually, adding features over the course of development
Phonetic knowledge does not entail phonological knowledge
Roman Jakobson,

53. Summary of Development
Memory
Surface 10 months
Memory 18 months
Constructed
Lexical
Surface
Auditory
Phonetic
Articulatory
Innate

54. Outline Categories & Abstraction Speech Perception in Infancy
Electrophysiology: Mismatch Paradigm
Speech Discrimination in Adult Brains
Speech Categorization in Adult Brains
Conclusion

55. KIT-Maryland MEG System (160 SQUID detectors)
Brain Magnetic Fields (MEG)
KIT-Maryland MEG System (160 SQUID detectors)

56. Brain Magnetic Fields (MEG)
SQUID detectors measure brain magnetic fields around 100 billion times weaker than earth’s steady magnetic field.

57. pickup coil & SQUID assembly
160 SQUID whole-head array

58. It’s safe…

59. Origin of the signal - noninvasive measurement - direct measurement.
skull
CSF
tissue
MEG
EEG B
orientation
of magnetic
field
recording
surface
scalp
current
flow
- noninvasive measurement
- direct measurement.

60. How small is the signal? Biomagnetism Intensity of magnetic signal(T)
Earth field
EYE (retina)
Steady activity
Evoked activity
LUNGS
Magnetic contaminants
LIVER
Iron stores
FETUS
Cardiogram
LIMBS
Steady ionic current
BRAIN (neurons)
Spontaneous activity
Evoked by sensory stimulation
SPINAL COLUMN (neurons)
HEART
Cardiogram (muscle)
Timing signals (His Purkinje system)
GI TRACK
Stimulus response
Magnetic contaminations
MUSCLE
Under tension
Intensity of magnetic signal(T)
Urban noise
Contamination at lung
Heart QRS
Biomagnetism
Fetal heart
Muscle
Spontaneous signal
(a-wave)
Signal from retina
Evoked signal
Intrinsic noise of SQUID

61. Electroencephalography (EEG/ERP)

62. Event-Related Potentials (ERPs)
John is laughing.
s1 s2 s3

63. Mismatch Response
X X X X X Y X X X X Y X X X X X X Y X X X Y X X X...

64. Mismatch Response
X X X X X Y X X X X Y X X X X X X Y X X X Y X X X...

65. Mismatch Response X X X X X Y X X X X Y X X X X X X Y X X X Y X X X...
Latency: msec.
Localization: Supratemporal auditory cortex
Many-to-one ratio between standards and deviants

66. Localization of Mismatch Response
(Phillips, Pellathy, Marantz et al., 2000)

67. Basic MMN elicitation
© Risto Näätänen

68. Basic MMN elicitation
MMN
P300
Näätänen et al. 1978

69. MMN Amplitude Variation
Sams et al. 1985

70. How does MMN latency, amplitude vary with frequency difference
How does MMN latency, amplitude vary with frequency difference? 1000Hz tone std.
Tiitinen et al. 1994

71. Different Dimensions of Sounds
Length
Amplitude
Pitch
…you name it …
Amplitude of mismatch response can be used as a measure of perceptual distance

72. Outline Categories & Abstraction Speech Perception in Infancy
Electrophysiology: Mismatch Paradigm
Speech Discrimination in Adult Brains
Speech Categorization in Adult Brains
Conclusion

73. ‘Vowel Space’

74. Näätänen et al. (1997)
e e/ö ö õ o

75. Place of Articulation
[bæ]
Formant Transition Cues
[dæ]

76. Place of Articulation
Sharma et al. 1993

77. When does Change Occur?
Hindi and Salish contrasts tested on English kids
Janet Werker
U. of British Columbia

78. 6-12 Months: What Changes?

79. Structure Changing
Patricia Kuhl U. of Washington

80. Place of Articulation Non-native continuum b -- d -- D
3 contrasts Native b -- d Non-native d -- D Non-phonetic b1 -- b5
Conflicting results!

81. Place of Articulation Non-native continuum b -- d -- D
3 contrasts Native b -- d Non-native d -- D Non-phonetic b1 -- b5
Conflicting results!
Dehaene-Lambertz 1997

82. Place of Articulation Non-native continuum b -- d -- D
3 contrasts Native b -- d Non-native d -- D Non-phonetic b1 -- b5
Conflicting results!
Rivera-Gaxiola et al. 2000

83. Place of Articulation Non-native continuum b -- d -- D
3 contrasts Native b -- d Non-native d -- D Non-phonetic b1 -- b5
Conflicting results!
Tsui et al. 2000

84. Interim Conclusion MMN/MMF is a sensitive measure of discrimination
In some - but not all - cases, MMN amplitude tracks native language discrimination patterns
When MMN fails to show native language category effects…
could reflect that MMN accesses only low-level acoustic representations
could reflect that MMN accesses multiple levels of representation, but response is dominated by acoustic representation
These studies all implicate phonetic categories

85. Outline Categories & Abstraction Speech Perception in Infancy
Electrophysiology: Mismatch Paradigm
Speech Discrimination in Adult Brains
Speech Categorization in Adult Brains
Conclusion

86. Objective
Isolate phonological categories, not phonetic categories

87. A Category

88. Another Category 3
III

89. Auditory Cortex Accesses Phonological Categories: An MEG Mismatch Study
Colin Phillips, Tom Pellathy, Alec Marantz, Elron Yellin, et al. Journal of Cognitive Neuroscience, 2000

90. Voice Onset Time (VOT)
60 msec

91. Categorical Perception

92. Design Fixed Design - Discrimination 20ms 40ms 60ms 0ms 8ms 16ms 24ms
Grouped Design - Categorization

96. Phonological Features in Auditory Cortex
Colin Phillips Tom Pellathy Alec Marantz

97. Sound Groupings
(Phillips, Pellathy & Marantz 2000)

98. Phonological Features
Phonological Natural Classes exist because...
Phonemes are composed of features - the smallest building blocks of language
Phonemes that share a feature form a natural class
Effect of Feature-based organization observed in…
Language development
Language disorders
Historical change
Synchronic processes
Roman Jakobson,

99. Voicing in English

100. Japanese - Rendaku s  z ts  z t  d k  g f  b [-voice]  [+voice]
take + sao  takezao
cito + tsuma  citozuma
hon + tana  hondana
yo: + karasi  yo:garasi
asa + furo  asaburo
Second member of compound word
s  z
ts  z
t  d
k  g
f  b
[-voice]  [+voice]

101. Sound Groupings in the Brain
pæ, tæ, tæ, kæ, dæ, pæ, kæ, tæ, pæ, kæ, bæ, tæ...
(Phillips, Pellathy & Marantz 2000)

102. Sound Groupings in the Brain
pæ, tæ, tæ, kæ, dæ, pæ, kæ, tæ, pæ, kæ, bæ, tæ...
(Phillips, Pellathy & Marantz 2000)

103. Feature Mismatch: Stimuli
(Phillips, Pellathy & Marantz 2000)

104. Feature Mismatch
Design
(Phillips, Pellathy & Marantz 2000)

105. Sound Groupings in the Brain
pæ tæ tæ kæ dæ pæ kæ tæ pæ kæ bæ tæ ...
(Phillips, Pellathy & Marantz 2000)

106. Sound Groupings in the Brain
pæ tæ tæ kæ dæ pæ kæ tæ pæ kæ bæ tæ ...
– – – – [+voi]
(Phillips, Pellathy & Marantz 2000)

107. Sound Groupings in the Brain
pæ tæ tæ kæ dæ pæ kæ tæ pæ kæ bæ tæ ...
– – – – [+voi] – – – – – [+voi] – …
(Phillips, Pellathy & Marantz 2000)

108. Sound Groupings in the Brain
pæ tæ tæ kæ dæ pæ kæ tæ pæ kæ bæ tæ ...
– – – – [+voi] – – – – – [+voi] – …
Voiceless phonemes are in many-to-one ratio with [+voice] phonemes
No other many-to-one ratio in this sequence
Notice, there is no many-to-one ratio in this sequence unless we appeal to features, here [+voice]
So, it means that if there is a MM between devi and standards, it’s due to the feature [+voice]. In other words, if your brain is surprised to head B’s, d’s and g’s, it’s because these sounds were rare and because all of them share this feature [+voice] that was absent in the frequent/standard sounds.
(Phillips, Pellathy & Marantz 2000)

109. Feature Mismatch
(Phillips, Pellathy & Marantz 2000)

110. Feature Mismatch Left Hemisphere Right Hemisphere
(Phillips, Pellathy & Marantz 2000)

111. Control Experiment - ‘Acoustic Condition’
Identical acoustical variability
No phonological many-to-one ratio
(Phillips, Pellathy & Marantz 2000)

112. Feature Mismatch
(Phillips, Pellathy & Marantz 2000)

113. Hemispheric Contrast in MMF
Studies of acoustic and phonetic contrasts consistently report bilateral mismatch responses Paavilainen, Alho, Reinikainen et al. 1991; Näätänen & Alho, 1995; Levänen, Ahonen, Hari et al. 1996; Alho, Winkler, Escera et al. 1998; Ackermann, Lutzenberger & Hertrich, 1999; Opitz, Mecklinger, von Cramon et al. 1999, etc., etc.
Striking difference in our finding of a left-hemisphere only mismatch response elicited by phonological feature contrast
Our studies probe a more abstract level of phonological representation

114. Colin Phillips Shani Abada Daniel Garcia-Pedrosa Nina Kazanina
EEG Measures of Discrimination and Categorization of Speech Sound Contrasts
Colin Phillips Shani Abada Daniel Garcia-Pedrosa Nina Kazanina

115. Design Fixed Design - Discrimination 20ms 40ms 60ms 0ms 8ms 16ms 24ms
Grouped Design - Categorization

116. Voice Onset Time (VOT) Mismatch Negativity (MMN): An ERP study
MMN: acoustic or perceptual phenomenon?
Does an across-category distinction (20ms VOT /da/, 40ms VOT /ta/ evoke a greater MMN than a within-category distinction (40ms VOT /ta/, 60ms VOT /ta/)?
Sharma & Dorman (1999): MMN only across categories; MMN represents perceptual, not physical, difference between stimuli; Double N100 for long VOT
‘Oddball’ paradigm - 7:1 ratio of standards to deviants
Fixed condition
standard VOT = 20ms, deviant VOT = 40 ms (across), or standard VOT = 40ms, deviant VOT = 60ms (within)
Grouped condition
no specific standard VOT,
but 7/8 fall into either /da/ or /ta/
20st,40dv st,60dv Dst,Tdv Tst,Ddv

117. Discrimination and Categorization of Vowels and Tones
Daniel Garcia-Pedrosa Colin Phillips

118. Two Concerns Are the category effects an artifact:
it is very hard to discriminate different members of the same category on a voicing scale
subjects are forming ad hoc groupings of sounds during the experiment, and are not using their phonological representations?

119. Discrimination A More Systematic Test D D D T T T Same/Different
0ms ms
0ms
20ms D
20ms
40ms T
Same/Different
0ms ms T T
40ms
60ms
Same/Different
Within-Category Discrimination is Hard
40ms 40ms

120. Vowels
Vowels show categorical perception effects in identification tasks
…but vowels show much better discriminability of within-category pairs

121. Method: Materials Tones: 290Hz, 300Hz, 310 Hz…470Hz Vowels
First formant (F1) varies along the same Hz continuum
F0, F2, voicing onset, etc. all remain constant

122. Method: Procedure Subject’s category boundary determined by pretest
Grouped mismatch paradigm
Standard stimulus (7/8) = 4 exemplars from one category
Deviant stimulus (1/8) = 4 exemplars from other category
MMN response therefore = deviance from a category, not from a single stimulus
Tones and vowels presented in separate blocks

123. Results: Vowels

124. Results: Vowels

125. Results: Tones

126. Results: Tones

127. Preliminary conclusions
MMN appears about 150ms post-stimulus in vowel but not in tone condition
Higher amplitude N100 for deviants in both conditions. Is this evidence for categorization of tones or just the result of habituation?
Acoustic differences may be responsible for greater N100, while categorization elicits the MMN

128. Phonemic vs. Allophonic Contrasts
Nina Kazanina Colin Phillips in progress

129. Cross-Language Differences
Focus on meaning-relevant sound contrasts Russian d t Korean d t

130. Cross-Language Differences
Focus on meaning-relevant sound contrasts Russian d t Korean d t

131. Cross-Language Differences
Focus on meaning-relevant sound contrasts Russian d t Korean d t …ada ada ada ada ada ada ata ada ada ada ata…

132. EXECTIVE SUITE

133. Phonology - Syllables Japanese versus French
Pairs like “egma” and “eguma”
Difference is possible in French, but not in Japanese

134. Behavioral Results Japanese have difficulty hearing the difference
Dupoux et al. 1999

135. ERP Results Sequences: egma, egma, egma, egma, eguma
French have 3 mismatch responses
Early, middle, late
Japanese only have late response
Dehaene-Lambertz et al. 2000

136. ERP Results - 2
Early response
Dehaene-Lambertz et al. 2000

137. ERP Results - 3
Middle response
Dehaene-Lambertz et al. 2000

138. ERP Results - 4
Late response
Dehaene-Lambertz et al. 2000

139. Cross-language Differences
Thai speakers:
Thai *words*: [da] [ta] DIFFERENT
English *words*: [daz] [taz] SAME
Imsri (2001)

140. Varying Pronunciations
Voiceless stops /p, t, k/
Aspirated at start of syllable; unaspirated after [s]
pit
spit
bit
tack
stack
dack

141. Outline Categories & Abstraction Speech Perception in Infancy
Electrophysiology: Mismatch Paradigm
Speech Discrimination in Adult Brains
Speech Categorization in Adult Brains
Conclusion

142. Conclusion
Sound representations involve (multiple degrees of) abstraction
Different levels of representation develop independently from 0-18 months of age
Although much is known about course of development, many open questions about how change proceeds
Possibility of a connection between adult electrophysiology and infant developmental findings