Doğu Erdener Auditory-visual speech perception in young children: language and age- specific factors @ 6 December 2007
Development of Speech Perception First 6 months phonetically-based, language-general Beyond 1 st year phonologically-based, language-specific Consonants 7-11 months (Best, 1995) Vowels 4-6 months (Kuhl et al., 1992) Tones 6-9 months (Burnham & Mattock, 2006) A working heuristic (Burnham et al., 2002) Acoustic / Phonetic Stage (0-6 months) Phonological Stage (6-12 months) Semantic Stage (12-24 months) Orthographic Stage (6-8 years)
Amodal (e.g.,Schwartz, et al., 1997) Phonetic (e.g.,Burnham & Dodd, 2004) Phonological / Postcategorical (e.g.,Massaro, 1998) none/some/all of the above? Pre-categorical Auditory-Visual Speech Perception (AVSP): Integration in early infancy (till ~6 months) is phonetically based beyond first year of life phonological Recall that auditory-only speech perception …. AVSP integration research: We need differential and ontogentic data! AVSP integration research: We need differential and ontogentic data!
AVSP: Cross-Language (Differential) Studies Cross-language phonemic differences e.g., Werker et al. (1992) Visual influence from /th/ in the McGurk stimulus [A]-/ba/ + [V]-/tha/ : anglophones > francophones more experienced francophones > less experienced francophones Amount of visual speech influence across langauges: American > Japanese (Sekiyama & Tohkura, 1993) Japanese > Mandarin (Sekiyama, 1997) Sekiyama studies Possible reasons: Cross-cultural studies: Less eye/face contact in some cultures + Japanese Ss tend to look at the eye area whereas American Ss on the mouth area (Yuki et al., 2007). Linguistic 1: Tonal (e.g. Cantonese) and pitch-accent (e.g., Japanese) information are not visually salient. Linguistic 2: Fewer visually distinct consonant clusters and vowels in Japanese than in English.
Development of Auditory-Visual Speech Perception Infancy Infants match auditory and visual speech information (Burnham & Dodd, 2004, Rosenblum et al., 1997) Childhood McGurk & MacDonald (1976) 3 to 8 years + adults Massaro et al. (1986) 4-6 years + adults Adulthood AV integration adults > children (Sekiyama & Burnham, in press; Massaro et al.,1986) Late adulthood older Ss > younger Ss on vısual speech information when auditory sensitivity is diminished (Cienkowski & Carney, 2002).
Focus: Developmental Data from Infants and Children Why is there an age-related increase? Infancy and childhood are marked with several novel language-relevant and language-specific challenges, such as phonetic phonological semantic vocabulary reading
Japanese and Australian English Speakers Three child (6-, 8-, and 11-year-olds) groups and a control group of adults. Method McGurk stimuli DV: Visual Speech Index (VSI) Score based on the proportion of auditory-based responses Identification Task: /ba/, /da/, or /ga/? Sekiyama & Burnham (in press): an ontogenetic + differential study
AV+ minus A Augmentation A minus AV- Interference Visual Speech Index (VSI) [AV+] minus [AV-] 6-yos 8-yos 11-yos Adults
Sekiyama and Burnham (in press) found: Developmental increase in visual speech influence with age… In English speakers but Not in Japanese speakers So….. What factors modulate the development of auditory-visual speech perception (measured by VSI), especially between 6 and 8 years of age?
Experiment 1: Factors Investigated Language Specific Speech Perception (LSSP) Effect of phonological experience in native language: Native Speech Perception minus Non-native Speech Perception (N-NN) (Burnham, 2003). Reading Reading ability is positively related to LSSP in children (Burnham, 2003). Evidence for reading-lipreading link: (de Gelder & Vroomen, 1998; Cavé et al., 2007) Articulation ability is positively related to visual speech influence in children (Desjardins et al., 1997), and in adults with cerebral palsy (Siva et al., 1995)
Experiment 1: Method Language Specific Speech Perception (LSSP) Test: (Native minus Non-native speech perception N-NN) 18 N [voiced vs. voiceless] + 18 NN [voiced vs. prevoiced] Reading Test WRAT-3 reading subtest: 15 letters + 42 words = 57 items Articulation Test Queensland Articulation Test (QAT) Naming task: 64 pictures Aus English consonants: initial, medial, & final positions Auditory-Visual Speech Perception (AVSP) Test McGurk paradigm: A & V combinations of /ba/, /da/, and /ga/ 48 McGurk stimuli: (24 AV, 12 AO, and 12 VO), 3 consonants x 4 talkers (2 Eng & 2 Jap) x 2 congruence types x 2 background noise (Clear & noisy [+4 dB]) versions Visual Speech Index (VSI) AV (+) minus AV (-)
Experiment 1: Method Participants n=96; Australian English 5-, 6-, 7-, and 8-year-olds n=48; Australian English adults N=144 Procedure Individual testing (~60 minutes) Task orders counterbalanced Clear & Noise (+4dB SN Ratio) Each stimulus presented twice
Exp. 1 Results: AVSP Test VSI scores: Linear increase with age (p <.01) + Quadratic between 5 and 6 years (p <.01). VO scores: Linear increase with age (p <.01). AO scores: Linear increase with age (p <.01).
Exp. 1 Results: LSSP, reading and articulation LSSP scores: No age-based differences (p >.01) but native > non-native (p<.01) Reading scores: Increase with age (p<.01) Articulation scores: Increase with age (p children (p <.01).
Children Age + AO + VO + LSSP + Articulation + Reading VSI Adults Age + AO + VO + LSSP +Articulation + Reading VSI Exp. 1 Results: Regression Analyses (DV = VSI Scores)
Experiment 1: Discussion AVSP and lipreading (VO) increase with age This increase is sharp at around 6 years of age, around the onset of reading instruction. No age-based LSSP differences native and non-native difference is largest at 6 years Reading instruction a new challenge new sources of speech information used? Regression Analyses Children: lipreading + LSSP VSI Adults : AO VSI Correlation Analyses Articulation is correlated with AO, lipreading (VO), and reading.
Exp. 1 Discussion Experiment 2 Experiment 1: LSSP & Lipreading AVSP Schooling and/or new language-specific challenges seem to increase visual speech influence. Towards Experiment 2 (3- & 4-year-olds): ? Status of LSSP AVSP link in pre-orthographic stages? Are AVSP and basic Cognitive abilities related? Are AVSP and Vocabulary Knowledge related?
Experiment 2: Method Participants: N=48; 3-year-olds & 4-year-olds AVSP – material, stimuli and procedure AVSP: AX discrimination task Three conditions AV(36), AO(12), VO(12) AX-VSI (/1): based on different trials LSSP – material, stimuli and procedure 18 Native [pa-p h a] & 18 Nonnative [ba-pa] trials. AX-based category change paradigm. Vocabulary Test – material, stimuli, and procedure PPVT (Dunn & Dunn,1997) age-based word sets of 12 items on average 5-6 sets. Cognitive Flexibility – material, stimuli and procedure FIST (Jacques & Zelazo, 2001) 15 items (shapes, sizes & colours)
Exp. 2 Results: AVSP Test VSI-AX scores: 4-year-olds > 3-year-olds (p <.01) VO scores: 4-year-olds ≈ 3-year-olds (p =.07) AO scores: 4-year-olds > 3-year-olds (p <.05)
Experiment 2: Discussion AVSP, LSSP, vocabulary and FIST improve with age AVSP improves between 3 and 4 years. Lipreading p=.07 – same age-based trend, but small age difference? Regression Analyses AO & FIST VSI-AX similar to adults in Exp. 1 (AO VSI) Why does AO predict VSI-AX? AVSP product of AO speech perception Three and 4 years: object-word pairing is a challenge, but not as much as reading – so no need for extra info. AVSP is determined by inherent abilities? e.g., AO speech perception and cognitive function.
Regression Analyses: LSSP, age, language tests regressed onto AVSP 5,6,7, 8 years LSSP + VO + AO + Articulation + Reading AVSP AVSP is related to LSSP and lipreading Adults LSSP + VO + AO +Articulation + Reading AVSP AVSP is related only to auditory speech perception 3, 4 years LSSP + VO + AO+ Cognitive + Vocab. AVSP AVSP is related only to auditory speech perception
General Discussion Visual speech influence develops with age AV: e.g., Massaro et al. (1986) VO: e.g., Desjardins et al. (1997) (though p=.07 in Exp.2) AVSP follows a phonetic phonological trend AVSP is amodal and phonetic (Burnham & Dodd, 2004; Rosenblum et al., 1997) AVSP is phonological (Massaro, 1998) But more importantly, AVSP seems to occur at all levels of language processing.
AVSP- Articulation Link Vocabulary and articulation challenges in early childhood Tests of AVSP, LSSP, and articulation for children (12-24 months), with and without speech disorder (Dodd et al.,in press). AVSP-Reading Link Burnham (2003): reading LSSP LSSP AVSP So, integrative ability phonemic/phonological processing This may be the case for English but not for other languages? Around school age, language-relevant challenges emerge Reading phoneme-grapheme mapping LSSP intensifies (orthographic stage: Burnham et al., 2002) Children use all speech information available Hyperarticulated speech styles (Lees & Burnham, 2005). e.g., teacherese (Håkansson, 1987). Cognitive skills activated + maturation General Discussion
Past Current Future AVSP and reading link (Cave et al., 2007; de Gelder & Vroomen, 1998) Burnham (2003) Articulation-lipreading link (Desjardins et al.,1997)
Some questions awaiting answer Speech Production-Perception Link: the perception-production link: a developmental perspective; e.g., testing children with speech disorders (Dodd et al., in press). Visual Speech and Reading Link: Research on the link between study strong correlations between visual aspects of speech and reading This study correlations between reading and AO, lipreading and articulation. Cavé et al. (2007); de Gelder & Vroomen (1998). Cross-language reading x AVSP studies.
Acknowledging……. Prof. Denis Burnham Prof. Barbara Dodd, Asc.Prof. Chris Davis, Prof.Kaoru Sekiyama MARCS Auditory Laboratories & MARCS Baby Lab. College of Arts, University of Western Sydney Australian Postgraduate Award Aprica Foundation, Osaka, Japan Participating schools, teachers, parents and kids in Sydney