in Single and Multiple Modalities, Invariance Detection in Single and Multiple Modalities, Word-Mapping and Vocabulary Development Lakshmi J. Gogate1, Divya Awal1, & Christopher G. Prince2 1SUNY Health Science Center at Brooklyn Brooklyn, NY and 2University of Minnesota Duluth Duluth, MN

Main Points For the infant, mapping a word onto an object is a multimodal activity involving at least the auditory and visual modalities. Young infants perceive invariant unimodal and multimodal properties of stimuli during word mapping. (e.g. temporal synchrony when a word is spoken at the same time with a moving object, phonetic invariance given multiple tokens of words, and invariant object motion when a mother moves it during naming). By describing the invariant unimodal and multimodal properties occurring in combination in multisensory communication, and elucidating how infants interact with their environment to perceive these invariant properties, we can explain word-mapping development in preverbal infants.

Main Objective To show using three different studies of syllable-object mapping that preverbal infants attend to multimodal and unimodal invariance in word-mapping tasks. A longitudinal study of 2 month-old preterm and full-term infants’ sensitivity to temporal synchrony and vocabulary development at 12 months. A study of 7- and 8-month-olds mapping of minimal versus non-minimal pair syllables onto objects and a computational model of synchrony detection. A study of object motion during maternal bimodal naming and word mapping at 6- to 8 months.

Study 1: Questions Are there differences in full-term versus preterm infants’ sensitivity to invariant temporal structure when spoken syllables co-occur with moving objects? Might these differences account for language delay in preterm infants?

Participants The sample thus far - 40 2-month-old full-terms, 30 2-month-old preterms at uncorrected age (EGA range 31-36 weeks, mean EGA 34.37 weeks), 13 of the same preterms at 2 months corrected age, and 10 each of the same full-terms and preterms at 12 months. All infants in the sample were reported to have normal hearing and vision, and no major health complications during or after birth (e. g., preterm infants with IVH or acute respiratory distress were not included).

Method Infant-Controlled Habituation Procedure at 2 Months- Preterms – 2 visits, full-terms 1 visit only. Habituation Phase- (1 S-O pair) e.g.,/gah/-porcupine until criterion decrement in visual fixation. Preterms at visit 1 received a different syllable-object pairing than at visit 2. Test Phase – Syllable-change e.g., /tah/-porcupine Object-change e.g.,/gah/-crab MacArthur-Bates Communicative Development Inventory at 12 months (Fenson et al, 1998; Infant version).

Figure 1. The crab, the porcupine (above), the lamb chop and the star (below).

Synchronous Presentation

Results – The Habituation Phase At uncorrected age 2 months, preterm infants attended less to invariant temporal synchrony in syllable-object pairs relative to their full-term counterparts during habituation (Table 1). At 2 months corrected age, preterms attended to synchronous syllable-object pairings similar to full-term infants.

Table 1. Two month-old full-term and preterm infants' looking time (sd) during infant-controlled habituation to a temporally synchronous syllable-object pair. Full-terms (n = 40) Preterms Uncorrected Age (n = 30) Corrected Age (n = 13) Baseline (mean of first two habituation trials) 38.11a (8.97) 23.38b (8.86) 31.21 (9.53) Mean looking on the last two habituation trials 10.01a (6.01) 5.81b (3.44) 10.18 (4.00) Mean seconds to reach habituation criterion 247.75a (98.77) 141.22 b (75.29) 237.65 (112.24) a b Independent samples t-tests, p < .01

Results- The Test Phase Replicating our findings from Gogate, Prince & Matatyaho (under review), the full-term infants looked longer to a change in the syllable and in the object. In contrast, the preterm infants, at uncorrected or corrected age 2 months did not look longer to either change.

Figure 2: Preterm and full-term 2-month-olds’ mean looking (and SD) to temporally synchronous syllable-object displays. /tah/-/gah/ Mean Looking Time (sec.) **p < .01 *p <.05

Results: Relation between Sensitivity to Temporal Synchrony at 2 mos Results: Relation between Sensitivity to Temporal Synchrony at 2 mos. and Vocabulary Development at 12 mos. Receptive and productive vocabulary at 1 year on the MacArthur-Bates Communicative Development Inventory is significantly lower for pre-terms compared to full-terms (Figure 3). Detection of temporal invariance between a spoken syllable and a moving object at 2 months chronological age on the habituation task is significantly correlated with receptive vocabulary at 12 months for full- and pre-term infants taken together (Pearson r (19) = .56, p < .05).

Figure 3. Vocabulary at 12 months on the MCDI

Study 1: Conclusions Invariance detection across auditory and visual modalities is integral to word learning and might predict language delay in preterm infants. Young full-term infants perceive invariant properties such as temporal synchrony presented simultaneously to multiple sensory modalities. Preterms have problems. Infants also likely perceive invariant properties of the visual object in motion, and the acoustic-phonetic properties of the syllables. More direct evidence . . .

Study 2- Infants’ Detection of Unimodal Invariance During Word Mapping- Minimal versus Non-Minimal Pair Syllables (Gogate & Lee, in prep.). Participants: 2 groups of 8-month-olds (n = 16 in each) and 3 groups of 7-month-olds (n = 16). Infant-Controlled Habituation Procedure: Habituation Phase (Synchronous or Asynchronous Condition)- (2 S-O pairs) e.g., /gah/-porcupine and /tah/-crab until criterion decrement in visual fixation. Test Phase - Switch trials e.g., /tah/-porcupine and /gah/-crab Control trials - /gah/-porcupine and /tah/-crab

MINIMAL PAIRS /tah/-/gah/ Figure 4: Exp 1. Eight-month-old infants’ mean looking (SD) to syllable-object displays as a function of trial type and condition. MINIMAL PAIRS /tah/-/gah/ 21.8 (13.47) Mean Looking Time (secs.) **p < .01 1 Visual recovery is a difference score between visual fixation during the switch trials versus visual fixation during the control trials.

MINIMAL PAIRS- /tah/-/gah/ Figure 5: Exp 2. Seven-month-old infants’ mean looking (SD) to syllable-object displays as a function of trial type and condition. MINIMAL PAIRS- /tah/-/gah/ 47.3 (15.84) Mean Looking Time (secs.) 1 Visual recovery is a difference score between visual fixation during the switch trials versus visual fixation during the control trials.

Figure 6: Exp 3. Seven month olds’ mean looking (SD) to synchronous syllable-object displays as a function of trial type. NON-MINIMAL PAIRS /tah/-/gih/ or /gah/-/tih/ **paired sample t-test p < .01 1 Visual recovery is a difference score between visual fixation during the switch trials versus visual fixation during the control trials.

Study 2: Conclusions Eight-month-olds map two similar sounding syllables onto objects when the syllables are spoken in temporal synchrony with object motions. In contrast, 7-month-olds map distinct syllables onto objects under the same stimulus presentation conditions. This preference for more distinct syllables at the earlier age shows that in addition to attending to bimodal invariants (temporal synchrony), infants attend to unimodal (acoustic/phonetic) invariants in syllables during word mapping.

A Sensory-Oriented Computational Model of Infant Bimodal Invariance Detection The SenseStream Program used digital video files of our experiments as input and computed the degree of A-V synchrony defined as Gaussian mutual information between A and V (Hershey & Movellan, 2000; Prince & Hollich, 2005). The model estimated the degree to which the visual feature values (each grayscale pixel), over a length of window S (e.g., 15 frames of video), co-varied with the audio feature values (MFCC audio features). This was displayed as a mixelgram - a saliency map of the amodal (invariant) information in the audio and visual inputs. Image processing techniques produced a number, an estimate of the degree of audio-visual synchrony for each mixelgram. These numbers, averaged for each of 16 video clips, were compared across synchronous and asynchronous clips.

Figure 7: The mean synchrony estimates for each of the video clips Note. Independent samples t (39,471) = 10.29, p < .0001. The synchronous (synch) and the asynchronous (asynch) video clips are not specifically paired. We present the two plots (synch and asynch) together to illustrate the differences.

Conclusion The model detects invariance in auditory-visual stimuli in a manner analogous to preverbal infants.

Study 3: Maternal Object Motion During Object Naming Predicts Infants’ Word-Mapping Method: Mothers (N = 24) taught their 6- to 8-month old infants the names for two objects during a 3-min play episode. The episodes were video-taped and coded for maternal temporal synchrony and other bimodal naming styles, and infants’ attention to maternal naming. Next, infants were given an intermodal word-mapping test to examine their learning of the word-object pairings. PFLs on a two-choice word-mapping test were measured (Gogate, Bolzani, & Betancourt, 2006, Infancy). Later, the episodes were coded for type of object motion during maternal naming, e.g., forward, backward, upward, shaking motion in the infants’ line of sight (Matatyaho & Gogate, 2006, ICDL, Bloomington).

Figure 8 :The objects mothers used to teach infants 2 word-object pairs chi/gow gow/chi

Figure 9: Maternal Motion in Bimodal Naming to 6- to 8-month-old Infants p < .001 p < .001

Results Shaking and looming motions occurred more often in temporally synchronous maternal naming than other motions (Figure 9). In ANCOVAs, shaking and looming motions in the presence of temporal synchrony were significant covariates of 6- to 8-month-olds ability to map two word-object relations on an intermodal word mapping test (p < .05).

Study 3: Conclusions Six to 8-month-old infants attend to invariant visual motions of objects during word mapping tasks.

General Conclusions During word mapping infants detect invariants in bimodal and unimodal patterns of stimulation. These invariant properties occur in combination as we have seen. By charting the interactions between these bimodal and unimodal invariants in communication, and explaining how infants utilize them, we are able to explain how preverbal infants develop word mapping abilities.

Acknowledgements Dean’s Research Initiative Grant, College of Medicine, SUNY, and The Thrasher Research Fund (#02819-1). Staff: Samantha Berkule, Elsa Lee, Dalit Matatyaho, Li-Fen Chen, and Linda Yoo. The parents and infants who participated.