Anaphoric dependencies : A window into the architecture of the language system Eye tracking experiments Eric Reuland Frank Wijnen Arnout Koornneef

EYE-TRACKING BY ARNOUT KOORNNEEF

OVERVIEW OF LECTURE Part 1: discussion of method - eye-tracking while reading Part 2: discussion of current research - eye-tracking experiments

THE EYE-TRACKER

monitors eye-movements from millisecond to millisecond provides information about where people look and for how long

TWO MAIN EYE- TRACKING PARADIGMS eye-tracking while reading eye-tracking while listening (not discussed)

EYE-TRACKING WHILE READING

SOME FACTS ABOUT READING people do not read a text “smoothly”, but fixate a particular word ( msec) and jump to the next a jump (or saccade) covers 7-9 letter spaces during a saccade visual input is reduced readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) readers regress (look back) readers often undershoot on return sweeps (going from the end of a line to the next line) the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING people do not read a text “smoothly”, but fixate a particular word ( msec) and jump to the next a jump (or saccade) covers 7-9 letter spaces during a saccade visual input is reduced readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) readers regress (look back) readers often undershoot on return sweeps (going from the end of a line to the next line) the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING people do not read a text “smoothly”, but fixate a particular word ( msec) and jump to the next a jump (or saccade) covers 7-9 letter spaces during a saccade visual input is reduced readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) readers regress (look back) readers often undershoot on return sweeps (going from the end of a line to the next line) the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING people do not read a text “smoothly”, but fixate a particular word ( msec) and jump to the next a jump (or saccade) covers 7-9 letter spaces during a saccade visual input is reduced readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) readers regress (look back) readers often undershoot on return sweeps (going from the end of a line to the next line) the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING people do not read a text “smoothly”, but fixate a particular word ( msec) and jump to the next a jump (or saccade) covers 7-9 letter spaces during a saccade visual input is reduced readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) readers regress (look back) readers often undershoot on return sweeps (going from the end of a line to the next line) the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING people do not read a text “smoothly”, but fixate a particular word ( msec) and jump to the next a jump (or saccade) covers 7-9 letter spaces during a saccade visual input is reduced readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) readers regress (look back) readers often undershoot on return sweeps (going from the end of a line to the next line) the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING people do not read a text “smoothly”, but fixate a particular word ( msec) and jump to the next a jump (or saccade) covers 7-9 letter spaces during a saccade visual input is reduced readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) readers regress (look back) readers often undershoot on return sweeps (going from the end of a line to the next line) the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING people do not read a text “smoothly”, but fixate a particular word ( msec) and jump to the next a jump (or saccade) covers 7-9 letter spaces during a saccade visual input is reduced readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) readers regress (look back) readers often undershoot on return sweeps (going from the end of a line to the next line) the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

A TYPICAL READING EXPERIMENT Garden-path sentence Since Jay always jogs a mile seems like a short distance to him. Control sentence Since Jay always jogs a mile this seems like a short distance to him.

READING PATTERN (Garden-path sentence) SinceJayalwaysjogsamileseems likeashortdistancetohim.

READING PATTERN (Garden-path sentence) Since Jay always jogs a mile like a short distance to him. = fixation after progressive saccade (first-pass) = fixation after regressive saccade = fixation after progressive saccade (second-pass) seems

READING PATTERN If readers experience some sort of trouble they may fixate the difficult region longer and the may even regress to earlier parts of the sentence/text.

HOW DO WE INTERPRET THE READING PATTERNS?

DIFFERENT MEASURES First fixation duration: duration of first fixation in a region First-pass duration: time spent in a region before moving on or looking back Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time Second-pass duration: duration of re-fixations Total duration: the sum of all fixations in a region Probability of a regression: the percentage of regressive eye-movements out of a region

DIFFERENT MEASURES First fixation duration: duration of first fixation in a region First-pass duration: time spent in a region before moving on or looking back Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time Second-pass duration: duration of re-fixations Total duration: the sum of all fixations in a region Probability of a regression: the percentage of regressive eye-movements out of a region

DIFFERENT MEASURES First fixation duration: duration of first fixation in a region First-pass duration: time spent in a region before moving on or looking back Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time Second-pass duration: duration of re-fixations Total duration: the sum of all fixations in a region Probability of a regression: the percentage of regressive eye-movements out of a region

DIFFERENT MEASURES First fixation duration: duration of first fixation in a region First-pass duration: time spent in a region before moving on or looking back Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time Second-pass duration: duration of re-fixations Total duration: the sum of all fixations in a region Probability of a regression: the percentage of regressive eye-movements out of a region

DIFFERENT MEASURES First fixation duration: duration of first fixation in a region First-pass duration: time spent in a region before moving on or looking back Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time Second-pass duration: duration of re-fixations Total duration: the sum of all fixations in a region Probability of a regression: the percentage of regressive eye-movements out of a region

DIFFERENT MEASURES First fixation duration: duration of first fixation in a region First-pass duration: time spent in a region before moving on or looking back Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time Second-pass duration: duration of re-fixations Total duration: the sum of all fixations in a region Probability of a regression: the percentage of regressive eye-movements out of a region

DIFFERENT MEASURES First fixation duration: duration of first fixation in a region First-pass duration: time spent in a region before moving on or looking back Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time Second-pass duration: duration of re-fixations Total duration: the sum of all fixations in a region Probability of a regression: the percentage of regressive eye-movements out of a region

EXPLANATION OF DIFFERENT MEASURES Bart annoyed Homer because… Reading Times for word 3 (Homer) First fixation duration = 3 First-pass duration = Regression Path duration = Second-pass duration = 6 Total duration =

DIFFERENT FIRST-PASS MEASURES FIRST FIXATION DURATION FIRST-PASS DURATION REGRESSION PATH DURATION TIME

HOW DO WE INTERPRET THE READING TIMES?

THE LINKING PROBLEM eyemind

EYE-MIND ASSUMPTION (JUST & CARPENTER, 1980) Readers retain fixation on a word until processing is completed This includes processes like word recognition, syntactic parsing, semantic integration, referential integration

AN IDEAL WORLD (VAN BERKUM, 2004) Snowwhitekissedadwarf WPSR WPSR WPSR WPSR TIME W = word recognition; P = parsing; S = semantic integration; R = referential integration

THE REAL WORLD IS A REAL MESS (VAN BERKUM, 2004) Snowwhitekissedadwarf WPSR WPSR WPSR WPSR TIME W = word recognition; P = parsing; S = semantic integration; R = referential integration

THE REAL WORLD IS A REAL MESS In the real world the processing of word X continues while fixating word X + 1 (and possibly while fixating word X + 2 etc.) Thus, the effects of a manipulation are often visible more downstream (i.e. after the critical word or region). This is called spill-over.

THE REAL WORLD IS A REAL MESS In the real world the processing of word X continues while fixating word X + 1 (and possibly while fixating word X + 2 etc.) Thus, the effects of a manipulation are often visible more downstream (i.e. after the critical word or region). This is called spill-over.

LINKING ASSUMPTION (BOLAND, 2004) The eyes do not leave a word until it has been structurally integrated (tree building). Therefore, constraints that control structure-building influence first-pass reading time. other measures (e.g., regression path duration) are sensitive for higher level processes (semantic integration, discourse processes)

SOLUTION LINKING PROBLEM? Perhaps the different measures can provide information about what is happening? (this is an empirical question)

A DISADVANTAGE OF THE READING PARADIGM You can only use skilled readers (no children or language-disordered populations). This is possible in spoken language paradigms.

CAVEAT: the eyes tell us that something is happening at a specific point in time, but not what that something is!

TO CONCLUDE Eye-tracking (while reading and listening) excels in the “when” question. Not really suited for “what” question (use EEG/MEG instead).

QUESTIONS?

PART 2: THEORY

CONSTRUCTING ANAPHORIC DEPENDENCIES: VARIABLE BINDING vs. CO-REFERENCE

BASIC ARCHITECTURE PROCESSING SYSTEM distinct modules  syntax, semantics, discourse economy principle: cross-modular operations carry a cost (Reuland 2001) syntax first in time-course-model of processing phases (Friederici 1995-)  syntax is cheaper than semantics  semantics is cheaper than discourse

CROSS-MODULAR OPERATIONS Discourse storage (values)a  a C-I objects (variables)x 1 x 2 Syntactic objects (chains)C 1 C 2 Basic expressionsαβ Discourse storage (values)a C-I objects (variables)x 1  x 1 Syntactic objects (chains)C 1 C 2 Basic expressionsαβ Discourse storage (values)a C-I objects (variables)x 1 Syntactic objects (chains)C 1  C 1 Basic expressionsαβ

TIME-COURSE SIMPLIFIED SYNTAX SEMANTICS DISCOURSE

SEMANTICS VS. DISCOURSE variable binding (semantic dependency)  Every clown thinks that he is not funny. co-reference (discourse dependency)  The clown has a big problem. He is not funny.

PREDICTIONS OF THEORY variable binding is cheaper/faster than co-reference in an ambiguous situation variable binding has precedence over co-reference

END OF INTRO