Towards a stochastic theory of saccadic eye movement GURU Oct 1, 2001 Jonathan Nelson

Thanks especially to Wendy Ark Gary Cottrel Carrie Joyce Javier Movellan Marty Sereno Ruth Williams

Outline Function of eye movement in vision Traditional analyses of eye movement Three papers using stochastic processes to model eye movement Research towards a better generative model

Why model eye movement? for cognitive science and economics: how do a person’s knowledge and goals influence their information-gathering in perception? (applications also to entertainment and interface design) for medical diagnosis: how does a physician/technician read an x-ray? Can we automate or assist the physician? for understanding and treating pathological conditions: autism, williams’ syndrome. Can training appropriate eye movement help autistics in social situations? for artificial perception: are the visual data that humans acquire also informative for active, sequential artificial vision systems?

Saccadic eye movement Eccentricity and image resolution Saccades and fixations Relationship to task, applications

Traditional analyses Large number of papers Inspired by behavioral statistics: lose dynamic character, or dichotomize/polychotomize time Yet unexplained qualitative results: “highly individual scanpaths” “high individual consistency from trial to trial”

Towards a stochastic theory Regions of interest in an image Timescale

Regions of interest:

Timescale Discrete or continuous? Every point sampled by the eye tracker? Every nth millisecond? Every fixation?

Timescale Nearby fixations grouped together Saccade points rejected

What kind of SP? Discrete state, usually with regions of interest that contain current point of gaze as state Discrete time, with points of meaningful fixations forming the time step Time-homogenous Markov chain

White et al.: mammograms How do experts read mammograms? Can experts’ patterns in reading mammograms suggest means of computer aided diagnosis?

from

White et al.’s modeling One model per subject, per image A sequential clustering algorithm to determine regions of interest Exclude transitions to same state, and fixations outside of regions of interest “Not enough data”

Joyce: face perception Goal: augment or corroborate eyewitness testimony, using eye movement EEG GSR Task: view a face recognize it as not novel, after a delay

Task: “Study this face”

Task: “Have you seen this face before?”

Joyce’s regions of interest 10 ROI: hair right eye nose etc. Normalized faces Nearest neighbor

Joyce’s model Markov States: each of 10 regions of interest Time: from entering region till leaving region is one discrete step

Joyce’s results More entropy in saccade sequences when viewing a novel face Matches qualitative findings

Eye-typing: Salvucci Goal: have users type a pre-selected word, by looking at an on-screen keyboard Models used: saccade or fixation HMM letter HMM grammar

Saccade-fixation HMM If previous time is fixation, current time is most likely fixation If low velocity, current time most likely fixation “Standard HMM parameter estimation” (Rabiner, 1989) Viterbi to optimize

Fixations and centroids

“Centroid submodel HMM” States: peaked bivariate distribution for (x, y) diffuse bivariate distribution for (x, y)

Grammar for words Which word gives the highest likelihood of the centroid data?

Results 92% accuracy with 1000 word vocabulary too computationally intensive for realistic-size (e.g. 50,000 words) vocabulary, for next few years … a nice proof of concept (previous systems required >750ms between fixations, and 4 degrees between targets) (for simpler multiple-letter models, use a string-edit distance algorithm to find the nearest vocabulary word)

Future work More temporal dependence: try 3rd+ order model Can a sophisticated string-edit distance algorithm correct for bias in limited sampling?

Future work Systematic exploration of time index Improvement of saccade-fixation HMM Distribution of fixations within regions What is the nature of individual differences? Relationship of knowledge and goals to task

Sources Cited White, KP; Hutson, TL; Hutchinson, TE (1997). Modeling human eye behavior during mammographic scanning: preliminary results. IEEE Transactions on Systems, Man and Cybernetics, A, 27, Salvucci, DD (1999). Inferring intent in eye-based interfaces: tracing eye movements with process models. Proceedings of the 1999 Computer Human Interaction Conference. Joyce, CA (2000). Saving Faces: Using Eye Movement, ERP, and SCR Measures of Face Processing and Recognition to Investigate Eyewitness Identification. Ph.D. dissertation, University of California, San Diego.