Explaining human multiple object tracking as resource-constrained approximate inference in a dynamic probabilistic model Ed Vul Mike Frank George Alvarez Josh Tenenbuam Support from: ONR-MURI (PI: Bavelier); NDSEG (Vul); NSF (Vul)

XNL PFJ DQR GAU

Multiple object tracking What limits performance?

What limits our performance? Juice? (Sperling, 1960; Vogel Machizawa 2003) Number tracked out of 16 total (Alvarez, Franconeri, Cavanagh) Slots?

Explanations of success and failure Harder when objects are faster (Alvarez & Franconeri, 2007) Easier when they are further apart. (Franconeri et al., 2008) We remember object velocity (Horowitz, 2008) But we don’t seem to use it (Keane & Pylyshyn, 2005) Well, maybe sometimes (Fencsik et al. 2005) Keeping track of features is hard (Pylyshyn, 2004) But we can track in color space (Blaser, Pylyshyn, & Holcombe, 2000) And unique colors help (Makovski & Jiang, 2009) We can only track a few objects (Pylyshyn & Storm, 1988; Intrilligator et al., 2001) But we can track more if they are slower (Alvarez & Franconeri, 2007) Crowding? Spotlights? Inhibitory surrounds? FINSTs? Slots? Juice? Juice boxes? Speed limits? Must specify performance of an unconstrained observer before postulating limitations. PhenomenaExplanations

Stark contrast with the real world

Puzzles What limits human performance in object tracking? Discrepancy between poor performance in the lab and robust behavior in the real world?

Our approach Look to engineering to see how people should track objects. Do people track objects this way? What resources would limit performance? How should these resources be allocated? Do people allocate resources in this manner?

An ideal observer for MOT ? ? ? ? … ? ? ? ? A B C D S0S0 m0m0 α0α0 S1S1 m1m1 α1α1 dynamics StSt mtmt αtαt … S: states m: observations α: data association

Dynamics and process uncertainty S0S0 m0m0 α0α0 S1S1 m1m1 α1α1 dynamics StSt mtmt αtαt … Parameterize by inertia and spacing / velocity variance of stationary distribution.

Measurement uncertainty Eccentricity scaling of position errors. Weber scaling of velocity errors S0S0 m0m0 α0α0 S1S1 m1m1 α1α1 dynamics StSt mtmt αtαt … S0S0 m0m0 α0α0 S1S1 m1m1 α1α1 StSt mtmt αtαt … ? ? ? ? Unknown data-associations

An ideal observer for MOT A B C D A B C D A B C D A B C D Measurement Assignment Estimate Prediction P(α t | m t, S’ t ) (sampled by particle filter) P(S t | α t, m t, S’ t ) (Kalman filter) P(S’ t+1 | S t, dynamics) (Kalman filter) mtmt

An ideal observer for MOT

Speed/space for people (Franconeri et al., 2008)

Speed/space for the ideal observer The correspondence problem is harder as speed increases or spacing decreases. faster closer

Speed/space for the ideal observer 10 0 Model thresholds Faster  Closer  Spatial concentration (1/σ x ) Velocity variance (σ v )

Speed/space tradeoff experiment Adjust minimal spacing for given a fixed speed. Track 3 of 6

Speed/space tradeoff Model thresholds Human thresholds 10 0 Faster  Closer  Spatial concentration (1/σ x ) Velocity variance (σ v ) People make similar tradeoffs between speed and space as the ideal observer.

Incorporating extra dynamic cues

10 0 Model thresholds Human thresholds Faster  Closer  Spatial concentration (1/σ x ) Velocity variance (σ v ) People make similar tradeoffs between speed/space and color as the ideal observer. Color-drift = 0.02π Color-drift = 0.2π

Predictive use of inertia 10 0 Inertia stable Inertia unstable Model thresholds Human thresholds Inertia = 0.7 Inertia = 0.9 Faster  Closer  Spatial concentration (1/σ x ) Velocity variance (σ v ) People use velocity when appropriate

Number tracked out of 16 total Ideal object tracking Speed Spacing Intermittent use of inertia Use of additional cues Track duration Additional distracters Phenomena explained We can only track a few objects (Pylyshyn & Storm, 1988; Intrilligator et al., 2001) But we can track more if they are slower (Alvarez & Franconeri, 2007) ???

Resources in tracking objects S t+1 StSt mtmt m t+1 α t+1 αtαt dynamics Limited ability to propagate state estimates: Limited memory fidelity P(S’ t+1 | S t + w m, dynamics) S w[i] = S w / ( 1 + √a i ) A resource that reduces noise in propagation of state estimates:

Number of targets Limited memory produces the speed-number tradeoff. Number tracked out of 16 total Human thresholds (Alvarez & Franconeri, 2007) Number of targets Number tracked out of 16 total Model thresholds Faster  Velocity variance (σ v )

Strategic allocation of resources Allocating resources Planning by greedy 1-step look-ahead sampling. Make one object more demanding. modelhuman Position Velocity TargetsDistractors Object 1 Object n … TargetsDistractors Position Velocity Object 1 Object n …

Contributions Ideal observer for human object tracking based on common engineering models… …Accounts for effects of speed, space, inertia, time, additional cues, distracters. A resource such as finite memory for state estimates accounts for target # effects. Teaser about strategic allocation of resources.

Position Velocity TargetsDistractors Object 1 Object n … TargetsDistractors Position Velocity Object 1 Object n …

How do people track objects? Harder to track when objects are faster (Alvarez & Franconeri, 2007) Easier when they are further apart. (Franconeri et al., 2008) We remember object velocity (Horowitz, 2008) But we don’t seem to use it (Keane & Pylyshyn, 2005) Yet it helps us track (Fencsik et al. 2005) Keeping track of identity is hard (Pylyshyn, 2004) But we can track in color space (Blaser, Pylyshyn, & Holcombe, 2000) And unique colors help (Makovski & Jiang, 2009) We can only track a few objects (Pylyshyn & Storm, 1988; Intrilligator et al., 2001) But we can track more if they are slower (Alvarez & Franconeri, 2007)

Inertia and trajectories Assumed “inertia” predicts future dot position. With inertia Without inertia

Inertia and trajectories With inertia Without inertia Model without inertia Inertia = 0.7 Inertia = 0.8 Inertia = σxσx Model with inertia Inertia = 0.7 Inertia = 0.8 Inertia = σxσx faster  Velocity std. dev. σvσv σxσx more space  Position std. dev. Inertia = 0.7 Inertia = 0.9 Inertia varied Across subjects Inertia varied Across trials People use inertia / extrapolated trajectories to track like the ideal observer: Only when it is stable and predictive.

Why only use inertia sometimes? Underestimating inertia under uncertainty?

Allocating a finite resource What allocation policies do people consider? Just Target vs. Non-Target? Or can people more flexibly adapt to task demands –Allocate to individual items Position Velocity TargetsDistractors Beta(p,v) Beta(T, D) Dirichlet(Α) Object 1 Object n …

How flexible is resource allocation? Track 4 of 8 1 tracked object (and one distractor) is “crazy charlie”: –Charlie moves faster/slower than the other targets. –Measure performance on other 3 targets. Key question: –Can we flexibly allocate to individual objects based on need? If so, accuracy on other targets should decrease as Charlie’s speed increases. –If we only allocate to targets vs. non-targets, Charlie’s speed should not matter.

Optimal Crazy Charlie performance Log10(Charlie speed / Other speed) Accuracy on non-Charlie targets

Crazy Charlie experimental results Slow “Charlie” Fast “Charlie” Speed of “key” target alters performance on other targets Tracking one harder target “steals” resources from the others

Multiple object tracking (Pylyshyn & Storm, 1988; and many others.)

Tracking in non-spatial dimensions Are spatial and non-spatial features combined to track objects? Blaser, Holcombe, Pylyshyn, 2000

Tracking in non-spatial dimensions

Are spatial and non-spatial features combined to track objects? Blaser, Holcombe, Pylyshyn, 2000 Model thresholds Color-drift = 0.02π Color-drift = 0.2π 10 0 σxσx Color-drift = 0.02π Color-drift = 0.2π Human settings faster  Velocity std. dev. σvσv σxσx more space  Position std. dev. People trade off spatial and non- spatial information when tracking.

An ideal observer for MOT S0S0 m0m0 α0α0 S1S1 m1m1 α1α1 dynamics StSt mtmt αtαt … Given: 1) Starting state: α 0, m 0 2) Unlabeled measurements: m 1, … m t 3) Model of the dynamics. Find: Final labels: α t Final state: S t Invert to compute: P(S t, α t |m 0,…m t-1, α 0 ) ? ? ? ? … ? ? ? ? A B C D How should people track objects given available information?