CSSE463: Image Recognition Day 29 This week This week Today: Surveillance and finding motion vectors Today: Surveillance and finding motion vectors Tomorrow: motion and tracking Tomorrow: motion and tracking Lab 7: due Wednesday night Lab 7: due Wednesday night Thursday: Kalman filtering Thursday: Kalman filtering Friday: project workday in class, status report due Friday: project workday in class, status report due Questions? Questions?

Motion New domain: image sequences. New domain: image sequences. Additional dimension: time Additional dimension: time Cases: Cases: Still camera, moving objects Still camera, moving objects Detection, recognition Detection, recognition Surveillance Surveillance Moving camera, constant scene Moving camera, constant scene 3D structure of scene 3D structure of scene Moving camera, several moving objects Moving camera, several moving objects Robot car navigation through traffic Robot car navigation through traffic

Surveillance Applications: Applications: Military Military Hospital halls during night Hospital halls during night Stationary camera, moving objects Stationary camera, moving objects Separate background from objects Separate background from objects Q1

Finding moving objects Subtract images Subtract images What next… What next… How could you use this to find moving objects? How could you use this to find moving objects? Discuss with a classmate Discuss with a classmate Share with class Share with class Q2

Processing ideas Subtract images Subtract images Mark those pixels that changed significantly (over threshold) Mark those pixels that changed significantly (over threshold) Connected components. Fill? Connected components. Fill? Toss small regions Toss small regions Morphological closing to merge neighboring regions Morphological closing to merge neighboring regions Return bounding box Return bounding box

Issues with image subtraction Background model Background model Simplest: previous frame Simplest: previous frame General: find mean M and variance of many frames General: find mean M and variance of many frames Consider the hospital hallway with a window Consider the hospital hallway with a window How to handle “drift” due to illumination changes? How to handle “drift” due to illumination changes? For each pixel p with mean M: M new =  M old + (1-  )p For each pixel p with mean M: M new =  M old + (1-  )p Consider what happens when a person enters the scene Consider what happens when a person enters the scene Background model adapts to her Background model adapts to her What happens when she leaves? What happens when she leaves? Mean changes, so detects background as foreground Mean changes, so detects background as foreground Variance remains high, so can’t detect new arrivals. Variance remains high, so can’t detect new arrivals. Answer: multiple models Answer: multiple models

Issues with image subtraction Can you ignore distracting (expected) motion? Can you ignore distracting (expected) motion? Example: monitoring computer lab Example: monitoring computer lab

Motion vectors Difference in motion of specific objects Difference in motion of specific objects Show examples for pan. Show examples for pan. Create ones for zoom in/out. Create ones for zoom in/out. How to find? How to find? 2 techniques 2 techniques

What is image flow? Notice that we can take partial derivatives with respect to x, y, and time. Notice that we can take partial derivatives with respect to x, y, and time.

Image flow equations Goal: to find where each pixel in frame t moves in frame t+  t Goal: to find where each pixel in frame t moves in frame t+  t E.g. for 2 adjacent frames,  t = 1 E.g. for 2 adjacent frames,  t = 1 That is,  x,  y are unknown That is,  x,  y are unknown Assume: Assume: Illumination of object doesn’t change Illumination of object doesn’t change Distances of object from camera or lighting don’t change Distances of object from camera or lighting don’t change Each small intensity neighborhood can be observed in consecutive frames: f(x,y,t)  f(x+  x, y+  y, t+  t) for some  x,  y (the correct motion vector). Each small intensity neighborhood can be observed in consecutive frames: f(x,y,t)  f(x+  x, y+  y, t+  t) for some  x,  y (the correct motion vector). Compute a Taylor-series expansion around a point in (x,y,t) coordinates. Compute a Taylor-series expansion around a point in (x,y,t) coordinates. Gives edge gradient and temporal gradient Gives edge gradient and temporal gradient Solve for (  x,  y) Solve for (  x,  y)

Limitations Assumptions don’t always hold in real- world images. Assumptions don’t always hold in real- world images. Doesn’t give a unique solution for flow Doesn’t give a unique solution for flow Sometimes motion is ambiguous Sometimes motion is ambiguous “Live demo” “Live demo”