1. Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 19: Tracking
C280, Computer Vision
Prof. Trevor Darrell
Lecture 19: Tracking

2. Tracking scenarios Follow a point Follow a template
Follow a changing template
Follow all the elements of a moving person, fit a model to it.

3. Things to consider in tracking
What are the dynamics of the thing being tracked?
How is it observed?

4. Three main issues in tracking

5. Simplifying Assumptions

6. Kalman filter graphical model and corresponding factorized joint probabilityx1 x2 x3 y1 y2 y3

7. Tracking as induction Make a measurement starting in the 0th frame
Then: assume you have an estimate at the ith frame, after the measurement step.
Show that you can do prediction for the i+1th frame, and measurement for the i+1th frame.

8. Base case

9. Prediction step
given

10. Update step
given

11. The Kalman Filter Key ideas:
Linear models interact uniquely well with Gaussian noise - make the prior Gaussian, everything else Gaussian and the calculations are easy
Gaussians are really easy to represent --- once you know the mean and covariance, you’re done

12. Recall the three main issues in tracking
(Ignore data association for now)

13. The Kalman Filter
[figure from

14. The Kalman Filter in 1D Dynamic Model Notation Predicted mean
Corrected mean

15. The Kalman Filter

16. Prediction for 1D Kalman filter
The new state is obtained by
multiplying old state by known constant
adding zero-mean noise
Therefore, predicted mean for new state is
constant times mean for old state
Old variance is normal random variable
variance is multiplied by square of constant
and variance of noise is added.

18. The Kalman Filter

19. Measurement update for 1D Kalman filter
Notice:
if measurement noise is small,
we rely mainly on the measurement,
if it’s large, mainly on the prediction
s does not depend on y

21. Kalman filter for computing an on-line average
What Kalman filter parameters and initial conditions should we pick so that the optimal estimate for x at each iteration is just the average of all the observations seen so far?

22. Kalman filter model
Initial conditions
Iteration 1

23. What happens if the x dynamics are given a non-zero variance?

24. Kalman filter model
Initial conditions
Iteration 1

25. Linear dynamic models A linear dynamic model has the form
This is much, much more general than it looks, and extremely powerful

26. Examples of linear state space models
Drifting points
assume that the new position of the point is the old one, plus noise
D = Identity
cic.nist.gov/lipman/sciviz/images/random3.gif

27. Constant velocity We have Stack (u, v) into a single state vector
(the Greek letters denote noise terms)
Stack (u, v) into a single state vector
which is the form we had above
Here u is position, v velocity xi
Di-1
xi-1

28. Constant Velocity Model velocity position position time
measurement,position
Constant
Velocity
Model
These are lifts from figure The caption for that figure explains them
time

29. Constant acceleration
We have
(the Greek letters denote noise terms)
Stack (u, v) into a single state vector
which is the form we had above
Same notation as the last slide xi
Di-1
xi-1

30. Constant Acceleration Model velocity position position time
These are figure see caption for that figure
Constant
Acceleration
Model

31. Periodic motion
Assume we have a point, moving on a line with a periodic movement defined with a differential eq:
can be defined as
with state defined as stacked position and velocity u=(p, v)

32. Periodic motion
Take discrete approximation….(e.g., forward Euler integration with Dt stepsize.)
xi-1 xi
Di-1

33. n-D
Generalization to n-D is straightforward but more complex.

34. n-D
Generalization to n-D is straightforward but more complex.

35. n-D Prediction
Generalization to n-D is straightforward but more complex.
Prediction:
Multiply estimate at prior time with forward model:
Propagate covariance through model and add new noise:

36. n-D Correction
Generalization to n-D is straightforward but more complex.
Correction:
Update a priori estimate with measurement to form a posteriori

37. n-D correction Find linear filter on innovations
which minimizes a posteriori error covariance:
K is the Kalman Gain matrix. A solution is

38. Kalman Gain Matrix
As measurement becomes more reliable, K weights residual more heavily,
As prior covariance approaches 0, measurements are ignored:

40. Constant Velocity Model
position
These are lifts from figure The caption for that figure explains them
position
time
Constant Velocity Model

41. position
These are lifts from figure The caption for that figure explains them
time

42. This is figure 17. 3 of Forsyth and Ponce
This is figure 17.3 of Forsyth and Ponce. The notation is a bit involved, but is logical. We
plot the true state as open circles, measurements as x’s, predicted means as *’s
with three standard deviation bars, corrected means as +’s with three
standard deviation bars.
position
This is figure 17.3 of Forsyth and Ponce. The notation is a bit involved, but is logical. We
plot state as open circles, measurements as x’s, predicted means as *’s
with three standard deviation bars, corrected means as +’s with three
standard deviation bars.
time

43. The o-s give state, x-s measurement.
position
This is figure The notation is a bit involved, but is logical. We
plot state as open circles, measurements as x’s, predicted means as *’s
with three standard deviation bars, corrected means as +’s with three
standard deviation bars.
time
The o-s give state, x-s measurement.

44. Smoothing
Idea
We don’t have the best estimate of state - what about the future?
Run two filters, one moving forward, the other backward in time.
Now combine state estimates
The crucial point here is that we can obtain a smoothed estimate by viewing the backward filter’s prediction as yet another measurement for the forward filter

45. The o-s give state, x-s measurement. time
Forward estimates.
position
Forward filter, using notation as above; figure is top left of The error bars are 1 standard deviation, not 3 (sorry!)
The o-s give state, x-s measurement.
time

46. The o-s give state, x-s measurement. time
Backward estimates.
position
Backward filter, top right of 17.5, The error bars are 1 standard deviation, not 3 (sorry!)
The o-s give state, x-s measurement.
time

47. Combined forward-backward estimates.
position
Smoothed estimate, bottom of Notice how good the state estimates are, despite the very
lively noise. The error bars are 1 standard deviation, not 3 (sorry!)
The o-s give state, x-s measurement.
time

48. 2-D constant velocity example from Kevin Murphy’s Matlab toolbox
[figure from

49. 2-D constant velocity example from Kevin Murphy’s Matlab toolbox
MSE of filtered estimate is 4.9; of smoothed estimate. 3.2.
Not only is the smoothed estimate better, but we know that it is better, as illustrated by the smaller uncertainty ellipses
Note how the smoothed ellipses are larger at the ends, because these points have seen less data.
Also, note how rapidly the filtered ellipses reach their steady-state (“Ricatti”) values.
[figure from

50. Resources Kalman filter homepage http://www.cs.unc.edu/~welch/kalman/
(kalman filter demo applet)
Kevin Murphy’s Matlab toolbox:

51. Embellishments for tracking
Richer models of P(xn|xn-1)
Richer models of P(yn|xn)
Richer model of probability distributions

52. Abrupt changes What if environment is sometimes unpredictable?
Do people move with constant velocity?
Test several models of assumed dynamics, use the best.

53. Multiple model filters
Test several models of assumed dynamics
[figure from Welsh and Bishop 2001]

54. MM estimate Two models: Position (P), Position+Velocity (PV)
[figure from Welsh and Bishop 2001]

55. P likelihood
[figure from Welsh and Bishop 2001]

56. No lag
[figure from Welsh and Bishop 2001]

57. Smooth when still
[figure from Welsh and Bishop 2001]

58. Embellishments for tracking
Richer models of P(yn|xn)
Richer model of probability distributions

59. Jepson, Fleet, and El-Maraghi tracker

60. Wandering, Stable, and Lost appearance model
Introduce 3 competing models to explain the appearance of the tracked region:
A stable model—Gaussian with some mean and covariance.
A 2-frame motion tracker appearance model, to rebuild the stable model when it gets lost
An outlier model—uniform probability over all appearances.
Use an on-line EM algorithm to fit the (changing) model parameters to the recent appearance data.

61. Jepson, Fleet, and El-Maraghi tracker for toy 1-pixel image model
Red line: observations
Blue line: true appearance state
Black line: mean of the stable process

62. Mixing probabilities for Stable (black), Wandering (red) and Lost (green) processes

63. Non-toy image representation
Phase of a steerable quadrature pair (G2, H2). Steered to 4 different orientations, at 2 scales.

64. The motion tracker
Motion prior, , prefers slow velocities and small accelerations.
The WSL appearance model gives a likelihood for each possible new position, orientation, and scale of the tracked region.
They combine that with the motion prior to find the most probable position, orientation, and scale of the tracked region in the next frame.
Gives state-of-the-art tracking results.

65. Jepson, Fleet, and El-Maraghi tracker

66. Add fleet&jepson tracking slides
Jepson, Fleet, and El-Maraghi tracker
Add fleet&jepson tracking slides
Far right column: the stable component’s mixing probability. Note its behavior at occlusions.

67. Show videos
Play freeman#results.mpg

68. Embellishments for tracking
Richer model of P(yn|xn)
Richer model of probability distributions
Particle filter models, applied to tracking humans

69. (KF) Distribution propagation
prediction from previous time frame
Noise added to that prediction
Make new measurement at next time frame
[Isard 1998]

70. Distribution propagation
[Isard 1998]

71. Representing non-linear Distributions

72. Representing non-linear Distributions
Unimodal parametric models fail to capture real-world densities…

73. Discretize by evenly sampling over the entire state space
Tractable for 1-d problems like stereo, but not for high-dimensional problems.

74. Representing Distributions using Weighted Samples
Rather than a parametric form, use a set of samples to represent a density:

75. Representing Distributions using Weighted Samples
Rather than a parametric form, use a set of samples to represent a density:
Sample positions
Probability mass at each sample
This gives us two knobs to adjust when representing a probability density by samples: the locations of the samples, and the probability weight on each sample.

76. Representing distributions using weighted samples, another picture
[Isard 1998]

77. Sampled representation of a probability distribution
You can also think of this as a sum of dirac delta functions, each of weight w:

78. Tracking, in particle filter representationx1 x2 x3 y1 y2 y3
Prediction step
Update step

79. Particle filter
Let’s apply this sampled probability density machinery to generalize the Kalman filtering framework.
More general probability density representation than uni-modal Gaussian.
Allows for general state dynamics, f(x) + noise

80. Sampled Prediction
= ? ~=
Drop elements to marginalize to get

81. Sampled Correction (Bayes rule)
Prior  posterior
Reweight every sample with the likelihood of the observations, given that sample:
yielding a set of samples describing the probability distribution after the correction (update) step:

82. Naïve PF Tracking Start with samples from something simple (Gaussian)
Repeat
Take each particle from the prediction step and modify the old weight by multiplying by the new likelihood
Correct
Predict s
Run every particle through the dynamics function and add noise.
But doesn’t work that well because of sample impoverishment…

83. Sample impoverishment
10 of the 100 particles, along with the true Kalman filter track, with variance:
time

84. Resample the prior
In a sampled density representation, the frequency of samples can be traded off against weight:
These new samples are a representation of the same density.
I.e., make N draws with replacement from the original set of samples, using the weights as the probability of drawing a sample.
s.t. …

85. Resampling concentrates samples

86. A practical particle filter with resampling

87. Pictorial view
[Isard 1998]

90. Animation of condensation algorithm
[Isard 1998]

91. Applications Tracking hands bodies Leaves What might we expect?
Reliable, robust, slow

92. Contour tracking
[Isard 1998]

93. Picture of the states represented by the top weighted particles
Head tracking
Picture of the states represented by the top weighted particles
The mean state
[Isard 1998]

94. Leaf tracking
[Isard 1998]

95. Hand tracking
[Isard 1998]

96. Go to hedvig slides

97. Desired operations with a probability density
Expected value
Marginalization
Bayes rule

98. Computing an expectation using sampled representation

99. Marginalizing a sampled density
If we have a sampled representation of a joint density
and we wish to marginalize over one variable:
we can simply ignore the corresponding components of the samples (!):

100. Marginalizing a sampled density

101. Sampled Bayes rule
k p(V=v0|U)p(U)