2. Stanford CS223B Computer Vision, Winter 2007 Lecture 12 Tracking Motion Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens

3. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

4. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Tracking Problem n Can we estimate the position of the object? n Can we estimate its velocity? n Can we predict future positions? Image 4Image 1Image 2Image 3

5. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Tracking Problem n Given Sequence of Images n Find center of moving object n Camera might be moving or stationary n We assume: We can find object in individual images. n The Problem: Track across multiple images. n Is a fundamental problem in computer vision

6. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Methods Bayes Filter Particle Filter Uncented Kalman Filter Kalman Filter Extended Kalman Filter

7. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Further Reading…

8. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Example: Moving Object

9. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Tracking

10. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filter Tracking

11. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Mixture of KF / PF (Unscented PF)

12. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

13. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Slow Down! Example of Bayesian Inference ? Environment prior p(staircase) = 0.1 Bayesian inference p(staircase | image) p(image | staircasse) p(staircase) p(im | stair) p(stair) + p(im | no stair) p(no stair) = 0.7  0.1 / (0.7  0.1 + 0.2  0.9) = 0.28 Sensor model p(image | staircase) = 0.7 p(image | no staircase) = 0.2 p(staircase) = 0.28 Cost model cost(fast walk | staircase) = $1,000 cost(fast walk | no staircase) = $0 cost(slow+sense) = $1 Decision Theory E[cost(fast walk)] = $1,000  0.28 = $280 E[cost(slow+sense)] = $1 =

14. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filter Definition n Environment state x t n Measurement z t n Can we calculate p(x t | z 1, z 2, …, z t ) ?

15. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters Illustrated

16. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters: Essential Steps n Belief: Bel(x t ) n Measurement update: Bel(x t )  Bel(x t ) p(z t |x t ) n Time update: Bel(x t+1 )  Bel(x t )  p(x t+1 |u t,x t ) 

17. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 x = state t = time z = observation u = action  = constant Bayes Filters Markov Bayes Markov

18. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters x = state t = time z = observation u = action

19. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters Illustrated

20. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters n Initial Estimate of State n Iterate –Receive measurement, update your belief (uncertainty shrinks) –Predict, update your belief (uncertainty grows)

21. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Methods Bayes Filter Particle Filter Uncented Kalman Filter Kalman Filter Extended Kalman Filter

22. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

23. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filters: Basic Idea (equality for ) set of n particles X t

24. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filter Explained

25. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Basic Particle Filter Algorithm Initialization: X 0  n particles x 0 [i] ~ p(x 0 ) particleFilters( X t  1 ){ for i=1 to n x t [i] ~ p(x t | x t  1 [i] ) (prediction) w t [i] = p(z t | x t [i] ) (importance weights) endfor for i=1 to n include x t [i] in X t with probability  w t [i] (resampling) }

26. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Weight samples: w = f / g Importance Sampling

27. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filter By Frank Dellaert

28. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Case Study: Track moving objects from Helicopter 1. Harris Corners 2. Optical Flow (with clustering) 3. Motion likelihood function 4. Particle Filter 5. Centroid Extraction David Stavens, Andrew Lookingbill, David Lieb, CS223b Winter 2004

29. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 1. Harris Corner Extraction David Stavens, Andrew Lookingbill, David Lieb, CS223b Winter 2004

30. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 2. Optical Flow + Motion Detection David Stavens, Andrew Lookingbill, David Lieb, CS223b Winter 2004

31. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 3. Motion Likelihood Function David Stavens, Andrew Lookingbill, David Lieb, CS223b Winter 2004

32. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 4. Particle Filters David Stavens, Andrew Lookingbill, David Lieb, CS223b Winter 2004

33. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 5. Extract Centroid David Stavens, Andrew Lookingbill, David Lieb, CS223b Winter 2004

34. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 More Particle Filter Tracking David Stavens, Andrew Lookingbill, David Lieb, CS223b Winter 2004

35. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Some Robotics Examples n Tracking Hands, People n Mobile Robot localization n People localization n Car localization n Mapping

36. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Examples Particle Filter Siu Chi Chan McGill University

37. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Another Example Mike Isard and Andrew Blake

38. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Tracking Fast moving Objects K. Toyama, A.Blake

39. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filters: Illustration With: Wolfram Burgard, Dieter Fox, Frank Dellaert

40. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filters (1)

41. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filters (2)

42. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particles = Robustness

43. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Tracking People from Moving Platform  robot location (particles)  people location (particles)  laser measurements (wall) With Michael Montemerlo

44. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Tracking People from Moving Platform With Michael Montemerlo  robot location (particles)  people location (particles)  laser measurements (wall)

45. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filters for Tracking Cars With Anya Petrovskaya

46. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Mapping Environments (SFM) With Dirk Haehnel

47. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

48. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Tracking with KFs: Gaussians! updateinitial estimte x y x y prediction x y measurement x y

49. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filters

50. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filters prior Measurement evidence posterior

51. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 A Quiz prior Measurement evidence posterior?

52. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Gaussians --   Univariate  Multivariate

53. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Properties of Univariate Gaussians

54. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Measurement Update Derived

55. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 n We stay in the “Gaussian world” as long as we start with Gaussians and perform only linear transformations. Properties Multivariate Gaussians Essentially the same as in the 1-D case, but with more general notation

56. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Linear Kalman Filter Estimates the state x of a discrete-time controlled process that is governed by the linear stochastic difference equation with a measurement

57. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Components of a Kalman Filter Matrix (n  n) that describes how the state evolves from t to t+1 without controls or noise. Matrix (n  i) that describes how the control u t changes the state from t to t+1. Matrix (k  n) that describes how to map the state x t to an observation z t. Random variables representing the process and measurement noise that are assumed to be independent and normally distributed with covariance R t and Q t respectively.

58. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Algorithm 1. Algorithm Kalman_filter(  t-1,  t-1, u t, z t ): 2. Prediction: 3. 4. 5. Correction: 6. 7. 8. 9. Return  t,  t

59. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Updates in 1D old belief measurement belief new belief belief measurement

60. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Updates in 1D old belief new belief measurement

61. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Updates in 1D old belief new belief measurement new belief newest belief

62. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Updates belief latest belief belief measurement belief

63. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Prediction-Correction-Cycle Prediction

64. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Prediction-Correction-Cycle Correction

65. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Prediction-Correction-Cycle Correction Prediction

66. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Summary n Highly efficient: Polynomial in measurement dimensionality k and state dimensionality n: O(k 2.376 + n 2 ) n Optimal for linear Gaussian systems! n Most robotics systems are nonlinear!

67. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

68. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Let’s Apply KFs to Tracking Problem Image 4Image 1Image 2Image 3

69. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter with 2-Dim Linear Model n Linear Change (Motion) n Linear Measurement Model What is A, B, Q? What is C, D, R?

70. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Algorithm 1. Algorithm Kalman_filter(  t-1,  t-1, u t, z t ): 2. Prediction: 3. 4. 5. Correction: 6. 7. 8. 9. Return  t,  t

71. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter in Detail n Measurements n Change n Prediction n Measurement Update updateinitial position x y x y prediction x y measurement x y

72. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Can We Do Better?

73. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman, Better! initial positionpredictionmeasurement next prediciton update

74. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 We Can Estimate Velocity! past measurements prediction

75. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter For 2D Tracking n Linear Measurement model (now with 4 state variables) n Linear Change

76. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Putting It Together Again n Measurements n Change n Prediction n Measurement Update

77. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary Kalman Filter n Estimates state of a system –Position –Velocity –Many other continuous state variables possible n KF maintains –Mean vector for the state –Covariance matrix of state uncertainty n Implements –Time update = prediction –Measurement update n Standard Kalman filter is linear-Gaussian –Linear system dynamics, linear sensor model –Additive Gaussian noise (independent) –Nonlinear extensions: extended KF, unscented KF: linearize n More info: –CS226 –Probabilistic Robotics (Thrun/Burgard/Fox, MIT Press)

78. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary Kalman Filter n Estimates state of a system –Position –Velocity –Many other continuous state variables possible n KF maintains –Mean vector for the state –Covariance matrix of state uncertainty n Implements –Time update = prediction –Measurement update n Standard Kalman filter is linear-Gaussian –Linear system dynamics, linear sensor model –Additive Gaussian noise (independent) –Nonlinear extensions: extended KF, unscented KF: linearize n More info: –CS226 –Probabilistic Robotics (Thrun/Burgard/Fox, MIT Press) Particle and discrete set of particles (example states) = predictive sampling = resampling, importance weights fully nonlinear easy to implement

79. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters