1. Motion from image and inertial measurements Dennis Strelow Carnegie Mellon University

2. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 2 Committee Dr. Sanjiv Singh (chair) Dr. Martial Hebert Dr. Michael Lewicki Dr. Rama Chellappa (University of Maryland)

3. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 3 Many thanks Jeff Mishler, Henele Adams, Tom Minka this talk: many others for helping with experiments or making experiments possible Henele Adams Dan Bartz Joseph Djugash Daniel Huber Bart Nabbe Sanjiv Singh Arne Suppe

4. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 4 On the web Related materials: these slides related papers movies VRML models at: http://www.cs.cmu.edu/~dstrelow/thesis-oral

5. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 5 Introduction (1) From an image sequence, we can recover: 6 degree of freedom (DOF) camera motion without knowledge of the camera’s surroundings without GPS

6. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 6 Introduction (2) Fitzgibbon

7. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 7 Introduction (3) Potential applications include: modeling from video Yuji Uchida

8. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 8 Introduction (4) micro air vehicles (MAVs) AeroVironment Black WidowAeroVironment Microbat

9. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 9 Introduction (5) Mars rover navigation Mars Exploration Rovers (MER) Hyperion

10. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 10 Introduction (6) search and rescue robots Rhex (movies: http://ai.eecs.umich.edu/Rhex/Movies)

11. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 11 Introduction (7) NASA Personal Satellite Assistant (PSA)

12. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 12 Introduction (8) For these problems, we want: 6 DOF motion in unknown environments without GPS or other absolute positioning

13. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 13 Introduction (8) For these problems, we want: 6 DOF motion in unknown environments without GPS or other absolute positioning using small, light, and cheap sensors

14. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 14 Introduction (8) For these problems, we want: 6 DOF motion in unknown environments without GPS or other absolute positioning using small, light, and cheap sensors over the long term

15. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 15 Introduction (9) Long-term motion estimation: distance or time is long MAVs: ½ hour endurance, 2 km range Hyperion: 1 km on 1 command only a small fraction of the scene is visible at any one time modeling from video search and rescue robots personal satellite assistant

16. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 16 Introduction (10) given these requirements, cameras are promising sensors… …and many algorithms for motion from images already exist

17. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 17 Introduction (11) But, where are the systems for estimating the motion of: over the long term?

18. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 18 Introduction (12) …and for automatically modeling rooms buildings cities from a handheld camera?

19. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 19 Introduction (13) Motion from images suffers from some long- standing difficulties This work attacks these problems by… exploiting image and inertial measurements robust image feature tracking recognizing previously mapped locations exploiting omnidirectional images

20. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 20 Outline Motion from image measurements refresher bundle adjustment difficulties Motion from image and inertial measurements Robust image feature tracking Long-term motion estimation Conclusion

21. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 21 Motion estimation from images: refresher (1) A two-step process is common: sparse feature tracking estimation

22. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 22 Motion estimation from images: refresher (1) A two-step process is common: sparse feature tracking estimation Sparse feature tracking: inputs: raw images outputs: projections

23. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 23 Motion estimation from images: refresher (2)

24. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 24 Motion estimation from images: refresher (3) Template matching: correlation tracking Lucas-Kanade (Lucas and Kanade, 1981) Extraction and matching: Harris features (Harris, 1992) Scale Invariant Feature Transform (SIFT) keypoints (Lowe, 2004)

25. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 25 Motion estimation from images: refresher (4) The second step is estimation: inputs: projections outputs: 6 DOF camera position at the time of each image 3D position of each tracked point

26. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 26 Motion estimation from images: refresher (5)

27. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 27 Motion estimation from images: refresher (6) bundle adjustment (various, 1950’s) Kalman filtering (Broida, Chandrashekhar, and Chellappa, 1990) variable state dimension filter (VSDF) (McLauchlan, 1996) two- and three-frame methods (Hartley and Zisserman, 2000; Nister, et al. 2004)

28. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 28 Motion estimation from images: refresher (7) batch bundle adjustment factorization (Tomasi and Kanade, 1992) recursive Kalman filtering variable state dimension filter (VSDF) sequential Monte Carlo (Qian and Chellappa, 2001) othertwo- and three-frame methods

29. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 29 Motion estimation from images: refresher (7) batch bundle adjustment factorization (Tomasi and Kanade, 1992) recursive Kalman filtering variable state dimension filter (VSDF) sequential Monte Carlo (Qian and Chellappa, 2001) othertwo- and three-frame methods the error that bundle adjustment minimizes is one leg of my image-and-inertial algorithm

30. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 30 Motion estimation from image: bundle adjustment (1) From tracking, we have the image locations x ij for each point j and each image i

31. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 31 Motion estimation from image: bundle adjustment (2) Suppose we also have estimates of: the camera rotation ρ i and translation t i at time of each image 3D point positions X j of each tracked point Then, we can compute reprojections:

32. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 32 Motion estimation from image: bundle adjustment (3)

33. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 33 Motion estimation from image: bundle adjustment (4)

34. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 34 Motion estimation from image: bundle adjustment (5) So, minimize: with respect to all the ρ i, t i, X j

35. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 35 Motion estimation from image: bundle adjustment (5) So, minimize: with respect to all the ρ i, t i, X j

36. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 36 Motion estimation from images: difficulties (1) Estimation step can be very sensitive to… incorrect or insufficient image feature tracking camera modeling and calibration errors outlier detection thresholds sequences with degenerate camera motions

37. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 37 Motion estimation from images: difficulties (2) Iterative batch methods have poor convergence or may fail to converge if: observations are missing the initial estimate is poor

38. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 38 Motion estimation from images: difficulties (3) Recursive methods suffer from: poor prior assumptions on the motion poor approximations in state error modeling

39. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 39 Motion estimation from images: difficulties (4) Resulting errors are: gross local errors long term drift

40. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 40 Motion estimation from images: difficulties (5)

41. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 41 Motion estimation from images: difficulties (6) 151 images, 23 points manually corrected Lucas-Kanade

42. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 42 Motion estimation from images: difficulties (7)  squares: ground truth points  dash-dotted line: accurate estimate  solid line: image-only, bundle adjustment estimate

43. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 43 Outline Motion from image measurements Motion from image and inertial measurements inertial sensors algorithms and results related work Robust image feature tracking Long-term motion estimation Conclusion

44. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 44 Motion from image and inertial measurements: inertial sensors (1) inertial sensors can be integrated to estimate six degree of freedom motion

45. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 45 Motion from image and inertial measurements: inertial sensors (2) But many applications require small, light, and cheap sensors

46. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 46 Motion from image and inertial measurements: inertial sensors (3) Integrating the outputs of these low grade sensors will produce drifting motion because of: noise unmodeled nonlinearities

47. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 47 Motion from image and inertial measurements: inertial sensors (4) And, we can’t even integrate until we can separate the effects of… rotation ρ gravity g acceleration a slowly changing bias b a noise n …in the accelerometer measurements

48. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 48 Motion from image and inertial measurements: inertial sensors (5) Image and inertial measurements are highly complementary With inertial measurements we can: decrease sensitivity in image-only estimates establish two rotation angles without drift establish the global scale

49. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 49 Motion from image and inertial measurements: inertial sensors (5) Image and inertial measurements are highly complementary With inertial measurements we can: decrease sensitivity in image-only estimates establish two rotation angles without drift establish the global scale …even with our low-grade sensors

50. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 50 Motion from image and inertial measurements: inertial sensors (6) With image measurements, we can: reduce the drift in integrating inertial data distinguish between… rotation ρ gravity g acceleration a bias b a noise n …in accelerometer measurements

51. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 51 Motion from image and inertial measurements: algorithms and results (1) This work has developed both: batch recursive algorithms for motion from image and inertial measurements

52. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 52 Motion from image and inertial measurements: algorithms and results (2) Gyro measurements: ω’, ω: measured and actual angular velocity b ω : gyro bias n: gaussian noise

53. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 53 Motion from image and inertial measurements: algorithms and results (3) Accelerometer measurements: ρ: rotation a’, a: measured and actual acceleration g: gravity vector b a : accelerometer bias n: gaussian noise

54. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 54 Motion from image and inertial measurements: algorithms and results (4) batch algorithm minimizes a combined error:

55. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 55 Motion from image and inertial measurements: algorithms and results (5) image term E image is the same as before

56. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 56 Motion from image and inertial measurements: algorithms and results (6) inertial error term E inertial is:

57. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 57 Motion from image and inertial measurements: algorithms and results (6) inertial error term E inertial is:

58. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 58 Motion from image and inertial measurements: algorithms and results (6) inertial error term E inertial is:

59. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 59 Motion from image and inertial measurements: algorithms and results (7) timeτ i-1 (time of image i - 1) t i-1 titi I(t i-1, …) τ i (time of image i) translation ( : translation estimate for image i – 1) ( : translation estimate for image i) ( : translation integrated from previous estimate)

60. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 60 Motion from image and inertial measurements: algorithms and results (8) time τ0τ0 translation τ1τ1 τ2τ2 τ5τ5 τ3τ3 τ4τ4 τ f-3 τ f-2 τ f-1

61. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 61 Motion from image and inertial measurements: algorithms and results (9)

62. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 62 Motion from image and inertial measurements: algorithms and results (10) I t (τ i-1, τ i,…, t i-1 ) depends on: τ i-1, τ i (known) all inertial measurements for times τ i-1 < τ < τ i (known) ρ i-1, t i-1 g b ω, b a camera linear velocities: v i

63. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 63 Motion from image and inertial measurements: algorithms and results (11) So minimize with respect to: ρ i-1, t i-1 X j g b ω, b a v i-1

64. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 64 Motion from image and inertial measurements: algorithms and results (12)  dash-dotted line: batch estimate from image and inertial  solid line: image-only, bundle adjustment estimate  squares: ground truth points

65. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 65 Motion from image and inertial measurements: algorithms and results (13) IEKF for the same sensors, unknowns  dash-dotted line: batch estimate  solid line: IEKF estimate

66. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 66 Motion from image and inertial measurements: algorithms and results (14) Difficulties with IEKF for this application: prior assumptions about motion smoothness cannot model relative error between adjacent camera positions So, converting the batch algorithm into a variable state dimension filter (VSDF) is a promising future direction

67. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 67 Motion from image and inertial measurements: algorithms and results (15) IEKF assumptions on motion smoothness  dash-dotted line: batch estimate  solid line: IEKF estimate  right: IEKF propagation variances too strict  left: IEKF propagation variances just right

68. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 68 Motion from image and inertial measurements: related work (1) simultaneous localization and mapping (SLAM)…typically: recovers planar motion, 2D scene structure from odometry, range and bearing sensors shape-from-motion (SFM): recovers 6 DOF motion, 3D scene structure from image measurements (bearing) only this work: recovers 6 DOF motion, 3D scene structure, etc. from image and inertial measurements

69. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 69 Motion from image and inertial measurements: related work (2) Motion from image and inertial measurements batch Deans and Hebert, 2001 Jung and Taylor, 2001 recursive Deans and Hebert, 2001 Huster and Rock, 2003 Qian, Chellappa, and Zhang, 2001 Foxlin, 2002 otherMukai and Ohnishi, 1999

70. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 70 Motion from image and inertial measurements Recap: image, gyro, and accelerometer measurements batch algorithm recursive algorithm experiments evaluate batch and recursive algorithms establish basic facts about motion from image and inertial measurements

71. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 71 Motion from image and inertial measurements See document for: recursive algorithm details many additional experiments more details on related work

72. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 72 Outline Motion from image measurements Motion from image and inertial measurements Robust image feature tracking Lucas-Kanade and real sequences The “smalls” tracker Long-term motion estimation Conclusion

73. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 73 Robust image feature tracking: Lucas- Kanade and real sequences (1) Combining image and inertial measurements improves our situation, but… we still need accurate feature tracking tracking some sequences do not come with inertial measurements

74. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 74 Robust image feature tracking: Lucas- Kanade and real sequences (2) better feature tracking for improved 6 DOF motion estimation remaining results will be image-only

75. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 75 Robust image feature tracking: Lucas- Kanade and real sequences (3) Lucas-Kanade has been the go-to feature tracker for shape-from-motion minimizes a correlation-like matching error using general minimization evaluates the matching error at only a few locations subpixel resolution

76. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 76 Robust image feature tracking: Lucas- Kanade and real sequences (4) Additional heuristics used to apply Lucas- Kanade to shape-from-motion: task:heuristic: choose features to trackhigh image texture identify mistracked, occluded, no-longer-visible convergence, matching error handle large motionsimage pyramid

77. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 77 Robust image feature tracking: Lucas- Kanade and real sequences (5) But Lucas-Kanade performs poorly on many real sequences…

78. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 78 Robust image feature tracking: the “smalls” tracker (1) smalls is a new feature tracker targeted at 6 DOF motion estimation exploits the rigid scene assumption eliminates the heuristics normally used with Lucas-Kanade SIFT is an enabling technology here

79. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 79 Robust image feature tracking: the “smalls” tracker (2) Leonard Smalls – tracker, manhunter

80. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 80 Robust image feature tracking: the “smalls” tracker (3) First step: epipolar geometry estimation use SIFT to establish matches between the two images get the 6 DOF camera motion between the two images get the epipolar geometry relating the two images

81. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 81 Robust image feature tracking: the “smalls” tracker (4)

82. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 82 Robust image feature tracking: the “smalls” tracker (5)

83. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 83 Robust image feature tracking: the “smalls” tracker (6) Second step: track along epipolar lines use nearby SIFT matches to get initial position on epipolar line exploits the rigid scene assumption eliminates heuristic: pyramid

84. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 84 Robust image feature tracking: the “smalls” tracker (7) Third step: prune features geometrically inconsistent features are marked as mistracked and removed clumped features are pruned eliminates heuristic: detecting mistracked features based on convergence, error

85. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 85 Robust image feature tracking: the “smalls” tracker (8) Fourth step: extract new features spatial image coverage is the main criterion required texture is minimal when tracking is restricted to the epipolar lines eliminates heuristic: extracting only textured features

86. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 86 Robust image feature tracking: the “smalls” tracker (9)

87. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 87 Robust image feature tracking: the “smalls” tracker (10) left: odometry onlyright: images only average error: 1.74 m maximum error: 5.14 m total distance: 230 m

88. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 88 Robust image feature tracking: the “smalls” tracker (11) Recap: exploiting rigid scene and eliminating heuristics allows hands-free tracking for real sequences can still be defeated by textureless areas or repetitive texture

89. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 89 Outline Motion from image measurements Motion from image and inertial measurements Robust image feature tracking Long-term motion estimation proof of concept system experiment Conclusion

90. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 90 Long-term motion estimation: proof of concept system (1) Image-based motion estimates from any system will drift: if the features we see are always changing given sufficient time if we don’t recognize when we’ve revisited a location

91. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 91 Long-term motion estimation: proof of concept system (2)

92. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 92 Long-term motion estimation: proof of concept system (3) To limit drift: recognize when we’ve returned to a previous location exploit the return A proof of concept system demonstrates these capabilities

93. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 93 Long-term motion estimation: proof of concept system (4) “smalls” tracker state: 2D feature history for images in I variable state dimension filter (VSDF) state for images in I: 6 DOF camera positions, covariances for images in I 3D positions for features visible in I SIFT keypoints for image i n system state S image indices: I = {i 1, …, i n }

94. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 94 Long-term motion estimation: proof of concept system (5) 012345678 {0, 1} {0}{0, 1, 2}{0, 1, …, 8}

95. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 95 Long-term motion estimation: proof of concept system (6) 012345678 {0, 1} {0}{0, 1, 2}{0, 1, …, 8} non-rollback rollback States:

96. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 96 Long-term motion estimation: proof of concept system (7) 012345678 {0, 1} {0}{0, 1, 2}{0, 1, …, 8} 8 non-rollback rollback States:

97. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 97 Long-term motion estimation: proof of concept system (8) 012345678 {0, 1} {0}{0, 1, 2} 8 {0, 1, 2, 3, 8} non-rollback rollback pruned States:

98. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 98 Long-term motion estimation: proof of concept system (9) 012345678 891011 121314 151617 181920 {0, …, 6, 11, 12, 17, …, 20} non-rollback rollback pruned States:

99. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 99 Long-term motion estimation: proof of concept system (10) When to “roll back”? examine the camera covariances for the current state and the candidate rollback state check the number of SIFT matches extend from the candidate state examine the camera covariances for the current state and the resulting extended state

100. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 100 Long-term motion estimation: experiment (1) CMU FRC highbay views; 945 images total

101. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 101 Long-term motion estimation: experiment (2) CMU FRC highbay

102. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 102 Long-term motion estimation: experiment (2) CMU FRC highbay (first forward pass)

103. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 103 Long-term motion estimation: experiment (2) CMU FRC highbay (first forward pass)

104. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 104 Long-term motion estimation: experiment (2) CMU FRC highbay (first forward pass)

105. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 105 Long-term motion estimation: experiment (2) CMU FRC highbay (first backward pass)

106. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 106 Long-term motion estimation: experiment (2) CMU FRC highbay (second forward pass)

107. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 107 Long-term motion estimation: experiment (2) CMU FRC highbay (second backward pass)

108. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 108 Long-term motion estimation: experiment (2) CMU FRC highbay (third forward pass)

109. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 109 Long-term motion estimation: experiment (2) CMU FRC highbay (third backward pass)

110. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 110 Long-term motion estimation: experiment (3) 012345678 891011 121314 151617 181920 non-rollback rollback pruned States: normally, the system produces a general tree of states

111. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 111 Long-term motion estimation: experiment (4) … 01234567 13141514 16171817 non-rollback rollback pruned States: for this example, the “rollback” states are restricted to the first forward pass 89 10111214 214

112. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 112 Long-term motion estimation: experiment (5) movie…bottom half is smalls output:

113. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 113 Long-term motion estimation: experiment (6) movie…top half is motion estimates:

114. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 114 Long-term motion estimation: experiment (7) movie…top half is motion estimates:

115. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 115 Outline Motion from image measurements Motion from image and inertial measurements Robust image feature tracking Long-term motion estimation Conclusion contributions remaining issues

116. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 116 Conclusion: contributions (1) Image and inertial batch algorithm fully exploits both modalities requires no prior knowledge of the unknowns requires no assumptions about the motion, e.g., smoothness tool for establishing basic facts about the problem

117. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 117 Conclusion: contributions (2) Image and inertial recursive algorithm retains most advantages of the batch algorithm suitable for long sequences accommodates missing observations in a principled way

118. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 118 Conclusion: contributions (3) Image and inertial experiments evaluation of specific algorithms, but also… fundamental facts about motion from image and inertial measurements

119. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 119 Conclusion: contributions (4) The “smalls” tracker exploits rigid scene eliminates the poor heuristics normally used with Lucas-Kanade allows robust, hands-free motion from real sequences

120. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 120 Conclusion: contributions (5) Proof of concept system smalls, VSDF, and SIFT for long-term motion

121. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 121 Conclusion: contributions (6) Omnidirectional vision noncentral catadioptric camera model motion from noncentral cameras precise catadioptric camera calibration

122. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 122 Conclusion: remaining issues all: system is experimental, not optimized for speed image and inertial: should use VSDF “smalls”: integration of gyro, more robustness to poor texture needed long-term: recorded states grow linearly with sequence

123. Dennis Strelow -- Motion estimation from image and inertial measurements -- November 8, 2004 123 Thanks! Related materials: these slides related papers movies VRML models at: http://www.cs.cmu.edu/~dstrelow/thesis-oral