1 Long-term image-based motion estimation Dennis Strelow and Sanjiv Singh

2 On the web Related materials: these slides related papers movies VRML models at:

3 Introduction (1) micro air vehicle (MAV) navigation AeroVironment Black WidowAeroVironment Microbat

4 Introduction (2) mars rover navigation Mars Exploration Rovers (MER)Hyperion

5 Introduction (3) robotic search and rescue Rhex Center for Robot-Assisted Search and Rescue, U. of South Florida

6 Introduction (4) NASA ISS personal satellite assistant

7 Introduction (5) Each of these problems requires: 6 DOF motion in unknown environments without GPS or other absolute positioning over the long term …and some of the problems require: small, light, and cheap sensors

8 Introduction (6) Monocular, image-based motion estimation is a good candidate In particular, simultaneous estimation of: multiframe motion sparse scene structure is the most promising approach

9 Outline Image-based motion estimation Improving image-based motion estimation Improving feature tracking Reacquisition

10 Outline Image-based motion estimation refresher difficulties Improving image-based motion estimation Improving feature tracking Reacquisition

11 Image-based motion estimation: refresher (1) A two-step process is typical… First, sparse feature tracking: Inputs: raw images Outputs: projections

12 Image-based motion estimation: refresher (2)

13 Image-based motion estimation: refresher (3) Second, estimation: Input: Outputs:  6 DOF camera position at the time of each image  3D position of each tracked point  projections from tracker

14 Image-based motion estimation: refresher (4)

15 Image-based motion estimation: refresher (5) Algorithms exist For tracking: Lucas-Kanade (Lucas and Kanade, 1981)

16 Image-based motion estimation: refresher (6) For estimation: SVD-based factorization (Tomasi and Kanade, 1992) bundle adjustment (various, 1950’s) Kalman filtering (Broida and Chellappa, 1990) variable state dimension filter (McLauchlan, 1996)

17 Image-based motion estimation: difficulties (1) So, the problem is solved?

18 Image-based motion estimation: difficulties (2) If so, where are the automatic systems for estimating the motion of: in unknown environments? from images in unknown environments?

19 Image-based motion estimation: difficulties (3) …and for automatically modeling rooms buildings cities from a handheld camera?

20 Image-based motion estimation: difficulties (4) Estimation step can be very sensitive to: incorrect or insufficient image feature tracking camera modeling and calibration errors outlier detection thresholds sequences with degenerate camera motions

21 Image-based motion estimation: difficulties (5) …and for recursive methods in particular: poor prior assumptions on the motion poor approximations in state error modeling

22 Image-based motion estimation: difficulties (6) 151 images, 23 points

23 Image-based motion estimation: difficulties (7)

24 Image-based motion estimation: difficulties (8) For long-term motion estimation, these errors accumulate

25 Outline Image-based motion estimation Improving image-based motion estimation overview image and inertial measurements Improving feature tracking Reacquisition

26 Improving image-based motion estimation: overview

27 Improving image-based motion estimation: overview

28 Improving image-based motion estimation: image and inertial (1) Image and inertial measurements are highly complimentary Inertial measurements can: resolve the ambiguities in image-only estimates establish the global scale

29 Improving image-based motion estimation: image and inertial (2) Images measurements can: reduce the drift in integrating inertial measurements distinguish between rotation, gravity, acceleration, bias, noise in accelerometer readings

30 Improving image-based motion estimation: image and inertial (3)

31 Improving image-based motion estimation: image and inertial (4)

32 Improving image-based motion estimation: image and inertial (5) Other examples: global scale typically within 5% better convergence than image-only estimation

33 Improving image-based motion estimation: image and inertial (6) Many more details in: Dennis Strelow and Sanjiv Singh. Motion estimation from image and inertial measurements. IJRR, September 2004.

34 Outline Image-based motion estimation Improving image-based motion estimation Improving feature tracking Lucas-Kanade and real sequences The “smalls” tracker Reacquisition

35 Improving feature tracking: Lucas- Kanade and real sequences (1) Lucas-Kanade is the “go to” sparse feature tracker: iterative minimization of the intensity matching error function applied at several image resolutions to handle large motions features extracted based on image texture feature death based on iteration convergence and correlation error

36 Improving feature tracking: Lucas- Kanade and real sequences (2) Advantages: fast subpixel resolution can handle some large motions well uses general minimization, so easily extendible

37 Improving feature tracking: Lucas- Kanade and real sequences (3) 0.1 average pixel reprojection error!

38 Improving feature tracking: Lucas- Kanade and real sequences (4) But, Lucas-Kanade has some flaws: does not exploit the rigid scene poor heuristics for:  large motions  extracting features  detecting feature mistracking

39 Improving feature tracking: Lucas- Kanade and real sequences (5)

40 Improving feature tracking: Lucas- Kanade and real sequences (6)

41 Improving feature tracking: Lucas- Kanade and real sequences (7)

42 Improving feature tracking: Lucas- Kanade and real sequences (7)

43 Improving feature tracking: Lucas- Kanade and real sequences (8)

44 Improving feature tracking: the “smalls” tracker (1) smalls is a new sparse image feature tracker designed to address these issues i.e., designed for long-term motion estimation

45 Improving feature tracking: the “smalls” tracker (2) Leonard Smalls: tracker, lone biker of the apocalypse

46 Improving feature tracking: the “smalls” tracker (3) epipolar geometry 1-D correlation matching along epipolar lines geometric mistracking detection feature death and birth outputto 6 DOF SIFT featuresestimation features

47 Improving feature tracking: the “smalls” tracker (4) SIFT keypoints (Lowe, IJCV 2004): image interest points can be extracted despite of large changes in viewpoint to subpixel accuracy A keypoint’s feature vectors in two images usually match

48 Improving feature tracking: the “smalls” tracker (5) Epipolar geometry between adjacent images is determined using… SIFT extraction and matching two-frame bundle adjustment RANSAC epipolar geometry SIFT features

49 Improving feature tracking: the “smalls” tracker (6) initial search position from nearby SIFT matches discrete SSD search (e.g.,  60 pixels) 1-D Lucas-Kanade refines the match 1-D correlation matching along epipolar lines

50 Improving feature tracking: the “smalls” tracker (7) To check for mistracking, use only three- frame geometric consistency… geometric mistracking detection …determined using: three-frame bundle adjustment RANSAC

51 Improving feature tracking: the “smalls” tracker (8) After tracking in each image: features are pruned to maintain a minimum separation new features are selected in those parts of the image not already covered feature death and birth outputto 6 DOF featuresestimation

52 Improving feature tracking: the “smalls” tracker (9)

53 Improving feature tracking: the “smalls” tracker (10)

54 Improving feature tracking: the “smalls” tracker (11)

55 Improving feature tracking: the “smalls” tracker (12)

56 Outline Image-based motion estimation Improving image-based motion estimation Improving feature tracking Reacquisition

57 Reacquisition (1) Closing the loop to deal with drift: (1) recognizing revisited features (2) exploiting revisited features in the estimation

58 Future work (2)

59 Reacquisition (3) Image 149 close to… …image 95? Yes, but closer to image 84.

60 Reacquisition (4) …image 84? Yes, but closer to image 83. Image 149 close to…

61 Reacquisition (5) …image 83? Yes, and 83 is closest. Image 149 close to…

62 Thanks! Related materials: these slides related papers movies VRML models at: