1. Computer and Robot Vision II
Chapter 15 Motion and Surface Structure from Time Varying Image Sequences
Presented by: 傅楸善 & 王林農
指導教授: 傅楸善 博士

2. 15.1 Introduction
Motion analysis involves estimating the relative motion of objects with respect to each other and the camera given two or more perspective projection images in a time sequence.
DC & CV Lab.
CSIE NTU

3. 15.1 Introduction (cont’) Real-world applications:
industrial automation and inspection, robot assembly, autonomous vehicle navigation, biomedical engineering, remote sensing, general 3D-scene understanding
DC & CV Lab.
CSIE NTU

4. 15.1 Introduction (cont’)
object motion and surface structure recovery from:
observed optic flow
point correspondences
DC & CV Lab.
CSIE NTU

5. 15.2 The Fundamental Optic Flow Equation
(x, y, z): 3D point on moving rigid body
(u, v): perspective projection on the image plane
f: camera constant
(u, v): velocity of the point (u, v) . .
DC & CV Lab.
CSIE NTU

6. 15.2 The Fundamental Optic Flow Equation (cont’)
take time derivatives of both sides
yields the fundamental optic flow equation: . . . . . . . . . . .
DC & CV Lab.
CSIE NTU

7. 15.2 The Fundamental Optic Flow Equation (cont’)
general solution: (λ is a free variable) . . . . .
DC & CV Lab.
CSIE NTU

8. 15.2.1 Translational Motion . . . . . Known: Unknown:
N-point optic flow field:
Unknown:
corresponding unknown 3D points:
all points moving with same but unknown velocity
(x, y, z)
can be solved up to a multiplicative constant . . . . .
DC & CV Lab.
CSIE NTU

9. 15.2.2 Focus of Expansion and Contraction
Known:
3D motion is translational
one 2D projected point (u, v) has no motion:
thus translational motion is in a direction along the ray of sight . .
DC & CV Lab.
CSIE NTU

10. 15.2.2 Focus of Expansion and Contraction (cont’)
focus of expansion (FOE): if 3D point field moving toward camera
FOE: motion-field vectors radiate outward from that point
focus of contraction (FOC): if 3D point field moving away from camera
FOC: vectors radiate inward toward diametrically opposite point flow pattern of the motion field of a forward-moving observer
DC & CV Lab.
CSIE NTU

11. DC & CV Lab.
CSIE NTU

12. 15.2.3 Moving Line Segment . . . Known:
fixed distance between two unknown 3D points
translational motion with common velocity
(x, y, z)
corresponding optic flow: . . .
DC & CV Lab.
CSIE NTU

13. 15.2.3 Moving Line Segment (cont’)
Unknown:
: two unknown 3D points
common velocity: (x, y, z) . . .
DC & CV Lab.
CSIE NTU

14. 15.2.3 Moving Line Segment (cont’)
From the perspective projection equations:
From the optic flow equation:
DC & CV Lab.
CSIE NTU

15. 15.2.3 Moving Line Segment (cont’)
From the known length of the line segment:
The optic flow equation (15.9) permits us to obtain a least squares solution for z in terms of z1 and z2, from
DC & CV Lab.
CSIE NTU

16. 15.2.3 Moving Line Segment (cont’)
We obtain
Substituting this back into the equation, we can solve z2 in terms of z1:z2=kz1
DC & CV Lab.
CSIE NTU

17. 15.2.3 Moving Line Segment (cont’)
Substitute the relations for (x1, y1, z1) from equations into Eq.(15.10) to obtain
Hence:
DC & CV Lab.
CSIE NTU

18. 15.2.4 Optic Flow Acceleration Invariant
Since
differentiating general solution in Sec 15.2 and solve for (x, y, z) .. .. .. . .. .. . .. . .. .. ..
DC & CV Lab.
CSIE NTU

19. joke
DC & CV Lab.
CSIE NTU

20. 15.3 Rigid-Body Motion
Rigid-body motion: no relative motion of points w.r.t. (with respect to) one another
Rigid-body motion: points maintain fixed position relative to one another
Rigid-body motion: all points move with the body as a whole
DC & CV Lab.
CSIE NTU

21. 15.3 Rigid-Body Motion (cont’)
R(t): rotation matrix
T(t): translation vector
p(0): initial position of given point
R(0)=I, T(0)=0
p(t): position of given point at time t
DC & CV Lab.
CSIE NTU

22. 15.3 Rigid-Body Motion (cont’)
Rigid-body motion in displacement vectors:
velocity vector: time derivative of its position: . . .
DC & CV Lab.
CSIE NTU

23. 15.3 Rigid-Body Motion (cont’)
Since
(a) translational-motion field under projection onto hemispherical surface only translational-component motion useful in determining scene structure
(b) rotational-motion field under projection onto hemispherical surface rotational-motion field provides no information about scene structure . . .
DC & CV Lab.
CSIE NTU

24. 15.3 Rigid-Body Motion (cont’)
DC & CV Lab.
CSIE NTU

25. 15.3 Rigid-Body Motion (cont’)
we can describe rigid-body motion in instantaneous velocity by .
DC & CV Lab.
CSIE NTU

26. 15.3 Rigid-Body Motion (cont’)
: angular velocities in three axes
: translational velocities in three axes
from rigid-body-motion equation . . .
DC & CV Lab.
CSIE NTU

27. 15.3 Rigid-Body Motion (cont’)
and perspective projection equation
we can determine an expression for z: . .
DC & CV Lab.
CSIE NTU

28. 15.3 Rigid-Body Motion (cont’)
after simplification . .
DC & CV Lab.
CSIE NTU

29. 15.3 Rigid-Body Motion (cont’)
image velocity: expressed as sum of translational field and rotational field
(x, y, z): 3D coordinate before rigid-body motion in displacement vectors
(x’, y’, z’): 3D coordinate after rigid-body motion in displacement vectors
: rotation angles in three axes
: translation in three axes
DC & CV Lab.
CSIE NTU

30. 15.3 Rigid-Body Motion (cont’)
Rigid-body motion in displacement vectors:
DC & CV Lab.
CSIE NTU

31. 15.3 Rigid-Body Motion (cont’)
motion in displacement vector and instantaneous velocity is different:
e.g. moon encircling earth
instantaneous velocity: first order approximation of displacement vector
first order approximation: when small,
DC & CV Lab.
CSIE NTU

32. 15.3 Rigid-Body Motion (cont’)
first order approximation: when time=1 thus
x=(x’ - x)/1
first order approximation: .
DC & CV Lab.
CSIE NTU

33. joke
DC & CV Lab.
CSIE NTU

34. 15.4 Linear Algorithms for Motion and Surface Structure from Optic Flow
The Planar Patch Case
: arbitrary object point on planar patch at time t
: central projective coordinates of p(t) onto image plane z= f
DC & CV Lab.
CSIE NTU

35. The Planar Patch Case . .
: instantaneous velocity of moving image point
: optic flow image point
: instantaneous rotational angular velocity
: instantaneous translational velocity . .
DC & CV Lab.
CSIE NTU

36. 15.4.1 The Planar Patch Case (cont’)
unit vector n(t): orthogonal to moving planar patch
rigid planar patch motion represented by rigid-motion constraint: .
DC & CV Lab.
CSIE NTU

37. 15.4.1 The Planar Patch Case (cont’)
from above two equations:
Let
Rigid-motion constraint could be written as
DC & CV Lab.
CSIE NTU

38. 15.4.1 The Planar Patch Case (cont’)
denote the 3 x 3 matrix by W and its three row vectors by
W: called planar motion parameter matrix
since skew symmetric
DC & CV Lab.
CSIE NTU

39. 15.4.1 The Planar Patch Case (cont’)
above equation can be written as
from perspective projection equations:
taking time derivatives of these equations we have . . . . . . . . .
DC & CV Lab.
CSIE NTU

40. 15.4.1 The Planar Patch Case (cont’)
substitute equations into above equations:
from third row
substitute z to obtain optical flow-planar motion equation . . . . . . . . .
DC & CV Lab.
CSIE NTU

41. 15.4.1 The Planar Patch Case (cont’)
we have 2N linear equations: n=1,…,N:
optic flow-planar motion recovery: first solve W then find . .
DC & CV Lab.
CSIE NTU

42. 15.4.2 General Case Optic Flow-Motion Equation
1. set up optic flow-motion equation not involving depth information
2. solve it by using linear least-squares technique
DC & CV Lab.
CSIE NTU

43. 15.4.3 A Linear Algorithm for Solving Optic FlowMotion Equations
DC & CV Lab.
CSIE NTU

44. 15.5.4 Mode of Motion, Direction of Translation, and Surface Structure
mode of motion: whether translation k=0 or not
direction of translation: direction of k
surface structure: relative depth when k 0
DC & CV Lab.
CSIE NTU

45. 15.4.5 Linear Optic Flow-Motion Algorithm and Simulation Results
motion and shape recovery algorithms should answer three questions:
minimum number of points to compute motion and shape
what set of optic flow points violate rank assumption e.g. collinearity…
What’s the accuracy of estimated motion from noisy optic flow?
DC & CV Lab.
CSIE NTU

46. joke
DC & CV Lab.
CSIE NTU

47. 15.5 The Two View-Linear Motion Algorithm
DC & CV Lab.
CSIE NTU

48. 15.5.1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review
Two View-Planar Motion Equation
imaging geometry for two view-planar motion
rigid planar patch in motion in half-space z< 0
DC & CV Lab.
CSIE NTU

49. DC & CV Lab.
CSIE NTU

50. 15.5.1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review (cont’)
: arbitrary object point before motion
: same object point after motion
: central projective coordinates of
f : camera constant
DC & CV Lab.
CSIE NTU

51. 15.5.1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review (cont’)
R0: 3 X 3 rotational matrix, R0’R0=I,|R0|=1
t0: 3 X 1 translational vector
n0: 3 X 1 normal vector
DC & CV Lab.
CSIE NTU

52. 15.5.1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review (cont’)
Rigid-body-motion equation relates p1 to p2 as follows:
planarity constrains p1 by
combining two equations produces planar rigid-body-motion-equation
DC & CV Lab.
CSIE NTU

53. 15.5.1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review (cont’)
Projecting the planar rigid-body motion onto the image plane z = f produces
Let the planar rigid –motion parameter matrix be defined by
Where bi, i=1,2,3, are three row vectors of B.
DC & CV Lab.
CSIE NTU

54. 15.5.1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review (cont’)
Then above equation could be written as
DC & CV Lab.
CSIE NTU

55. 15.5.1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review (cont’)
From above we derive the two view-planar motion equation
With the natural constraint
DC & CV Lab.
CSIE NTU

56. 15.5.1 Planar Patch Motion Recovery from Two Perspective Views: A Brief Review (cont’)
Now the planar motion recovery problem involves first solving the planar rigid-motion parameter matrix B and then estimating
DC & CV Lab.
CSIE NTU

57. 15.5.2 General Curved Patch Motion Recovery from Two Perspective Views A Simplified Linear Algorithm
discard planar patch assumption, consider general curved patch
DC & CV Lab.
CSIE NTU

58. 15.5.3 Determining Translational Orientation
DC & CV Lab.
CSIE NTU

59. 15.5.4 Determining Mode of Motion and Relative Depths
DC & CV Lab.
CSIE NTU

60. 15.5.5 A Simplified Two View-Motion Linear Algorithm
DC & CV Lab.
CSIE NTU

61. 15.5.6 Discussion and Summary
when no noise appears: algorithm extremely accurate
when small noise appears: it works well except mode of motion incorrect
DC & CV Lab.
CSIE NTU

62. 15.6 Linear Algorithm for Motion and Structure from Three Orthographic Views
Ullman (1979) showed that for the orthographic case four-point correspondences over three views are sufficient to determine the motion and structure of the four-point rigid configuration
DC & CV Lab.
CSIE NTU

63. Shimon Ullman, The Interpretation of Visual Motion
The MIT Press, Cambridge MA. 1979
DC & CV Lab.
CSIE NTU

64. 15.6 Linear Algorithm for Motion and Structure from Three Orthographic Views
to infer depth information: translation needed in perspective projection
to infer depth information: rotation useless in perspective projection
to infer depth information: rotation needed in orthographic projection
to infer depth information translation useless in orthographic projection
DC & CV Lab.
CSIE NTU

65. DC & CV Lab.
CSIE NTU

66. Problem Formulation
image plane stationary three orthographic views at time
(x, y, z): object-space coordinates of point P at t1
(x’, y’, z’): object-space coordinates of point P at t2
(x”, y”, z”): object-space coordinates of point P at t3
(u, v): image-space coordinates of P at t1
(u’, v’): image-space coordinates of P at t2
(u”, v”): image-space coordinates of P at t3
DC & CV Lab.
CSIE NTU

67. 15.6.1 Problem Formulation (cont’)
: rotation matrix
: translation vector
(x’, y’, z’)’ = R(x’, y’, z’)+Tr
(x”, y”, z”)” = S(x”, y”, z”)+Ts
DC & CV Lab.
CSIE NTU

68. 15.6.1 Problem Formulation (cont’)
Known:
four image-point correspondences
Unkown:
DC & CV Lab.
CSIE NTU

69. 15.6.1 Problem Formulation (cont’)
note that with orthographic projections
therefore it is obvious that tr3, ts3 can never be determined
we are trying to determine:
DC & CV Lab.
CSIE NTU

70. Determining
DC & CV Lab.
CSIE NTU

71. 15.6.3 Solving a Unique Orthonormal Matrix R
DC & CV Lab.
CSIE NTU

72. 15.6.4 Linear Algorithm to Uniquely Solve R, s, a3
DC & CV Lab.
CSIE NTU

73. Summary
Given two orthographic views, one cannot finitely determine the motion and structure of a rigid body, no matter how many point correspondences are used, as shown by Huang.
DC & CV Lab.
CSIE NTU

74. Joke
DC & CV Lab.
CSIE NTU

75. 15.7 Developing a Highly Robust Estimator for General Regression
DC & CV Lab.
CSIE NTU

76. 15.7.1 Inability of the Classical Robust M-Estimator to Render High Robustness
Classical robust estimator, such as M-, L-, or R-estimator:
1. optimal or nearly optimal at assumed noise distribution
2. relatively small performance degradation with small number of outliers
3. larger deviations from assumed distribution do not cause catastrophe
MF-estimator with new property much stronger than property 3
relatively small performance degradation with larger deviations from assumed distribution
DC & CV Lab.
CSIE NTU

77. 15.7.2 Partially Modeling Log Likelihood Function by Using Heuristics
MF-estimator: robust regression more appropriate model-fitting
DC & CV Lab.
CSIE NTU

78. 15.7.3 Discussion M-, L-, R and MF-estimator: all residual based
DC & CV Lab.
CSIE NTU

79. MF-Estimator
MF-estimator: combine Bayes statistical decision rule with heuristics
DC & CV Lab.
CSIE NTU

80. 15.8 Optic Flow-Instantaneous Rigid-Motion Segmentation and Estimation
formulate optic flow-single rigid-motion estimation into general regression
DC & CV Lab.
CSIE NTU

81. Single Rigid Motion
P(t): position vector of an object point at the time t
[X(t), Y(t)]: central projective coordinate of P(t)
DC & CV Lab.
CSIE NTU

82. 15.8.1 Single Rigid Motion (cont’)
DC & CV Lab.
CSIE NTU

83. 15.8.1 Single Rigid Motion (cont’)
: noisy optic flow image point
Instantaneous representation of the rigid motion is described by
Instantaneous rotational angular velocity of rigid motion:
Instantaneous translational angular velocity of rigid motion:
DC & CV Lab.
CSIE NTU

84. 15.8.1 Single Rigid Motion (cont’)
Differentiating above equation
where for simplicity the time variable t has been omitted
DC & CV Lab.
CSIE NTU

85. 15.8.1 Single Rigid Motion (cont’)
Combine above two equations:
DC & CV Lab.
CSIE NTU

86. 15.8.2 Multiple Rigid Motions
We turn the optic flow-multiple rigid –motion segmentation and estimation problem into a number of successive optic flow-single rigid-motion estimation problems.
DC & CV Lab.
CSIE NTU

87. joke
DC & CV Lab.
CSIE NTU

88. 15.9 Experimental Protocol
Simulate simplest location estimation
Simulate Optic flow-rigid-motion segmentation and estimation
DC & CV Lab.
CSIE NTU

89. 15.9 Experimental Protocol (cont’)
DC & CV Lab.
CSIE NTU

90. 15.9 Experimental Protocol (cont’)
DC & CV Lab.
CSIE NTU

91. 15.9 Experimental Protocol (cont’)
DC & CV Lab.
CSIE NTU

92. 15.10 Motion and Surface Structure from Line Correspondences
Discussion concerns only the general rigid motion of straight-line structure
DC & CV Lab.
CSIE NTU

93. Problem Formulation
Cartesian reference system-central projection
DC & CV Lab.
CSIE NTU

94. 15.10.1 Problem Formulation (cont’)
DC & CV Lab.
CSIE NTU

95. 15.10.1 Problem Formulation (cont’)
l: line in 3D space
L: projection of the line on image plane z = f
z = f : image frame
: known plane line L is in; projective plane of l
: set of lines in 3D space
: lines moved by rigid motion
(R’ , T’)’ at time t’
(R” , T”)” at time t”
DC & CV Lab.
CSIE NTU

96. 15.10.1 Problem Formulation (cont’)
: projections of lines
; respective projective planes
DC & CV Lab.
CSIE NTU

97. 15.10.1 Problem Formulation (cont’)
Known:
K triples of line correspondences in three views
Unkown:
rotations and translations:
3D lines
DC & CV Lab.
CSIE NTU

98. 15.10.2 Solving Rotation Matrices R’, R” and Translations T’,R”
: the normals of the ith projective planes
Then
DC & CV Lab.
CSIE NTU

99. 15.10.2 Solving Rotation Matrices R’, R” and Translations T’,R”
Above two equations define a unique three-dimensional line solution if and only if
DC & CV Lab.
CSIE NTU

100. 15.10.3 Solving Three-Dimensional Line Structure
Once rotation martrices R’, R” and translations T’, T” are solved, each three-dimensional line li can be determined
DC & CV Lab.
CSIE NTU

101. 15.11 Multiple Rigid Motions from Two Perspective Views
Problem Statement
imaging geometry for two-view-motion
DC & CV Lab.
CSIE NTU

102. DC & CV Lab.
CSIE NTU

103. Problem Statement
How many good point correspondences are needed in order to apply the nonlinear least-squares estimator?
DC & CV Lab.
CSIE NTU

104. 15.11.2 Simulated Experiments
DC & CV Lab.
CSIE NTU

105. 15.11.2 Simulated Experiments (cont’)
DC & CV Lab.
CSIE NTU

106. 15.11.2 Simulated Experiments (cont’)
DC & CV Lab.
CSIE NTU

107. 15.12 Rigid Motion from Three Orthographic Views
DC & CV Lab.
CSIE NTU

108. 15.12.1 Problem Formulation and Algorithm
same as Sec. 15.6, instead of linear algorithms, formulate model-fitting problem
DC & CV Lab.
CSIE NTU

109. 15.12.2 Simulated Experiments
DC & CV Lab.
CSIE NTU

110. 15.12.2 Simulated Experiments (cont’)
DC & CV Lab.
CSIE NTU

111. 15.12.2 Simulated Experiments (cont’)
DC & CV Lab.
CSIE NTU

112. 15.12.3 Further Research on the MF-Estimator
two problems to be solved for MF-estimator to be practically useful:
distance problem
requirement for a good initial approximation
DC & CV Lab.
CSIE NTU

113. difficulty of motion and shape recovery: ambiguity of displacement field
Fuh. Ph.D. Thesis, Fig 4.1
DC & CV Lab.
CSIE NTU

114. DC & CV Lab.
CSIE NTU

115. 15.13 Literature Review 15.13.1 Inferring Motion and Surface Structure
DC & CV Lab.
CSIE NTU

116. 15.13.1 Inferring Motion and Surface Structure
classifications for methods of inferring 3D motion and shape
use of individual sets of feature points
use of local optic flow information about a single point
use of the entire optic flow field
DC & CV Lab.
CSIE NTU

117. 15.13.1 Inferring Motion and Surface Structure
Despite all the results obtained over the years, almost none of these inference techniques have been successfully applied to feature-point correspondences calculated from real imagery
DC & CV Lab.
CSIE NTU

118. Joke
DC & CV Lab.
CSIE NTU

119. 15.13.2 Computing Optic Flow or Image-Point Correspondences
problem source contains abundant information
occlusion boundaries
specular points
near a focus of expansion
noise and digitization effects in image formation
DC & CV Lab.
CSIE NTU

120. 15.13.2 Computing Optic Flow or Image-Point Correspondences (cont’)
motion parallax: apparent relative motion between objects and observer
points in observer’s direction of translation remain relatively unchanged information available to a moving observer
DC & CV Lab.
CSIE NTU

121. DC & CV Lab.
CSIE NTU

122. 15.13.2 Computing Optic Flow or Image-Point Correspondences (cont’)
impart time dimension to image data
spatiotemporal image data block
DC & CV Lab.
CSIE NTU

123. DC & CV Lab.
CSIE NTU

124. 15.13.2 Computing Optic Flow or Image-Point Correspondences (cont’)
motion field: assignment of vectors to image points representing motion
angular velocity of fixed scene: inversely proportional to distance pilot in straight-ahead level flight on an overcast day
DC & CV Lab.
CSIE NTU

125. DC & CV Lab.
CSIE NTU

126. DC & CV Lab.
CSIE NTU

127. 15.13.2 Computing Optic Flow or Image-Point Correspondences (cont’)
motion field of pilot looking straight ahead in motion direction
zero image velocity: at approach point and at infinity (along horizon)
DC & CV Lab.
CSIE NTU

128. DC & CV Lab.
CSIE NTU

129. 15.13.2 Computing Optic Flow or Image-Point Correspondences (cont’)
motion field of pilot looking to the right in level flight
focus of expansion here: at infinity to the left
focus of contraction here: at infinity to the right of the figure
DC & CV Lab.
CSIE NTU

130. DC & CV Lab.
CSIE NTU

131. 15.13.2 Computing Optic Flow or Image-Point Correspondences (cont’)
spatiotemporal image data acquired by a camera,- caption -
straight streaks at block top due to translating parallel to image plane
DC & CV Lab.
CSIE NTU

132. DC & CV Lab.
CSIE NTU

133. joke
DC & CV Lab.
CSIE NTU

134. B.K.P. Horn, Robot Vision, The MIT Press, Cambridge, MA, 1986
Chapter 12 Motion Field & Optical Flow
optic flow: apparent motion of brightness patterns during relative motion
DC & CV Lab.
CSIE NTU

135. 12.1 Motion Field
motion field: assigns velocity vector to each point in the image
Po: some point on the surface of an object
Pi: corresponding point in the image
vo: object point velocity relative to camera
vi: motion in corresponding image point
DC & CV Lab.
CSIE NTU

136. 12.1 Motion Field (cont’)
ri: distance between perspectivity center and image point
ro: distance between perspectivity center and object point
f’: camera constant
z: depth axis, optic axis
object point displacement causes corresponding image point displacement
DC & CV Lab.
CSIE NTU

137. 12.1 Motion Field (cont’)
DC & CV Lab.
CSIE NTU

138. 12.1 Motion Field (cont’) Velocities: where ro and ri are related by
DC & CV Lab.
CSIE NTU

139. 12.1 Motion Field (cont’)
differentiation of this perspective projection equation yields
DC & CV Lab.
CSIE NTU

140. joke
DC & CV Lab.
CSIE NTU

141. 12.2 Optical Flow
optical flow need not always correspond to the motion field
(a) perfectly uniform sphere rotating under constant illumination:
no optical flow, yet nonzero motion field
(b) fixed sphere illuminated by moving light source:
nonzero optical flow, yet zero motion field
DC & CV Lab.
CSIE NTU

142. DC & CV Lab.
CSIE NTU

143. 12.2 Optical Flow (cont’)
not easy to decide which P’ on contour C’ corresponds to P on C
DC & CV Lab.
CSIE NTU

144. DC & CV Lab.
CSIE NTU

145. 12.2 Optical Flow (cont’)
optical flow: not uniquely determined by local information in changing
irradiance at time t at image point (x, y)
components of optical flow vector
DC & CV Lab.
CSIE NTU

146. 12.2 Optical Flow (cont’) assumption: irradiance the same at time
fact: motion field continuous almost everywhere
DC & CV Lab.
CSIE NTU

147. 12.2 Optical Flow (cont’) expand above equation in Taylor series
e: second- and higher-order terms in
cancelling E( x, y, t), dividing through by
DC & CV Lab.
CSIE NTU

148. 12.2 Optical Flow (cont’)
which is actually just the expansion of the equation
abbreviations:
DC & CV Lab.
CSIE NTU

149. 12.2 Optical Flow (cont’) we obtain optical flow constraint equation:
flow velocity (u, v): lies along straight line perpendicular to intensity gradient
DC & CV Lab.
CSIE NTU

150. DC & CV Lab.
CSIE NTU

151. 12.2 Optical Flow (cont’) rewrite constraint equation:
aperture problem: cannot determine optical flow along isobrightness contour
DC & CV Lab.
CSIE NTU

152. 12.3 Smoothness of the Optical Flow
motion field: usually varies smoothly in most parts of image
try to minimize a measure of departure from smoothness
DC & CV Lab.
CSIE NTU

153. 12.3 Smoothness of the Optical Flow (cont’)
error in optical flow constraint equation should be small
overall, to minimize
DC & CV Lab.
CSIE NTU

154. 12.3 Smoothness of the Optical Flow (cont’)
large if brightness measurements are accurate
small if brightness measurements are noisy
DC & CV Lab.
CSIE NTU

155. 12.4 Filling in Optical Flow Information
regions of uniform brightness: optical flow velocity cannot be found locally
brightness corners: reliable information is available
DC & CV Lab.
CSIE NTU

156. 12.5 Boundary Conditions
Well-posed problem: solution exists and is unique
partial differential equation: infinite number of solution unless with boundary
DC & CV Lab.
CSIE NTU

157. joke
DC & CV Lab.
CSIE NTU

158. 12.6 The Discrete Case
first partial derivatives of u, v: can be estimated using difference
DC & CV Lab.
CSIE NTU

159. DC & CV Lab.
CSIE NTU

160. 12.6 The Discrete Case (cont’)
measure of departure from smoothness:
error in optical flow constraint equation:
to seek set of values that minimize
DC & CV Lab.
CSIE NTU

161. 12.6 The Discrete Case (cont’)
dieffrentiating e with respect to
DC & CV Lab.
CSIE NTU

162. 12.6 The Discrete Case (cont’)
where are local average of u, v (9 neighbors? )
extremum occurs where the above derivatives of e are zero:
DC & CV Lab.
CSIE NTU

163. 12.6 The Discrete Case (cont’)
determinant of 2x2 coefficient matrix:
so that
DC & CV Lab.
CSIE NTU

164. 12.6 The Discrete Case (cont’)
suggests iterative scheme such as
new value of (u, v): average of surrounding values minus adjustment
DC & CV Lab.
CSIE NTU

165. DC & CV Lab.
CSIE NTU

166. 12.6 The Discrete Case (cont’)
first derivatives estimated using first differences in 2x2x2 cube
DC & CV Lab.
CSIE NTU

167. DC & CV Lab.
CSIE NTU

168. 12.6 The Discrete Case (cont’)
consistent estimates of three first partial derivatives:
DC & CV Lab.
CSIE NTU

169. 12.6 The Discrete Case (cont’)
four successive synthetic images of rotating sphere
DC & CV Lab.
CSIE NTU

170. DC & CV Lab.
CSIE NTU

171. 12.6 The Discrete Case (cont’)
estimated optical flow after 1, 4, 16, and 64 iterations
DC & CV Lab.
CSIE NTU

172. DC & CV Lab.
CSIE NTU

173. 12.6 The Discrete Case (cont’)
(a) estimated optical flow after several more iterations
(b) computed motion field
DC & CV Lab.
CSIE NTU

174. DC & CV Lab.
CSIE NTU

175. 12.7 Discontinuities in Optical Flow
discontinuities in optical flow: on silhouettes where occlusion occurs
DC & CV Lab.
CSIE NTU

176. Joke
DC & CV Lab.
CSIE NTU

177. Project due May 2
implementing Horn & Schunck optical flow estimation as above
synthetically translate lena.im one pixel to the right and downward
Try
DC & CV Lab.
CSIE NTU