1. Multidimensional scaling
© Alexander & Michael Bronstein,
© Michael Bronstein, 2010
tosca.cs.technion.ac.il/book
Advanced topics in vision
Processing and Analysis of Geometric Shapes
EE Technion, Spring 2010 1

2. If it doesn’t fit, you must acquit!
Image: Associated Press

3. Metric model Shape = metric space Similarity = isometry w.r.t.
EXTRINSIC GEOMETRY
Euclidean metric
Invariant to rigid motion
Extrinsic geometry is not invariant under inelastic deformations

4. Similarity = isometry w.r.t.
Metric model
Shape = metric space
Similarity = isometry w.r.t.
EXTRINSIC GEOMETRY
INTRINSIC GEOMETRY
Euclidean metric
Invariant to rigid motion
Geodesic metric
Invariant to inelastic deformation

5. in a common metric space
Intrinsic vs. extrinsic similarity
EXTRINSIC SIMILARITY
INTRINSIC SIMILARITY
Part of the same metric space
Two different metric spaces
SOLUTION: Find a representation of and
in a common metric space

6. Canonical forms
Isometric embedding

7. ? Canonical form distance INTRINSIC SIMILARITY Compute canonical forms
EXTRINSIC SIMILARITY OF CANONICAL FORMS
= INTRINSIC SIMILARITY

8. Mapmaker’s problem ?

9. Mapmaker’s problem
A sphere has non-zero curvature, therefore, it is not isometric to the plane
(a consequence of Theorema egregium)
Karl Friedrich Gauss
( )

10. Conclusion: generally, isometric embedding does not exist!
Linial’s example A A 1 1 C D 2 A C D 1 B C A 1 1 C D B B 2 C D D 1 1 B
Conclusion: generally, isometric embedding does not exist!
No contradiction to Nash embedding theorem: Nash guarantees that any Riemannian structure can be realized as a length metric induced by Euclidean metric. We try to realize it using restricted Euclidean metric

11. Minimum distortion embedding

12. Multidimensional scaling
Find an embedding into that distorts the distances the least by solving the optimization problem
where are the coordinates of the canonical form
The function measuring the distortion of distances is called stress
The problem is called multidimensional scaling (MDS)

13. Stress
The stress is a function of the canonical form coordinates and the distances
L2-stress
Lp-stress
L-stress (distortion)

14. Matrix expression of L2-stress
Some notation:
- an matrix of canonical form coordinates (each row
corresponds to a point)
Shorthand notation for Euclidean distances
Write the stress as 1 2

15. Term 1

16. Term 1 Matrices commute under trace operator
where is and matrix with elements

17. Term 2
For and
zero otherwise

18. Term 2
where is and matrix-valued function with elements

19. LS-MDS Minimization of L2-stress is called least squares MDS (LS-MDS)
variables
Non-linear
Non-convex (thus liable to local convergence)
Optimum defined up to Euclidean transformation

20. Gradient of L2-stress Quadratic in Nonlinear in Constant in
Recall exercise
in optimization

21. Complexity Stress computation complexity:
Stress gradient computation complexity:
Steepest descent
with line search may be prohibitively expensive (requires multiple
evaluations of the stress and the gradient)

22. Observation By Cauchy-Schwarz inequality for all and
Equality is achieved for
Write it differently:

23. Observation
Consider the nonlinear term in the stress

24. Majorizing inequality
Equality is achieved for
[de Leeuw, 1977]
We have a quadratic majorizing function to use in iterative majorization algorithm!
Jan de Leeuw

25. Iterative majorization
Construct a majorizing function satisfying .
Majorizing inequality: for all
is convex

26. Iterative majorization
Start with some
Find such that
Update iterate
Increment iteration counter
Solution
Until
convergence

27. Analytic expression for
Iterative majorization
The majorizing function
is quadratic!
Analytic expression for the minimim
Analytic expression for
pseudoinverse:

28. SMACOF algorithm Start with some Update iterate
Increment iteration counter
Solution
Until
convergence
The algorithm is called SMACOF (Scaling by Minimizing A COnvex Function)

29. SMACOF algorithm vs steepest descent
SMACOF is a constant-step gradient descent!
Majorization guarantees monotonically decreasing sequence of stress
values (untypical behavior for constant-step steepest descent)
No guarantee of global convergence
Iteration cost:

30. MATLAB® intermezzo
SMACOF algorithm
Canonical forms

31. Near-isometric deformations of a shape
Examples of canonical forms
Near-isometric deformations of a shape
Canonical forms

32. Visualization of intrinsic similarity
Examples of canonical forms
Visualization of intrinsic similarity

33. Weighted stress where are some fixed weights Matrix expression
where is and matrix with elements
SMACOF algorithm can be used to minimize weighted stress!

34. Variations on the stress theme
Generic form of the stress
where is some norm
For example, gives the Lp-stress
The necessary condition for to be the minimizer of is

35. Variations on the stress theme
Idea: minimize weighted stress instead of the generic stress and choose the weights such that the two are equivalent
If the weights are selected as
the minimizers of and will coincide
Since is unknown in advance, iterate!

36. Iteratively reweighted LS
Start with some and
Find
using as an initialization
Update weights
Increment iteration counter
Until
convergence
The algorithm is called Iteratively Reweighted Least Squares (IRLS)

37. L-stress Optimization problem
can be written equivalently as a constrained problem
is called an artificial variable

38. Complexity, bis MDS N=1000 points x5 less points x25 lower complexity

39. Multiresolution MDS
Bottom-up approach: solve coarse level MDS problem to initialize fine
level problem
Reduce complexity (less fine-level iterations)
Reduce the risk of local convergence (good initialization)
Can be performed on multiple resolution levels

40. Solve coarse level problem
Two-resolution MDS
Grid
Data
Interpolation operator to transfer solution from
coarse level to fine level
Can be represented as an sparse matrix
Solve fine
level problem
Interpolate
Solve coarse level problem

41. Solve -st level problem Solve coarsest level problem
Multiresolution MDS
Solve finest
level problem
Interpolate
Solve st level problem
Interpolate
Solve coarsest level problem

42. Multiresolution MDS algorithm
Hierarchy of grids
Hierarchy of data
Interpolation operators
Start with coarsest-level initialization
Solve the l-th level MDS problem
using as initialization
Interpolate to next level
For
Solve finest level problem
using as initialization

43. Top-down approach Start with a fine-level initialization
Decimate fine-level initialization to the coarse grid
Solve the coarse-level MDS problem
using as initialization
Improve the fine-level solution propagating the coarse-level error
Typically

44. A problem
Fine level
Coarse level

45. Correction The fine- and the coarse-level minimizers do not coincide!
is called a residual
Force consistency of the coarse- and fine-level problems by introducing a correction term into the stress
which gives the gradient

46. Correction
Fine level
Coarse level

47. Correction In order to guarantee consistency, must satisfy at
Chain rule
which gives the correction term
Verify: is the minimizer of coarse-level problem with correction,

48. Modified stress Another problem: the coarse grid problem
Can grow arbitrarily
is unbounded for
Fix the translation ambiguity by adding a quadratic penalty to the stress
is called the modified stress [Bronstein et al., 05]
The modified coarse grid problem is bounded

49. Solve corrected coarse level problem
Two-grid MDS
Iteratively improve the fine-level solution using coarse grid residual
External iteration is called cycle
Correct fine
level solution
Decimate
Interpolate
Solve corrected coarse level problem

50. Solve corrected coarse level problem
Two-grid MDS
Perform a few optimization iterations at fine level between cycles
(relaxation)
Correct fine
level solution
Relax
Decimate
Interpolate
Solve corrected coarse level problem

51. Two-grid MDS algorithm
Start with fine-level initialization
Produce an improved fine-level solution by
making a few iterations on initialized with
Decimate fine-level solution
Correction:
Solve the coarse-level corrected MDS problem
using as initialization
Transfer residual:
Update iterate
Solution
Until
convergence

52. Solve coarsest level problem
Multigrid MDS
Apply the two-grid algorithm recursively
Different cycles possible, e.g. V-cycle
Relax
Relax
Decimate
Interpolate
correct
Relax
Relax
Decimate
Interpolate
correct
Solve coarsest level problem

53. MATLAB® intermezzo
Multigrid MDS

54. Convergence of SMACOF and Multigrid MDS (N=2145)
Convergence example
Stress
Time (sec)
Convergence of SMACOF and Multigrid MDS (N=2145)

55. How to accelerate convergence?
Let be a sequence of iterates produced by some optimization algorithms converging to
Denote by the remainder
Construct a new sequence converging faster to
in the sense

56. Vector extrapolation
Construct the new sequence as a transformation of iterates
Particular choice: linear combination
Question: how to choose the coefficients ?
Ideally, hence

57. Reduced rank extrapolation (RRE)
Problem: depends on unknown
Eliminate this dependence by considering the difference
which ideally should vanish.
Result: solve the constrained linear system
of equations in variables
Typically hence the system is over-determined and must be solved approximately using least-squares

58. Optimization with vector extrapolation
Start with some and
Generate iterates
using optimization on initialized by
Extrapolate from
Update
Increment iteration counter
Until
convergence

59. Safeguarded optimization with vector extrapolation
Start with some and
Generate iterates
using optimization on initialized by
Extrapolate from If
else
Increment iteration counter
Until
convergence

60. MATLAB® intermezzo
RRE MDS