1. Extending metric multidimensional scaling with Bregman divergences
Jigang Sun and Colin Fyfe

2. Visualising 18 dimensional data

3. Outline Bregman divergence. Multidimensional scaling(MDS).
Extending MDS with Bregman divergences.
Relating the Sammon mapping to mappings with Bregman divergences. Comparison of effects and explanation.
Conclusion

4. Strictly Convex function
Pictorially, the strictly convex function F(x) lies below segment connecting two points q and p.

5. Bregman Divergences is the Bregman divergence between x and y based on
convex function, φ.
Taylor Series expansion is

6. Bregman Divergences

7. Euclidean distance is a Bregman divergence

8. Kullback Leibler Divergence

9. Generalised Information Divergence
φ(z)=z log(z)

11. Other Divergences Itakura-Saito Divergence Mahalanobis distance
Logistic loss
Any convex function

12. Some Properties dφ(x,y)≥0, with equality iff x==y.
Not a metric since dφ(x,y)≠ dφ(y,x)
(Though d(x,y)=(dφ(x,y)+dφ(y,x)) is symmetric)
Convex in the first parameter.
Linear, dφ+aγ(x,y)= dφ(x,y) + a.dγ(x,y)

13. Multidimensional Scaling
Creates one latent point for each data point.
The latent space is often 2 dimensional.
Positions the latent points so that they best represent the data distances.
Two latent points are close if the two
corresponding data points are close.
Two latent points are distant if the two corresponding data points are distant.

14. Classical/Basic Metric MDS
We minimise the stress function
data space
Latent space

15. Sammon Mapping (1969)
Focuses on small distances: for the same error, the smaller distance is given bigger stress.

16. Possible Extensions
Bregman divergences in both data space and latent space
Or even

17. Metric MDs with Bregman divergence between distances
Euclidean distance on latents.
Any divergence on data
Itakura-Saito divergence between them:
(Sammon-like)
to minimise
divergence.

18. Moving the Latent Points
F1 for I.S. divergence, F2 for euclidean , F3 any divergence

19. The algae data set

20. The algae data set

21. Two representations The standard Bregman representation:
Concentrating on the residual errors:

22. Basic MDS is a special BMMDS
Base convex function is chosen as
And higher order derivatives are So
is derived as

23. Sammon Mapping
Select
Then

24. Example 2: Extended Sammon
Base convex function
This is equivalent to
The Sammon mapping is rewritten as

25. Sammon and Extended Sammon
The common term
The Sammon mapping is thus an approximation to the Extended Sammon mapping via the common term.
The Extended Sammon mapping will do more adjustments on the basis of the higher order terms.

26. An Experiment on Swiss roll data set

27. Distance preservation

28. Relative standard deviation

29. Relative standard deviation
On short distances, Sammon has smaller variance than BasicMDS, Extended Sammon has smaller variance than Sammon, i.e. control of small distances is enhanced.
Large distances are given more and more freedom in the same order as above.

30. LCMC: local continuity meta-criterion (L. Chen 2006)
A common measure assesses projection quality of different MDS methods.
In terms of neighbourhood preservation.
Value between 0 and 1, the higher the better.

31. Quality accessed by LCMC

32. Why Extended Sammon outperforms Sammon
Stress formation

33. Features of the base convex function
Recall that the base convex function for the Extended Sammon mapping is
Higher order derivatives are
Even orders are positive and odd ones are negative.

34. Stress comparison between Sammon and Extended Sammon

35. Stress configured by Sammon, calculated and mapped by Extended Sammon

36. Stress configured by Sammon, calculated and mapped by Extended Sammon
The Extended Sammon mapping calculates stress on the basis of the configuration found by the Sammon mapping.
For , the mean stresses calculated by the Extended Sammon are much higher than mapped by the Sammon mapping.
For , the calculated mean stresses are obviously lower than that of the Sammon mapping.
The Extended Sammon makes shorter mapped distance even more short, longer even more long.

37. Stress formation by items

38. Generalisation: from MDS to Bregman divergences
A group of MDS is generalised as
C is a normalisation scalar which is used for quantitative comparison purposes. It does not affect the mapping results.
Weight function for missing samples
The Basic MDS and the Sammon mapping belong to this group.

39. Generalisation: from MDS to Bregman divergences
If C=1, then set
Then the generalised MDS is the first term of BMMDS and BMMDS is an extension of MDS.
Recall that BMMDS is equivalent to

40. Criterion for base convex function selection
In order to focus on local distances and concentrate less on long distances, the base convex function must satisfy
Not all convex functions can be considered, such as F(x)=exp(x).
The 2nd order derivative is primarily considered. We wish it to be big for small distances and small for long distances. It represents the focusing power on local distances.

41. Two groups of Convex functions
The even order derivatives are positive, odd order ones are negative.
No 1 is that of the Extended Sammon mapping.

42. Focusing power

43. Different strategies for focusing power
Vertical axis is logarithm of 2nd order derivative.
These use different strategies for increasing focusing power.
In the first group, the second order derivatives are higher and higher for small distances and lower and lower for long distances.
In the second group, second order derivatives have limited maximum values for very small distances, but derivatives are drastically lower and lower for long distances when λ increases.

44. Two groups of Bregman divergences
Elastic scaling(Victor E McGee, 1966)

45. Experiment on Swiss roll: The FirstGroup

46. Experiment on Swiss roll: FirstGroup
For Extended Sammon, Itakura-Saito,
, local distances are mapped better and better, long distances are stretched such that unfolding trend is obvious.

47. Distances mapping : FirstGroup

48. Standard deviation : FirstGroup

49. LCMC measure : FirstGroup

50. Experiment on Swiss roll:SecondGroup

51. Distance mapping: SecondGroup

52. StandardDeviation: SecondGroup

53. LCMC: SecondGroup

54. OpenBox, Sammon and FirstGroup

55. SecondGroup on OpenBox

56. Distance mapping: two groups

57. LCMC: two groups

58. Standard deviation: two groups

59. Swiss roll distances distribution

60. OpenBox distances distribution

61. Swiss roll vs OpenBox Distances formation:
Swiss roll: proportion of longer distances is greater than that of the shorter distances.
OpenBox: Very large quantity of a set of medium distances, small distances take much of the rest.
Mapping results:
Swiss roll: Long distances are stretched and local distances are usually mapped shorter.
The OpenBox: the longest distances are not stretched obviously, perhaps even compressed. Small distances are mapped longer than original values in data space by some methods.
Conclusion: Tug of war between local and long distances. Trying to get the opportunities to be mapped to their original values in data space.

62. Left and right Bregman divergences
All of this is with left divergences – latent points are in left position in divergence, ...
We can show that right divergences produce extensions of curvilinear component analysis.
(Sun et al, ESANN2010)

63. Conclusion Applied Bregman divergences to multidimensional scaling.
Shown that basic MMDS is a special case and Sammon mapping approximates a BMMDS.
Improved upon both with 2 families of divergences.
Shown results on two artificial data sets.