1. Graph matching algorithms
Ilchae Jung

2. Object matching
Slide from “Linear solution to scale and rotation invariant object matching”, Hao Jiang, Stella X. Yu, CVPR 09

3. Non-parametric models
Many approaches
Deformation
Challenges
Non-parametric models
Parametric models
Icp = iterative closest points 를 주기적으로 찾는것
AAM/ASM = Active appearance model/ active shape model
Pictorial structures
Graph matching
RANSAC / ICP
ASM / AAM

4. Graph matching formulation
Slide from “Learning Graphs to Match”, Minsu Cho, Karteek Alahari, and Jean Ponce,ICCV 13

5. Graph matching formulation
Finding y maximizing score S (Integer quadratic programming) 𝑦 ∗ = arg max 𝑦 𝑆 𝐺, 𝐺 ′ ,𝑦 = arg max 𝑦 𝑦 𝑇 𝑊𝑦 𝑠.𝑡 𝑊 𝑖𝑎;𝑗𝑏 = 𝑆 𝑉 𝒂 𝑖 , 𝒂 𝑎 ′ 𝑖𝑓 𝑖=𝑗, 𝑎=𝑏 𝑆 𝐸 𝒂 𝑖𝑗 , 𝒂 𝑎𝑏 ′ 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
Slide from “Learning Graphs to Match”, Minsu Cho, Karteek Alahari, and Jean Ponce,ICCV 13

6. Problems Hardness of application -progressive graph matching
NP-hard problem -relaxed program
Loss of relaxation

7. papers
Robust feature matching with alternate hough and Inverted Hough Transform (CVPR 2013) - Hsin-Yi Chen, Yen-Yu Lin, Bing-Yu Chen
A Path Following Algorithm for the Graph Matching Problem (PAMI 09) -Mikhail Zaslavskiy, Francis Bach, Jean-Philippe Vert

8. Robust feature matching with alternate hough and Inverted Hough Transform (CVPR 2013) - Hsin-Yi Chen, Yen-Yu Lin, Bing-Yu Chen

9. framework - High precision with hough voting
Correspondence 단에서는 가까이 있는 점들은 homography가 비슷할 것이라는 가정을 한다.
그렇기 때문에 가까이 있는 점들과 source image에서
- High precision with hough voting
- High recall with inverted hough voting

10. Intialization
Hough space - make a homography , 𝐻 𝑖𝑖′ between two inter-frame points 𝑣 𝑖 ∈ 𝑉 𝑃 , 𝑣 𝑖 ′ ∈ 𝑉 𝑄 𝐻 𝑖𝑖′ =𝑇 𝑣 𝑖 ′ 𝑄 ∗𝑇 𝑣 𝑖 𝑃 −1
Initial correspondences - choose top-r matchings by appr. similarity - make a matching candidate set, 𝑀 𝑖 for points 𝑀 𝑖 = 𝑚 𝑖𝑘 = 𝑣 𝑖 𝑃 , 𝑣 𝑖𝑘 𝑄 , 𝐻 𝑖 𝑖 𝑘 𝑘=1 𝑟
𝑉 𝑃 =𝑘𝑒𝑦𝑝𝑜𝑖𝑛𝑡𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑖𝑚𝑎𝑔𝑒 𝑃 𝑉 𝑄 =𝑘𝑒𝑦𝑝𝑜𝑖𝑛𝑡𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑖𝑚𝑎𝑔𝑒 𝑄 𝑣 𝑖 = 𝑥 𝑖 =𝑐𝑒𝑛𝑡𝑒𝑟, 𝑅 𝑖 =𝑟𝑒𝑔𝑖𝑜𝑛 𝑇 𝑣 =𝑎𝑓𝑓𝑖𝑛𝑒 𝑡𝑟𝑎𝑛𝑠𝑓𝑜𝑟𝑚

11. correspondences 𝑀 𝑖 ={ 𝑚 𝑗 𝑗 ′ ∗ | 𝑣 𝑗 𝑃 ∈𝐺( 𝑣 𝑖 𝑃 )}
From 𝑀 𝑖 , choose a matching with top high prob. - Gather neighbors of 𝑀 𝑖 𝑅 𝑣 𝑖 𝑃 = 𝑣 𝑗 𝑃 ∈𝐺( 𝑣 𝑖 𝑃 ) 𝑀 𝑗
𝑠.𝑡 𝐺 𝑣 𝑖 𝑃 = 𝑣 𝑗 𝑃 𝜋 𝑣 𝑖 𝑃 ∩𝜋 𝑣 𝑗 𝑃 ≠∅ Calculate prob. By KDE
𝑚 𝑖 𝑖 ′ ∗ =𝑎𝑟𝑔 max 𝑚 𝑖𝑖′ ∈ 𝑀 𝑖 1 |𝑅( 𝑣 𝑖 𝑃 )| 𝑚∈𝑅( 𝑣 𝑖 𝑝 ) exp⁡(− 𝑑( 𝑚 𝑖 𝑖 ′ ,𝑚) 𝜎 ) Gather the result by set, 𝑀 𝑖
𝑀 𝑖 ={ 𝑚 𝑗 𝑗 ′ ∗ | 𝑣 𝑗 𝑃 ∈𝐺( 𝑣 𝑖 𝑃 )}
𝑉 𝑃 =𝑘𝑒𝑦𝑝𝑜𝑖𝑛𝑡𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑖𝑚𝑎𝑔𝑒 𝑃, 𝑉 𝑄 =𝑘𝑒𝑦𝑝𝑜𝑖𝑛𝑡𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑖𝑚𝑎𝑔𝑒 𝑄 𝑣 𝑖 = 𝑥 𝑖 =𝑐𝑒𝑛𝑡𝑒𝑟, 𝑅 𝑖 =𝑟𝑒𝑔𝑖𝑜𝑛 , 𝑇 𝑣 =𝑎𝑓𝑓𝑖𝑛𝑒 𝑡𝑟𝑎𝑛𝑠𝑓𝑜𝑟𝑚 𝑀 𝑖 = 𝑚 𝑖𝑘 = 𝑣 𝑖 𝑃 , 𝑣 𝑖𝑘 𝑄 , 𝐻 𝑖 𝑖 𝑘 𝑘=1 𝑟

12. Inverted hough transform
Revise the result because of Many false positive - Each 𝑣 𝑖 𝑃 add new kernel to 𝑀 𝑖 𝟏. 𝒓𝒆𝒑𝒓𝒆𝒔𝒆𝒏𝒕𝒂𝒕𝒊𝒗𝒆 𝒐𝒇 𝒏𝒆𝒊𝒈𝒉𝒃𝒐𝒓𝒔(𝒂𝒔𝒔𝒖𝒎𝒆 𝒊𝒕 𝒊𝒔 𝒓𝒆𝒂𝒍𝒍𝒚 𝒓𝒊𝒈𝒉𝒕) → 𝒎 𝒋𝒋′ =𝒂𝒓𝒈 𝐦𝐚𝐱 𝒎 𝒋 𝒋 ′ ∗ ∈ 𝑴 𝒊 𝒎∈ 𝑴 𝒊 𝒆𝒙𝒑⁡(− 𝒅( 𝒎 𝒋 𝒋 ′ ∗ ,𝒎) 𝝈 ) 𝟐. 𝑵𝒆𝒘 𝒌𝒆𝒓𝒏𝒆𝒍, 𝒎 𝒊𝒌 = 𝒗 𝒊 𝑷 , 𝒗 𝒌 𝑸 𝒊𝒔 𝒔𝒆𝒍𝒆𝒄𝒕𝒆𝒅 𝒃𝒚 𝒇𝒊𝒏𝒅 𝒎𝒐𝒔𝒕 𝒔𝒊𝒎𝒊𝒍𝒂𝒓 𝒌𝒆𝒚𝒑𝒐𝒊𝒏𝒕 𝒗 𝒌 𝑸 𝒊𝒏 𝒕𝒉𝒆 𝒊𝒎𝒂𝒈𝒆 𝑸 → 𝒗 𝒌 𝑸 =𝒂𝒓𝒈 𝒎𝒂𝒙 𝒗 𝒌 𝑸 ∈ 𝑽 𝑸 𝑺∩ 𝑺 𝒌 𝑸 𝑺∪ 𝑺 𝒌 𝑸 𝒔.𝒕 𝒗 𝒋 ′ 𝑸 = 𝒙 𝒋 ′ ,𝑺 𝟑. 𝑨𝒅𝒅 𝒎 𝒊𝒌 𝒕𝒐 𝑴 𝒊 여기에 수식을 입력하십시오.
𝑽 𝑷 =𝒌𝒆𝒚𝒑𝒐𝒊𝒏𝒕𝒔 𝒊𝒏 𝒕𝒉𝒆 𝒊𝒎𝒂𝒈𝒆 𝑷 𝑽 𝑸 =𝒌𝒆𝒚𝒑𝒐𝒊𝒏𝒕𝒔 𝒊𝒏 𝒕𝒉𝒆 𝒊𝒎𝒂𝒈𝒆 𝑸 𝒗 𝒊 = 𝒙 𝒊 =𝒄𝒆𝒏𝒕𝒆𝒓, 𝑹 𝒊 =𝒓𝒆𝒈𝒊𝒐𝒏 𝑻 𝒗 =𝒂𝒇𝒇𝒊𝒏𝒆 𝒕𝒓𝒂𝒏𝒔𝒇𝒐𝒓𝒎

13. Whole algorithm

14. Results

15. Results

16. A Path Following Algorithm for the Graph Matching Problem (PAMI 09) -Mikhail Zaslavskiy, Francis Bach, Jean-Philippe Vert

17. Double-Stochastic Approximation
Previous Works
Not Convex
(K is Indefinite)
Double-Stochastic Approximation
Chui & Rangarajan, 2003
Cho et al, 2010
Leordeanu et al, 2009
Not Discrete
Gradient Method
is More Accurate
A. Rangarajan
M. Cho
K. Lee
M. Leordeanu
M. Hebert
R. Sukthankar

18. Hardness of graph matching
QAP(quadratic assignment problem) : NP-hard
->Relaxed problem : gradient algorithm with doubly stochastic matrix
Not Convex
(K is Indefinite)
Not Discrete
Outperforming rounding algorithm(continous -> discrete) with two equivalent convex/concave relaxation

19. Framework 𝑷→𝑫 Original QAP Convex minimization Concave minimization
Relaxed Convex minimization
Relaxed Concave minimization
𝑷→𝑫
𝑿∈𝑷
𝑿∈𝑫
Path following algorithm
Permutatation matrix space 상에서 convex와 concav는 똑같은 minimization term을 가진다.
또한 alpha=1 에서 permutation matri로 가는 것도 명확한 것이 concav의 minimum은 0,1 즉 constraint의 boundary에서 나타나므로 가능하다.
𝒑 ∗ 𝟎 =𝒂𝒓𝒈𝒎𝒊𝒏 𝑭 𝟎 (𝒑)
𝑭 𝝀 𝒑 =𝝀 𝑭 𝟏 𝒑 + 𝟏−𝝀 𝑭 𝟎 (𝒑) 𝝀 1
𝒑 ∗ 𝝀 =𝒍𝒐𝒄𝒂𝒍𝒎𝒊𝒏 𝑭 𝝀 (𝒑)
𝒑 ∗ 𝟏 =𝒍𝒐𝒄𝒂𝒍𝒎𝒊𝒏 𝑭 𝝀 (𝒑) ⋱
𝑿∈𝑷
𝑷=𝒔𝒑𝒂𝒄𝒆 𝒐𝒇 𝒑𝒆𝒓𝒎𝒂𝒕𝒂𝒕𝒆 𝒎𝒂𝒕𝒓𝒊𝒙
𝑫=𝒔𝒑𝒂𝒄𝒆 𝒐𝒇 𝒅𝒐𝒖𝒃𝒍𝒚 𝒔𝒕𝒐𝒄𝒉𝒂𝒔𝒕𝒊𝒄 𝒎𝒂𝒕𝒓𝒊𝒙
𝑭 𝟎 𝒑 =𝒄𝒐𝒏𝒗𝒆𝒙 , 𝑭 𝟏 𝒑 =𝒄𝒐𝒏𝒄𝒂𝒗𝒆, 𝒑∈𝑷, 𝑷=𝒑𝒆𝒓𝒎𝒖𝒕𝒂𝒕𝒊𝒐𝒏 𝒎𝒂𝒕𝒓𝒊𝒙 𝒔𝒑𝒂𝒄𝒆

20. Example : path following algorithm
<𝒍𝒐𝒄𝒂𝒍 𝒎𝒊𝒏𝒊𝒎𝒂 𝒔𝒆𝒂𝒓𝒄𝒉 𝑭𝒓𝒂𝒏𝒌−𝑾𝒐𝒍𝒇𝒆 >
1. 𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒 𝑡ℎ𝑒 𝑔𝑟𝑎𝑑𝑖𝑒𝑛𝑡 𝛻 𝐹 𝜆 𝑝 𝑛 2. 𝑟𝑒𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛 𝑜𝑓 𝑡ℎ𝑒 𝑙𝑖𝑛𝑒𝑎𝑟 𝑝𝑟𝑜𝑔𝑟𝑎𝑚 𝑝 𝑛 ∗ =𝑎𝑟𝑔 min 𝑝∈𝐷 <𝛻 𝐹 𝜆 𝑝 𝑛 ,𝑃> 3. 𝑓𝑖𝑛𝑑𝑖𝑛𝑔 𝑡ℎ𝑒 𝑚𝑖𝑛𝑖𝑚𝑢𝑚 𝑜𝑓 𝐹 𝜆 𝑝 𝑜𝑛 𝑡ℎ𝑒 [ 𝑝 𝑛 𝑝 𝑛 ∗ ]

21. Experiment
Objective :

22. Experiment
Objective :

23. Experiment: generated graph
1. Fixing the degree of nodes by Prob.(node degree=k)=VD(k)

24. Experiment: generated graphs
Noise는 기본 graph에 얼마나 많은 noised edge를 더하는 지에 대한 척도

25. Experiment: generated graphs
-time complexity
Noise는 기본 graph에 얼마나 많은 noised edge를 더하는 지에 대한 척도

26. Discussion Needs for progressive graph matching
Limitation of relaxation program
High time complexity