1. Object Matching Using a Locally Affine Invariant and Linear Programming Techniques - H. Li, X. Huang, L. He
Ilchae Jung

2. Object matching
Slide from “Linear solution to scale and rotation invariant object matching”, Hao Jiang, Stella X. Yu, CVPR 09

3. Problem formulation
Find the matching function 𝒎 ∙ maximizing the objective
𝑚 ∙ = arg min 𝑚(∙) 𝑖=1 𝑛 𝑡 {𝑐( 𝑝 𝑖 , 𝑚 𝑝 𝑖 +𝜆∙𝑔( 𝑝 𝑖 , 𝑁 𝑝 𝑖 ;𝑚 𝑝 𝑖 , 𝑁 𝑚( 𝑝 𝑖 ) )}
𝑛 𝑡 =𝑡ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡𝑠 𝑛 𝑠 =𝑡ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠𝑐𝑒𝑛𝑒 𝑝𝑜𝑖𝑛𝑡𝑠
𝑝 𝑖 = 𝑥 𝑖 , 𝑦 𝑖 𝑇 𝜖 𝑅 2 = 𝑖 𝑡ℎ 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡 𝑞 𝑗 𝜖 𝑅 2 = 𝑗 𝑡ℎ 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡
𝑆𝜖 𝑅 𝑛 𝑆 ×2 =𝑠𝑒𝑡 𝑜𝑓 𝑠𝑐𝑒𝑛𝑒 𝑝𝑜𝑖𝑛𝑡𝑠 𝑇𝜖 𝑅 𝑛 𝑡 ×2 =𝑠𝑒𝑡 𝑜𝑓 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡𝑠
𝑁 𝑝 𝑖 =𝑡ℎ𝑒 𝑠𝑒𝑡 𝑜𝑓 𝑜𝑟𝑑𝑒𝑟𝑒𝑑 𝑝𝑜𝑖𝑛𝑡𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟ℎ𝑜𝑜𝑑 𝑜𝑓 𝑝 𝑖 𝑚 𝑝 𝑖 =𝑚𝑎𝑡𝑐ℎ𝑖𝑛𝑔 𝑓𝑢𝑛𝑐𝑡𝑢𝑖𝑜𝑛

4. Problem formulation Photometric similarity 𝑝 𝑖 𝑚(𝑝 𝑖 )
𝑖=1 𝑛 𝑡 {𝑐( 𝑝 𝑖 , 𝑚 𝑝 𝑖 +𝜆∙𝑔( 𝑝 𝑖 , 𝑁 𝑝 𝑖 ;𝑚 𝑝 𝑖 , 𝑁 𝑚( 𝑝 𝑖 ) )}
𝑛 𝑡 =𝑡ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡𝑠 𝑛 𝑠 =𝑡ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠𝑐𝑒𝑛𝑒 𝑝𝑜𝑖𝑛𝑡𝑠
𝑝 𝑖 = 𝑥 𝑖 , 𝑦 𝑖 𝑇 𝜖 𝑅 2 = 𝑖 𝑡ℎ 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡 𝑞 𝑗 𝜖 𝑅 2 = 𝑗 𝑡ℎ 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡
𝑆𝜖 𝑅 𝑛 𝑆 ×2 =𝑠𝑒𝑡 𝑜𝑓 𝑠𝑐𝑒𝑛𝑒 𝑝𝑜𝑖𝑛𝑡𝑠 𝑇𝜖 𝑅 𝑛 𝑡 ×2 =𝑠𝑒𝑡 𝑜𝑓 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡𝑠
𝑁 𝑝 𝑖 =𝑡ℎ𝑒 𝑠𝑒𝑡 𝑜𝑓 𝑜𝑟𝑑𝑒𝑟𝑒𝑑 𝑝𝑜𝑖𝑛𝑡𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟ℎ𝑜𝑜𝑑 𝑜𝑓 𝑝 𝑖 𝑚 𝑝 𝑖 =𝑚𝑎𝑡𝑐ℎ𝑖𝑛𝑔 𝑓𝑢𝑛𝑐𝑡𝑢𝑖𝑜𝑛

5. Problem formulation geometric similarity Photometric similarity 𝑝 𝑖
𝑚(𝑝 𝑖 )
𝑖=1 𝑛 𝑡 {𝑐( 𝑝 𝑖 , 𝑚 𝑝 𝑖 +𝜆∙𝑔( 𝑝 𝑖 , 𝑁 𝑝 𝑖 ;𝑚 𝑝 𝑖 , 𝑁 𝑚( 𝑝 𝑖 ) )}
𝑛 𝑡 =𝑡ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡𝑠 𝑛 𝑠 =𝑡ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠𝑐𝑒𝑛𝑒 𝑝𝑜𝑖𝑛𝑡𝑠
𝑝 𝑖 = 𝑥 𝑖 , 𝑦 𝑖 𝑇 𝜖 𝑅 2 = 𝑖 𝑡ℎ 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡 𝑞 𝑗 𝜖 𝑅 2 = 𝑗 𝑡ℎ 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡
𝑆𝜖 𝑅 𝑛 𝑆 ×2 =𝑠𝑒𝑡 𝑜𝑓 𝑠𝑐𝑒𝑛𝑒 𝑝𝑜𝑖𝑛𝑡𝑠 𝑇𝜖 𝑅 𝑛 𝑡 ×2 =𝑠𝑒𝑡 𝑜𝑓 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡𝑠
𝑁 𝑝 𝑖 =𝑡ℎ𝑒 𝑠𝑒𝑡 𝑜𝑓 𝑜𝑟𝑑𝑒𝑟𝑒𝑑 𝑝𝑜𝑖𝑛𝑡𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟ℎ𝑜𝑜𝑑 𝑜𝑓 𝑝 𝑖 𝑚 𝑝 𝑖 =𝑚𝑎𝑡𝑐ℎ𝑖𝑛𝑔 𝑓𝑢𝑛𝑐𝑡𝑢𝑖𝑜𝑛

6. Supplement: Difference from graph matching
Slide from “Learning Graphs to Match”, Minsu Cho, Karteek Alahari, and Jean Ponce,ICCV 13

7. Supplement: Difference from graph matching
Finding y maximizing score S (Integer quadratic programming) 𝑦 ∗ = arg max 𝑦 𝑆 𝐺, 𝐺 ′ ,𝑦 = arg max 𝑦 𝑦 𝑇 𝑊𝑦 𝑠.𝑡 𝑊 𝑖𝑎;𝑗𝑏 = 𝑆 𝑉 𝒂 𝑖 , 𝒂 𝑎 ′ 𝑖𝑓 𝑖=𝑗, 𝑎=𝑏 𝑆 𝐸 𝒂 𝑖𝑗 , 𝒂 𝑎𝑏 ′ 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
Slide from “Learning Graphs to Match”, Minsu Cho, Karteek Alahari, and Jean Ponce,ICCV 13

8. contribution Locally affine invariant geometric constraint
Efficient linearization to solve an original objective
Successive refinement for accurate matching

9. Problem formulation geometric similarity Photometric similarity 𝑝 𝑖
𝑚(𝑝 𝑖 )
𝑖=1 𝑛 𝑡 {𝑐( 𝑝 𝑖 , 𝑚 𝑝 𝑖 +𝜆∙𝑔( 𝑝 𝑖 , 𝑁 𝑝 𝑖 ;𝑚 𝑝 𝑖 , 𝑁 𝑚( 𝑝 𝑖 ) )}
𝑛 𝑡 =𝑡ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡𝑠 𝑛 𝑠 =𝑡ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠𝑐𝑒𝑛𝑒 𝑝𝑜𝑖𝑛𝑡𝑠
𝑝 𝑖 = 𝑥 𝑖 , 𝑦 𝑖 𝑇 𝜖 𝑅 2 = 𝑖 𝑡ℎ 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡 𝑞 𝑗 𝜖 𝑅 2 = 𝑗 𝑡ℎ 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡
𝑆𝜖 𝑅 𝑛 𝑆 ×2 =𝑠𝑒𝑡 𝑜𝑓 𝑠𝑐𝑒𝑛𝑒 𝑝𝑜𝑖𝑛𝑡𝑠 𝑇𝜖 𝑅 𝑛 𝑡 ×2 =𝑠𝑒𝑡 𝑜𝑓 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡𝑠
𝑁 𝑝 𝑖 =𝑡ℎ𝑒 𝑠𝑒𝑡 𝑜𝑓 𝑜𝑟𝑑𝑒𝑟𝑒𝑑 𝑝𝑜𝑖𝑛𝑡𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟ℎ𝑜𝑜𝑑 𝑜𝑓 𝑝 𝑖 𝑚 𝑝 𝑖 =𝑚𝑎𝑡𝑐ℎ𝑖𝑛𝑔 𝑓𝑢𝑛𝑐𝑡𝑢𝑖𝑜𝑛

10. The modeling of the feature matching function (photometric)
𝑖=1 𝑛 𝑡 {𝑐( 𝑝 𝑖 , 𝑚 𝑝 𝑖 )} = 𝑖 𝑛 𝑡 𝑗 𝑛 𝑠 𝐶 𝑖𝑗 𝑋 𝑖𝑗 =𝑡𝑟 𝐶 𝑇 𝑋
𝐶 𝑖𝑗 = min 𝑠, 𝜃 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒(𝑓𝑒𝑎𝑡𝑢𝑟𝑒 𝑝 𝑖 , 𝑓𝑒𝑎𝑡𝑢𝑟𝑒(𝑇 𝑞 𝑗 ;𝑠, 𝜃 )
𝑋 𝜖 0, 1 𝑛 𝑡 × 𝑛 𝑠 =𝑟𝑒𝑝𝑟𝑒𝑠𝑒𝑛𝑡𝑎𝑡𝑖𝑜𝑛 𝑜𝑓 𝑚 ∙ 𝑠.𝑡 𝑋 1 𝑛 𝑠 = 1 𝑛 𝑡
𝐶 𝜖 𝑅 𝑛 𝑡 × 𝑛 𝑠 =𝑐𝑜𝑠𝑡𝑠 𝑜𝑓 𝑚𝑎𝑡𝑐ℎ𝑖𝑛𝑔

11. A locally affine-invariant constraint
Every template point have at least three neighbors
Neighbors must not be collinear
𝑝 𝑖 = 𝑝 𝑗 𝜖 𝑁 𝑝 𝑖 𝑊 𝑖𝑗 𝑝 𝑗
𝑊 𝑖𝑗 = 𝑊 𝑖𝑙 𝑖𝑓 𝑝 𝑗 𝑖𝑠 𝑡ℎ𝑒 𝑙 𝑡ℎ 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑜𝑓 𝑝 𝑖 0 𝑖𝑓 𝑝 𝑗 𝑖𝑠 𝑛𝑜𝑡 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑜𝑓 𝑝 𝑖
𝑝 1 𝑝 𝑝 𝑊 𝑖 𝑇 =Q 𝑊 𝑖 𝑇 = 𝑝 𝑖 1
⇒ 𝑊 𝑖 𝑇 = 𝑄 −1 𝑝 𝑖 𝑇

12. A locally affine-invariant constraint
Every template point have at least three neighbors
Neighbors must not be collinear
All the points are moving in the same scale and rotation
𝑝 𝑖 = 𝑝 𝑗 𝜖 𝑁 𝑝 𝑖 𝑊 𝑖𝑗 𝑝 𝑗 ⇒ 0= 𝑝 𝑖 − 𝑗 𝑊 𝑖𝑗 𝑝 𝑗 ⇒ 0= 𝐼−𝑊 𝑇
𝑋= arg min 𝑋 𝑔 𝑝 𝑖 , 𝑁 𝑝 𝑖 ;𝑚 𝑝 𝑖 , 𝑁 𝑚 𝑝 𝑖 = arg min 𝑋 𝐼−𝑊 𝑋𝑆
The objective is affine-invariant!!
𝑠.𝑡. 𝐼=𝑖𝑑𝑒𝑛𝑡𝑖𝑡𝑦 𝑚𝑎𝑡𝑟𝑖𝑥 , 𝑇 𝜖 𝑅 𝑛 𝑡 ×2 , 𝑆𝜖 𝑅 𝑛 𝑆 ×2
𝑋𝑆=𝑚𝑎𝑡𝑐ℎ𝑒𝑑 𝑠𝑐𝑒𝑛𝑒 𝑝𝑜𝑖𝑛 𝑡 ′ 𝑠 𝑐𝑜𝑜𝑟𝑑𝑖𝑛𝑎𝑡𝑒𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑠𝑎𝑚𝑒 𝑜𝑟𝑑𝑒𝑟 𝑎𝑠 𝑡ℎ𝑒 𝑡𝑒𝑚𝑝𝑙𝑎𝑡𝑒 𝑝𝑜𝑖𝑛𝑡𝑠

13. Overall objective function
𝑚𝑖𝑛𝑖𝑚𝑖𝑧 𝑒 𝑋 tr C T X +𝜆 𝐼−𝑊 𝑋𝑆
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑋 𝜖 0,1 𝑛 𝑡 × 𝑛 𝑠 𝑋 1 𝑛 𝑠 = 1 𝑡 𝑡 𝑋 𝑇 1 𝑛 𝑡 ≤ 𝑤 𝑛 𝑠
Difficult to solve : NP-hard
Solution : convert an objective to linear programming algorithm
𝑚 ∙ = arg min 𝑚(∙) 𝑖=1 𝑛 𝑡 {𝑐( 𝑝 𝑖 , 𝑚 𝑝 𝑖 +𝜆∙𝑔( 𝑝 𝑖 , 𝑁 𝑝 𝑖 ;𝑚 𝑝 𝑖 , 𝑁 𝑚( 𝑝 𝑖 ) )}

14. Linearization and relaxation
Linear programming
- Linear objective and linear constraints - Reduction of search space
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑐 𝑇 𝑥 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝐴𝑥≤𝑏 𝑎𝑛𝑑 𝑥≥0
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑋 𝜖 0,1 𝑛 𝑡 × 𝑛 𝑠 𝑋 1 𝑛 𝑠 = 1 𝑡 𝑡 𝑋 𝑇 1 𝑛 𝑡 ≤ 𝑤 𝑛 𝑠
𝑋 𝜖 0, 1 𝑛 𝑡 × 𝑛 𝑠
𝑚𝑖𝑛𝑖𝑚𝑖𝑧 𝑒 𝑋 tr C T X +𝜆 𝐼−𝑊 𝑋𝑆

15. Linearization and relaxation
Linear approximation
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑖 𝑁 𝑥 𝑖 ⇔ 𝑚𝑖𝑛𝑖𝑚𝑖𝑧 𝑒 𝑥 𝑖 , 𝑢 𝑖 𝑖=1 𝑁 𝑢 𝑖 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑥 𝑖 ≤ 𝑢 𝑖 , 𝑥 𝑖 ≥− 𝑢 𝑖 𝑢 𝑖 ≥0 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑖=1,⋯,𝑁
𝑚𝑖𝑛𝑖𝑚𝑖𝑧 𝑒 𝑋 𝜆 𝐼−𝑊 𝑋𝑆 ⇔ 𝑚𝑖𝑛𝑖𝑚𝑖𝑧 𝑒 𝑋,𝑈 𝜆 1 𝑛 𝑡 𝑇 𝑈 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑋≥0, 𝑈≥0 𝐼−𝑊 𝑋𝑆≤𝑈 𝐼−𝑊 𝑋𝑆≥−𝑈

16. Linearization and relaxation
Final objective
𝑚𝑖𝑛𝑖𝑚𝑖𝑧 𝑒 𝑋,𝑈 𝑓 𝑋 =𝑡𝑟 𝐶 𝑇 𝑋 +𝜆 1 𝑛 𝑡 𝑇 𝑈 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑋≥0, 𝑈≥0 𝑋 1 𝑛 𝑠 = 1 𝑛 𝑡 𝐼−𝑊 𝑋𝑆≤𝑈 𝐼−𝑊 𝑋𝑆≥−𝑈 𝑋 𝑇 1 𝑛 𝑡 ≤ 𝑤 𝑛 𝑠
More problem : X is continuous
Solution : successive linear programming with search space reduction

17. Removing unnecessary variables
Trust region shrinkage with neighbor search
Lower convex hull search
각 template point에 대해서 search space를 줄여준다.

18. Trust region shrinkage with neighbor search
For each iteration - smaller diameter r
- Exclude the variables outside its trust region
Finishing condition - For proper iteration step, For each row, fixing other rows, find a discrete solution for the low minimizing an objective function

19. The lower convex hull property
Cost surfaces can be replaced by their lower convex hulls without changing the LP solution

20. Experiments
Time complexity

21. Experiments
Matching correctness

22. Experiments
Rotation invariance

23. Experiments
In videos

24. Discussion Assumption of scale and rotation Distinctive feature points
Occlusion handling
Appropriate weights
Time complexity