1. Locality-constrained Linear Coding for Image Classification
Presenter : Han-Mu Park

2. Contents Introduction Coding methods Proposed method
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Contents
Introduction
Coding methods
Proposed method
Experimental results
Conclusion
References

3. Introduction Bag-of-Words (BoW) model
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Introduction
Bag-of-Words (BoW) model
An image is represented as a collection of visual words.
Generally, to represent the collection, histogram of words form is used.

4. Introduction General Spatial Pyramid Matching frameworks
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Introduction
General Spatial Pyramid Matching frameworks
Feature extraction
SIFT
HOG
etc
Coding
Vector Quantization
Sparse coding
Pooling
Max pooling
Sum pooling
Spatial Pyramid Matching framework [J.Wang2010]

5. Introduction General Spatial Pyramid Matching frameworks
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Introduction
General Spatial Pyramid Matching frameworks
Spatial Pyramid Matching framework [J.Wang2010]

6. Coding methods Coding methods Vector quantization (VQ)
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Coding methods
Coding methods
Vector quantization (VQ)
Sparce coding (SC)
Locality-constrained Linear Coding (LLC)
[J.Wang2010]

7. Coding methods Vector quantization (VQ) Hard quantization method
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Coding methods
Vector quantization (VQ)
Hard quantization method
A set of 𝐷-dimensional local descriptors
𝑋= 𝑥 1 , 𝑥 2 , …,𝑥 𝑁 ∈ 𝑅 𝐷×𝑁
Codebook with 𝑀 entries
𝐵= 𝑏 1 , 𝑏 2 ,…, 𝑏 𝑀 ∈ 𝑅 𝐷×𝑀
Objective function
𝑎𝑟𝑔 min 𝐶 𝑖=1 𝑁 𝑥 𝑖 −𝐵 𝑐 𝑖 2
𝑠.𝑡. 𝑐 𝑖 𝑙 0 =1, 𝑐 𝑖 𝑙 1 =1, 𝑐 𝑖 ≽0, ∀𝑖
Where 𝐶= 𝑐 1 , 𝑐 2 ,…, 𝑐 𝑁 is the set of codes for X

8. Coding methods Sparse coding (SC) Soft quantization method
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Coding methods
Sparse coding (SC)
Soft quantization method
Relaxed the cardinality constraint
Objective function
𝑎𝑟𝑔 min 𝐶 𝑖=1 𝑁 𝑥 𝑖 −𝐵 𝑐 𝑖 𝜆 𝑐 𝑖 𝑙 1
The roles of sparsity regularization term
Because the codebook 𝐵 is usually over-complete 𝑀>𝐷 , it is necessary to ensure that the under-determined system has a unique solution.
Sparsity allows the learned representation to capture salient patterns of local descriptors.
The sparse coding can achieve much less quantization error than VQ.

9. Proposed method Locality-constrained Linear Coding (LLC)
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Proposed method
Locality-constrained Linear Coding (LLC)
Replaced the sparsity regularization term with new constraint.
Objective function
𝑎𝑟𝑔 min 𝐶 𝑖=1 𝑁 𝑥 𝑖 −𝐵 𝑐 𝑖 𝜆 𝑑 𝑖 ⊙ 𝑐 𝑖 2
𝑠.𝑡. 1 𝑇 𝑐 𝑖 =1, ∀𝑖
⊙ : the element-wise multiplication
𝑑 𝑖 ∈ 𝑅 𝑀
𝑑 𝑖 = exp 𝑑𝑖𝑠𝑡 𝑥 𝑖 ,𝐵 𝜎
Where 𝑑𝑖𝑠𝑡 𝑥 𝑖 ,𝐵 = 𝑑𝑖𝑠𝑡 𝑥 𝑖 , 𝑏 1 ,𝑑𝑖𝑠𝑡 𝑥 𝑖 , 𝑏 2 ,…,𝑑𝑖𝑠𝑡 𝑥 𝑖 , 𝑏 𝑀 𝑇

10. Proposed method Properties of LLC Better reconstruction
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Proposed method
Properties of LLC
Better reconstruction
Because LLC represents each descriptor by using multiple weighted bases (codewords), it has less reconstruction error than VQ.
Local smooth sparsity
Because the regularization term of 𝑙 1 𝑛𝑜𝑟𝑚 in SC is not smooth, therefore, SC loses correlations between codes.
Analytical solution
The solution of LLC can be derived analytically by
𝑐 𝑖 = 1 𝐶 𝑖 +𝜆𝑑𝑖𝑎𝑔 𝑑
𝑐 𝑖 = 𝑐 𝑖 1 𝑇 𝑐 𝑖
Where 𝐶 𝑖 = 𝐵−1 𝑥 𝑖 𝑇 𝐵−1 𝑥 𝑖 𝑇 𝑇

11. Proposed method Approximated LLC for fast encoding
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Proposed method
Approximated LLC for fast encoding
The LLC selects the local bases for each descriptor to form a local coordinate system.
To speedup the encoding process, authors used 𝐾 (𝐾<𝐷<𝑀) nearest neighbors of 𝑥 𝑖 as the local bases 𝐵 𝑖 , and solve a much smaller linear system to get the codes
min 𝐶 𝑖=1 𝑁 𝑥 𝑖 − 𝑐 𝑖 𝐵 𝑖 2
𝑠.𝑡. 1 𝑇 𝑐 𝑖 =1, ∀𝑖
The reduced computation complexity
𝑂 𝑀 2 ⟶𝑂 𝑀+ 𝐾 2 , where 𝐾≪𝑀

12. Proposed method Codebook optimization
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Proposed method
Codebook optimization
To improve the accuracy, authors trained the codebook to optimize for LLC codes.
The optimal codebook 𝐵 ∗ can be obtained by
𝑎𝑟𝑔 min 𝐶,𝐵 𝑖=1 𝑁 𝑥 𝑖 −𝐵 𝑐 𝑖 2 +𝜆 𝑑 𝑖 ⊙ 𝑐 𝑖 2
𝑠.𝑡. 1 𝑇 𝑐 𝑖 =1, ∀𝑖
𝑏 𝑗 2 ≤1, ∀𝑗
This can be solved by using Coordinate Descent method.
However, because the number of training descriptors 𝑁 is usually very large, the huge memory space is needed to solve that problem.

13. Proposed method Incremental codebook optimization
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Proposed method
Incremental codebook optimization
First, initialize 𝐵 by using K-means clustering.
Then loop through all the training descriptors to update 𝐵 incrementally.
In each iteration, we take in a single (or a small set of) examples 𝑥 𝑖 , and solve original objective function to obtain the corresponding LLC codes.
[J.Wang2010]

14. Proposed method Incremental codebook optimization
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Proposed method
Incremental codebook optimization
Then select bases 𝐵 𝑖 whose corresponding weights are larger than predefined threshold, and refit 𝑥 𝑖 without the locality constraint.
The obtained code is used to update the basis in a gradient descent fashion.
[J.Wang2010]

15. Experimental results Performance of codebook
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Experimental results
Performance of codebook

16. Experimental results Performance under different neighbors
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Experimental results
Performance under different neighbors

17. Experimental results Results using Pascal VOC 2007
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Experimental results
Results using Pascal VOC 2007

18. Conclusion Contribution
Locality-constrained Linear Coding for Image Classification, CVPR 2010
Conclusion
Contribution
In this paper, the Locality-constrained Linear Coding (LCC) method is proposed
Better reconstruction
Local smooth sparsity
Analytical solution
For speedup, K-nearest neighbors algorithm is used.
To optimize the accuracy, incremental codebook learning is proposed for LCC.

19. Locality-constrained Linear Coding for Image Classification, CVPR 2010
References
[1] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, Y. Gong, “Locality-constrained Linear Coding for Image Classification,” CVPR 2010.

20. Thank you!