1. Image classification
by sparse coding

2. Feature learning problem
Given a 14x14 image patch x, can represent it using 196 real numbers.
Problem: Can we find a learn a better representation for this?

3. Unsupervised feature learning
Given a set of images, learn a better way to represent image than pixels.

4. First stage of visual processing in brain: V1
The first stage of visual processing in the brain (V1) does “edge detection.”
Green: Responds to white dot.
Red: Responds to black dot.
Schematic of simple cell
Actual simple cell
Also used in image compression and denoising.
“Gabor functions.”
[Images from DeAngelis, Ohzawa & Freeman, 1995]

5. Learning an image representation
Sparse coding (Olshausen & Field,1996) Input: Images x(1), x(2), …, x(m) (each in Rn x n) Learn: Dictionary of bases f1, f2, …, fk (also Rn x n), so that each input x can be approximately decomposed as: s.t. aj’s are mostly zero (“sparse”) Use to represent 14x14 image patch succinctly, as [a7=0.8, a36=0.3, a41 = 0.5]. I.e., this indicates which “basic edges” make up the image.
[NIPS 2006, 2007]

6. Sparse coding illustration
Natural Images
Learned bases (f1 , …, f64): “Edges”
Test example
» 0.8 * * *
x » 0.8 * f * f * f63
[0, 0, …, 0, 0.8, 0, …, 0, 0.3, 0, …, 0, 0.5, …]
= [a1, …, a64] (feature representation)
Compact & easily
interpretable

7. » 0.6 * + 0.8 * + 0.4 * » 1.3 * + 0.9 * + 0.3 * More examples
Represent as: [0, 0, …, 0, 0.6, 0, …, 0, 0.8, 0, …, 0, 0.4, …] Represent as: [0, 0, …, 0, 1.3, 0, …, 0, 0.9, 0, …, 0, 0.3, …]
» 0.6 * * *
  37
» 1.3 * * *
  29
Method hypothesizes that edge-like patches are the most “basic” elements of a scene, and represents an image in terms of the edges that appear in it.
Use to obtain a more compact, higher-level representation of the scene than pixels.

8. Digression: Sparse coding applied to audio
Efficient Kernels
Here is a collection of revcor filters, some of which you saw earlier, recorded from cat auditory nerve fibers
For each optimized kernel function we’ll pick the revcor filter it best matches and overlaid them [click]
[Evan Smith & Mike Lewicki, 2006]

9. Digression: Sparse coding applied to audio
Efficient Kernels
Here you can see that for nearly all of the optimized kernel functions, they closely match the detailed structure of an individual ANF revcor filter [click]
[Evan Smith & Mike Lewicki, 2006]

10. Input: Images x(1), x(2), …, x(m) (each in Rn x n)
Sparse coding details
Input: Images x(1), x(2), …, x(m) (each in Rn x n)
L1 sparsity term
(causes most s to be 0)
Alternating minimization:
Alternately minimize with respect to fi‘s (easy)
and a’s (harder).

11. How to scale this algorithm up?
Solving for bases
Early versions of sparse coding were used to learn about this many bases:
32 learned bases
How to scale this algorithm up?

12. Input: Images x(1), x(2), …, x(m) (each in Rn x n)
Sparse coding details
Input: Images x(1), x(2), …, x(m) (each in Rn x n)
L1 sparsity term
Alternating minimization:
Alternately minimize with respect to fi‘s (easy)
and a’s (harder).

13. Feature sign search (solve for ai’s)
Goal: Minimize objective with respect to ai’s.
Simplified example:
Suppose I tell you:
Problem simplifies to:
This is a quadratic function of the ai’s. Can be solved efficiently in closed form.
Algorithm:
Repeatedly guess sign (+, - or 0) of each of the ai’s.
Solve for ai’s in closed form. Refine guess for signs.

14. The feature-sign search algorithm: Visualization
Current guess:
Starting from zero (default)

15. The feature-sign search algorithm: Visualization
Current guess:
1: Activate a2
with “+” sign
Active set ={a2}
Starting from zero (default)

16. The feature-sign search algorithm: Visualization
Current guess:
1: Activate a2
with “+” sign
Active set ={a2}
Starting from zero (default)

17. The feature-sign search algorithm: Visualization
Current guess:
2: Update a2 (closed form)
1: Activate a2
with “+” sign
Active set ={a2}
Starting from zero (default)

18. The feature-sign search algorithm: Visualization
3: Activate a1
with “+” sign
Active set ={a1,a2}
Current guess:
Starting from zero (default)

19. The feature-sign search algorithm: Visualization
3: Activate a1
with “+” sign
Active set ={a1,a2}
Current guess:
4: Update a1 & a2 (closed form)
Starting from zero (default)

20. Before feature sign search
32 learned bases

21. With feature signed search

22. Recap of sparse coding for feature learning
SIFT descriptors x(1), x(2), …, x(m) (each in R128)
R128.
Recap of sparse coding for feature learning
Input: Images x(1), x(2), …, x(m) (each in Rn x n)
Learn: Dictionary of bases f1, f2, …, fk (also Rn x n).
Relate to histograms view, and so sparse-coding on top of SIFT features.
Training time
Test time
Input: Novel image x (in Rn x n) and previously learned fi’s.
Output: Representation [a1, a2, …, ak] of image x.
» 0.8 * * *
x » 0.8 * f * f * f63
Represent as: [0, 0, …, 0, 0.8, 0, …, 0, 0.3, 0, …, 0, 0.5, …]

23. x » 0.8 * f36 + 0.3 * f42 + 0.5 * f63 Sparse coding recap
» 0.8 * * *
[0, 0, …, 0, 0.8, 0, …, 0, 0.3, 0, …, 0, 0.5, …]
x » 0.8 * f * f * f63
Much better than pixel representation. But still not competitive with SIFT, etc.
Three ways to make it competitive:
Combine this with SIFT.
Advanced versions of sparse coding (LCC).
Deep learning.

24. Combining sparse coding with SIFT
Input: Images x(1), x(2), …, x(m) (each in Rn x n)
Learn: Dictionary of bases f1, f2, …, fk (also Rn x n).
SIFT descriptors x(1), x(2), …, x(m) (each in R128)
R128.
Test time: Given novel SIFT descriptor, x (in R128), represent as

25. Relate to histograms view, and so sparse-coding on top of SIFT features.
Putting it together
Feature representation
Learning
algorithm
Suppose you’ve already learned bases f1, f2, …, fk. Here’s how you represent an image.
Learning
algorithm or
x(1)
x(2)
x(3) …
E.g., 73-75% on Caltech 101 (Yang et al., 2009, Boreau et al., 2009) …
a(1)
a(2)
a(3)

26. K-means vs. sparse coding
Centroid 1
Centroid 2
Centroid 3
Represent as:

27. K-means vs. sparse coding
Intuition: “Soft” version of k-means (membership in multiple clusters).
K-means vs. sparse coding
K-means
Sparse coding
Centroid 1
Basis f1
Centroid 2
Basis f2
Centroid 3
Basis f3
Represent as:
Represent as:

28. K-means vs. sparse coding
Rule of thumb: Whenever using k-means to get a dictionary, if you replace it with sparse coding it’ll often work better.