Presentation is loading. Please wait.

Presentation is loading. Please wait.

An Introduction to Sparse dictionary learning

Similar presentations


Presentation on theme: "An Introduction to Sparse dictionary learning"— Presentation transcript:

1 An Introduction to Sparse dictionary learning
SUBTITLE HERE

2 1. Definition of Sparsity
Theoretical studies suggest that primary visual cortex (area V1) uses a sparse code to efficiently represent natural scenes. Sparse coding is a way of representing some phenomenon with as few variables as possible. Usage: Compression, Analysis, Denoising

3 2. Sparse Representation
𝑥 ≈ 𝐷 ∗𝑎 𝑥∈ ℛ 𝑛 is a signal Non-zero entries Near-zero or zero entries 𝐷=[𝑑1,𝑑2..𝑑𝑛]∈ ℛ 𝑛𝑥𝑚 𝑑 𝑇 𝑑=1 is a set of normalized “basis vectors” or “atoms” Non-zero entries * Near-zero or zero entries 𝑎∈ ℛ 𝑚 is a sparse vector that solves the equation Non-zero entries Signal Dictionary Sparse Vector

4 3. Why Sparsity? Provides the ability to represent large amount of data by keeping less information. (saves memory/storage) But the main property that sparsity has… “If you can create a model such that the objects can be represented with much less information than originally, but still have high fidelity to the original, it means that you have created a good model, because you have replaced redundancies in the original representation.”

5 4. Sparse Representation Overview
Δεν μ αρεσει ο τιτλος εδω When using sparse representation as a way of feature extraction, you may wonder: Does there exists the sparsity property in the data? Does sparse feature really come up with better results? Does it contain any semantic meaning? In a short answer “sometimes, not always” Below is a list of successful applications: Image Denoising Deblurring Inpainting Super-Resolution Restoration Quality Assessment Classification Segmentation Signal Processing Object Tracking Texture Classification Image Retrieval Bioinformatics Biometrics And Many Other Artificial Intelligence Sections

6 4. Sparse Representation Overview
Δεν μ αρεσει ο τιτλος εδω Assumes that there exists a sparse property in the data, otherwise sparse representation means nothing. D is used to be the feature of x D can be used to efficiently store and reconstruct x

7 4. Sparse Representation Overview
The general framework of sparse representation is: To exploit the linear combination of some samples or “atoms” . To represent the probe sample. To calculate the representation solution. i.e. Find the representation coefficients 𝑎 of features 𝑑 𝑖 for the samples 𝑥, and then utilize the representation solution to reconstruct the desired results.

8 4. Sparse Representation Categories
Sparse Representation, can be dominated by the regularizer imposed on the representation solution. For that reason it can be grouped into five general categories in terms of the different norms: l0-norm minimization: 𝛼 ∗ =𝑎𝑟𝑔𝑚𝑖𝑛 α 0 l1-norm minimization: 𝛼 ∗ =𝑎𝑟𝑔𝑚𝑖𝑛 α 1 lp-norm minimization: 𝛼 ∗ =𝑎𝑟𝑔𝑚𝑖𝑛 α 𝑝 𝑝 l2-norm minimization: 𝛼 ∗ =𝑎𝑟𝑔𝑚𝑖𝑛 α 2 2 l2,1-norm minimization: 𝛼 ∗ =𝑎𝑟𝑔𝑚𝑖𝑛 𝑋−𝐷𝛼 2,1 +𝜇 𝛼 2,1 function [ out ] = l_norm(varargin) A = varargin{1}; p = varargin{2}; if nargin == 2 if p == 0 out = nnz(A); else out = sum(abs(A).^p)^(1/p); end q = varargin{3}; out = sum(sum(abs(A').^p).^(q/p))^(1/q);

9 Built-in feature selection
4.1. Comparison of L-norms Before the algorithms that will find the sparse vector 𝑎, of a given 𝑥,𝐷. Let’s take a close look at 𝐿0/𝐿𝑝/𝐿1/𝐿2/𝐿2,1 norm L0-regularization Lp-regularization L1-regularization L2-regularization L2,1-regularization 𝑥 0 =#(𝑖| 𝑥 𝑖 ≠0) 𝑥 𝑝 = 𝑝 𝑖 |𝑥 𝑖 ​ 𝑝 𝑥 1 = 𝑖 | 𝑥 𝑖 | 𝑥 2 = 𝑖 𝑥 𝑖 2 𝑋 𝑝,𝑞 = 𝑗=1 𝑛 𝑖=1 𝑚 𝑥 𝑖𝑗 𝑝 𝑞 𝑝 /𝑞 𝑥 is a vector 𝑋 is a matrix Counts nonzero elements A p-th-root of a summation of all elements to the p-th power Sparse outputs No Sparse outputs Is more robust since the error for each data point (a column) is not squared. It is used in robust data analysis and sparse coding. Built-in feature selection No feature selection

10 4.1. Why solution uniqueness is important?
A unique linear regression estimate can have quite poor predictive accuracy. For example: The green line (L2-norm) is the unique shortest path, while the red, blue, yellow (L1- norm) are all same length for the same route. L2-norm has unique solutions while L1-norm does not.

11 4.1. Why Built-in feature selection is important?
Suppose the model have 1000 coefficients but only 10 of them have non-zero coefficients, this is effectively saying that “the other 990 predictors are useless in predicting the target values”. This is actually a result of the L1-norm, which tends to produces sparse coefficients.

12 5. Methods of Solving Sparse Coding
The following algorithms are trying to find a sparse vector α οf a given signal x and dictionary D. Greedy Optimization Strategy Constrained Optimization Strategy Proximity Algorithm Based Optimization Strategy Homotopy Algorithm Based Sparse Representation Matching Pursuit Orthogonal Matching Pursuit Orthogonal Matching Pursuit for large matrices Gradient Projection Sparse Reconstruction Interior-point method based sparse representation strategy Alternating direction method (ADM) based sparse representation strategy Soft thresholding or shrinkage operator Sparse reconstruction by separable approximation Iterative shrinkage thresholding algorithm l1/2-norm regularization based sparse representation Fast Iterative shrinkage thresholding algorithm Augmented Lagrange Multiplier based optimization strategy LASSO homotopy BPDN homotopy Iterative Reweighting l1-norm minimization via homotopy

13 5.1. Matching Pursuit Algorithm
It is a coordinate descent algorithm that iteratively selects one entry of the current estimate 𝛼 at a time and updates it. It is closely related to projection pursuit algorithms from the statistics literature. input Initialize 𝑎 = 0 select the coordinate with maximum partial Derivative 𝐽 = 𝑎𝑟𝑔𝑚𝑎𝑥| 𝑑 𝑗 𝑡 (𝑥−𝐷𝑎) D, χ 𝑎 0 < 𝑘 output α Lnorm0 function [a] = matching_pursuit(x,D,K) [n,m] = size (D) ; a = zeros(m,1); while l_norm(a,0) < K [argval_k,argmax_k] = max(abs(D'*(x-D*a))); a(argmax_k) = a(argmax_k) + D(:,argmax_k)'*(x-D*a); end

14 5.2. Orthogonal Matching Pursuit Algorithm
It is similar in spirit to matching pursuit, but enforces the residual to be always orthogonal to all previously selected variables, which is equivalent to saying that the algorithm reoptimizes the value of all non-zero coefficients once it selects a new variable. input Select dk with max projection on residue αk = arg min ||χ-Dkαk|| D, χ output Check terminating condition Update residue r = χ - Dkαk α Lnorm0 There is an optimized version of this algorithm which is presented as matlab source above.

15 5.3. Orthogonal Matching Pursuit Algorithm – for large matrices
function [a] = orthogonal_matching_pursuit (x,D,k) [m,n] = size (D) ; Q = zeros (m,k); R = zeros (k,k); Rinv = zeros (k,k) ; w = zeros (m,k) ; a = zeros (1,n) ; Res = x; for J = 1 : k [argval_k ,argmax_k(J)] = max(abs(D'*Res)) ; w (:,J) = D (:,argmax_k(J)) ; for I = 1 : J-1 if (J-1 ~= 0) R (I,J) = Q (:,I)' * w (:,J) ; w (:,J) = w (:,J) - R (I,J) * Q (:,I) ; end R (J,J) = norm (w (:,J)) ; Q (:,J) = w (:,J) / R (J,J) ; Res = Res - (Q (:,J) * Q (:,J)' * Res) ; Rinv (J,J) = 1 / R (J,J) ; Rinv (I,J) = -Rinv (J,J) * (Rinv (I,1:J-1) * R (1:J-1,J)); xx = Rinv * Q' * x ; a(argmax_k) = xx ;

16 6. Dictionary Learning Now let’s look at the reverse problem: could we design dictionaries based on learning? In the preceding slides, we generally assume that the (over-complete) bases D is existed and known. However in practice, we usually need to build it. Our goal is to find the dictionary D that yields sparse representations for the training signals.

17 6. How can we get the dictionary D?
Ideal case of Dictionary Expect: 𝑋 – 𝐷∗𝑎 = 0 Until now no algorithm is guaranteed to find a global minimizer. Know categories of dictionaries: Predefined dictionary based on various types of wavelets Dictionary Learning

18 6.1. Predefined dictionary based on various types of wavelets Algorithms
DCT/Wavelet dictionaries Time-frequency dictionaries

19 6.2. Dictionary Learning Algorithms
Supervised dictionary learning. Some representative algorithms are the following: Unsupervised dictionary learning. Discriminative KSVD Metaface dictionary learning Label consistent KSVD Discriminative non-parametric dictionary learning Fisher discrimination dictionary learning K-SVD Nonparametric Bayesian dictionary learning Tree-structured dictionary learningLabel consistent KSVD Locality constrained linear coding for unsupervised dictionary learning

20 Initialize Dictionary
K-SVD Algorithm Select atoms from input Atoms can be patches from the image Patches are overlapping Initialize Dictionary Sparse Coding (OMP) Update Dictionary One atom at a time

21 Initialize Dictionary
K-SVD Algorithm Use OMP or any other fast method Output gives sparse code for all signals Minimize error in representation Initialize Dictionary Sparse Coding (OMP) Update Dictionary One atom at a time

22 Initialize Dictionary
K-SVD Algorithm Replace unused atom with minimally represented signal Identify signals that use k-th atom (non zero entries in rows of X) Initialize Dictionary Sparse Coding (OMP) Update Dictionary One atom at a time

23 Initialize Dictionary
K-SVD Algorithm Initialize Dictionary Deselect k-th atom from dictionary Find coding error matrix of these signals Minimize this error matrix with rank- 1 approx from SVD Sparse Coding (OMP) Update Dictionary One atom at a time

24 Initialize Dictionary
K-SVD Algorithm Initialize Dictionary [U,S,V] = svd(Ek) Replace coeff of atom dk in X with entries of s1v1 dk = u1/||u1||2 Sparse Coding (OMP) Update Dictionary One atom at a time

25 Applications

26 7.1. Super Resolution using Sparse Coding
This application performs sparse coding to increase(x2) the resolution of a given image. Open and execute the Main.m file Select an image (jpg,bmp,png) Wait for the results. The application is using a predefined dictionary created by several natural images.

27 7.1. Super Resolution using Sparse Coding

28 7.2. Image Denoising using Sparse Coding
This application performs sparse coding to denoise a given image. Open and execute the Main.m file Select an image (jpg,bmp,png) Wait for the results. The application creates a dictionary using the K-SVD algorithm from the noisy image and afterwards reconstructs the image by its coefficients

29 7.2. Image Denoising using Sparse Coding


Download ppt "An Introduction to Sparse dictionary learning"

Similar presentations


Ads by Google