Presentation on theme: "Independent Component Analysis"— Presentation transcript:
1 Independent Component Analysis CMPUT 466/551Nilanjan Ray
2 The Origin of ICA: Factor Analysis Multivariate data are often thought to be indirect measurements arising from some underlying sources, which cannot be directly measured/observed.ExamplesEducational and psychological tests use the answers to questionnaires to measure the underlying intelligence and other mental abilities of subjectsEEG brain scans measure the neuronal activity in various parts of the brain indirectly via electromagnetic signals recorded at sensors placed at various positions on the head.Factor analysis is a classical technique developed in statistical literature that aims at identifying these latent sources.Independent component analysis (ICA) is a kind of factor analysis that can uniquely identify the latent variables.
3 Latent Variables and Factor Analysis Latent variable model:or,Observed variableLatent componentsMixing matrixFactor analysis attempts to find out both the mixing coefficients and thelatent components given some instances of observed variables
4 Latent Variables and Factor Analysis… Typically we require the latent variables to have unit variance and to be uncorrelated.Thus, in the following model, cov(S) = I.This representation has an ambiguity. Consider, for example an orthogonal matrix R:So, is also a factor model with unit variance, uncorrelated latent variables.Classical factor analysis cannot remove this ambiguity; ICA can remove this ambiguity.
5 Classical Factor Analysis Model:’s are zero mean, uncorrelated Gaussian noise.q < p, i.e., the number of underlying latent factor is assumed less thanthe number of observed components.Diagonal matrixThe covariance matrix takes this form:Maximum likelihood estimation is used to estimate A.However, still the previous problem of ambiguity remains here too…
6 Independent Component Analysis Step 1: Center data:Step 2: Whiten data: compute SVD of the centered data matrixAfter whitening in the factor model, the covariance of x, cov(x) = I, and A become orthogonalStep 3: Find out orthogonal A and unit variance, non-Gaussian and independent SPCA
7 Example: PCA and ICA Model: Blind source separation (cocktail party problem)
8 PCA vs. ICA PCA: Find projections to minimize reconstruction error Variance of projected data is as large as possible2nd-order statistics needed (cov(x))ICA:Find “interesting” projectionsProjected data look as non-Gaussian, independent as possibleHigher-order statistics needed to measure degree of independence
9 Computing ICAModel:Step 3: Find out orthogonal A and unit variance, non-Gaussian and independent S.The computational approaches are mostly based on information theoretic criterion.Kullback-Leibler (KL) divergenceNegentropyAnother different approach emerged recently is called “Product Density Approach”
10 ICA: KL Divergence Criterion x is zero-mean and whitenedKL divergence measures “distance” between two probability densitiesFind A such that KL(.) is minimized:Joint densityBefore whitening, ICA means finding components as non-Gaussian as possibleAfter whitening procedure: Cov(X) = I, ICA means finding components as independent as possible.Independent densityH is differential entropy:
11 ICA: KL Divergence Criterion… Theorem for random variable transformation says:So,Verify formula: why I got + log |det(A)| instead of –ve log |det(A)|Hence,Minimize with respect to orthogonal A
12 ICA: Negentropy Criterion Differential entropy H(.) is not invariant to scaling of variableNegentropy is a scale-normalized version of H(.):Negentropy measures the departure of a r.v. s from a Gaussian r.v. with same varianceOptimization criterion:I would think that normalized entropy would be better if (gaussian) noise is present. (not sure if the claim is true)
13 ICA: Negentropy Criterion… Approximate the negentropy from data by:FastICA (http://www.cis.hut.fi/projects/ica/fastica/) is based on negentropy. Free software in Matlab, C++, Python…
14 ICA Filter Bank for Image Processing An image patch is modeled as a weighted sum of basis images (basis functions):Image patchBasis functions (a.k.a. ICA filter bank)Rows of AT are filtersColumns of A are filtersFilter responsesJenssen and Eltoft, “ICA filter bank for segmentation of textured images,” 4th International symposium on ICA and BSS, Nara, Japan, 2003
15 Texture and ICA Filter Bank Training textures12x12 ICA basis functions or ICA filtersJenssen and Eltoft, “ICA filter bank for segmentation of textured images,” 4th International symposium on ICA and BSS, Nara, Japan, 2003
16 Segmentation By ICA FB ICA Filter Bank With n filters I1, I2,…, In Theseare filterresponsesImage, ISegmented image, CClusteringAbove is an unsupervised setting.Segmentation (i.e., classification in this context) can also be performed bya supervised method on the output feature images I1, I2 , …, In.A texture imageSegmentationJenssen and Eltoft, “ICA filter bank for segmentation of textured images,” 4th International symposium on ICA and BSS, Nara, Japan, 2003
17 On PCA and ICA PCA & ICA differ in choosing projection directions: Different principle: least-square (PCA), independence (ICA)For data compression, PCA would be a good choiceFor discovering structures of data, ICA would be a reasonable choice
Your consent to our cookies if you continue to use this website.