# EE645: Independent Component Analysis

## Presentation on theme: "EE645: Independent Component Analysis"— Presentation transcript:

EE645: Independent Component Analysis
Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003

Presentation Overview
ICA Motivation Mathematical Formulation Fast ICA Algorithm Applications Noise Separation and Feature Extraction Digital Watermarking

Motivation for ICA Cocktail Party Problem
Suppose you are in a crowded room with many people. How do you understand what any one person is saying? Separation of Independent Signals Similar to Blind Source Separation Little knowledge of the signals Access to mixed signals only

Cocktail Party Problem
??

Cocktail Party Problem
ICA Separation Algorithm Separation of Speech Signals Humans can separate multiple signals with only two ears/sensors ICA needs as many ears/sensors as message signals Here we assume he has four ears!

Recovered Messages

Mathematical Formulation Overview
ICA Definition ICA Assumptions Independent Signals Non-Gaussian ICA Limitations Scaling Permutations No. of Sensors

ICA Definition Mixed Signals in Matrix Notation Variable Definitions

ICA Solution Signal Separation Find using the ICA Algorithm

ICA Block Diagram (2 Signals)
Multiplexed Signal #1 Multiplexed Signal #2

ICA Assumption #1: Independence
Probability Density Definition Expected Value Definition

ICA Assumption #2: Non-Gaussian
Property of Gaussian signals Addition of two independent Gaussian random variables is another single Gaussian random variable. Information Lost! Kurtosis Function Special Case: kurt(N) = 0

Limitation #1: Scaling ICA maximizes independence between signals.
Scaling Factor for Signal #1 Assuming the Separated Signals are Independent Scaling Factor for Signal #2

Limitation #2: Signal Permutations
The mixing matrix and independent components are unknown.

Limitation #3: No. of Sensors
Sensor Requirement The number of separated signals cannot be larger than the number of inputs. Current research is being done to reduce this constraint.

ICA Separation Technique
Central Limit Theorem If two random (non-Gaussian) signals added, the resulting signal will be more Gaussian than the original two random signals ICA Separation Concept Central Limit Theorem (in Reverse) Maximizing Non-Gaussianity Results in separating the two signals

Fast ICA Algorithm Overview
“Fixed-Point” Algorithm Implementation Fast ICA algorithm Extensions Algorithm Speed & Performance Currently the fastest Most Commonly Used

Fast ICA Algorithm Choose a random initial weight vector. Let,
Repeat until converges.

Fast ICA Extensions Preprocessing Activation Functions
Normalize mean to zero Pre-Whitening Activation Functions g(u)=u^3 g(u)=u^2 g(u)=tanh(a1*u) g(u)=u*exp(-a2*u^2/2)

Noise Separation Example
Separation of Noise Impulsive Noise Additive White Gaussian Noise Implementation Two Sensor Setup Fast ICA Algorithm

Noise Generation AWGN Gaussian R.V. Impulsive Noise Poisson R.V.

Noise Separation Example

Noise Separation Example

Noise Separation Example

Noise Separation Performance

Noise Separation Performance

Noise Separation & Feature Extraction
ICA performs well in Blind Source Separation ICA for Feature Extraction Reduce Complexity of the Neural Network Train only on the appropriate signal Detection and Estimation of Hidden Signals

Noise Detection and Estimation: Method #1
Fast ICA N.N. (BP, RBF, SVM) Sensor #1 Feature #1 Sensor #2 Feature #2 Sensor #3 Feature #3 Sensor #4 Feature #4

Noise Detection and Estimation: Method #2
Fast ICA N.N. (BP, RBF, SVM) Sensor #1 Feature #1 Decision Device Sensor #2 Feature #2 Sensor #3 Feature #3 Sensor #4 Feature #4

Noise Detection and Estimation: Conclusion
Additional Preprocessing Segmentation of Impulsive Noise (Time-Limited) Possible Inputs to the Neural Network Statistical Moments Signal Samples Possible Neural Networks Back Propagation SVM Radial Basis Functions Problem Need to train Neural Network in Matlab

Digital Watermarking of Music
Motivation Popularity of Digital Storage Devices Reliable, Fast, Ease of duplication, etc. How to protect copyrighted information? Leaving digital signatures of its artist Essential Properties for Watermarking Undetectable Irremovable Resilient

Detection & Estimation of Watermarks
Detection of Watermark Authenticate copyrighted music Estimation of Watermark Information on artist, producer, etc.

Watermarking Model Process
Mix the original musical data with watermark Keep watermark Power relatively low Ensure high quality of the watermarked music Watermark is better hidden

Popular Digital Formats
Wave Format Mp3 Format Bit Rate 1411 [kbps] 128 [kbps] Channels 2 Audio Sample Rate 44 [kHz]

Detection of Watermark
Watermarking Detection Algorithm Use the ICA model to randomly mix the watermark and music file. Save the watermarked music in the popular *.wav format Read the saved *.wave file. Separate the watermark and the music file. Identify the watermark using statistical methods (mean, std, etc.) Performance Statistic Correlation Coefficient (Absolute Value)

Detection Performance

Estimation of Watermark
Watermarking Estimation Algorithm Use the ICA model to randomly mix the watermark and music file. Save the watermarked music in the popular *.wav format Read the saved *.wave file. Separate the watermark and the music file. Identify the watermark using statistical methods (mean, std, etc.) Digitize the watermark signal. Performance Statistic Bit Error Rate

Estimation Performance

Is the Music Content Preserved?
“Hero” – Mariah Carey Original Wave file from CD Hero.wav 29 [s] “Hero” – Mariah Carey Watermarked Wave file HeroWatermarked.wav 29 [s]

Resilience of Proposed Watermark
Resiliency Previous Simulations show that wav format is resilient to 8-bit and 16-bit quantization. Can the watermark be detected after Mp3 compression and decompression?

Mp3 Compression/Decompression
Actual Mp3 compression program used CDex Version 1.40 Release Mp3 (lossy) Compression Down Sampling Filter banks Mp3 Decompression Up Sampling Reconstruction Filter

Detection and Estimation Performance
Bad News Correlation Coefficients MSE Possible Problems with Mp3 Compression Down Sampling Watermark information is lost Quantization Noise Watermark information absorbs into the quantization noise Lossy Compression 11:1 Compression Rate

How to Improve its Resilience?
Alternative Approaches Synchronization of the music data Time shift in Mp3 compression? Storing watermark in certain frequencies (where less quantization occurs) Error Coding Hamming Reed-Solomon

Conclusion Wave to Wave Format Mp3 Compression
Very good performance (even for 8-bit wave files) SNR is very low. Music Integrity is excellent Mp3 Compression Very bad performance Alternative methods need to be found! Need a greater understanding of current Mp3 Compression Algorithms

References [1] Araki and others. Suband Based Blind Source Separation with Appropriate Processing for Each Frequency Band. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003. [2] Hoyer and Hyvarinen. Independent Component Analysis Applied to Feature Extraction from Colour and Stereo Images. August 2000. [3] Hyvarinen, Aapo. The Fixed-Point Algorithm and Maximum Likelihood Estimation for Independent Component Analysis. [4] Hyvarinen, Aapo. Fast and Robust Fixed-Point Algorithms for Independent Component Analysis. April 1999. [5] Hyvarinen and Oja. Independent Component Analysis: A Tutorial. April 1999. [6] Introduction to Blind Source Separation. [7] Liu and others. A Digital Watermarking Scheme based on ICA Detection. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003. [8] Mitra, Sanjit K. Digital Signal Processing: Second Addition. McGraw Hill, 1998. [9] Shen and others. A Method for Digital Image Watermarking Using ICA. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.