Presentation on theme: "EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003."— Presentation transcript:
EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003
Presentation Overview ICA Motivation Mathematical Formulation Fast ICA Algorithm Applications Noise Separation and Feature Extraction Digital Watermarking
Motivation for ICA Cocktail Party Problem Suppose you are in a crowded room with many people. How do you understand what any one person is saying? Separation of Independent Signals Similar to Blind Source Separation Little knowledge of the signals Access to mixed signals only
Cocktail Party Problem ICA Separation Algorithm Separation of Speech Signals Humans can separate multiple signals with only two ears/sensors ICA needs as many ears/sensors as message signals Here we assume he has four ears!
ICA Definition Mixed Signals in Matrix Notation Variable Definitions
ICA Solution Signal Separation Find using the ICA Algorithm
ICA Block Diagram (2 Signals) Signal #1Signal #2 Multiplexed Signal #2 Multiplexed Signal #1 a 11 a 12 a 21 a 22
ICA Assumption #1: Independence Probability Density Definition Expected Value Definition
ICA Assumption #2: Non-Gaussian Property of Gaussian signals Addition of two independent Gaussian random variables is another single Gaussian random variable. Information Lost! Kurtosis Function Special Case: kurt(N) = 0
Limitation #1: Scaling ICA maximizes independence between signals. Assuming the Separated Signals are Independent Scaling Factor for Signal #1 Scaling Factor for Signal #2
Limitation #2: Signal Permutations The mixing matrix and independent components are unknown.
Limitation #3: No. of Sensors Sensor Requirement The number of separated signals cannot be larger than the number of inputs. Current research is being done to reduce this constraint.
ICA Separation Technique Central Limit Theorem If two random (non-Gaussian) signals added, the resulting signal will be more Gaussian than the original two random signals ICA Separation Concept Central Limit Theorem (in Reverse) Maximizing Non-Gaussianity Results in separating the two signals
Fast ICA Algorithm Overview “Fixed-Point” Algorithm Implementation Fast ICA algorithm Extensions Algorithm Speed & Performance Currently the fastest Most Commonly Used
Fast ICA Algorithm 1. Choose a random initial weight vector. 2. Let, 3. Let, 4. Repeat until converges.
Fast ICA Extensions Preprocessing Normalize mean to zero Pre-Whitening Activation Functions g(u)=u^3 g(u)=u^2 g(u)=tanh(a1*u) g(u)=u*exp(-a2*u^2/2)
Noise Separation Example Separation of Noise Impulsive Noise Additive White Gaussian Noise Implementation Two Sensor Setup Fast ICA Algorithm
Noise Separation & Feature Extraction ICA performs well in Blind Source Separation ICA for Feature Extraction Reduce Complexity of the Neural Network Train only on the appropriate signal Detection and Estimation of Hidden Signals
Noise Detection and Estimation: Conclusion Additional Preprocessing Segmentation of Impulsive Noise (Time-Limited) Possible Inputs to the Neural Network Statistical Moments Signal Samples Possible Neural Networks Back Propagation SVM Radial Basis Functions Problem Need to train Neural Network in Matlab
Digital Watermarking of Music Motivation Popularity of Digital Storage Devices Reliable, Fast, Ease of duplication, etc. How to protect copyrighted information? Leaving digital signatures of its artist Essential Properties for Watermarking Undetectable Irremovable Resilient
Detection & Estimation of Watermarks Detection of Watermark Authenticate copyrighted music Estimation of Watermark Authenticate copyrighted music Information on artist, producer, etc.
Watermarking Model Process Mix the original musical data with watermark Keep watermark Power relatively low Ensure high quality of the watermarked music Watermark is better hidden
Popular Digital Formats Wave FormatMp3 Format Bit Rate1411 [kbps]128 [kbps] Channels22 Audio Sample Rate 44 [kHz]
Detection of Watermark Watermarking Detection Algorithm 1. Use the ICA model to randomly mix the watermark and music file. 2. Save the watermarked music in the popular *.wav format 3. Read the saved *.wave file. Separate the watermark and the music file. 4. Identify the watermark using statistical methods (mean, std, etc.) Performance Statistic Correlation Coefficient (Absolute Value)
Estimation of Watermark Watermarking Estimation Algorithm 1. Use the ICA model to randomly mix the watermark and music file. 2. Save the watermarked music in the popular *.wav format 3. Read the saved *.wave file. Separate the watermark and the music file. 4. Identify the watermark using statistical methods (mean, std, etc.) 5. Digitize the watermark signal. Performance Statistic Bit Error Rate
Is the Music Content Preserved? “Hero” – Mariah Carey Original Wave file from CD Hero.wav 29 [s] “Hero” – Mariah Carey Watermarked Wave file HeroWatermarked.wav 29 [s]
Resilience of Proposed Watermark Resiliency Previous Simulations show that wav format is resilient to 8-bit and 16-bit quantization. Can the watermark be detected after Mp3 compression and decompression?
Mp3 Compression/Decompression Actual Mp3 compression program used CDex Version 1.40 Release Mp3 (lossy) Compression Down Sampling Filter banks Mp3 Decompression Up Sampling Reconstruction Filter
Detection and Estimation Performance Bad News Correlation Coefficients 0 MSE 0.5 Possible Problems with Mp3 Compression Down Sampling Watermark information is lost Quantization Noise Watermark information absorbs into the quantization noise Lossy Compression 11:1 Compression Rate
How to Improve its Resilience? Alternative Approaches Synchronization of the music data Time shift in Mp3 compression? Storing watermark in certain frequencies (where less quantization occurs) Error Coding Hamming Reed-Solomon
Conclusion Wave to Wave Format Very good performance (even for 8-bit wave files) SNR is very low. Music Integrity is excellent Mp3 Compression Very bad performance Alternative methods need to be found! Need a greater understanding of current Mp3 Compression Algorithms
References  Araki and others. Suband Based Blind Source Separation with Appropriate Processing for Each Frequency Band. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.  Hoyer and Hyvarinen. Independent Component Analysis Applied to Feature Extraction from Colour and Stereo Images. August 2000.  Hyvarinen, Aapo. The Fixed-Point Algorithm and Maximum Likelihood Estimation for Independent Component Analysis. http://www.cis.hut.fi/~aapo/.http://www.cis.hut.fi/~aapo/  Hyvarinen, Aapo. Fast and Robust Fixed-Point Algorithms for Independent Component Analysis. http://www.cis.hut.fi/~aapo/. April 1999.http://www.cis.hut.fi/~aapo/  Hyvarinen and Oja. Independent Component Analysis: A Tutorial. http://www.cis.hut.fi/projects/ica/. April 1999. http://www.cis.hut.fi/projects/ica/  Introduction to Blind Source Separation. http://www.cnl.salk.edu/~tewon/Blind/.http://www.cnl.salk.edu/~tewon/Blind/  Liu and others. A Digital Watermarking Scheme based on ICA Detection. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.  Mitra, Sanjit K. Digital Signal Processing: Second Addition. McGraw Hill, 1998.  Shen and others. A Method for Digital Image Watermarking Using ICA. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.