1. EE645: Independent Component Analysis
Elliot Taniguchi
Advisor: Dr. Kuh
May 16, 2003

2. Presentation Overview
ICA Motivation
Mathematical Formulation
Fast ICA Algorithm
Applications
Noise Separation and Feature Extraction
Digital Watermarking

3. Motivation for ICA Cocktail Party Problem
Suppose you are in a crowded room with many people. How do you understand what any one person is saying?
Separation of Independent Signals
Similar to Blind Source Separation
Little knowledge of the signals
Access to mixed signals only

4. Cocktail Party Problem??

5. Cocktail Party Problem
ICA Separation Algorithm
Separation of Speech Signals
Humans can separate multiple signals with only two ears/sensors
ICA needs as many ears/sensors as message signals
Here we assume he has four ears!

6. Recovered Messages

7. Mathematical Formulation Overview
ICA Definition
ICA Assumptions
Independent Signals
Non-Gaussian
ICA Limitations
Scaling
Permutations
No. of Sensors

8. ICA Definition
Mixed Signals in Matrix Notation
Variable Definitions

9. ICA Solution
Signal Separation
Find using the ICA Algorithm

10. ICA Block Diagram (2 Signals)
Multiplexed
Signal #1
Multiplexed
Signal #2

11. ICA Assumption #1: Independence
Probability Density Definition
Expected Value Definition

12. ICA Assumption #2: Non-Gaussian
Property of Gaussian signals
Addition of two independent Gaussian random variables is another single Gaussian random variable.
Information Lost!
Kurtosis Function
Special Case: kurt(N) = 0

13. Limitation #1: Scaling ICA maximizes independence between signals.
Scaling Factor
for Signal #1
Assuming the
Separated Signals
are Independent
Scaling Factor
for Signal #2

14. Limitation #2: Signal Permutations
The mixing matrix and independent components are unknown.

15. Limitation #3: No. of Sensors
Sensor Requirement
The number of separated signals cannot be larger than the number of inputs.
Current research is being done to reduce this constraint.

16. ICA Separation Technique
Central Limit Theorem
If two random (non-Gaussian) signals added, the resulting signal will be more Gaussian than the original two random signals
ICA Separation Concept
Central Limit Theorem (in Reverse)
Maximizing Non-Gaussianity
Results in separating the two signals

17. Fast ICA Algorithm Overview
“Fixed-Point” Algorithm
Implementation
Fast ICA algorithm
Extensions
Algorithm Speed & Performance
Currently the fastest
Most Commonly Used

18. Fast ICA Algorithm Choose a random initial weight vector. Let,
Repeat until converges.

19. Fast ICA Extensions Preprocessing Activation Functions
Normalize mean to zero
Pre-Whitening
Activation Functions
g(u)=u^3
g(u)=u^2
g(u)=tanh(a1*u)
g(u)=u*exp(-a2*u^2/2)

20. Noise Separation Example
Separation of Noise
Impulsive Noise
Additive White Gaussian Noise
Implementation
Two Sensor Setup
Fast ICA Algorithm

21. Noise Generation
AWGN
Gaussian R.V.
Impulsive Noise
Poisson R.V.

22. Physical Setup
Impulsive
Noise
Gaussian
Noise
Receiver #1
Receiver #2

23. Noise Separation Example

24. Noise Separation Example

25. Noise Separation Example

26. Noise Separation Performance

27. Noise Separation Performance

28. Noise Separation & Feature Extraction
ICA performs well in Blind Source Separation
ICA for Feature Extraction
Reduce Complexity of the Neural Network
Train only on the appropriate signal
Detection and Estimation of Hidden Signals

29. Noise Detection and Estimation: Method #1
Fast
ICA
N.N.
(BP,
RBF,
SVM)
Sensor #1
Feature #1
Sensor #2
Feature #2
Sensor #3
Feature #3
Sensor #4
Feature #4

30. Noise Detection and Estimation: Method #2
Fast
ICA
N.N.
(BP,
RBF,
SVM)
Sensor #1
Feature #1
Decision
Device
Sensor #2
Feature #2
Sensor #3
Feature #3
Sensor #4
Feature #4

31. Noise Detection and Estimation: Conclusion
Additional Preprocessing
Segmentation of Impulsive Noise (Time-Limited)
Possible Inputs to the Neural Network
Statistical Moments
Signal Samples
Possible Neural Networks
Back Propagation
SVM
Radial Basis Functions
Problem
Need to train Neural Network in Matlab

32. Digital Watermarking of Music
Motivation
Popularity of Digital Storage Devices
Reliable, Fast, Ease of duplication, etc.
How to protect copyrighted information?
Leaving digital signatures of its artist
Essential Properties for Watermarking
Undetectable
Irremovable
Resilient

33. Detection & Estimation of Watermarks
Detection of Watermark
Authenticate copyrighted music
Estimation of Watermark
Information on artist, producer, etc.

34. Watermarking Model Process
Mix the original musical data with watermark
Keep watermark Power relatively low
Ensure high quality of the watermarked music
Watermark is better hidden

35. Popular Digital Formats
Wave Format
Mp3 Format
Bit Rate
1411 [kbps]
128 [kbps]
Channels 2
Audio Sample Rate
44 [kHz]

36. Detection of Watermark
Watermarking Detection Algorithm
Use the ICA model to randomly mix the watermark and music file.
Save the watermarked music in the popular *.wav format
Read the saved *.wave file. Separate the watermark and the music file.
Identify the watermark using statistical methods (mean, std, etc.)
Performance Statistic
Correlation Coefficient (Absolute Value)

37. Detection Performance

38. Estimation of Watermark
Watermarking Estimation Algorithm
Use the ICA model to randomly mix the watermark and music file.
Save the watermarked music in the popular *.wav format
Read the saved *.wave file. Separate the watermark and the music file.
Identify the watermark using statistical methods (mean, std, etc.)
Digitize the watermark signal.
Performance Statistic
Bit Error Rate

39. Estimation Performance

40. Is the Music Content Preserved?
“Hero” – Mariah Carey
Original Wave file from CD
Hero.wav
29 [s]
“Hero” – Mariah Carey
Watermarked Wave file
HeroWatermarked.wav
29 [s]

41. Resilience of Proposed Watermark
Resiliency
Previous Simulations show that wav format is resilient to 8-bit and 16-bit quantization.
Can the watermark be detected after Mp3 compression and decompression?

42. Mp3 Compression/Decompression
Actual Mp3 compression program used
CDex Version 1.40 Release
Mp3 (lossy) Compression
Down Sampling
Filter banks
Mp3 Decompression
Up Sampling
Reconstruction Filter

43. Detection and Estimation Performance
Bad News
Correlation Coefficients
MSE
Possible Problems with Mp3 Compression
Down Sampling
Watermark information is lost
Quantization Noise
Watermark information absorbs into the quantization noise
Lossy Compression
11:1 Compression Rate

44. How to Improve its Resilience?
Alternative Approaches
Synchronization of the music data
Time shift in Mp3 compression?
Storing watermark in certain frequencies (where less quantization occurs)
Error Coding
Hamming
Reed-Solomon

45. Conclusion Wave to Wave Format Mp3 Compression
Very good performance (even for 8-bit wave files)
SNR is very low.
Music Integrity is excellent
Mp3 Compression
Very bad performance
Alternative methods need to be found!
Need a greater understanding of current Mp3 Compression Algorithms

46. References
[1] Araki and others. Suband Based Blind Source Separation with Appropriate Processing for Each Frequency Band. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.
[2] Hoyer and Hyvarinen. Independent Component Analysis Applied to Feature Extraction from Colour and Stereo Images. August 2000.
[3] Hyvarinen, Aapo. The Fixed-Point Algorithm and Maximum Likelihood Estimation for Independent Component Analysis.
[4] Hyvarinen, Aapo. Fast and Robust Fixed-Point Algorithms for Independent Component Analysis. April 1999.
[5] Hyvarinen and Oja. Independent Component Analysis: A Tutorial. April 1999.
[6] Introduction to Blind Source Separation.
[7] Liu and others. A Digital Watermarking Scheme based on ICA Detection. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.
[8] Mitra, Sanjit K. Digital Signal Processing: Second Addition. McGraw Hill, 1998.
[9] Shen and others. A Method for Digital Image Watermarking Using ICA. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.