1. Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando Fujitsu Laboratories Ltd., Atsugi, 243-0197 Japan ) MSc in RSIP Jie Zhang s0567061 10/02/06

2. Background  Why Lip reading?  Human computer interaction  Automatically recognize speech contents by processing lip movement images  Potential application for disable facilities  What is the principle?  Dealing with frame images sampled from video  Lip Feature extraction  Lip Feature recognition  Various methodology

3. System overview  Lip colour extraction algorithm  Colour characteristics of lip feature  Roughly mouth detection with colour algorithm  Precise mouth position detection  Eigen image  Eigen template  Matching algorithm  Eigenwaveform recognition  Mouth status  Dictionary data  Thresholding Training Lip images Test Lip images Colour extraction Eigenvector template build Colour extraction Eigentemplate Detection Eigenwaveform recognition

4. Colour extraction algorithm  Colour components of face feature Strongly brightness dependence !  Colour system  HSV or RGB  Evenly luminance  Brightness normalization  Lip region colour characteristics  Three classes: lip, skin, teeth  Looking for the separation of spectrum components Figure 1

5. Colour extraction algorithm Figure 2: Brightness dependent face feature colour distribution of normalised RGB image. Well distinguished in R &G!

6. Colour extraction algorithm  Lip Extraction with colour distribution  Threshold function of R and G to separate skin and mouth  Label and Extract largest bright area to appropriate size as it is deem to be mouth  Remove teeth and lips with second R and G threshold function  Position determination with four key points of the lips Figure 3 Lip edge is the combination of teeth and mouth cavity.

7. Eigentemplate method  Aim ----- precisely detect the location of lip!  Creation of eigen image  Use appropriate colour extracted image (trimmed image)  PCA + one dimension vector for a single trimmed image  Form image series matrix with x vectors

8. Eigentemplate method  Calculate eigenvector  Convert eigenvectors to eigenimage

9. Eigentemplate method Figure 4

10. Eigentemplate method  Eigentemplate  Trimmed test image  Recover eigenimage to template image  Similarity calculation Searching and trimming the image, compare trimmed image will template. Largest similarity gives the exact location.

11. Eigentemplate method Figure 5 Figure 6

12. Eigenwaveform recognition  Brief introduction to preprocessing  A threshold method to detect mouth states and hence recognize particular mouth shape  Define mouth state: i.e. open, wide open, close, tight closed…  Aspect ratio: distance b between upper and lower edge points over distance a between left and right edge points  Projection vector components: project the image of mouth and lip into eigen space through time scale.  Thresholding

13. Eigenwaveform recognition  Dictionary matching Figure 7 Small difference sw recognize the test utterance as the same as the template utterance

14. Summary & Analysis  Lip reading system  Colour extraction method roughly classify lip and other face feature  Eigentemplate method precisely detect the location  Eigenwaveform algorithm recognize the utterance  Analysis  Widely used image processing technique  Hard to get high precision  Difficult for language different, culture different, appearance difference user  Potential problems of algorithm: brightness dependence; overlap between lip & skin; similar words; uncommon mouth shape; various speak speed and so on.  Need more improvement!

15. That’s all! Thank you very much! Question?