Presentation on theme: "Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando."— Presentation transcript:
Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando Fujitsu Laboratories Ltd., Atsugi, Japan ) MSc in RSIP Jie Zhang s /02/06
Background Why Lip reading? Human computer interaction Automatically recognize speech contents by processing lip movement images Potential application for disable facilities What is the principle? Dealing with frame images sampled from video Lip Feature extraction Lip Feature recognition Various methodology
System overview Lip colour extraction algorithm Colour characteristics of lip feature Roughly mouth detection with colour algorithm Precise mouth position detection Eigen image Eigen template Matching algorithm Eigenwaveform recognition Mouth status Dictionary data Thresholding Training Lip images Test Lip images Colour extraction Eigenvector template build Colour extraction Eigentemplate Detection Eigenwaveform recognition
Colour extraction algorithm Colour components of face feature Strongly brightness dependence ! Colour system HSV or RGB Evenly luminance Brightness normalization Lip region colour characteristics Three classes: lip, skin, teeth Looking for the separation of spectrum components Figure 1
Colour extraction algorithm Figure 2: Brightness dependent face feature colour distribution of normalised RGB image. Well distinguished in R &G!
Colour extraction algorithm Lip Extraction with colour distribution Threshold function of R and G to separate skin and mouth Label and Extract largest bright area to appropriate size as it is deem to be mouth Remove teeth and lips with second R and G threshold function Position determination with four key points of the lips Figure 3 Lip edge is the combination of teeth and mouth cavity.
Eigentemplate method Aim precisely detect the location of lip! Creation of eigen image Use appropriate colour extracted image (trimmed image) PCA + one dimension vector for a single trimmed image Form image series matrix with x vectors
Eigentemplate method Eigentemplate Trimmed test image Recover eigenimage to template image Similarity calculation Searching and trimming the image, compare trimmed image will template. Largest similarity gives the exact location.
Eigentemplate method Figure 5 Figure 6
Eigenwaveform recognition Brief introduction to preprocessing A threshold method to detect mouth states and hence recognize particular mouth shape Define mouth state: i.e. open, wide open, close, tight closed… Aspect ratio: distance b between upper and lower edge points over distance a between left and right edge points Projection vector components: project the image of mouth and lip into eigen space through time scale. Thresholding
Eigenwaveform recognition Dictionary matching Figure 7 Small difference sw recognize the test utterance as the same as the template utterance
Summary & Analysis Lip reading system Colour extraction method roughly classify lip and other face feature Eigentemplate method precisely detect the location Eigenwaveform algorithm recognize the utterance Analysis Widely used image processing technique Hard to get high precision Difficult for language different, culture different, appearance difference user Potential problems of algorithm: brightness dependence; overlap between lip & skin; similar words; uncommon mouth shape; various speak speed and so on. Need more improvement!