Download presentation
Presentation is loading. Please wait.
Published byPercival Price Modified over 2 years ago
1
Converting sign language gestures from digital images to text
ASL2TXT Converting sign language gestures from digital images to text George Corser
2
Presentation Overview
Concept Foundation: Barkoky & Charkari (2011) Segmentation Thinning My Contribution: Corser (2012) Segmentation (similar to Barkoky) CED: Canny Edge Dilation (Minus Errors) Assumption: User trains his own phone
3
Concept Deaf and hearing people talking on the phone, each using their natural language Sign-activated commands like voice-activated
4
Situation: Drive Thru Window
Think: Stephen Hawking Deaf person signs order Phone speaks order Confirmation on screen
5
Process Flow Requires several conversion processes
Many have been accomplished Remaining: ASL2TXT
6
Goal: Find an Algorithm
Find an image processing algorithm that recognizes ASL alphabet = A Web site
7
Barkoky: Segmentation & Thinning
Barkoky counts endpoints to determine sign (doesn’t work for ASL)
8
Barkoky Process Segmentation Thinning Capture RGB image Rescale
Extract using colors Reduce noise Crop at wrist Result: hand segment Input: hand segment Apply thinning Find endpoints, joints Calculate lengths Clean short lengths Identify gesture by counting endpoints
9
1. Capture RGB Image 2. Rescale
% Capture RGB image a = imread('DSC04926.JPG'); figure('Name','RGB image'),imshow(a); % Rescale image to 205x154 a10 = imresize(a, 0.1); figure('Name','Rescaled image'),imshow(a10);
10
3. Extract Hand Using Colors
% Extract hand using color abw10 = zeros(205,154,1); for i=1:205, for j=1:154, if a10(i,j,2)<140 && a10(i,j,3)<100, abw10(i,j,1)=255; end; figure('Name','Extracted'),imshow(abw10); Note: Color threshold code differs from Barkoky
11
Colors: Training Set Histograms
12
Colors: Training Set (2)
Red Green Blue Excel
13
Colors: Test Set Histograms
14
4. Reduce Noise % Reduce noise for i=2:204, for j=1:154, if abw10(i-1,j,1)==0 if abw10(i+1,j,1)==0, abw10(i,j,1)=0; end; end; if abw10(i-1,j,1)==255 if abw10(i+1,j,1)==255, end; end; abw10 = imfill(abw10,'holes');
15
5. Identify Wrist Position
% Identify wrist position for i=204:-1:1, for j=1:154, if abw10(i,j,1)==255, break; end; end; if j ~= 154 && abw10(i+1,j,1)~=255, wristi=i+1; wristj=j+1; break;
16
Wrist Detection Algorithm searches bottom-to-top of image
Finds a leftmost white pixel above black pixel Sets wrist position SE of found white pixel
17
Corser: Segmentation & CED
Segmentation (similar to Barkoky) Color threshold technique slightly different American Sign Language (ASL) alphabet, not Persian Sign Language (PSL) numbers Image Comparison: Tried Several Methods Full Threshold (Minus Errors) Diced Segments (Minus Errors) Endpoint Count Difference CED: Canny Edge Dilation
18
ASL Training Set Hit-or-miss: 23% Barkoky: 8%
19
ASL Test Set MATLAB
20
A
21
A
22
B
23
B
24
C
25
C
26
D
27
D
28
E
29
E
30
F
31
F
32
G
33
G
34
H
35
H
36
I
37
I
38
J
39
J
40
K
41
K
42
L
43
L
44
M
45
M
46
N
47
N
48
O
49
O
50
P
51
P
52
Q
53
Q
54
R
55
R
56
S
57
S
58
T
59
T
60
U
61
U
62
V
63
V
64
W
65
W
66
X
67
X
68
Y
69
Y
70
Z
71
Z
72
Z
73
Hybrid Algorithm Example
% MATLAB Code matchtotal = 0; if abs(x10range - x20range) < 20, matchtotal = matchtotal + 10; end; if abs(y10range - y20range) < 20, matchtotal = matchtotal + 11; matchtotal = matchtotal - abs(h10 - h20); % h10, h20 are vector magnitudes -----
74
Erosion Subtraction
75
Canny Edge
76
Canny Edge Dilation Code
% MATLAB Code se = strel('disk',5); a10 = edge(a10,'canny'); a20 = edge(a20,'canny'); a10 = imdilate(a10,se); a20 = imdilate(a20,se); % Then calculate matches minus errors
77
Experimental Results Technique Correct Full Threshold (Minus Errors)
19% (27%) Diced Segments (Minus Errors) 23% (27%) Barkoky Endpoint Count Diff. 8% Hybrid - Height/Width/Endpoints 19% Erosion Subtraction 15% Canny Edge Dilation (Minus Errors) 12% (35%)
78
Disadvantages Dependent on lighting conditions
Fails with flesh-tone backgrounds Requires calibration to a specific user Limited applications: text messaging, activation (“sign” similar to voice activation) ASL numbers (A=10, D=1, O=0, V=2, W=6) Alphabet is tiny portion of full translation: complete translation maybe many years away
79
Future Work Barkoky claims flesh tones can be detected, but I have yet to replicate (even Barkoky changed his color detection scheme) Could write letter-by-letter algorithm Could use range camera to compute distance of finger instead of shape of hand Motion analysis or edge count Many possibilities… we’ve only just begun! Cue: music
80
The End
Similar presentations
© 2018 SlidePlayer.com Inc.
All rights reserved.
Ppt on first conditional form Ppt on fibonacci series Ppt on icici bank history Ppt on balancing redox reactions Download ppt on sets for class 11 Ppt on zener diode theory Ppt on structural changes in chromosomes Ppt on nitrogen cycle and nitrogen fixation Ppt on road accidents in sri Ppt on famous indian entrepreneurs in usa