Download presentation

Presentation is loading. Please wait.

Published byPercival Price Modified about 1 year ago

1
ASL2TXT Converting sign language gestures from digital images to text George Corser

2
Presentation Overview Concept Foundation: Barkoky & Charkari (2011) – Segmentation – Thinning My Contribution: Corser (2012) – Segmentation (similar to Barkoky) – CED: Canny Edge Dilation (Minus Errors) – Assumption: User trains his own phone

3
Concept Deaf and hearing people talking on the phone, each using their natural language Sign-activated commands like voice-activated

4
Situation: Drive Thru Window 1. Deaf person signs order 2. Phone speaks order 3. Confirmation on screen Think: Stephen Hawking

5
Process Flow Requires several conversion processes Many have been accomplished Remaining: ASL2TXT

6
Goal: Find an Algorithm Find an image processing algorithm that recognizes ASL alphabet Web site

7
Barkoky: Segmentation & Thinning

8
Barkoky Process Segmentation 1.Capture RGB image 2.Rescale 3.Extract using colors 4.Reduce noise 5.Crop at wrist 6.Result: hand segment Thinning 7.Input: hand segment 8.Apply thinning 9.Find endpoints, joints 10.Calculate lengths 11.Clean short lengths 12.Identify gesture by counting endpoints

9
1. Capture RGB Image 2. Rescale % ---------- 1. Capture RGB image a = imread('DSC04926.JPG'); figure('Name','RGB image'),imshow(a); % ---------- 2. Rescale image to 205x154 a10 = imresize(a, 0.1); figure('Name','Rescaled image'),imshow(a10);

10
3. Extract Hand Using Colors % ---------- 3. Extract hand using color abw10 = zeros(205,154,1); for i=1:205, for j=1:154, if a10(i,j,2)<140 && a10(i,j,3)<100, abw10(i,j,1)=255; end; figure('Name','Extracted'),imshow(abw10); Note: Color threshold code differs from Barkoky

11
Colors: Training Set Histograms

12
Colors: Training Set (2) Excel

13
Colors: Test Set Histograms

14
4. Reduce Noise % ---------- 4. Reduce noise for i=2:204, for j=1:154, if abw10(i-1,j,1)==0 if abw10(i+1,j,1)==0, abw10(i,j,1)=0; end; end; if abw10(i-1,j,1)==255 if abw10(i+1,j,1)==255, abw10(i,j,1)=0; end; end; end; abw10 = imfill(abw10,'holes');

15
5. Identify Wrist Position % ---------- 5. Identify wrist position for i=204:-1:1, for j=1:154, if abw10(i,j,1)==255, break; end; end; if j ~= 154 && abw10(i+1,j,1)~=255, wristi=i+1; wristj=j+1; break; end;

16
Wrist Detection Algorithm searches bottom-to-top of image Finds a leftmost white pixel above black pixel Sets wrist position SE of found white pixel

17
Corser: Segmentation & CED Segmentation (similar to Barkoky) – Color threshold technique slightly different – American Sign Language (ASL) alphabet, not Persian Sign Language (PSL) numbers Image Comparison: Tried Several Methods – Full Threshold (Minus Errors) – Diced Segments (Minus Errors) – Endpoint Count Difference – CED: Canny Edge Dilation

18
ASL Training Set

19
ASL Test Set MATLAB

20
A

21
A

22
B

23
B

24
C

25
C

26
D

27
D

28
E

29
E

30
F

31
F

32
G

33
G

34
H

35
H

36
I

37
I

38
J

39
J

40
K

41
K

42
L

43
L

44
M

45
M

46
N

47
N

48
O

49
O

50
P

51
P

52
Q

53
Q

54
R

55
R

56
S

57
S

58
T

59
T

60
U

61
U

62
V

63
V

64
W

65
W

66
X

67
X

68
Y

69
Y

70
Z

71
Z

72
Z

73
Hybrid Algorithm Example % ---------- MATLAB Code ------------------- matchtotal = 0; if abs(x10range - x20range) < 20, matchtotal = matchtotal + 10; end; if abs(y10range - y20range) < 20, matchtotal = matchtotal + 11; end; matchtotal = matchtotal - abs(h10 - h20); % ----- h10, h20 are vector magnitudes -----

74
Erosion Subtraction

75
Canny Edge

76
Canny Edge Dilation Code % ---------- MATLAB Code ------------------- se = strel('disk',5); a10 = edge(a10,'canny'); a20 = edge(a20,'canny'); a10 = imdilate(a10,se); a20 = imdilate(a20,se); % ----- Then calculate matches minus errors

77
Experimental Results TechniqueCorrect Full Threshold (Minus Errors)19% (27%) Diced Segments (Minus Errors)23% (27%) Barkoky Endpoint Count Diff.8% Hybrid - Height/Width/Endpoints19% Erosion Subtraction15% Canny Edge Dilation (Minus Errors)12% (35%)

78
Disadvantages Dependent on lighting conditions Fails with flesh-tone backgrounds Requires calibration to a specific user Limited applications: text messaging, activation (“sign” similar to voice activation) ASL numbers (A=10, D=1, O=0, V=2, W=6) Alphabet is tiny portion of full translation: complete translation maybe many years away

79
Future Work Barkoky claims flesh tones can be detected, but I have yet to replicate (even Barkoky changed his color detection scheme) Could write letter-by-letter algorithm Could use range camera to compute distance of finger instead of shape of hand Motion analysis or edge count Many possibilities… we’ve only just begun! http://www.youtube.com/watch?v=__VQX2Xn7tI

80
The End

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google