Presentation is loading. Please wait.

Presentation is loading. Please wait.

Converting sign language gestures from digital images to text

Similar presentations

Presentation on theme: "Converting sign language gestures from digital images to text"— Presentation transcript:

1 Converting sign language gestures from digital images to text
ASL2TXT Converting sign language gestures from digital images to text George Corser

2 Presentation Overview
Concept Foundation: Barkoky & Charkari (2011) Segmentation Thinning My Contribution: Corser (2012) Segmentation (similar to Barkoky) CED: Canny Edge Dilation (Minus Errors) Assumption: User trains his own phone

3 Concept Deaf and hearing people talking on the phone, each using their natural language Sign-activated commands like voice-activated

4 Situation: Drive Thru Window
Think: Stephen Hawking Deaf person signs order Phone speaks order Confirmation on screen

5 Process Flow Requires several conversion processes
Many have been accomplished Remaining: ASL2TXT

6 Goal: Find an Algorithm
Find an image processing algorithm that recognizes ASL alphabet = A Web site

7 Barkoky: Segmentation & Thinning
Barkoky counts endpoints to determine sign (doesn’t work for ASL)

8 Barkoky Process Segmentation Thinning Capture RGB image Rescale
Extract using colors Reduce noise Crop at wrist Result: hand segment Input: hand segment Apply thinning Find endpoints, joints Calculate lengths Clean short lengths Identify gesture by counting endpoints

9 1. Capture RGB Image 2. Rescale
% Capture RGB image a = imread('DSC04926.JPG'); figure('Name','RGB image'),imshow(a); % Rescale image to 205x154 a10 = imresize(a, 0.1); figure('Name','Rescaled image'),imshow(a10);

10 3. Extract Hand Using Colors
% Extract hand using color abw10 = zeros(205,154,1); for i=1:205, for j=1:154, if a10(i,j,2)<140 && a10(i,j,3)<100, abw10(i,j,1)=255; end; figure('Name','Extracted'),imshow(abw10); Note: Color threshold code differs from Barkoky

11 Colors: Training Set Histograms

12 Colors: Training Set (2)
Red Green Blue Excel

13 Colors: Test Set Histograms

14 4. Reduce Noise % Reduce noise for i=2:204, for j=1:154, if abw10(i-1,j,1)==0 if abw10(i+1,j,1)==0, abw10(i,j,1)=0; end; end; if abw10(i-1,j,1)==255 if abw10(i+1,j,1)==255, end; end; abw10 = imfill(abw10,'holes');

15 5. Identify Wrist Position
% Identify wrist position for i=204:-1:1, for j=1:154, if abw10(i,j,1)==255, break; end; end; if j ~= 154 && abw10(i+1,j,1)~=255, wristi=i+1; wristj=j+1; break;

16 Wrist Detection Algorithm searches bottom-to-top of image
Finds a leftmost white pixel above black pixel Sets wrist position SE of found white pixel

17 Corser: Segmentation & CED
Segmentation (similar to Barkoky) Color threshold technique slightly different American Sign Language (ASL) alphabet, not Persian Sign Language (PSL) numbers Image Comparison: Tried Several Methods Full Threshold (Minus Errors) Diced Segments (Minus Errors) Endpoint Count Difference CED: Canny Edge Dilation

18 ASL Training Set Hit-or-miss: 23% Barkoky: 8%

19 ASL Test Set MATLAB

20 A

21 A

22 B

23 B

24 C

25 C

26 D

27 D

28 E

29 E

30 F

31 F

32 G

33 G

34 H

35 H

36 I

37 I

38 J

39 J

40 K

41 K

42 L

43 L

44 M

45 M

46 N

47 N

48 O

49 O

50 P

51 P

52 Q

53 Q

54 R

55 R

56 S

57 S

58 T

59 T

60 U

61 U

62 V

63 V

64 W

65 W

66 X

67 X

68 Y

69 Y

70 Z

71 Z

72 Z

73 Hybrid Algorithm Example
% MATLAB Code matchtotal = 0; if abs(x10range - x20range) < 20, matchtotal = matchtotal + 10; end; if abs(y10range - y20range) < 20, matchtotal = matchtotal + 11; matchtotal = matchtotal - abs(h10 - h20); % h10, h20 are vector magnitudes -----

74 Erosion Subtraction

75 Canny Edge

76 Canny Edge Dilation Code
% MATLAB Code se = strel('disk',5); a10 = edge(a10,'canny'); a20 = edge(a20,'canny'); a10 = imdilate(a10,se); a20 = imdilate(a20,se); % Then calculate matches minus errors

77 Experimental Results Technique Correct Full Threshold (Minus Errors)
19% (27%) Diced Segments (Minus Errors) 23% (27%) Barkoky Endpoint Count Diff. 8% Hybrid - Height/Width/Endpoints 19% Erosion Subtraction 15% Canny Edge Dilation (Minus Errors) 12% (35%)

78 Disadvantages Dependent on lighting conditions
Fails with flesh-tone backgrounds Requires calibration to a specific user Limited applications: text messaging, activation (“sign” similar to voice activation) ASL numbers (A=10, D=1, O=0, V=2, W=6) Alphabet is tiny portion of full translation: complete translation maybe many years away

79 Future Work Barkoky claims flesh tones can be detected, but I have yet to replicate (even Barkoky changed his color detection scheme) Could write letter-by-letter algorithm Could use range camera to compute distance of finger instead of shape of hand Motion analysis or edge count Many possibilities… we’ve only just begun! Cue: music

80 The End

Download ppt "Converting sign language gestures from digital images to text"

Similar presentations

Ads by Google