Presentation is loading. Please wait.

Presentation is loading. Please wait.

ASL2TXT Converting sign language gestures from digital images to text George Corser.

Similar presentations

Presentation on theme: "ASL2TXT Converting sign language gestures from digital images to text George Corser."— Presentation transcript:

1 ASL2TXT Converting sign language gestures from digital images to text George Corser

2 Presentation Overview Concept Foundation: Barkoky & Charkari (2011) – Segmentation – Thinning My Contribution: Corser (2012) – Segmentation (similar to Barkoky) – CED: Canny Edge Dilation (Minus Errors) – Assumption: User trains his own phone

3 Concept Deaf and hearing people talking on the phone, each using their natural language Sign-activated commands like voice-activated

4 Situation: Drive Thru Window 1. Deaf person signs order 2. Phone speaks order 3. Confirmation on screen Think: Stephen Hawking

5 Process Flow Requires several conversion processes Many have been accomplished Remaining: ASL2TXT

6 Goal: Find an Algorithm Find an image processing algorithm that recognizes ASL alphabet Web site

7 Barkoky: Segmentation & Thinning

8 Barkoky Process Segmentation 1.Capture RGB image 2.Rescale 3.Extract using colors 4.Reduce noise 5.Crop at wrist 6.Result: hand segment Thinning 7.Input: hand segment 8.Apply thinning 9.Find endpoints, joints 10.Calculate lengths 11.Clean short lengths 12.Identify gesture by counting endpoints

9 1. Capture RGB Image 2. Rescale % Capture RGB image a = imread('DSC04926.JPG'); figure('Name','RGB image'),imshow(a); % Rescale image to 205x154 a10 = imresize(a, 0.1); figure('Name','Rescaled image'),imshow(a10);

10 3. Extract Hand Using Colors % Extract hand using color abw10 = zeros(205,154,1); for i=1:205, for j=1:154, if a10(i,j,2)<140 && a10(i,j,3)<100, abw10(i,j,1)=255; end; figure('Name','Extracted'),imshow(abw10); Note: Color threshold code differs from Barkoky

11 Colors: Training Set Histograms

12 Colors: Training Set (2) Excel

13 Colors: Test Set Histograms

14 4. Reduce Noise % Reduce noise for i=2:204, for j=1:154, if abw10(i-1,j,1)==0 if abw10(i+1,j,1)==0, abw10(i,j,1)=0; end; end; if abw10(i-1,j,1)==255 if abw10(i+1,j,1)==255, abw10(i,j,1)=0; end; end; end; abw10 = imfill(abw10,'holes');

15 5. Identify Wrist Position % Identify wrist position for i=204:-1:1, for j=1:154, if abw10(i,j,1)==255, break; end; end; if j ~= 154 && abw10(i+1,j,1)~=255, wristi=i+1; wristj=j+1; break; end;

16 Wrist Detection Algorithm searches bottom-to-top of image Finds a leftmost white pixel above black pixel Sets wrist position SE of found white pixel

17 Corser: Segmentation & CED Segmentation (similar to Barkoky) – Color threshold technique slightly different – American Sign Language (ASL) alphabet, not Persian Sign Language (PSL) numbers Image Comparison: Tried Several Methods – Full Threshold (Minus Errors) – Diced Segments (Minus Errors) – Endpoint Count Difference – CED: Canny Edge Dilation

18 ASL Training Set

19 ASL Test Set MATLAB

20 A

21 A

22 B

23 B

24 C

25 C

26 D

27 D

28 E

29 E

30 F

31 F

32 G

33 G

34 H

35 H

36 I

37 I

38 J

39 J

40 K

41 K

42 L

43 L

44 M

45 M

46 N

47 N

48 O

49 O

50 P

51 P

52 Q

53 Q

54 R

55 R

56 S

57 S

58 T

59 T

60 U

61 U

62 V

63 V

64 W

65 W

66 X

67 X

68 Y

69 Y

70 Z

71 Z

72 Z

73 Hybrid Algorithm Example % MATLAB Code matchtotal = 0; if abs(x10range - x20range) < 20, matchtotal = matchtotal + 10; end; if abs(y10range - y20range) < 20, matchtotal = matchtotal + 11; end; matchtotal = matchtotal - abs(h10 - h20); % h10, h20 are vector magnitudes -----

74 Erosion Subtraction

75 Canny Edge

76 Canny Edge Dilation Code % MATLAB Code se = strel('disk',5); a10 = edge(a10,'canny'); a20 = edge(a20,'canny'); a10 = imdilate(a10,se); a20 = imdilate(a20,se); % Then calculate matches minus errors

77 Experimental Results TechniqueCorrect Full Threshold (Minus Errors)19% (27%) Diced Segments (Minus Errors)23% (27%) Barkoky Endpoint Count Diff.8% Hybrid - Height/Width/Endpoints19% Erosion Subtraction15% Canny Edge Dilation (Minus Errors)12% (35%)

78 Disadvantages Dependent on lighting conditions Fails with flesh-tone backgrounds Requires calibration to a specific user Limited applications: text messaging, activation (“sign” similar to voice activation) ASL numbers (A=10, D=1, O=0, V=2, W=6) Alphabet is tiny portion of full translation: complete translation maybe many years away

79 Future Work Barkoky claims flesh tones can be detected, but I have yet to replicate (even Barkoky changed his color detection scheme) Could write letter-by-letter algorithm Could use range camera to compute distance of finger instead of shape of hand Motion analysis or edge count Many possibilities… we’ve only just begun!

80 The End

Download ppt "ASL2TXT Converting sign language gestures from digital images to text George Corser."

Similar presentations

Ads by Google