Presentation is loading. Please wait.

Presentation is loading. Please wait.

ASL2TXT Converting sign language gestures from digital images to text George Corser.

Similar presentations


Presentation on theme: "ASL2TXT Converting sign language gestures from digital images to text George Corser."— Presentation transcript:

1 ASL2TXT Converting sign language gestures from digital images to text George Corser

2 Presentation Overview Concept Foundation: Barkoky & Charkari (2011) – Segmentation – Thinning My Contribution: Corser (2012) – Segmentation (similar to Barkoky) – CED: Canny Edge Dilation (Minus Errors) – Assumption: User trains his own phone

3 Concept Deaf and hearing people talking on the phone, each using their natural language Sign-activated commands like voice-activated

4 Situation: Drive Thru Window 1. Deaf person signs order 2. Phone speaks order 3. Confirmation on screen Think: Stephen Hawking

5 Process Flow Requires several conversion processes Many have been accomplished Remaining: ASL2TXT

6 Goal: Find an Algorithm Find an image processing algorithm that recognizes ASL alphabet Web site

7 Barkoky: Segmentation & Thinning

8 Barkoky Process Segmentation 1.Capture RGB image 2.Rescale 3.Extract using colors 4.Reduce noise 5.Crop at wrist 6.Result: hand segment Thinning 7.Input: hand segment 8.Apply thinning 9.Find endpoints, joints 10.Calculate lengths 11.Clean short lengths 12.Identify gesture by counting endpoints

9 1. Capture RGB Image 2. Rescale % ---------- 1. Capture RGB image a = imread('DSC04926.JPG'); figure('Name','RGB image'),imshow(a); % ---------- 2. Rescale image to 205x154 a10 = imresize(a, 0.1); figure('Name','Rescaled image'),imshow(a10);

10 3. Extract Hand Using Colors % ---------- 3. Extract hand using color abw10 = zeros(205,154,1); for i=1:205, for j=1:154, if a10(i,j,2)<140 && a10(i,j,3)<100, abw10(i,j,1)=255; end; figure('Name','Extracted'),imshow(abw10); Note: Color threshold code differs from Barkoky

11 Colors: Training Set Histograms

12 Colors: Training Set (2) Excel

13 Colors: Test Set Histograms

14 4. Reduce Noise % ---------- 4. Reduce noise for i=2:204, for j=1:154, if abw10(i-1,j,1)==0 if abw10(i+1,j,1)==0, abw10(i,j,1)=0; end; end; if abw10(i-1,j,1)==255 if abw10(i+1,j,1)==255, abw10(i,j,1)=0; end; end; end; abw10 = imfill(abw10,'holes');

15 5. Identify Wrist Position % ---------- 5. Identify wrist position for i=204:-1:1, for j=1:154, if abw10(i,j,1)==255, break; end; end; if j ~= 154 && abw10(i+1,j,1)~=255, wristi=i+1; wristj=j+1; break; end;

16 Wrist Detection Algorithm searches bottom-to-top of image Finds a leftmost white pixel above black pixel Sets wrist position SE of found white pixel

17 Corser: Segmentation & CED Segmentation (similar to Barkoky) – Color threshold technique slightly different – American Sign Language (ASL) alphabet, not Persian Sign Language (PSL) numbers Image Comparison: Tried Several Methods – Full Threshold (Minus Errors) – Diced Segments (Minus Errors) – Endpoint Count Difference – CED: Canny Edge Dilation

18 ASL Training Set

19 ASL Test Set MATLAB

20 A

21 A

22 B

23 B

24 C

25 C

26 D

27 D

28 E

29 E

30 F

31 F

32 G

33 G

34 H

35 H

36 I

37 I

38 J

39 J

40 K

41 K

42 L

43 L

44 M

45 M

46 N

47 N

48 O

49 O

50 P

51 P

52 Q

53 Q

54 R

55 R

56 S

57 S

58 T

59 T

60 U

61 U

62 V

63 V

64 W

65 W

66 X

67 X

68 Y

69 Y

70 Z

71 Z

72 Z

73 Hybrid Algorithm Example % ---------- MATLAB Code ------------------- matchtotal = 0; if abs(x10range - x20range) < 20, matchtotal = matchtotal + 10; end; if abs(y10range - y20range) < 20, matchtotal = matchtotal + 11; end; matchtotal = matchtotal - abs(h10 - h20); % ----- h10, h20 are vector magnitudes -----

74 Erosion Subtraction

75 Canny Edge

76 Canny Edge Dilation Code % ---------- MATLAB Code ------------------- se = strel('disk',5); a10 = edge(a10,'canny'); a20 = edge(a20,'canny'); a10 = imdilate(a10,se); a20 = imdilate(a20,se); % ----- Then calculate matches minus errors

77 Experimental Results TechniqueCorrect Full Threshold (Minus Errors)19% (27%) Diced Segments (Minus Errors)23% (27%) Barkoky Endpoint Count Diff.8% Hybrid - Height/Width/Endpoints19% Erosion Subtraction15% Canny Edge Dilation (Minus Errors)12% (35%)

78 Disadvantages Dependent on lighting conditions Fails with flesh-tone backgrounds Requires calibration to a specific user Limited applications: text messaging, activation (“sign” similar to voice activation) ASL numbers (A=10, D=1, O=0, V=2, W=6) Alphabet is tiny portion of full translation: complete translation maybe many years away

79 Future Work Barkoky claims flesh tones can be detected, but I have yet to replicate (even Barkoky changed his color detection scheme) Could write letter-by-letter algorithm Could use range camera to compute distance of finger instead of shape of hand Motion analysis or edge count Many possibilities… we’ve only just begun! http://www.youtube.com/watch?v=__VQX2Xn7tI

80 The End


Download ppt "ASL2TXT Converting sign language gestures from digital images to text George Corser."

Similar presentations


Ads by Google