Download presentation

Presentation is loading. Please wait.

Published byPercival Price Modified over 2 years ago

1
**Converting sign language gestures from digital images to text**

ASL2TXT Converting sign language gestures from digital images to text George Corser

2
**Presentation Overview**

Concept Foundation: Barkoky & Charkari (2011) Segmentation Thinning My Contribution: Corser (2012) Segmentation (similar to Barkoky) CED: Canny Edge Dilation (Minus Errors) Assumption: User trains his own phone

3
Concept Deaf and hearing people talking on the phone, each using their natural language Sign-activated commands like voice-activated

4
**Situation: Drive Thru Window**

Think: Stephen Hawking Deaf person signs order Phone speaks order Confirmation on screen

5
**Process Flow Requires several conversion processes**

Many have been accomplished Remaining: ASL2TXT

6
**Goal: Find an Algorithm**

Find an image processing algorithm that recognizes ASL alphabet = A Web site

7
**Barkoky: Segmentation & Thinning**

Barkoky counts endpoints to determine sign (doesn’t work for ASL)

8
**Barkoky Process Segmentation Thinning Capture RGB image Rescale**

Extract using colors Reduce noise Crop at wrist Result: hand segment Input: hand segment Apply thinning Find endpoints, joints Calculate lengths Clean short lengths Identify gesture by counting endpoints

9
**1. Capture RGB Image 2. Rescale**

% Capture RGB image a = imread('DSC04926.JPG'); figure('Name','RGB image'),imshow(a); % Rescale image to 205x154 a10 = imresize(a, 0.1); figure('Name','Rescaled image'),imshow(a10);

10
**3. Extract Hand Using Colors**

% Extract hand using color abw10 = zeros(205,154,1); for i=1:205, for j=1:154, if a10(i,j,2)<140 && a10(i,j,3)<100, abw10(i,j,1)=255; end; figure('Name','Extracted'),imshow(abw10); Note: Color threshold code differs from Barkoky

11
**Colors: Training Set Histograms**

12
**Colors: Training Set (2)**

Red Green Blue Excel

13
**Colors: Test Set Histograms**

14
4. Reduce Noise % Reduce noise for i=2:204, for j=1:154, if abw10(i-1,j,1)==0 if abw10(i+1,j,1)==0, abw10(i,j,1)=0; end; end; if abw10(i-1,j,1)==255 if abw10(i+1,j,1)==255, end; end; abw10 = imfill(abw10,'holes');

15
**5. Identify Wrist Position**

% Identify wrist position for i=204:-1:1, for j=1:154, if abw10(i,j,1)==255, break; end; end; if j ~= 154 && abw10(i+1,j,1)~=255, wristi=i+1; wristj=j+1; break;

16
**Wrist Detection Algorithm searches bottom-to-top of image**

Finds a leftmost white pixel above black pixel Sets wrist position SE of found white pixel

17
**Corser: Segmentation & CED**

Segmentation (similar to Barkoky) Color threshold technique slightly different American Sign Language (ASL) alphabet, not Persian Sign Language (PSL) numbers Image Comparison: Tried Several Methods Full Threshold (Minus Errors) Diced Segments (Minus Errors) Endpoint Count Difference CED: Canny Edge Dilation

18
ASL Training Set Hit-or-miss: 23% Barkoky: 8%

19
ASL Test Set MATLAB

20
A

21
A

22
B

23
B

24
C

25
C

26
D

27
D

28
E

29
E

30
F

31
F

32
G

33
G

34
H

35
H

36
I

37
I

38
J

39
J

40
K

41
K

42
L

43
L

44
M

45
M

46
N

47
N

48
O

49
O

50
P

51
P

52
Q

53
Q

54
R

55
R

56
S

57
S

58
T

59
T

60
U

61
U

62
V

63
V

64
W

65
W

66
X

67
X

68
Y

69
Y

70
Z

71
Z

72
Z

73
**Hybrid Algorithm Example**

% MATLAB Code matchtotal = 0; if abs(x10range - x20range) < 20, matchtotal = matchtotal + 10; end; if abs(y10range - y20range) < 20, matchtotal = matchtotal + 11; matchtotal = matchtotal - abs(h10 - h20); % h10, h20 are vector magnitudes -----

74
Erosion Subtraction

75
Canny Edge

76
**Canny Edge Dilation Code**

% MATLAB Code se = strel('disk',5); a10 = edge(a10,'canny'); a20 = edge(a20,'canny'); a10 = imdilate(a10,se); a20 = imdilate(a20,se); % Then calculate matches minus errors

77
**Experimental Results Technique Correct Full Threshold (Minus Errors)**

19% (27%) Diced Segments (Minus Errors) 23% (27%) Barkoky Endpoint Count Diff. 8% Hybrid - Height/Width/Endpoints 19% Erosion Subtraction 15% Canny Edge Dilation (Minus Errors) 12% (35%)

78
**Disadvantages Dependent on lighting conditions**

Fails with flesh-tone backgrounds Requires calibration to a specific user Limited applications: text messaging, activation (“sign” similar to voice activation) ASL numbers (A=10, D=1, O=0, V=2, W=6) Alphabet is tiny portion of full translation: complete translation maybe many years away

79
Future Work Barkoky claims flesh tones can be detected, but I have yet to replicate (even Barkoky changed his color detection scheme) Could write letter-by-letter algorithm Could use range camera to compute distance of finger instead of shape of hand Motion analysis or edge count Many possibilities… we’ve only just begun! Cue: music

80
The End

Similar presentations

OK

ECE 8443 – Pattern Recognition EE 3512 – Signals: Continuous and Discrete Objectives: Spectrograms Revisited Feature Extraction Filter Bank Analysis EEG.

ECE 8443 – Pattern Recognition EE 3512 – Signals: Continuous and Discrete Objectives: Spectrograms Revisited Feature Extraction Filter Bank Analysis EEG.

© 2018 SlidePlayer.com Inc.

All rights reserved.

Ads by Google

Ppt on first conditional form Ppt on fibonacci series Ppt on icici bank history Ppt on balancing redox reactions Download ppt on sets for class 11 Ppt on zener diode theory Ppt on structural changes in chromosomes Ppt on nitrogen cycle and nitrogen fixation Ppt on road accidents in sri Ppt on famous indian entrepreneurs in usa