1. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Sign Language Recognition using Webcams

2. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Overview Average person’s typing speed  Composing: ~19 words per minute  Transcribing: ~33 words per minute Sign speaker  Full sign language: ~200 words per minute  Spelling out: estimate: 50 words per minute  Up to 3x faster

3. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Purpose and Scope Native signers can input faster Benefits:  Hearing & speaking disabled  Sign interpreters Just letters & numbers for now  Additional complexity too much to handle  Would require smaller distinctions

4. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Research Related projects  Using mechanical gloves, colored gloves  Tracking body parts  Neural network-based application Still images: 92% accuracy Motion: less than 50% accuracy  Feature vector-based application Also about 90% accuracy on stills No motion tests

5. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 More Research Image techniques:  Edge detection (Robert’s Cross)‏  Line detection (Hough transform)‏  Line interpretation methods Chaining groups of lines Macro-scale templates  Residual math Memory management

6. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Testing Model Human interaction necessary General testing model: bhood@testing: ~/syslab-tech $ \ >./main images/hand.png [DEBUG] Edge detect time: 29 ms Errors: 0 Warnings: 0

7. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Program Architecture Webcam capture Edge detection Line detection Interpretation Attribute matching IMAGE FEATURE OUTLINE LINE LIST FINGER POSITONS SERVER PROCESS

8. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Edge Detection Results Results:  Outlines the important edges and not much besides  Robert’s Cross balances detection of major and minor lines Original image (800 x 703)‏ Final image (800 x 703)‏

9. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Cropping Results Original (800 x 703)‏ Result (633 x 645)‏ Remove useless rows & columns with no features Better contrast Very large optimization  Memory savings  Area difference means order n 2

10. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Line detection Finished!  Recently finished tweaking sensitivities  Still a few potential memory issues

11. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Line Grouping Part of line detection Large optimization  Iterate over an order of magnitude fewer items  Easier to handle, more pronounced trends Examples of line groups, called “chains”

12. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Line Interpretation Chaining groups of lines Templates  Generation  Template-based comparison Line residuals  Use point coordinate averages  Calculate average offset from average  Easy to find height of finger

13. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Sample Output After a typical run: 10 days, 6:12:19 until graduation!! bhood@testing ~/syslab-tech/src $./main hand.png Edge detection took 0.04 sec Image cropping took 0.00 sec Line detection took 0.17 sec (detected 1424 lines)‏ Line chaining took 0.25 sec (detected 130 chains)‏ Getting orientation took -0.08 src (1 => ORIENTATION_FORWARD)‏ Getting pinky pos. took 0.00 sec (2 => FINGER_BENT)‏ Getting ring pos. took 0.00 sec (2 => FINGER_BENT)‏ Getting middle pos. took 0.01 sec (2 => FINGER_BENT)‏ Getting index pos. took -0.00 sec (4 => FINGER_TUCKED)‏ Overall process took 0.47 sec [TOTALCOUNT] allocated: 10718901, freed: 10364880; leaked: 354021.

14. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Timing Timing data from runs:  To nearest hundredth of a second (1)Edge detection: 0.04 sec (2)Image cropping: 0.00 sec (3)Line detection: 0.17 sec (4)Line chaining: 0.25 sec (5)orientation: 0.08 sec (6)Pinky finger: 0.00 sec (7)Ring finger: 0.00 sec (8)Middle finger: 0.01 sec (9)Index finger: 0.00 sec (10)Overall process: 0.47 sec  A little slow considering goal of real-time

15. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 The Mysterious Future Perfect line interpretation Work on memory management  Am leaking large quantities (~50K) of memory  Aggressive profiling needed Finish camera-computer interaction  Device control must be precise, picky

16. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 The End! Code will be available to future years Contact me for a copy: byron@phareware.com