Translation of Sign Language to Speech Sensor Glove Development Name:Garry Rank Student Number: Supervisor:Mr Iain Murray

Australian Sign Language (ASL) Brief History –Developed by whom. –Various standards and forms of sign language. One handed alphabet. Two handed alphabet. Full word gestures.

Why do we want translation? To help people to communicate. It can not replace full understanding of sign language. There are many elements of ASL that are simply too subtle! Speed, force, facial expressions, body language can all change meaning of a sign.

Glove Specifics Why is a glove required? Image data extraction not yet powerful enough to handle everything. Using a glove is more effective than a CCD for: –Speed –Sensitivity –Accuracy

Design Requirements Fast Accurate and sensitive. Robust Low cost Low profile – the glove should not impede signing. Be able to detect everything necessary for translation. These are all problems that make most virtual reality gloves unsuitable for the task!

Required Data What sensors are required? Less is better General hand locations important to ASL: Top of handSide and palm of hand

Amount of bend of base and first knuckle of each digit. Pitch of the wrist. Detection of finger splaying. Finger’s crossed. General area contacts.

Fingers that Bend Properties of a bending finger –Vertical compression over the knuckle. –Horizontal sliding.

Fingers that Bend contd… Capacitors lend a natural solution. –Sliding motion to change surface area. –Capacitance is changed. –Detectable by a micro-controller’s analogue to digital converter (ADC).

Fingers that Bend contd… Apply voltage. Wait for capacitor charge. Turn off voltage. Wait a short time and read voltage level. Probably not required.

Capacitor Discharge From the discharge rate of the circuit, the voltage at time T can be used to determine amount of capacitance. T Higher capacitance Lower capacitance

Testing An m16c microprocessor development board provided the ADC. 8bit resolution mode used for its increased speed. Results showed 10° accuracy possible, but interference may produce erroneous values. 10bit resolution on the ADC offered a dramatically increased reliability and accuracy.

Contact Sensors Primary and Secondary contact locations. Primary –Used for fast determination of hand shape. –Finger splaying. –Touching the palm. –Fingers crossed. Secondary –Fingers touching elsewhere. –Chest, head, arms or elsewhere on the body. Different types of contact sensors for primary and secondary locations.

Primary Contact Used to supplement finger bend data to improve speed of sign recognition. Amount of finger bend is redundant data if it is known a finger is touching the palm. Minimum set required – more buttons greatly increases complexity and chance of equipment failure.

Primary Contact contd… Positive terminal on palm and tops of fingers. Negative terminal on finger tips. When the fingers touch the palm a circuit is closed. Same reasoning can be used for finger splaying.

Primary Contact contd…

What to use as materials on the glove? –Flexible to move with the hand. –Conductive Carbon rubber. –Used on keypads. –Flexible and conductive. –Requires outer surface to be kept clean. –Not as sensitive to contact as metal. Probably not required.

Secondary Contact A simple button used to detect when a finger is touching something not detectable by the glove. Not as sensitive to soft contact –Some signs could be missed. Used to supplement overall sign recognition of the system.

Secondary Contact contd…

Putting it all Together M16c microcontroller development board linked to a PC via serial cable. Developed under cygwin programming environment – but should work for any Unix type OS. Bend sensor: –1MΩ Resistor. –Brass tubing. –Old mouse cord for over-the-finger exoskeleton.

Software Flowchart Wait for capacitor charge Set output pin low (0V) Wait some time T Use ADC for voltage reading Transmit value to PC Set output pin high (5V) Start Here

Timing A real time system – at least 10 full glove readings per second. Including one for the wrist, three for the thumb and two for each finger, 24 bend sensors may be required. Conversion timing is important! Approximate 6us conversion time. With up to 24 bend sensors, this is means a 144us bend sensor clock time. Well within acceptable limits!!!

ADC Input Multiplexing The problem now is that your standard micro-controller doesn’t have enough ADC inputs for all the required bend-sensors! Inputs must be multiplexed. Diodes can be used to prevent all capacitors multiplexed from being charged. Probably not required.

Multiplexed Input If only A, only B or only C is turned on, then only one capacitor will affect the value as read from the voltage probe (ADC input). Probably not required.

Contact Sensors Much of this work has already been done by Craig Newman in his thesis “Translating Sign Language to Speech”, Most notably, pull-down resistors are used to prevent indeterminate values when the contacts are not touching.

Contact Sensor Circuit Primary contact Secondary contact +5V Input Pin Note that the voltage sources are isolated – this is so that it can be known if the primary and/or secondary contacts are closed. Much of this work has already been done by Craig Newman in his thesis “Translating Sign Language to Speech”, Most notably, pull-down resistors are used to prevent indeterminate values when the contacts are not touching.

Two Gloves The sensors so far assume only one glove. Two handed interaction should also be detected. More contact sensors seems like a good idea, or is it? Pins on microcontrollers are I/O. –They can be input or output. –By controlling this, much more data can be obtained from the same number of sensors. –Incurs a time penalty.

Problems The glove needs to be specially made. –Material moves too much with the hand. –Capacitors sewn onto the glove move with the material. Bending of the wrist causes the most problems here. –Wrist bending is calculated and so these errors can be algorithmically reduced Primary and secondary contacts become more sensitive with small hardpoints. –Thin plastic can be used as an exoskeleton on which to mount sensors. –Straps to keep the glove fixed on the hand. Electrical interference can induce currents that produce errors in results.

Thanks for Listening! Feel free to ask some questions.