1. SigninGlove SDP 2016 February 29, 2016 Cumulative Design Review
Department of Electrical and Computer Engineering
Advisor: Professor Jackson

2. SigninGlove Team
John Gontowicz
Mathew Lau
Aaron Gilbert
Kacey Looney

3. Sign Language Translation
Currently need translator
Difficult to perform everyday activities
Ordering food, paying at shops, etc.
SigninGlove is a portable translator
Especially useful when a translator is not available
Kacey:
Currently a deaf person needs a translator to accompany them when they are trying to interact with the hearing world. However in everyday situations, such as going grocery shopping or eating at a restaurant, translators are not available. In these situations there is usually a lot of confusion and misunderstandings occurring between the deaf individual and the hearing person they are trying to communicate with. The SigninGlove would help alleviate these misunderstandings and make it easier for the deaf person to be understood. This would be a great help in situations where a translator is not available.

4. Our Solution: Block Diagram
Glove
Raspberry Pi 2
Android App
Sensor Data Transmission
Sensor Inputs
Sign Lookup Table Storage
Connect to RPi GPIO
Power Supply
GUI
Sensor filtering/ waveshaping
Processed Data Transmission
Data Collection
Sensors
Send over wireless connection
Flex Sensor
Sensor Data Processing
Incoming Data Processing
Palm/Finger contact sensors
Flex, Contact sensor output processing
Sign lookup
Gyroscope
Accelerometer, Gyro Signal Analysis
String building
Accelerometer

5. MDR Review Demonstrated end to end data transmission
Used 3 flex sensors and 3 pressure sensors
Sent sensor data to Raspberry Pi
Raspberry Pi sent translated value to Android App
Android App looked up value and displayed corresponding sign
Kacey

6. Sensors Conductive Ink-Based Flex Sensors Force Sensitive Resistor
0 if straight, 1 if bent
Force Sensitive Resistor
0 if no pressure, 1 if pressure is applied
Kacey

7. Accelerometer/Gyroscope
Berry IMU
LSM9DS0
3 acceleration channels
2/4/6/8/16 g linear acceleration
3 angular rate channels
245/500/2000 dps angular rate
Kacey

8. Battery Selection 5V, 2600 mAh=13Wh Powers Raspberry Pi and Sensors
Majority of power consumed by Raspberry Pi
Slightly more than 5W, battery would last about 5 hours
Easily replaceable
Mat

9. Raspberry Pi - Gyroscope
Connects to RPi via I2C Bus
Reads X, Y angles of IMU
Using Complementary Filter
Accounts for Gyro settling
RPi Records min and max of XY angles (0 neg. angle, 1 positive)
Example: Gyro Y angle
Aaron
max angle
min angle

10. Raspberry Pi - Accelerometer
Reads force acting upon IMU in g’s (set to 0-2g range)
Gyro angle used to cancel gravity
Records max/min acceleration in y, z direction → possible 0, 1 or 2
Example: Accel Y force
Aaron
max → 2
min → 0

11. Android App - New Table
Currently searches through lookup table for flex/contact sensors
Signs with same flex/contact but different gyro/accelerometer data then have a linked table
Tree will be implemented since similar behavior
Input
Letter or Table
Input→ Letter Table
Key
Value a b
k table
John
...
...

12. Android App - Auto Update
The app now updates automatically when a new packet is received; enabled using Handler class
This will allow the user to use left hand to control when a new sign is starting/ending
GUI Thread
Waits idle until Handler’s message
Updates GUI with Handler’s value
Network Thread
Continuously
waiting
for packets
Sends packet value to Handler Class
UDP Packet via WIFI
John
Handler

13. Demo - “G” and “Q”

14. Demo - “K” & “P”

15. Demo - “Z”

16. Future Work

17. Turing Test List of sentences commonly used in real world
Test our glove side by side with an interpreter
See how often the glove is understood compared to interpreter
Test both pre-programmed sentences and on the fly sentences
Kacey

18. FPR/Demo Day Presentation
Entirely Portable
Form sentences using pre-programmed words
Add words/sentences in real time
Button is part of the app
App shows current and previous signs
All

19. Adding of Words and Sentences
The next step will be to add the functionality of adding new words to the app
Holding the button will record the data on the RasPi and then send binary values to app
App will then insert the values in the proper place within the tree for future use Pi
Pi records data until button is released
Turns values into binary values
Sends to app as packet
APP
Holding button records data on Pi
Pi sends back binary values
App inserts data into tree
Command to record
Send packet
Insert into tree

20. Android button
The app will also have a button that will tell the Pi when the user is done with that sign
This will eliminate the physical button and allow smoother signing Pi
Pi records data until button is released
Turns values into binary values
Sends to app as packet
APP
Holding button samples data on Pi
Pi sends back binary values
App searches tree for correct sign
Command to sample
Search tree
Send packet
Show sign on GUI

21. Current Challenges/ Solutions
Challenge: All the wires complicate signing right now Solution: PCB will eliminate wires and make portable along with battery
Challenge: We can get some corrupt data through UDP Solution: TCP and error checking will eliminate any corrupt data
Challenge: Accelerometer/Gyro data is not 100% consistent Solution: Calibrate threshold and digitization of data; implement most similar algorithm on app

22. Thank You
Questions?

23. Portability of Glove
Back
Front Pi
PCB
Battery

24. Sample Turing Test Sentences
Where’s the restroom?
My name is _______.
Yes No
Can I have a _______? (Ordering)
Hello
I agree
Please
Thank you

25. Sensor Outputs Flex Sensor Pressure Sensor (Circle)
Straight (3.2mV): <1.1V
Flexed (3V): >1.1V
Pressure Sensor (Circle)
No Pressure (2.9mV): <2.4V
Pressure (2.9V): >2.4V
Pressure Sensor (Square)
No Pressure (3.1mV): <3.2V
Pressure (3V): >3.2V

26. Sensor Power Consumption
Resistor
Resistance
Power (W)
R1 (+)
15k
7.13E-04
R1 (G)
5.6k
3.26E-04 R2
27k
1.23E-04 R3
20k
9.94E-05 RM
100k
1.35E-04
Sensor
25k-150k
4.49E-05

27. Raspberry Pi - Inputs Flex Sensors Contact Sensors Berry IMU
Sensor Power
Push Button