1. Interactive Mirror Display
ECE 445 Spring 2019
Group 29: Hiraal Doshi, Stephen Hurwit, Pamela Sanan

3. Introduction
The Interactive Mirror Display provides you with convenient access to information and media through a discrete gesture and speech controlled device

4. Objectives
Present information and provide media content through intuitive user interface
Recognize gestures including directional swipes and button presses
Recognize speech and interpret specific voice commands

8. Hardware - Control Unit
Raspberry Pi
Runs user interface software
Sends display and audio data to monitor over HDMI
Accepts audio input from microphone over USB
Microcontroller (ATmega328-P)
Receives data from proximity sensors over I2C
Receives data from passive infrared (PIR) sensors over GPIO
Powers LEDs over GPIO
Communicates with Raspberry Pi over UART

9. Hardware - Visual Sensor
Proximity Sensors (VCNL4200)
Provide distance data to microcontroller over I2C
Powered by 3.3V
PIR Sensors (HC-SR501)
Indicate when motion is detected to microcontroller over GPIO
Powered by 5V

10. Hardware - Power AC/DC Power Converter Converts 120V AC to 12V DC
Linear Voltage Regulators
Converts 12V DC to 5V DC
Converts 12V DC to 3.3V DC
Power Switch
Enables/Disables power to microcontroller and visual sensor

11. Software - User Interface
The user interface was developed on the Electron platform
HTML/CSS/JavaScript (Node.js)
Runs on Raspberry Pi

12. User Interface - Features
Provides convenient access to information and media
Personal schedule
Weather
News
Music
Videos

13. User Interface - Diagram

14. User Interface - Screenshots

15. User Interface - Media Integrations
SoundCloud
Provide music content through SoundCloud Widget API
YouTube
Provide video content through YouTube Widget API
Google Calendar
Provide personal schedule information through Google Calendar API

16. User Interface - Gesture Response

17. Software - Voice Commands
Uses annyang speech recognition library for JavaScript to recognize and respond to spoken commands
“Search news ‘University of Illinois’”
“Play playlist ‘jazz’”
“Play video ‘cats’”
“Play/Pause music”
“Skip/Previous song”

18. Software - Vision Sensor
Arduino program for ATmega328P
Communicates with proximity and motion sensors using GPIO and I2C
Interprets gestures using timestamps
Relays gesture to Raspberry Pi over UART serial communication

19. Software - Serial Host
Receives gesture data from ATmega328P and relay it to user interface
Required to access data from within user interface
Receives serial data over UART
Relays messages over TCP socket
Written in Python

20. Verification - Control Unit
Proximity Sensor
Proximity sensors output data to microcontroller ATmega328-P over the I2C bus
I2C mux allows several slave devices with same address to share I2C bus
Microcontroller
Performs I2C read from proximity sensor register to retrieve distance data
Polls GPIO pins to determine when PIR sensors activate
Raspberry Pi
(Serial Host)
Microcontroller sends data over UART to the Raspberry Pi with a baud rate of 9600
Host program transmits messages to user interface over TCP
User Interface
User interface displayed on the Raspberry Pi changes after receiving the data

21. Verification - Visual Sensors
Type of Sensor
Operating Voltage (V)
Range (cm)
PIR Sensor 5
Proximity Sensor
3.3
0 - 30
Each PIR sensor is 2.4cm x 3.2cm

22. Visual Sensor - Button Presses
Detect button presses with infrared proximity sensors
Uses I2C protocol to transmit distance
Represents 1.5 meter range with 16 bit precision
Programmatically detect when hand is held within 30cm of the sensor for more than 2 seconds

23. Visual Sensor - Swipe Gestures
Detect swipe gestures with an array of passive infrared (PIR) motion sensors
Use array of four PIR sensors to detect directionality of swipe gesture
Limit the sensitivity of the PIR sensors by taping them to get a certain field of view angle

24. Visual Sensor - PIR
dmax_op is the furthest tangential distance from the sensor at which the user is expected to be able to perform the gesture
dsensor is the distance between sensors

25. Visual Sensor - Detection Range
1 = high 0 = low

26. Verification - Camera / Mic / Status LEDs
Uses the Raspberry Pi Camera v2 to take high quality pictures that are ed to the user
Camera Status LED
Green LED on left side of the device
Privacy
In order to address ethical concerns surrounding user privacy, we enable status LEDs to notify the user whenever the microphone or camera becomes active
Microphone
Mini USB microphone captures speech allowing the device to recognize voice commands
Microphone Status LED
Red LED on right side of the device

27. Verification - Power Module
AC/DC Converter
VIN (V)
VOUT (V)
120
12.18
Mechanical Switch
VIN (V)
VOUT (V) On
12.18
Off
0.01
Voltage Regulator
VIN (V)
VOUT (V)
3.3V 12
3.313 5V
5.01

28. Conclusion & Future Work
The user interface was developed and responded to gesture and voice input accordingly
The proximity sensors functioned as expected
The PIR sensors did not function as expected
Future Work:
Replace the PIR sensors with a more reliable gesture recognition system
Expand upon the user interface and features of the mirror
Develop companion mobile app

29. Questions?