1. Smart Home Design Based On Voice Recognition
Using Galileo and Raspberry Pi
Written in Python script language
Semesterial project
Students : Matan Cohen & Sefi Cohen
Supervisor : Mony Orbach
Spring 2014

2. Motivation
Most smart homes today require a remote instead of being hands-off
We want to automate home life
We want to control appliances from afar without remotes
Smart home products are expensive
Smart home products aren’t extendible

3. Project Goals To build an infrastructure of an interactive home
It’ll need to be hardware extendible – Can add more components in the future easily
It’ll need to be software extendible – Can add more software features easily
To build a prototype which turns on a led with the right command.
Voice recognition based commands
Since most products out there require a remote (which is a hassle), ours will be hands-off.
Communicate with the user – Using TTS we can have the system speak back and give the user indications on actions being performed
To experience designing hardware and software product from scratch.
To get our hands dirty with both hardware and programming at the hardware & software environments.

4. Project Vision Sketch (for example)
BEDROOM
LIVINGROOM
MIC
MIC
Raspberry Pi
Raspberry Pi
Galileo
Galileo I N T E R TV
SERVER
(RP or PC)
Electric Blinds
WHEREVER

5. Hardware Environment Galileo Raspberry Pi Based on ARM Linux based
Open Source
On board GPU
On board sound card
Ethernet
Based on Intel Quark
Arduino based
Open Source
Ethernet

6. Software Environment Raspberry Pi - Linux environment
Using Python script language.
Galileo – Linux environment
Using implemented library of Arduino in Python.
Both are controlled via SSH protocol.

7. Hardware Architecture
Linked through internet

8. Software Architecture
Raspberry Pi - Linux environment
Listener program – record the voice from the microphone and send the wav file to voice recognition unit .
Voice recognition unit – based on WIT learning speech recognition web service online.
RabbitMQ SERVER – Linux environment
Server SW– receive and send the analyzed messages from client to client.
Command Parser– parse the received string and send a signal to the right Galileo to run the right add-on script
Galileo – Arduino environment
Add-ons scripts written in python using Arduino library .
Scripts like lights and blinds are already implemented.

9. Configuration file
We need to initialize the HW component with information that it needs to function properly.
When we add a new component (recording like Rpi or commands invoker like Galileo) we have to write a “system config” file which contain the following :
Mic – insert microphone dry details : Format, Rate, Channels number and Chunk size.
Voice Energy Threshold – which the microphone will start recording from this energy and above.
WIT Key – that serial number is needed for connecting to the WIT web service.
Name – The component's name. It takes part in routing the message to the right component via RabbitMQ.
Commands – The addons that are supported on this component.

10. External SW Components
RabbitMQ – Server Implementation
There is a lot of different implementation of servers – we choose “Routing” which in this implement every target has a queue of his own and he gets only the messages that’s addressed to him
In the code the RabbitMQ is called “AMQP” (Advanced Message Queuing Protocol) which is the name of the protocol which RabbitMQ implements .

11. External SW Components cont.
wit.ai – Voice recognition web service
enables developers to add a language interface to their app or device in minutes.
It’s a learning system and as one we need first to provide a few examples of queries to get the expected responses.
We use their API to send queries and get the response as structured information back (JSON)
Py Galileo Library – Python library for Galileo board
Helps us write code in python that control the pins on the board like Arduino does.

12. Main Software Components
Text To Speech Module – Given a string, create an audio file using a Google TTS service that says that string aloud. Plays that file.
Wit Query Analyzer – using WIT web service. Given an audio recording of a command, the WIT service returns a JSON that can be worked with containing information about what was said.
Client - Manages sending parsed commands to the correct HW component.

13. Main Software Components cont.
Receiver - Receives a command from another HW components and starts processing it by sending it to the Addon Manager
Addon Manager – when the Receiver gets a command the Addon Manager is responsible for checking the validity of the command and executing the right script

14. Flow of events - System POV
Voice message is received in Raspberry Pi
The message is analyzed by WIT web service online and a json file returns as a string
This json file is wrapped by private object called wit_response – locally on Raspberry Pi
From Rpi the analyzed message is sent to the right target by the RabbitMQ server which send the message according to the target’s name

15. Flow of events - System POV cont.
The target (Galileo or other Rpi) receives the wit response and transfers it to the add-on manager
The add-on manager that runs on the target checks if (in the system configuration file) the target supports the command. It does this by checking if the "intent" field in the wit response object is found in the "commands" field list in the component's JSON configuration file.
If there is a match the add-on manager executes the right script (and pass to the add-on the original wit response since the wit response includes information needed by the addon itself as well.)

16. Demonstration Equipment
1 Raspberry pi
2 Galileo
1 PC (the SERVER)
1 SD card (for Rpi)
2 mini SD card (for Galileo)
1 USB sound card
3 Ethernet cables
2 USB microphone
1 Speakers
1 Led circuit
1 Buzzer circuit

17. Demonstration Environment
MIC
SPK
BUZZ
LED
USB Connected
Electric Circuit
Electric Circuit
Galileo RP
Galileo
Ethernet
Connection
SSH Connection over TCP/IP
INTERNET
RabbitMQ SERVER

18. Starting Demonstration
Burning the Linux version on SD cards for all the hardware components.
Connecting the all the Demo environment as the sketch in the last slide.
Connect the Rpi and Galileo through direct Ethernet connection.
Run the command Nmap from any PC in order to find the hardware IP’s
Connected to all the hardware components through SSH protocol over TCP/IP.

19. Starting Demonstration cont.
Write a configuration file for each component :
Measure the energy threshold using script.
Insert the Microphone factory properties .
Inset the WIT key.
Give a name to the component.
Enter the commands that are available for this “room”/HW component.
Execute the RabbitMQ server on PC or on one of the Raspberry Pi and “tell” the other components his IP.
Execute on each Raspberry Pi in the all system the client “audio_client” program.
Execute on each Galileo the “receiver_client” program.

20. FAQ Why python ? Why do we need the server in the middle ?
It’s a strong language with all the libraries we need.
It’s easier to write code in.
Why do we need the server in the middle ?
Because we want to make Hardware extendible infrastructure and we need a server to control all the units in the house .
How can we configure new microphone ?
In the system configuration file we can edit the mic factory settings (format, rate, channels number and chunk size).
How can we configure energy threshold in the config file ?
Special script written for that matter ,named energy_threshold.py, which measure the silence and loud energies in the room (will be shown in demo).

21. FAQ cont. What about the delay of the all system?
We add to our software “performance counters” for getting information about the time of each action.
The voice recognition proccess (using WIT) is taken between 6 to 12 seconds depend on the network speed and the load on WIT server.
The sketch upload to the right galileo device is taken between 1 to 2 seconds.
How can we do the initialize step hands off ?
At our home we could give the server static IP and write the IP in the configuration file so that all the other devices will know who is the server.
Then we need to define in each device the right scripts that’s needs to be uploaded from the start.

22. Future Upgrades Make the voice recognition SW run locally
It will shorten the delay significantly.
“Sphinx” – library that implement voice recognition locally.
Use Zero-configuration networking
In our project its important that each device is getting an IP address.
In this methodology DHCP server is not necessary for network configuration and there is no need in manual operator intervention or other special configuration servers.
Make the all system wireless

23. Future Possible Projects
Add camera and use facial recognition algorithms
If its not someone familiar turn on the alarm.
If its someone familiar turn on the A/C and his favorite music.
Add a web application of user interface
Easy to configure hardware components.
Easy to add software add-ons.
Control away from home will be enabled.

24. Questions ?
Thank you all very much !!!