Smart Home Design Based On Voice Recognition Using Galileo and Raspberry Pi Written in Python script language Semesterial project Students : Matan Cohen & Sefi Cohen Supervisor : Mony Orbach Spring 2014

Motivation Most smart homes today require a remote instead of being hands-off We want to automate home life We want to control appliances from afar without remotes Smart home products are expensive Smart home products aren’t extendible

Project Goals To build an infrastructure of an interactive home It’ll need to be hardware extendible – Can add more components in the future easily It’ll need to be software extendible – Can add more software features easily To build a prototype which turns on a led with the right command. Voice recognition based commands Since most products out there require a remote (which is a hassle), ours will be hands-off. Communicate with the user – Using TTS we can have the system speak back and give the user indications on actions being performed To experience designing hardware and software product from scratch. To get our hands dirty with both hardware and programming at the hardware & software environments.

Project Vision Sketch (for example) BEDROOM LIVINGROOM MIC MIC Raspberry Pi Raspberry Pi Galileo Galileo I N T E R TV SERVER (RP or PC) Electric Blinds WHEREVER

Hardware Environment Galileo Raspberry Pi Based on ARM Linux based Open Source On board GPU On board sound card Ethernet Based on Intel Quark Arduino based Open Source Ethernet

Software Environment Raspberry Pi - Linux environment Using Python script language. Galileo – Linux environment Using implemented library of Arduino in Python. Both are controlled via SSH protocol.

Hardware Architecture Linked through internet

Software Architecture Raspberry Pi - Linux environment Listener program – record the voice from the microphone and send the wav file to voice recognition unit . Voice recognition unit – based on WIT learning speech recognition web service online. RabbitMQ SERVER – Linux environment Server SW– receive and send the analyzed messages from client to client. Command Parser– parse the received string and send a signal to the right Galileo to run the right add-on script Galileo – Arduino environment Add-ons scripts written in python using Arduino library . Scripts like lights and blinds are already implemented.

Configuration file We need to initialize the HW component with information that it needs to function properly. When we add a new component (recording like Rpi or commands invoker like Galileo) we have to write a “system config” file which contain the following : Mic – insert microphone dry details : Format, Rate, Channels number and Chunk size. Voice Energy Threshold – which the microphone will start recording from this energy and above. WIT Key – that serial number is needed for connecting to the WIT web service. Name – The component's name. It takes part in routing the message to the right component via RabbitMQ. Commands – The addons that are supported on this component.

External SW Components RabbitMQ – Server Implementation There is a lot of different implementation of servers – we choose “Routing” which in this implement every target has a queue of his own and he gets only the messages that’s addressed to him In the code the RabbitMQ is called “AMQP” (Advanced Message Queuing Protocol) which is the name of the protocol which RabbitMQ implements .

External SW Components cont. wit.ai – Voice recognition web service enables developers to add a language interface to their app or device in minutes. It’s a learning system and as one we need first to provide a few examples of queries to get the expected responses. We use their API to send queries and get the response as structured information back (JSON) Py Galileo Library – Python library for Galileo board Helps us write code in python that control the pins on the board like Arduino does.

Main Software Components Text To Speech Module – Given a string, create an audio file using a Google TTS service that says that string aloud. Plays that file. Wit Query Analyzer – using WIT web service. Given an audio recording of a command, the WIT service returns a JSON that can be worked with containing information about what was said. Client - Manages sending parsed commands to the correct HW component.

Main Software Components cont. Receiver - Receives a command from another HW components and starts processing it by sending it to the Addon Manager Addon Manager – when the Receiver gets a command the Addon Manager is responsible for checking the validity of the command and executing the right script

Flow of events - System POV Voice message is received in Raspberry Pi The message is analyzed by WIT web service online and a json file returns as a string This json file is wrapped by private object called wit_response – locally on Raspberry Pi From Rpi the analyzed message is sent to the right target by the RabbitMQ server which send the message according to the target’s name

Flow of events - System POV cont. The target (Galileo or other Rpi) receives the wit response and transfers it to the add-on manager The add-on manager that runs on the target checks if (in the system configuration file) the target supports the command. It does this by checking if the "intent" field in the wit response object is found in the "commands" field list in the component's JSON configuration file. If there is a match the add-on manager executes the right script (and pass to the add-on the original wit response since the wit response includes information needed by the addon itself as well.)

Demonstration Equipment 1 Raspberry pi 2 Galileo 1 PC (the SERVER) 1 SD card (for Rpi) 2 mini SD card (for Galileo) 1 USB sound card 3 Ethernet cables 2 USB microphone 1 Speakers 1 Led circuit 1 Buzzer circuit

Demonstration Environment MIC SPK BUZZ LED USB Connected Electric Circuit Electric Circuit Galileo RP Galileo Ethernet Connection SSH Connection over TCP/IP INTERNET RabbitMQ SERVER

Starting Demonstration Burning the Linux version on SD cards for all the hardware components. Connecting the all the Demo environment as the sketch in the last slide. Connect the Rpi and Galileo through direct Ethernet connection. Run the command Nmap from any PC in order to find the hardware IP’s Connected to all the hardware components through SSH protocol over TCP/IP.

Starting Demonstration cont. Write a configuration file for each component : Measure the energy threshold using script. Insert the Microphone factory properties . Inset the WIT key. Give a name to the component. Enter the commands that are available for this “room”/HW component. Execute the RabbitMQ server on PC or on one of the Raspberry Pi and “tell” the other components his IP. Execute on each Raspberry Pi in the all system the client “audio_client” program. Execute on each Galileo the “receiver_client” program.

FAQ Why python ? Why do we need the server in the middle ? It’s a strong language with all the libraries we need. It’s easier to write code in. Why do we need the server in the middle ? Because we want to make Hardware extendible infrastructure and we need a server to control all the units in the house . How can we configure new microphone ? In the system configuration file we can edit the mic factory settings (format, rate, channels number and chunk size). How can we configure energy threshold in the config file ? Special script written for that matter ,named energy_threshold.py, which measure the silence and loud energies in the room (will be shown in demo).

FAQ cont. What about the delay of the all system? We add to our software “performance counters” for getting information about the time of each action. The voice recognition proccess (using WIT) is taken between 6 to 12 seconds depend on the network speed and the load on WIT server. The sketch upload to the right galileo device is taken between 1 to 2 seconds. How can we do the initialize step hands off ? At our home we could give the server static IP and write the IP in the configuration file so that all the other devices will know who is the server. Then we need to define in each device the right scripts that’s needs to be uploaded from the start.

Future Upgrades Make the voice recognition SW run locally It will shorten the delay significantly. “Sphinx” – library that implement voice recognition locally. Use Zero-configuration networking In our project its important that each device is getting an IP address. In this methodology DHCP server is not necessary for network configuration and there is no need in manual operator intervention or other special configuration servers. Make the all system wireless

Future Possible Projects Add camera and use facial recognition algorithms If its not someone familiar turn on the alarm. If its someone familiar turn on the A/C and his favorite music. Add a web application of user interface Easy to configure hardware components. Easy to add software add-ons. Control away from home will be enabled.

Questions ? Thank you all very much !!!