1. Augmented Reality: Internet of Things
GROUP 5
Adam Sanche • Andrei Usenka • Jiawei Wu • Satyen Akolkar
Introduce ourselves and give a quick gist of what we’ve been working on….

2. IoT - Background Internet of Things:
Embedded devices control other devices over the internet
Used in various areas such as:
Environmental Monitoring
Infrastructure Management
Manufacturing
Energy Management
Hair brushing
etc...
Environmental Monitoring: monitoring of things such as water quality, soil conditions, movement of wildlife.
Infrastructure Management: monitoring and controlling things like bridges/ railways/ wind-farms.
Manufacturing: process control / asset management / plant safety optimization
Energy Management: integrates sensing and actuation systems. Improve efficiency. Better energy optimization downstream helps energy companies balance power generation upstream.
The list goes on… our project focuses on incorporating iot devices to help maintainance of the iot device itself.

3. Hair Brushing? Yes… Hair brushing
There is a tendency with IoT to do it just because you can.
Yes… Hair brushing
Whenever you choose to do something involving IoT, the question should never be about whether or not you can do it. If you have the sensors, it’s likely that you can do it. Therefore the real question should be why you should do it. In our project we have made a large effort to focus on the why, and in answering these questions, finding the direction to go in.

4. Project Overview Expected delivery Our Project March 4, 2017
Build IoT device
Integrate it with Microsoft HoloLens
Allow users to query environmental data
Display that data to user in real-time, through the HoloLens
Project
Overview
In our project we are going to be building an Internet of Things device and integrating it with a microsoft HoloLens. The user would then see sensor information and be able to respond faster and with a greater level of safety to issues at hand.

5. Project Overview Specific Overview Working with a breaker box for demo
Query for Amperage and Voltage for 3 individual breakers. (One breaker has 2 wires)
How does this help anyone?
Describe our circuit breaker setup here...

6. Project Overview General Overview of our IoT project
Allows devices to be queried for real-time or past data.
Companies rely on component MTTF and replace them past expiry
Information saves money
Say you live at home and you trip a breaker. It is easy enough to go to the breaker box and flip it on again.
What if you are now working for a company and you have to drive half hour each way to service a breaker for a client. Now you have to ask a question of “why is it tripping?”.
There is a chance that the breaker is worn out and trips at too low of a current draw. However, how does one know if that’s the case without testing the system.
Big companies rely on Mean Time to Failure of components and simply replace them when time is up. What if that components life time is actually not over yet. That’s lost money.
This is the issue that our project focuses on solving. We are using IoT and HoloLens to provide maintenance information about specific components in devices. With extra information about the devices, the company might have the knowledge to just flip that breaker back on, instead of either testing it or replacing it. Alternatively it might tell them that it is in-fact time to replace the breaker instead of sending technicians out every time.

7. The Team

8. The Team - Work Breakdown
Adam Sanche
Creating the ability to use IoT sensors in WorkLink.
Polling the server for the appropriate sensor data while WorkLink is running.
Jiawei Wu
Create a server to poll sensors for readings.
Create API for client (WorkLink) to read sensor data from.
Andrei Usenka
Wiring, batteries, sensors and fpga installation
LCD implementation on fpga
Satyen Akolkar
Setting up the environment on the DE0- nano SoC.
Linux, NodeJS, ExpressJS
Drivers for sensors / IO

9. Detailed Overview

10. Hardware Overview System Block Diagram 3 Hardware Subsystems
FPGA
Embedded ARM (HPS)
Microsoft HoloLens
FPGA and HPS subsystems are customized and focused on for Hardware Development
Hardware Overview
System Block Diagram

11. Hardware Overview Cyclone V FPGA
Programmed during boot by Hard Processor System with .rbf (analogous to .pof files)
Analog Pin Reading Components (ADC Header)
Voltage Reading Components (GPIO)
Character LCD Interfacing Component (GPIO)
Hardware Overview
Cyclone V FPGA

12. Hardware Overview ARM Cortex A9 & Linux U-Boot Bootloader from Altera
Programs FPGA using .rbf file generated in Quartus II
Linux Kernel Compiled for ARM reads device tree binary for board and FPGA components
Debian RootFS allows access to Debian and Raspbian software repositories for ARM
Custom Device Drivers for FPGA component Reads/Writes
Hardware Overview
ARM Cortex A9 & Linux

13. WorkLink Microsoft HoloLens’ Application & WorkLink
Augmented Reality content “authoring” platform and “player”.
The application (Android, iOS or WSA on Hololens) uses the authoring platform as a library as to reduce redundant code.
This means most additions are done to create more tools for authoring utility, and thusly allow for the additions to be viewed in the device application
Microsoft HoloLens’ Application & WorkLink

14. Functionality FPGA HoloLens Act as a server
Read and store data from Sensors
Connect to LCD to display IP address
Connect with Wifi module to respond to information requests from hololens
Establish connection with a router through wifi module
Uses microSD card to run linux with debian, and store the sensor data.
HoloLens
Provides user with access to server
Allows user to follow instructions on servicing a breaker box
Allows user to query information from the server
Going to use client’s system to produce Augmented Reality visuals and write our own routines to show our data visually
Just summarize the design how everything fits together, to make sure everyone understands.

15. Server Breakdown

16. Why Node.js? Node.js Server Fast
Written in C/C++, Using Google’s V8 JS engine
Asynchronous IO and Event Driven Programming
No thread synchronization
Non-blocking
Crossplatform
Linux, OSX, Windows
Standardized Communication Protocol (HTTP and JSON)
Rest
Expansive library of open-source modules (NPM)
Security (HTTPS - HTTP over TLS/SSL)
Botnet of Things
Node.js Server

17. Why Express.js? Express.js Framework
Express is an open-source framework that provides abstracts the Node.js HTTP module
Simplifies routing
Refactors large request handlers into smaller functions known as Middleware
Adds helpful conveniences to Node.js HTTP module
Parsing Request bodies, URL, etc.
Sending Files
Minimal and unbiased
Express.js Framework

18. Server Code Snippet (app.js)
app.get("/sensor_1", function(request, response) { if (typeof sensor1JSON != "undefined") { response.status(202).json(sensor1JSON); } else { response.status(404).send("No File Found"); } }); app.get("/sensor_2", function(request, response) { if (typeof sensor2JSON != "undefined") { response.status(202).json(sensor2JSON); } else { response.status(404).send("No File Found"); } })
var express = require("express"); var path = require("path"); var fs = require("fs"); var spawn = require("child_process").spawn; var app = express(); var cmd = "./generateJSON.o"; var sensorPath = path.resolve(__dirname, "sensors"); var sensor1JSON; var sensor2JSON; app.use(express.static(path.resolve(__dirname, "public"))); app.set("views", path.resolve(__dirname, "views")); app.set("view engine", "ejs"); app.get("/", function(request, response) { response.render("index"); });
app.use(function(request, response) { response.status(404).render("404"); });
app.listen(3000, function() { console.log("App started on port 3000"); spawn(cmd); setInterval(function() { fs.readFile(sensorPath + "/sensor_1.json", "utf8", function (err, data) { if (err) { throw err; } sensor1JSON = JSON.parse(data); }); }, 1000); setInterval(function() { fs.readFile(sensorPath + "/sensor_2.json", "utf8", function (err, data) { if (err) { throw err; } sensor2JSON = JSON.parse(data); }); }, 1000); });
Server Code Snippet (app.js)

19. Test Plan

20. General Testing Outline
C# unit tests for classes and methods made in WorkLink
Test asynchronous polling with mock data
Testing on Android or in Unity editor to decrease Workflow overhead (building to Hololens can be time consuming)
Component Testing
Do our drivers work as we intended? Do we get accurate values?
Stress testing the server
Load testing. Handling bad requests etc...

21. Test Results
Benchmarking done with Apache HTTP Server Benchmarking Tool
VM set to 1GB of RAM and 2 cores to better represent the hardware that is available to us.
This is ApacheBench, Version 2.3 <$Revision: $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd,
Licensed to The Apache Software Foundation,
Server Software:
Server Hostname:
Server Port:
Document Path: /
Document Length: bytes
Concurrency Level:
Time taken for tests: seconds
Complete requests:
Failed requests:
Total transferred: bytes
HTML transferred: bytes
Requests per second: [#/sec] (mean)
Time per request: [ms] (mean)
Time per request: [ms] (mean, across all concurrent requests)
Transfer rate: [Kbytes/sec] received
Percentage of the requests served within a certain time (ms)
50%
66%
75%
80%
90%
95%
98%
99%
100% (longest request)

22. Next Steps

23. Additional Features Features to add: Features to remove:
Add more sensors without modifying code
Can access server from a remote location
Web GUI
Features to remove:
Wifi module - replace it with ethernet because coding a wifi driver might be hard
User can not retrieve past measurements from server.
Additional Features

25. References https://en.wikipedia.org/wiki/Internet_of_things
Every body dance.
Clap your hands.