Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *, Zhigang Liu **, Andrew T. Campbell * * CS Department – Dartmouth College ** Nokia Research Center – Palo Alto
Miluzzo
Miluzzo evolution of sensing and inference on mobile phones
Emiliano Miluzzo PR time
Emiliano Miluzzo
Miluzzo
Miluzzo
Miluzzo
Miluzzo
ok… so what ?? Miluzzo
Miluzzo density
Miluzzo accelerometer digital compass microphone WiFi/bluetooth GPS …. light sensor/camera sensing
Miluzzo accelerometer digital compass microphone WiFi/bluetooth GPS light sensor/camera gyroscope air quality / pollution sensor sensing ….
Miluzzo - free SDK - multitasking - free SDK - multitasking programmability
Miluzzo MHz CPU - up to 1GB application memory MHz CPU - up to 1GB application memory hardware computation capability is increasing
Miluzzo application distribution
Miluzzo application distribution deploy apps onto millions of phones at the blink of an eye
Miluzzo application distribution collect huge amount of data for research purposes deploy apps onto millions of phones at the blink of an eye
Miluzzo cloud infrastructure cloud - backend support
Miluzzo cloud infrastructure cloud - backend support
Miluzzo cloud infrastructure cloud - backend support we want to push intelligence to the phone
Miluzzo cloud infrastructure cloud - backend support preserve the phone user experience (battery lifetime, ability to make calls, etc.) preserve the phone user experience (battery lifetime, ability to make calls, etc.)
Miluzzo cloud infrastructure cloud - backend support - sensing - run machine learning algorithms locally (feature extraction + inference)
Miluzzo cloud infrastructure cloud - backend support - sensing - run machine learning algorithms locally (feature extraction + inference) run machine learning algorithms (learning)
Miluzzo cloud infrastructure cloud - backend support store and crunch big data (fusion) run machine learning algorithms (learning) - sensing - run machine learning algorithms locally (feature extraction + inference)
Miluzzo cloud infrastructure cloud - backend support run machine learning algorithms (learning) store and crunch big data (fusion) 3 to 5 years from now our phones will be as powerful as a - sensing - run machine learning algorithms locally (feature extraction + inference)
Miluzzo cloud infrastructure cloud - backend support run machine learning algorithms (learning) store and crunch big data (fusion) 3 to 5 years from now our phones will be as powerful as a - sensing - run machine learning algorithms locally (feature extraction + inference)
Miluzzo cloud infrastructure cloud - backend support run machine learning algorithms (learning) store and crunch big data (fusion) 3 to 5 years from now our phones will be as powerful as a - sensing - run machine learning algorithms locally (feature extraction + inference)
Miluzzo cloud infrastructure cloud - backend support - Sensing - run machine learning algorithms locally (feature extraction + learning + inference) run machine learning algorithms (learning) store and crunch big data (fusion) 3 to 5 years from now our phones will be as powerful as a
Miluzzo sensing programmability cloud infrastructure
Miluzzo sensing programmability cloud infrastructure ??
Miluzzo societal scale sensing global mobile sensor network reality mining using mobile phones will play a big role in the future reality mining using mobile phones will play a big role in the future
end of PR – now darwin Emiliano
a small building block towards the big vision Emiliano
Emiliano Miluzzo from motes to mobile phones
Miluzzo evolution of sensing and inference on mobile phones from motes to mobile phones
Miluzzo evolution of sensing and inference on mobile phones from motes to mobile phones darwin - classification model evolution - classification model pooling - collaborative inference
Miluzzo microphone camera GPS/WiFi/ cellular air quality pollution sensingapps social context audio / pollution / RF fingerprinting image / video manipulation darwin applies distributed computing and collaborative inference concepts to mobile sensing systems darwin - classification model evolution - classification model pooling - collaborative inference
why darwin? Miluzzo mobile phone sensing today
why darwin? Miluzzo train classification model X in the lab mobile phone sensing today
why darwin? Miluzzo deploy classifier X mobile phone sensing today train classification model X in the lab
why darwin? Miluzzo train classification model X in the lab deploy classifier X train classification model X in the lab mobile phone sensing today
why darwin? Miluzzo deploy classifier X mobile phone sensing today train classification model X in the lab train classification model X in the lab
why darwin? Miluzzo train classification model X in the lab deploy classifier X a fully supervised approach doesnt scale! mobile phone sensing today train classification model X in the lab
why darwin? a same classifier does not scale to multiple environments (e.g., quiet and noisy env) Miluzzo
why darwin? a same classifier does not scale to multiple environments (e.g., quiet and noisy env) Miluzzo
why darwin? a same classifier does not scale to multiple environments (e.g., quiet and noisy env) Miluzzo
why darwin? a same classifier does not scale to multiple environments (e.g., quiet and noisy env) Miluzzo darwin creates new classification models transparently from the user (classification model evolution ) darwin creates new classification models transparently from the user (classification model evolution )
Miluzzo why darwin? ability for an application to rapidly scale to many devices
Miluzzo why darwin? ability for an application to rapidly scale to many devices darwin re-uses classification models when possible (classification model pooling ) darwin re-uses classification models when possible (classification model pooling )
Miluzzo why darwin? leverage the large ensemble of in-situ resources
Miluzzo why darwin? leverage the large ensemble of in-situ resources darwin exploits spatial diversity and co- operate to alleviate the sensing context problem ( collaborative inference ) darwin exploits spatial diversity and co- operate to alleviate the sensing context problem ( collaborative inference )
Miluzzo darwin design
Miluzzo speaker recognition (subject to audio noise, sensing context, etc.)
Miluzzo darwin phases
Miluzzo darwin phases initial training (derive model seed) supervised
Miluzzo darwin phases initial training (derive model seed) classification model evolution supervised unsupervised
Miluzzo darwin phases initial training (derive model seed) classification model evolution classification model pooling supervised unsupervised
Miluzzo darwin phases initial training (derive model seed) classification model evolution classification model pooling collaborative inference supervised unsupervised
Miluzzo classification model training sensed event
Miluzzo classification model training sensed event filtering (silence suppression + voicing)
Miluzzo classification model training sensed event filtering (silence suppression + voicing) feature extraction (MFCC) feature extraction (MFCC)
Miluzzo classification model training filtering (silence suppression + voicing) feature extraction (MFCC) feature extraction (MFCC) model training (GMM) model training (GMM) model baseline sensed event send model + baseline back to phone send MFCC to backend to train the model backend
Miluzzo classification model training phone : feature extraction (low computation) backend backend : model training (high computation)
Miluzzo classification model evolution phone : determines when to evolve
Miluzzo classification model evolution phone : determines when to evolve training
Miluzzo classification model evolution phone : determines when to evolve trainingsampled
Miluzzo classification model evolution phone : determines when to evolve match? YES do not evolve
Miluzzo classification model evolution phone : determines when to evolve match? NO evolve ( train new model using backend as before)
Miluzzo classification model evolution new speaker voice model training
Miluzzo classification model evolution new speaker voice model training
Miluzzo classification model evolution new speaker voice model training
Miluzzo classification model pooling
Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker C s model Speaker B s model Speaker C s model
Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker C s model Speaker B s model Speaker C s model we have two options 1. train a new classifier for each speaker (costly for power, inference delay)
Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker C s model Speaker B s model Speaker C s model we have two options 1. train a new classifier for each speaker (costly for power, inference delay) 2. re-use already available classifiers
Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker C s model Speaker B s model Speaker C s model we have two options 1. train a new classifier for each speaker (costly for power, inference delay) 2. re-use already available classifiers
Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker C s model Speaker B s model Speaker C s model
Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker Bs model Speaker Cs model Speaker B s model Speaker A s model Speaker C s model
Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker Bs model Speaker Cs model Speaker B s model Speaker As model Speaker Cs model Speaker A s model Speaker B s model
Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker Bs model Speaker Cs model Speaker As model Speaker Bs model Speaker As model Speaker Cs model
Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker Bs model Speaker Cs model Speaker As model Speaker Bs model Speaker As model Speaker Cs model ready to run the collaborative inference algorithm - local inference first - final inference later ready to run the collaborative inference algorithm - local inference first - final inference later
Miluzzo collaborative inference two phases
Miluzzo collaborative inference 1. local inference (running independently in parallel on each mobile phone) two phases
Miluzzo collaborative inference 1. local inference (running independently in parallel on each mobile phone) two phases 2. final inference (after collecting Local Inference results, to get better confidence about the final classification result)
local inference (LI) Miluzzo collaborative inference Phone A Phone B Phone C
Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! local inference (LI)
Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! local inference (LI) As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10
Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! local inference (LI) As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10
Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 local inference (LI) Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10
Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 local inference (LI) Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10
Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! local inference (LI) As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10
Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! local inference (LI) As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10 individual classification can be misleading!
final inference (FI) Miluzzo collaborative inference Phone A Phone B Phone C each phone gathers LI results As LI results Cs LI results Bs LI results As LI results Cs LI results Bs LI results
final inference (FI) Miluzzo collaborative inference on each phone As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10
Miluzzo As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10 x x x x x x final inference (FI) collaborative inference on each phone
Miluzzo As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10 x x x x x x FI results (normalized): Confidence (A speaking) = 1 Confidence (B speaking) = 0.12 Confidence (C speaking) = = final inference (FI) collaborative inference on each phone
Miluzzo As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10 x x x x x x = FI results (normalized): Confidence (A speaking) = 1 Confidence (B speaking) = 0.12 Confidence (C speaking) = final inference (FI) collaborative inference on each phone
Miluzzo As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10 x x x x x x = collaborative inference compensates the inaccuracies of individual inferences FI results (normalized): Confidence (A speaking) = 1 Confidence (B speaking) = 0.12 Confidence (C speaking) = final inference (FI) collaborative inference on each phone
Miluzzo evaluation
Miluzzo evaluation C/C++ & implemented on Nokia N97 and iPhone in support of a speaker recognition app
Miluzzo evaluation C/C++ & unix server implemented on Nokia N97 and iPhone in support of a speaker recognition app
Miluzzo evaluation C/C++ & unix server lightweight reliable protocol to transfer models from the server and between phones implemented on Nokia N97 and iPhone in support of a speaker recognition app
Miluzzo evaluation C/C++ & UDP multicast protocol to distribute local inference results between phones implemented on Nokia N97 and iPhone in support of a speaker recognition app
Miluzzo experimental scenarios up to eight people in conversation in three different scenarios (quiet indoor, down the street, in a restaurant)
Miluzzo some numerical results
Miluzzo need for evolution train indoor, evaluate outdoor
Miluzzo need for evolution accuracy improvement after evolution accuracy
Miluzzo indoor quiet scenario 8 people talking around a table
Miluzzo indoor quiet scenario 8 people talking around a table
Miluzzo indoor quiet scenario 8 people talking around a table
Miluzzo indoor quiet scenario 8 people talking around a table
Miluzzo indoor quiet scenario 8 people talking around a table
Miluzzo indoor quiet scenario 8 people talking around a table
Miluzzo indoor quiet scenario 8 people talking around a table collaborative inference + classification model evolution boost the performance of a mobile sensing app
Miluzzo impact of the number of mobile phones
Miluzzo impact of the number of mobile phones
Miluzzo impact of the number of mobile phones
Miluzzo impact of the number of mobile phones
Miluzzo impact of the number of mobile phones the larger the number of mobile phones collaborating, the better the final inference result
Miluzzo battery lifetime Vs inference responsiveness
Miluzzo battery lifetime Vs inference responsiveness
Miluzzo high responsiveness battery lifetime Vs inference responsiveness
Miluzzo short battery life battery lifetime Vs inference responsiveness
Miluzzo longer battery duration battery lifetime Vs inference responsiveness
Miluzzo low responsiveness battery lifetime Vs inference responsiveness
Miluzzo battery lifetime Vs inference responsiveness smart duty-cycling techniques and machine learning algorithms with better performance in terms of energy usage on mobile phones need to be identified
Miluzzo a quick recap smartphones are everywhere, lets exploit their collective sensing and computation capabilities
Miluzzo a quick recap smartphones are everywhere – lets exploit their collective sensing and computation capabilities smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities
Miluzzo a quick recap smartphones are everywhere – lets exploit their collective sensing and computation capabilities continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques) smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities
Miluzzo a quick recap smartphones are everywhere – lets exploit their collective sensing and computation capabilities continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques) ML algorithms should perform reliably in the wild smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities
Miluzzo a quick recap smartphones are everywhere – lets exploit their collective sensing and computation capabilities continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques) ML algorithms should perform reliably in the wild smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities ok I think Im done…
Miluzzo a quick recap smartphones are everywhere – lets exploit their collective sensing and computation capabilities continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques) ML algorithms should perform reliably in the wild smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities but please bear in mind…
Miluzzo Mobile Phone Sensing is the Next Big Thing!
Thank you!! Miluzzo Mobile Sensing Group