Presentation is loading. Please wait.

Presentation is loading. Please wait.

Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,

Similar presentations


Presentation on theme: "Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,"— Presentation transcript:

1 Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *, Zhigang Liu **, Andrew T. Campbell * * CS Department – Dartmouth College ** Nokia Research Center – Palo Alto

2 Miluzzo

3 Miluzzo evolution of sensing and inference on mobile phones

4 Emiliano Miluzzo PR time

5 Emiliano Miluzzo

6 Miluzzo

7 Miluzzo

8 Miluzzo

9 Miluzzo

10 ok… so what ?? Miluzzo

11 Miluzzo density

12 Miluzzo accelerometer digital compass microphone WiFi/bluetooth GPS …. light sensor/camera sensing

13 Miluzzo accelerometer digital compass microphone WiFi/bluetooth GPS light sensor/camera gyroscope air quality / pollution sensor sensing ….

14 Miluzzo - free SDK - multitasking - free SDK - multitasking programmability

15 Miluzzo MHz CPU - up to 1GB application memory MHz CPU - up to 1GB application memory hardware computation capability is increasing

16 Miluzzo application distribution

17 Miluzzo application distribution deploy apps onto millions of phones at the blink of an eye

18 Miluzzo application distribution collect huge amount of data for research purposes deploy apps onto millions of phones at the blink of an eye

19 Miluzzo cloud infrastructure cloud - backend support

20 Miluzzo cloud infrastructure cloud - backend support

21 Miluzzo cloud infrastructure cloud - backend support we want to push intelligence to the phone

22 Miluzzo cloud infrastructure cloud - backend support preserve the phone user experience (battery lifetime, ability to make calls, etc.) preserve the phone user experience (battery lifetime, ability to make calls, etc.)

23 Miluzzo cloud infrastructure cloud - backend support - sensing - run machine learning algorithms locally (feature extraction + inference)

24 Miluzzo cloud infrastructure cloud - backend support - sensing - run machine learning algorithms locally (feature extraction + inference) run machine learning algorithms (learning)

25 Miluzzo cloud infrastructure cloud - backend support store and crunch big data (fusion) run machine learning algorithms (learning) - sensing - run machine learning algorithms locally (feature extraction + inference)

26 Miluzzo cloud infrastructure cloud - backend support run machine learning algorithms (learning) store and crunch big data (fusion) 3 to 5 years from now our phones will be as powerful as a - sensing - run machine learning algorithms locally (feature extraction + inference)

27 Miluzzo cloud infrastructure cloud - backend support run machine learning algorithms (learning) store and crunch big data (fusion) 3 to 5 years from now our phones will be as powerful as a - sensing - run machine learning algorithms locally (feature extraction + inference)

28 Miluzzo cloud infrastructure cloud - backend support run machine learning algorithms (learning) store and crunch big data (fusion) 3 to 5 years from now our phones will be as powerful as a - sensing - run machine learning algorithms locally (feature extraction + inference)

29 Miluzzo cloud infrastructure cloud - backend support - Sensing - run machine learning algorithms locally (feature extraction + learning + inference) run machine learning algorithms (learning) store and crunch big data (fusion) 3 to 5 years from now our phones will be as powerful as a

30 Miluzzo sensing programmability cloud infrastructure

31 Miluzzo sensing programmability cloud infrastructure ??

32 Miluzzo societal scale sensing global mobile sensor network reality mining using mobile phones will play a big role in the future reality mining using mobile phones will play a big role in the future

33 end of PR – now darwin Emiliano

34 a small building block towards the big vision Emiliano

35 Emiliano Miluzzo from motes to mobile phones

36 Miluzzo evolution of sensing and inference on mobile phones from motes to mobile phones

37 Miluzzo evolution of sensing and inference on mobile phones from motes to mobile phones darwin - classification model evolution - classification model pooling - collaborative inference

38 Miluzzo microphone camera GPS/WiFi/ cellular air quality pollution sensingapps social context audio / pollution / RF fingerprinting image / video manipulation darwin applies distributed computing and collaborative inference concepts to mobile sensing systems darwin - classification model evolution - classification model pooling - collaborative inference

39 why darwin? Miluzzo mobile phone sensing today

40 why darwin? Miluzzo train classification model X in the lab mobile phone sensing today

41 why darwin? Miluzzo deploy classifier X mobile phone sensing today train classification model X in the lab

42 why darwin? Miluzzo train classification model X in the lab deploy classifier X train classification model X in the lab mobile phone sensing today

43 why darwin? Miluzzo deploy classifier X mobile phone sensing today train classification model X in the lab train classification model X in the lab

44 why darwin? Miluzzo train classification model X in the lab deploy classifier X a fully supervised approach doesnt scale! mobile phone sensing today train classification model X in the lab

45 why darwin? a same classifier does not scale to multiple environments (e.g., quiet and noisy env) Miluzzo

46 why darwin? a same classifier does not scale to multiple environments (e.g., quiet and noisy env) Miluzzo

47 why darwin? a same classifier does not scale to multiple environments (e.g., quiet and noisy env) Miluzzo

48 why darwin? a same classifier does not scale to multiple environments (e.g., quiet and noisy env) Miluzzo darwin creates new classification models transparently from the user (classification model evolution ) darwin creates new classification models transparently from the user (classification model evolution )

49 Miluzzo why darwin? ability for an application to rapidly scale to many devices

50 Miluzzo why darwin? ability for an application to rapidly scale to many devices darwin re-uses classification models when possible (classification model pooling ) darwin re-uses classification models when possible (classification model pooling )

51 Miluzzo why darwin? leverage the large ensemble of in-situ resources

52 Miluzzo why darwin? leverage the large ensemble of in-situ resources darwin exploits spatial diversity and co- operate to alleviate the sensing context problem ( collaborative inference ) darwin exploits spatial diversity and co- operate to alleviate the sensing context problem ( collaborative inference )

53 Miluzzo darwin design

54 Miluzzo speaker recognition (subject to audio noise, sensing context, etc.)

55 Miluzzo darwin phases

56 Miluzzo darwin phases initial training (derive model seed) supervised

57 Miluzzo darwin phases initial training (derive model seed) classification model evolution supervised unsupervised

58 Miluzzo darwin phases initial training (derive model seed) classification model evolution classification model pooling supervised unsupervised

59 Miluzzo darwin phases initial training (derive model seed) classification model evolution classification model pooling collaborative inference supervised unsupervised

60 Miluzzo classification model training sensed event

61 Miluzzo classification model training sensed event filtering (silence suppression + voicing)

62 Miluzzo classification model training sensed event filtering (silence suppression + voicing) feature extraction (MFCC) feature extraction (MFCC)

63 Miluzzo classification model training filtering (silence suppression + voicing) feature extraction (MFCC) feature extraction (MFCC) model training (GMM) model training (GMM) model baseline sensed event send model + baseline back to phone send MFCC to backend to train the model backend

64 Miluzzo classification model training phone : feature extraction (low computation) backend backend : model training (high computation)

65 Miluzzo classification model evolution phone : determines when to evolve

66 Miluzzo classification model evolution phone : determines when to evolve training

67 Miluzzo classification model evolution phone : determines when to evolve trainingsampled

68 Miluzzo classification model evolution phone : determines when to evolve match? YES do not evolve

69 Miluzzo classification model evolution phone : determines when to evolve match? NO evolve ( train new model using backend as before)

70 Miluzzo classification model evolution new speaker voice model training

71 Miluzzo classification model evolution new speaker voice model training

72 Miluzzo classification model evolution new speaker voice model training

73 Miluzzo classification model pooling

74 Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker C s model Speaker B s model Speaker C s model

75 Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker C s model Speaker B s model Speaker C s model we have two options 1. train a new classifier for each speaker (costly for power, inference delay)

76 Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker C s model Speaker B s model Speaker C s model we have two options 1. train a new classifier for each speaker (costly for power, inference delay) 2. re-use already available classifiers

77 Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker C s model Speaker B s model Speaker C s model we have two options 1. train a new classifier for each speaker (costly for power, inference delay) 2. re-use already available classifiers

78 Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker C s model Speaker B s model Speaker C s model

79 Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker Bs model Speaker Cs model Speaker B s model Speaker A s model Speaker C s model

80 Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker Bs model Speaker Cs model Speaker B s model Speaker As model Speaker Cs model Speaker A s model Speaker B s model

81 Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker Bs model Speaker Cs model Speaker As model Speaker Bs model Speaker As model Speaker Cs model

82 Miluzzo classification model pooling Speaker A s model Phone A Phone B Phone C Speaker Bs model Speaker Cs model Speaker As model Speaker Bs model Speaker As model Speaker Cs model ready to run the collaborative inference algorithm - local inference first - final inference later ready to run the collaborative inference algorithm - local inference first - final inference later

83 Miluzzo collaborative inference two phases

84 Miluzzo collaborative inference 1. local inference (running independently in parallel on each mobile phone) two phases

85 Miluzzo collaborative inference 1. local inference (running independently in parallel on each mobile phone) two phases 2. final inference (after collecting Local Inference results, to get better confidence about the final classification result)

86 local inference (LI) Miluzzo collaborative inference Phone A Phone B Phone C

87 Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! local inference (LI)

88 Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! local inference (LI) As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10

89 Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! local inference (LI) As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10

90 Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 local inference (LI) Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10

91 Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 local inference (LI) Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10

92 Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! local inference (LI) As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10

93 Miluzzo collaborative inference Phone A Phone B Phone C speaker A speaking!!! local inference (LI) As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10 individual classification can be misleading!

94 final inference (FI) Miluzzo collaborative inference Phone A Phone B Phone C each phone gathers LI results As LI results Cs LI results Bs LI results As LI results Cs LI results Bs LI results

95 final inference (FI) Miluzzo collaborative inference on each phone As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10

96 Miluzzo As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10 x x x x x x final inference (FI) collaborative inference on each phone

97 Miluzzo As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10 x x x x x x FI results (normalized): Confidence (A speaking) = 1 Confidence (B speaking) = 0.12 Confidence (C speaking) = = final inference (FI) collaborative inference on each phone

98 Miluzzo As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10 x x x x x x = FI results (normalized): Confidence (A speaking) = 1 Confidence (B speaking) = 0.12 Confidence (C speaking) = final inference (FI) collaborative inference on each phone

99 Miluzzo As LI results: Prob(A speaking) = 0.65 Prob(B speaking) = 0.25 Prob(C speaking) = 0.10 Cs LI results: Prob(A speaking) = 0.30 Prob(B speaking) = 0.67 Prob(C speaking) = 0.03 Bs LI results: Prob(A speaking) = 0.79 Prob(B speaking) = 0.11 Prob(C speaking) = 0.10 x x x x x x = collaborative inference compensates the inaccuracies of individual inferences FI results (normalized): Confidence (A speaking) = 1 Confidence (B speaking) = 0.12 Confidence (C speaking) = final inference (FI) collaborative inference on each phone

100 Miluzzo evaluation

101 Miluzzo evaluation C/C++ & implemented on Nokia N97 and iPhone in support of a speaker recognition app

102 Miluzzo evaluation C/C++ & unix server implemented on Nokia N97 and iPhone in support of a speaker recognition app

103 Miluzzo evaluation C/C++ & unix server lightweight reliable protocol to transfer models from the server and between phones implemented on Nokia N97 and iPhone in support of a speaker recognition app

104 Miluzzo evaluation C/C++ & UDP multicast protocol to distribute local inference results between phones implemented on Nokia N97 and iPhone in support of a speaker recognition app

105 Miluzzo experimental scenarios up to eight people in conversation in three different scenarios (quiet indoor, down the street, in a restaurant)

106 Miluzzo some numerical results

107 Miluzzo need for evolution train indoor, evaluate outdoor

108 Miluzzo need for evolution accuracy improvement after evolution accuracy

109 Miluzzo indoor quiet scenario 8 people talking around a table

110 Miluzzo indoor quiet scenario 8 people talking around a table

111 Miluzzo indoor quiet scenario 8 people talking around a table

112 Miluzzo indoor quiet scenario 8 people talking around a table

113 Miluzzo indoor quiet scenario 8 people talking around a table

114 Miluzzo indoor quiet scenario 8 people talking around a table

115 Miluzzo indoor quiet scenario 8 people talking around a table collaborative inference + classification model evolution boost the performance of a mobile sensing app

116 Miluzzo impact of the number of mobile phones

117 Miluzzo impact of the number of mobile phones

118 Miluzzo impact of the number of mobile phones

119 Miluzzo impact of the number of mobile phones

120 Miluzzo impact of the number of mobile phones the larger the number of mobile phones collaborating, the better the final inference result

121 Miluzzo battery lifetime Vs inference responsiveness

122 Miluzzo battery lifetime Vs inference responsiveness

123 Miluzzo high responsiveness battery lifetime Vs inference responsiveness

124 Miluzzo short battery life battery lifetime Vs inference responsiveness

125 Miluzzo longer battery duration battery lifetime Vs inference responsiveness

126 Miluzzo low responsiveness battery lifetime Vs inference responsiveness

127 Miluzzo battery lifetime Vs inference responsiveness smart duty-cycling techniques and machine learning algorithms with better performance in terms of energy usage on mobile phones need to be identified

128 Miluzzo a quick recap smartphones are everywhere, lets exploit their collective sensing and computation capabilities

129 Miluzzo a quick recap smartphones are everywhere – lets exploit their collective sensing and computation capabilities smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities

130 Miluzzo a quick recap smartphones are everywhere – lets exploit their collective sensing and computation capabilities continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques) smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities

131 Miluzzo a quick recap smartphones are everywhere – lets exploit their collective sensing and computation capabilities continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques) ML algorithms should perform reliably in the wild smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities

132 Miluzzo a quick recap smartphones are everywhere – lets exploit their collective sensing and computation capabilities continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques) ML algorithms should perform reliably in the wild smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities ok I think Im done…

133 Miluzzo a quick recap smartphones are everywhere – lets exploit their collective sensing and computation capabilities continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user experience (need for energy efficient ML algorithms and smart duty-cycling techniques) ML algorithms should perform reliably in the wild smartphone sensing opens up new frontiers: applications can be spread and big data collected at unprecedented scale enabling endless research opportunities but please bear in mind…

134 Miluzzo Mobile Phone Sensing is the Next Big Thing!

135 Thank you!! Miluzzo Mobile Sensing Group


Download ppt "Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,"

Similar presentations


Ads by Google