Presentation is loading. Please wait.

Presentation is loading. Please wait.

SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones -Hong LU, Wei Pan, Nicholas D. Lane, Tanzeem Choudhury and Andrew T.

Published byGary Houston Modified over 9 years ago

Similar presentations

Presentation on theme: "SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones -Hong LU, Wei Pan, Nicholas D. Lane, Tanzeem Choudhury and Andrew T."— Presentation transcript:

1 SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones -Hong LU, Wei Pan, Nicholas D. Lane, Tanzeem Choudhury and Andrew T. Campbell -Department of Computer Science Dartmouth College, 2009, number of pages 14 Presented By: Rene Chacon

2 Current use of Sound Sensing in mobile phones. SoundSense, a scalable framework for modeling sounds events on mobile phones. Implemented on Apple iPhone, supervised and unsupervised learning techniques for classification. Sound Sense (Abstract) 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

3 Sounds used to dictate a person’s surrounding environment. Audio stream degrades more gracefully (phone in pocket) and as such is still useful. Sound Sense (Introduction) 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

4 Audio Sensing Scalability Problem -Voice, Music, Ambient sounds -hierarchical classification architecture “Goal of SoundSense is to support robust sound processing and classifications under different phone context conditions.” Sound Sense (Section 2) 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

5 Privacy Issues and Resource Limitations -SoundSense processes all audio data on the phone. -Consider the accuracy and cost tradeoff. -Microphone: 4kHz Sound Sense (Section 2) 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

6 Sound Sense (Section 3) 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

7 Preprocessing Framing (64ms) Frame Admission Control rms threshold & entropy threshold Sound Sense (Section 4) 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

8 Coarse Category Classification [voice, music, ambient] -Zero Crossing Rate, human voice has high variation while music and ambient sounds have low. -Spectral Flux (SF) measures the changes in the shape of the spectrum. -Spectral Rolloff (SRF), Spectral Centroid (SC), and more. Sound Sense (Section 4) 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

9 Multi Level Classification – Use decision tree classifier -Utilizing algorithms, Gaussian Mixture Models (GMMs), Hidden Markov Models (HMMs) and more. Sound Sense (Section 4) 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

10 Finer Intra-Category Classification Unsupervised Ambient Sound Learning -utilize Mel Frequency Cepstral Coefficient (MFCC) in depth details of frequency bands sensitive to human ear. -Sound Rank and Bin Replacement Strategy -User Annotation of New Sounds Sound Sense (Section 4) 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

11 Sound Sense (Section 5) 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones PCM data: 8kHz, 16 bit Frame: 512 samples ; lack of acoustic event: long duty cycle state Either an intra-category classifier or unsupervised adaptive classifier

12 Determining buffer sizes -Ex. Output of the decision tree classifier before input to Markov model recognizer. -Ex. MFCC frame length Sound Sense (Section 5) 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

13 SoundSense on iPhone. <60% of the CPU utilized roughly 5MB compare to 30MB Coarse Category Classifier Finer Intra-Category Classifier Sound Sense (Section 6) 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

14 Unsupervised Adaptive Ambient Sound Learning -Future Work, incorporate other types of contextual information such as GPS to further clarify sounds. Sound Sense (Section 6) 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

15 Audio Daily Diary based on Opportunistic Sensing 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

16 Audio Daily Diary based on Opportunistic Sensing 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones http://www.youtube.com/watch?v=VK9AE_ 7dhc4

17 Discover events with music-like sounds (parties, concerts, music in a room). Camera is used to take picture or location and associate. Music Detector based on Participatory Sensing 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

18 Sound Processing and classification is performed real- time as oppose to off-line / external. SoundSense works solely on the mobile phone. Tackles Classification issues with Ambient sounds. Significance of Work 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

19 Paper: SoundSense: Scalable Sound Sensing for People-Centric Application on Mobile Phones Link: http://www.cs.dartmouth.edu/~sensorlab/pubs/s3_m obisys09.pdf http://www.cs.dartmouth.edu/~sensorlab/pubs/s3_m obisys09.pdf MetroSense Project that explores SoundSense http://metrosense.cs.dartmouth.edu/projects.html References 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

20 Questions 1 Paper: SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

Download ppt "SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones -Hong LU, Wei Pan, Nicholas D. Lane, Tanzeem Choudhury and Andrew T."

Similar presentations

Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,

Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,
Outline Activity recognition applications

Outline Activity recognition applications
Voiceprint System Development Design, implement, test unique voiceprint biometric system Research Day Presentation, May 3 rd 2013 Rahul Raj (Team Lead),

Voiceprint System Development Design, implement, test unique voiceprint biometric system Research Day Presentation, May 3 rd 2013 Rahul Raj (Team Lead),
Entropy and Dynamism Criteria for Voice Quality Classification Applications Authors: Peter D. Kukharchik, Igor E. Kheidorov, Hanna M. Lukashevich, Denis.

Entropy and Dynamism Criteria for Voice Quality Classification Applications Authors: Peter D. Kukharchik, Igor E. Kheidorov, Hanna M. Lukashevich, Denis.
Ear-Phone: An End-to-End Participatory Urban Noise Mapping System -Rajib Kumar Rana, Chun Tung Chou, Salil S. Kanhere, Nirupama Bulusu, Wen Hu -School.

Ear-Phone: An End-to-End Participatory Urban Noise Mapping System -Rajib Kumar Rana, Chun Tung Chou, Salil S. Kanhere, Nirupama Bulusu, Wen Hu -School.
Acoustic Model Adaptation Based On Pronunciation Variability Analysis For Non-Native Speech Recognition Yoo Rhee Oh, Jae Sam Yoon, and Hong Kook Kim Dept.

Acoustic Model Adaptation Based On Pronunciation Variability Analysis For Non-Native Speech Recognition Yoo Rhee Oh, Jae Sam Yoon, and Hong Kook Kim Dept.
THE JIGSAW CONTINUOUS SENSING ENGINE FOR MOBILE PHONE APPLICATIONS Hong Lu,† Jun Yang,! Zhigang Liu,! Nicholas D. Lane,† Tanzeem Choudhury,† Andrew T.

THE JIGSAW CONTINUOUS SENSING ENGINE FOR MOBILE PHONE APPLICATIONS Hong Lu,† Jun Yang,! Zhigang Liu,! Nicholas D. Lane,† Tanzeem Choudhury,† Andrew T.
Pitch Prediction From MFCC Vectors for Speech Reconstruction Xu shao and Ben Milner School of Computing Sciences, University of East Anglia, UK Presented.

Pitch Prediction From MFCC Vectors for Speech Reconstruction Xu shao and Ben Milner School of Computing Sciences, University of East Anglia, UK Presented.
Urban Sensing Systems: Opportunistic or Participatory?

Urban Sensing Systems: Opportunistic or Participatory?
Dual-domain Hierarchical Classification of Phonetic Time Series Hossein Hamooni, Abdullah Mueen University of New Mexico Department of Computer Science.

Dual-domain Hierarchical Classification of Phonetic Time Series Hossein Hamooni, Abdullah Mueen University of New Mexico Department of Computer Science.
D u k e S y s t e m s Sensing Meets Mobile Social Networks: The Design, Implementation and Evaluation of the CenceMe Application Emiliano Miluzzo†, Nicholas.

D u k e S y s t e m s Sensing Meets Mobile Social Networks: The Design, Implementation and Evaluation of the CenceMe Application Emiliano Miluzzo†, Nicholas.
DARWIN PHONES: THE EVOLUTION OF SENSING AND INFERENCE ON MOBILE PHONES PRESENTED BY: BRANDON OCHS Emiliano Miluzzo, Cory T. Cornelius, Ashwin Ramaswamy,

DARWIN PHONES: THE EVOLUTION OF SENSING AND INFERENCE ON MOBILE PHONES PRESENTED BY: BRANDON OCHS Emiliano Miluzzo, Cory T. Cornelius, Ashwin Ramaswamy,
NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,

NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,
A Survey of Mobile Phone Sensing

A Survey of Mobile Phone Sensing
Feature Vector Selection and Use With Hidden Markov Models to Identify Frequency-Modulated Bioacoustic Signals Amidst Noise T. Scott Brandes IEEE Transactions.

Feature Vector Selection and Use With Hidden Markov Models to Identify Frequency-Modulated Bioacoustic Signals Amidst Noise T. Scott Brandes IEEE Transactions.
Toward Semantic Indexing and Retrieval Using Hierarchical Audio Models Wei-Ta Chu, Wen-Huang Cheng, Jane Yung-Jen Hsu and Ja-LingWu Multimedia Systems,

Toward Semantic Indexing and Retrieval Using Hierarchical Audio Models Wei-Ta Chu, Wen-Huang Cheng, Jane Yung-Jen Hsu and Ja-LingWu Multimedia Systems,
Chapter 1: Introduction to Pattern Recognition

Chapter 1: Introduction to Pattern Recognition
Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John.

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John.
SENSING MEETS MOBILE SOCIAL NETWORKS: THE DESIGN, IMPLEMENTATION AND EVALUATION OF THE CENCEME APPLICATION Emiliano Miluzzo†, Nicholas D. Lane†, Kristóf.

SENSING MEETS MOBILE SOCIAL NETWORKS: THE DESIGN, IMPLEMENTATION AND EVALUATION OF THE CENCEME APPLICATION Emiliano Miluzzo†, Nicholas D. Lane†, Kristóf.
A Practical Approach to Recognizing Physical Activities Jonathan Lester, Tanzeem Choudhury, and Gaetano Borriello In Proceedings of the Fourth International.

A Practical Approach to Recognizing Physical Activities Jonathan Lester, Tanzeem Choudhury, and Gaetano Borriello In Proceedings of the Fourth International.

Similar presentations

Ads by Google