Deep Learning and its applications to Speech EE 225D - Audio Signal Processing in Humans and Machines Oriol Vinyals UC Berkeley.

Slides:

Advertisements

Similar presentations

Thomas Trappenberg Autonomous Robotics: Supervised and unsupervised learning.

Advertisements

Advanced topics.

Rajat Raina Honglak Lee, Roger Grosse Alexis Battle, Chaitanya Ekanadham, Helen Kwong, Benjamin Packer, Narut Sereewattanawoot Andrew Y. Ng Stanford University.

1 Neural Networks - Basics Artificial Neural Networks - Basics Uwe Lämmel Business School Institute of Business Informatics

An introduction to: Deep Learning aka or related to Deep Neural Networks Deep Structural Learning Deep Belief Networks etc,

Neural NetworksNN 11 Neural Networks Teacher: Elena Marchiori R4.47 Assistant: Kees Jong S2.22

F 鍾承道 Acoustic Features for Speech Recognition: From Mel-Frequency Cepstrum Coefficients (MFCC) to BottleNeck Features(BNF)

Supervised and Unsupervised learning and application to Neuroscience Cours CA6b-4.

How to do backpropagation in a brain

Machine Learning and having it deep and structured

Face Recognition Using Neural Networks Presented By: Hadis Mohseni Leila Taghavi Atefeh Mirsafian.

Stanford CS224S Spring 2014 CS 224S / LINGUIST 285 Spoken Language Processing Andrew Maas Stanford University Spring 2014 Lecture 16: Acoustic Modeling.

CSC 4510 – Machine Learning Dr. Mary-Angela Papalaskari Department of Computing Sciences Villanova University Course website:

Traffic Sign Recognition Using Artificial Neural Network Radi Bekker

LLNL-PRES This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.

Using Neural Networks in Database Mining Tino Jimenez CS157B MW 9-10:15 February 19, 2009.

Chapter 3 Neural Network Xiu-jun GONG (Ph. D) School of Computer Science and Technology, Tianjin University

Qual Presentation Daniel Khashabi 1. Outline  My own line of research  Papers:  Fast Dropout training, ICML, 2013  Distributional Semantics Beyond.

Mestrado em Ciência de Computadores Mestrado Integrado em Engenharia de Redes e Sistemas Informáticos VC 14/15 – TP19 Neural Networks & SVMs Miguel Tavares.

Neural Network Introduction Hung-yi Lee. Review: Supervised Learning Training: Pick the “best” Function f * Training Data Model Testing: Hypothesis Function.

From Machine Learning to Deep Learning. Topics that I will Cover (subject to some minor adjustment) Week 2: Introduction to Deep Learning Week 3: Logistic.

Building high-level features using large-scale unsupervised learning Anh Nguyen, Bay-yuan Hsu CS290D – Data Mining (Spring 2014) University of California,

Dr. Z. R. Ghassabi Spring 2015 Deep learning for Human action Recognition 1.

EE459 Neural Networks Examples of using Neural Networks Kasin Prakobwaitayakit Department of Electrical Engineering Chiangmai University.

Akram Bitar and Larry Manevitz Department of Computer Science

Introduction to Neural Networks Introduction to Neural Networks Applied to OCR and Speech Recognition An actual neuron A crude model of a neuron Computational.

Neural Networks Presented by M. Abbasi Course lecturer: Dr.Tohidkhah.

Neural Networks Teacher: Elena Marchiori R4.47 Assistant: Kees Jong S2.22

Introduction to Deep Learning

CS 188: Artificial Intelligence Learning II: Linear Classification and Neural Networks Instructors: Stuart Russell and Pat Virtue University of California,

ImageNet Classification with Deep Convolutional Neural Networks Presenter: Weicong Chen.

Object Recognizing. Deep Learning Success in 2012 DeepNet and speech processing.

1 LING 696B: Final thoughts on nonparametric methods, Overview of speech processing.

Introduction to Convolutional Neural Networks

City Forensics: Using Visual Elements to Predict Non-Visual City Attributes Sean M. Arietta, Alexei A. Efros, Ravi Ramamoorthi, Maneesh Agrawala Presented.

Big data classification using neural network

Convolutional Neural Network

CS 388: Natural Language Processing: LSTM Recurrent Neural Networks

The Relationship between Deep Learning and Brain Function

Deep Learning Amin Sobhani.

Data Mining, Neural Network and Genetic Programming

CSE 190 Neural Networks: How to train a network to look and see

ECE 5424: Introduction to Machine Learning

Computer Science and Engineering, Seoul National University

Article Review Todd Hricik.

Many slides and slide ideas thanks to Marc'Aurelio Ranzato and Michael Nielson.

Matt Gormley Lecture 16 October 24, 2016

Lecture 24: Convolutional neural networks

Intelligent Information System Lab

Neural networks (3) Regularization Autoencoder

ECE 6504 Deep Learning for Perception

Deep learning and applications to Natural language processing

RECURRENT NEURAL NETWORKS FOR VOICE ACTIVITY DETECTION

AV Autonomous Vehicles.

Deep Belief Networks Psychology 209 February 22, 2013.

"Playing Atari with deep reinforcement learning."

Deep Learning Hierarchical Representations for Image Steganalysis

Deep learning Introduction Classes of Deep Learning Networks

XOR problem Input 2 Input 1

A Proposal Defense On Deep Residual Network For Face Recognition Presented By SAGAR MISHRA MECE

On Convolutional Neural Network

Deep Learning for Non-Linear Control

Visualizing and Understanding Convolutional Networks

Deep Learning Some slides are from Prof. Andrew Ng of Stanford.

Neural networks (3) Regularization Autoencoder

Deep Learning Authors: Yann LeCun, Yoshua Bengio, Geoffrey Hinton

An introduction to: Deep Learning aka or related to Deep Neural Networks Deep Structural Learning Deep Belief Networks etc,

Sanguthevar Rajasekaran University of Connecticut

CSC 578 Neural Networks and Deep Learning

Machine Learning.

Presentation transcript:

Deep Learning and its applications to Speech EE 225D - Audio Signal Processing in Humans and Machines Oriol Vinyals UC Berkeley

● This is my biased view about deep learning and, more generally, machine learning past and current research! Disclaimer

● It’s a hot topic… isn’t it? ● Why this talk?

● Let x be a signal (or features in machine learning jargon), want to find a function f that maps x to an output y: ● Waveform “x” to sentence “y” (ASR) ● Image “x” to face detection “y” (CV) ● Weather measurements “x” to forecast “y” (…) ● Machine learning approach: ● Get as many (x,y) pairs as possible, and find f minimizing some loss over the training pairs ● Supervised ● Unsupervised Let’s step back to a ML formulation

(slide credit: Eric Xing, CMU) NN

● Universal approximation thm.: ● We can approximate any (continuous) function on a compact set with a single hidden neural network Can’t we do everything with NNs?

● It has two (possibly more) meanings: ● Use many layers in a NN ● Train each layer in an unsupervised fashion ● G. Hinton (U. of T.) et al made these two ideas famous in his 2006 Science paper. Deep Learning

2006 Science paper (G. Hinton et al)

Great results using Deep Learning

Deep Learning in Speech Feature extraction Phone probabilities HMM

● Small scale (TIMIT) ● Many papers, most recent: [Deng et al, Interspeech11] ● Small scale (Aurora) ● 50% rel. impr. [Vinyals et al, ICASSP11/12] ● ~Med/Lg scale (Switchboard) ● 30% rel. impr. [Seide et al, Interspeech11] ● … more to come Some interesting ASR results

● Model strength vs. generalization error ● Deep architectures: more parameters more efficiently… Why? Why is deep better?

● Most relevant work by B. Olshausen (1997!) “Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1?” ● Take a bunch of random natural images, do unsupervised learning, you recover filters that look exactly the same as V1! Is this how the brain really works?

● People knew about NN for very long, why the hype now? ● Computational power? ● More data available? ● Connection with neuroscience? ● Can we computationally emulate a brain? ● ~10^11 neurons, ~10^15 connections ● Biggest NN: ~10^4 neurons, ~10^8 connections ● Many connections flow backwards ● Brain understanding is far from complete Criticisms/open questions

Questions?