Presentation is loading. Please wait.

Presentation is loading. Please wait.

Comp 3503 / 5013 Dynamic Neural Networks Daniel L. Silver March, 2014 1.

Published byAnn Conley Modified over 9 years ago

Similar presentations

Presentation on theme: "Comp 3503 / 5013 Dynamic Neural Networks Daniel L. Silver March, 2014 1."— Presentation transcript:

1 Comp 3503 / 5013 Dynamic Neural Networks Daniel L. Silver March, 2014 1

2 Outline Hopfield Networks Boltzman Machines Mean Field Theory Restricted Boltzman Machines (RBM) 2

3 Dynamic Neural Networks See handout for image of spider, beer and dog The search for a model or hypothesis can be considered the relaxation of a dynamic system into a state of equilibrium This is the nature of most physical systems – Pool of water – Air in a room Mathematics is that of thermal-dynamics – Quote from John Von Neumann 3

4 Hopfield Networks See hand out 4

5 Hopfield Networks Hopfield Network video intro – http://www.youtube.com/watch?v=gfPUWwBkXZ Y http://www.youtube.com/watch?v=gfPUWwBkXZ Y – http://faculty.etsu.edu/knisleyj/neural/ http://faculty.etsu.edu/knisleyj/neural/ Try these Applets: – http://lcn.epfl.ch/tutorial/english/hopfield/html/i ndex.html http://lcn.epfl.ch/tutorial/english/hopfield/html/i ndex.html – http://www.cbu.edu/~pong/ai/hopfield/hopfielda pplet.html http://www.cbu.edu/~pong/ai/hopfield/hopfielda pplet.html 5

6 Hopfield Networks Basics with Geoff Hinton: Introduction to Hopfield Nets – http://www.youtube.com/watch?v=YB3-Hn-inHI http://www.youtube.com/watch?v=YB3-Hn-inHI Storage capacity of Hopfield Nets – http://www.youtube.com/watch?v=O1rPQlKQBLQ http://www.youtube.com/watch?v=O1rPQlKQBLQ 6

7 Hopfield Networks Advanced concepts with Geoff Hinton: Hopfield nets with hidden units – http://www.youtube.com/watch?v=bOpddsa4BPI http://www.youtube.com/watch?v=bOpddsa4BPI Necker Cube – http://www.cs.cf.ac.uk/Dave/JAVA/boltzman/Neck er.html http://www.cs.cf.ac.uk/Dave/JAVA/boltzman/Neck er.html Adding noise to improve search – http://www.youtube.com/watch?v=kVgT2Eaa6KA http://www.youtube.com/watch?v=kVgT2Eaa6KA 7

8 Boltzman Machine -See Handout -http://www.scholarpedia.org/article/Boltzmann_machinehttp://www.scholarpedia.org/article/Boltzmann_machine Basics with Geoff Hinton Modeling binary data – http://www.youtube.com/watch?v=MKdvJst8a6k http://www.youtube.com/watch?v=MKdvJst8a6k BM Learning Algorithm – http://www.youtube.com/watch?v=QgrFsnHFeig http://www.youtube.com/watch?v=QgrFsnHFeig 8

9 Limitations of BMs BM Learning does not scale well This is due to several factors, the most important being: – The time the machine must be run in order to collect equilibrium statistics grows exponentially with the machine's size = number of nodes For each example – sample nodes, sample states – Connection strengths are more plastic when the units have activation probabilities intermediate between zero and one. Noise causes the weights to follow a random walk until the activities saturate (variance trap). 9

10 Potential Solutions Use a momentum term as in BP: Add a penalty term to create sparse coding (encourage shorter encodings for different inputs) Use implementation tricks to do more in memory – batches of examples Restrict number of iterations in + and – phases Restrict connectivity of network 10 w ij (t+1)=w ij (t) +ηΔw ij +αΔw ij (t-1)

11 Restricted Boltzman Machine 11 Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ SF/Fantasy Oscar Winner w ij j i Σ j =w ij v i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Recall = Relaxation Σ i =w ij h j v o or h o

12 Restricted Boltzman Machine 12 Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ SF/Fantasy Oscar Winner w ij j i Σ j =w ij v i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Recall = Relaxation Σ i =w ij h j v o or h o

13 Restricted Boltzman Machine 13 Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ SF/Fantasy Oscar Winner j i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Σ i =w ij h j v o or h o Oscar Winner SF/Fantasy Recall = Relaxation w ij Σ j =w ij v i

14 Restricted Boltzman Machine 14 Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ SF/Fantasy Oscar Winner j i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Σ i =w ij h j v o or h o Oscar Winner SF/Fantasy Recall = Relaxation w ij Σ j =w ij v i

15 Restricted Boltzman Machine 15 Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ SF/Fantasy Oscar Winner j i Σ i =w ij h j h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Learning = ~ Gradient Descent = Constrastive Divergence Update hidden units P=P+v i h j v o or h o Σ j =w ij v i

16 Restricted Boltzman Machine 16 Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ SF/Fantasy Oscar Winner j i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Learning = ~ Gradient Descent = Constrastive Divergence Reconstruct visible units v o or h o Σ j =w ij v i Σ i =w ij h j

17 Restricted Boltzman Machine 17 Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ SF/Fantasy Oscar Winner j i Σ j =w ij v i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Learning = ~ Gradient Descent = Constrastive Divergence Reupdate hidden units v o or h o Σ i =w ij h j N=N+v i h j

18 Restricted Boltzman Machine 18 Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ SF/Fantasy Oscar Winner Δw ij = - j i Σ j =w ij v i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Σ i =w ij h j v o or h o w ij =w ij +ηΔw ij Learning = ~ Gradient Descent = Constrastive Divergence Update weights

19 Restricted Boltzman Machine RBM Overview: – http://blog.echen.me/2011/07/18/introduction- to-restricted-boltzmann-machines/ http://blog.echen.me/2011/07/18/introduction- to-restricted-boltzmann-machines/ Wikipedia on DLA and RBM: – http://en.wikipedia.org/wiki/Deep_learning http://en.wikipedia.org/wiki/Deep_learning RBM Details and Code: – http://www.deeplearning.net/tutorial/rbm.html http://www.deeplearning.net/tutorial/rbm.html 19

20 Restricted Boltzman Machine Geoff Hinton on RBMs: RBMs and Constrastive Divergence Algorithm – http://www.youtube.com/watch?v=fJjkHAuW0Yk http://www.youtube.com/watch?v=fJjkHAuW0Yk An example of RBM Learning – http://www.youtube.com/watch?v=Ivj7jymShN0 http://www.youtube.com/watch?v=Ivj7jymShN0 RBMs applied to Collaborative Filtering – http://www.youtube.com/watch?v=laVC6WFIXjg http://www.youtube.com/watch?v=laVC6WFIXjg 20

21 Additional References Coursera course – Neural Networks fro Machine Learning: – https://class.coursera.org/neuralnets-2012- 001/lecture https://class.coursera.org/neuralnets-2012- 001/lecture ML: Hottest Tech Trend in next 3-5 Years – http://www.youtube.com/watch?v=b4zr9Zx5WiE http://www.youtube.com/watch?v=b4zr9Zx5WiE 21

Download ppt "Comp 3503 / 5013 Dynamic Neural Networks Daniel L. Silver March, 2014 1."

Similar presentations

Bioinspired Computing Lecture 16

Bioinspired Computing Lecture 16
Greedy Layer-Wise Training of Deep Networks

Greedy Layer-Wise Training of Deep Networks
Thomas Trappenberg Autonomous Robotics: Supervised and unsupervised learning.

Thomas Trappenberg Autonomous Robotics: Supervised and unsupervised learning.
NEURAL NETWORKS Backpropagation Algorithm

NEURAL NETWORKS Backpropagation Algorithm
Deep Learning Bing-Chen Tsai 1/21.

Deep Learning Bing-Chen Tsai 1/21.
Stochastic Neural Networks Deep Learning and Neural Nets Spring 2015.

Stochastic Neural Networks Deep Learning and Neural Nets Spring 2015.
Advanced topics.

Advanced topics.
CS 678 –Boltzmann Machines1 Boltzmann Machine Relaxation net with visible and hidden units Learning algorithm Avoids local minima (and speeds up learning)

CS 678 –Boltzmann Machines1 Boltzmann Machine Relaxation net with visible and hidden units Learning algorithm Avoids local minima (and speeds up learning)
Tuomas Sandholm Carnegie Mellon University Computer Science Department

Tuomas Sandholm Carnegie Mellon University Computer Science Department
Learning in Recurrent Networks Psychology 209 February 25, 2013.

Learning in Recurrent Networks Psychology 209 February 25, 2013.
Machine Learning: Connectionist McCulloch-Pitts Neuron Perceptrons Multilayer Networks Support Vector Machines Feedback Networks Hopfield Networks.

Machine Learning: Connectionist McCulloch-Pitts Neuron Perceptrons Multilayer Networks Support Vector Machines Feedback Networks Hopfield Networks.
Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl.

Structure learning with deep neuronal networks 6 th Network Modeling Workshop, 6/6/2013 Patrick Michl.
How to do backpropagation in a brain

How to do backpropagation in a brain
INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, 2004 Lecture Slides for.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.
Artificial Neural Networks ML 4.6-4.9 Paul Scheible.

Artificial Neural Networks ML Paul Scheible.
Deep Belief Networks for Spam Filtering

Deep Belief Networks for Spam Filtering
MAE 552 Heuristic Optimization Instructor: John Eddy Lecture #31 4/17/02 Neural Networks.

MAE 552 Heuristic Optimization Instructor: John Eddy Lecture #31 4/17/02 Neural Networks.
MACHINE LEARNING 12. Multilayer Perceptrons. Neural Networks Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

MACHINE LEARNING 12. Multilayer Perceptrons. Neural Networks Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)
CSC321: Introduction to Neural Networks and Machine Learning Lecture 20 Learning features one layer at a time Geoffrey Hinton.

CSC321: Introduction to Neural Networks and Machine Learning Lecture 20 Learning features one layer at a time Geoffrey Hinton.
Comp 5013 Deep Learning Architectures Daniel L. Silver March, 2014 1.

Comp 5013 Deep Learning Architectures Daniel L. Silver March,

Similar presentations

Ads by Google