1. Deep learning
Tsai bing-chen
10/22

2. Neural networks Back-propagation targets
hidden layers(unknown features)
input datas

3. What’s wrong with back-propagation?
It requires labeled training data.
Almost all data is unlabeled.
The learning time does not scale well
It is very slow in networks with multiple hidden layers.
It can get stuck in poor local optima.

4. Deep belief networks Learn P( image) not P(label | image)
It is generative model
Graphical model
hidden
visible

5. Graphical model Undirected graphical model Directed graphical model
Boltzmann machines
Directed graphical model
Belief nets
𝑃 𝑿 = 1 𝑍 exp 𝑖𝑗 𝑊 𝑖𝑗 𝑥 𝑖 𝑥 𝑗 + 𝑖 𝑥 𝑖 𝑏 𝑖
𝑃 𝑿 = 𝑖 𝑃( 𝑥 𝑖 |𝑝𝑎𝑟𝑒𝑛𝑡 𝑠 𝑖 )

6. Graphical model Undirected graphical model Directed graphical model
Restricted boltzmann machines
Directed graphical model
Sigmoid belief nets
𝑃 𝒗,𝒉 = 1 𝑍 exp 𝑖𝑗 𝑊 𝑖𝑗 𝑣 𝑖 ℎ 𝑗 + 𝑖 𝑣 𝑖 𝑏 𝑖 + 𝑗 ℎ 𝑗 𝑐 𝑗

7. Deep belief nets Two problem
Learning problem: Adjust the interactions between variables to make the network more likely to generate the visible data.
Inference problem: Infer the states of the unobserved variables.
biases
要介紹一下weight , biases和hidden 的感覺
hidden layers
𝑊 2 𝑤𝑒𝑖𝑔ℎ𝑡𝑠
𝑊 1 𝑤𝑒𝑖𝑔ℎ𝑡𝑠
visible

8. … … … Deep belief nets 𝑡ℎ𝑒 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝑟𝑎𝑡𝑒 𝑖𝑠 ∆ W jk ∝ 𝑣 𝑘 − 𝑝 𝑘 ℎ 𝑗
Use gibbs sampling to sample distribution
P(h | v) is complex
hidden_1
hidden _ infinity … …
P(h | v)
P(v | h) …
visible_1
visible_2
visible _ infinity
𝑡ℎ𝑒 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝑟𝑎𝑡𝑒 𝑖𝑠 ∆ W jk ∝ 𝑣 𝑘 − 𝑝 𝑘 ℎ 𝑗

9. … … … Deep belief nets Complementary priors P(h) 𝑃 𝒉 𝒗 = 𝑖 𝑃( ℎ 𝑖 |𝒗)
No explaining away
P(h|v) is factorial
P(h) is not factorial
𝑃 𝒉 𝒗 = 𝑖 𝑃( ℎ 𝑖 |𝒗) … …
P(h|v)
P(h|v) …

10. . . . Deep belief nets It is infinite sigmoid belief net
P(v | h) is sigmoid function , but P(h | v) not .
P(v |h) . .
把整個training的問題拉成一個infinite belief nets 看
雖然沒有explaining away 的問題
但是p(h|v)還是不好解,所以會想簡化問題 讓p(h|v)變成一個sigmoid的function
P(v |h)
P(h |v)

11. Deep belief nets Restricted boltzmann machines
P(v | h) and P(h | v) are sigmoid function
𝑃 𝑣,ℎ = 1 𝑍 exp⁡( 𝑖𝑗 𝑊 𝑖𝑗 𝑣 𝑖 ℎ 𝑗 + 𝑖 𝑣 𝑖 𝑏 𝑖 + 𝑗 ℎ 𝑗 𝑐 𝑗 )

12. . . . Deep belief nets RBM is a infinity belief nets𝑊 .
𝑊 𝑇 .
𝑃 ℎ 𝑖 𝑣 =𝜎( 𝑊 𝑇 𝑣+𝑐) . 𝑊
𝑊 𝑇
𝑃 𝑣 𝑖 ℎ =𝜎(𝑊ℎ+𝑏)
𝑊 𝑇 𝑊 𝑊
𝑊 𝑇

13. Deep belief nets Use RBM to training every layer Train W4 Train W3
freeze W3
Train W2
freeze W2
freeze W2
Train W1
freeze W1
freeze W1
freeze W1

14. classification 2000 top- level nodes 10 label nodes 500 nodes
28*28 pixel image
A fast learning algorithm for deep belief nets“
G. E. Hinton, S. Osindero, and Y. W. Teh,2006

15. reference "A fast learning algorithm for deep belief nets“
G. E. Hinton, S. Osindero, and Y. W. Teh,2006
"Deep boltzmann machines"
R. Salakhutdinov and G. E. Hinton,2009
"Efficient learning of deep boltzmann machines“
R. Salakhutdinov and H. Larochelle,2010
“An Efficient Learning Procedure for Deep Boltzmann Machines“
R. Salakhutdinov and G. E. Hinton,2010
" NIPS tutorial , Deep Belief Nets“
G. E. Hinton,2007
" MLSS Tutorial,Deep belief nets “
Marcus Frean,2010