Deep learning Tsai bing-chen 10/22

Neural networks Back-propagation targets hidden layers(unknown features) input datas

What’s wrong with back-propagation? It requires labeled training data. Almost all data is unlabeled. The learning time does not scale well It is very slow in networks with multiple hidden layers. It can get stuck in poor local optima.

Deep belief networks Learn P( image) not P(label | image) It is generative model Graphical model hidden visible

Graphical model Undirected graphical model Directed graphical model Boltzmann machines Directed graphical model Belief nets 𝑃 𝑿 = 1 𝑍 exp 𝑖𝑗 𝑊 𝑖𝑗 𝑥 𝑖 𝑥 𝑗 + 𝑖 𝑥 𝑖 𝑏 𝑖 𝑃 𝑿 = 𝑖 𝑃( 𝑥 𝑖 |𝑝𝑎𝑟𝑒𝑛𝑡 𝑠 𝑖 )

Graphical model Undirected graphical model Directed graphical model Restricted boltzmann machines Directed graphical model Sigmoid belief nets 𝑃 𝒗,𝒉 = 1 𝑍 exp 𝑖𝑗 𝑊 𝑖𝑗 𝑣 𝑖 ℎ 𝑗 + 𝑖 𝑣 𝑖 𝑏 𝑖 + 𝑗 ℎ 𝑗 𝑐 𝑗

Deep belief nets Two problem Learning problem: Adjust the interactions between variables to make the network more likely to generate the visible data. Inference problem: Infer the states of the unobserved variables. biases 要介紹一下weight , biases和hidden 的感覺 hidden layers 𝑊 2 𝑤𝑒𝑖𝑔ℎ𝑡𝑠 𝑊 1 𝑤𝑒𝑖𝑔ℎ𝑡𝑠 visible

… … … Deep belief nets 𝑡ℎ𝑒 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝑟𝑎𝑡𝑒 𝑖𝑠 ∆ W jk ∝ 𝑣 𝑘 − 𝑝 𝑘 ℎ 𝑗 Use gibbs sampling to sample distribution P(h | v) is complex hidden_1 hidden _ infinity … … P(h | v) P(v | h) … visible_1 visible_2 visible _ infinity 𝑡ℎ𝑒 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝑟𝑎𝑡𝑒 𝑖𝑠 ∆ W jk ∝ 𝑣 𝑘 − 𝑝 𝑘 ℎ 𝑗

… … … Deep belief nets Complementary priors P(h) 𝑃 𝒉 𝒗 = 𝑖 𝑃( ℎ 𝑖 |𝒗) No explaining away P(h|v) is factorial P(h) is not factorial 𝑃 𝒉 𝒗 = 𝑖 𝑃( ℎ 𝑖 |𝒗) … … P(h|v) P(h|v) …

. . . Deep belief nets It is infinite sigmoid belief net P(v | h) is sigmoid function , but P(h | v) not . P(v |h) . . 把整個training的問題拉成一個infinite belief nets 看 雖然沒有explaining away 的問題 但是p(h|v)還是不好解,所以會想簡化問題 讓p(h|v)變成一個sigmoid的function P(v |h) P(h |v)

Deep belief nets Restricted boltzmann machines P(v | h) and P(h | v) are sigmoid function 𝑃 𝑣,ℎ = 1 𝑍 exp⁡( 𝑖𝑗 𝑊 𝑖𝑗 𝑣 𝑖 ℎ 𝑗 + 𝑖 𝑣 𝑖 𝑏 𝑖 + 𝑗 ℎ 𝑗 𝑐 𝑗 )

. . . Deep belief nets RBM is a infinity belief nets 𝑊 . 𝑊 𝑇 . 𝑃 ℎ 𝑖 𝑣 =𝜎( 𝑊 𝑇 𝑣+𝑐) . 𝑊 𝑊 𝑇 𝑃 𝑣 𝑖 ℎ =𝜎(𝑊ℎ+𝑏) 𝑊 𝑇 𝑊 𝑊 𝑊 𝑇

Deep belief nets Use RBM to training every layer Train W4 Train W3 freeze W3 Train W2 freeze W2 freeze W2 Train W1 freeze W1 freeze W1 freeze W1

classification 2000 top- level nodes 10 label nodes 500 nodes 28*28 pixel image A fast learning algorithm for deep belief nets“ G. E. Hinton, S. Osindero, and Y. W. Teh,2006

reference "A fast learning algorithm for deep belief nets“ G. E. Hinton, S. Osindero, and Y. W. Teh,2006 "Deep boltzmann machines" R. Salakhutdinov and G. E. Hinton,2009 "Efficient learning of deep boltzmann machines“ R. Salakhutdinov and H. Larochelle,2010 “An Efficient Learning Procedure for Deep Boltzmann Machines“ R. Salakhutdinov and G. E. Hinton,2010 " NIPS tutorial , Deep Belief Nets“ G. E. Hinton,2007 " MLSS Tutorial,Deep belief nets “ Marcus Frean,2010