2. Neural Network and Deep Learning 王强昌 2015-11-24 MLA lab

3. Neural Network and Deep Learning  Artificial Neural Network  Why we go deep?  Amazing achievement  Deep learning-getting started

4. Artificial Neural Network  How do ANNs work?  Feed-forward process  About Weights Gradient Descent Back-propagation process  Summaries

5.  Artificial Neural Network (ANN) is a technique for solving problems by constructing software that works like our brains.

6.  Our brains are a huge network of processing elements. A typical brain contains a network of 10 billion neurons.

7. An artificial neuron is an imitation of a human neuron

8.  Now, let us have a look at the model of an artificial neuron.

9. Transfer Function (Activation Function) Output x1x1 x2x2 xmxm ∑ y Neuron Input w1w1 w2w2 wmwm Weights...... f(v k ).....

10.  An example Sum : (1  0.25) + (0.5  (-1.5)) = 0.25 + (-0.75) = - 0.5 Transfer function : if we get then

11.  Transfer (Activation, Squash) function: Limits node output; enhances Non-linearity. For the function below, we limit the output in the range [0,1]. An example:

12. Artificial Neural Network  How do ANNs work?  Feed-forward process  About Weights Gradient Descent Back-propagation process  Summaries

13. Feed-forward process  Information flow is unidirectional Data is presented to input layer  Data example Pixel intensity (for image classification). Share prices (for stock market prediction ). Passed on to hidden Layer Passed on to output layer Hidden layer: internal representation (interpretation) of data. Layer

14. Picture below illustrates how data is propagated through the network.  w (xm)n represent weights of connections between network input x m and neuron n in next layer.  y n represents output of neuron n. Input layerHidden layer Output layer

15. Input layerHidden layer Output layer

16. Propagation of data through the hidden layer. w mn represent weights of connections between neuron m and neuron n in the next layer. Output layer Hidden layer Input layer

17. Propagation of data through the output layer. Input layerHidden layer Output layer

18. Artificial Neural Network  How do ANNs work?  Feed-forward process  About Weights Gradient Descent Back-propagation process  Summaries

19. About Weights  Weights w settings determine the behaviour of a network  How can we find the right weights ?

20. Example: Voice Recognition  Task: Learn to discriminate between two different voices saying “Hello”  Data  Sources Steve David  Input data Frequency distribution (60 bins)

21.  Network architecture  Feed forward network (predefined) 60 input units (one for every frequency bin) 6 hidden units 2 output units (0-1 for “Steve”, 1-0 for “David”)

22.  Presenting the data Steve David

23.  Presenting the data (untrained network) Steve 0.43 0.57 David 0.7 0.3

24.  Calculate error (suppose the error function is absolute value function) Steve:0-1 |0.43 – 0| = 0.43 |0.57 – 1| = 0.43 David:1-0 |0.7 – 1| = 0.3 |0.3 – 0| = 0.3

25.  Backprop error and adjust weights (just the last hidden layer) Steve |0.43 – 0| = 0.43 |0.57 – 1| = 0.43 David |0.7 – 1| = 0.3 |0.3 – 0| = 0.3 How do we adjust the weights ? weights

26. Artificial Neural Network  How do ANNs work?  Feed-forward process  About Weights Gradient Descent Back-propagation process  Summaries

27. Gradient Descent  Think of (w 0,w 1,…,w n-1 ) as a point in an n-dimensional space.  Suppose the error function is E(w 0,w 1,…,w n-1 ).  Try to minimize error E(w 0,w 1,…,w n-1 ) by changing the point position on the “error surface”.

28.  How do we change w i ? Change i-th weight by η w i = η*   : direction of going down.   η: length of going down, a constant. w i (new)=w i (old)+ w i

29.  Repeat the procedure above, we can finally get the minimum. But we need to compute derivative first ! Grad E =[,, …, ]

30. Artificial Neural Network  How do ANNs work?  Feed-forward process  About Weights Gradient Descent Back-propagation process  Summaries

31. Back-propagation process  the output of the network y is compared with the desired output z (the target), compute the error, suppose we get error function

32.  The idea is to propagate error back to all neurons.  The weights' coefficients w mn used to propagate errors back are equal to this used during feed-forward process.  The direction of data flow is changed (signals are propagated from output to inputs one after the other). depends on what function f(e) is. if f(e)=e, then 链式求导 w 56

33.  If propagated errors came from few neurons they are added. The illustration is below: w 46 链式求导 depends on what function f(e) is. if f(e)=e, then w 24 w 34

34.  Continue to propagate the error, we can modify the weights for the inputs nodes:

35. Artificial Neural Network  How do ANNs work?  Feed-forward process  About Weights Gradient Descent Back-propagation process  Summaries

36. Summaries  1. Initialize network with random weights.  2. For all training cases, repeat:  a. Feed-forward process: present training inputs to network and calculate output.  b.  b. Back-propagation process: for all layers (starting with output layer, back to input layer): Computes the error term for the output units using the observed error. From output layer, repeat - propagating the error term back to the previous layer - updating the weights between the two layers until the earliest hidden layer is reached.

37. Neural Network and Deep Learning  Artificial Neural Network  Why we go deep?  Amazing achievement  Deep learning-getting started

38. Why we go deep?  Learning multiple levels of representation

39.  Learning Non-Linear Features

41.  Learning features, not just handcrafting them. Most ML systems use very carefully hand-designed features and representations. So, many practitioners are very experienced – and good at such feature design (or kernel design). Hand-crafting features are brittle, incomplete.

42.  Highly varying functions can be efficiently represented with deep architectures.  Problems which can be represented with a polynomial number of nodes with k layers, may require an exponential number of nodes with k-1 layers.

43. Neural Network and Deep Learning  Artificial Neural Network  Why we go deep?  Amazing achievement  Deep learning-getting started

44. Amazing achievement on ImageNet classification  Database: part of ImageNet database, 1000 categories, 1.2 million training images, 150,000 testing images.  Task: classify testing image into one of 1000 categories.  Examples of two categories needed to be differentiated: Differentiate 波斯猫 布偶猫

45. Machine is as good as human !!! A New Era Begins: Deep Convolutional Neural Network (DCNN)

46. Neural Network and Deep Learning  Artificial Neural Network  Why we go deep?  Amazing achievement  Deep learning-getting started

47. Deep learning-getting started  http://blog.csdn.net/zouxy09/article/details/8775360 http://blog.csdn.net/zouxy09/article/details/8775360 了解一些 deep learning 基本方法的思想  http://ufldl.stanford.edu/wiki/index.php/UFLDL 教程 deep learning 大牛 Andrew Ng 所写 ，还 有 实验、 源代 码， 推 荐 细读

48. Thank You!