Neural Network and Deep Learning 王强昌 MLA lab

Neural Network and Deep Learning  Artificial Neural Network  Why we go deep?  Amazing achievement  Deep learning-getting started

Artificial Neural Network  How do ANNs work?  Feed-forward process  About Weights Gradient Descent Back-propagation process  Summaries

 Artificial Neural Network (ANN) is a technique for solving problems by constructing software that works like our brains.

 Our brains are a huge network of processing elements. A typical brain contains a network of 10 billion neurons.

An artificial neuron is an imitation of a human neuron

 Now, let us have a look at the model of an artificial neuron.

Transfer Function (Activation Function) Output x1x1 x2x2 xmxm ∑ y Neuron Input w1w1 w2w2 wmwm Weights f(v k ).....

 An example Sum : (1  0.25) + (0.5  (-1.5)) = (-0.75) = Transfer function : if we get then

 Transfer (Activation, Squash) function: Limits node output; enhances Non-linearity. For the function below, we limit the output in the range [0,1]. An example:

Artificial Neural Network  How do ANNs work?  Feed-forward process  About Weights Gradient Descent Back-propagation process  Summaries

Feed-forward process  Information flow is unidirectional Data is presented to input layer  Data example Pixel intensity (for image classification). Share prices (for stock market prediction ). Passed on to hidden Layer Passed on to output layer Hidden layer: internal representation (interpretation) of data. Layer

Picture below illustrates how data is propagated through the network.  w (xm)n represent weights of connections between network input x m and neuron n in next layer.  y n represents output of neuron n. Input layerHidden layer Output layer

Input layerHidden layer Output layer

Propagation of data through the hidden layer. w mn represent weights of connections between neuron m and neuron n in the next layer. Output layer Hidden layer Input layer

Propagation of data through the output layer. Input layerHidden layer Output layer

Artificial Neural Network  How do ANNs work?  Feed-forward process  About Weights Gradient Descent Back-propagation process  Summaries

About Weights  Weights w settings determine the behaviour of a network  How can we find the right weights ?

Example: Voice Recognition  Task: Learn to discriminate between two different voices saying “Hello”  Data  Sources Steve David  Input data Frequency distribution (60 bins)

 Network architecture  Feed forward network (predefined) 60 input units (one for every frequency bin) 6 hidden units 2 output units (0-1 for “Steve”, 1-0 for “David”)

 Presenting the data Steve David

 Presenting the data (untrained network) Steve David

 Calculate error (suppose the error function is absolute value function) Steve:0-1 |0.43 – 0| = 0.43 |0.57 – 1| = 0.43 David:1-0 |0.7 – 1| = 0.3 |0.3 – 0| = 0.3

 Backprop error and adjust weights (just the last hidden layer) Steve |0.43 – 0| = 0.43 |0.57 – 1| = 0.43 David |0.7 – 1| = 0.3 |0.3 – 0| = 0.3 How do we adjust the weights ? weights

Artificial Neural Network  How do ANNs work?  Feed-forward process  About Weights Gradient Descent Back-propagation process  Summaries

Gradient Descent  Think of (w 0,w 1,…,w n-1 ) as a point in an n-dimensional space.  Suppose the error function is E(w 0,w 1,…,w n-1 ).  Try to minimize error E(w 0,w 1,…,w n-1 ) by changing the point position on the “error surface”.

 How do we change w i ? Change i-th weight by η w i = η*   : direction of going down.   η: length of going down, a constant. w i (new)=w i (old)+ w i

 Repeat the procedure above, we can finally get the minimum. But we need to compute derivative first ! Grad E =[,, …, ]

Artificial Neural Network  How do ANNs work?  Feed-forward process  About Weights Gradient Descent Back-propagation process  Summaries

Back-propagation process  the output of the network y is compared with the desired output z (the target), compute the error, suppose we get error function

 The idea is to propagate error back to all neurons.  The weights' coefficients w mn used to propagate errors back are equal to this used during feed-forward process.  The direction of data flow is changed (signals are propagated from output to inputs one after the other). depends on what function f(e) is. if f(e)=e, then 链式求导 w 56

 If propagated errors came from few neurons they are added. The illustration is below: w 46 链式求导 depends on what function f(e) is. if f(e)=e, then w 24 w 34

 Continue to propagate the error, we can modify the weights for the inputs nodes:

Artificial Neural Network  How do ANNs work?  Feed-forward process  About Weights Gradient Descent Back-propagation process  Summaries

Summaries  1. Initialize network with random weights.  2. For all training cases, repeat:  a. Feed-forward process: present training inputs to network and calculate output.  b.  b. Back-propagation process: for all layers (starting with output layer, back to input layer): Computes the error term for the output units using the observed error. From output layer, repeat - propagating the error term back to the previous layer - updating the weights between the two layers until the earliest hidden layer is reached.

Neural Network and Deep Learning  Artificial Neural Network  Why we go deep?  Amazing achievement  Deep learning-getting started

Why we go deep?  Learning multiple levels of representation

 Learning Non-Linear Features

 Learning features, not just handcrafting them. Most ML systems use very carefully hand-designed features and representations. So, many practitioners are very experienced – and good at such feature design (or kernel design). Hand-crafting features are brittle, incomplete.

 Highly varying functions can be efficiently represented with deep architectures.  Problems which can be represented with a polynomial number of nodes with k layers, may require an exponential number of nodes with k-1 layers.

Neural Network and Deep Learning  Artificial Neural Network  Why we go deep?  Amazing achievement  Deep learning-getting started

Amazing achievement on ImageNet classification  Database: part of ImageNet database, 1000 categories, 1.2 million training images, 150,000 testing images.  Task: classify testing image into one of 1000 categories.  Examples of two categories needed to be differentiated: Differentiate 波斯猫 布偶猫

Machine is as good as human !!! A New Era Begins: Deep Convolutional Neural Network (DCNN)

Neural Network and Deep Learning  Artificial Neural Network  Why we go deep?  Amazing achievement  Deep learning-getting started

Deep learning-getting started  了解一些 deep learning 基本方法的思想  教程 deep learning 大牛 Andrew Ng 所写 ，还 有 实验、 源代 码， 推 荐 细读

Thank You!