1. Deep Feedforward Networks
6 ~ 6.2
Kyunghyun Lee

2. 목차 Introduction Example: Learning XOR Gradient-Based Learning
Cost Functions
Output Units

3. 1. Introduction Deep Feedforward Networks
A feedforward network defines a mapping
Feedforward : information flows x to f(x), there are no feedback connections
Networks : composing together many different functions

4. 1. Introduction Deep Feedforward Networks depth first layer,
input layer
output layer
second layer,
hidden later

5. 1. Introduction Deep Feedforward Networks Hidden layers
training examples specify directly what the output layer must do
behavior of the other layers(hidden layers) is not directly specified by the training data
learning algorithm must decide how to use these layers to best implement an approximation
Each hidden layer of the network is typically vector-valued
dimensionality of these hidden layers determines the width

6. model’s output is always 0.5!
2. Example: Learning XOR
Learning the XOR function
Use linear regression
loss function
linear model
learning result x1 x2 f 1
fitting
model
XOR
model’s output is always 0.5!

7. 2. Example: Learning XOR
Add a different feature space : h space
model

8. 2. Example: Learning XOR Add a different feature space : h space
we must use a nonlinear function to describe the features
activation function : rectified linear unit or ReLU

9. 2. Example: Learning XOR Add a different feature space : h space
solution
let
calculation
activation
function
multiplying by w

10. 2. Example: Learning XOR Conclusion
we simply specified the solution, then showed that it obtained zero error
In a real situation, there might be billions of model parameters and billions of training examples
Instead, a gradient-based optimization algorithm can find parameters that produce very little error

11. 3. Gradient-Based Learning
Neural network vs Linear models
Designing and training a neural network is not much different from training any other machine learning model with gradient descent
The largest difference between the linear models we have seen so far and neural networks is that the nonlinearity of a neural network causes most interesting loss functions to become non- convex

12. 3. Gradient-Based Learning
Non-convex optimization
no such convergence guarantee
sensitive to the values of the initial parameters
For feedforward neural networks
important to initialize all weights to small random values
biases may be initialized to zero or to small positive values

13. 3. Gradient-Based Learning
Gradient descent
train models such as linear regression and support vector machines with gradient descent
must choose a cost function
must choose how to represent the output of the model

14. 3.1 Cost Functions
An important aspect of the design of a deep neural network is the choice of the cost function
Parametric model defines a distribution
maximum likelihood
predict some statistic of y conditioned on x
specialized loss functions

15. 3.1 Cost Functions
Learning Conditional Distributions with Maximum Likelihood
Most modern neural networks are trained using maximum likelihood
cost function is simply the negative log-likelihood
removes the burden of designing cost functions for each model
model
cost function

16. 3.1 Cost Functions Learning Conditional Statistics
Instead of learning a full probability distribution we often want to learn just one conditional statistic
ex) mean of y
two results derived using calculus of variations(19.4.2)
mean squared error :
mean absolute error :

17. 3.2 Output Units
The choice of cost function is tightly coupled with the choice of output unit
suppose that the feedforward network provides a set of hidden features
role of the output layer is then to provide some additional transformation from the features to complete the task

18. 3.2 Output Units Linear Units for Gaussian Output Distributions
Given features h, a layer of linear output units produces a vector
Linear output layers are often used to produce the mean of a conditional Gaussian distribution
Maximizing the log-likelihood is then equivalent to minimizing the mean squared error
may be used with gradient-based optimization algorithms and a wide variety of optimization algorithms (linear units do not saturate)

19. 3.2 Output Units Sigmoid Units for Bernoulli Output Distributions
for predicting the value of a binary variable
ex) classification problems with two classes
Bernoulli distribution
neural net needs to predict only
For this number to be a valid probability, it must lie in the interval [0, 1]
linear unit with threshold
Any time that strayed outside the unit interval, the gradient of the output of the model with respect to its parameters would be 0

20. 3.2 Output Units Sigmoid Units for Bernoulli Output Distributions
sigmoid output units combined with maximum likelihood
sigmoid output unit :
Probability distributions :
assumption : unnormalized log probabilities are linear in y and z

21. 3.2 Output Units Sigmoid Units for Bernoulli Output Distributions
cost function with sigmoid approach
cost function used with maximum likelihood is
loss function :
gradient-based learning is available

22. 3.2 Output Units Softmax Units for Multinoulli Output Distributions
generalization of the Bernoulli distribution
probability distribution over a discrete variable with n possible values
Softmax function
often used as the output of a classifier

23. 3.2 Output Units Softmax Units for Multinoulli Output Distributions
log softmax function
first term : input zi always has a direct contribution to the cost function
second term : roughly approximated by
negative log-likelihood cost function always strongly penalizes the most active incorrect prediction
for correct answer

24. 3.2 Output Units Softmax Units for Multinoulli Output Distributions
softmax saturation
when the differences between input values become extreme
saturates to 1
input is maximal and greater than all of the other inputs
saturates to 0
input is not maximal and the maximum is much greater
if the loss function is not designed to compensate for softmax saturation, it can cause similar difficulties for learning

25. 3.2 Output Units Softmax Units for Multinoulli Output Distributions
softmax argument : z
can be produced in two different ways
using the linear layer
this approach actually overparametrizes the distribution
constraint that the n outputs must sum to 1 means that only n−1 parameters are necessary
impose a requirement that one element of z be fixed
example : sigmoid - softmax with a two-dimensional z

26. 3.2 Output Units Other Output Types
In general, we can think of the neural network as representing a function
outputs of this function are not direct predictions of the value y
provides the parameters for a distribution over y

27. 3.2 Output Units Other Output Types example : variance
learn the variance of a conditional Gaussian for y, given x
negative log-likelihood will provide a cost function
make optimization procedure incrementally learn the variance

28. 3.2 Output Units Other Output Types example : mixture density networks
Neural networks with Gaussian mixtures as their output
The neural network must have three outputs
vector : mixture components
matrix : means
tensor : covariances

29. 3.2 Output Units Other Output Types example : mixture density networks
mixture components
multinoulli distribution over the n different components
can typically be obtained by a softmax over an n-dimensional vector
means
indicate the center associated with the i-th Gaussian component
Learning these means with maximum likelihood is slightly more complicated than learning the means of a distribution with only one output mode

30. 3.2 Output Units Other Output Types example : mixture density networks
covariances
specify the covariance matrix for each component i
maximum likelihood is complicated by needing to assign partial responsibility for each point to each mixture component