Download presentation

Presentation is loading. Please wait.

Published byJustin Hayes Modified over 3 years ago

1
Neural Networks and Kernel Methods

2
How are we doing on the pass sequence? We can now track both men, provided with –Hand-labeled coordinates of both men in 30 frames –Hand-extracted features (stripe detector, white blob detector) –Hand-labeled classes for the white-shirt tracker We have a framework for how to optimally make decisions and track the men Generally, this will take a lot longer than 24 hours… We need to avoid doing this by hand!

3
Recall: Multi-input linear regression y(x,w) = w 0 + w 1 1 (x) + w 2 2 (x) + … + w M M (x) x can be an entire scan-line or image! We could try to uniformly distribute basis functions in the input space: This is futile, because of the curse of dimensionality x = entire scan line …

4
Neural networks and kernel methods Two main approaches to avoiding the curse of dimensionality: –Neural networks Parameterize the basis functions and learn their locations Can be nested to create a hierarchy Regularize the parameters or use Bayesian learning –Kernel methods The basis functions are associated with data points, limiting complexity A subset of data points may be selected to further limit complexity

5
Neural networks and kernel methods Two main approaches to avoiding the curse of dimensionality: –Neural networks Parameterize the basis functions and learn their locations Can be nested to create a hierarchy Regularize the parameters or use Bayesian learning –Kernel methods The basis functions are associated with data points, limiting complexity A subset of data points may be selected to further limit complexity

6
Two-layer neural networks Before, we used Replace each j with a variable z j, where and h() is a fixed activation function The outputs are obtained from where () is another fixed function In all, we have (simplifying biases):

7
Typical activation functions Logistic sigmoid, aka logit: h(a) = (a) = 1/(1+e -a ) Hyperbolic tangent: h(a) = tanh(a) = (e a -e -a )/(e a +e -a ) Cumulative Gaussian (error function): h(a) = 2 x - a N (x|0,1)dx - 1 –This one has a lighter tail h(a)h(a) Normalized to have same range and slope at a=0 a As above, but h is on a log-scale

8
Examples of functions learned by a neural network (3 tanh hidden units, one linear output unit)

9
Multi-layer neural networks Only weights corresponding to the feed- forward topology are instantiated The sum is over those values of j with instantiated weights w kj From now on, well denote all activation functions by h

10
Learning neural networks As for regression, we consider a squared error cost function: E(w) = ½ n k ( t nk – y k (x n,w) ) 2 which corresponds to a Gaussian density p(t|x) We can substitute and use a general purpose optimizer to estimate w, but it is illustrative and useful to study the derivatives of E …

11
Learning neural networks E(w) = ½ n k ( t nk – y k (x n,w) ) 2 Recall that for linear regression: E(w)/ w m = - n ( t n - y n ) x nm Well use the chain rule of differentiation to derive a similar-looking expression, where –Local input signals are forward-propagated from the input –Local error signals are back-propagated from the output Error signalInput signal Weight in-between error signal and input signal

12
Local signals needed for learning For clarity, consider the error for one training case: To compute E n / w ji, note that w ji appears in only one term of the overall expression, namely Using the chain rule of differentiation, we have where if w ji is in the 1st layer, z i is actually input x i Weight Local error signal Local input signal

13
Forward-propagating local input signals Forward propagation gives all the a s and z s

14
Back-propagating local error signals Back-propagation gives all the s t2t2 t1t1

15
Back-propagating error signals To compute E n / a j ( j ), note that a j appears in all those expressions a k = i w ki h(a i ) that depend on a j Using the chain rule, we have The sum is over k s.t. unit j is connected to unit k and for each such term, a k / a j = w kj h (a j ) Noting that E n / a k = k, we get the back-propagation rule: For output units: -

16
Putting the propagations together For each training case n, apply forward propagation and back-propagation to compute for each weight w ji Sum these over training cases to compute Use these derivatives for steepest descent learning or as input to a conjugate gradients optimizer, etc On-line learning: After each pattern presentation, use the above gradient to update the weights

17
The number of hidden units determines the complexity of the learned function ( M = # hidden units)

18
The effect of local minima Because of random weight initialization, each training run will find a different solution M Validation error

19
Regularizing neural networks Demonstration of over-fitting ( M = # hidden units)

20
Regularizing neural networks Use cross-validation to select the network architecture (number of layers, number of units per layer) Add to E a term ( / 2 ) ji w ji 2 that penalizes large weights, so Use cross-validation to select Use early-stopping and cross-validation (next slide) Take a Bayesian approach: Put a prior on the w s and integrate over them to make predictions Over-fitting:

21
Early stopping The weights start at small values and grow Perhaps the number of learning iterations is a surrogate for model complexity? This works for some learning tasks Number of learning iterations Training error Validation error

22
Can we use a standard neural network to automatically learn the features needed for tracking? x is 320-dimensional, so the number of parameters would be at least 320 We have only 15 data points (setting aside 15 for cross validation) so over-fitting will be an issue We could try weight decay, Bayesian learning, etc, but a little thinking reveals that our approach is wrong… In fact, we want the weights connecting different positions in the scan line to use the same feature (eg, stripes) x = entire scan line

23
Convolutional neural networks Recall that a short portion of the scan line was sufficient for tracking the striped shirt We can use this idea to build a convolutional network Same set of weights used for all hidden units With constrained weights, the number of free parameters is now only ~ one dozen, so… We can use Bayesian/regularized learning to automatically learn the features

24
Convolutional neural networks in 2-D (from Le Cun et al, 1989)

25
Neural networks and kernel methods Two main approaches to avoiding the curse of dimensionality: –Neural networks Parameterize the basis functions and learn their locations Can be nested to create a hierarchy Regularize the parameters or use Bayesian learning –Kernel methods The basis functions are associated with data points, limiting complexity A subset of data points may be selected to further limit complexity

26
Kernel methods Basis functions offer a way to enrich the feature space, making simple methods (such as linear regression and linear classifiers) much more powerful Example: Input x ; Features x, x 2, x 3, sin(x), … There are two problems with this approach –Computational efficiency: Generally, the appropriate features are not known, so there is a huge (possibly infinite) number of them to search over –Regularization: Even if we could search over the huge number of features, how can we select appropriate features so as to prevent overfitting? The kernel framework enables efficient approaches to both problems

27
Kernel methods x1x1 x2x2 1 2

28
Definition of a kernel Suppose (x) is a mapping from the D -dimensional input vector x to a high (possibly infinite) dimensional feature space Many simple methods rely on inner products of feature vectors, (x 1 ) T (x 2 ) For certain feature spaces, the kernel trick can be used to compute (x 1 ) T (x 2 ) using the input vectors directly: (x 1 ) T (x 2 ) = k(x 1, x 2 ) k(x 1, x 2 ) is referred to as a kernel If a function satisfies Mercers conditions (see textbook), it can be used as a kernel

29
Examples of kernels k(x 1, x 2 ) = x 1 T x 2 ( is symmetric positive definite) k(x 1, x 2 ) = exp(-||x 1 -x 2 || 2 k(x 1, x 2 ) = exp(-½ x 1 T x 2 ( is symmetric positive definite) k(x 1, x 2 ) = p(x 1 )p(x 2 )

30
Gaussian processes Recall that for linear regression: Using a design matrix, our prediction vector is Lets use a simple prior on w : Then K is called the Gram matrix, where Result: The correlation between two predictions equals the kernel evaluated for the corresponding inputs Example

31
Gaussian processes: Learning and prediction As before, we assume The target vector likelihood is Using, we can obtain the marginal predictive distribution over targets: where Predictions are based on where, = is Gaussian with

32
Example: Samples from

33
Example: Learning and prediction

34
Sparse kernel methods and SVMs Idea: Identify a small number of training cases, called support vectors, which are used to make predictions See textbook for details Support vector

35
Questions?

36
How are we doing on the pass sequence? We can now automatically learn the features needed to track both people Same set of weights used for all hidden units

37
How are we doing on the pass sequence? Pretty good! We can now automatically learn the features needed to track both people But, it sucks that we need to hand-label the coordinates of both men in 30 frames and hand- label the 2 classes for the white-shirt tracker Same set of weights used for all hidden units

38

39
Lecture 5 Appendix

40
Constructing kernels Provided with a kernel or a set of kernels, we can construct new kernels using any of the rules:

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google