Download presentation

1
**Supervised Learning Recap**

Machine Learning

2
Last Time Support Vector Machines Kernel Methods

3
**Today Review of Supervised Learning Unsupervised Learning**

(Soft) K-means clustering Expectation Maximization Spectral Clustering Principle Components Analysis Latent Semantic Analysis

4
**Supervised Learning Linear Regression Logistic Regression**

Graphical Models Hidden Markov Models Neural Networks Support Vector Machines Kernel Methods

5
**Major concepts Gaussian, Multinomial, Bernoulli Distributions**

Joint vs. Conditional Distributions Marginalization Maximum Likelihood Risk Minimization Gradient Descent Feature Extraction, Kernel Methods

6
**Some favorite distributions**

Bernoulli Multinomial Gaussian

7
Maximum Likelihood Identify the parameter values that yield the maximum likelihood of generating the observed data. Take the partial derivative of the likelihood function Set to zero Solve NB: maximum likelihood parameters are the same as maximum log likelihood parameters

8
**Maximum Log Likelihood**

Why do we like the log function? It turns products (difficult to differentiate) and turns them into sums (easy to differentiate) log(xy) = log(x) + log(y) log(xc) = c log(x)

9
**Risk Minimization Pick a loss function**

Squared loss Linear loss Perceptron (classification) loss Identify the parameters that minimize the loss function. Take the partial derivative of the loss function Set to zero Solve

10
**Frequentists v. Bayesians**

Point estimates vs. Posteriors Risk Minimization vs. Maximum Likelihood L2-Regularization Frequentists: Add a constraint on the size of the weight vector Bayesians: Introduce a zero-mean prior on the weight vector Result is the same!

11
**L2-Regularization Frequentists: Bayesians:**

Introduce a cost on the size of the weights Bayesians: Introduce a prior on the weights

12
**Types of Classifiers Generative Models Discriminative Models**

Highest resource requirements. Need to approximate the joint probability Discriminative Models Moderate resource requirements. Typically fewer parameters to approximate than generative models Discriminant Functions Can be trained probabilistically, but the output does not include confidence information

13
Linear Regression Fit a line to a set of points

14
**Linear Regression Extension to higher dimensions Polynomial fitting**

Arbitrary function fitting Wavelets Radial basis functions Classifier output

15
**Logistic Regression Fit gaussians to data for each class**

The decision boundary is where the PDFs cross No “closed form” solution to the gradient. Gradient Descent

16
Graphical Models General way to describe the dependence relationships between variables. Junction Tree Algorithm allows us to efficiently calculate marginals over any variable.

17
**Junction Tree Algorithm**

Moralization “Marry the parents” Make undirected Triangulation Remove cycles >4 Junction Tree Construction Identify separators such that the running intersection property holds Introduction of Evidence Pass slices around the junction tree to generate marginals

18
**Hidden Markov Models Sequential Modeling**

Generative Model Relationship between observations and state (class) sequences

19
**Perceptron Step function used for squashing.**

Classifier as Neuron metaphor.

20
**Perceptron Loss Classification Error vs. Sigmoid Error**

Loss is only calculated on Mistakes Perceptrons use strictly classification error

21
Neural Networks Interconnected Layers of Perceptrons or Logistic Regression “neurons”

22
Neural Networks There are many possible configurations of neural networks Vary the number of layers Size of layers

23
**Support Vector Machines**

Maximum Margin Classification Small Margin Large Margin

24
**Support Vector Machines**

Optimization Function Decision Function

25
**Visualization of Support Vectors**

26
Questions? Now would be a good time to ask questions about Supervised Techniques.

27
**Clustering Identify discrete groups of similar data points**

Data points are unlabeled

28
**Recall K-Means Algorithm Select K – the desired number of clusters**

Initialize K cluster centroids For each point in the data set, assign it to the cluster with the closest centroid Update the centroid based on the points assigned to each cluster If any data point has changed clusters, repeat

29
k-means output

30
Soft K-means In k-means, we force every data point to exist in exactly one cluster. This constraint can be relaxed. Minimizes the entropy of cluster assignment

31
Soft k-means example

32
Soft k-means We still define a cluster by a centroid, but we calculate the centroid as the weighted mean of all the data points Convergence is based on a stopping threshold rather than changed assignments

33
**Gaussian Mixture Models**

Rather than identifying clusters by “nearest” centroids Fit a Set of k Gaussians to the data. p(x) = \pi_0f_0(x) + \pi_1f_1(x) + \pi_2f_2(x) + \ldots + \pi_kf_k(x)

34
GMM example

35
**Gaussian Mixture Models**

Formally a Mixture Model is the weighted sum of a number of pdfs where the weights are determined by a distribution,

36
**Graphical Models with unobserved variables**

What if you have variables in a Graphical model that are never observed? Latent Variables Training latent variable models is an unsupervised learning application uncomfortable amused sweating laughing

37
Latent Variable HMMs We can cluster sequences using an HMM with unobserved state variables We will train the latent variable models using Expectation Maximization

38
**Expectation Maximization**

Both the training of GMMs and Gaussian Models with latent variables are accomplished using Expectation Maximization Step 1: Expectation (E-step) Evaluate the “responsibilities” of each cluster with the current parameters Step 2: Maximization (M-step) Re-estimate parameters using the existing “responsibilities” Related to k-means

39
Questions One more time for questions on supervised learning…

40
Next Time Gaussian Mixture Models (GMMs) Expectation Maximization

Similar presentations

© 2020 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google