# Regression “A new perspective on freedom” TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAA A A A A AAA A A.

## Presentation on theme: "Regression “A new perspective on freedom” TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAA A A A A AAA A A."— Presentation transcript:

Regression “A new perspective on freedom” TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAA A A A A AAA A A

Classification

? CatDog

Cleanliness Size

? \$\$\$\$\$\$\$\$\$\$

Regression

\$ \$\$ \$\$\$ \$\$\$\$ Price Top speed x y

Regression Data Goal: given, predict i.e. find a prediction function

Nearest neighbor -50510152025 -10 -5 0 5 10 15

Nearest neighbor To predict x –Find the data point x i closest to x –Choose y = y i + No training – Finding closest point can be expensive – Overfitting

Kernel Regression To predict X –Give data point x i weight –Normalize weights –Let e.g.

Kernel Regression -50510152025 -10 -5 0 5 10 15 [matlab demo]

Kernel Regression + No training + Smooth prediction – Slower than nearest neighbor – Must choose width of

Linear regression

0 10 20 30 40 0 10 20 30 20 22 24 26 Temperature [start Matlab demo lecture2.m] Given examples Predict given a new point 0 10 20 30 40 0 10 20 30 20 22 24 26 Temperature

0 10 20 30 40 0 10 20 30 20 22 24 26 Temperature Linear regression Prediction

Linear Regression Error or “residual” Prediction Observation Sum squared error

Linear Regression n d Solve the system (it’s better not to invert the matrix)

Minimize the sum squared error Sum squared error Linear equation Linear system

LMS Algorithm (Least Mean Squares) where Online algorithm

Beyond lines and planes everything is the same with still linear in 01020 0 40

Linear Regression [summary] n d Let For example Let Minimize by solving Given examples Predict

Probabilistic interpretation Likelihood

Overfitting 02468101214161820 -15 -10 -5 0 5 10 15 20 25 30 [Matlab demo] Degree 15 polynomial

Ridge Regression (Regularization) 02468101214161820 -10 -5 0 5 10 15 Effect of regularization (degree 19) with “small” Minimize Solve Let

Probabilistic interpretation Likelihood Prior Posterior

Locally Linear Regression

[source: http://www.cru.uea.ac.uk/cru/data/temperature] 1840186018801900192019401960198020002020 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 Global temperature increase

Locally Linear Regression To predict X –Give data point x i weight –Let e.g.

Locally Linear Regression + Good even at the boundary (more important in high dimension) – Solve linear system for each new prediction – Must choose width of To minimize Solve Predict where

[source: http://www.cru.uea.ac.uk/cru/data/temperature] Locally Linear Regression Gaussian kernel 180

[source: http://www.cru.uea.ac.uk/cru/data/temperature] Locally Linear Regression Laplacian kernel 180

L1 Regression

Sensitivity to outliers High weight given to outliers Influence function

L 1 Regression Linear program Influence function

Spline Regression Regression on each interval 5200540056005800 50 60 70

Spline Regression With equality constraints 5200540056005800 50 60 70

Spline Regression With L 1 cost 5200540056005800 50 60 70

To learn more The Elements of Statistical Learning, Hastie, Tibshirani, Friedman, Springer

Download ppt "Regression “A new perspective on freedom” TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAA A A A A AAA A A."

Similar presentations