Presentation is loading. Please wait.

Presentation is loading. Please wait.

Jia-Bin Huang Virginia Tech

Similar presentations


Presentation on theme: "Jia-Bin Huang Virginia Tech"โ€” Presentation transcript:

1 Jia-Bin Huang Virginia Tech
Linear Regression Jia-Bin Huang Virginia Tech ECE-5424G / CS-5824 Spring 2019

2 BRACE YOURSELVES WINTER IS COMING

3 BRACE YOURSELVES HOMEWORK IS COMING

4 Administrative Office hour Feedback (Thanks!) Chen Gao Shih-Yang Su
Notation? More descriptive slides? Video/audio recording? TA hours (uniformly spread over the week)?

5 Recap: Machine learning algorithms
Supervised Learning Unsupervised Learning Discrete Classification Clustering Continuous Regression Dimensionality reduction

6 Recap: Nearest neighbor classifier
Training data ๐‘ฅ 1 , ๐‘ฆ 1 , ๐‘ฅ 2 , ๐‘ฆ 2 , โ‹ฏ, ๐‘ฅ ๐‘ , ๐‘ฆ ๐‘ Learning Do nothing. Testing โ„Ž ๐‘ฅ = ๐‘ฆ (๐‘˜) , where ๐‘˜= argmin i ๐ท(๐‘ฅ, ๐‘ฅ (๐‘–) )

7 Recap: Instance/Memory-based Learning
1. A distance metric Continuous? Discrete? PDF? Gene data? Learn the metric? 2. How many nearby neighbors to look at? 1? 3? 5? 15? 3. A weighting function (optional) Closer neighbors matter more 4. How to fit with the local points? Kernel regression Slide credit: Carlos Guestrin

8 Slide credit: CS231 @ Stanford
Validation set Spliting training set: A fake test set to tune hyper-parameters Slide credit: Stanford

9 Slide credit: CS231 @ Stanford
Cross-validation 5-fold cross-validation -> split the training data into 5 equal folds 4 of them for training and 1 for validation Slide credit: Stanford

10 Things to remember Supervised Learning k-NN
Training/testing data; classification/regression; Hypothesis k-NN Simplest learning algorithm With sufficient data, very hard to beat โ€œstrawmanโ€ approach Kernel regression/classification Set k to n (number of data points) and chose kernel width Smoother than k-NN Problems with k-NN Curse of dimensionality Not robust to irrelevant features Slow NN search: must remember (very large) dataset for prediction

11 Todayโ€™s plan: Linear Regression
Model representation Cost function Gradient descent Features and polynomial regression Normal equation

12 Linear Regression Model representation Cost function Gradient descent
Features and polynomial regression Normal equation

13 Training set Learning Algorithm โ„Ž ๐‘ฅ ๐‘ฆ Regression real-valued output
Size of house Hypothesis Estimated price

14 House pricing prediction
Price ($) in 1000โ€™s 500 1000 1500 2000 2500 100 200 300 400 Size in feet^2

15 Slide credit: Andrew Ng
Training set Size in feet^2 (x) Price ($) in 1000โ€™s (y) 2104 460 1416 232 1534 315 852 178 โ€ฆ Notation: ๐‘š = Number of training examples ๐‘ฅ = Input variable / features ๐‘ฆ = Output variable / target variable (๐‘ฅ, ๐‘ฆ) = One training example ( ๐‘ฅ (๐‘–) , ๐‘ฆ (๐‘–) ) = ๐‘– ๐‘กโ„Ž training example ๐‘š = 47 Examples: ๐‘ฅ (1) = 2104 ๐‘ฅ (2) = 1416 ๐‘ฆ (1) = 460 Slide credit: Andrew Ng

16 Slide credit: Andrew Ng
Model representation ๐‘ฆ= โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ Shorthand โ„Ž ๐‘ฅ Training set Learning Algorithm โ„Ž ๐‘ฅ ๐‘ฆ Hypothesis Size of house Estimated price Price ($) in 1000โ€™s 500 1000 1500 2000 2500 100 200 300 400 Size in feet^2 Univariate linear regression Slide credit: Andrew Ng

17 Linear Regression Model representation Cost function Gradient descent
Features and polynomial regression Normal equation

18 Slide credit: Andrew Ng
Training set Size in feet^2 (x) Price ($) in 1000โ€™s (y) 2104 460 1416 232 1534 315 852 178 โ€ฆ Hypothesis โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ ๐œƒ 0 , ๐œƒ 1 : parameters/weights How to choose ๐œƒ ๐‘– โ€™s? ๐‘š = 47 Slide credit: Andrew Ng

19 Slide credit: Andrew Ng
โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ ๐‘ฆ ๐‘ฆ ๐‘ฆ 1 2 3 1 2 3 1 2 3 ๐‘ฅ ๐‘ฅ ๐‘ฅ ๐œƒ 0 =1.5 ๐œƒ 1 =0 ๐œƒ 0 =0 ๐œƒ 1 =0.5 ๐œƒ 0 =1 ๐œƒ 1 =0.5 Slide credit: Andrew Ng

20 Slide credit: Andrew Ng
Cost function minimize 1 2๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– 2 ๐œƒ 0 , ๐œƒ 1 Idea: Choose ๐œƒ 0 , ๐œƒ 1 so that โ„Ž ๐œƒ ๐‘ฅ is close to ๐‘ฆ for our training example (๐‘ฅ,๐‘ฆ) โ„Ž ๐œƒ ๐‘ฅ ๐‘– = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ (๐‘–) ๐ฝ ๐œƒ 0 , ๐œƒ 1 = 1 2๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– 2 Price ($) in 1000โ€™s 500 1000 1500 2000 2500 100 200 300 400 Size in feet^2 ๐‘ฅ ๐‘ฆ minimize ๐ฝ ๐œƒ 0 , ๐œƒ 1 Cost function ๐œƒ 0 , ๐œƒ 1 Slide credit: Andrew Ng

21 Slide credit: Andrew Ng
Simplified Hypothesis: โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ Parameters: ๐œƒ 0 , ๐œƒ 1 Cost function: ๐ฝ ๐œƒ 0 , ๐œƒ 1 = 1 2๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– 2 Goal: minimize ๐ฝ ๐œƒ 0 , ๐œƒ 1 Hypothesis: โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 1 ๐‘ฅ ๐œƒ 0 =0 Parameters: ๐œƒ 1 Cost function: ๐ฝ ๐œƒ 1 = 1 2๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– 2 Goal: minimize ๐ฝ ๐œƒ 1 ๐œƒ 0 , ๐œƒ 1 ๐œƒ 0 , ๐œƒ 1 Slide credit: Andrew Ng

22 Slide credit: Andrew Ng
โ„Ž ๐œƒ ๐‘ฅ , function of ๐‘ฅ ๐ฝ ๐œƒ 1 , function of ๐œƒ 1 1 2 3 ๐‘ฅ ๐‘ฆ 1 2 3 ๐ฝ ๐œƒ 1 ๐œƒ 1 Slide credit: Andrew Ng

23 Slide credit: Andrew Ng
โ„Ž ๐œƒ ๐‘ฅ , function of ๐‘ฅ ๐ฝ ๐œƒ 1 , function of ๐œƒ 1 1 2 3 ๐‘ฅ ๐‘ฆ 1 2 3 ๐ฝ ๐œƒ 1 ๐œƒ 1 Slide credit: Andrew Ng

24 Slide credit: Andrew Ng
โ„Ž ๐œƒ ๐‘ฅ , function of ๐‘ฅ ๐ฝ ๐œƒ 1 , function of ๐œƒ 1 1 2 3 ๐‘ฅ ๐‘ฆ 1 2 3 ๐ฝ ๐œƒ 1 ๐œƒ 1 Slide credit: Andrew Ng

25 Slide credit: Andrew Ng
โ„Ž ๐œƒ ๐‘ฅ , function of ๐‘ฅ ๐ฝ ๐œƒ 1 , function of ๐œƒ 1 1 2 3 ๐‘ฅ ๐‘ฆ 1 2 3 ๐ฝ ๐œƒ 1 ๐œƒ 1 Slide credit: Andrew Ng

26 Slide credit: Andrew Ng
โ„Ž ๐œƒ ๐‘ฅ , function of ๐‘ฅ ๐ฝ ๐œƒ 1 , function of ๐œƒ 1 1 2 3 ๐‘ฅ ๐‘ฆ 1 2 3 ๐ฝ ๐œƒ 1 ๐œƒ 1 Slide credit: Andrew Ng

27 Slide credit: Andrew Ng
Hypothesis: โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ Parameters: ๐œƒ 0 , ๐œƒ 1 Cost function: ๐ฝ ๐œƒ 0 , ๐œƒ 1 = 1 2๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– 2 Goal: minimize ๐ฝ ๐œƒ 0 , ๐œƒ 1 ๐œƒ 0 , ๐œƒ 1 Slide credit: Andrew Ng

28 Slide credit: Andrew Ng
Cost function Slide credit: Andrew Ng

29 How do we find good ๐œƒ 0 , ๐œƒ 1 that minimize ๐ฝ ๐œƒ 0 , ๐œƒ 1 ?
Slide credit: Andrew Ng

30 Linear Regression Model representation Cost function Gradient descent
Features and polynomial regression Normal equation

31 Slide credit: Andrew Ng
Gradient descent Have some function ๐ฝ ๐œƒ 0 , ๐œƒ 1 Want argmin ๐ฝ ๐œƒ 0 , ๐œƒ 1 Outline: Start with some ๐œƒ 0 , ๐œƒ 1 Keep changing ๐œƒ 0 , ๐œƒ 1 to reduce ๐ฝ ๐œƒ 0 , ๐œƒ 1 until we hopefully end up at minimum ๐œƒ 0 , ๐œƒ 1 Slide credit: Andrew Ng

32 Slide credit: Andrew Ng

33 Slide credit: Andrew Ng
Gradient descent Repeat until convergence{ ๐œƒ ๐‘— โ‰” ๐œƒ ๐‘— โˆ’๐›ผ ๐œ• ๐œ• ๐œƒ ๐‘— ๐ฝ ๐œƒ 0 , ๐œƒ 1 (for ๐‘—=0 and ๐‘—=1) } ๐›ผ: Learning rate (step size) ๐œ• ๐œ• ๐œƒ ๐‘— ๐ฝ ๐œƒ 0 , ๐œƒ 1 : derivative (rate of change) Slide credit: Andrew Ng

34 Slide credit: Andrew Ng
Gradient descent Correct: simultaneous update temp0 โ‰”๐œƒ 0 โˆ’๐›ผ ๐œ• ๐œ• ๐œƒ 0 ๐ฝ ๐œƒ 0 , ๐œƒ 1 temp1 โ‰”๐œƒ 1 โˆ’๐›ผ ๐œ• ๐œ• ๐œƒ 1 ๐ฝ ๐œƒ 0 , ๐œƒ 1 ๐œƒ 0 โ‰” temp0 ๐œƒ 1 โ‰” temp1 Incorrect: temp0 โ‰”๐œƒ 0 โˆ’๐›ผ ๐œ• ๐œ• ๐œƒ 0 ๐ฝ ๐œƒ 0 , ๐œƒ 1 ๐œƒ 0 โ‰” temp0 temp1 โ‰”๐œƒ 1 โˆ’๐›ผ ๐œ• ๐œ• ๐œƒ 1 ๐ฝ ๐œƒ 0 , ๐œƒ 1 ๐œƒ 1 โ‰” temp1 Slide credit: Andrew Ng

35 Slide credit: Andrew Ng
๐œƒ 1 โ‰” ๐œƒ 1 โˆ’๐›ผ ๐œ• ๐œ• ๐œƒ 1 ๐ฝ ๐œƒ 1 1 2 3 ๐ฝ ๐œƒ 1 ๐œƒ 1 ๐œ• ๐œ• ๐œƒ 1 ๐ฝ ๐œƒ 1 <0 ๐œ• ๐œ• ๐œƒ 1 ๐ฝ ๐œƒ 1 >0 Slide credit: Andrew Ng

36 Learning rate

37 Gradient descent for linear regression
Repeat until convergence{ ๐œƒ ๐‘— โ‰” ๐œƒ ๐‘— โˆ’๐›ผ ๐œ• ๐œ• ๐œƒ ๐‘— ๐ฝ ๐œƒ 0 , ๐œƒ (for ๐‘—=0 and ๐‘—=1) } Linear regression model โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ ๐ฝ ๐œƒ 0 , ๐œƒ 1 = 1 2๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– 2 Slide credit: Andrew Ng

38 Computing partial derivative
๐œ• ๐œ• ๐œƒ ๐‘— ๐ฝ ๐œƒ 0 , ๐œƒ 1 = ๐œ• ๐œ• ๐œƒ ๐‘— 1 2๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– 2 = ๐œ• ๐œ• ๐œƒ ๐‘— 1 2๐‘š ๐‘–=1 ๐‘š ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ (๐‘–) โˆ’ ๐‘ฆ ๐‘– 2 ๐‘—=0: ๐œ• ๐œ• ๐œƒ 0 ๐ฝ ๐œƒ 0 , ๐œƒ 1 = 1 ๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– ๐‘—=1: ๐œ• ๐œ• ๐œƒ 1 ๐ฝ ๐œƒ 0 , ๐œƒ 1 = 1 ๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– ๐‘ฅ ๐‘– Slide credit: Andrew Ng

39 Gradient descent for linear regression
Repeat until convergence{ ๐œƒ 0 โ‰” ๐œƒ 0 โˆ’๐›ผ 1 ๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– ๐œƒ 1 โ‰” ๐œƒ 1 โˆ’๐›ผ 1 ๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– ๐‘ฅ ๐‘– } Update ๐œƒ 0 and ๐œƒ 1 simultaneously Slide credit: Andrew Ng

40 Batch gradient descent
โ€œBatchโ€: Each step of gradient descent uses all the training examples Repeat until convergence{ ๐œƒ 0 โ‰” ๐œƒ 0 โˆ’๐›ผ 1 ๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– ๐œƒ 1 โ‰” ๐œƒ 1 โˆ’๐›ผ 1 ๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– ๐‘ฅ ๐‘– } ๐‘š: Number of training examples Slide credit: Andrew Ng

41 Linear Regression Model representation Cost function Gradient descent
Features and polynomial regression Normal equation

42 Slide credit: Andrew Ng
Training dataset Size in feet^2 (x) Price ($) in 1000โ€™s (y) 2104 460 1416 232 1534 315 852 178 โ€ฆ โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ Slide credit: Andrew Ng

43 Multiple features (input variables)
Size in feet^2 ( ๐‘ฅ 1 ) Number of bedrooms ( ๐‘ฅ 2 ) Number of floors ( ๐‘ฅ 3 ) Age of home (years) ( ๐‘ฅ 4 ) Price ($) in 1000โ€™s (y) 2104 5 1 45 460 1416 3 2 40 232 1534 30 315 852 36 178 โ€ฆ Notation: ๐‘› = Number of features ๐‘ฅ (๐‘–) = Input features of ๐‘– ๐‘กโ„Ž training example ๐‘ฅ ๐‘— (๐‘–) = Value of feature ๐‘— in ๐‘– ๐‘กโ„Ž training example ๐‘ฅ 3 (2) =? ๐‘ฅ 3 (4) =? Slide credit: Andrew Ng

44 Slide credit: Andrew Ng
Hypothesis Previously: โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ Now: โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ 1 + ๐œƒ 2 ๐‘ฅ 2 +๐œƒ 3 ๐‘ฅ 3 + ๐œƒ 4 ๐‘ฅ 4 Slide credit: Andrew Ng

45 Slide credit: Andrew Ng
โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ 1 + ๐œƒ 2 ๐‘ฅ 2 +โ€ฆ+ ๐œƒ ๐‘› ๐‘ฅ ๐‘› For convenience of notation, define ๐‘ฅ 0 =1 ( ๐‘ฅ 0 (๐‘–) =1 for all examples) ๐’™= ๐‘ฅ 0 ๐‘ฅ 1 ๐‘ฅ 2 โ‹ฎ ๐‘ฅ ๐‘› โˆˆ ๐‘… ๐‘› ๐œฝ= ๐œƒ 0 ๐œƒ 1 ๐œƒ 2 โ‹ฎ ๐œƒ ๐‘› โˆˆ ๐‘… ๐‘›+1 โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ 1 + ๐œƒ 2 ๐‘ฅ 2 +โ€ฆ+ ๐œƒ ๐‘› ๐‘ฅ ๐‘› = ๐œฝ โŠค ๐’™ Slide credit: Andrew Ng

46 Slide credit: Andrew Ng
Gradient descent Previously (๐‘›=1) Repeat until convergence{ ๐œƒ 0 โ‰” ๐œƒ 0 โˆ’๐›ผ 1 ๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– ๐œƒ 1 โ‰” ๐œƒ 1 โˆ’๐›ผ 1 ๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– ๐‘ฅ ๐‘– } New algorithm (๐‘›โ‰ฅ1) Repeat until convergence{ ๐œƒ ๐‘— โ‰” ๐œƒ ๐‘— โˆ’๐›ผ 1 ๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– ๐‘ฅ ๐‘— ๐‘– } Simultaneously update ๐œƒ ๐‘— , for ๐‘—=0, 1, โ‹ฏ,๐‘› Slide credit: Andrew Ng

47 Gradient descent in practice: Feature scaling
Idea: Make sure features are on a similar scale (e.g,. โˆ’1โ‰ค๐‘ฅ ๐‘– โ‰ค1) E.g. ๐‘ฅ 1 = size ( feat^2) ๐‘ฅ 2 = number of bedrooms (1-5) 1 2 3 ๐œƒ 1 ๐œƒ 2 1 2 3 ๐œƒ 1 ๐œƒ 2 Slide credit: Andrew Ng

48 Gradient descent in practice: Learning rate
Automatic convergence test ๐›ผ too small: slow convergence ๐›ผ too large: may not converge To choose ๐›ผ, try 0.001, โ€ฆ 0.01, โ€ฆ, 0.1, โ€ฆ , 1 Image credit:

49 House prices prediction
โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ร—frontage+ ๐œƒ 2 ร—depth Area ๐‘ฅ= frontageร—depth โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ Slide credit: Andrew Ng

50 Polynomial regression
Price ($) in 1000โ€™s 500 1000 1500 2000 2500 100 200 300 400 Size in feet^2 โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ 1 + ๐œƒ 2 ๐‘ฅ 2 + ๐œƒ 3 ๐‘ฅ 3 = ๐œƒ 0 + ๐œƒ 1 ๐‘ ๐‘–๐‘ง๐‘’ + ๐œƒ 2 ๐‘ ๐‘–๐‘ง๐‘’ 2 + ๐œƒ 3 ๐‘ ๐‘–๐‘ง๐‘’ 3 ๐‘ฅ 1 = (size) ๐‘ฅ 2 = (size)^2 ๐‘ฅ 3 = (size)^3 Slide credit: Andrew Ng

51 Linear Regression Model representation Cost function Gradient descent
Features and polynomial regression Normal equation

52 Slide credit: Andrew Ng
( ๐‘ฅ 0 ) Size in feet^2 ( ๐‘ฅ 1 ) Number of bedrooms ( ๐‘ฅ 2 ) Number of floors ( ๐‘ฅ 3 ) Age of home (years) ( ๐‘ฅ 4 ) Price ($) in 1000โ€™s (y) 1 2104 5 45 460 1416 3 2 40 232 1534 30 315 852 36 178 โ€ฆ ๐‘ฆ= ๐œƒ= ( ๐‘‹ โŠค ๐‘‹) โˆ’1 ๐‘‹ โŠค ๐‘ฆ Slide credit: Andrew Ng

53 Least square solution ๐ฝ ๐œƒ = 1 2๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– = 1 2๐‘š ๐‘–=1 ๐‘š ๐œƒ โŠค ๐‘ฅ (๐‘–) โˆ’ ๐‘ฆ ๐‘– = 1 2๐‘š ๐‘‹๐œƒโˆ’๐‘ฆ 2 2 ๐œ• ๐œ•๐œƒ ๐ฝ ๐œƒ =0 ๐œƒ= ( ๐‘‹ โŠค ๐‘‹) โˆ’1 ๐‘‹ โŠค ๐‘ฆ

54 Justification/interpretation 1
Loss minimization ๐ฝ ๐œƒ = 1 2๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– = 1 ๐‘š ๐‘–=1 ๐‘š ๐ฟ ๐‘™๐‘  โ„Ž ๐œƒ ๐‘ฅ ๐‘– , ๐‘ฆ ๐‘– ๐ฟ ๐‘™๐‘  ๐‘ฆ, ๐‘ฆ = ๐‘ฆ โˆ’ ๐‘ฆ : Least squares loss Empirical Risk Minimization (ERM) 1 ๐‘š ๐‘–=1 ๐‘š ๐ฟ ๐‘™๐‘  ๐‘ฆ ๐‘– , ๐‘ฆ

55 Justification/interpretation 2
Probabilistic model Assume linear model with Gaussian errors ๐‘ ๐œƒ ๐‘ฆ ๐‘– ๐‘ฅ ๐‘– = 1 2๐œ‹ ๐œŽ 2 expโก(โˆ’ 1 2 ๐œŽ 2 ( ๐‘ฆ ๐‘– โˆ’ ๐œƒ โŠค ๐‘ฅ ๐‘– ) Solving maximum likelihood argmin ๐œƒ ๐‘–=1 ๐‘š ๐‘ ๐œƒ ๐‘ฆ ๐‘– ๐‘ฅ ๐‘– argmin ๐œƒ logโก( ๐‘–=1 ๐‘š ๐‘ ๐‘ฆ ๐‘– ๐‘ฅ ๐‘– ) = argmin ๐œƒ ๐œŽ 2 ๐‘–=1 ๐‘š ๐œƒ โŠค ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– 2 Image credit: CS

56 Justification/interpretation 3
๐’š ๐‘ฟ๐œฝโˆ’๐’š Geometric interpretation ๐‘ฟ= 1 โ† ๐’™ (1) โ†’ 1 โ‹ฎ 1 โ† ๐’™ (2) โ†’ โ‹ฎ โ† ๐’™ (๐‘š) โ†’ = โ†‘ ๐’› ๐ŸŽ โ†‘ ๐’› ๐Ÿ โ†‘ ๐’› ๐Ÿ โ‹ฏ โ†‘ ๐’› ๐’ โ†“ โ†“ โ†“ โ†“ ๐‘ฟ๐œฝ: column space of ๐‘ฟ or span({ ๐’› ๐ŸŽ , ๐’› ๐Ÿ , โ‹ฏ, ๐’› ๐’ }) Residual ๐‘ฟ๐œฝโˆ’๐ฒ is orthogonal to the column space ofย ๐‘ฟ ๐‘ฟ โŠค ๐‘ฟ๐œฝโˆ’๐ฒ =0โ†’ (๐‘ฟ โŠค ๐‘ฟ)๐œฝ= ๐‘ฟ โŠค ๐’š ๐‘ฟ๐œฝ column space ofย ๐‘ฟ

57 ๐‘š training examples, ๐‘› features
Gradient Descent Need to choose ๐›ผ Need many iterations Works well even when ๐‘› is large Normal Equation No need to choose ๐›ผ Donโ€™t need to iterate Need to compute ( ๐‘‹ โŠค ๐‘‹) โˆ’1 Slow if ๐‘› is very large Slide credit: Andrew Ng

58 Things to remember Model representation
โ„Ž ๐œƒ ๐‘ฅ = ๐œƒ 0 + ๐œƒ 1 ๐‘ฅ 1 + ๐œƒ 2 ๐‘ฅ 2 +โ€ฆ+ ๐œƒ ๐‘› ๐‘ฅ ๐‘› = ๐œƒ โŠค ๐‘ฅ Cost function ๐ฝ ๐œƒ = 1 2๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– 2 Gradient descent for linear regression Repeat until convergence { ๐œƒ ๐‘— โ‰” ๐œƒ ๐‘— โˆ’๐›ผ 1 ๐‘š ๐‘–=1 ๐‘š โ„Ž ๐œƒ ๐‘ฅ ๐‘– โˆ’ ๐‘ฆ ๐‘– ๐‘ฅ ๐‘— ๐‘– } Features and polynomial regression Can combine features; can use different functions to generate features (e.g., polynomial) Normal equation ๐œƒ= ( ๐‘‹ โŠค ๐‘‹) โˆ’1 ๐‘‹ โŠค ๐‘ฆ

59 Next Naรฏve Bayes, Logistic regression, Regularization


Download ppt "Jia-Bin Huang Virginia Tech"

Similar presentations


Ads by Google