1. CSCI B609: “Foundations of Data Science”
Lecture 13/14: Gradient Descent, Boosting and Learning from Experts
Slides at
Grigory Yaroslavtsev

2. Constrained Convex Optimization
Non-convex optimization is NP-hard:
𝑖 𝑥 𝑖 − 𝑥 𝑖 2 =0 ⇔∀𝑖: 𝑥 𝑖 ∈ 0,1
Knapsack:
Minimize 𝑖 𝑐 𝑖 𝑥 𝑖
Subject to: 𝑖 𝑤 𝑖 𝑥 𝑖 ≤𝑊
Convex optimization can often be solved by ellipsoid algorithm in 𝑝𝑜𝑙𝑦(𝑛) time, but too slow

3. Convex multivariate functions
Convexity:
∀𝑥,𝑦∈ ℝ 𝑛 : 𝑓 𝑥 ≥𝑓 𝑦 + 𝑥 −𝑦 𝛻f(y)
∀𝑥,𝑦∈ ℝ 𝑛 , 0≤𝜆≤1:
𝑓 𝜆𝑥+ 1−𝜆 𝑦 ≤𝜆𝑓 𝑥 + 1−𝜆 𝑓(𝑦)
If higher derivatives exist:
𝑓 𝑥 =𝑓 𝑦 +𝛻𝑓 𝑦 ⋅ 𝑥−𝑦 + 𝑥−𝑦 𝑇 𝛻 2 𝑓 𝑥 𝑥−𝑦 +…
𝛻 2 𝑓 𝑥 𝑖𝑗 = 𝜕 2 𝑓 𝜕 𝑥 𝑖 𝜕 𝑥 𝑗 is the Hessian matrix
𝑓 is convex iff it’s Hessian is positive semidefinite, 𝑦 𝑇 𝛻 2 𝑓𝑦≥0 for all 𝑦.

4. Examples of convex functions
ℓ 𝑝 -norm is convex for 1≤𝑝≤∞:
𝜆𝑥+ 1−𝜆 𝑦 𝑝 ≤ 𝜆𝑥 𝑝 −𝜆 𝑦 𝑝 =𝜆 𝑥 𝑝 + 1 −𝜆 𝑦 𝑝
𝑓 𝑥 = log ( 𝑒 𝑥 1 + 𝑒 𝑥 2 +⋯ 𝑒 𝑥 𝑛 )
max 𝑥 1 ,…, 𝑥 𝑛 ≤𝑓 𝑥 ≤ max 𝑥 1 ,…, 𝑥 𝑛 + log 𝑛
𝑓 𝑥 = 𝑥 𝑇 𝐴𝑥 where 𝐴 is a p.s.d. matrix, 𝛻 2 𝑓=𝐴
Examples of constrained convex optimization:
(Linear equations with p.s.d. constraints):
minimize: 𝑥 𝑇 𝐴𝑥− 𝑏 𝑇 𝑥 (solution satisfies 𝐴𝑥=𝑏)
(Least squares regression):
Minimize: 𝐴𝑥−𝑏 = x T A T A x −2 Ax T b+ b T b

5. Constrained Convex Optimization
General formulation for convex 𝑓 and a convex set 𝐾:
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒: 𝑓 𝑥 subject to: 𝑥∈𝐾
Example (SVMs):
Data: 𝑋 1 ,…, 𝑋 𝑁 ∈ ℝ 𝑛 labeled by 𝑦 1 ,…, 𝑦 𝑁 ∈ −1,1 (spam / non-spam)
Find a linear model:
𝑊⋅ 𝑋 𝑖 ≥1⇒ 𝑋 𝑖 is spam
𝑊⋅ 𝑋 𝑖 ≤−1⇒ 𝑋 𝑖 is non-spam
∀𝑖:1 − 𝑦 𝑖 𝑊 𝑋 𝑖 ≤0
More robust version:
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒: 𝑖 𝐿𝑜𝑠𝑠 1−𝑊 𝑦 𝑖 𝑋 𝑖 +𝜆 𝑊 2
E.g. hinge loss Loss(t)=max(0,t)
Another regularizer: 𝜆 𝑊 1 (favors sparse solutions)

6. Gradient Descent for Constrained Convex Optimization
(Projection): 𝑥∉𝐾→𝑦∈𝐾
y= argmin z∈𝐾 𝑧−𝑥 2
Easy to compute for ⋅ : 𝑦=𝑥/ 𝑥 2 2
Let 𝛻𝑓 𝑥 2 ≤𝐺, max 𝑥,𝑦∈𝐾 𝑥−𝑦 2 ≤𝐷.
Let 𝑇= 4 𝐷 2 𝐺 2 𝜖 2
Gradient descent (gradient + projection oracles):
Let 𝜂=𝐷/𝐺 𝑇
Repeat for 𝑖=0, …, 𝑇:
𝑦 (𝑖+1) = 𝑥 (𝑖) −𝜂𝛻𝑓 𝑥 𝑖
𝑥 (𝑖+1) = projection of 𝑦 (𝑖+1) on 𝐾
Output 𝑧= 1 𝑇 𝑖 𝑥 (𝑖)

7. Gradient Descent for Constrained Convex Optimization
𝑥 𝑖+1 − 𝑥 ∗ ≤ 𝑦 𝑖+1 − 𝑥 ∗
= 𝑥 𝑖 − 𝑥 ∗ −𝜂𝛻𝑓 𝑥 𝑖
= 𝑥 𝑖 − 𝑥 ∗ 𝜂 2 𝛻𝑓 𝑥 𝑖 −2𝜂𝛻𝑓 𝑥 𝑖 ⋅ 𝑥 𝑖 − 𝑥 ∗
Using definition of 𝐺:
𝛻𝑓 𝑥 𝑖 ⋅ 𝑥 𝑖 − 𝑥 ∗ ≤ 1 2𝜂 𝑥 𝑖 − 𝑥 ∗ − 𝑥 𝑖+1 − 𝑥 ∗ 𝜂 2 𝐺 2
𝑓 𝑥 𝑖 −𝑓 𝑥 ∗ ≤ 1 2𝜂 𝑥 𝑖 − 𝑥 ∗ − 𝑥 𝑖+1 − 𝑥 ∗ 𝜂 2 𝐺 2
Sum over 𝑖=1,…, 𝑇:
𝑖=1 𝑇 𝑓 𝑥 𝑖 −𝑓 𝑥 ∗ ≤ 1 2𝜂 𝑥 0 − 𝑥 ∗ − 𝑥 𝑇 − 𝑥 ∗ 𝑇𝜂 2 𝐺 2

8. Gradient Descent for Constrained Convex Optimization
𝑖=1 𝑇 𝑓 𝑥 𝑖 −𝑓 𝑥 ∗ ≤ 1 2𝜂 𝑥 0 − 𝑥 ∗ − 𝑥 𝑇 − 𝑥 ∗ 𝑥 0 − 𝑥 ∗ − 𝑥 𝑇 − 𝑥 ∗ 𝑇𝜂 2 𝐺 2
𝑓 1 𝑇 𝑖 𝑥 𝑖 ≤ 1 𝑇 𝑖 𝑓 𝑥 𝑖 :
𝑓 1 𝑇 𝑖 𝑥 𝑖 −𝑓 𝑥 ∗ ≤ 𝐷 2 2𝜂𝑇 + 𝜂 2 𝐺 2
Set 𝜂= 𝐷 𝐺 𝑇 ⇒ RHS ≤ 𝐷𝐺 𝑇 ≤𝜖

9. Online Gradient Descent
Gradient descent works in a more general case:
𝑓→ sequence of convex functions 𝑓 1 , 𝑓 2 …, 𝑓 𝑇
At step 𝑖 need to output 𝑥 (𝑖) ∈𝐾
Let 𝑥 ∗ be the minimizer of 𝑖 𝑓 𝑖 (𝑤)
Minimize regret:
𝑖 𝑓 𝑖 𝑥 𝑖 − 𝑓 𝑖 ( 𝑥 ∗ )
Same analysis as before works in online case.

10. Stochastic Gradient Descent
(Expected gradient oracle): returns 𝑔 such that 𝔼 𝑔 𝑔 =𝛻𝑓 𝑥 .
Example: for SVM pick randomly one term from the loss function.
Let 𝑔 𝑖 be the gradient returned at step i
Let 𝑓 𝑖 = 𝑔 𝑖 𝑇 𝑥 be the function used in the i-th step of OGD
Let 𝑧= 1 𝑇 𝑖 𝑥 (𝑖) and 𝑥 ∗ be the minimizer of 𝑓.

11. Stochastic Gradient Descent
Thm. 𝔼 𝑓 𝑧 ≤𝑓 𝑥 ∗ + 𝐷𝐺 𝑇 where 𝐺 is an upper bound of any gradient output by oracle.
𝑓 𝑧 −𝑓 𝑥 ∗ ≤ 1 𝑇 𝑖 (𝑓 𝑥 𝑖 −𝑓( 𝑥 ∗ )) (convexity)
≤ 1 𝑇 𝑖 𝛻𝑓 𝑥 𝑖 𝑥 𝑖 − 𝑥 ∗
= 1 𝑇 𝑖 𝔼 [ 𝑔 𝑖 𝑇 ( 𝑥 𝑖 − 𝑥 ∗ )] (grad. oracle)
= 1 𝑇 𝑖 𝔼[ 𝑓 𝑖 ( 𝑥 𝑖 )− 𝑓 𝑖 (𝑥 ∗ )]
= 1 𝑇 𝔼[ 𝑖 𝑓 𝑖 ( 𝑥 𝑖 )− 𝑓 𝑖 (𝑥 ∗ )]
𝔼[] = regret of OGD , always ≤𝜖

12. VC-dim of combinations of concepts
For 𝒌 concepts ℎ 1 ,…, ℎ 𝒌 + a Boolean function 𝑓:
𝑐𝑜𝑚 𝑏 𝑓 ℎ 1 ,… ℎ 𝒌 ={𝑥∈𝑋:𝑓 ℎ 1 𝑥 ,… ℎ 𝒌 𝑥 =1}
Ex: 𝐻 = lin. separators, 𝑓=𝐴𝑁𝐷 / 𝑓=𝑀𝑎𝑗𝑜𝑟𝑖𝑡𝑦
For a concept class 𝐻 + a Boolean function 𝑓:
𝐶𝑂𝑀 𝐵 𝑓,𝒌 𝐻 ={𝑐𝑜𝑚 𝑏 𝑓 ℎ 1 ,… ℎ 𝒌 : ℎ 𝑖 ∈𝐻}
Lem. If 𝑉𝐶-dim(𝐻)=𝒅 then for any 𝑓:
𝑉𝐶-dim 𝐶𝑂𝑀 𝐵 𝑓,𝒌 𝐻 ≤𝑂(𝒌𝒅 log(𝒌𝒅))

13. VC-dim of combinations of concepts
Lem. If 𝑉𝐶-dim(𝐻)=𝑑 then for any 𝑓:
𝑉𝐶-dim 𝐶𝑂𝑀 𝐵 𝑓,𝒌 𝐻 ≤𝑂(𝒌𝒅 log(𝒌𝒅))
Let 𝑛=𝑉𝐶−dim 𝐶𝑂𝑀 𝐵 𝑓,𝒌 𝐻
⇒∃ set 𝑆 of 𝑛 points shattered by 𝐶𝑂𝑀 𝐵 𝑓,𝒌 𝐻
Sauer’s lemma ⇒ ≤ 𝑛 𝑑 ways of labeling 𝑆 by 𝐻
Each labeling in 𝐶𝑂𝑀 𝐵 𝑓,𝒌 𝐻 determined by 𝒌 labelings of 𝑆 by 𝐻 ⇒ ≤ (𝑛 𝒅 ) 𝒌 = 𝑛 𝒌𝒅 labelings
2 𝑛 ≤ 𝑛 𝒌𝒅 ⇒𝑛≤𝒌𝒅 log 𝑛 ⇒𝑛≤2𝒌𝒅 log 𝒌𝒅

14. Back to the batch setting
Classification problem
Instance space 𝑋: 0,1 𝒅 or ℝ 𝒅 (feature vectors)
Classification: come up with a mapping 𝑋→{0,1}
Formalization:
Assume there is a probability distribution 𝐷 over 𝑋
𝒄 ∗ = “target concept” (set 𝒄 ∗ ⊆𝑋 of positive instances)
Given labeled i.i.d. samples from 𝐷 produce 𝒉⊆𝑋
Goal: have 𝒉 agree with 𝒄 ∗ over distribution 𝐷
Minimize: 𝑒𝑟 𝑟 𝐷 𝒉 = Pr 𝐷 [𝒉 Δ 𝒄 ∗ ]
𝑒𝑟 𝑟 𝐷 𝒉 = “true” or “generalization” error

15. Boosting Strong learner: succeeds with prob. ≥1 −𝜖
Weak learner: succeeds with prob. ≥ 1 2 +𝛾
Boosting (informal): weak learner that works under any distribution ⇒ strong learner
Idea: run weak leaner 𝑨 on sample 𝑆 under reweightings focusing on misclassified examples

16. VC-dim ≤𝑂( 𝒕 𝟎 VC−dim 𝐻 log ( 𝒕 𝟎 VC−dim 𝐻 ))
Boosting (cont.)
𝐻 = class of hypothesis produced by 𝑨
Apply majority rule to ℎ 1 ,…, ℎ 𝒕 𝟎 ∼𝐻:
VC-dim ≤𝑂( 𝒕 𝟎 VC−dim 𝐻 log ( 𝒕 𝟎 VC−dim 𝐻 ))
Algorithm:
Given 𝑆=( 𝑥 1 ,…, 𝑥 𝑛 ) set 𝑤 𝑖 =1 in 𝒘=( 𝑤 1 ,…, 𝑤 𝑛 )
For 𝑡=1,…, 𝒕 𝟎 do:
Call weak learner on 𝑆,𝒘 ⇒ hypothesis ℎ 𝑡
For misclassified 𝑥 𝑖 multiply 𝑤 𝑖 by 𝛼=( 𝛾)/( 1 2 −𝛾)
Output: MAJ( ℎ 1 , …, ℎ 𝒕 𝟎 )

17. Boosting: analysis
Def (𝜸-weak learner on sample): For labeled examples 𝑥 𝑖 weighted by 𝑤 𝑖 with weight of correct
≥ 𝛾 𝑖=1 𝑛 𝑤 𝑖
Thm. If 𝐴 is 𝛾-weak learner on 𝑆 ⇒
for 𝒕 𝟎 =𝑂 1 𝛾 2 𝑙𝑜𝑔 𝑛 boosting achieves 0 error on 𝑆.
Proof. m = # mistakes of the final classifier
Each was misclassified ≥ 𝒕 𝟎 2 times ⇒ weight ≥ 𝛼 𝒕 𝟎 /2
Total weight ≥𝑚 𝛼 𝒕 𝟎 /2
Total weight at 𝑡=W t
𝑊 𝑡+1 ≤ 𝛼 1 2 −𝛾 𝛾 𝑊 𝑡 = 1+2𝛾 𝑊(𝑡)

18. Boosting: analysis (cont.)
𝑊 0 =𝑛⇒𝑊 𝒕 𝟎 ≤𝑛 1+2𝛾 𝒕 𝟎
𝑚 𝛼 𝒕 𝟎 /2 ≤𝑊 𝒕 𝟎 ≤𝑛 1+2𝛾 𝒕 𝟎
𝛼=( 𝛾)/( 1 2 −𝛾) =(1+2𝛾)/(1−2𝛾)
𝑚≤𝑛 1−2𝛾 𝒕 𝟎 / 𝛾 𝒕 𝟎 /2 = 𝑛 1−4 𝛾 2 𝒕 𝟎 /2
1−𝑥≤ 𝑒 −𝑥 ⇒ 𝑚≤𝑛 𝑒 −2 𝛾 2 𝒕 𝟎 ⇒ 𝒕 𝟎 =𝑂 1 𝛾 2 log 𝑛
Comments:
Applies even if the weak learners are adversarial
VC-dim bounds ⇒𝑛= 𝑂 1 𝜖 𝑉𝐶−dim⁡(𝐻) 𝛾 2