1. Introduction to Machine Learning 236756
Prof. Nir Ailon
Lecture 4:
Computational Complexity of Learning
&
Surrogate Losses

2. Efficient PAC Learning
Until now we were mostly worried about sample complexity
How many examples do we need in order to Probably Approximately Correctly learn from a specific concept class?
This is a CS course, not a stats course. We can’t ignore the computational price!
How much computational effort is needed for PAC learning?

3. Definition of Efficient PAC Learning
Of course we want “polynomial”. But in what?
1st Attempt:
In the number of training examples 𝑚
2nd Attempt:
In 1 𝛿 , 1 𝜀
The number of necessary examples 𝑚 is a function of the complexity of ℋ, and parameters 𝛿,𝜀 (which we really care about). If 𝑚 is large, we can throw away the excess.
Learning:
procedure one-step-ERM 𝑆= 𝑥 1 , 𝑦 1 ,…, 𝑥 𝑚 , 𝑦 𝑚
return ℎ 𝑆
Predicting:
ℎ 𝑆 𝑥 := argmin ℎ′∈ℋ 𝐿 𝑆 (ℎ′) (𝑥)
Must output an efficient prediction rule

4. …Polynomial In What?
We will define complexity of a learning algorithm 𝒜 with respect to 𝛿,𝜀 and another parameter 𝑛 which is related to the ``size’’ of ℋ,𝒳
Parameter 𝑛 can be embedding dimension. Example: If we decide to use 𝑛 features to describe objects, how will that increase runtime?
The standard way to do this is by defining a sequence of pairs ( 𝒳 𝑛 , ℋ 𝑛 ) 𝑛=1 ∞ , and studying asymptotic complexity of learning 𝒳 𝑛 , ℋ 𝑛 as 𝑛 grows
Important to remember: 𝒜 does not get distribution as part of input

5. Formal Definition: Efficient PAC Learning [Valiant 1984]
A sequence ( 𝒳 𝑛 , ℋ 𝑛 ) 𝑛=1 ∞ is efficient PAC learnable if ∃ algorithm 𝒜(𝑆,𝜀,𝛿) and a polynomial 𝑝(𝑛, 1 𝜀 , 1 𝛿 ) s.t.
For all 𝑛, 𝒟 𝒳 𝑛 ,𝒴 s.t. ∃ℎ: 𝐿 𝒟 ℎ =0 and ∀𝜀,𝛿: 𝒜 receives as input 𝑆∼ 𝒟 𝑚 with 𝑚≤𝑝 𝑛, 1 𝜀 , 1 𝛿 , runs in time at most 𝑝 𝑛, 1 𝜀 , 1 𝛿 , and outputs a predictor ℎ: 𝒳 𝑛 ↦𝒴 that can be evaluated in time 𝑝 𝑛, 1 𝜀 , 1 𝛿 and w.p. ≥1−𝛿: 𝐿 𝒟 ℎ ≤𝜀
Over sample and/or algorithm randomization

6. Efficient PAC Learning Using CONSISTENT
Reminder: CONSISTENT ℋ 𝑆=( 𝑥 1 , 𝑦 1 ,…,( 𝑥 𝑚 , 𝑦 𝑚 )
If ∃ℎ∈ℋ s.t. ℎ 𝑥 𝑖 = 𝑦 𝑖 for all 𝑖, output such ℎ
Otherwise, output “doesn’t exist”.
If VCdim(ℋ)≤𝐷 can learn (in realizable case) using CONSISTENT on 𝑂 𝐷+log 1 𝛿 /𝜀 samples
Conclusion: If for sequence ( 𝒳 𝑛 , ℋ 𝑛 ) 𝑛=1 ∞ we have (1) VCdim ℋ 𝑛 ≤poly(𝑛) and (2) CONSISTENT polytime computable (in sample size) then sequence is efficient PAC learnable.

7. Example: If 𝑛 encodes | ℋ 𝑛 |
Assume ∀𝑛 |ℋ 𝑛 |=𝑛
∀ 𝑥,ℎ ∈( 𝒳 𝑛 , ℋ 𝑛 ), ℎ(𝑥) computable in polynomial time 𝑞(𝑛)
CONSISTENT computable in time 𝑚𝑛⋅𝑞(𝑛)
Problem efficient PAC learnable in 𝑂 𝑞 𝑛 ×𝑛× 𝜀 −1 log 𝑛 𝛿 time
More generally
If VCDim grows polynomially with 𝑛
And CONSISTENT computable in polynomial time
 Problem efficient PAC learnable

8. Exponential Size (or Infinite) Classes
Axis-aligned rectangles in 𝑛 dimensions?
Halfspaces in 𝑛 dimensions?
Boolean functions on 𝑛 variables

9. Axis-Aligned Rectangles in 𝑛 Dimensions
𝑛=2
VCDim ℋ 𝑛 =𝑂(𝑛)
CONSISTENT solvable in time 𝑂 𝑛𝑚

10. Halfspaces in 𝑛 Dimensions
𝑛=2
Reminder: ℎ 𝑤 𝑥 = sign 𝑤,𝑥
VCDim ℋ 𝑛 =𝑂(𝑛)
CONSISTENT solvable in time 𝑂 poly 𝑛,𝑚

11. Boolean Conjunctions 𝒳 𝑛 = 0,1 𝑛 𝒴= 0,1
Note: ℋ = 3 𝑛 +1
𝒳 𝑛 = 0,1 𝑛 𝒴= 0,1
ℋ 𝑛 = ℎ 𝑖 1 .. 𝑖 𝑘 , 𝑗 1 .. 𝑗 𝑟 𝑖 1 ,.., 𝑖 𝑘 , 𝑗 1 ,.., 𝑗 𝑟 ∈ 𝑛 } ℎ 𝑖 1 .. 𝑖 𝑘 , 𝑗 1 .. 𝑗 𝑟 𝑥 1 ,…, 𝑥 𝑛 = 𝑥 𝑖 1 ∧⋯∧ 𝑥 𝑖 𝑘 ∧ ¬𝑥 𝑗 1 ∧⋯∧¬ 𝑥 𝑗 𝑟
Can we solve CONSISTENT efficiently?
Yes!
Start with ℎ=ℎ 1,..𝑛,1,..𝑛
Scan samples in any order
Ignore samples 𝑥,0
Given sample 𝑥,1 : Fix ℎ by removing violating literals
What is the running time?
Literals

12. 3-term DNFs
Note: |ℋ|= 3 3𝑛
𝒳 𝑛 = 0,1 𝑛 𝒴= 0,1
ℋ 𝑛 = ℎ 𝐴 1 , 𝐴 2 , 𝐴 𝐴 1 , 𝐴 2 , 𝐴 3 conjunctions} ℎ 𝐴 1 , 𝐴 2 , 𝐴 3 𝑥 = 𝐴 1 𝑥 ∨ 𝐴 2 𝑥 ∨ 𝐴 3 𝑥
Can we still solve CONSISTENT efficiently?
Probably not! It’s NP-Hard

13. Exponential Size (or Infinite) Classes
Axis-aligned rectangles in 𝑛 dimensions
Halfspaces in 𝑛 dimensions
Conjunctions on 𝑛 variables
3-term DNF’s
Python programs of size at most 𝑛
Python programs that run in poly(n) time
Decision trees of size at most 𝑛
Circuits of size at most 𝑛
Even circuits of depth at most log 𝑛
CONSISTENT:
Poly-time
What does this
imply?
CONSISTENT:
NP-Hard
What does THIS
imply???

14. Implication Of “CONSISTENT is Hard”
CONSISTENT computes ℎ∈ ℋ 𝑛 consistent with sample
Efficient PAC learnability allows outputting any function as prediction (as long as it is efficiently computable)
“CONSISTENT is hard”  “learning is hard”
“CONSISTENT is hard”  “proper learning is hard”
Def: A proper learning algorithm is a learning algorithm that must output ℎ∈ ℋ 𝑛
We already saw improper learning… When?
The halving algorithm!
Is there a problem that is not efficient PAC properly learnable, but still efficient PAC (improperly) learnable?

15. Improper Learning of 3-term DNFs
Distribution rule: 𝑎∧𝑏 ∨ 𝑐∧𝑑 = 𝑎∨𝑐 ∧ 𝑎∨𝑑 ∧ 𝑏∨𝑐 ∧(𝑏∨𝑑)
𝐴 1 ∨ 𝐴 2 ∨ 𝐴 3 = 𝑢∈ 𝐴 1 ,𝑣∈ 𝐴 2 ,𝑤∈ 𝐴 3 (𝑢∨𝑣∨𝑤)
Can view as conjunction over 2𝑛 3 variables
At most 3 ((2𝑛) 3 ) +1 possible conjunctions
Can solve using CONSISTENT for conjunctions in dimension 2𝑛 3 .
Can efficient PAC learn 3-term DNFs
Pay polynomially in sample complexity, gain exponentially in computational complexity
At most 2𝑛 3 possibilities

16. CONSISTENT Computational-Complexity
Improper Learning
3-term DNF
Conjunctions over 𝟎,𝟏 𝟐𝒏 𝟑
Sample complexity
𝑂 𝑛+𝑙𝑜𝑔 1 𝛿 𝜀
𝑂 𝑛 3 +𝑙𝑜𝑔 1 𝛿 𝜀
CONSISTENT Computational-Complexity
NP-Hard
𝑂 𝑛 3 × 𝑛 3 +𝑙𝑜𝑔 1 𝛿 𝜀
Conjunctions over 0,1 2𝑛 3

17. (Smaller concept class, hard CONSISTENT)
Improper Learning
Conjunctions over 0,1 2𝑛 3
(Larger concept class,
easy CONSISTENT)
3-term DNF over 0,1 𝑛
(Smaller concept class, hard CONSISTENT)

18. So How Do We Prove Hardness of Learning?
Hardness of learning is reminiscent of cryptography
The business of cryptography is preventing from an adversary to uncover a secret (key) even using partial observations
Much of cryptography is based on existence of trapdoor one-way functions 𝑓: 0,1 𝑛 ↦{0,1 } 𝑛 s.t.
(1) Easy to compute 𝑓
(2) Hard to compute 𝑓 −1 , even only with high probability
[(3) Easy to compute 𝑓 −1 given trapdoor 𝑠 𝑓 ]
Given family ℱ 𝑛 of trapdoor one-way functions 𝑓, each with a distinct trapdoor 𝑠 𝑓 ∈ 0,1 𝑝𝑜𝑙𝑦 𝑛
Define ℋ 𝑛 = ℎ=𝑓 −1 :𝑓∈ ℱ 𝑛
Efficient PAC learning ℋ 𝑛 would violate (2) because thanks to (1) we could simulate a sample 𝑥 1 =𝑓( 𝑦 1 ), 𝑦 1 ,…,( 𝑥 𝑚 =𝑓 𝑦 𝑚 , 𝑦 𝑚 )

19. Breaking a Crypto System Given Efficient PAC Learnability
break-crypto-system 𝑥∈ 0,1 𝑛 , 𝑓∈ ℱ 𝑛 draw 𝑦 1 ,…, 𝑦 𝑚 ∈ 0,1 𝑛 randomly // 𝑚=𝑝𝑜𝑙𝑦(𝑛) compute 𝑥 1 =𝑓 𝑦 1 ,…, 𝑥 𝑚 =𝑓( 𝑦 𝑚 ) // easy send 𝑥 1 , 𝑦 1 ,…,( 𝑥 𝑚 , 𝑦 𝑚 ) to efficient PAC learner, obtain ℎ // efficiently computable return ℎ(𝑥)
By efficient PAC learnability, succeeds for most 𝑥’s with high probability

20. The Cubic Root Problem
Fix integer 𝑁 of 𝑛≈ log 𝑁 bits s.t. 𝑁=𝑝𝑞 for two primes 𝑝,𝑞
Let 𝑓 𝑥 = 𝑥 3 𝑚𝑜𝑑 𝑁
Under mild number-theoretic assumption, 𝑓 −1 well defined
Given 𝑝,𝑞, easy to compute 𝑓 −1
Using a python program of polynomial size, running in 𝑝𝑜𝑙𝑦 𝑛 time
Using a circuit of O(𝑙𝑜𝑔 𝑛 ) depth
Without 𝑝,𝑞, believed to be hard to compute 𝑓 −1
 No efficient PAC learning of
Short python programs
Efficient python programs
Logarithmic depth circuits

21. The Agnostic Case
If you thought realizable was hard agnostic is even harder
ERM for halfspaces is NP-Hard.
Improperly learning halfspaces is crypto-Hard.
Everything “interesting” we know of is hard in the agnostic case (except maybe for intervals…)
So what to do?

22. What is the Source of Hardness?
Is it the fact that the concept classes are large?
No – We saw that over squared losses (over the reals) we can efficiently optimize linear classifiers
Working with discrete valued losses is what makes a hard problem hard
ℓ 𝑠𝑞𝑟 (−1,ℎ 𝑥 )
ℓ 0−1 (−1,ℎ 𝑥 ) 1 1 -1
ℎ 𝑥 ∈ℝ
ℎ 𝑥 ∈ℝ
Why is this easier?
It’s continuous
It’s convex
All of the above

23. Convexity
Definition (convex set): A set 𝐶 in a vector space is convex if ∀𝑢,𝑣∈𝐶 and for all 𝛼∈ 0,1 : 𝛼𝑢+ 1−𝛼 𝑣∈𝐶

24. Convexity
Definition (convex function): A function 𝑓:𝐶↦ ℝ for convex domain 𝐶 is convex if ∀𝑢,𝑣∈𝐶,𝛼∈ 0,1 : 𝑓 𝛼𝑢+ 1−𝛼 𝑣 ≤𝛼𝑓 𝑢 + 1−𝛼 𝑓(𝑣)
𝑓(𝑣)
𝛼𝑓(𝑢)+ 1−𝛼 𝑓(𝑣)
𝑓(𝑢)
𝑓(𝛼𝑢+ 1−𝛼 𝑣) 𝑢 𝑣 𝐶
𝛼𝑢+ 1−𝛼 𝑣

25. Minimizing Convex Functions
Any local minimum is also a global minimum
See proof in book!
We can greedily search for increasingly better solutions in small neighborhoods.
When we’re stuck, we’re done!

26. Surrogate Loss Functions
Instead of minimizing ℓ 0−1 minimize something that is
Convex (for any fixed 𝑦)
An upper bound of ℓ 0−1
For example:
ℓ 𝑠𝑞𝑟 𝑦,ℎ 𝑥 = 𝑦−ℎ 𝑥 2
ℓ 𝑙𝑜𝑔𝑖𝑠𝑡𝑖𝑐 𝑦,ℎ 𝑥 =log⁡(1+ exp −𝑦ℎ 𝑥 )
ℓ ℎ𝑖𝑛𝑔𝑒 (𝑦,ℎ(𝑥))=max{0,1−𝑦⋅ℎ(𝑥)}
Easier to minimize
If surrogate small, so is ℓ 0−1
The bigger the violation, the more you pay
ℓ 0−1
ℓ ℎ𝑖𝑛𝑔𝑒 1 1
Hinge loss for linear predictors minimizable using LP (in poly time)
ℎ 𝑥 ∈ℝ
ℎ 𝑥 ∈ℝ
𝑦=1
𝑦=−1
No payment if sign correct by margin

27. Summary: Learning Using Convex Surrogates
If your problem is hard to learn w.r.t. ℓ:𝒴×𝒴↦ ℝ
Predict using ℎ:𝒳↦ 𝒴
Define surrogate ℓ 𝑠𝑢𝑟 :𝒴× 𝒴 ↦ ℝ
𝒴 convex set
ℓ 𝑠𝑢𝑟 (𝑦, 𝑦 ) convex function in 𝑦 for all 𝑦
ℓ≤ ℓ 𝑠𝑢𝑟
Can now naturally define 𝐿 𝒟 ℎ = E 𝑥∼𝒟 ℓ 𝑠𝑢𝑟 (𝑦,ℎ 𝑥 ) ≤ 𝐿 𝒟 (ℎ)
𝐿 𝑆 ℎ = 1 𝑚 𝑖=1 𝑚 ℓ 𝑠𝑢𝑟 ( 𝑦 𝑖 ,ℎ 𝑥 𝑖 ) ≤ 𝐿 𝑆 (ℎ)
𝑠𝑢𝑟
𝑠𝑢𝑟

28. Sample Bounds for Probably Approximately Optimizing over Surrogates
𝑠𝑢𝑟
Can we approximately, probably minimize 𝐿 𝒟 by minimizing 𝐿 𝑆 over some class of functions {ℎ:𝒳↦ 𝒴 } ?
If 𝒴 = −𝑀,𝑀 (bounded functions), then Hoeffding bound can be used (as in binary case) to say that, for any ℎ: Pr 𝑆∼ 𝒟 𝑚 𝐿 𝒟 (ℎ)− 𝐿 𝑆 (ℎ) >𝑡 ≤ 2𝑒 − 𝑡 2 2 𝑀 2 𝑚
Union bound won’t work here (class of functions typically uncountable)
VC-Subgraph is a method for extending VC theory to real valued functions
𝑠𝑢𝑟
𝑠𝑢𝑟
𝑠𝑢𝑟