# Learning intersections and thresholds of halfspaces Adam Klivans (MIT/Harvard) Ryan O’Donnell (MIT) Rocco Servedio (Harvard)

## Presentation on theme: "Learning intersections and thresholds of halfspaces Adam Klivans (MIT/Harvard) Ryan O’Donnell (MIT) Rocco Servedio (Harvard)"— Presentation transcript:

Learning intersections and thresholds of halfspaces Adam Klivans (MIT/Harvard) Ryan O’Donnell (MIT) Rocco Servedio (Harvard)

Learning We consider the PAC model of [Valiant-84], in which learning a “concept class” C of boolean functions means: - a function f in C is selected, and also a probability distribution D over {+1,−1} n - the learning algorithm gets access to random examples, where the x ’s are drawn from D - goal: efficiently output a hypothesis h such that w.h.p., Pr x← D [ f(x) ≠ h(x)] < ε.

Learning example Example: C is the class of all conjunctions of variables. Perhaps the concept selected is: x 1 AND x 2 AND x 4. One might see examples: What is a learning algorithm for this class?

Halfspaces Let h be a hyperplane in R n : h = {x : ∑w i x i = θ}. h naturally induces a boolean function: f : {+1,−1} n → {+1,−1}, f (x) = sgn(∑w i x i − θ). We call such a function a boolean halfspace, or a weighted majority. The majority function itself is an example ( w i ≡ 1, θ = 0). i=1 n

Learning halfspaces Learning halfspaces is a very old problem; dates back to models for the brain from the ’50s: [Agmon-54, Rosenblatt-58, Block-62]. The concept class of halfspaces has long been known to be PAC learnable in polynomial time via Linear Programming [BEHW-89]. Indeed, this works over any distribution on R n, including those singling out {+1, −1} n.

Learning halfspaces Basic idea: given a bunch of examples, find a halfspace which classifies them correctly. By some learning theory technology (“Occam’s Razor”), this is a good algorithm. Consider the coefficients of a hypothesis halfspace to be unknowns, a 1, …, a n, θ. Each example induces some linear constraints: e.g., induces a 1 +a 2 −a 3 +a 4 −a 5 −a 6 > θ. Solve LP.

Learning intersections of halfspaces The next logical extension of this, and a very important one, is learning intersections of halfspaces. Intersections of halfspaces form a very rich concept class: all convex bodies, CNF formulas… Learning them is also an important problem for computer vision, study of perceptrons. But very little is known.

Prior work - [Baum91]: poly time algorithm for intersection of two halfspaces through the origin under symmetric distributions (those satisfying D (x) = D (−x) ). - [BlumKannan,Vempala97] learn an intersection of O(1) halfspaces in poly time over near-uniform distributions on the Euclidean sphere: - not relevant for boolean halfspaces -[KwekPitt98] gave a polynomial time alg., but requires membership queries -also not relevant for boolean halfspaces

Our results Theorem 1: The concept class of arbitrary functions of k boolean halfspaces over {+1,−1} n is learnable under the uniform distribution to accuracy 1−ε in time: n O(k²/ε²). This is polynomial time if k = O(1), ε = Ω(1). (Prior to this, no algorithm could learn even an intersection of 2 arbitrary boolean halfspaces under the uniform distribution in subexponential time.)

Our results Theorem 2: The concept class of intersections of k boolean halfspaces with weight bound W is learnable under any probability distribution to accuracy 1−ε in time: n O(k log k log W) / ε. So if the weights are polynomially bounded, one can learn an intersection of log many halfspaces in quasipolynomial time.

More results FunctionHalfspacesDistrib.Time any fcn. of k weight W any n O(k² log k logW) /ε weight k threshold (e.g., inters. of k ) weight W any n O(k log k logW) /ε intersection of k weight W any n O(√W log k) /ε read-once intersection of k arbitraryuniform n O((log(k)/ε)²) read-once majority of k arbitraryuniform n Õ((log(k)/ε) ) 4

Sketch of techniques For arbitrary distribution results: show that functions of low weight halfspaces have low degree polynomial threshold representations. For uniform distribution results: show that functions of halfspaces have low noise sensitivity. Both conclusions imply learning results generically.

Talk outline Plan for the rest of the talk: 1.Prove n O(k log k log W) bound for learning intersections of k weight- W halfspaces under arbitrary distributions. (Sketch other arbit. dist. results.) 2.Prove n O(k²/ε²) bound for learning arbitrary functions of k halfspaces under the uniform distribution. (Sketch other unif. dist. results.)

Polynomial threshold functions A (multilinear) polynomial p : R n → R is a PTF for f if it sign-represents f : f(x) = sgn(p(x)) for all x  {+1, −1} n. - every boolean halfspace is a degree 1 PTF for itself - every boolean function has a degree n PTF By linear programming [KS01]: if every function in a class C has a PTF of degree d, then C is learnable in time n O(d) /ε.

PTFs for intersections of halfspaces Suppose f and g are hyperplanes, f(x) = ∑w i x i −θ, g(x) = ∑w i ' x i −θ'. We would like a PTF for sgn(f)  sgn(g). Failed attempt 1: - try f(x)g(x) :is >0 if f(x)>0 and g(x)>0 is >0 if f(x)<0 and g(x)<0  Failed attempt 2: - try f(x)+g(x) : is >0 if f(x)>0 and g(x)>0 is 0 and g(x)<0 

PTFs for intersections of halfspaces The solution: apply a (polynomial?) function to f and g to make them look more like their sign. Assume ∑|w i | < W. Then for all x  {+1,−1} n, f(x), g(x)  [-W,-1] ∪ [1,W]. Beigel et al. [BRS95] showed how to construct a univariate rational function which is an essentially optimal approximator of the sgn function on [-W,-1] ∪ [1,W].

BRS’s sgn -approximator p(x)=(x-1)(x-2) 2 (x-4) 2 (x-8) 2 (x-16) 2 (x-32) 2 q(x) = Q is a rational function of degree O(log k log W) such that: Q(x)  [1, 1+1/k] for x  [1,W], Q(x)  [-1-1/k, -1] for x  [-W,-1]. p(-x)-p(x) p(-x)+p(x)

PTFs for intersections of halfspaces Now given weight W halfspaces h 1, …, h k, sgn(Q(h 1 (x)) + … + Q(h k (x)) − (k−½)) is a rational function which sign-represents the intersection. Once taken to a common denominator, it has degree O(k log k log W). Easy to get a polynomial: sgn(p/q)=sgn(pq). So we have a PTF for the intersection of k weight- W halfspaces of degree O(k log k log W). Hence a learning algorithm running in time n O(k log k log W).

Talk outline Plan for the talk: 1.Prove n O(k log k log W) bound for learning intersections of k weight- W halfspaces under arbitrary distributions.  2.Prove n O(k²/ε²) bound for learning functions of k halfspaces under the uniform distribution. 

Noise sensitivity Let f : {+1,−1} n → {+1,−1} be a boolean function. Pick x  {+1,−1} n uniformly at random, and let y be an ε -corruption of x : flip each bit of x independently with probability ε. defn: The noise sensitivity of f is: NS ε (f) = Pr[f(x) ≠ f(y)].

Noise sensitivity examples Let f be a projection to one bit, f(x 1, …, x n ) = x 1. Then NS ε (f) = ε. Suppose f depends on only k bits. Then NS ε (f) ≤ k ε. PARITY is the most noise-sensitive function: NS ε (PARITY n ) = ½ − ½(1−2ε) n.

Noise sensitivity – study and apps. [Benjamini-Kalai-Schramm-98] – percolation, low-level circuit complexity [Kahn-Kalai-Linial-88] – random walks on the hypercube [Håstad-97] – probabilistically checkable proofs [Bshouty-Jackson-Tamon-99] – learning theory under noise [O-02] – Yao’s XOR Lemma, average case hardness of NP [Bourgain-02, Kindler-Safra-02, FKRSS-02] – study of juntas, Fourier analysis of boolean fcns.

Low noise sens.  fast learning We show that if the noise sensitivity of all f in C is uniformly bounded: NS ε (f) ≤ α(ε), then C is learnable under the uniform distribution in time: n O(1) / α (ε/3). Intuition: if f is not too noise sensitive, nearby points are highly correlated, so a net of examples works. −1

Proof of NS-learning connection Actually, the intuition is wrong. Here is the proper proof sketch: Low noise sensitivity  Fourier spectrum concentrated at low levels; this uses the formula: NS ε (f ) = ½−½ Σ(1−2ε) |S| f(S) 2 and a Markovish inequality. Low level Fourier concentration  efficient uniform distribution learning; this is by the “Low degree” Fourier sampling learning algorithm of [Linial-Mansour-Nisan-93]. ˆ

Noise sensitivity of halfspaces Function NS ε proof one boolean halfspace O(√ε) Y. Peres, ’98 any function of k halfspaces O(k√ε) union bound read-once intersection of k halfspaces O(√ε log k) difficult probabilistic analysis read-once majority of k halfspaces Õ((ε log k) ¼ )

Consequences Let C be the class of functions of k boolean halfspaces. Take α(ε) = O(k√ε), so all f  C have NS ε (f) ≤ α(ε). α −1 (ε/3) = O(ε 2 /k 2 ). Hence we get Theorem 1: a uniform distribution learning algorithm running in time n O(k²/ε²).

Noise sensitivity of a halfspace We now sketch Peres’s beautiful proof that the noise sensitivity of a single halfspace is O(√ε). Suppose the halfspace is f = sgn(∑w i x i −θ). Without (much) loss of generality, one can assume θ = 0. Recall that x i ’s are selected randomly from {+1,−1} and the sum is formed; then each x i is flipped indep. with prob. ε. We want to show that the prob. the sums land on opposite sides of 0 – call this a “flop”, prob. P – is O(√ε).

Noise sensitivity of a halfspace With high probability, the number of flipped bits is about k := εn. Let’s assume we always flip exactly k random bits, and that k divides n. (Both assumptions are easily removed.) We now model the problem thus: Pick signs x i at random. Randomly permute the weights. Divide the weights into n/k blocks of size k. Form the n/k block sums, X 1 = ∑w i x i, X 2 = ∑w i x i, etc. i=1…k i=k+1…2k

Noise sensitivity of a halfspace Write S = X 1 + … + X n/k for the initial sum. Because of the permutation, we may assume that the random signs in the first block are the “flips”. Put S' = S − X 1, so the sum before flipping is S'+X 1, and the sum after flipping is S'−X 1. We are trying to bound the probability P that these two sums have opposite signs (a flop). Note that this happens iff |S'| < |X 1 |.

Noise sensitivity of a halfspace sgn(X 1 ) and S' are independent, so: Pr[sgn(X 1 ) ≠ sgn(S')] = ½. sgn(X 1 ) and |X 1 | are independent, so: Pr[sgn(X 1 ) ≠ sgn(S') | |S'| > |X 1 |] = ½  Pr[sgn(X 1 ) ≠ sgn(S) | |S'| > |X 1 |] = ½  Pr[sgn(X 1 ) ≠ sgn(S) & no flop ] = ½(1−P)  Pr[sgn(X 1 ) ≠ sgn(S)] = ½(1−P)  P = 2 E[½ – I[sgn(X 1 ) ≠ sgn(S)]].

Noise sensitivity of a halfspace Of course, there was nothing special about block X 1 as opposed to any other block. So in fact, P = 2 E[½ – I[sgn(X i ) ≠ sgn(S)]]. for all i = 1…n/k. Write τ=sgn(S), σ i =sgn(X i ), and average: P = 2 E[½ – (k/n) ∑ i I[τ ≠ σ i ]].

Noise sensitivity of a halfspace P = 2 E[½ – (k/n) ∑ i I[τ ≠ σ i ]] The quantity inside the expectation is some random variable, a number which is either ½ – (k/n) ∑ i I[1 ≠ σ i ] or ½ – (k/n) ∑ i I[−1 ≠ σ i ]. If I tell you a number is either a or b, then assuredly it’s at most |a| + |b|. Applying this to the expectation, pointwise: P ≤ 2 E[|½ – (k/n) ∑ i I[σ i =1]| + |½ – (k/n) ∑ i I[σ i =−1]|].

Noise sensitivity of a halfspace P ≤ 2 E[ |½ – ε ∑ I[σ i =1]| + |½ – ε ∑ I[σ i =−1]| ] But the σ i ’s are simply independent, uniformly random signs. Hence both quantities in the expectation are merely the expected absolute deviation from the mean in 1/ε samples of an unbiased 0/1 random variable – i.e., O(√ε). i=1…1/ε

Extensions This concludes the proof that a single halfspace has noise sensitivity O(√ε), from which the uniform distribution learning algorithm for functions of k halfspaces follows. To get the extended learning algorithms, must work harder at analyzing noise sensitivity. Key result: if a halfspace h is biased – say, the probability of + is p < ½, then: NS ε (h ) ≤ min{2p, C p (ε log(1/p)) ½ }.

Talk outline Plan for the talk: 1.Prove n O(k log k log W) bound for learning intersections of k weight- W halfspaces under arbitrary distributions.  2.Prove n O(k²/ε²) bound for learning functions of k halfspaces under the uniform distribution. 

Open technical challenges Give an upper bound on the degree necessary for a PTF which represents the AND of two arbitrary halfspaces. (For a new lower bound, see my talk tomorrow!) Give a better analysis of the noise sensitivity of the intersection of k halfspaces on n bits. Is it O((ε log k) ½ ) ?

The huge open problem It still remains open how to learn an intersection of two arbitrary boolean halfspaces under an arbitrary distribution in subexponential time!

Download ppt "Learning intersections and thresholds of halfspaces Adam Klivans (MIT/Harvard) Ryan O’Donnell (MIT) Rocco Servedio (Harvard)"

Similar presentations