1. 1 Computational Complexity (Ctnd.) ORF 523 Lecture 15 Instructor: Amir Ali Ahmadi TA: G. Hall Spring 2015

2. 2 Open questions in complexity for optimization

3. 3

4. 4

5. 5

6. 6 Convexity This talk: -- Given a multivariate polynomial, can we efficiently decide if it is convex? -- Given a basic semialgebraic set, can we efficiently decide if it is a convex set? “In fact the great watershed in optimization isn't between linearity and nonlinearity, but convexity and nonconvexity.” Rockafellar, ’93: But how easy is it to distinguish between convexity and nonconvexity?

7. 7 Data separation with convex sublevel sets Convex envelope approximation Convex fitting Search over Convex Polynomials: Applications Other Applications: - Convex Lyapunov functions in robust control - Norms defined by polynomials

8. 8 Complexity of deciding convexity  Input to the problem: an ordered list of the coefficients (all rational)  Degree d odd: trivial  d=1, always convex  d>1 and odd, never convex  d=2, i.e., p(x)=x T Qx+q T x+c : poly time  check if Q is PSD  d=4, first interesting case  Question of N. Z. Shor

9. 9 What we are going to prove: Thm: Deciding convexity of quartic polynomials is (strongly) NP-hard. This is true even if the polynomials are restricted to be homogeneous (i.e., forms).

10. 10 Sequence of reductions STABLE SET Nonconvex QP Nonnegativity of biquadratic forms Convexity of quartic forms CLIQUE Any problem in NP Circuit SAT SAT 3SAT Cook-Levin Theorem Saw in class Flip edges Motzkin- Strauss Thm. Next (straightforward, see [DPV])

11. Any problem in NP  Circuit SAT (review) 11  The Cook-Levin theorem.  In a way a very deep theorem.  At the same time almost a tautology.  An example of a problem in NP: STABLE SET

12. 12 Sequence of reductions STABLE SET Nonconvex QP Nonnegativity of biquadratic forms Convexity of quartic forms CLIQUE Any problem in NP Circuit SAT SAT 3SAT Cook-Levin Theorem Saw in class Flip edges Motzkin- Strauss Thm. Next (straightforward, see [DPV])

13. 13 NP-hardness of deciding convexity of quartics  Reduction from problem of deciding “nonnegativity of biquadratic forms” Thm: Deciding convexity of quartic forms is NP-hard.  Biquadratic form:  These are degree-4 polynomials of very special structure.  So our previous proof that testing nonnegativity of quartic (homogeneous) polynomials is NP-hard does not suffice…

14. 14 CLIQUE  Biquadratic Nonnegativity

15. 15 Why this problem? And why aren’t we done?  Biquadratic form:  Can write any biquadratic form as where is a matrix whose entries are quadratic forms  Example: with  This is exactly how the Hessian of a quartic form looks like!  So why aren’t we done?

16. 16 Why aren’t we done? The Hessian has very special structure!  above is not a valid Hessian:

17. 17 From biquadratic forms to biquadratic Hessian forms  We give a constructive procedure to go from any biquadratic form to a biquadratic Hessian form by doubling the number of variables, such that:  In fact, we construct the polynomial f(x,y) that has H(x,y) as its Hessian directly  Let’s see this construction…

18. 18 The main reduction Thm: Given any biquadratic form Let Then

19. 19 Observation on the Hessian of a biquadratic form

20. 20 Proof of correctness of the reduction Start with Let Claim:

21. 21 Reduction on an instance A 6x6 Hessian with quadratic form entries

22. 22 Questions for you  Why does the result imply testing convexity is NP-hard also for degree-6 polynomials? (and degree-8, etc.)  (Hint: add an independent variable)  Why does the result imply testing convexity of basic semialgebraic sets (i.e., sets defined by polynomial inequalities) is NP-hard?  (Hint: epigraphs)

23. 23 Other notions of interest in optimization  Strong convexity  “Hessian uniformly bounded away from zero”  Appears e.g. in convergence analysis of Newton-type methods  Strict convexity  “curve strictly below the line”  Guarantees uniqueness of optimal solution  Convexity  Pseudoconvexity  “Relaxation of first order characterization of convexity”  Any point where gradient vanishes is a global min  Quasiconvexity  “Convexity of sublevel sets”  Deciding convexity of basic semialgebraic sets    

24. 24 Summary of complexity results

25. 25 Quasiconvexity A multivariate polynomial p(x)=p(x 1,…,x n ) is quasiconvex if all its sublevel sets are convex. Convexity  Quasiconvexity (converse fails)  Deciding quasiconvexity of polynomials of even degree 4 or larger is (strongly) NP-hard  Quasiconvexity of odd degree polynomials can be decided in polynomial time

26. 26 Quasiconvexity of even degree forms Proof outline:  A homogeneous quasiconvex polynomial is nonnegative (why?)  The unit sublevel sets of p(x) and p(x) 1/d are the same convex set  p(x) 1/d is the Minkowski (semi)-norm defined by this convex set and hence a convex function  A convex nonnegative function raised to a power d larger than one remains convex (why?) Corollaries: -- Deciding quasiconvexity is NP-hard -- Deciding convexity of basic semialgebraic sets is NP-hard Lemma: A homogeneous polynomial p(x) of even degree d is quasiconvex if and only if it is convex.

27. 27 Proof:  Show super level sets must also be convex sets  Only convex set whose complement is also convex is a halfspace Quasiconvexity of odd degree polynomials This representation can be checked in polynomial time Thm: The sublevel sets of a quasiconvex polynomial p(x) of odd degree are halfspaces. Thm: A polynomial p(x) of odd degree d is quasiconvex iff it can be written as

28. As we’ve seen several times by now… 28  The boundary between the tractable and the intractable is very delicate!  Testing quasiconvexity of odd vs. even degree polynomials  Mincut vs. Maxcut  Testing matrix positive semidefiniteness versus matrix copositivity  Robust stability of interval polynomials versus robust stability of interval matrices  Testing primality versus finding the factors (?)  Many many others…

29. Let’s recall what we are achieving by an NP-hardness proof 29 [Garey, Johnson]

30. NP-complete problems are everywhere… 30  Today we have thousands of NP-complete problems. In all areas of science and engineering.

31. The domino effect 31  All NP-complete problems reduce to each other!  If you solve one in polynomial time, you solve ALL in polynomial time!

32. The $1M question! 32 Most people believe the answer is NO! Philosophical reason: If a proof of the Goldbach conjecture were to fly from the sky, we could certainly efficiently verify it. But should this imply that we can find this proof efficiently? P=NP would imply the answer is yes.

33. Nevertheless, there are believers too… 33 Over 100 wrong proofs have appeared so far (in both directions)! See http://www.win.tue.nl/~gwoegi/P-versus-NP.htm

34. Coping with NP-completeness 34  Solving special cases exactly  Approximation algorithms – suboptimal solutions with worst-case guarantees  At least certifying gap to optimality on an instance-by-instance basis  Can easily be done by finding bounds using convex optimization (think e.g., SDP for STABLE SET)  Challenge whether worst-case analysis is the right notion  But propose rigorous alternatives  Average-case analysis, provable behavior on random instances, etc.  Heuristics  OK if you are truly solving a real-life problem  Easy for practitioners to become overly optimistic about their performance  On what fraction of the input instances do you think a poly-time heuristic should fail?  Just a few contrived examples that theoreticians can construct?  Constant fraction?  Linearly many?  Polynomially many?  Results in complexity theory from the 80s and 90s: even the last answer would imply P=NP

35. Main messages… 35  Computational complexity theory beautifully classifies many problems of optimization theory as easy or hard  At the most basic level, when faced with a new optimization problem, you should try to prove membership to the class P or establish NP-hardness.  After that, you can still study the complexity of your problem in further detail (e.g., quality/hardness of approximation)  Complexity theory is important to the theory of convex optimization  Arguably the most rigorous tool we have for proving that certain problems do not have an efficient convex reformulation  The boundary between tractable and intractable problems is very delicate:  MINCUT vs. MAXCUT, PSD-ness vs. copositivity, LP vs. IP,...  P=NP?  Maybe one of you guys will tell us one day.

36. References 36 -NP-hardness of testing convexity: http://web.mit.edu/~a_a_a/Public/Publications/convexity_nphard.pdf http://web.mit.edu/~a_a_a/Public/Publications/convexity_nphard.pdf -NP-hardness of testing nonnegativity of biquadratic forms: http://www.math.ucsd.edu/~njw/PUBLICPAPERS/LNQY_july02.pdf http://www.math.ucsd.edu/~njw/PUBLICPAPERS/LNQY_july02.pdf -Survey on performance of heuristic algorithms: http://arxiv.org/abs/1210.8099v1 http://arxiv.org/abs/1210.8099v1 - [DPV08] S. Dasgupta, C. Papadimitriou, and U. Vazirani. Algorithms. McGraw Hill, 2008. -[GJ79] D.S. Johnson and M. Garey. Computers and Intractability: a guide to the theory of NP-completeness, 1979.