1. Convex Programming Brookes Vision Reading Group

2. Huh? What is convex ??? What is programming ??? What is convex programming ???

3. Huh? What is convex ??? What is programming ??? What is convex programming ???

4. Convex Function f(t x + (1-t) y) <= t f(x) + (1-t) f(y)

5. Convex Function Is a linear function convex ???

6. Convex Set Region above a convex function is a convex set.

7. Convex Set Is the set of all positive semidefinite matrices convex??

8. Huh? What is convex ??? What is programming ??? What is convex programming ???

9. Programming Objective function to be minimized/maximized. Constraints to be satisfied. Example Objective function Constraints

10. Example Feasible region Vertices Objective function Optimal solution

11. Huh? What is convex ??? What is programming ??? What is convex programming ???

12. Convex Programming Convex optimization function Convex feasible region Why is it so important ??? Global optimum can be found in polynomial time. Many practical problems are convex Non-convex problems can be relaxed to convex ones.

13. Convex Programming Convex optimization function Convex feasible region Examples ??? Linear Programming Refer to Vladimir/Pushmeets reading group Second Order Cone Programming What ??? Semidefinite Programming All this sounds Greek and Latin !!!!

14. Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP 2 out of 3 is not bad !!!

15. Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

16. Second Order Cone || u || < t u - vector of dimension d-1 t - scalar Cone lies in d dimensions Second Order Cone defines a convex set Example: Second Order Cone in 3D x 2 + y 2 <= z 2

17. Hmmm ICE CREAM !!

18. Second Order Cone Programming Minimize f T x Subject to || A i x+ b i || <= c i T x + d i i = 1, …, L Linear Objective Function Affine mapping of SOC Constraints are SOC of n i dimensions Feasible regions are intersections of conic regions

19. Example

20. Why SOCP ?? A more general convex problem than LP – LP SOCP Fast algorithms for finding global optimum – LP - O(n 3 ) – SOCP - O(L 1/2 ) iterations of O(n 2 n i ) Many standard problems are SOCP-able

21. SOCP-able Problems Convex quadratically constrained quadratic programming Sum of norms Maximum of norms Problems with hyperbolic constraints

22. SOCP-able Problems Convex quadratically constrained quadratic programming Sum of norms Maximum of norms Problems with hyperbolic constraints

23. QCQP Minimize x T P 0 x + 2 q 0 T x + r 0 Subject to x T P i x + 2 q i T x + r i P i >= 0 || P 0 1/2 x + P 0 -1/2 x || 2 + r 0 -q 0 T P 0 -1 p 0

24. QCQP Minimize x T P 0 x + 2 q 0 T x + r 0 Subject to x T P i x + 2 q i T x + r i Minimize t Subject to || P 0 1/2 x + P 0 -1/2 x || < = t || P 0 1/2 x + P 0 -1/2 x || < = (r 0 -q 0 T P 0 -1 p 0 ) 1/2

25. SOCP-able Problems Convex quadratically constrained quadratic programming Sum of norms Maximum of norms Problems with hyperbolic constraints

26. Sum of Norms Minimize || F i x + g i || Minimize t i Subject to || F i x + g i || <= t i Special Case: L-1 norm minimization

27. SOCP-able Problems Convex quadratically constrained quadratic programming Sum of norms Maximum of norms Problems with hyperbolic constraints

28. Maximum of Norms Minimize max || F i x + g i || Minimize t Subject to || F i x + g i || <= t Special Case: L-inf norm minimization

29. You werent expecting a question, were you ??

30. SOCP-able Problems Convex quadratically constrained quadratic programming Sum of norms Maximum of norms Problems with hyperbolic constraints

31. Hyperbolic Constraints w 2 <= xy x >= 0, y >= 0 || [2w; x-y] || <= x+y

32. Lets see if everyone was awake !

33. Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

34. Semidefinite Programming Minimize C X Subject to A i X = b i X >= 0 Linear Objective Function Linear Constraints Linear Programming on Semidefinite Matrices

35. Why SDP ?? A more general convex problem than SOCP – LP SOCP SDP Generality comes at a cost though – SOCP - O(L 1/2 ) iterations of O(n 2 n i ) – SDP - O((n i ) 1/2 ) iterations of O(n 2 n i 2 ) Many standard problems are SDP-able

36. SDP-able Problems Minimizing the maximum eigenvalue Class separation with ellipsoids

37. SDP-able Problems Minimizing the maximum eigenvalue Class separation with ellipsoids

38. Minimizing the Maximum Eigenvalue Matrix M(z) To find vector z* such that max is minimized. Let max (M(z)) <= n max (M(z)-nI) <= 0 min (nI - M(z)) >= 0 nI - M(z) >= 0

39. Minimizing the Maximum Eigenvalue Matrix M(z) To find vector z* such that max is minimized. Max -n nI - M(z) >= 0

40. SDP-able Problems Minimizing the maximum eigenvalue Class separation with ellipsoids

43. Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

44. Non-Convex Problems Minimize x T Q 0 x + 2q 0 T x + r 0 Subject to x T Q i x + 2q i T x + r i < = 0 Q i >= 0 => Convex Redefine x in homogenous coordinates. y = (1; x) Non-Convex Quadratic Programming Problem !!!

45. Non-Convex Problems Minimize x T Q 0 x + 2q 0 T x + r 0 Subject to x T Q i x + 2q i T x + r i < = 0 Minimize y T M 0 y Subject to y T M i y < = 0 M i = [ r i q i T ; q i Q i ] Lets solve this now !!!

46. Non-Convex Problems Problem is NP-hard. Lets relax the problem to make it convex. Pray !!!

47. Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

48. SDP Relaxation Minimize y T M 0 y Subject to y T M i y < = 0 Minimize M 0 Y Subject to M i Y < = 0 Y = yy T Bad Constraint !!!! No donut for you !!!

49. SDP Relaxation Minimize y T M 0 y Subject to y T M i y < = 0 Minimize M 0 Y Subject to M i Y < = 0 Y >= 0 SDP Problem Nothing left to do …. but Pray Note that we have squared the number of variables.

50. Example - Max Cut Graph: G=(V,E) Maximum-Cut

51. Graph: G=(V,E) Maximum-Cut Example - Max Cut - x i = -1 - x i = +1

52. Graph: G=(V,E) Maximum-Cut Example - Max Cut Alright !!! So its an integer programming problem !!! Doesnt look like quadratic programming to me !!!

53. Max Cut as an IQP Max Cut problem can be written as Naah !! Lets get it into the standard quadratic form.

55. Max Cut as an IQP Max Cut problem can be written as Naah !! Lets get it into the standard quadratic form.

56. Solving Max Cut using SDP Relaxations To the white board. (You didnt think Ill prepare slides for this, did you??)

57. Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

58. SOCP Relaxation Minimize y T M 0 y Subject to y T M i y < = 0 Minimize M 0 Y Subject to M i Y < = 0 Y >= 0 Remember Y = [1 x T ; x X] X - xx T >= 0

59. SOCP Relaxation Say youre given C = { C 1, C 2, … C n } such that C j >= 0 C j (X - xx T ) >= 0 (Ux) T (Ux) <= C j X Wait.. Isnt this a hyperbolic constraint Therefore, its SOCP-able.

60. SOCP Relaxation Minimize y T M 0 y Subject to y T M i y < = 0 Minimize Q 0 X + 2q 0 T x + r 0 Subject to Q i X + 2q i T x + r i < = 0 C j (X - xx T ) >= 0 C j C

61. SOCP Relaxation If C is the infinite set of all semidefinite matrices SOCP Relaxation = SDP Relaxation If C is finite, SOCP relaxation is looser than SDP relaxation. Then why SOCP relaxation ??? Efficiency - Accuracy Tradeoff

62. Choice of C Remember we had squared the number of variables. Lets try to reduce them with our choice of C. For a general problem - Kim and Kojima Using the structure of a specific problem - e.g. Muramatsu and Suzuki for Max Cut

63. Choice of C Minimize c T x Subject to Q i X + 2q i T x + r i < = 0 Q X + 2q T x + r <= 0 Q = n i u i u i T Let 1 >= 2 >= …. k >= 0 >= k+1 >= n

64. Choice of C C = Q + = k i u i u i T Q X + 2q T x + r <= 0 x T Q + x - Q + X <= 0 x T Q + x + k+1 i u i u i T X + 2q T x + r <= 0 zizi

65. Choice of C C = Q + = k i u i u i T x T Q + x + k+1 i z i + 2q T x + r <= 0 u i u i T i = k+1, k+2, … n x T u i u i T x - u i u i T X <= 0 Q X + 2q T x + r <= 0

66. Choice of C C = Q + = k i u i u i T x T Q + x + k+1 i z i + 2q T x + r <= 0 u i u i T i = k+1, k+2, … n x T u i u i T x - z i <= 0 Q X + 2q T x + r <= 0

67. Specific Problem Example - Max Cut e i = [0 0 …. 1 0 …0] u ij = e i + e j v ij = e i - e j C = e i e i T i = 1, …, |V| u ij u ij T (i,j) E v ij v ij T (i,j) E

68. Specific Problem Example - Max Cut Warning: Scary equations to follow.

69. Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

70. Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

71. Back to work now !!!