1. Continuous optimization Problems and successes
Tijl De Bie
Intelligent Systems Laboratory MVSE, University of Bristol United Kingdom

2. Motivation
Back-propagation algorithm for training neural networks (gradient descent)
Support vector machines
Convex optimization `boom’ (NIPS, also ICML, KDD...)
What explains this success?
(Is it really a success?)
(Mainly for CP-ers not familiar with continuous optimization)

3. (Convex) continuous optimization
Convex optimization:

4. Convex optimization

5. Convex optimization General convex optimization approach
Start with a guess, iteratively improve until optimum found
E.g. Gradient descent, conjugate gradient, Newton method, etc
For constrained convex optimization: Interior point methods
Provably efficient (worst-case, typical case even better)
Iteration complexity:
Complexity per iteration: polynomial
Out-of-the-box tools exist (SeDuMi, SDPT3, MOSEK...)
Purely declarative
Book: Convex Optimization (Boyd & Vandenberghe)

6. Convex optimization Convex optimization Logdet Cone Programming LP QP
Geometric programming
SOCP
SDP

7. Linear Programming (LP)
Linear objective Linear inequality constraints Affine equality constraints
Applications:
Relaxations of Integer LP’s
Classification: linear support vector machines (SVM), forms of boosting
(Lots outside DM/ML)

8. Convex Quadratic Programming (QP)
Convex Quadratic constraints
LP is a special case where
Applications:
Classification/regression: SVM
Novelty detection: minimum volume enclosing hypersphere
Regression + feature selection: lasso
Structured prediction problems

9. Second-Order Cone Programming (SOCP)
Second Order Cone constraints
QCQP is a special case where
Applications:
Metric learning
Fermat-Weber problem: find a point in a plane with minimal sum of distances to a set of points
Robust linear programming

10. Semi-Definite Programming (SDP)
Constraints requiring a matrix to be Positive Semi-Definite:
SOCP is a special case:
Applications:
Metric learning
Low rank matrix approximations (dimensionality reduction)
Very tight relaxations of graph labeling problems (e.g. Max-cut)
Semi-supervised learning
Approximate inference in difficult graphical models

11. Geometric programming
Objective and constraints of the form:
Applications:
Maximum entropy modeling with moment constraints
Maximum likelihood fitting of exponential family distributions

12. Log Determinant Optimization (Logdet)
Objective is the log determinant of a matrix:
= -volume of parallelepiped spanned by columns of X
Applications:
Novelty detection: minimum volume enclosing ellipsoid
Experimental design / active learning (which labels for which data points are likely to be most informative)

13. Eigenvalue problems
Eigenvalue problems are not convex optimization problems
Still, a relatively efficient and globally convergent, and a useful primitive:
Dimensionality reduction (PCA)
Finding relations between datasets (CCA)
Spectral clustering
Metric learning
Relaxations of combinatorial problems

14. The hype Very popular in conferences like NIPS, ICML, KDD
These model classes are sufficiently rich to do sophisticated things
Sparsity: L1 norm/linear constraints  feature selection
Low-rank of matrices: SDP constraint and trace norm (sparse PCA, labeling problems...)
Declarative nature, little expertise needed
Computational complexity is easy to understand

15. After the hype But: Tendency toward other paradigms:
Polynomial-time, often with a high exponent E.g. SDP: and sometimes
Convex constraints can be too limitative
Tendency toward other paradigms:
Convex-concave programming (Few guarantees, but works well in practice)
Submodular optimization (Approximation guarantees, works well in practice)

16. CP vs Convex Optimization
“CP: Choosing the best model is an art” (Helmut) “CP requires skill and ingenuity” (Barry)
I understand in CP there is a hierarchy of propagation methods, but...
Is there a hierarchy of problem complexities?
How hard is it to see if a constraint will propagate well?
Does it depend on the implementation?
...