1. Peter Richtarik School of Mathematics Optimization with Big Data * in a billion dimensional space on a foggy day Extreme* Mountain Climbing =

2. BIG DATA digital images & videos transaction records government records health records defence internet activity (social media, wikipedia,...) scientific measurements (physics, climate models,...) BIG Volume BIG Velocity BIG Variety Sources BIG Volume BIG Velocity BIG Variety

3. Western General Hospital ( Creutzfeldt-Jakob Disease) Arup (Truss Topology Design) Ministry of Defence dstl lab (Algorithms for Data Simplicity) Royal Observatory (Optimal Planet Growth)

4. GOD’S Algorithm = Teleportation

5. If you are not a God... x0x0 x1x1 x2x2 x3x3

6. Optimization as Lock Breaking Setup: Combination maximizing F opens the lock x = (x 1, x 2, x 3, x 4 )F(x) = F(x 1, x 2, x 3, x 4 ) A number representing the “quality” of a combination Optimization Problem: Find combination maximizing F

7. Optimization Algorithm

8. How to Open a Lock with Billion Interconnected Dials? F : R n R # variables/dials = n = 10 9 x1x1 x2x2 Assumption: F = F 1 + F 2 +... + F n ----------------------- F j depends on the neighbours of x j only x3x3 x4x4 Example: F 1 depends on x 1, x 2, x 3 and x 4 F 2 depends on x 1 and x 2,... xnxn

9. Optimization Methods Computing Architectures Multicore CPUs GP GPU accelerators Clusters / Clouds Effectivity Efficiency Scalability Parallelism Distribution Asynchronicity Randomization

10. Optimization Methods for Big Data Randomized Coordinate Descent –P. R. and M. Takac: Parallel coordinate descent methods for big data optimization, ArXiv:1212.0873 [can solve a problem with 1 billion variables in 2 hours using 24 processors] Stochastic (Sub) Gradient Descent –P. R. and M. Takac: Randomized lock-free methods for minimizing partially separable convex functions [can be applied to optimize an unknown function] Both of the above M. Takac, A. Bijral, P. R. and N. Srebro: Mini-batch primal and dual methods for SVMs, ArXiv:1302.xxxx

11. Theory vs Reality

12. Parallel Coordinate Descent

13. TOOLS Probability Machine LearningMatrix Theory HPC