Accelerated, Parallel and PROXimal coordinate descent Moscow February 2014 APPROX Peter Richtárik (Joint work with Olivier Fercoq - arXiv:1312.5799)

Slides:



Advertisements
Similar presentations
Subspace Embeddings for the L1 norm with Applications Christian Sohler David Woodruff TU Dortmund IBM Almaden.
Advertisements

Peter Richtárik Parallel coordinate Simons Institute for the Theory of Computing, Berkeley Parallel and Distributed Algorithms for Inference and Optimization,
05/11/2005 Carnegie Mellon School of Computer Science Aladdin Lamps 05 Combinatorial and algebraic tools for multigrid Yiannis Koutis Computer Science.
Regularized risk minimization
Semi-Stochastic Gradient Descent Methods Jakub Konečný University of Edinburgh Lehigh University December 3, 2014.
Semi-Stochastic Gradient Descent Methods Jakub Konečný University of Edinburgh ETH Zurich November 3, 2014.
Semi-Stochastic Gradient Descent Peter Richtárik ANC/DTC Seminar, School of Informatics, University of Edinburgh Edinburgh - November 4, 2014.
Peter Richtarik Why parallelizing like crazy and being lazy can be good.
Accelerated, Parallel and PROXimal coordinate descent IPAM February 2014 APPROX Peter Richtárik (Joint work with Olivier Fercoq - arXiv: )
Regularization David Kauchak CS 451 – Fall 2013.
Semi-Stochastic Gradient Descent Methods Jakub Konečný (joint work with Peter Richtárik) University of Edinburgh.
Semi-Stochastic Gradient Descent Methods Jakub Konečný University of Edinburgh BASP Frontiers Workshop January 28, 2014.
SA-1 Probabilistic Robotics Planning and Control: Partially Observable Markov Decision Processes.
CSCI 347 / CS 4206: Data Mining Module 07: Implementations Topic 03: Linear Models.
Peter Richtárik Distributed Coordinate Descent For Big Data Optimization Numerical Algorithms and Intelligent Software - Edinburgh – December 5,
Optimization Tutorial
Peter Richtarik School of Mathematics Optimization with Big Data * in a billion dimensional space on a foggy day Extreme* Mountain Climbing =
Graph Laplacian Regularization for Large-Scale Semidefinite Programming Kilian Weinberger et al. NIPS 2006 presented by Aggeliki Tsoli.
Peter Richtárik Parallel coordinate NIPS 2013, Lake Tahoe descent methods.
Distributed Optimization with Arbitrary Local Solvers
1 Reinforcement Learning Introduction & Passive Learning Alan Fern * Based in part on slides by Daniel Weld.
Classification and Prediction: Regression Via Gradient Descent Optimization Bamshad Mobasher DePaul University.
Coefficient Path Algorithms Karl Sjöstrand Informatics and Mathematical Modelling, DTU.
Support Vector Machines (and Kernel Methods in general)
Zheng Qu University of Edinburgh Optimization & Big Data Workshop Edinburgh, 6 th to 8 th May, 2015 Randomized dual coordinate ascent with arbitrary sampling.
Efficient and Numerically Stable Sparse Learning Sihong Xie 1, Wei Fan 2, Olivier Verscheure 2, and Jiangtao Ren 3 1 University of Illinois at Chicago,
Semi-Stochastic Gradient Descent Methods Jakub Konečný (joint work with Peter Richtárik) University of Edinburgh SIAM Annual Meeting, Chicago July 7, 2014.
Blitz: A Principled Meta-Algorithm for Scaling Sparse Optimization Tyler B. Johnson and Carlos Guestrin University of Washington.
Peter Richtárik (joint work with Martin Takáč) Distributed Coordinate Descent Method AmpLab All Hands Meeting - Berkeley - October 29, 2013.
Machine Learning Week 4 Lecture 1. Hand In Data Is coming online later today. I keep test set with approx test images That will be your real test.
Fast Max–Margin Matrix Factorization with Data Augmentation Minjie Xu, Jun Zhu & Bo Zhang Tsinghua University.
CS 8751 ML & KDDSupport Vector Machines1 Support Vector Machines (SVMs) Learning mechanism based on linear programming Chooses a separating plane based.
Mathematical formulation XIAO LIYING. Mathematical formulation.
Efficient and Numerically Stable Sparse Learning Sihong Xie 1, Wei Fan 2, Olivier Verscheure 2, and Jiangtao Ren 3 1 University of Illinois at Chicago,
CS Statistical Machine learning Lecture 18 Yuan (Alan) Qi Purdue CS Oct
Center for Evolutionary Functional Genomics Large-Scale Sparse Logistic Regression Jieping Ye Arizona State University Joint work with Jun Liu and Jianhui.
Ch 4. Linear Models for Classification (1/2) Pattern Recognition and Machine Learning, C. M. Bishop, Summarized and revised by Hee-Woong Lim.
Map-Reduce for Machine Learning on Multicore C. Chu, S.K. Kim, Y. Lin, Y.Y. Yu, G. Bradski, A.Y. Ng, K. Olukotun (NIPS 2006) Shimin Chen Big Data Reading.
Biointelligence Laboratory, Seoul National University
1  The Problem: Consider a two class task with ω 1, ω 2   LINEAR CLASSIFIERS.
ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Ensemble Methods: Bagging, Boosting Readings: Murphy 16.4; Hastie 16.
Massive Support Vector Regression (via Row and Column Chunking) David R. Musicant and O.L. Mangasarian NIPS 99 Workshop on Learning With Support Vectors.
Learning by Loss Minimization. Machine learning: Learn a Function from Examples Function: Examples: – Supervised: – Unsupervised: – Semisuprvised:
Coordinate Descent Algorithms
Combining Models Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya.
For convex optimization
StingyCD: Safely Avoiding Wasteful Updates in Coordinate Descent
The role of optimization in machine learning
Sathya Ronak Alisha Zach Devin Josh
Zhu Han University of Houston Thanks for Dr. Mingyi Hong’s slides
Multiplicative updates for L1-regularized regression
Zhu Han University of Houston Thanks for Dr. Mingyi Hong’s slides
Lecture 07: Soft-margin SVM
Announcements HW4 due today (11:59pm) HW5 out today (due 11/17 11:59pm)
Machine Learning Basics
Nonnegative polynomials and applications to learning
ECE 5424: Introduction to Machine Learning
For convex optimization
Lecture 07: Soft-margin SVM
CSCI B609: “Foundations of Data Science”
Logistic Regression & Parallel SGD
Parallel and Distributed Block Coordinate Frank Wolfe
PEGASOS Primal Estimated sub-GrAdient Solver for SVM
Lecture 08: Soft-margin SVM
Lecture 07: Soft-margin SVM
Support Vector Machines
CS639: Data Management for Data Science
Linear Discrimination
ADMM and DSO.
Presentation transcript:

Accelerated, Parallel and PROXimal coordinate descent Moscow February 2014 APPROX Peter Richtárik (Joint work with Olivier Fercoq - arXiv: )

Optimization Problem

Problem Convex (smooth or nonsmooth) Convex (smooth or nonsmooth) - separable - allow Loss Regularizer

Regularizer: examples No regularizerWeighted L1 norm Weighted L2 norm Box constraints e.g., SVM dual e.g., LASSO

Loss: examples Quadratic loss L-infinity L1 regression Exponential loss Logistic loss Square hinge loss BKBG’11 RT’11b TBRS’13 RT ’13a FR’13

RANDOMIZED COORDINATE DESCENT IN 2D

Find the minimizer of 2D Optimization Contours of a function Goal:

Randomized Coordinate Descent in 2D N S E W

1 N S E W

1 N S E W 2

1 2 3 N S E W

N S E W

N S E W 5

N S E W

N S E W 6 7 S O L V E D !

CONTRIBUTIONS

Variants of Randomized Coordinate Descent Methods Block – can operate on “blocks” of coordinates – as opposed to just on individual coordinates General – applies to “general” (=smooth convex) functions – as opposed to special ones such as quadratics Proximal – admits a “nonsmooth regularizer” that is kept intact in solving subproblems – regularizer not smoothed, nor approximated Parallel – operates on multiple blocks / coordinates in parallel – as opposed to just 1 block / coordinate at a time Accelerated – achieves O(1/k^2) convergence rate for convex functions – as opposed to O(1/k) Efficient – complexity of 1 iteration is O(1) per processor on sparse problems – as opposed to O(# coordinates) : avoids adding two full vectors

Brief History of Randomized Coordinate Descent Methods + new long stepsizes

APPROX

“PROXIMAL” “PARALLEL” “ACCELERATED” APPROX

PCDM (R. & Takáč, 2012) = APPROX if we force

APPROX: Smooth Case Want this to be as large as possible Update for coordinate i Partial derivative of f

CONVERGENCE RATE

Convergence Rate average # coordinates updated / iteration # coordinates # iterations implies Theorem [FR’13b] Key assumption

Special Case: Fully Parallel Variant all coordinates are updated in each iteration # normalized weights (summing to n) # iterations implies

Special Case: Effect of New Stepsizes Average degree of separability “Average” of the Lipschitz constants With the new stepsizes (will mention later!), we have:

“EFFICIENCY” OF APPROX

Cost of 1 Iteration of APPROX Assume N = n (all blocks are of size 1) and that Sparse matrix Then the average cost of 1 iteration of APPROX is Scalar function: derivative = O(1) arithmetic ops = average # nonzeros in a column of A

Bottleneck: Computation of Partial Derivatives maintained

PRELIMINARY EXPERIMENTS

L1 Regularized L1 Regression Dorothea dataset: Gradient Method Nesterov’s Accelerated Gradient Method SPCDM APPROX

L1 Regularized L1 Regression

L1 Regularized Least Squares (LASSO) KDDB dataset: PCDM APPROX

Training Linear SVMs Malicious URL dataset:

Choice of Stepsizes: How (not) to Parallelize Coordinate Descent

Convergence of Randomized Coordinate Descent Strongly convex F (Simple Mehod) Smooth or ‘simple’ nonsmooth F (Accelerated Method) ‘Difficult’ nonsmooth F (Accelerated Method) or smooth F (Simple method) ‘Difficult’ nonsmooth F (Simple Method) Focus on n (big data = big n)

Parallelization Dream Depends on to what extent we can add up individual updates, which depends on the properties of F and the way coordinates are chosen at each iteration SerialParallel What do we actually get? WANT

“Naive” parallelization Do the same thing as before, but for MORE or ALL coordinates & ADD UP the updates

Failure of naive parallelization 1a 1b 0

Failure of naive parallelization 1 1a 1b 0

Failure of naive parallelization 1 2a 2b

Failure of naive parallelization 1 2a 2b 2

Failure of naive parallelization 2 O O P S !

1 1a 1b 0 Idea: averaging updates may help S O L V E D !

Averaging can be too conservative 1a 1b 0 1 2a 2b 2 a n d s o o n...

Averaging may be too conservative 2 WANT BAD!!! But we wanted:

What to do? Averaging: Summation: Update to coordinate i i-th unit coordinate vector Figure out when one can safely use:

ESO: Expected Separable Overapproximation

5 Models for f Admitting Small Smooth partially separable f [RT’11b ] Nonsmooth max-type f [FR’13] f with ‘bounded Hessian’ [BKBG’11, RT’13a ]

5 Partially separable f with block smooth components [FR’13b] 5 Models for f Admitting Small 4 Partially separable f with smooth components [NC’13]

Randomized Parallel Coordinate Descent Method Random set of coordinates (sampling) Current iterateNew iteratei-th unit coordinate vector Update to i-th coordinate

ESO: Expected Separable Overapproximation Definition [RT’11b] 1. Separable in h 2. Can minimize in parallel 3. Can compute updates for only Shorthand: Minimize in h

Convergence Rate of APPROX average # coordinates updated / iteration # coordinates # iterations implies Theorem [FR’13b] Key assumption

Convergence Rate of PCDM: Convex f average # updated coordinates per iteration # coordinatesstepsize parameter error tolerance # iterations implies Theorem [RT’11b]

Convergence Rate of PCDM: strongly convex f implies Strong convexity constant of the regularizer Strong convexity constant of the loss f Theorem [RT’11b]

PART II. ADDITIONAL TOPICS

Partial Separability and Doubly Uniform Samplings

Serial uniform sampling Probability law:

-nice sampling Probability law: Good for shared memory systems

Doubly uniform sampling Probability law: Can model unreliable processors / machines

ESO for partially separable functions and doubly uniform samplings Theorem [RT’11b] 1 Smooth partially separable f [RT’11b ]

PCDM: Theoretical Speedup Much of Big Data is here! degree of partial separability # coordinates # coordinate updates / iter WEAK OR NO SPEEDUP: Non-separable (dense) problems LINEAR OR GOOD SPEEDUP: Nearly separable (sparse) problems

n = 1000 (# coordinates) Theory

Practice n = 1000 (# coordinates)

PCDM: Experiment with a 1 billion-by-2 billion LASSO problem

Optimization with Big Data * in a billion dimensional space on a foggy day Extreme* Mountain Climbing =

Coordinate Updates

Iterations

Wall Time

Distributed-Memory Coordinate Descent

Distributed -nice sampling Probability law: Machine 2Machine 1Machine 3 Good for a distributed version of coordinate descent

ESO: Distributed setting Theorem [RT’13b] 3 f with ‘bounded Hessian’ [BKBG’11, RT’13a ] spectral norm of the data

Bad partitioning at most doubles # of iterations spectral norm of the partitioning Theorem [RT’13b] # nodes # iterations = implies # updates/node

LASSO with a 3TB data matrix 128 Cray XE6 nodes with 4 MPI processes (c = 512) Each node: 2 x 16-cores with 32GB RAM = # coordinates

Shai Shalev-Shwartz and Ambuj Tewari, Stochastic methods for L1-regularized loss minimization. JMLR Yurii Nesterov, Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22(2): , [RT’11b] P.R. and Martin Takáč, Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. Mathematical Prog., Rachael Tappenden, P.R. and Jacek Gondzio, Inexact coordinate descent: complexity and preconditioning, arXiv: , Ion Necoara, Yurii Nesterov, and Francois Glineur. Efficiency of randomized coordinate descent methods on optimization problems with linearly coupled constraints. Technical report, Politehnica University of Bucharest, Zhaosong Lu and Lin Xiao. On the complexity analysis of randomized block- coordinate descent methods. Technical report, Microsoft Research, References: serial coordinate descent

[BKBG’11] Joseph Bradley, Aapo Kyrola, Danny Bickson and Carlos Guestrin, Parallel Coordinate Descent for L1-Regularized Loss Minimization. ICML 2011 [RT’12] P.R. and Martin Takáč, Parallel coordinate descen methods for big data optimization. arXiv: , 2012 Martin Takáč, Avleen Bijral, P.R., and Nathan Srebro. Mini-batch primal and dual methods for SVMs. ICML 2013 [FR’13a] Olivier Fercoq and P.R., Smooth minimization of nonsmooth functions with parallel coordinate descent methods. arXiv: , 2013 [RT’13a] P.R. and Martin Takáč, Distributed coordinate descent method for big data learning. arXiv: , 2013 [RT’13b] P.R. and Martin Takáč, On optimal probabilities in stochastic coordinate descent methods. arXiv: , 2013 References: parallel coordinate descent Good entry point to the topic (4p paper)

P.R. and Martin Takáč, Efficient serial and parallel coordinate descent methods for huge-scale truss topology design. Operations Research Proceedings Rachael Tappenden, P.R. and Burak Buke, Separable approximations and decomposition methods for the augmented Lagrangian. arXiv: , Indranil Palit and Chandan K. Reddy. Scalable and parallel boosting with MapReduce. IEEE Transactions on Knowledge and Data Engineering, 24(10): , [FR’13b] Olivier Fercoq and P.R., Accelerated, Parallel and Proximal coordinate descent. arXiv: , 2013 References: parallel coordinate descent