Download presentation

Presentation is loading. Please wait.

Published bySantos Stovall Modified over 2 years ago

1
Concurrency Control for Machine Learning Joseph E. Gonzalez Post-doc, UC Berkeley AMPLab jegonzal@eecs.berkeley.edu In Collaboration with Xinghao Pan, Stefanie Jegelka, Tamara Broderick, Michael I. Jordan

2
Data Model Parameters Serial Machine Learning Algorithm

3
Data Model Parameters Parallel Machine Learning

4
Data Model Parameters ! ! Parallel Machine Learning Concurrency: more machines = less time Correctness: serial equivalence

5
Data Model Parameters Coordination-free

6
Data Model Parameters Concurrency Control

7
Data Model Parameters Serializability

8
Research Summary Coordination Free (e.g., Hogwild): Provably fast and correct under key assumptions. Concurrency Control (e.g., Mutual Exclusion): Provably correct and fast under key assumptions. Research Focus

9
Stability & Correctness Concurrency Coordination- free Mutual exclusion High LowHigh Low Optimistic Concurrency Control ?

10
Optimistic Concurrency Control to parallelize: Non-Parametric Clustering and Sub-modular Maximization

11
Data Model Parameters Optimistic Concurrency Control Optimistic updates Validation: detect conflict Resolution: fix conflict ! ! Hsiang-Tsung Kung and John T Robinson. On optimistic methods for concurrency control. ACM Transactions on Database Systems (TODS), 6(2):213–226, 1981. Concurrency Correctness

12
Example: Serial DP-means Clustering Sequential! Brian Kulis and Michael I. Jordan. Revisiting k-means: New algorithms via Bayesian nonparametrics. In Proceedings of 23rd International Conference on Machine Learning, 2012.

13
Validation Resolution First proposal wins Assumption No new cluster created nearby Example: OCC DP-means Clustering

14
Optimistic Concurrency Control for DP-means Theorem: OCC DP-means is serializable. Corollary: OCC DP-means preserves theoretical properties of DP-means. Theorem: Expected overhead of OCC DP- means, in terms of number of rejected proposals, does not depend on size of data set. Correctness Concurren cy

15
Evaluation: Amazon EC2 OCC DP-means Runtime Projected Linear Scaling ~140 million data points; 1, 2, 4, 8 machines

16
Optimistic Concurrency Control to parallelize Non-Parametric Clustering Summary Sub-modular Maximization Next

17
Motivating Example Bidding on Keywords: Apple iPhone Android Games xBox Samsung Microwave Appliances Keywords How big is Apple iPhone iPhone vs Android best Android and iPhone games Samsung sues Apple over iPhone Samsung Microwaves Appliance stores in SF Playing games on a Samsung TV xBox game of the year Common Queries

18
Motivating Example Bidding on Keywords: Apple iPhone Android Games xBox Samsung Microwave Appliances Keywords How big is Apple iPhone iPhone vs Android best Android and iPhone games Samsung sues Apple over iPhone Samsung Microwaves Appliance stores in SF Playing games on a Samsung TV xBox game of the year Common Queries A B C D E F G H Keywords Queries 1 2 3 4 5 6 7 8

19
Motivating Example Bidding on Keywords: Keywords Queries A B C D E F G H 1 2 3 4 5 6 7 8 $2 $5 $1 $2 $5 $1 $4 $2 Costs $2 $4 $3 $6 $5 $1 Value

20
Purchas e Motivating Example Bidding on Keywords: Keywords Queries A C D E F G H 5 6 7 8 $2 $5 $1 $2 $5 $1 $4 $2 Costs $2 $4 $3 $6 $5 $1 Value B 1 2 3 4 Cover $5 - Cost: $1 2 Revenue: $7 Profit:

21
Purchas e Motivating Example Bidding on Keywords: Keywords Queries A D E F G H 5 6 7 8 $2 $5 $1 $2 $5 $1 $4 $2 Costs $2 $4 $3 $6 $5 $1 Value B 1 4 Cover C 2 3 $1 2 $5 - Cost: Revenue: $7 Profit: +1 $6 Submodularity = Diminishing Returns

22
Purchas e Motivating Example Bidding on Keywords: Keywords Queries A B C D E F G H 1 2 3 4 5 6 7 8 $2 $5 $1 $2 $5 $1 $4 $2 Costs $2 $4 $3 $6 $5 $1 Value $2 0 $1 0 - Cost: Revenue: $1 0 Profit:

23
Purchas e Motivating Example Bidding on Keywords: Keywords Queries A B C D E F G H 1 2 3 4 5 6 7 8 $2 $5 $1 $2 $5 $1 $4 $2 Costs $2 $4 $3 $6 $5 $1 Value $2 0 $1 0 - Cost: Revenue: $1 0 Profit: - 4 +6 $2 0 NP-Hard in General

24
Submodular Maximization NP-Hard in General Buchbinder et al. [FOCS12] proposed the double-greedy randomized algorithm which is provably optimal.

25
f(, X, Y ) = Double Greedy Algorithm Process keywords serially Keywords Queries A B C D E F 1 2 3 4 5 6 Set X Set Y A B C D E F Add XRem. Y 01 A rand A Keywords to purchase

26
f(, X, Y ) = Double Greedy Algorithm Process keywords serially Keywords Queries A B C D E F 1 2 3 4 5 6 Set X Set Y A B C D E F Add XRem. Y 01 B rand A Keywords to purchase

27
f(, X, Y ) = Double Greedy Algorithm Process keywords serially Keywords Queries A B C D E F 1 2 3 4 5 6 Set X Set Y A C D E F Add XRem. Y 01 C rand A C Keywords to purchase

28
Concurrency Control Double Greedy Algorithm Process keywords in parallel Keywords Queries A B C D E F 1 2 3 4 5 6 Set X Set Y A C D E F B Within each processor: f(, X bnd,Y bnd )= Add XRem. Y 01 A Subset of true X Superset of true Y Uncertaint y Keywords to purchase Sets X and Y are shared by all processors.

29
Concurrency Control Double Greedy Algorithm Process keywords in parallel Keywords Queries A B C D E F 1 2 3 4 5 6 Set X Set Y A C D E F B Within each processor: f(, X bnd,Y bnd )= Add XRem. Y 01 A Subset of true X Superset of true Y Uncertaint y rand A Safe rand Unsafe Must Validate Keywords to purchase Sets X and Y are shared by all processors.

30
Concurrency Control Double Greedy Algorithm System Design Implemented in multicore (shared memory): Model Server (Validator) Set X Set Y A C D E F A Validation Queue Published Bounds (X,Y) Published Bounds (X,Y) Bound (X,Y) D Trx. Add X D Bound (X,Y) E Fail E Thread 1 f(, X bnd,Y bnd )= Add X Rem. Y 01 D Uncertainty Thread 2 f(, X bnd,Y bnd )= Add XRem. Y 01 E Uncertaint y

31
Provable Properties Theorem: CC double greedy is serializable. Corollary: CC double greedy preserves optimal approximation guarantee of ½OPT. Lemma: CC has bounded overhead. set cover with costs: 2τ sparse max cut:2cτ/n Correctness Concurren cy

32
Provable Properties – coord free? Theorem: CF double greedy is serializable. Lemma: CF double greedy achieves approximation guarantee of ½OPT – ¼ Lemma: CC has bounded overhead. set cover with costs: 2τ sparse max cut:2cτ/n Correctness Concurren cy depends on uncertainty region similar order of CC overhead!

33
Provable Properties – coord free? Theorem: CF double greedy is serializable. Lemma: CF double greedy achieves approximation guarantee of ½OPT – ¼ CF: no coordination overhead. Correctness Concurren cy depends on uncertainty region similar order of CC overhead!

34
Early Results

35
Runtime and Strong-Scaling IT-2004: Italian Web-graph (41M Vertices, 1.1B Edges) UK-2005: UK Web-graph (39M, 921M Edges) Arabic-2005: Arabic Web-graph (22M, 631M Edges) Coordination Free Concurrency Ctrl.

36
Coordination and Guarantees IT-2004: Italian Web-graph (41M Vertices, 1.1B Edges) UK-2005: UK Web-graph (39M, 921M Edges) Arabic-2005: Arabic Web-graph (22M, 631M Edges) Increase in Coordination Bad Decrease in Objective

37
Summary New primitives for robust parallel algorithm design –Exploit properties in ML algorithms Introduced parallel algorithms for: –DP-Means –Submodular Maximization Future Work: Integrate with Velox Model Server

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google