Presentation is loading. Please wait.

Presentation is loading. Please wait.

High Performance Parallel Stochastic Gradient Descent

Similar presentations


Presentation on theme: "High Performance Parallel Stochastic Gradient Descent"— Presentation transcript:

1 High Performance Parallel Stochastic Gradient Descent
IN SHARED MEMORY 1,2Scott Sallinen, 2Nadathur Satish, 2Mikhail Smelyanskiy, 2Samantika Sury, 3Christopher Ré 1University of British Columbia, 2Intel Corporation, 3Stanford University

2 Overview of Regression and Stochastic gradient descent
Parallelizing SGD Overview of Regression & SGD Experimental Results Comparison to State of the Art Overview of Regression and Stochastic gradient descent

3 Regression Goal of regression is to model and analyze data
Many types: Linear, Polynomial, Least Squares, Logistic… Generally Sparse, Convex Problems Training Strategy is representative of many machine learning training techniques

4 Single Model Regression
Want to create a model M, based on input dataset X, and their corresponding training labels Y. # Features D 1 # Features D X Y 1 M # Samples N # Samples N Dataset Labels Model (weights)

5 Stochastic Gradient Descent
Stochastic Gradient Method [Robbins & Monro, 1951] Select index i 𝑀 𝑡+1 = 𝑀 𝑡 − 𝑎 𝑡 ∗ 𝑔 𝑖 ( 𝑀 𝑡 ) Much faster than full gradient descent Note: Each model is a direct descendant of the previous. # Features D 1 # Features D i X Y 1 M # Samples N # Samples N Dataset Labels Model (weights)

6 Stochastic Gradient Descent (Convex)
Training looks something like this: Source: Adam Harley

7 Many faces of SGD Not only for Logistic regression. Select Sample(s)
Other Regressions Support Vector Machines Least Mean Squares Neural Network Training And more... There are many options for each step. Select Sample(s) Compute Gradient Update Model

8 Parallelizing Stochastic Gradient Descent
Parallelizing SGD Overview of Regression & SGD Experimental Results Comparison to State of the Art Parallelizing Stochastic Gradient Descent

9 Opportunities for Parallelism
Typically problems (data sets) are sparse. Amount of parallelism for the vector computation is dynamic (number of non- zeros in the given row), and is typically small: using threads here is non-ideal. We should parallelize across samples. Select Sample(s): ~ Compute Gradient: Logistic Function Update Model: Learning Rate (𝛼/sqrt(iter)) All Samples (GD) 1 Sample (SGD) S Samples (MINIBATCH) T Samples (HOGWILD)

10 Mini-Batching Select index b 𝑀 𝑡+𝑆 = 𝑀 𝑡 − 𝑎 𝑡 ∗ 𝑖=𝑏 𝑏+𝑆 𝑔 𝑖 ( 𝑀 𝑡 )
𝑀 𝑡+𝑆 = 𝑀 𝑡 − 𝑎 𝑡 ∗ 𝑖=𝑏 𝑏+𝑆 𝑔 𝑖 ( 𝑀 𝑡 ) Convergence between full Gradient Descent, and stochastic, depending on size of the batch. Parallelize across samples within the batch. (Similar to SpMDV). # Features D 1 # Features D X Y 1 M b S T1 T2 # Samples N # Samples N Dataset Labels Model (weights)

11 Mini-Batching: Key Aspects
 One model update per batch size  Private thread gradient vectors  Reduction  Thread barrier between updates (synchronization steps)  No conflicts, but “stale”: Update is a descendant of the model from the previous batch, rather than previous sample.  # Features D 1 # Features D X Y 1 M b S T1 T2 # Samples N # Samples N Dataset Labels Model (weights)

12 Hogwild Each thread: Select index i 𝑀 𝑡+1 = 𝑀 𝑡 − 𝑎 𝑡 ∗ 𝑔 𝑖 ( 𝑀 𝑡 )
𝑀 𝑡+1 = 𝑀 𝑡 − 𝑎 𝑡 ∗ 𝑔 𝑖 ( 𝑀 𝑡 ) Each model update not guaranteed a direct descendant. Write-races acceptable since the algorithm “refines” anyway. However when sparse updates touch different indices in the weight vector, there is no dependency. # Features D 1 # Features D X Y 1 M # Samples N # Samples N Dataset Labels Model (weights)

13 Hogwild: Key Aspects # Features D 1 # Features D  Thread asynchronicity  No reductions: async. SpVDV  One update for every sample  Potential for conflict…  … But not stale due to sharing the model  Cross-core false sharing of the model vector X Y 1 M False sharing: Indices reside on the same cache line, but are not actually conflicting. # Samples N # Samples N Dataset Labels Model (weights) Parallel for-all print  

14 Summary: Hogwild vs Mini-Batching
 Thread asynchronicity  One update for every sample  No reductions, async. update  Cross core sharing model vector  Potential for conflict, but not stale  Thread barrier between updates  One update per batch size  Reduction  Private gradient vectors  No conflicts, but stale unify

15 Introducing: Hogwild Batching (HogBatching)
Each thread: Select index b 𝑀 𝑡+𝑆 = 𝑀 𝑡 − 𝑎 𝑡 ∗ 𝑖=𝑏 𝑏+𝑆 𝑔 𝑖 ( 𝑀 𝑡 ) Parallelize across batches instead of samples. Statistical Efficiency (Convergence): At best as good as Hogwild, at worst as poor as Mini-Batching. # Features D 1 # Features D X Y 1 M S T1 S T2 # Samples N # Samples N S T3 Dataset Labels Model (weights)

16 HogBatching: Hierarchal Parallelism
To extend to many-core, we need to map more parallelism. # Features D 1 # Features D X Y 1 M S S Solution: Divide parallelism to inner and outer parallelism. # Samples N # Samples N There are many HogBatches running asynchronously S The internals of a HogBatch is just a small SGD problem Dataset Labels Model (weights)

17 HogBatching: Hierarchal Parallelism
Groups work on batches as Outer parallelism (parallel across batches). The entire group applies only one update to the Model vector. Workers within the groups perform Inner parallelism, in the form of a small Mini-Batch (parallel across samples). SIMD within the workers process the vector of the sample (parallel across elements). Perfect for multiple threads on the same core and cache (e.g. hyperthreading)

18 HogBatch: Note on Algorithm Identities
Creates nice ‘bridge’ or identity between the three methods: HogBatch, with a batch size of 1, is just Hogwild. HogBatch, with outer parallelism of 1, is just Mini-Batching. Further: All three methods are functionally equivalent to Serial SGD when executed with one thread. HogBatching is a general solution, in between the previous two methods. unify

19 Experimental Results Overview of Regression & SGD Parallelizing SGD
Comparison to State of the Art Experimental Results

20 Parallel scaling of Strategies Sparse Problem (0.155% nz)
RCV1-test dataset. Good case for Hogwild Poor case for batching Large feature size - Large reduction - Large dense update ? Features (column) tend to be power law. Sparse does not imply uniformly sparse Dataset Name Examples Features NNZ Sparsity (%) NNZ/Row Range Avg NNZ/Row rcv1-test 677,399 47,236 49,556,258 0.155 4 to 1,224 73.157

21 Parallel scaling of Strategies Sparse Problem (0.155% nz)
Raw “time to solution” Time-per-Datapass x Number-of-Datapasses Impact of measuring it this way is important! Mini-Batching has excellent hardware efficiency (low time-per-datapass). Hogwild has excellent statistical efficiency (low number-of-datapasses). … but each suffers in the other aspect. HogBatching takes the best of both worlds for a 2.4x improvement. Speedup compared to serial, on ‘effective time to solution’ of 99.5% closeness to optimal. This is a product of both statistical and hardware efficiency. Dataset Name Examples Features NNZ Sparsity (%) NNZ/Row Range Avg NNZ/Row rcv1-test 677,399 47,236 49,556,258 0.155 4 to 1,224 73.157

22 Parallel scaling of Strategies Denser Problem (22% nz)
Raw “time to solution” Time-per-Datapass x Number-of-Datapasses Hogwild suffered in hardware efficiency due to false sharing a small model. Mini-Batching suffered in hardware efficiency due to synchronizing threads constantly during small and rapid batches. (Note: A larger batch caused oscillations or poorer performance). HogBatching suffers no hardware inefficiencies: achieving a 20x performance improvement. Speedup compared to serial, on ‘effective time to solution’ of 99.5% closeness to optimal. This is a product of both statistical and hardware efficiency. Constant write pressure on shared, small weight vector (size of #Features). Batches must be small to get good convergence. Batches are completed quickly due to small feature size. Dataset Name Examples Features NNZ Sparsity (%) NNZ/Row Range Avg NNZ/Row covtype 581,012 54 6,940,438 22.121 9 to 12 11.945

23 Comparison to state-of-the-art
Parallelizing SGD Overview of Regression & SGD Experimental Results Comparison to State of the Art Comparison to state-of-the-art

24 BIDMach State-of-the-art framework making noise for the GPU being much faster than CPU solutions.

25 Single Model Comparison
Comparison to BIDMach Apples to Apples comparison.* We discussed with the authors, and were told that single model is not an optimized case for BIDMach. * We set all parameters (learning rate, batch size, regularization) to the same as BIDMach. We run BIDMach on our machine. We use the ADAGRAD update, since it is what they use. We discussed with the authors to ensure we were evaluating correctly. Implementation Hardware Time/Pass (ms) BidMach TITAN X 723 Sandy Bridge 14,190 Intel (Batching) 289 Intel (Hogwild) 253 Intel (HogBatching) 147 Haswell 111 Dataset: RCV1-V2, 1 model, ADAGRAD update. Single precision. 1 Socket CPU. Dataset Name Examples Features NNZ Sparsity (%) NNZ/Row Range Avg NNZ/Row RCV1-V2 781,265 276,544 60,534,218 0.028 4 to 1585 77.482

26 Multi Model Comparison
Comparison to BIDMach Apples to Apples comparison. Time per pass per model begins to level off as SIMD units become saturated. 2 CPUs, ~64 models each: very good scaling! This also shows the importance of using strong baselines in CPU vs GPU comparisons. Hardware Models Time/Pass (ms) BidMach TITAN X 103 2,170 Sandy Bridge 120,720 Intel 2,010 Haswell 1,283 2x Haswell 724 Dataset: RCV1-V2. Time per pass for 103 models. Single precision. 1 Socket CPU except the last result. Dataset Name Examples Features NNZ Sparsity (%) NNZ/Row Range Avg NNZ/Row RCV1-V2 781,265 276,544 60,534,218 0.028 4 to 1585 77.482

27 Questions? Lessons Learned
Hardware Efficiency is extremely important during training phases, and needs to be considered as well. Don’t only focus on Statistical Efficiency, as total time is a product of both: “time-per-datapass” x “number-of-datapasses” When bounded by inter-core communication, using both privatization and asynchronicity together is key to improving performance. Questions? Contact:

28 Backup slides

29 Multi Model Regression
Parallelizing SGD Overview of Regression & SGD Experimental Results Comparison to State of the Art Multi Model Regression Multi Model Regression

30 SGD, Multi Model Case Select index i Easy to parallelize X Y M
𝑚=1 𝑀 𝑀 𝑚 [𝑡+1] = 𝑀 𝑚 [𝑡] − 𝑎 𝑡 ∗ 𝑔 𝑚 [𝑖] ( 𝑀 𝑚 [𝑡] ) Easy to parallelize # Features D m # Features D X Y M m # Samples N Dataset Labels Models (weights)

31 Each written index in the model is SIMD friendly.
SGD, Multi Model Case Easy to parallelize SIMD units work across static models, instead of across dynamically sized vectors. Batching is no longer a viable strategy due to increased size of labels and models. They are dense, and now 2 dimensional, no longer able to be duplicated as thread private temporary vectors which fit in core cache. HogBatching is not useful.  # Features D m # Features D X Y M m # Samples N Each written index in the model is SIMD friendly. Dataset Labels Models (weights)

32 Multi Model Scaling Parallelism increase offers near- linear scaling until a saturation at ~16 models. Also shown is the ADAGRAD update. Convergence comparisons and arguments for which is better, is best left for another time. Note: Log-Log Axis. Single precision. Dataset Name Examples Features NNZ Sparsity (%) NNZ/Row Range Avg NNZ/Row RCV1-V2 781,265 276,544 60,534,218 0.028 4 to 1585 77.482

33 Calculating error: Logistic Regression
A common model to perform Regression Logistic Function: 1 1+ 𝑒 −𝑡

34 Average Gradient (SAG)
Many faces of SGD Application of gradient to model weights. Select Sample(s) Learning rate (SGD) Compute Gradient ADAGRAD Average Gradient (SAG) Update Model Momentum

35 Support Vector Machine
Many faces of SGD Gradient Computation. Select Sample(s) Linear Function Compute Gradient Logistic Function Support Vector Machine Update Model Least Squares

36 Many faces of SGD How we choose samples to update. Select Sample(s)
All Samples (GD) Compute Gradient 1 Sample (SGD) B Samples (BATCH SGD) Update Model T Samples (HOGWILD)

37 Single Model Regression
Deterministic Gradient Method [Cauchy, 1847] 𝑀 𝑡+1 = 𝑀 𝑡 − 𝑎 𝑡 𝑁 ∗ 𝑖=1 𝑁 𝑔 𝑖 ( 𝑀 𝑡 ) Iteration cost linear in N x D # Features D 1 # Features D X X Y Y 1 M # Samples N # Samples N Dataset Labels Model (weights)

38 Serial SGD: Application Pattern
Semi - expanded algorithm (Logistic Regression) Gather for (index = 0; index < Samples; index++) { } ** regularization is not shown for simplicity for (non-zero indices j of X[index]) { dotProduct += X[index][j] * model[j] } Apply g = y[index] / 1 + exp(y[index] * dotProduct) update = a/sqrt(index) * g for (non-zero indices j of X[index]) { model[j] = model[j] – update } Scatter

39 Serial SGD: Dependency Pattern
Expanded Logistic Regression # Features D 1 # Features D X Y 1 M for (index = 0; index < Samples; index++) { for (non-zero indices j of X[index]) { dotProd += X[index][j] * model[j] } g = y[index] / 1 + exp(y[index] * dotProd) update = a/sqrt(index) * g model[j] = model[j] – update ** regularization is not shown for simplicity # Samples N # Samples N Update only touches sparse indices in the vector Dataset Labels Model (weights)

40 Batching Model is read-only during the batch
for (st = 0; st < num_samples/SIZE; st += SIZE) { #pragma omp parallel for for (index = st; index < SIZE; index++) { // Sparse vector operation. g_tid[TID] += a * Gradient(model, index) } // (implicit thread barrier) for (f = 0; f < num_features; f++) { for (t = 0; t < NUM_THREADS; t++) model[f] = model[f] - g_tid[t][f] } Model is written once at the end of the batch, by all threads

41 Introducing: HogBatching
Model is read-only (to current thread) during the batch #pragma omp parallel for schedule(dynamic) for (st = 0; st < num_samples/SIZE; st += SIZE) { for (index = st; index < SIZE; index++) { // Sparse vector operation. g_tid[TID] += a * Gradient(model, index) } for (f = 0; f < num_features; f++) model[f] = model[f] - g_tid[TID][f] Model is written once at the end of the batch by the thread

42 Across datasets HogBatch offers speedup across most datasets that we tested on, varying sparsity of 3 orders of magnitude. Speedup trend seems correlated to smaller feature size and denser problems. No free lunch! Pure Hogwild can be a better solution for massively sparse problems with an obnoxiously large feature size. Dataset # Features Sparsity (%) Speedup vs Best news20 1,355,191 0.034 0.86x rcv1-v2 276,544 0.028 1.87x rcv1-test 47,236 0.155 2.43x real-sim 20,958 0.245 3.85x w8a 300 3.884 8.97x connect4 126 33.333 5.81x covtype 54 22.121 20.16x Speedup of HogBatch over best alternative solution of Serial, SGD Batch, or Hogwild. Run with all 28 threads on 14 cores.

43 Staleness of Updates For best convergence, we want the update to the model to be based off of the most recent model possible. (As in the case of serial SGD.) Staleness is our measure of a threads’ un-seen model updates between computing and applying it’s own update. T = #Threads, S = #Samples per SGDBatch, HS = #Samples per HogBatch Min Stale (For Final Sample in Batch) (best case: divide out asynchronicity) Max Stale (worst case) Example T = 8, S = 1024, HS = (S/T) <min, max> Hogwild T-1 < 0, 7> SGD Batch S <1024, 1024> HogBatch HS (T*HS) < 0, 1024>

44 Improvements to Staleness
Improves staleness for batches. Experimentally: up to 30% improvement in convergence per unit time. T = #Threads, S = #Samples per SGDBatch, HS = #Samples per HogBatch Min Stale (For Final Sample in Batch) (subtract staleness of samples per thread) Max Stale Example T = 8, S = 1024, HS = (S/T) <min, max> Hogwild T-1 < 0, 7> SGDBatch (S) - (S/T) (S) - (S/T) <896, 896> HogBatch (HS) – (HS) (T*HS) – (HS) < 0, 896>

45 Improvements to Staleness
for (st = 0; st < Samples/SIZE; st += SIZE) { #pragma omp parallel for for (index = st; index < SIZE; index++) { p_tid += w – a * g(index, [w + p_tid]) } // synchronize #pragma omp parallel for ** for (t = 0; t < numThreads; t++) { model = model + p_tid ** actual reduction done without conflict #pragma omp parallel for schedule(dynamic) for (st = 0; st < Samples/SIZE; st += SIZE) { for (index = st; index < SIZE; index++) { p_tid += w – a * g(index, [w + p_tid]) } model = model + p_tid Reduces staleness by a factor of [size / numThreads] Since each thread aggregates local update.

46 Improvements to Staleness
for (st = 0; st < Samples/SIZE; st += SIZE) { #pragma omp parallel for for (index = st; index < SIZE; index++) { p_tid += w – a * g(index, [w + p_tid]) } // synchronize #pragma omp parallel for ** for (t = 0; t < numThreads; t++) { model = model + p_tid ** actual reduction done without conflict #pragma omp parallel for schedule(dynamic) for (st = 0; st < Samples/SIZE; st += SIZE) { for (index = st; index < SIZE; index++) { p_tid += w – a * g(index, [w + p_tid]) } model = model + p_tid Keep w read only. Write w once at the end of the batch.

47 HogBatch: Note On Batch Size
Mathematically, should be regular mini-batch size divided by number of threads, to achieve same number of “samples in flight”. e.g.: 4 threads working together on a batch of 100, or 4 threads working individually on batches of size 25 each. Due to reduction in average staleness: not necessarily true. In practice we can choose a larger batch size.

48 Parallel scaling of Strategies Sparse Problem
RCV1 test dataset: Frequency Hogwild Better to double frequency than double core count. core-to-core interconnect speedup is needed. Batching Nearly equivalent to double frequency or core count. This is due to private vectors not causing core-to-core traffic. Time (in seconds) to 99.5% optimal. Dataset Name Examples Features NNZ Sparsity (%) NNZ/Row Range Avg NNZ/Row rcv1-test 677,399 47,236 49,556,258 0.155 4 to 1,224 73.157

49 Raw time data RCV1 Covtype


Download ppt "High Performance Parallel Stochastic Gradient Descent"

Similar presentations


Ads by Google