Statistical Modeling of Feedback Data in an Automatic Tuning System Richard Vuduc, James Demmel (U.C. Berkeley, EECS) Jeff Bilmes (Univ. of Washington, EE) December 10, 2000 Workshop on Feedback-Directed Dynamic Optimization

Context: High Performance Libraries n Libraries can isolate performance issues –BLAS/LAPACK/ScaLAPACK (linear algebra) –VSIPL (signal and image processing) –MPI (distributed parallel communications) n Can we implement libraries … –automatically and portably –incorporating machine-dependent features –matching performance of hand-tuned implementations –leveraging compiler technology –using domain-specific knowledge

Generate and Search: An Automatic Tuning Methodology n Given a library routine n Write parameterized code generators –parameters machine (e.g., registers, cache, pipeline) input (e.g., problem size) problem-specific transformations –output high-level source (e.g., C code) n Search parameter spaces –generate an implementation –compile using native compiler –measure performance (feedback)

Tuning System Examples n Linear algebra –PHiPAC (Bilmes, et al., 1997) –ATLAS (Whaley and Dongarra, 1998) –Sparsity (Im and Yelick, 1999) n Signal Processing –FFTW (Frigo and Johnson, 1998) –SPIRAL (Moura, et al., 2000) n Parallel Communcations –Automatically tuned MPI collective operations (Vadhiyar, et al. 2000) n Related: Iterative compilation (Bodin, et al., 1998)

Road Map n Context n The Search Problem n Problem 1: Stopping searches early n Problem 2: High-level run-time selection n Summary

The Search Problem in PHiPAC n PHiPAC (Bilmes, et al., 1997) –produces dense matrix multiply (matmul) implementations –generator parameters include size and depth of fully unrolled core matmul rectangular, multi-level cache tile sizes 6 flavors of software pipelining scaling constants, transpose options, precisions, etc. n An experiment –fix software pipelining method –vary register tile sizes –500 to 2500 reasonable implementations on 6 platforms

A Needle in a Haystack, Part I

Road Map n Context n The Search Problem n Problem 1: Stopping searches early n Problem 2: High-level run-time selection n Summary

Problem 1: Stopping Searches Early n Assume –dedicated resources limited –near-optimal implementation okay n Recall the search procedure –generate implementations at random –measure performance n Can we stop the search early? –how early is early? –guarantees on quality?

An Early Stopping Criterion n Performance scaled from 0 (worst) to 1 (best) Goal: Stop after t implementations when Prob[ M t <= 1- ] < –M t max observed performance at t – proximity to best – error tolerance –example: find within top 5% with error 10% =.05, =.1 Can show probability depends only on F(x) = Prob[ performance <= x ] Idea: Estimate F(x) using observed samples

Stopping time (300 MHz Pentium-II)

Stopping Time (Cray T3E Node)

Road Map n Context n The Search Problem n Problem 1: Stopping searches early n Problem 2: High-level run-time selection n Summary

Problem 2: Run-Time Selection n Assume –one implementation is not best for all inputs –a few, good implementations known –can benchmark n How do we choose the best implementation at run-time? n Example: matrix multiply, tuned for small (L1), medium (L2), and large workloads CA M K B K N C = C + A*B

Truth Map (Sun Ultra-I/170)

A Formal Framework n Given –m implementations –n sample inputs (training set) –execution time n Find –decision function f(s) –returns best implementation on input s – f(s) cheap to evaluate

Solution Techniques (Overview) n Method 1: Cost Minimization –minimize overall execution time on samples (boundary modeling) pro: intuitive, f(s) cheap con: ad hoc, geometric assumptions n Method 2: Regression (Brewer, 1995) –model run-time of each implementation e.g., T a (N) = b 3 N 3 + b 2 N 2 + b 1 N + b 0 pro: simple, standard con: user must define model n Method 3: Support Vector Machines –statistical classification pro: solid theory, many successful applications con: heavy training and prediction machinery

Results 1: Cost Minimization

Results 2: Regression

Results 3: Classification

Quantitative Comparison Note: Cost of regression and cost-min prediction ~O(3x3 matmul) Cost of SVM prediction ~O(32x32 matmul)

Road Map n Context n The Search Problem n Problem 1: Stopping searches early n Problem 2: High-level run-time selection n Summary

Conclusions n Search beneficial n Early stopping –simple (random + a little bit) –informative criteria n High-level run-time selection –formal framework –error metrics n To do –other stopping models (cost-based) –large design space for run-time selection

Extra Slides More detail (time and/or questions permitting)

PHiPAC Performance (Pentium-II)

PHiPAC Performance (Ultra-I/170)

PHiPAC Performance (IBM RS/6000)

PHiPAC Performance (MIPS R10K)

Needle in a Haystack, Part II

Performance Distribution (IBM RS/6000)

Performance Distribution (Pentium II)

Performance Distribution (Cray T3E Node)

Performance Distribution (Sun Ultra-I)

Cost Minimization n Decision function n Minimize overall execution time on samples n Softmax weight (boundary) functions

Regression n Decision function n Model implementation running time (e.g., square matmul of dimension N) n For general matmul with operand sizes (M, K, N), we generalize the above to include all product terms –MKN, MK, KN, MN, M, K, N

Support Vector Machines n Decision function n Binary classifier

Proximity to Best (300 MHz Pentium-II)

Proximity to Best (Cray T3E Node)