1. Relational Verification to SIMD Loop Synthesis Mark Marron – IMDEA & Microsoft Research Sumit Gulwani – Microsoft Research Gilles Barthe, Juan M. Crespo, Cesar Kunz – IMDEA

2. General SIMD Compilation Compilers struggle to utilize SIMD operations in general purpose code ◦Text processing, web browser, compiler, etc. ◦Standard library code (C++ STL,.Net BCL) they utilize Challenges ◦Data structure layouts of composite types ◦Complex data driven control flow ◦Wide ranging code restructuring is often needed We know most of the needed “tricks” but: ◦Time and implementation effort too large to identify and implement all of them ◦“An Evaluation of Vectorizing Compilers” PACT ‘11

3. Example (Exists Function) struct { int tag; int score; } widget; int exists(widget* vals, int len, int t, int s) { for(int i = 0; i < len; ++i) { int tagok = vals[i].tag == t; int scoreok = vals[i].score > s; int andok = tagok & scoreok; if(andok) return 1; } return 0; }

4. SIMD Example (Exists Function) … for(; i < (len - 3); i += 4) { m128i blck1 = load_128(vals, i); m128i blck2 = load_128(vals, i + 4); m128i tagvs = shuffle_i32(blck1, blck2, ORDER(0, 2, 0, 2)); m128i scorevs = shuffle_i32(blck1, blck2, ORDER(1, 3, 1, 3)); m128i cmptag = cmpeq_i32(vectv, tagvs); m128i cmpscore = cmpgt_i32(vecsv, scorevs); m128i cmpr = and_i128(cmptag, cmpscore); int match = !allzeros(cmpr); if (match) return 1; } … [t i, s i, t i+1, s i+1 ] [t i+2, s i+2, t i+3, s i+3 ] [t i, t i+1, t i+2, t i+3 ] [s i, s i+1, s i+2, s i+3 ] [t i ==t ? 0xF…F : 0x0, …, t i+3 ==t ? 0xF…F : 0x0] [s i >s ? 0xF…F : 0x0, …, s i+3 >s ? 0xF…F : 0x0] [cmptag 0 & cmpscore 0, …, cmptag 3 & cmpscore 3 ] (cmpr 0 !=0 | cmpr 1 !=0 | cmpr 2 !=0 | cmpr 3 !=0)

5. Performance Impact

6. Overview of Approach Deductive Rewriting of program source to: ◦Identify high-level structures of interest ◦Rewrite to expose latent parallelism (split, unroll, etc.) and straighten hot-paths Relational Verification techniques used to: ◦Construct the needed synthesis conditions (for code involving loops!) ◦Produce proof for semantic equivalence of input and result code Inductive Synthesis of SIMD program fragments to: ◦Identify the best SIMD realizations of the synthesis conditions ◦Produce proofs of correctness wrt. synthesis conditions Methodology more general than just SIMD Loops!

7. From Verification to Synthesis Condition Generation Relational Verification: ◦Prove two programs equivalent under equivalence relations on states ◦y = x ◦y = x 1 + x 2 + x 3 + x 4 ◦y = 5 ◦Only a few standard equivalence relations needed in practice Prove results of two programs are equivalent by showing: ◦If the programs are synchronously executed then at synchronization points the program states are always equivalent under the relations ◦For our purposes at the start and end of the loop body

8. Relational Verification int suml = 0; for(int i = 0; i < len; i+=4) { suml = suml + A[i]; suml = suml + A[i+1]; suml = suml + A[i+2]; suml = suml + A[i+3]; } int sumr = 0; int as0, as1, as2, as3 = 0; for(int i = 0; i < len; i+=4) { as0 = as0 + A[i]; as1 = as1 + A[i+1]; as2 = as2 + A[i+2]; as3 = as3 + A[i+3]; } sumr = as0 + as1 + as2 + as3;

9. From Verification to Condition Generation

10. Relational Synthesis Condition int suml = 0; for(int i = 0; i < len; i+=4) { suml = suml + A[i]; suml = suml + A[i+1]; suml = suml + A[i+2]; suml = suml + A[i+3]; }

11. Resulting Synthesis Condition Pre-condition: ◦ac == [v 1, v 2, v 3, v 4 ] Post-condtion: ◦ac == [v 1 + A[i], v 2 + A[i+1], v 3 + A[i+2], v 4 + A[i+3]]

12. Instruction Sequence Search Search space for SIMD instruction sequences is large ◦Length: frequently need 8 or more instructions ◦Branching: SSE has 200+ instructions Concrete state space exploration ◦Explore program states instead of instruction sequences ◦Use concrete execution to quickly exclude many candidate instruction sequences Query SMT solver for a counter example input ◦Eventually either no counter examples or give up Search for alternative sequences ◦Can generate multiple solutions to find best performance on varying data sizes

13. Optimize Search Cost model provides upper bound on depth of search ◦Also used to pick best operation to explore next and to pick shortest path from input to output state Incrementally expand available instruction set ◦Start with standard operations (and those seen in input code) ◦Add more specialized operations if desired Generate multiple initial input-output pairs ◦One per path in original loop body Stack machine construction to reduce the branching factor

14. Cost Model Do not want to compute absolute costs ◦A very hard problem Compute relative costs ◦Both programs run on the same data so same cache misses and branch taken/not taken ◦Build simple machine model to encapsulate instruction costs Cost function a polynomial in terms of loop counts and branch rates ◦Use conservative static estimates for synthesis ◦Can use runtime data for selection in JIT setting

15. Complete Algorithm Synthesize Synth. Cond. Generation Cost Ranking Function Correctness Proof Body Synthesis Cond. Optimistic Vectorize Restructure Loop Merge & Cleanup Cost Score Simulation Relation (Eq) CPU Model

16. SIMD Standard Library Synthesize SIMD implementations of C++ STL and.Net BCL code Consistent performance improvements ◦Between 2x-4x on large inputs ◦Avoid performance degradation on small inputs Cost model accurately predicts performance ◦Can pick best implementation based on hardware and input data

17. Library Function Performance

18. String Processing Synthesize standard string functions using PCMPESTRI ◦Packed Compare Explicit Length Strings, Return Index Encoded semantics and provided them to synthesizer ◦Synthesized range of common string functions with no other changes ◦Speedup of 3.4x for String.Equals ◦Speedup up to 9.5x for String.IndexOfAny

19. Impact In Practice 483.Xalan (SPEC CPU) XML processing framework written in C++ Replaced STL calls with our SIMD implementations Performance sensitive to input data ◦Previous work replacing these calls with set structures was +15% to -20% on different data Synthesized SIMD code produces consistent 2%-5% speedup ◦Indicates a 1.15x to 1.5x speedup in the STL code which is inline with cost model predictions

20. Benefits of Approach Proof of correctness from original loop and SIMD version Separation of correctness and optimization ◦Transform for performant code structure ◦If incorrect proof (or synthesis) will fail later Approach consistently produces fast SIMD code ◦Robust to details of SIMD instruction set and loop patterns ◦2x-4x speedups obtained from synthesized SIMD code

21. Future Work Pointers and object structures ◦Scatter-Gather support will help ◦Compact object graphs into arrays (current work) ◦Can we do local data structure transformations? Apply technique to larger structures and more generally ◦What about loops with small inner-loops (HashTable lookup)? ◦Can we use synthesis as part of general code-gen?

22. Big Picture Conclusions Big challenges and big benefits using specialized hardware ◦Both performance and power! Synthesis complements compilation ◦Small step vs. big step code generation ◦Verification structures synthesis (and eliminates compilation bugs) ◦Can we apply ideas to other compiler actions? Target other hardware? Idea more general than just compilers or SIMD synthesis ◦Expert provided deductive structure ◦Inductive synthesis driven by underlying semantics ◦A powerful combination for approaching problems

23. Questions