1. EECS 583 – Class 17 Research Topic 1 Decoupled Software Pipelining
University of Michigan
November 12, 2012

2. Announcements + Reading Material
1st paper review due today
Should have submitted to andrew.eecs.umich.edu:/y/submit
Next Monday – Midterm exam in class
See room assignments
Today’s class reading
“Automatic Thread Extraction with Decoupled Software Pipelining,” G. Ottoni, R. Rangan, A. Stoler, and D. I. August, Proceedings of the 38th IEEE/ACM International Symposium on Microarchitecture, Nov
Next class
“Spice: Speculative Parallel Iteration Chunk Execution,” E. Raman, N. Vachharajani, R. Rangan, and D. I. August, Proc 2008 Intl. Symposium on Code Generation and Optimization, April 2008.

3. Midterm Exam When: Monday, Nov 19, 2012, 10:40-12:30 Where
1003 EECS
Uniquenames (not first or last names!) starting with A-E
1005 EECS
Uniquenames starting with F-N
3150 Dow (our classroom)
Uniquenames starting with P-Z
What to expect
Open book/notes, no laptops
Apply techniques we discussed in class on examples
Reason about solving compiler problems – why things are done
A couple of thinking problems
No LLVM code
Reasonably long but you should finish

4. Midterm Exam Last 2 years exams are posted on the course website
Note – Past exams may not accurately predict future exams!!
Office hours between now and Monday if you have questions
James: Tue, Thu, Fri: 1-3pm
Scott: Tue 4:30-5:30pm, Fri 5-6pm (extra slot)
Studying
Yes, you should study even though its open notes
Lots of material that you have likely forgotten from early this semester
Refresh your memories
No memorization required, but you need to be familiar with the material to finish the exam
Go through lecture notes, especially the examples!
If you are confused on a topic, go through the reading
If still confused, come talk to me or James
Go through the practice exams as the final step

5. Exam Topics Control flow analysis Predicated execution
Control flow graphs, Dom/pdom, Loop detection
Trace selection, superblocks
Predicated execution
Control dependence analysis, if-conversion, hyperblocks
Can ignore control height reduction
Dataflow analysis
Liveness, reaching defs, DU/UD chains, available defs/exprs
Static single assignment
Optimizations
Classical: Dead code elim, constant/copy prop, CSE, LICM, induction variable strength reduction
ILP optimizations - unrolling, renaming, tree height reduction, induction/accumulator expansion
Speculative optimization – like HW 1

6. Exam Topics - Continued
Acyclic scheduling
Dependence graphs, Estart/Lstart/Slack, list scheduling
Code motion across branches, speculation, exceptions
Can ignore sentinel scheduling
Software pipelining
DSA form, ResMII, RecMII, modulo scheduling
Make sure you can modulo schedule a loop!
Execution control with LC, ESC
Register allocation
Live ranges, graph coloring
Research topics
Can ignore automatic parallelization

7. Last Class Scientific codes – Successful parallelization C programs
KAP, SUIF, Parascope, gcc w/ Graphite
Affine array dependence analysis
DOALL parallelization
C programs
Not dominated by array accesses – classic parallelization fails
Speculative parallelization – Hydra, Stampede, Speculative multithreading
Profiling to identify statistical DOALL loops
But not all loops DOALL, outer loops typically not!!
This class – Parallelizing loops with dependences

8. What About Non-Scientific Codes???
Scientific Codes (FORTRAN-like)
General-purpose Codes (legacy C/C++)
for(i=1; i<=N; i++) // C
a[i] = a[i] + 1; // X
while(ptr = ptr->next) // LD
ptr->val = ptr->val + 1; // X 1 2 3 4 5
C:1
X:1
C:2
X:2
C:4
X:4
C:3
X:3
C:5
X:5
C:6
X:6
Core 1
Core 2
Core 1
Core 2
Independent
Multithreading
(IMT)
Example: DOALL
parallelization
Cyclic Multithreading
(CMT)
Example: DOACROSS
[Cytron, ICPP 86]
LD:1 1
X:1
LD:2 2
LD:3
X:2 3
X:3
LD:4 4
LD:5
X:4 5
X:5
LD:6 7

9. Alternative Parallelization Approaches
while(ptr = ptr->next) // LD
ptr->val = ptr->val + 1; // X 1 2 3 4 5
LD:1
X:1
LD:2
X:2
LD:4
X:4
LD:3
X:3
LD:5
X:5
LD:6
Core 1
Core 2 1 2 3 4 5
LD:1
LD:2
X:1
X:2
X:3
X:4
LD:3
LD:4
LD:5
LD:6
X:5
Core 1
Core 2
Cyclic
Multithreading
(CMT)
Pipelined
Multithreading (PMT)
Example: DSWP
[PACT 2004] 8

10. Comparison: IMT, PMT, CMT1 2 3 4 5
LD:1
LD:2
X:1
X:2
X:3
X:4
LD:3
LD:4
LD:5
LD:6
X:5
Core 1
Core 2
PMT 1 2 3 4 5
LD:1
X:1
LD:2
X:2
LD:4
X:4
LD:3
X:3
LD:5
X:5
LD:6
Core 1
Core 2
CMT 1 2 3 4 5
C:1
X:1
C:2
X:2
C:4
X:4
C:3
X:3
C:5
X:5
C:6
X:6
Core 1
Core 2 9

11. Comparison: IMT, PMT, CMT1 2 3 4 5
C:1
X:1
C:2
X:2
C:4
X:4
C:3
X:3
C:5
X:5
C:6
X:6
Core 1
Core 2
IMT 1 2 3 4 5
LD:1
LD:2
X:1
X:2
X:3
X:4
LD:3
LD:4
LD:5
LD:6
X:5
Core 1
Core 2
PMT
1 iter/cycle 1 2 3 4 5
LD:1
X:1
LD:2
X:2
LD:4
X:4
LD:3
X:3
LD:5
X:5
LD:6
Core 1
Core 2
CMT
1 iter/cycle
lat(comm) = 1:
1 iter/cycle 10

12. Comparison: IMT, PMT, CMT1 2 3 4 5
C:1
X:1
C:2
X:2
C:4
X:4
C:3
X:3
C:5
X:5
C:6
X:6
Core 1
Core 2
IMT 1 2 3 4 5
LD:1
LD:2
X:1
X:2
X:3
X:4
LD:3
LD:4
LD:5
LD:6
Core 1
Core 2
PMT 1 2 3 4 5
LD:1
X:1
LD:2
X:2
LD:3
X:3
Core 1
Core 2
CMT
lat(comm) = 1:
1 iter/cycle
1 iter/cycle
1 iter/cycle
lat(comm) = 2:
1 iter/cycle
1 iter/cycle
0.5 iter/cycle 11

13. Comparison: IMT, PMT, CMT
Thread-local Recurrences  Fast Execution
IMT
PMT
CMT
Core 1
Core 2
Core 1
Core 2
Core 1
Core 2 1 2 3 4 5
C:1
X:1
C:2
X:2
C:4
X:4
C:3
X:3
C:5
X:5
C:6
X:6
LD:1
LD:1 1 1
LD:2
X:1 2 2
LD:3
X:1
LD:2 3 3
LD:4
X:2
X:2 4 4
LD:5
X:3
LD:3 5 5
LD:6
X:4
X:3
Cross-thread Dependences  Wide Applicability 12

14. Our Objective: Automatic Extraction of Pipeline Parallelism using DSWP
197.parser
This is parser with two input sets shown in green. The red dots are the historic performance trend lines, our goal, shown the first slide. We can automatically parallelize parser in a scalable way - all we need is to have the programmer remove or mark the custom allocator. That is so much easier than rewriting parser in a new language.
Find
English
Sentences
Parse
Sentences
(95%)
Emit
Results
Decoupled Software Pipelining
PS-DSWP (Spec DOALL Middle Stage) 13

15. Decoupled Software Pipelining

16. Inter-thread communication latency is a one-time cost
Decoupled Software Pipelining (DSWP)
[MICRO 2005]
A: while(node)
B: ncost = doit(node);
C: cost += ncost;
D: node = node->next;
Dependence
Graph
DAGSCC
A D
Thread 1
Thread 2 D A B A B D B C C C
intra-iteration
loop-carried
register
control
communication queue
Inter-thread communication
latency is a one-time cost 15

17. Implementing DSWP L1: DFG Aux: register control memory intra-iteration
Choose a valid partitioning
Goals: balance load, reduce sync
Transform the code
intra-iteration
loop-carried
register
memory
control

18. Optimization: Node Splitting To Eliminate Cross Thread Control
intra-iteration
loop-carried
register
memory
control

19. Optimization: Node Splitting To Reduce Communication
intra-iteration
loop-carried
register
memory
control

20. Constraint: Strongly Connected Components
Consider:
Solution: DAGSCC
Choose a valid partitioning
Goals: balance load, reduce sync
Transform the code
intra-iteration
loop-carried
register
memory
control
Eliminates pipelined/decoupled property

21. 2 Extensions to the Basic Transformation
Speculation
Break statistically unlikely dependences
Form better-balanced pipelines
Parallel Stages
Execute multiple copies of certain “large” stages
Stages that contain inner loops perfect candidates

22. Why Speculation? A: while(node) B: ncost = doit(node);
C: cost += ncost;
D: node = node->next;
Dependence
Graph
DAGSCC
A D D B A B
intra-iteration
loop-carried
register
control
communication queue C C

23. Why Speculation? A: while(cost < T && node) B: ncost = doit(node);
C: cost += ncost;
D: node = node->next;
Dependence
Graph
DAGSCC
A D D B
A B C D A B
intra-iteration
loop-carried
register
control
communication queue C C
Predictable
Dependences

24. Why Speculation? A: while(cost < T && node) B: ncost = doit(node);
C: cost += ncost;
D: node = node->next;
Dependence
Graph
DAGSCC D D B A B C
intra-iteration
loop-carried
register
control
communication queue C
Predictable
Dependences A

25. Misspeculation Recovery
Execution Paradigm
DAGSCC D B
Misspeculation
detected C
Misspeculation Recovery
Rerun Iteration 4 A
intra-iteration
loop-carried
register
control
communication queue 24

26. Understanding PMT Performance1 2 3 4 5
A:1
A:2
B:1
B:2
B:3
B:4
A:3
A:4
A:5
A:6
B:5
Core 1
Core 2 1 2 3 4 5
A:1
B:1
C:1
C:3
A:2
B:2
A:3
B:3
Core 1
Core 2
Idle Time
Rate ti is at least as large as the longest dependence recurrence.
NP-hard to find longest recurrence.
Large loops make problem difficult in practice.
Slowest thread:
1 cycle/iter
2 cycle/iter
Iteration Rate:
1 iter/cycle
0.5 iter/cycle

27. Selecting Dependences To Speculate
A: while(cost < T && node)
B: ncost = doit(node);
C: cost += ncost;
D: node = node->next;
Dependence
Graph
DAGSCC D D
Thread 1 B
Thread 2 A B C
Thread 3
intra-iteration
loop-carried
register
control
communication queue C A
Thread 4

28. Detecting Misspeculation
A1: while(consume(4))
D : node = node->next
produce({0,1},node);
Thread 1
A2: while(consume(5))
B : ncost = doit(node);
produce(2,ncost);
D2: node = consume(0);
Thread 2
DAGSCC D B
A3: while(consume(6))
B3: ncost = consume(2);
C : cost += ncost;
produce(3,cost);
Thread 3 C A
A : while(cost < T && node)
B4: cost = consume(3);
C4: node = consume(1);
produce({4,5,6},cost < T
&& node);
Thread 4

29. Detecting Misspeculation
A1: while(TRUE)
D : node = node->next
produce({0,1},node);
Thread 1
A2: while(TRUE)
B : ncost = doit(node);
produce(2,ncost);
D2: node = consume(0);
Thread 2
DAGSCC D B
A3: while(TRUE)
B3: ncost = consume(2);
C : cost += ncost;
produce(3,cost);
Thread 3 C A
A : while(cost < T && node)
B4: cost = consume(3);
C4: node = consume(1);
produce({4,5,6},cost < T
&& node);
Thread 4

30. Detecting Misspeculation
A1: while(TRUE)
D : node = node->next
produce({0,1},node);
Thread 1
A2: while(TRUE)
B : ncost = doit(node);
produce(2,ncost);
D2: node = consume(0);
Thread 2
DAGSCC D B
A3: while(TRUE)
B3: ncost = consume(2);
C : cost += ncost;
produce(3,cost);
Thread 3 C A
A : while(cost < T && node)
B4: cost = consume(3);
C4: node = consume(1);
if(!(cost < T && node))
FLAG_MISSPEC();
Thread 4

31. Breaking False Memory Dependences
Oldest Version
Committed by
Recovery Thread
Dependence
Graph D A B
Memory
Version 3 C
false memory
intra-iteration
loop-carried
register
control
communication queue

32. Adding Parallel Stages to DSWP
Core 1
Core 2
Core 3
LD:1 1
LD:2
while(ptr = ptr->next) // LD
ptr->val = ptr->val + 1; // X 2
LD:3
X:1 3
LD:4
X:2
LD = 1 cycle
X = 2 cycles
The schedule here shows how PS-DSWP would operate when the latency of X and the inter-core communication latencies are both 2 cycles. PS-DSWP replicaes the stage containing X into two cores. Thus, two Xs from two different iterations can be executed concurrently in two different cores. This helps PS-DSWP to achieve a throughput of 1 iteration per cycle. This is in contrast to the ½ iteration per cycle throughput provided by DOACROSS and DSWP.
While in the example, we have shown X as a single statement with a ltency of 2, X could be a stage with varying latency. For instance, X could be an inner loop with varying number of iterations which could not be partitioned by DSWP if it is a single recurrence. As we will see later, one of the key observations that enable PS-DSWP to perform well is that a recurrence in a loop does not necessarily imply the presence of loop carried dependences. 4
LD:5
X:3
Comm. Latency = 2 cycles 5
LD:6
X:4
Throughput
DSWP: 1/2 iteration/cycle
DOACROSS: 1/2 iteration/cycle
PS-DSWP: 1 iteration/cycle 6
LD:7
X:5 7
LD:8 31

33. Thread Partitioning p = list; sum = 0; A: while (p != NULL) {
B: id = p->id;
E: q = p->inner_list;
C: if (!visited[id]) {
D: visited[id] = true;
F: while (foo(q))‏
G: q = q->next;
H: if (q != NULL)‏
I: sum += p->value; }
J: p = p->next; 10 10 10 10 5
PS-DSWP works on the program dependence graph representation the loop. I
n the figure, we can see the PDG of the loop on the left with the nodes of the PDG r
epresenting statementrs and the edges representing dependence between statements.
The actual code in the loop is not important for the purpose of our discussion, except
to note that it contains complex control flow and ireegular memory make accesses making
it hard for many techniques to parallelize them.
The nodes in the PDG are annotated with profile weights which represent the number
of times the corresponding statement is executed. The edges too have some annotations.
The self edge in node I is annotated as a reduction edge since sum reduction or
accumulator expansion can be applied to statement I.
Once the PDG is constructed, the nodes are divided into a set of strongly connected
components. The three non-trivial SCCs are highlighted in the figure 5 50 50 5
Reduction 3 32

34. Thread Partitioning: DAGSCC
The next step is to construct the DAG_SCC. As the name indicates, it is a DAG. The nodes represent SCCs in the original PDG. 33

35. Thread Partitioning Merging Invariants No cycles20
Merging Invariants
No cycles
No loop-carried dependence inside a doall node 10 15 5
The nodes in the DAG_SCC are annotated as DOALL or sequential nodes. A node in the DAG_SCC is a sequential node if the corresponding SCC nodes are connected by a loop carried dependence., unless that loop carried edge is marked as a reduction edge. All the other nodes are labeled as doall. A couple of interesting things to note in this graph. Node I is labeled doall even though it has a self arc, because that self arc is a reduction arc. Node FG, which corresponds to the entire inner loop is a doall.. The execution latency of this loop could be arbitrarily high depending on its trip count, but it could not be partitioned into multiple cores since it is a single SCC. The key observation here is that what matters while replicating a stage is whether it has any loop carried deps wrt the loop being parallelized, which is the outer loop in this case.
At this point, each node in the DAG_SCC can be made into a pipeline stage and if it is a doall stage, it could be replicated. But this would result in more stages than the threads available. So we want to merge nodes together till there are just a few nodes such that each node could be made into a stage. In our current partitioning algorithm, the goal is to get one large doall stage and a minimal number of sequential stages. This works well in many cases where the loop is almost a doall except for a small portion which is serialized. However, there are cases where this partitioning does not do well, and we are developing a new heuristic.
Since we want a large doall stage, we adapt a greedy strategy to merge the doall nodes first. The merging of doall nodes must respect two invariants. First, no cycles should be formed in the graph as a result of merging. Second, two doall nodes can not be merged together if they are connected by a loop carried dependence. We adapt a greedy strategy in doing the merge. We start with the largest doall node, FG in this case, and merge it with a neighbor with a large weight, E in this case. Note that this greedy strategy prevents nodes B and E to be merged which results in a slightly poor partitioning as would be seen later. Subsequently we merge nodes H and I also
100 5 3 34

36. Thread Partitioning 20 10 Treated as sequential 15 113
Thus we end up with a large doall node EFGHI and another doall node B. Since we are
looking for one single doall stage, we relabel B as a sequential node. If we had merged
E and B together, our single parallel stage would have had a slightly lesser weight. The
next step is to greedily merge sequential nodes taking care that there are no cycles in
the resulting graph.
113 35

37. Thread Partitioning 45
113
This results in a single sequential stage ABCDJ. We allot one thread for the seq stage
and the rest of the threads to the doall stage. From this partitioning, we generate
multi-threaded code by adapting the MTCG algorithm proposed by Ottoni et al.
Modified MTCG[Ottoni, MICRO’05] to generate code from partition 36

38. Discussion Point 1 – Speculation
How do you decide what dependences to speculate?
Look solely at profile data?
How do you ensure enough profile coverage?
What about code structure?
What if you are wrong? Undo speculation decisions at run-time?
How do you manage speculation in a pipeline?
Traditional definition of a transaction is broken
Transaction execution spread out across multiple cores

39. Discussion Point 2 – Pipeline Structure
When is a pipeline a good/bad choice for parallelization?
Is pipelining good or bad for cache performance?
Is DOALL better/worse for cache?
Can a pipeline be adjusted when the number of available cores increases/decreases?