1. Prof. Onur Mutlu Carnegie Mellon University Spring 2012, 3/19/2012
18-447: Computer Architecture Lecture 15: Approaches to Concurrency (OoO, DataFlow, Vector, VLIW)
Prof. Onur Mutlu
Carnegie Mellon University
Spring 2012, 3/19/2012

2. Reminder: Homeworks Homework 4 Homework 5 Due Today Out today
Due April 2
Topics: Out-of-order execution, dataflow, vector processing, memory, caches

3. Reminder: Lab Assignments
Due this Friday (March 23)
Implementing control flow and branch prediction in your pipelined simulator
You can get extra credit by optimizing your design to be the fastest
Just make sure you have one of the Top 3 lowest execution times
Lab Assignment 5
Will be out Wednesday
Due April 6

4. Don’t Forget: Feedback Sheet for Us
You can still turn this in
Until the end of this weekend (March 25)

5. Midterm I Grades Average: 175 / 335 (+40 bonus)
Median: 170 / 335 (+40 bonus)
Minimum: 98 / 335 (+40 bonus)
Maximum: 281 / 335 (+40 bonus)

6. Midterm Solutions Posted online
Check and make sure you understand everything
Remember: this is just the first midterm; there is one more
And then the final
TAs will cover all solutions during discussion sessions this week and early next week

7. Midterm I: Per-Question Statistics

8. Midterm I: Per-Question Statistics

9. Homework 3 Grades Average 91.46 Median 95.5 Max 104 Min 57
Max Possible Points
105
Total number of students 49

10. Lab 3 Grades Average 185.5 Median 190 Max 248 Min 90
Max Possible Points (w/o EC)
215
Total number of students 49

11. Prizes for Lab 3 Competition
8 students had fully-correct designs
4 students shared the “fastest” design
Earned 10% extra credit
Top 3 will soon earn a book each
Student
Cycles
Cycle Time (ns)
Execution Time (ns)
Ranking
Relative Exec. Time
rcai
29997
10.554
316588
1st Place 1
jasonli1
28693
11.109
318750
2nd Place
1.01
thuberty
28303
11.687
330777
3rd Place
1.04
jwloh
31151
10.73
334250
4th Place
1.06
twomack
28653
18.705
535954
5th Place
1.69
dhavals
29045
18.653
541776
6th Place
1.71
dliang
31327
27.632
865627
7th Place
2.73
pmpatel
57819
20.074
8th Place
3.67

12. Prizes for Lab 3 Competition
Rui Cai
Jason Lin
Tyler Huberty

13. Readings for Next Time Cache chapters from P&H: 5.1-5.3
Memory/cache chapters from Hamacher+:
First cache paper by Maurice Wilkes, 1965
Wilkes, “Slave Memories and Dynamic Storage Allocation,” IEEE Trans. On Electronic Computers, 1965.

14. Review of Last Lecture Precise exceptions Out-of-order execution
Reorder buffer
Future file
In-order pipeline with precise exceptions
Out-of-order execution
Tomasulo’s algorithm

15. Today
Wrap up out-of-order execution
Data flow
Vector processing

16. Out-of-Order Execution (Dynamic Instruction Scheduling)

17. Enabling OoO Execution
1. Link the consumer of a value to the producer
Register renaming: Associate a “tag” with each data value
2. Buffer instructions until they are ready
Insert instruction into reservation stations after renaming
3. Keep track of readiness of source values of an instruction
Broadcast the “tag” when the value is produced
Instructions compare their “source tags” to the broadcast tag  if match, source value becomes ready
4. When all source values of an instruction are ready, dispatch the instruction to functional unit (FU)
Wakeup and schedule the instruction

18. Review: An Exercise
Assume ADD (4 cycle execute), MUL (6 cycle execute)
Assume one adder and one multiplier
How many cycles
in a non-pipelined machine
in an in-order-dispatch pipelined machine with imprecise exceptions (no forwarding and full forwarding)
in an out-of-order dispatch pipelined machine imprecise exceptions (full forwarding)
MUL R3  R1, R2
ADD R5  R3, R4
ADD R7  R2, R6
ADD R10  R8, R9
MUL R11  R7, R10
ADD R5  R5, R11 F D E W

19. An Exercise, with Precise Exceptions
Assume ADD (4 cycle execute), MUL (6 cycle execute)
Assume one adder and one multiplier
How many cycles
in a non-pipelined machine
in an in-order-dispatch pipelined machine with reorder buffer (no forwarding and full forwarding)
in an out-of-order dispatch pipelined machine with reorder buffer (full forwarding)
MUL R3  R1, R2
ADD R5  R3, R4
ADD R7  R2, R6
ADD R10  R8, R9
MUL R11  R7, R10
ADD R5  R5, R11 F D E R W

20. Out-of-Order Execution with Precise Exceptions
Idea: Use a reorder buffer to reorder instructions before committing them to architectural state
An instruction updates the register alias table (essentially a future file) when it completes execution
An instruction updates the architectural register file when it is the oldest in the machine and has completed execution

21. Out-of-Order Execution with Precise Exceptions
Hump 1: Reservation stations (scheduling window)
Hump 2: Reordering (reorder buffer, aka instruction window or active window)
TAG and VALUE Broadcast Bus S C H E D U L R E O D
Integer add E
Integer mul E F D W
FP mul E E
. . .
Load/store
in order
out of order
in order

22. Summary of OoO Execution Concepts
Renaming eliminates false dependencies
Tag broadcast enables value communication between instructions  dataflow
An out-of-order engine dynamically builds the dataflow graph of a piece of the program
which piece?
Limited to the instruction window
Instruction window: the set of instructions that are currently in the machine, i.e. decoded but not yet retired
Can we do it for the whole program? Why would we like to?
How can we have a large instruction window efficiently?

23. Questions Why is OoO execution beneficial?
What if all operations take single cycle?
Latency tolerance: OoO execution tolerates the latency of multi-cycle operations by executing independent operations concurrently
What if an instruction takes 500 cycles?
How large of an instruction window do we need to continue decoding?
How many cycles of latency can OoO tolerate?
What limits the latency tolerance scalability of Tomasulo’s algorithm?
Active/instruction window size: determined by register file, scheduling window, reorder buffer

24. Food for Thought for You (I)
How can you implement branch prediction in an out-of-order execution machine?
Think about branch history register and PHT updates
Think about recovery from mispredictions
How to do this fast?
How can you combine superscalar execution with out-of-order execution?
These are different concepts
Concurrent renaming of instructions
Concurrent broadcast of tags
How can you combine superscalar + out-of-order + branch prediction?

25. Food for Thought for You (II)
How do you handle load and store execution in an out-of-order execution engine?
Problem: A younger load can have its address ready before an older store’s address is known
Known as the memory disambiguation problem or the unknown address problem
Approaches
Conservative: Stall the load until all previous stores have computed their addresses (or even retired from the machine)
Aggressive: Assume load is independent of unknown-address stores and schedule the load right away
Intelligent: Predict (with a more sophisticated predictor) if the load is dependent on the/any unknown address store

26. Handling of Store-Load Dependencies
A load’s dependence status is not known until all previous store addresses are available.
How does the OOO engine detect dependence of a load instruction on a previous store?
Option 1: Wait until all previous stores committed (no need to check)
Option 2: Keep a list of pending stores in a store buffer and check whether load address matches a previous store address
How does the OOO engine treat the scheduling of a load instruction wrt previous stores?
Option 1: Assume load dependent on all previous stores
Option 2: Assume load independent of all previous stores
Option 3: Predict the dependence of a load on an outstanding store

27. Memory Disambiguation (I)
Option 1: Assume load dependent on all previous stores
+ No need for recovery
-- Too conservative: delays independent loads unnecessarily
Option 2: Assume load independent of all previous stores
+ Simple and can be common case: no delay for independent loads
-- Requires recovery and re-execution of load and dependents on misprediction
Option 3: Predict the dependence of a load on an outstanding store
+ More accurate. Load store dependencies persist over time
-- Still requires recovery/re-execution on misprediction
Alpha : Initially assume load independent, delay loads found to be dependent
Moshovos et al., “Dynamic speculation and synchronization of data dependences,” ISCA 1997.
Chrysos and Emer, “Memory Dependence Prediction Using Store Sets,” ISCA 1998.

28. Memory Disambiguation (II)
Chrysos and Emer, “Memory Dependence Prediction Using Store Sets,” ISCA 1998.
Predicting store-load dependencies important for performance
Simple predictors (based on past history) can achieve most of the potential performance

29. Food for Thought for You (III)
Many other design choices
For example: Should reservation stations be centralized or distributed?
What are the tradeoffs?

30. Recommended Readings
Kessler, “The Alpha Microprocessor,” IEEE Micro, March-April 1999.
Smith and Sohi, “The Microarchitecture of Superscalar Processors,” Proc. IEEE, Dec

31. Data Flow: Exploiting Irregular Parallelism

32. Dataflow Graph for Our Example
In the last lecture
MUL R3  R1, R2
ADD R5  R3, R4
ADD R7  R2, R6
ADD R10  R8, R9
MUL R11  R7, R10
ADD R5  R5, R11

33. State of RAT and RS in Cycle 7

34. Dataflow Graph

35. Restricted Data Flow
An out-of-order machine is a “restricted data flow” machine
Dataflow-based execution is restricted to the microarchitecture level
ISA is still based on von Neumann model (sequential execution)
Remember the data flow model (at the ISA level):
Dataflow model: An instruction is fetched and executed in data flow order
i.e., when its operands are ready
i.e., there is no instruction pointer
Instruction ordering specified by data flow dependence
Each instruction specifies “who” should receive the result
An instruction can “fire” whenever all operands are received

36. Review: More on Data Flow
In a data flow machine, a program consists of data flow nodes
A data flow node fires (fetched and executed) when all it inputs are ready
i.e. when all inputs have tokens
Data flow node and its ISA representation

37. Data Flow Nodes

38. Dataflow Nodes (II)
A small set of dataflow operators can be used to define a general programming language
Fork
Primitive Ops
Switch
Merge + T F T F T T  + T F T F T T

39. Dataflow Graphs a b x y < ip , p , v >
{x = a + b;
y = b * 7 in
(x-y) * (x+y)} a b + *7 - * y x 1 2 3 4 5
Values in dataflow graphs are represented as tokens
An operator executes when all its input tokens are present; copies of the result token are distributed to the destination operators
ip = 3
p = L
token
< ip , p , v >
instruction ptr
port
data
no separate control flow

40. Example Data Flow Program
OUT

41. Control Flow vs. Data Flow

42. Data Flow Characteristics
Data-driven execution of instruction-level graphical code
Nodes are operators
Arcs are data (I/O)
As opposed to control-driven execution
Only real dependencies constrain processing
No sequential I-stream
No program counter
Operations execute asynchronously
Execution triggered by the presence of data

43. A Dataflow Processor

44. MIT Tagged Token Data Flow Architecture
Wait−Match Unit: try to match incoming token and context id and a waiting token with same instruction address
Success: Both tokens forwarded
Fail: Incoming token −−> Waiting Token Mem, bubble (no-op forwarded)

45. TTDA Data Flow Example

46. TTDA Data Flow Example

47. TTDA Data Flow Example

48. Manchester Data Flow Machine
Matching Store: Pairs together tokens destined for the same instruction
Large data set  overflow in overflow unit
Paired tokens fetch the appropriate instruction from the node store

49. Data Flow Advantages/Disadvantages
Very good at exploiting irregular parallelism
Only real dependencies constrain processing
Disadvantages
No precise state
Interrupt/exception handling is difficult
Debugging very difficult (no precise state)
Bookkeeping overhead (tag matching)
Too much parallelism? (Parallelism control needed)
Overflow of tag matching tables
Implementing dynamic data structures difficult

50. Data Flow Summary Availability of data determines order of execution
A data flow node fires when its sources are ready
Programs represented as data flow graphs (of nodes)
Data Flow at the ISA level has not been (as) successful
Data Flow implementations under the hood (while preserving sequential ISA semantics) have been very successful
Out of order execution
Hwu and Patt, “HPSm, a high performance restricted data flow architecture having minimal functionality,” ISCA 1986.

51. Further Reading on Data Flow
Gurd et al., “The Manchester prototype dataflow computer,” CACM 1985.

52. Vector Processing: Exploiting Regular (Data) Parallelism

53. Flynn’s Taxonomy of Computers
Mike Flynn, “Very High-Speed Computing Systems,” Proc. of IEEE, 1966
SISD: Single instruction operates on single data element
SIMD: Single instruction operates on multiple data elements
Array processor
Vector processor
MISD: Multiple instructions operate on single data element
Closest form: systolic array processor, streaming processor
MIMD: Multiple instructions operate on multiple data elements (multiple instruction streams)
Multiprocessor
Multithreaded processor

54. Data Parallelism
Concurrency arises from performing the same operations on different pieces of data
Single instruction multiple data (SIMD)
E.g., dot product of two vectors
Contrast with thread (“control”) parallelism
Concurrency arises from executing different threads of control in parallel
Contrast with data flow
Concurrency arises from executing different operations in parallel (in a data driven manner)
SIMD exploits instruction-level parallelism
Multiple instructions concurrent: instructions happen to be the same

55. SIMD Processing Single instruction operates on multiple data elements
In time or in space
Multiple processing elements
Time-space duality
Array processor: Instruction operates on multiple data elements at the same time
Vector processor: Instruction operates on multiple data elements in consecutive time steps

56. SIMD Array Processing vs. VLIW

57. SIMD Array Processing vs. VLIW
Array processor

58. Vector Processors A vector is a one-dimensional array of numbers
Many scientific/commercial programs use vectors
for (i = 0; i<=49; i++)
C[i] = (A[i] + B[i]) / 2
A vector processor is one whose instructions operate on vectors rather than scalar (single data) values
Basic requirements
Need to load/store vectors  vector registers (contain vectors)
Need to operate on vectors of different lengths  vector length register (VLEN)
Elements of a vector might be stored apart from each other in memory  vector stride register (VSTR)
Stride: distance between two elements of a vector

59. We did not cover the following slides in lecture
We did not cover the following slides in lecture. These are for your preparation for the next lecture.

60. Vector Processors (II)
A vector instruction performs an operation on each element in consecutive cycles
Vector functional units are pipelined
Each pipeline stage operates on a different data element
Vector instructions allow deeper pipelines
No intra-vector dependencies  no hardware interlocking within a vector
No control flow within a vector
Known stride allows prefetching of vectors into cache/memory

61. Vector Processor Advantages
+ No dependencies within a vector
Pipelining, parallelization work well
Can have very deep pipelines, no dependencies!
+ Each instruction generates a lot of work
Reduces instruction fetch bandwidth
+ Highly regular memory access pattern
Interleaving multiple banks for higher memory bandwidth
Prefetching
+ No need to explicitly code loops
Fewer branches in the instruction sequence

62. Vector Processor Disadvantages
-- Works (only) if parallelism is regular (data/SIMD parallelism) ++ Vector operations -- Very inefficient if parallelism is irregular -- How about searching for a key in a linked list?
Fisher, “Very Long Instruction Word architectures and the ELI-512,” ISCA 1983.

63. Vector Processor Limitations
-- Memory (bandwidth) can easily become a bottleneck, especially if 1. compute/memory operation balance is not maintained 2. data is not mapped appropriately to memory banks

64. Vector Registers Each vector data register holds N M-bit values
Vector control registers: VLEN, VSTR, VMASK
Vector Mask Register (VMASK)
Indicates which elements of vector to operate on
Set by vector test instructions
e.g., VMASK[i] = (Vk[i] == 0)
Maximum VLEN can be N
Maximum number of elements stored in a vector register
M-bit wide
M-bit wide
V0,0
V1,0
V0,1
V1,1
V0,N-1
V1,N-1

65. Vector Functional Units
Use deep pipeline (=> fast clock) to execute element operations
Simplifies control of deep pipeline because elements in vector are independent V1 V2 V3
Six stage multiply pipeline
V3 <- v1 * v2
Slide credit: Krste Asanovic

66. Vector Machine Organization (CRAY-1)
Russell, “The CRAY-1 computer system,” CACM 1978.
Scalar and vector modes
8 64-element vector registers
64 bits per element
16 memory banks
8 64-bit scalar registers
8 24-bit address registers