1. Module: Part 2 Dynamic Scheduling in Hardware - Tomasulo’s Algorithm
Spring 2005
© Krishna V. Palem, Weng Fai Wong, and Sudhakar Yalamanchili, Georgia Institute of Technology (slides contributed by Prof. Weng Fai Wong were prepared while visiting, and employed by, Georgia Tech)

2. Algorithms for Out-of-order Issue
Scoreboarding
Tomasulo’s Algorithm
Others
Spring 2005

3. In-Order Issue, Out-of-order Execution, Out-of-order Completion
I-Fetch
Execution Core
Retire
Spring 2005

4. Dynamic Scheduling
Hardware will detect and preserve dependencies (within a limited window of the instruction stream)
Hardware will check for resource availability
Independent instructions will be issued to the correct functional units
Spring 2005

5. Advantages Correctness of execution guaranteed by hardware
Independent of compiler optimizations
Backward compatibility
Software scheduling: different machine configuration necessitate recompilation (or at least rescheduling)
Spring 2005

6. IBM 360/91
Introduced in 1966
Introduced many important architectural innovations
pipelining
parallel functional units
out of order execution
imprecise interrupts
load/store buffers
Can execute programs 10 to 100 faster than its immediate predecessor (IBM 7090)
According to Hennessy and Patterson: “Many of the ideas in the 360/91 faded from use for nearly 25 years before being broadly employed in the 1990s.”
Spring 2005

7. IBM 360 Instruction Format
Known as the RX format
All instructions (except load and stores)are of the format where SOURCE may be a memory operand or a register while the SINK must be a register
SOURCE op SINK  SINK
Spring 2005

8. Tomasulo’s Algorithm
Credited to R.M. Tomasulo who presented it in a paper
Implemented for the floating point unit of the IBM 360/91
Spring 2005

9. IBM 360/91 FPU FP Registers FP Add (2 stage) FP Mul/Div (6 stage)
From Memory
From Instruction Unit 8 7 6 5 4 3 2 1
FP Ops
“Stack”
Load
Buffer 6 5 4 3 2 1 FP
Registers
Operand
Busses
Decoder
Operation Bus 3 2 1
To Memory 2 1
Reservation Stations
FP Add
(2 stage)
FP Mul/Div
(6 stage)
Store
Buffer 3 2 1
Common Data Bus
Spring 2005

10. IBM 360/91 FPU
From Memory
From Instruction Unit 8 7 6 5 4 3 2 1
FP Ops
“Stack”
Load
Buffer 6 5 4 3 2 1 FP
Registers
FP operations are sent by the instruction unit to the FPU into a “stack” (IBM terminology - actually a queue!)
Operand
Busses
Decoder
Operation Bus 3 2 1
To Memory 2 1
Reservation Stations
FP Add
(2 stage)
FP Mul/Div
(6 stage)
Store
Buffer 3 2 1
Common Data Bus
Spring 2005

11. IBM 360/91 FPU Decides if it is an add or a multiply/divide FP
From Memory
From Instruction Unit 8 7 6 5 4 3 2 1
Decides if it is an add or a multiply/divide
FP Ops
“Stack”
Load
Buffer 6 5 4 3 2 1 FP
Registers
Operand
Busses
Decoder
Operation Bus 3 2 1
To Memory 2 1
Reservation Stations
FP Add
(2 stage)
FP Mul/Div
(6 stage)
Store
Buffer 3 2 1
Common Data Bus
Spring 2005

12. IBM 360/91 FPU FP Registers 4 floating point registers FP Add
From Memory
From Instruction Unit 8 7 6 5 4 3 2 1
FP Ops
“Stack”
Load
Buffer 6 5 4 3 2 1 FP
Registers
4 floating point registers
Operand
Busses
Decoder
Operation Bus 3 2 1
To Memory 2 1
Reservation Stations
FP Add
(2 stage)
FP Mul/Div
(6 stage)
Store
Buffer 3 2 1
Common Data Bus
Spring 2005

13. IBM 360/91 FPU
From Memory
From Instruction Unit 8 7 6 5 4 3 2 1
FP Ops
“Stack”
Load
Buffer 6 5 4 3 2 1 FP
Registers
Buffers for load. Each load request that goes out to memory gets a buffer allocated.
Operand
Busses
Decoder
Operation Bus 3 2 1
To Memory 2 1
Reservation Stations
FP Add
(2 stage)
FP Mul/Div
(6 stage)
Store
Buffer 3 2 1
Common Data Bus
Spring 2005

14. IBM 360/91 FPU FP Registers The two floating point functional units.
From Memory
From Instruction Unit 8 7 6 5 4 3 2 1
FP Ops
“Stack”
Load
Buffer 6 5 4 3 2 1 FP
Registers
Operand
Busses
Decoder
The two floating point functional units.
Operation Bus 3 2 1
To Memory 2 1
Reservation Stations
FP Add
(2 stage)
FP Mul/Div
(6 stage)
Store
Buffer 3 2 1
Common Data Bus
Spring 2005

15. IBM 360/91 FPU FP Registers FP Add (2 stage) FP Mul/Div (6 stage)
From Memory
From Instruction Unit 8 7 6 5 4 3 2 1
FP Ops
“Stack”
Load
Buffer 6 5 4 3 2 1 FP
Registers
Operand
Busses
Decoder
Supplies operands to reservation stations. Each operand has a tag.
Operation Bus 3 2 1
To Memory 2 1
Reservation Stations
FP Add
(2 stage)
FP Mul/Div
(6 stage)
Store
Buffer 3 2 1
Common Data Bus
Spring 2005

16. IBM 360/91 FPU
From Memory
From Instruction Unit 8 7 6 5 4 3 2 1
FP Ops
“Stack”
Load
Buffer 6 5 4 3 2 1 FP
Registers
Each reservation station holds the two operands of a operation together with their tags as well as the busy bit (which indicates if the operand is available.)
Operand
Busses
Decoder
Operation Bus 3 2 1
To Memory 2 1
Reservation Stations
FP Add
(2 stage)
FP Mul/Div
(6 stage)
Store
Buffer 3 2 1
Common Data Bus
Spring 2005

17. Tags Each tag identify uniquely either
one of the 5 reservation stations
one of the 6 load buffers
Indicates the “producer” of an operand that is not available from the registers
A zero tag indicates that the operand is immediately available.
Spring 2005

18. Reservation Stations
Each reservation station contains the following fields:
the operation to be performed (also known as a CTRL field in IBM terminology)
the SOURCE
the tag for the SOURCE, together with the busy bit
the SINK
the tag for the SINK, together with the busy bit
Spring 2005

19. Data Structures
LD/SD buffers act as reservations stations for memory units
Instruction execution cannot start until all branches resolved
Reservation stations
Values Op Qj Qk Vj Vk A
Busy
Register
value Qi
Spring 2005

20. IBM 360/91 FPU
From Memory
From Instruction Unit 8 7 6 5 4 3 2 1
FP Ops
“Stack”
All operand transport occurs on the common data bus - only one operand may occupy the bus.
Load
Buffer 6 5 4 3 2 1 FP
Registers
Operand
Busses
Decoder
Operation Bus 3 2 1
To Memory 2 1
Reservation Stations
FP Add
(2 stage)
FP Mul/Div
(6 stage)
Store
Buffer 3 2 1
Common Data Bus
Spring 2005

21. Tomasulo’s Algorithm
Decode an operation at the head of the floating point operation stack
Look for an empty reservation station in the functional unit corresponding to the operation. If none exist, instruction issue stalls until one does exit
Read the source operands from the register file, bringing forward the tags
Spring 2005

22. Tomasulo’s Algorithm - cont’d
Mark the busy bit of the SINK in the register file. Also, the tag will be set to point to the selected reservation station
When the functional unit completes its execution, it will write its result and the corresponding reservation station number back to the register file via the common data bus
Spring 2005

23. Tomasulo’s Algorithm - cont’d
All units will listen to the bus and if it is one of the operands it need,
it will read it in
clear the busy bit
When a functional unit is free, it will examine its reservation stations. The one with both its operands’ busy bit clear will be selected for execution
Spring 2005

24. Flow Dependency Flow dependency is obeyed
The exclusivity and broadcast nature of the common data bus ensures that once an operand is produced, all operations requiring it will be notified
Anti and output dependencies are handled by implicit register renaming
Spring 2005

25. The Essence of Register Renaming
DIV.D F0, F2, F4
ADD.D F6, F0, F8
S.D F6, 0(R1)
SUB.D F8, F10, F14
MUL.D F6, F10, F8
DIV.D F0, F2, F4
ADD.D S, F0, F8
S.D S, 0(R1)
SUB.D T, F10, F14
MUL.D F6, F10, T
WAR
WAW
Renaming is performed by the hardware using additional storage  reservation stations
Equivalently, renaming can be performed by the compiler
Spring 2005

26. The Data Path and Functional Units
Register renaming
A form of “generalized” forwarding
The reservation stations can be viewed as “renaming registers” that are physically distributed among the functional units
Control logic for forwarding results is distributed among the function units
Serialization of broadcast of results enables correct operation
Spring 2005

27. Anti-dependence example
Considering the following anti-dependence
Suppose both S1 and S2 are issued and now reside in two distinct reservation stations, RS1 and RS2 say
Two possibilities:
either the operation producing R0 for S1 (let’s call this S0 and assume that it occupies RS0) has completed
S0 has not completed execution
S1: R0 + R1  R2
S2: R3 + R4  R0
Spring 2005

28. Anti-dependence example - cont’d
If S0 has completed
the value of R0 would be read during the issuing of S1 and would now reside in RS1
even if S2 completed and overwrites R0, there would be no effect
If S0 has not completed
when S0 completes, its result goes straight to RS1
R0 is not written by the value produced by S0 because R0 now points to RS2, not RS0
Spring 2005

29. Anti-dependence example - cont’d
R0 is mapped to two physical registers (one register and one reservation station field) depending on the instruction
Hardware implicit register renaming overcomes anti-dependence
Spring 2005

30. Output-dependence example
Considering the following output dependence
When S1 is issued off the FP op queue, it is assigned a reservation station, RS1 say
Suppose S2 is assigned RS2 when issued
S1: R1 + R2  R0
S2: R3 * R4  R0
Spring 2005

31. Output-dependence example - cont’d
If S1 completes before S2 is issued
the tag of R0 would point to RS1
when S2 completes its execution, it will overwrite R0
If S2 is issued and assigned RS2 before S1 completes its execution, the tag of R0 will point to RS2. We have to consider three sub-cases:
S1 completes before S2: since tag of R0 no longer points to RS1, the result of S1 will not overwrite R0
S2 completes before S1: S2’s result will be written into R0; when S1 completes, since the tag of R0 no longer points to RS1, the result of S1 will not be entered into R0
Spring 2005

32. Output-dependence example - cont’d
S1 and S2 completes at the same time: exclusivity of bus ownership ensures that only one of the two above sub-cases will occur
Renaming of R0 also prevents violation of output dependency
Spring 2005

33. Memory Disambiguation
Detection of RAW dependencies through memory
SD F6, 44(R4)
LD F8, 32(R8)
Loads must be checked with preceding stores (RAW)
Stores must be checked with preceding Loads and Stores (WAW and WAR)
A simple scheme: all effective address calculations are performed in program order
Buffers’ A field stores effective address
Can use forwarding directly to/from load/store buffers
RAW Dependency?
Spring 2005

34. Disadvantages Relies on a global bus - lack scalability
Modern CPUs have many more registers and buffers - tag comparison becomes expensive and this can impact the critical path of instruction processing
Spring 2005

35. Example 1 Functional unit latencies are as follows
FPADD = 3 cycles, FPMULT = 5 cycles, Integer/Branch = 1 cycle, LD/SD = 2 cycles
One of each type functional unit each with a single reservation station
Functional units are pipelined
If an operand is written over the CDB on one cycle, dependent operations execute on the next cycle
Spring 2005

36. Example 1 (cont.) Code Issue Execute Writeback L.D F2, 0(R1) 1-2 3
1-2 3
MUL.D F4, F2, F0 1
4-8 9
L.D F6, 0(R2) 4
5-6 7
ADD.D F6, F4, F6 5
10-12 13
S.D F6, 0(R2) 8
14-15
DADDUI R1, R1, #8 10 11
DADDIU R2, R2, #-8 12 14
BGT R1, #800 15 16
Only one reservation station
Only one reservation station
Spring 2005

37. Example 2
Now consider the status of the reservation stations, load/store buffers, and FP registers
Show the status of the data structures for the following program when the first MUL.D has completed execution but not yet written the results
Spring 2005

38. Example 2
Spring 2005

39. Example 2
Spring 2005

40. Note the potential conflict on the CDB in cycle 8
Example 2
Note the potential conflict on the CDB in cycle 8
Spring 2005

41. Overlapping Loop Iterations
Register renaming
Multiple iterations use different physical destinations for registers (dynamic loop unrolling).
Reservation stations
Permit instruction issue to advance past integer control flow operations
Also buffer old values of registers - totally avoiding the WAR stall that we saw in the scoreboard.
Alternative perspective: Tomasulo building data flow dependency graph on the fly.
Spring 2005

42. Additional Reference
Reference the Tomasulo Example in lecture notes for CS 252, Department of EECS, University of California, Berkeley, taught by Professor David Patterson and available at
Spring 2005

43. Conclusions Three key elements improve performance
Dynamic scheduling
Register renaming
Memory disambiguation
What limits instruction concurrency?
Hardware resources
Control flow
Spring 2005