1 IBM System 360. Common architecture for a set of machines. Robert Tomasulo worked on a high-end machine, the Model 91 (1967), on which they implemented his algorithm (today’s topic).

2 COMP 740: Computer Architecture and Implementation Montek Singh Tue, Feb 17, 2009 Topic: Instruction-Level Parallelism (Dynamic Scheduling: Tomasulo’s Algorithm)

3 Today’s Topic  Tomasulo’s algorithm for dynamic scheduling more sophisticated than scoreboarding more sophisticated than scoreboarding will enable scheduling multiple iterations of a loop (but still limited, more to come) will enable scheduling multiple iterations of a loop (but still limited, more to come) Reading: Ch Reading: Ch

4 Dynamic Scheduling: Tomasulo’s Algorithm  For IBM 360/91 (about three years after CDC 6600) long mem latency (before caches!) long mem latency (before caches!)  Goal: High performance without special compilers Small number of floating point registers (4 in 360 architecture) prevented interesting compiler scheduling of operations Small number of floating point registers (4 in 360 architecture) prevented interesting compiler scheduling of operations Led Tomasulo to try to figure out how to get more effective registers — renaming in hardware! Led Tomasulo to try to figure out how to get more effective registers — renaming in hardware!  Why Study 1967 Computer? Technique languished until 1990s, then rediscovered Technique languished until 1990s, then rediscovered The descendants have flourished The descendants have flourished  Alpha 21264, Pentium 4, AMD Opteron, Power 5, …

5 Tomasulo’s Algorithm  Differences between IBM 360 and CDC 6600 ISA IBM has only 2 register specifiers/instruction versus 3 in CDC 6600 IBM has only 2 register specifiers/instruction versus 3 in CDC 6600 IBM has 4 FP registers versus 8 in CDC 6600 IBM has 4 FP registers versus 8 in CDC 6600  Differences between Tomasulo Algorithm and Scoreboard Control and buffers distributed with Function Units versus centralized in scoreboard; called “reservation stations” Control and buffers distributed with Function Units versus centralized in scoreboard; called “reservation stations” Registers in instructions replaced by pointers to reservation stations Registers in instructions replaced by pointers to reservation stations  this is called register renaming  renaming helps avoid WAR and WAW hazards  more reservation stations than registers; so allow optzns compilers can’t do Common Data Bus broadcasts results to all FUs (forwarding!) Common Data Bus broadcasts results to all FUs (forwarding!) Load and Stores treated as FUs as well, with reservation stations for stores Load and Stores treated as FUs as well, with reservation stations for stores

6 Tomasulo: Organization FP adders Add1 Add2 Add3 FP multipliers Mult1 Mult2 From Mem FP Registers Reservation Stations Common Data Bus (CDB) To Mem FP Op Queue Load Buffers Store Buffers Load1 Load2 Load3 Load4 Load5 Load6

7 More Details of Tomasulo Organization  Entities that produce values are assigned 4-bit tags 1, 2, 3, 4, 5, 6 for load buffers 1, 2, 3, 4, 5, 6 for load buffers 8, 9 for multiplier reservation stations 8, 9 for multiplier reservation stations 10, 11, 12 for adder reservation stations 10, 11, 12 for adder reservation stations Tag 0 indicates presence of valid data Tag 0 indicates presence of valid data  FP registers have “busy bits” 0 means that register holds valid data 0 means that register holds valid data 1 means that it is waiting to receive value from source identified by its tag field 1 means that it is waiting to receive value from source identified by its tag field

8 Reservation Station Components Op:Operation to perform (e.g., + or –) Vj, Vk: Value of Source operands Buffer for Store Inst. has one V field, result to be stored Buffer for Store Inst. has one V field, result to be stored Qj, Qk: Reservation stations producing source registers (value to be written) Note: Qj,Qk=0 => ready Note: Qj,Qk=0 => ready Buffer for Store Inst. has single Q field, source of value Buffer for Store Inst. has single Q field, source of value Busy: Indicates reservation station or FU is busy Busy: Indicates reservation station or FU is busy Register result status—Indicates which functional unit will write each register, if one exists. Blank when no pending instructions that will write that register.

9 Tomasulo: Representing Data Dependences  Inputs Operand is a register with busy bit = 0 Operand is a register with busy bit = 0  Data copied immediately (through register bus) into reservation station  Tag field of RS set to 0 Operand is a register with busy bit = 1 Operand is a register with busy bit = 1  Tag field of RS receives a copy of the register tag field Operand is a load buffer that contains valid data Operand is a load buffer that contains valid data  Data copied into RS Operand is a load buffer that is awaiting data Operand is a load buffer that is awaiting data  Tag field of RS receives tag of load buffer  Outputs Output is a register Output is a register  Busy bit set to 1, tag set to RS tag Output is a store buffer Output is a store buffer  Tag set to RS tag, destination address set

10 Three Stages of Tomasulo Algorithm 1. Issue: get instruction from FP operation queue If reservation station free, the scoreboard issues instruction and sends operands (renames registers) If reservation station free, the scoreboard issues instruction and sends operands (renames registers) 2. Execution: operate on operands (EX) When both operands ready then execute; if not ready, watch CDB for result When both operands ready then execute; if not ready, watch CDB for result 3. Write Result: finish execution (WB) Write on Common Data Bus to all awaiting units; mark reservation station available Write on Common Data Bus to all awaiting units; mark reservation station available  Common Data Bus: data + source (“came from”) 64 bits of data + 4 bits of RS source address => broadcast 64 bits of data + 4 bits of RS source address => broadcast Different from “normal” data bus: data + destination (“go to” bus) Different from “normal” data bus: data + destination (“go to” bus)

11 Tomasulo Example Clock cycle counter FU count down Instruction stream 3 Load/Buffers 3 FP Adder R.S. 2 FP Mult R.S.

12 Tomasulo Example Cycle 1

13 Tomasulo Example Cycle 2 Note: Can have multiple loads outstanding

14 Tomasulo Example Cycle 3 Note: registers names are removed (“renamed”) in Reservation Stations; MULT issued Load1 completing; what is waiting for Load1?

15 Tomasulo Example Cycle 4 Load2 completing; what is waiting for Load2?

16 Tomasulo Example Cycle 5 Timer starts down for Add1, Mult1

17 Tomasulo Example Cycle 6 Issue ADDD here despite name dependency on F6?

18 Tomasulo Example Cycle 7 Add1 (SUBD) completing; what is waiting for it?

19 Tomasulo Example Cycle 8

20 Tomasulo Example Cycle 9

21 Tomasulo Example Cycle 10 Add2 (ADDD) completing; what is waiting for it?

22 Tomasulo Example Cycle 11 Write result of ADDD here? All quick instructions complete in this cycle!

23 Tomasulo Example Cycle 12

24 Tomasulo Example Cycle 13

25 Tomasulo Example Cycle 14

26 Tomasulo Example Cycle 15 Mult1 (MULTD) completing; what is waiting for it?

27 Tomasulo Example Cycle 16 Just waiting for Mult2 (DIVD) to complete

28 Skip some cycles…

29 Tomasulo Example Cycle 55

30 Tomasulo Example Cycle 56 Mult2 (DIVD) is completing; what is waiting for it?

31 Tomasulo Example Cycle 57 Once again: In-order issue, out-of-order execution and out-of-order completion.

32 Observations on Tomasulo’s Algorithm  Instructions: move from decoder to reservation stations in program order in program order dependences can be correctly recorded dependences can be correctly recorded  Data Flow Graph: The graph of pointers connecting the RS, registers, and memory buffers helps accomplish out-of-order sequencing of instructions helps accomplish out-of-order sequencing of instructions  Chief cost of this scheme: high-speed associative hardware RS hardware has to search for tags when CDB broadcasts some value with its tag RS hardware has to search for tags when CDB broadcasts some value with its tag  Full load bypassing is supported load and store buffers are treated just like functional units load and store buffers are treated just like functional units additional hardware on 360/91 also supported load forwarding additional hardware on 360/91 also supported load forwarding

33 Tomasulo: Example of Load Bypassing 200: F0  A 201: F0  F0 / F1 202: C  F0 203: F0  D 204: F0  F0 * F2 200: F0  A 201: F0  F0 / F1 202: C  F0 203: F0  D 204: F0  F0 * F2  Instruction 202 depends on instructions 200 and 201, so instruction 203 will start executing much before 202 (assuming C and D are found to be different memory addresses)  Work out details off-line

34 Tomasulo: “Loop Unrolling in Hardware”  360/91 supported limited kind of speculation Small loops could be held in a loop buffer Small loops could be held in a loop buffer Loop closing branches were predicted as taken Loop closing branches were predicted as taken  This has the effect of loop unrolling at run-time Given the small number of FP registers in machine, software loop unrolling was not a viable option Given the small number of FP registers in machine, software loop unrolling was not a viable option  Why exactly can it unroll loops? Register renaming Register renaming  Multiple iterations use different physical destinations for registers (dynamic loop unrolling). Reservation stations Reservation stations  Permit instruction issue to advance past integer control flow operations  Also buffer old values of registers - totally avoiding the WAR stall Other perspective: it’s building data flow dependency graph on the fly Other perspective: it’s building data flow dependency graph on the fly

35 Tomasulo Loop Example Loop: L.DF00R1 MULT.DF4F0F2 S.DF40R1 SUBIR1R1#8 BNEZR1Loop  Multiply takes 4 clocks  Loads slow (no caches!)

36 Loop Example Cycle 0

37 Loop Example Cycle 1

38 Loop Example Cycle 2

39 Loop Example Cycle 3

40 Loop Example Cycle 4

41 Loop Example Cycle 5

42 Loop Example Cycle 6 Load2

43 Loop Example Cycle 7

44 Loop Example Cycle 8

45 Loop Example Cycle 9

46 Loop Example Cycle 10

47 Loop Example Cycle 11

48 Loop Example Cycle 12 Structural hazard – no MULT unit available

49 Loop Example Cycle 13

50 Loop Example Cycle 14

51 Loop Example Cycle 15

52 Loop Example Cycle 16

53 Loop Example Cycle 17

54 Loop Example Cycle 18

55 Loop Example Cycle 19 …

56 Loop Example Cycle 20

57 Loop Example Cycle 21

58 Summary of Tomasulo’s Algorithm  Reservations stations: renaming to larger set of registers + buffering source operands Registers not bottleneck Registers not bottleneck Avoids WAR, WAW hazards of scoreboarding Avoids WAR, WAW hazards of scoreboarding Allows loop unrolling in hardware Allows loop unrolling in hardware  Not limited to basic blocks provided we have branch prediction provided we have branch prediction  Lasting contributions Dynamic scheduling Dynamic scheduling Register renaming Register renaming Load/store disambiguation Load/store disambiguation 360/91 descendants are Intel Pentium 4, IBM Power 5, AMD Athlon/Opteron, … 360/91 descendants are Intel Pentium 4, IBM Power 5, AMD Athlon/Opteron, …  Next: Dynamic branch prediction