1. March 1, 2012CS152, Spring 2012 CS 152 Computer Architecture and Engineering Lecture 12 - Advanced Out-of-Order Superscalars Krste Asanovic Electrical Engineering and Computer Sciences University of California at Berkeley http://www.eecs.berkeley.edu/~krste http://inst.eecs.berkeley.edu/~cs152

2. March 1, 2012CS152, Spring 2012 2 Last time in Lecture 11 Register renaming removes WAR, WAW hazards In-order fetch/decode, out-of-order execute, in-order commit gives high performance and precise exceptions Need to rapidly recover on branch mispredictions

3. March 1, 2012CS152, Spring 2012 3 “Data-in-ROB” Design (HP PA8000, Pentium Pro, Core2Duo, Nehalem) On dispatch into ROB, ready sources can be in regfile or in ROB dest (copied into src1/src2 if ready before dispatch) On completion, write to dest field and broadcast to src fields. On issue, read from ROB src fields Register File holds only committed state Reorder buffer Load Unit FU Store Unit t1t2..tnt1t2..tn Ins# use exec op p1 src1 p2 src2 pd dest data Commit

4. March 1, 2012CS152, Spring 2012 ROB Data Movement in Data-in-ROB Design 4 Architectural Register File Read operands during decode Src Operands Write sources after decode Read operands at issue Functional Units Write results at completion Read results, write to ARF at commit Bypass newer values at decode Result Data

5. March 1, 2012CS152, Spring 2012 Unified Physical Register File (MIPS R10K, Alpha 21264, Intel Pentium 4 & Sandy Bridge) Rename all architectural registers into a single physical register file during decode, no register values read Functional units read and write from single unified register file holding committed and temporary registers in execute Commit only updates mapping of architectural register to physical register, no data movement 5 Unified Physical Register File Read operands at issue Functional Units Write results at completion Committed Register Mapping Decode Stage Register Mapping

6. March 1, 2012CS152, Spring 2012 6 Pipeline Design with Physical Regfile Fetch Decode & Rename Reorder BufferPC Branch Prediction Commit Branch Resolution Branch Unit ALU MEM Store Buffer D$ Execute In-Order Out-of-Order Physical Reg. File kill

7. March 1, 2012CS152, Spring 2012 7 Lifetime of Physical Registers ld x1, (x3) addi x3, x1, #4 sub x6, x7, x9 add x3, x3, x6 ld x6, (x1) add x6, x6, x3 sd x6, (x1) ld x6, (x11) ld P1, (Px) addi P2, P1, #4 sub P3, Py, Pz add P4, P2, P3 ld P5, (P1) add P6, P5, P4 sd P6, (P1) ld P7, (Pw) Rename When can we reuse a physical register? When next write of same architectural register commits Physical regfile holds committed and speculative values Physical registers decoupled from ROB entries (no data in ROB)

8. March 1, 2012CS152, Spring 2012 8 Physical Register Management opp1PR1p2PR2exuseRdPRdLPRd P5 P6 P7 P0 Pn P1 P2 P3 P4 x5 P5 x6 P6 x7 x0 P8 x1 x2 P7 x3 x4 ROB Rename Table Physical Regs Free List ld x1, 0(x3) addi x3, x1, #4 sub x6, x7, x6 add x3, x3, x6 ld x6, 0(x1) p p p P0 P1 P3 P2 P4 (LPRd requires third read port on Rename Table for each instruction) P8 p

9. March 1, 2012CS152, Spring 2012 9 Physical Register Management opp1PR1p2PR2exuseRdPRdLPRd ROB ld x1, 0(x3) addi x3, x1, #4 sub x6, x7, x6 add x3, x3, x6 ld x6, 0(x1) Free List P0 P1 P3 P2 P4 P5 P6 P7 P0 Pn P1 P2 P3 P4 Physical Regs p p p P8 p x ld p P7 x1 P0 x5 P5 x6 P6 x7 x0 P8 x1 x2 P7 x3 x4 Rename Table P0 P8

10. March 1, 2012CS152, Spring 2012 10 Physical Register Management opp1PR1p2PR2exuseRdPRdLPRd ROB ld x1, 0(x3) addi x3, x1, #4 sub x6, x7, x6 add x3, x3, x6 ld x6, 0(x1) Free List P0 P1 P3 P2 P4 P5 P6 P7 P0 Pn P1 P2 P3 P4 Physical Regs p p p P8 p x ld p P7 x1 P0 x5 P5 x6 P6 x7 x0 P8 x1 x2 P7 x3 x4 Rename Table P0 P8 P7 P1 x addi P0 x3 P1

11. March 1, 2012CS152, Spring 2012 11 Physical Register Management opp1PR1p2PR2exuseRdPRdLPRd ROB ld x1, 0(x3) addi x3, x1, #4 sub x6, x7, x6 add x3, x3, x6 ld x6, 0(x1) Free List P0 P1 P3 P2 P4 P5 P6 P7 P0 Pn P1 P2 P3 P4 Physical Regs p p p P8 p x ld p P7 x1 P0 x5 P5 x6 P6 x7 x0 P8 x1 x2 P7 x3 x4 Rename Table P0 P8 P7 P1 x addi P0 x3 P1 P5 P3 x sub p P6 p P5 x6 P3

12. March 1, 2012CS152, Spring 2012 12 Physical Register Management opp1PR1p2PR2exuseRdPRdLPRd ROB ld x1, 0(x3) addi x3, x1, #4 sub x6, x7, x6 add x3, x3, x6 ld x6, 0(x1) Free List P0 P1 P3 P2 P4 P5 P6 P7 P0 Pn P1 P2 P3 P4 Physical Regs p p p P8 p x ld p P7 x1 P0 x5 P5 x6 P6 x7 x0 P8 x1 x2 P7 x3 x4 Rename Table P0 P8 P7 P1 x addi P0 x3 P1 P5 P3 x sub p P6 p P5 x6 P3 P1 P2 x add P1 P3 x3 P2

13. March 1, 2012CS152, Spring 2012 13 Physical Register Management opp1PR1p2PR2exuseRdPRdLPRd ROB ld x1, 0(x3) addi x3, x1, #4 sub x6, x7, x6 add x3, x3, x6 ld x6, 0(x1) Free List P0 P1 P3 P2 P4 P5 P6 P7 P0 Pn P1 P2 P3 P4 Physical Regs p p p P8 p x ld p P7 x1 P0 x5 P5 x6 P6 x7 x0 P8 x1 x2 P7 x3 x4 Rename Table P0 P8 P7 P1 x addi P0 x3 P1 P5 P3 x sub p P6 p P5 x6 P3 P1 P2 x add P1 P3 x3 P2 x ld P0 x6 P4P3 P4

14. March 1, 2012CS152, Spring 2012 14 opp1PR1p2PR2exuseRdPRdLPRd ROB x ld p P7 x1 P0 x addi P0 x3 P1 x sub p P6 p P5 x6 P3 x ld p P7 x1 P0 Physical Register Management ld x1, 0(x3) addi x3, x1, #4 sub x6, x7, x6 add x3, x3, x6 ld x6, 0(x1) Free List P0 P1 P3 P2 P4 P5 P6 P7 P0 Pn P1 P2 P3 P4 Physical Regs p p p P8 p x5 P5 x6 P6 x7 x0 P8 x1 x2 P7 x3 x4 Rename Table P0 P8 P7 P1 P5 P3 P1 P2 x add P1 P3 x3 P2 x ld P0 x6 P4P3 P4 Execute & Commit p p p P8 x

15. March 1, 2012CS152, Spring 2012 15 opp1PR1p2PR2exuseRdPRdLPRd ROB x sub p P6 p P5 x6 P3 x addi P0 x3 P1 Physical Register Management ld x1, 0(x3) addi x3, x1, #4 sub x6, x7, x6 add x3, x3, x6 ld x6, 0(x1) Free List P0 P1 P3 P2 P4 P5 P6 P7 P0 Pn P1 P2 P3 P4 Physical Regs p p p P8 x x ld p P7 x1 P0 x5 P5 x6 P6 x7 x0 P8 x1 x2 P7 x3 x4 Rename Table P0 P8 P7 P1 P5 P3 P1 P2 x add P1 P3 x3 P2 x ld P0 x6 P4P3 P4 Execute & Commit p p p P8 x p p P7

16. March 1, 2012CS152, Spring 2012 Separate Pending Instruction Window from ROB 16 Reorder buffer used to hold exception information for commit. The instruction window holds instructions that have been decoded and renamed but not issued into execution. Has register tags and presence bits, and pointer to ROB entry. opp1PR1p2PR2PRduseexROB# ROB is usually several times larger than instruction window – why? RdLPRdPCExcept? Ptr 2 next to commit Ptr 1 next available Done?

17. March 1, 2012CS152, Spring 2012 17 Reorder Buffer Holds Active Instructions (Decoded but not Committed) … ld x1, (x3) add x3, x1, x2 sub x6, x7, x9 add x3, x3, x6 ld x6, (x1) add x6, x6, x3 sd x6, (x1) ld x6, (x1) … (Older instructions) (Newer instructions) Cycle t … ld x1, (x3) add x3, x1, x2 sub x6, x7, x9 add x3, x3, x6 ld x6, (x1) add x6, x6, x3 sd x6, (x1) ld x6, (x1) … Commit Fetch Cycle t + 1 Execute

18. March 1, 2012CS152, Spring 2012 18 Superscalar Register Renaming During decode, instructions allocated new physical destination register Source operands renamed to physical register with newest value Execution unit only sees physical register numbers Rename Table OpSrc1Src2DestOpSrc1Src2Dest Register Free List OpPSrc1PSrc2PDestOpPSrc1PSrc2PDest Update Mapping Does this work? Inst 1Inst 2 Read Addresses Read Data Write Ports

19. March 1, 2012CS152, Spring 2012 19 Superscalar Register Renaming Rename Table OpSrc1Src2DestOpSrc1Src2Dest Register Free List OpPSrc1PSrc2PDestOpPSrc1PSrc2PDest Update Mapping Inst 1 Inst 2 Read Addresses Read Data Write Ports =? Must check for RAW hazards between instructions issuing in same cycle. Can be done in parallel with rename lookup. MIPS R10K renames 4 serially-RAW-dependent insts/cycle

20. March 1, 2012CS152, Spring 2012 20 CS152 Administrivia Quiz 2, Tuesday March 6 –Covers lectures 6-9, PS 2, Lab 2, readings Complete Undergrad EECS town hall survey at: –https://www.surveymonkey.com/s/HWGPSRThttps://www.surveymonkey.com/s/HWGPSRT –Town hall overlaps class on March 15 (1:30-3:30) 

21. March 1, 2012CS152, Spring 2012 21 Memory Dependencies sd x1, (x2) ld x3, (x4) When can we execute the load?

22. March 1, 2012CS152, Spring 2012 22 In-Order Memory Queue Execute all loads and stores in program order => Load and store cannot leave ROB for execution until all previous loads and stores have completed execution Can still execute loads and stores speculatively, and out-of-order with respect to other instructions Need a structure to handle memory ordering…

23. March 1, 2012CS152, Spring 2012 23 Conservative O-o-O Load Execution sd x1, (x2) ld x3, (x4) Split execution of store instruction into two phases: address calculation and data write Can execute load before store, if addresses known and x4 != x2 Each load address compared with addresses of all previous uncommitted stores –can use partial conservative check i.e., bottom 12 bits of address, to save hardware Don’t execute load if any previous store address not known (MIPS R10K, 16-entry address queue)

24. March 1, 2012CS152, Spring 2012 24 Address Speculation Guess that x4 != x2 Execute load before store address known Need to hold all completed but uncommitted load/store addresses in program order If subsequently find x4==x2, squash load and all following instructions => Large penalty for inaccurate address speculation sd x1, (x2) ld x3, (x4)

25. March 1, 2012CS152, Spring 2012 25 Memory Dependence Prediction (Alpha 21264) sd x1, (x2) ld x3, (x4) Guess that x4 != x2 and execute load before store If later find x4==x2, squash load and all following instructions, but mark load instruction as store-wait Subsequent executions of the same load instruction will wait for all previous stores to complete Periodically clear store-wait bits

26. March 1, 2012CS152, Spring 2012 26 Speculative Loads / Stores Just like register updates, stores should not modify the memory until after the instruction is committed - A speculative store buffer is a structure introduced to hold speculative store data.

27. March 1, 2012CS152, Spring 2012 27 Speculative Store Buffer On store execute: –mark entry valid and speculative, and save data and tag of instruction. On store commit: –clear speculative bit and eventually move data to cache On store abort: – clear valid bit Data Load Address Tags Store Commit Path Speculative Store Buffer L1 Data Cache Load Data TagDataSVTagDataSVTagDataSVTagDataSVTagDataSVTagDataSV

28. March 1, 2012CS152, Spring 2012 28 Speculative Store Buffer If data in both store buffer and cache, which should we use? Speculative store buffer If same address in store buffer twice, which should we use? Youngest store older than load Data Load Address Tags Store Commit Path Speculative Store Buffer L1 Data Cache Load Data TagDataSVTagDataSVTagDataSVTagDataSVTagDataSVTagDataSV

29. March 1, 2012CS152, Spring 2012 29 Fetch Decode & Rename Reorder BufferPC Branch Prediction Commit Datapath: Branch Prediction and Speculative Execution Branch Resolution Branch Unit ALU Reg. File MEM Store Buffer D$ Execute kill

30. March 1, 2012CS152, Spring 2012 30 Instruction Flow in Unified Physical Register File Pipeline Fetch –Get instruction bits from current guess at PC, place in fetch buffer –Update PC using sequential address or branch predictor (BTB) Decode/Rename –Take instruction from fetch buffer –Allocate resources to execute instruction: »Destination physical register, if instruction writes a register »Entry in reorder buffer to provide in-order commit »Entry in issue window to wait for execution »Entry in memory buffer, if load or store –Decode will stall if resources not available –Rename source and destination registers –Check source registers for readiness –Insert instruction into issue window+reorder buffer+memory buffer

31. March 1, 2012CS152, Spring 2012 31 Memory Instructions Split store instruction into two pieces during decode: –Address calculation, store-address –Data movement, store-data Allocate space in program order in memory buffers during decode Store instructions: –Store-address calculates address and places in store buffer –Store-data copies store value into store buffer –Store-address and store-data execute independently out of issue window –Stores only commit to data cache at commit point Load instructions: –Load address calculation executes from window –Load with completed effective address searches memory buffer –Load instruction may have to wait in memory buffer for earlier store ops to resolve

32. March 1, 2012CS152, Spring 2012 32 Issue Stage Writebacks from completion phase “wakeup” some instructions by causing their source operands to become ready in issue window –In more speculative machines, might wake up waiting loads in memory buffer Need to “select” some instructions for issue –Arbiter picks a subset of ready instructions for execution –Example policies: random, lower-first, oldest-first, critical-first Instructions read out from issue window and sent to execution

33. March 1, 2012CS152, Spring 2012 33 Execute Stage Read operands from physical register file and/or bypass network from other functional units Execute on functional unit Write result value to physical register file (or store buffer if store) Produce exception status, write to reorder buffer Free slot in instruction window

34. March 1, 2012CS152, Spring 2012 34 Commit Stage Read completed instructions in-order from reorder buffer –(may need to wait for next oldest instruction to complete) If exception raised –flush pipeline, jump to exception handler Otherwise, release resources: –Free physical register used by last writer to same architectural register –Free reorder buffer slot –Free memory reorder buffer slot

35. March 1, 2012CS152, Spring 2012 35 Acknowledgements These slides contain material developed and copyright by: –Arvind (MIT) –Krste Asanovic (MIT/UCB) –Joel Emer (Intel/MIT) –James Hoe (CMU) –John Kubiatowicz (UCB) –David Patterson (UCB) MIT material derived from course 6.823 UCB material derived from course CS252