CS 152 Computer Architecture and Engineering Lecture 13 - Out-of-Order Issue, Register Renaming, & Branch Prediction Krste Asanovic Electrical Engineering and Computer Sciences University of California at Berkeley http://www.eecs.berkeley.edu/~krste http://inst.eecs.berkeley.edu/~cs152 CS252 S05

Last time in Lecture 12 Pipelining is complicated by multiple and/or variable latency functional units Out-of-order and/or pipelined execution requires tracking of dependencies RAW WAR WAW Dynamic issue logic can support out-of-order execution to improve performance Last time, looked at simple scoreboard to track out-of-order completion Hardware register renaming can further improve performance by removing hazards. 3/12/2009 CS152-Spring’09 CS252 S05

Out-of-Order Issue IF ID WB ALU Mem Fadd Fmul Issue Issue stage buffer holds multiple instructions waiting to issue. Decode adds next instruction to buffer if there is space and the instruction does not cause a WAR or WAW hazard. Note: WAR possible again because issue is out-of-order (WAR not possible with in-order issue and latching of input operands at functional unit) Any instruction in buffer whose RAW hazards are satisfied can be issued (for now at most one dispatch per cycle). On a write back (WB), new instructions may get enabled. 3/12/2009 CS152-Spring’09 CS252 S05

Overcoming the Lack of Register Names Floating Point pipelines often cannot be kept filled with small number of registers. IBM 360 had only 4 floating-point registers Can a microarchitecture use more registers than specified by the ISA without loss of ISA compatibility ? Robert Tomasulo of IBM suggested an ingenious solution in 1967 using on-the-fly register renaming 3/12/2009 CS152-Spring’09 CS252 S05

Instruction-level Parallelism via Renaming latency 1 LD F2, 34(R2) 1 2 LD F4, 45(R3) long 3 MULTD F6, F4, F2 3 4 SUBD F8, F2, F2 1 5 DIVD F4’, F2, F8 4 6 ADDD F10, F6, F4’ 1 1 2 3 4 5 6 X In-order: 1 (2,1) . . . . . . 2 3 4 4 3 5 . . . 5 6 6 Out-of-order: 1 (2,1) 4 4 5 . . . 2 (3,5) 3 6 6 Any antidependence can be eliminated by renaming. (renaming  additional storage) Can it be done in hardware? 3/12/2009 CS152-Spring’09 CS252 S05

Register Renaming IF ID WB ALU Mem Fadd Fmul Issue Decode does register renaming and adds instructions to the issue stage reorder buffer (ROB)  renaming makes WAR or WAW hazards impossible Any instruction in ROB whose RAW hazards have been satisfied can be dispatched.  Out-of-order or dataflow execution 3/12/2009 CS152-Spring’09 CS252 S05

Dataflow execution Reorder buffer ptr2 next to deallocate prt1 next available Ins# use exec op p1 src1 p2 src2 t1 t2 . tn Reorder buffer Instruction slot is candidate for execution when: It holds a valid instruction (“use” bit is set) It has not already started execution (“exec” bit is clear) Both operands are available (p1 and p2 are set) 3/12/2009 CS152-Spring’09 CS252 S05

Renaming & Out-of-order Issue An example Renaming table Reorder buffer Ins# use exec op p1 src1 p2 src2 t1 t2 t3 t4 t5 . data / ti p data F1 F2 F3 F4 F5 F6 F7 F8 1 1 1 LD 1 1 0 LD t1 v1 2 0 2 1 0 LD 2 1 1 LD v1 3 1 0 MUL 1 v2 1 v1 3 1 0 MUL 0 t2 1 v1 t2 t5 4 0 4 1 1 SUB 1 v1 1 v1 4 1 0 SUB 1 v1 1 v1 5 1 0 DIV 1 v1 1 v4 5 1 0 DIV 1 v1 0 t4 t3 t4 v4 1 LD F2, 34(R2) 2 LD F4, 45(R3) 3 MULTD F6, F4, F2 4 SUBD F8, F2, F2 5 DIVD F4, F2, F8 6 ADDD F10, F6, F4 When are tags in sources replaced by data? When can a name be reused? 3/12/2009 CS152-Spring’09 CS252 S05

Data-Driven Execution Renaming table & reg file Ins# use exec op p1 src1 p2 src2 t1 t2 . tn Reorder buffer Replacing the tag by its value is an expensive operation Load Unit Store Unit FU FU < t, result > Instruction template (i.e., tag t) is allocated by the Decode stage, which also associates tag with register in regfile When an instruction completes, its tag is deallocated 3/12/2009 CS152-Spring’09 CS252 S05

Simplifying Allocation/Deallocation ptr2 next to deallocate prt1 next available Ins# use exec op p1 src1 p2 src2 t1 t2 . tn Reorder buffer Instruction buffer is managed circularly “exec” bit is set when instruction begins execution When an instruction completes its “use” bit is marked free ptr2 is incremented only if the “use” bit is marked free 3/12/2009 CS152-Spring’09 CS252 S05

IBM 360/91 Floating-Point Unit R. M. Tomasulo, 1967 2 3 4 5 6 p tag/data load buffers (from memory) ... instructions 1 2 3 4 Floating Point Reg p tag/data p tag/data p tag/data p tag/data p tag/data p tag/data p tag/data p tag/data p tag/data distribute instruction templates by functional units 1 2 3 p tag/data p tag/data 1 p tag/data p tag/data p tag/data p tag/data 2 p tag/data p tag/data p tag/data p tag/data Adder Mult < tag, result > Common bus ensures that data is made available immediately to all the instructions waiting for it. Match tag, if equal, copy value & set presence “p”. p tag/data store buffers (to memory) p tag/data p tag/data 3/12/2009 CS152-Spring’09 CS252 S05

Effectiveness? Renaming and Out-of-order execution was first implemented in 1969 in IBM 360/91 but did not show up in the subsequent models until mid-Nineties. Why ? Reasons 1. Effective on a very small class of programs 2. Memory latency a much bigger problem 3. Exceptions not precise! One more problem needed to be solved 3/12/2009 CS152-Spring’09 CS252 S05

Precise Interrupts It must appear as if an interrupt is taken between two instructions (say Ii and Ii+1) the effect of all instructions up to and including Ii is totally complete no effect of any instruction after Ii has taken place The interrupt handler either aborts the program or restarts it at Ii+1 . 3/12/2009 CS152-Spring’09 CS252 S05

Effect on Interrupts Out-of-order Completion I1 DIVD f6, f6, f4 I2 LD f2, 45(r3) I3 MULTD f0, f2, f4 I4 DIVD f8, f6, f2 I5 SUBD f10, f0, f6 I6 ADDD f6, f8, f2 out-of-order comp 1 2 2 3 1 4 3 5 5 4 6 6 restore f2 restore f10 Consider interrupts Precise interrupts are difficult to implement at high speed - want to start execution of later instructions before exception checks finished on earlier instructions 3/12/2009 CS152-Spring’09 CS252 S05

Exception Handling (In-Order Five-Stage Pipeline) Asynchronous Interrupts Exc D PC Inst. Mem Decode E M Data Mem W + Cause EPC Kill D Stage Kill F Stage Kill E Stage Illegal Opcode Overflow Data Addr Except PC Address Exceptions Kill Writeback Select Handler PC Commit Point Hold exception flags in pipeline until commit point (M stage) Exceptions in earlier pipe stages override later exceptions Inject external interrupts at commit point (override others) If exception at commit: update Cause and EPC registers, kill all stages, inject handler PC into fetch stage 3/12/2009 CS152-Spring’09 CS252 S05

Phases of Instruction Execution PC Fetch: Instruction bits retrieved from cache. I-cache Fetch Buffer Decode: Instructions placed in appropriate issue (aka “dispatch”) stage buffer Issue Buffer Execute: Instructions and operands sent to execution units . When execution completes, all results and exception flags are available. Func. Units Result Buffer Commit: Instruction irrevocably updates architectural state (aka “graduation” or “completion”). Arch. State 3/12/2009 CS152-Spring’09 CS252 S05

In-Order Commit for Precise Exceptions Out-of-order In-order Commit Fetch Decode Reorder Buffer Kill Kill Kill Exception? Execute Inject handler PC Instructions fetched and decoded into instruction reorder buffer in-order Execution is out-of-order (  out-of-order completion) Commit (write-back to architectural state, i.e., regfile & memory, is in-order Temporary storage needed to hold results before commit (shadow registers and store buffers) 3/12/2009 CS152-Spring’09 CS252 S05

Extensions for Precise Exceptions Inst# use exec op p1 src1 p2 src2 pd dest data cause ptr2 next to commit ptr1 next available Reorder buffer add <pd, dest, data, cause> fields in the instruction template commit instructions to reg file and memory in program order  buffers can be maintained circularly on exception, clear reorder buffer by resetting ptr1=ptr2 (stores must wait for commit before updating memory) 3/12/2009 CS152-Spring’09 CS252 S05

Rollback and Renaming Register File Reorder buffer < t, result > (now holds only committed state) Reorder buffer Load Unit FU Store < t, result > t1 t2 . tn Ins# use exec op p1 src1 p2 src2 pd dest data Commit Register file does not contain renaming tags any more. How does the decode stage find the tag of a source register? 3/12/2009 CS152-Spring’09 CS252 S05

Renaming Table tag Register File Rename valid bit Table Reorder buffer Load Unit FU Store < t, result > t1 t2 . tn Ins# use exec op p1 src1 p2 src2 pd dest data Commit Renaming table is a cache to speed up register name look up. It needs to be cleared after each exception taken. When else are valid bits cleared? 3/12/2009 CS152-Spring’09 CS252 S05

CS152 Administrivia 3/12/2009 CS152-Spring’09 CS252 S05

Control Flow Penalty Next fetch started PC Fetch I-cache Branch executed Next fetch started I-cache Fetch Buffer Issue Buffer Func. Units Arch. State Execute Decode Result Commit PC Fetch Modern processors may have > 10 pipeline stages between next PC calculation and branch resolution ! How much work is lost if pipeline doesn’t follow correct instruction flow? 3/12/2009 CS152-Spring’09 CS252 S05

MIPS Branches and Jumps Each instruction fetch depends on one or two pieces of information from the preceding instruction: 1) Is the preceding instruction a taken branch? 2) If so, what is the target address? Instruction Taken known? Target known? J JR BEQZ/BNEZ 3/12/2009 CS152-Spring’09 CS252 S05

Branch Penalties in Modern Pipelines UltraSPARC-III instruction fetch pipeline stages (in-order issue, 4-way superscalar, 750MHz, 2000) A PC Generation/Mux P Instruction Fetch Stage 1 F Instruction Fetch Stage 2 B Branch Address Calc/Begin Decode I Complete Decode J Steer Instructions to Functional units R Register File Read E Integer Execute Remainder of execute pipeline (+ another 6 stages) Branch Target Address Known Branch Direction & Jump Register Target Known 3/12/2009 CS152-Spring’09 CS252 S05

Reducing Control Flow Penalty Software solutions Eliminate branches - loop unrolling Increases the run length Reduce resolution time - instruction scheduling Compute the branch condition as early as possible (of limited value) Hardware solutions Find something else to do - delay slots Replaces pipeline bubbles with useful work (requires software cooperation) Speculate - branch prediction Speculative execution of instructions beyond the branch Like using compiler to avoid WAW hazards, one can use compiler to reduce control flow penalty. 3/12/2009 CS152-Spring’09 CS252 S05

Branch Prediction Motivation: Required hardware support: Branch penalties limit performance of deeply pipelined processors Modern branch predictors have high accuracy (>95%) and can reduce branch penalties significantly Required hardware support: Prediction structures: Branch history tables, branch target buffers, etc. Mispredict recovery mechanisms: Keep result computation separate from commit Kill instructions following branch in pipeline Restore state to state following branch 3/12/2009 CS152-Spring’09 CS252 S05

Static Branch Prediction Overall probability a branch is taken is ~60-70% but: JZ JZ backward 90% forward 50% ISA can attach preferred direction semantics to branches, e.g., Motorola MC88110 bne0 (preferred taken) beq0 (not taken) ISA can allow arbitrary choice of statically predicted direction, e.g., HP PA-RISC, Intel IA-64 typically reported as ~80% accurate 3/12/2009 CS152-Spring’09 CS252 S05

Dynamic Branch Prediction learning based on past behavior Temporal correlation The way a branch resolves may be a good predictor of the way it will resolve at the next execution Spatial correlation Several branches may resolve in a highly correlated manner (a preferred path of execution) 3/12/2009 CS152-Spring’09 CS252 S05

Branch Prediction Bits Assume 2 BP bits per instruction Change the prediction after two consecutive mistakes! ¬take wrong taken ¬ taken right take BP state: (predict take/¬take) x (last prediction right/wrong) 3/12/2009 CS152-Spring’09 CS252 S05

Branch History Table Fetch PC k BHT Index 2k-entry BHT, 2 bits/entry Fetch PC k BHT Index 2k-entry BHT, 2 bits/entry Taken/¬Taken? I-Cache Opcode offset Instruction Branch? Target PC + 4K-entry BHT, 2 bits/entry, ~80-90% correct predictions 3/12/2009 CS152-Spring’09 CS252 S05

Exploiting Spatial Correlation Yeh and Patt, 1992 if (x[i] < 7) then y += 1; if (x[i] < 5) then c -= 4; If first condition false, second condition also false History register, H, records the direction of the last N branches executed by the processor 3/12/2009 CS152-Spring’09 CS252 S05

Two-Level Branch Predictor Pentium Pro uses the result from the last two branches to select one of the four sets of BHT bits (~95% correct) k Fetch PC 2-bit global branch history shift register Shift in Taken/¬Taken results of each branch Taken/¬Taken? 3/12/2009 CS152-Spring’09 CS252 S05

Limitations of BHTs Only predicts branch direction. Therefore, cannot redirect fetch stream until after branch target is determined. Correctly predicted taken branch penalty A PC Generation/Mux P Instruction Fetch Stage 1 F Instruction Fetch Stage 2 B Branch Address Calc/Begin Decode I Complete Decode J Steer Instructions to Functional units R Register File Read E Integer Execute Remainder of execute pipeline (+ another 6 stages) Jump Register penalty UltraSPARC-III fetch pipeline 3/12/2009 CS152-Spring’09 CS252 S05

Branch Target Buffer Branch Target Buffer (2k entries) IMEM k PC predicted BPb target Branch Target Buffer (2k entries) IMEM k PC target BP BP bits are stored with the predicted target address. IF stage: If (BP=taken) then nPC=target else nPC=PC+4 later: check prediction, if wrong then kill the instruction and update BTB & BPb else update BPb 3/12/2009 CS152-Spring’09 CS252 S05

Address Collisions Assume a 128-entry BTB BPb target take 236 1028 Add ..... 132 Jump 100 Assume a 128-entry BTB Instruction Memory What will be fetched after the instruction at 1028? BTB prediction = Correct target =  236 1032 kill PC=236 and fetch PC=1032 Is this a common occurrence? Can we avoid these bubbles? 3/12/2009 CS152-Spring’09 CS252 S05

BTB is only for Control Instructions BTB contains useful information for branch and jump instructions only  Do not update it for other instructions For all other instructions the next PC is PC+4 ! How to achieve this effect without decoding the instruction? 3/12/2009 CS152-Spring’09 CS252 S05

Branch Target Buffer (BTB) 2k-entry direct-mapped BTB (can also be associative) I-Cache PC k Valid valid Entry PC = match predicted target target PC Keep both the branch PC and target PC in the BTB PC+4 is fetched if match fails Only taken branches and jumps held in BTB Next PC determined before branch fetched and decoded 3/12/2009 CS152-Spring’09 CS252 S05

Consulting BTB Before Decoding 1028 Add ..... 132 Jump 100 BPb target take 236 entry PC 132 The match for PC=1028 fails and 1028+4 is fetched  eliminates false predictions after ALU instructions BTB contains entries only for control transfer instructions  more room to store branch targets 3/12/2009 CS152-Spring’09 CS252 S05

Combining BTB and BHT BTB entries are considerably more expensive than BHT, but can redirect fetches at earlier stage in pipeline and can accelerate indirect branches (JR) BHT can hold many more entries and is more accurate BHT BHT in later pipeline stage corrects when BTB misses a predicted taken branch BTB A PC Generation/Mux P Instruction Fetch Stage 1 F Instruction Fetch Stage 2 B Branch Address Calc/Begin Decode I Complete Decode J Steer Instructions to Functional units R Register File Read E Integer Execute BTB/BHT only updated after branch resolves in E stage 3/12/2009 CS152-Spring’09 CS252 S05

Uses of Jump Register (JR) Switch statements (jump to address of matching case) Dynamic function call (jump to run-time function address) Subroutine returns (jump to return address) How well does BTB work for each of these cases? 3/12/2009 CS152-Spring’09 CS252 S05

Subroutine Return Stack Small structure to accelerate JR for subroutine returns, typically much more accurate than BTBs. fa() { fb(); } fb() { fc(); } fc() { fd(); } Pop return address when subroutine return decoded Push call address when function call executed k entries (typically k=8-16) &fd() &fc() &fb() 3/12/2009 CS152-Spring’09 CS252 S05

Mispredict Recovery In-order execution machines: Assume no instruction issued after branch can write-back before branch resolves Kill all instructions in pipeline behind mispredicted branch Out-of-order execution? Multiple instructions following branch in program order can complete before branch resolves 3/12/2009 CS152-Spring’09 CS252 S05

In-Order Commit for Precise Exceptions Out-of-order In-order Commit Fetch Decode Reorder Buffer Kill Kill Kill Exception? Execute Inject handler PC Instructions fetched and decoded into instruction reorder buffer in-order Execution is out-of-order (  out-of-order completion) Commit (write-back to architectural state, i.e., regfile & memory, is in-order Temporary storage needed in ROB to hold results before commit 3/12/2009 CS152-Spring’09 CS252 S05

Branch Misprediction in Pipeline Inject correct PC Branch Prediction Branch Resolution Kill Kill Kill Commit PC Fetch Decode Reorder Buffer Complete Execute Can have multiple unresolved branches in ROB Can resolve branches out-of-order by killing all the instructions in ROB that follow a mispredicted branch 3/12/2009 CS152-Spring’09 CS252 S05

Recovering ROB/Renaming Table Rename Table t v Rename Snapshots Register File t v t v r1 r2 Ptr2 next to commit Ins# use exec op p1 src1 p2 src2 pd dest data t1 t2 . tn rollback next available Ptr1 next available Reorder buffer Commit Load Unit Store Unit FU FU FU < t, result > Take snapshot of register rename table at each predicted branch, recover earlier snapshot if branch mispredicted 3/12/2009 CS152-Spring’09 CS252 S05

Speculating Both Directions An alternative to branch prediction is to execute both directions of a branch speculatively resource requirement is proportional to the number of concurrent speculative executions only half the resources engage in useful work when both directions of a branch are executed speculatively branch prediction takes less resources than speculative execution of both paths With accurate branch prediction, it is more cost effective to dedicate all resources to the predicted direction 3/12/2009 CS152-Spring’09 CS252 S05

Acknowledgements These slides contain material developed and copyright by: Arvind (MIT) Krste Asanovic (MIT/UCB) Joel Emer (Intel/MIT) James Hoe (CMU) John Kubiatowicz (UCB) David Patterson (UCB) MIT material derived from course 6.823 UCB material derived from course CS252 3/12/2009 CS152-Spring’09 CS252 S05