1. CS 152 Computer Architecture and Engineering Lecture 4 - Pipelining Krste Asanovic Electrical Engineering and Computer Sciences University of California at Berkeley http://www.eecs.berkeley.edu/~krste http://inst.eecs.berkeley.edu/~cs152

2. 2/5/2008CS152-Spring’08 2 Last time in Lecture 3 Microcoding became less attractive as gap between RAM and ROM speeds reduced Complex instruction sets difficult to pipeline, so difficult to increase performance as gate count grew Iron-law explains architecture design space –Trade instruction/program, cycles/instruction, and time/cycle Load-Store RISC ISAs designed for efficient pipelined implementations –Very similar to vertical microcode –Inspired by earlier Cray machines

3. 2/5/2008CS152-Spring’08 3 “Iron Law” of Processor Performance Time = Instructions Cycles Time Program Program * Instruction * Cycle –Instructions per program depends on source code, compiler technology, and ISA –Cycles per instructions (CPI) depends upon the ISA and the microarchitecture –Time per cycle depends upon the microarchitecture and the base technology MicroarchitectureCPIcycle time Microcoded>1short Single-cycle unpipelined1long Pipelined1short Lecture 3 This lecture

4. 2/5/2008CS152-Spring’08 4 An Ideal Pipeline All objects go through the same stages No sharing of resources between any two stages Propagation delay through all pipeline stages is equal The scheduling of an object entering the pipeline is not affected by the objects in other stages stage 1 stage 2 stage 3 stage 4 These conditions generally hold for industrial assembly lines. But can an instruction pipeline satisfy the last condition?

5. 2/5/2008CS152-Spring’08 5 Pipelined MIPS To pipeline MIPS: First build MIPS without pipelining with CPI=1 Next, add pipeline registers to reduce cycle time while maintaining CPI=1

6. 2/5/2008CS152-Spring’08 6 Pipelined Datapath Clock period can be reduced by dividing the execution of an instruction into multiple cycles t C > max {t IM, t RF, t ALU, t DM, t RW } ( = t DM probably) However, CPI will increase unless instructions are pipelined write -back phase fetch phase execute phase decode & Reg-fetch phase memory phase addr wdata rdata Data Memory we ALU Imm Ext 0x4 Add addr rdata Inst. Memory rd1 GPRs rs1 rs2 ws wd rd2 we IR PC

7. 2/5/2008CS152-Spring’08 7 Technology Assumptions Thus, the following timing assumption is reasonable A small amount of very fast memory (caches) backed up by a large, slower memory Fast ALU (at least for integers) Multiported Register files (slower!) t IM t RF t ALU t DM t RW A 5-stage pipelined Harvard architecture will be the focus of our detailed design

8. 2/5/2008CS152-Spring’08 8 5-Stage Pipelined Execution time t0t1t2t3t4t5t6t7.... instruction1IF 1 ID 1 EX 1 MA 1 WB 1 instruction2 IF 2 ID 2 EX 2 MA 2 WB 2 instruction3IF 3 ID 3 EX 3 MA 3 WB 3 instruction4 IF 4 ID 4 EX 4 MA 4 WB 4 instruction5 IF 5 ID 5 EX 5 MA 5 WB 5 Write - Back (WB) I-Fetch (IF) Execute (EX) Decode, Reg. Fetch (ID) Memory (MA) addr wdata rdata Data Memory we ALU Imm Ext 0x4 Add addr rdata Inst. Memory rd1 GPRs rs1 rs2 ws wd rd2 we IR PC

9. 2/5/2008CS152-Spring’08 9 5-Stage Pipelined Execution Resource Usage Diagram time t0t1t2t3t4t5t6t7.... IFI 1 I 2 I 3 I 4 I 5 IDI 1 I 2 I 3 I 4 I 5 EX I 1 I 2 I 3 I 4 I 5 MA I 1 I 2 I 3 I 4 I 5 WB I 1 I 2 I 3 I 4 I 5 Resources Write - Back (WB) I-Fetch (IF) Execute (EX) Decode, Reg. Fetch (ID) Memory (MA) addr wdata rdata Data Memory we ALU Imm Ext 0x4 Add addr rdata Inst. Memory rd1 GPRs rs1 rs2 ws wd rd2 we IR PC

10. 2/5/2008CS152-Spring’08 10 Pipelined Execution: ALU Instructions IR 31 PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add IR Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata addr wdata rdata Data Memory we Not quite correct! We need an Instruction Reg (IR) for each stage

11. 2/5/2008CS152-Spring’08 11 Pipelined MIPS Datapath without jumps IR 31 PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add IR Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we Data Memory wdata addr wdata rdata we OpSel ExtSelBSrc WBSrc MemWrite RegDst RegWrite FDEMW Control Points Need to Be Connected

12. 2/5/2008CS152-Spring’08 12 How Instructions can Interact with each other in a pipeline An instruction in the pipeline may need a resource being used by another instruction in the pipeline  structural hazard An instruction may depend on something produced by an earlier instruction –Dependence may be for a data value  data hazard –Dependence may be for the next instruction’s address  control hazard (branches, exceptions)

13. 2/5/2008CS152-Spring’08 13 Data Hazards... r1 r0 + 10 r4 r1 + 17... r1 is stale. Oops! r1 … r4 r1… IR 31 PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add IR Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata addr wdata rdata Data Memory we

14. 2/5/2008CS152-Spring’08 14 Resolving Data Hazards (1) Strategy 1: Wait for the result to be available by freezing earlier pipeline stages  interlocks

15. 2/5/2008CS152-Spring’08 15 Feedback to Resolve Hazards Later stages provide dependence information to earlier stages which can stall (or kill) instructions FB 1 stage 1 stage 2 stage 3 stage 4 FB 2 FB 3 FB 4 Controlling a pipeline in this manner works provided the instruction at stage i+1 can complete without any interference from instructions in stages 1 to i (otherwise deadlocks may occur)

16. 2/5/2008CS152-Spring’08 16 IR 31 PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add IR Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata addr wdata rdata Data Memory we nop Interlocks to resolve Data Hazards... r1 r0 + 10 r4 r1 + 17... Stall Condition

17. 2/5/2008CS152-Spring’08 17 stalled stages time t0t1t2t3t4t5t6t7.... IFI 1 I 2 I 3 I 3 I 3 I 3 I 4 I 5 IDI 1 I 2 I 2 I 2 I 2 I 3 I 4 I 5 EX I 1 nopnopnopI 2 I 3 I 4 I 5 MA I 1 nopnopnopI 2 I 3 I 4 I 5 WB I 1 nopnopnopI 2 I 3 I 4 I 5 Stalled Stages and Pipeline Bubbles time t0t1t2t3t4t5t6t7.... (I 1 ) r1 (r0) + 10IF 1 ID 1 EX 1 MA 1 WB 1 (I 2 ) r4 (r1) + 17IF 2 ID 2 ID 2 ID 2 ID 2 EX 2 MA 2 WB 2 (I 3 )IF 3 IF 3 IF 3 IF 3 ID 3 EX 3 MA 3 WB 3 (I 4 ) IF 4 ID 4 EX 4 MA 4 WB 4 (I 5 ) IF 5 ID 5 EX 5 MA 5 WB 5 Resource Usage nop  pipeline bubble

18. 2/5/2008CS152-Spring’08 18 IR 31 PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add IR Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata addr wdata rdata Data Memory we nop Interlock Control Logic Compare the source registers of the instruction in the decode stage with the destination register of the uncommitted instructions. stall C stall ws rs rt ?

19. 2/5/2008CS152-Spring’08 19 C dest Interlock Control Logic ignoring jumps & branches Should we always stall if the rs field matches some rd? IR PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add IR Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata addr wdata rdata Data Memory we 31 nop stall C stall ws rs rt ? we re1re2 C re wswe ws C dest we not every instruction writes a register we not every instruction reads a register re

20. 2/5/2008CS152-Spring’08 20 Source & Destination Registers source(s) destination ALUrd (rs) func (rt) rs, rtrd ALUirt (rs) op immrs rt LWrt M [(rs) + imm] rs rt SWM [(rs) + imm] (rt) rs, rt BZcond (rs) true:PC (PC) + immrs false: PC (PC) + 4rs JPC (PC) + imm JALr31 (PC), PC (PC) + imm31 JRPC (rs) rs JALRr31 (PC), PC (rs) rs31 R-type: op rs rt rd func I-type: op rs rt immediate16 J-type: op immediate26

21. 2/5/2008CS152-Spring’08 21 Deriving the Stall Signal C dest ws = Case opcode ALUrd ALUi, LWrt JAL, JALRR31 we = Case opcode ALU, ALUi, LW (ws  0) JAL, JALR on... off C re re1 = Case opcode ALU, ALUi, on off re2 = Case opcode on off LW, SW, BZ, JR, JALR J, JAL ALU, SW... C stall stall = ((rs D =ws E ).we E + (rs D =ws M ).we M + (rs D =ws W ).we W ). re1 D + ((rt D =ws E ).we E + (rt D =ws M ).we M + (rt D =ws W ).we W ). re2 D This is not the full story !

22. 2/5/2008CS152-Spring’08 22 Hazards due to Loads & Stores... M[(r1)+7]  (r2) r4  M[(r3)+5]... IR 31 PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add IR Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata addr wdata rdata Data Memory we nop Stall Condition Is there any possible data hazard in this instruction sequence? What if (r1)+7 = (r3)+5 ?

23. 2/5/2008CS152-Spring’08 23 Load & Store Hazards However, the hazard is avoided because our memory system completes writes in one cycle ! Load/Store hazards are sometimes resolved in the pipeline and sometimes in the memory system itself. More on this later in the course.... M[(r1)+7]  (r2) r4  M[(r3)+5]... (r1)+7 = (r3)+5  data hazard

24. 2/5/2008CS152-Spring’08 24 CS152 Administrivia Krste, office hours, Monday 1-3pm, 645 Soda –email for alternate time Henry office hours, 511 Soda –9:30-10:30AM Mondays –2:00-3:00PM Fridays First lab and practice problems this evening In-class quiz dates: –Q1: Tuesday February 19 (ISAs, microcode, simple pipelining) –Q2: Tuesday March 4 (memory hierarchies)

25. 2/5/2008CS152-Spring’08 25 Course Schedule Check website for schedule Generally, a lab, practice problem set, plus quiz every two weeks Lab and practice problems out on Tuesday (this evening for first) Lab overview in section next day (Wednesday) Practice problems due following Tuesday in class, solutions available on line Problem review in section next day (Wednesday) Lab due in class Thursday Quiz following Tuesday, with new PS and lab.

26. 2/5/2008CS152-Spring’08 26 Resolving Data Hazards (2) Strategy 2: Route data as soon as possible after it is calculated to the earlier pipeline stage  bypass

27. 2/5/2008CS152-Spring’08 27 Bypassing Each stall or kill introduces a bubble in the pipeline CPI > 1 time t0t1t2t3t4t5t6t7.... (I 1 ) r1 r0 + 10IF 1 ID 1 EX 1 MA 1 WB 1 (I 2 ) r4 r1 + 17IF 2 ID 2 ID 2 ID 2 ID 2 EX 2 MA 2 WB 2 (I 3 )IF 3 IF 3 IF 3 IF 3 ID 3 EX 3 MA 3 (I 4 ) stalled stagesIF 4 ID 4 EX 4 (I 5 ) IF 5 ID 5 timet0t1t2t3t4t5t6t7.... (I 1 ) r1 r0 + 10IF 1 ID 1 EX 1 MA 1 WB 1 (I 2 ) r4  r1 + 17IF 2 ID 2 EX 2 MA 2 WB 2 (I 3 )IF 3 ID 3 EX 3 MA 3 WB 3 (I 4 ) IF 4 ID 4 EX 4 MA 4 WB 4 (I 5 ) IF 5 ID 5 EX 5 MA 5 WB 5 A new datapath, i.e., a bypass, can get the data from the output of the ALU to its input

28. 2/5/2008CS152-Spring’08 28 Adding a Bypass ASrc... (I 1 )r1 r0 + 10 (I 2 )r4 r1 + 17 r4 r1r1  IR PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add IR Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata addr wdata rdata Data Memory we 31 nop stall D EMW When does this bypass help? r1 M[r0 + 10] r4 r1 + 17 JAL 500 r4 r31 + 17 yesno

29. 2/5/2008CS152-Spring’08 29 The Bypass Signal Deriving it from the Stall Signal ASrc = (rs D =ws E ).we E.re1 D we = Case opcode ALU, ALUi, LW (ws  0) JAL, JALR on... off No because only ALU and ALUi instructions can benefit from this bypass Is this correct? Split we E into two components: we-bypass, we-stall stall = ( ((rs D =ws E ).we E + (rs D =ws M ).we M + (rs D =ws W ).we W ).re1 D +((rt D =ws E ).we E + (rt D =ws M ).we M + (rt D =ws W ).we W ).re2 D ) ws = Case opcode ALUrd ALUi, LWrt JAL, JALRR31

30. 2/5/2008CS152-Spring’08 30 Bypass and Stall Signals we-bypass E = Case opcode E ALU, ALUi(ws  0)... off ASrc = (rs D =ws E ).we-bypass E. re1 D Split we E into two components: we-bypass, we-stall stall = ((rs D =ws E ).we-stall E + (rs D =ws M ).we M + (rs D =ws W ).we W ). re1 D +((rt D = ws E ).we E + (rt D = ws M ).we M + (rt D = ws W ).we W ). re2 D we-stall E = Case opcode E LW (ws  0) JAL, JALRon... off

31. 2/5/2008CS152-Spring’08 31 Fully Bypassed Datapath ASrc IR PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add IR ALU Imm Ext rd1 GPRs rs1 rs2 ws wd rd2 we wdata addr wdata rdata Data Memory we 31 nop stall D EMW PC for JAL,... BSrc Is there still a need for the stall signal ? stall = (rs D =ws E ). (opcode E =LW E ).(ws E 0 ).re1 D + (rt D =ws E ). (opcode E =LW E ).(ws E 0 ).re2 D

32. 2/5/2008CS152-Spring’08 32 Resolving Data Hazards (3) Strategy 3: Speculate on the dependence. Two cases: Guessed correctly  do nothing Guessed incorrectly  kill and restart

33. 2/5/2008CS152-Spring’08 33 Instruction to Instruction Dependence What do we need to calculate next PC: –For Jumps » Opcode, offset and PC –For Jump Register »Opcode and Register value –For Conditional Branches »Opcode, PC, Register (for condition), and offset –For all others »Opcode and PC

34. 2/5/2008CS152-Spring’08 34 time t0t1t2t3t4t5t6t7.... (I 1 ) r1 (r0) + 10IF 1 ID 1 EX 1 MA 1 WB 1 (I 2 ) r3 (r2) + 17 IF 2 IF 2 ID 2 EX 2 MA 2 WB 2 (I 3 )IF 3 IF 3 ID 3 EX 3 MA 3 WB 3 (I 4 ) IF 4 IF 4 ID 4 EX 4 MA 4 WB 4 time t0t1t2t3t4t5t6t7.... IFI 1 nop I 2 nop I 3 nop I 4 IDI 1 nop I 2 nop I 3 nop I 4 EX I 1 nopI 2 nopI 3 nopI 4 MA I 1 nopI 2 nopI 3 nopI 4 WB I 1 nopI 2 nopI 3 nopI 4 PC Calculation Bubbles Resource Usage nop  pipeline bubble

35. 2/5/2008CS152-Spring’08 35 Speculate next address is PC+4 I 1 096ADD I 2 100J 304 I 3 104ADD I 4 304ADD kill A jump instruction kills (not stalls) the following instruction stall How? I2I2 I1I1 104 IR PC addr inst Inst Memory 0x4 Add nop IR E M Add Jump? PCSrc (pc+4 / jabs / rind/ br)

36. 2/5/2008CS152-Spring’08 36 Pipelining Jumps I 1 096ADD I 2 100J 304 I 3 104ADD I 4 304ADD kill I2I2 I1I1 104 stall IR PC addr inst Inst Memory 0x4 Add nop IR E M Add Jump? PCSrc (pc+4 / jabs / rind/ br) IRSrc D = Case opcode D J, JAL nop... IM To kill a fetched instruction -- Insert a mux before IR Any interaction between stall and jump? nop IRSrc D I2I2 I1I1 304 nop

37. 2/5/2008CS152-Spring’08 37 time t0t1t2t3t4t5t6t7.... IFI 1 I 2 I 3 I 4 I 5 IDI 1 I 2 nop I 4 I 5 EX I 1 I 2 nop I 4 I 5 MA I 1 I 2 nop I 4 I 5 WB I 1 I 2 nop I 4 I 5 Jump Pipeline Diagrams time t0t1t2t3t4t5t6t7.... (I 1 ) 096: ADDIF 1 ID 1 EX 1 MA 1 WB 1 (I 2 ) 100: J 304IF 2 ID 2 EX 2 MA 2 WB 2 (I 3 ) 104: ADDIF 3 nop nop nop nop (I 4 ) 304: ADD IF 4 ID 4 EX 4 MA 4 WB 4 Resource Usage nop  pipeline bubble

38. 2/5/2008CS152-Spring’08 38 Pipelining Conditional Branches I 1 096ADD I 2 100BEQZ r1 +200 I 3 104ADD I 4 304ADD BEQZ? I2I2 I1I1 104 stall IR PC addr inst Inst Memory 0x4 Add nop IR E M Add PCSrc (pc+4 / jabs / rind / br) nop IRSrc D Branch condition is not known until the execute stage what action should be taken in the decode stage ? A Y ALU zero?

39. 2/5/2008CS152-Spring’08 39 Pipelining Conditional Branches I 1 096ADD I 2 100BEQZ r1 +200 I 3 104ADD I 4 304ADD stall IR PC addr inst Inst Memory 0x4 Add nop IR E M Add PCSrc (pc+4 / jabs / rind / br) nop IRSrc D A Y ALU zero? If the branch is taken - kill the two following instructions - the instruction at the decode stage is not valid  stall signal is not valid I2I2 I1I1 108 I3I3 BEQZ? ?

40. 2/5/2008CS152-Spring’08 40 Pipelining Conditional Branches I 1 096ADD I 2 100BEQZ r1 200 I 3 104ADD I 4 304ADD stall IR PC addr inst Inst Memory 0x4 Add nop IR E M PCSrc (pc+4/jabs/rind/br) nop A Y ALU zero? I2I2 I1I1 108 I3I3 BEQZ? Jump? IRSrc D IRSrc E If the branch is taken - kill the two following instructions - the instruction at the decode stage is not valid  stall signal is not valid Add PC

41. 2/5/2008CS152-Spring’08 41 New Stall Signal stall = ( ((rs D =ws E ).we E + (rs D =ws M ).we M + (rs D =ws W ).we W ).re1 D + ((rt D =ws E ).we E + (rt D =ws M ).we M + (rt D =ws W ).we W ).re2 D ). !((opcode E =BEQZ).z + (opcode E =BNEZ).!z) Don’t stall if the branch is taken. Why? Instruction at the decode stage is invalid

42. 2/5/2008CS152-Spring’08 42 Control Equations for PC and IR Muxes PCSrc = Case opcode E BEQZ.z, BNEZ.!z br...  Case opcode D J, JALjabs JR, JALRrind... pc+4 IRSrc D = Case opcode E BEQZ.z, BNEZ.!z nop...  Case opcode D J, JAL, JR, JALR nop... IM Give priority to the older instruction, i.e., execute stage instruction over decode stage instruction IRSrc E = Case opcode E BEQZ.z, BNEZ.!z nop... stall.nop + !stall.IR D

43. 2/5/2008CS152-Spring’08 43 time t0t1t2t3t4t5t6t7.... IFI 1 I 2 I 3 I 4 I 5 IDI 1 I 2 I 3 nop I 5 EX I 1 I 2 nop nop I 5 MA I 1 I 2 nop nop I 5 WB I 1 I 2 nop nop I 5 Branch Pipeline Diagrams (resolved in execute stage) time t0t1t2t3t4t5t6t7.... (I 1 ) 096: ADDIF 1 ID 1 EX 1 MA 1 WB 1 (I 2 ) 100: BEQZ 200IF 2 ID 2 EX 2 MA 2 WB 2 (I 3 ) 104: ADDIF 3 ID 3 nop nop nop (I 4 ) 108: IF 4 nop nop nop nop (I 5 ) 304: ADD IF 5 ID 5 EX 5 MA 5 WB 5 Resource Usage nop  pipeline bubble

44. 2/5/2008CS152-Spring’08 44 One pipeline bubble can be removed if an extra comparator is used in the Decode stage PC addr inst Inst Memory 0x4 Add IR nop E Add PCSrc (pc+4 / jabs / rind/ br) rd1 GPRs rs1 rs2 ws wd rd2 we nop Zero detect on register file output Pipeline diagram now same as for jumps D Reducing Branch Penalty ( resolve in decode stage)

45. 2/5/2008CS152-Spring’08 45 Branch Delay Slots (expose control hazard to software) Change the ISA semantics so that the instruction that follows a jump or branch is always executed –gives compiler the flexibility to put in a useful instruction where normally a pipeline bubble would have resulted. I 1 096ADD I 2 100BEQZ r1 200 I 3 104ADD I 4 304ADD Delay slot instruction executed regardless of branch outcome Other techniques include branch prediction, which can dramatically reduce the branch penalty... to come later

46. 2/5/2008CS152-Spring’08 46 Why an Instruction may not be dispatched every cycle (CPI>1) Full bypassing may be too expensive to implement –typically all frequently used paths are provided –some infrequently used bypass paths may increase cycle time and counteract the benefit of reducing CPI Loads have two cycle latency –Instruction after load cannot use load result –MIPS-I ISA defined load delay slots, a software-visible pipeline hazard (compiler schedules independent instruction or inserts NOP to avoid hazard). Removed in MIPS-II. Conditional branches may cause bubbles –kill following instruction(s) if no delay slots Machines with software-visible delay slots may execute significant number of NOP instructions inserted by the compiler.

47. 2/5/2008CS152-Spring’08 47 Acknowledgements These slides contain material developed and copyright by: –Arvind (MIT) –Krste Asanovic (MIT/UCB) –Joel Emer (Intel/MIT) –James Hoe (CMU) –John Kubiatowicz (UCB) –David Patterson (UCB) MIT material derived from course 6.823 UCB material derived from course CS252