Presentation on theme: "EENG449b/Savvides Lec 17.1 3/29/05 March 29, 2005 Prof. Andreas Savvides Spring 2005 g449b EENG 449bG/CPSC 439bG."— Presentation transcript:
EENG449b/Savvides Lec /29/05 March 29, 2005 Prof. Andreas Savvides Spring g449b EENG 449bG/CPSC 439bG Computer Systems Lecture 17 Instruction Level Parallelism III (Multiple Issue Processors and Speculation)
EENG449b/Savvides Lec /29/05 Why can Tomasulo overlap iterations of loops? Register renaming –Multiple iterations use different physical destinations for registers (dynamic loop unrolling). Reservation stations –Permit instruction issue to advance past integer control flow operations –Also buffer old values of registers - totally avoiding the WAR stall that we saw in the scoreboard. Other perspective: Tomasulo building data flow dependency graph on the fly.
EENG449b/Savvides Lec /29/05 Tomasulo’s scheme offers 2 major advantages (1)the distribution of the hazard detection logic –distributed reservation stations and the CDB (Common Data Bus) –If multiple instructions waiting on single result, & each instruction has other operand, then instructions can be released simultaneously by broadcast on CDB –If a centralized register file were used, the units would have to read their results from the registers when register buses are available. (2) the elimination of stalls for WAW and WAR hazards
EENG449b/Savvides Lec /29/05 Multiple Issue Processors Two main types: –Superscalar Processors »Issue variable number of instructions per clock cycle »Can be statically or dynamically scheduled –VLIW (Very Large Instruction Set) Processors »Issue a constant number of instructions formatted as a packet of smaller instructions »Parallelism across instructions is specifically indicated »Statically scheduled by the compiler
EENG449b/Savvides Lec /29/05 Multiple Issue Issues issue packet: group of instructions from fetch unit that could potentially issue in 1 clock –If instruction causes structural hazard or a data hazard either due to earlier instruction in execution or to earlier instruction in issue packet, then instruction does not issue –0 to N instruction issues per clock cycle, for N-issue Performing issue checks in 1 cycle could limit clock cycle time: O(n 2 -n) comparisons –=> issue stage usually split and pipelined –1st stage decides how many instructions from within this packet can issue, 2nd stage examines hazards among selected instructions and those already been issued –=> splitting leads to higher branch penalties => prediction accuracy important
EENG449b/Savvides Lec /29/05 Getting CPI < 1: Issuing Multiple Instructions/Cycle Superscalar MIPS: 2 instructions, 1 FP & 1 anything – Fetch 64-bits/clock cycle; Int on left, FP on right – Can only issue 2nd instruction if 1st instruction issues – More ports for FP registers to do FP load & FP op in a pair TypePipeStages Int. instructionIFIDEXMEMWB FP instructionIFIDEXMEMWB Int. instructionIFIDEXMEMWB FP instructionIFIDEXMEMWB Int. instructionIFIDEXMEMWB FP instructionIFIDEXMEMWB 1 cycle load delay expands to 3 instructions in SS –instruction in right half can’t use it, nor instructions in next slot
EENG449b/Savvides Lec /29/05 Dynamic Scheduling in Superscalar The easy way How to issue two instructions and keep in-order instruction issue for Tomasulo? –Assume 1 integer + 1 floating point –1 Tomasulo control for integer, 1 for floating point Issue 2X Clock Rate, so that issue remains in order Only loads/stores might cause dependency between integer and FP issue: –Replace load reservation station with a load queue; operands must be read in the order they are fetched –Load checks addresses in Store Queue to avoid RAW violation –Store checks addresses in Load Queue to avoid WAR,WAW
EENG449b/Savvides Lec /29/05 Superscalar Processors Require More Ambitious Scheduling Need to deal with preserving exception order Pipelining the issue stage will result in additional overheads –E.g in a superscalar pipeline the outcome of a load instruction cannot be used in the next 3 instructions –Because the issue stage is pipelined Without more ambitious scheduling, Superscalar processors do not have an advantage How can we extend Tomasulo’s algorithm to schedule a superscalar pipeline?
EENG449b/Savvides Lec /29/05 Hardware Speculation Tomasulo had: In-order issue, out-of-order execution, and out-of-order completion Need to “fix” the out-of-order completion aspect so that we can find precise breakpoint in instruction stream.
EENG449b/Savvides Lec /29/05 Relationship between precise interrupts and specultation: Speculation is a form of guessing. Important for branch prediction: –Need to “take our best shot” at predicting branch direction. –Go further than dynamic branch prediction – start executing If we speculate and are wrong, need to back up and restart execution to point at which we predicted incorrectly: –This is exactly same as precise exceptions! Technique for both precise interrupts/exceptions and speculation: in-order completion or commit Tomasulo’s algorithm can be extended to support speculation
EENG449b/Savvides Lec /29/05 Tomasulo Recap
EENG449b/Savvides Lec /29/05 Improvements to Tomasulo Algorithm Separate the bypassing of results among instructions –Allow an instruction to execute and bypass its results to other instructions –Do not allow the instruction to make changes that cannot be undone unless we know the instruction is no longer speculative –When an instruction is no longer speculative, we allow it to update the register file – instruction commit Speculation key idea –Execute instructions out of order but commit them in order
EENG449b/Savvides Lec /29/05 HW support for precise interrupts Need HW buffer for results of uncommitted instructions: reorder buffer (ROB) –3 fields: instr, destination, value –Use reorder buffer number instead of reservation station when execution completes –Supplies operands between execution complete & commit –(Reorder buffer can be operand source => more registers like RS) –Instructions commit –Once instruction commits, result is put into register –As a result, easy to undo speculated instructions on mispredicted branches or exceptions Reorder Buffer FP Op Queue FP Adder Res Stations FP Regs
EENG449b/Savvides Lec /29/05 Four Steps of Speculative Tomasulo Algorithm 1.Issue—get instruction from FP Op Queue If reservation station and reorder buffer slot free, issue instr & send operands & reorder buffer no. for destination (this stage sometimes called “dispatch”) 2.Execution—operate on operands (EX) When both operands ready then execute; if not ready, watch CDB for result; when both in reservation station, execute; checks RAW (sometimes called “issue”) 3.Write result—finish execution (WB) Write on Common Data Bus to all awaiting FUs & reorder buffer; mark reservation station available. 4.Commit—update register with reorder result When instr. at head of reorder buffer & result present, update register with result (or store to memory) and remove instr from reorder buffer. Mispredicted branch flushes reorder buffer (sometimes called “graduation”)
EENG449b/Savvides Lec /29/05 What are the hardware complexities with reorder buffer (ROB)? Reorder Buffer FP Op Queue FP Adder Res Stations FP Regs Compar network Need as many ports on ROB as register file Reorder Table Dest Reg ResultExceptions?Valid Program Counter
EENG449b/Savvides Lec /29/05 Summary Reservations stations: implicit register renaming to larger set of registers + buffering source operands –Prevents registers as bottleneck –Avoids WAR, WAW hazards of Scoreboard –Allows loop unrolling in HW Not limited to basic blocks (integer units gets ahead, beyond branches) Today, helps cache misses as well –Don’t stall for L1 Data cache miss (insufficient ILP for L2 miss?) Lasting Contributions –Dynamic scheduling –Register renaming –Load/store disambiguation
EENG449b/Savvides Lec /29/05 Register renaming, virtual registers versus Reorder Buffers Alternative to Reorder Buffer is a larger virtual set of registers and register renaming Virtual registers hold both architecturally visible registers + temporary values –replace functions of reorder buffer and reservation station Renaming process maps names of architectural registers to registers in virtual register set –Changing subset of virtual registers contains architecturally visible registers Simplifies instruction commit: mark register as no longer speculative, free register with old value Adds extra registers: Alpha, Pentium,… –Size limits no. instructions in execution (used until commit)
EENG449b/Savvides Lec /29/05 How much to speculate? Speculation Pro: uncover events that would otherwise stall the pipeline (cache misses) Speculation Con: speculate costly if exceptional event occurs when speculation was incorrect Typical solution: speculation allows only low- cost exceptional events (1st-level cache miss) When expensive exceptional event occurs, (2nd-level cache miss or TLB miss) processor waits until the instruction causing event is no longer speculative before handling the event Assuming single branch per cycle: future may speculate across multiple branches!
EENG449b/Savvides Lec /29/05 Limits to ILP Conflicting studies of amount –Benchmarks (vectorized Fortran FP vs. integer C programs) –Hardware sophistication –Compiler sophistication How much ILP is available using existing mechanisms with increasing HW budgets? Do we need to invent new HW/SW mechanisms to keep on processor performance curve? –Intel MMX, SSE (Streaming SIMD Extensions): 64 bit ints –Intel SSE2: 128 bit, including 2 64-bit Fl. Pt. per clock –Motorola AltaVec: 128 bit ints and FPs –Supersparc Multimedia ops, etc.
EENG449b/Savvides Lec /29/05 Limits to ILP Initial HW Model here; MIPS compilers. Assumptions for ideal/perfect machine to start: 1. Register renaming – infinite virtual registers => all register WAW & WAR hazards are avoided 2. Branch prediction – perfect; no mispredictions 3. Jump prediction – all jumps perfectly predicted 2 & 3 => machine with perfect speculation & an unbounded buffer of instructions available 4. Memory-address alias analysis – addresses are known & a store can be moved before a load provided addresses not equal Also: unlimited number of instructions issued/clock cycle; perfect caches; 1 cycle latency for all instructions (FP *,/);
EENG449b/Savvides Lec /29/05 More Realistic HW: Branch Impact Figure 3.38, Page 300 Change from Infinite window to examine to 2000 and maximum issue of 64 instructions per clock cycle ProfileBHT (512)TournamentPerfect No prediction FP: Integer: IPC
EENG449b/Savvides Lec /29/05 Realistic HW for ‘00: Window Impact (Figure 3.45, Page 309) Perfect disambiguation (HW), 1K Selective Prediction, 16 entry return, 64 registers, issue as many as window Infinite Integer: FP: IPC
EENG449b/Savvides Lec /29/05 How to Exceed ILP Limits of this study? WAR and WAW hazards through memory: eliminated WAW and WAR hazards through register renaming, but not in memory usage Unnecessary dependences (compiler not unrolling loops so iteration variable dependence) Overcoming the data flow limit: value prediction, predicting values and speculating on prediction –Address value prediction and speculation predicts addresses and speculates by reordering loads and stores; could provide better aliasing analysis, only need predict if addresses =
EENG449b/Savvides Lec /29/05 Workstation Microprocessors 3/2001 Source: Microprocessor Report, Max issue: 4 instructions (many CPUs) Max rename registers: 128 (Pentium 4) Max BHT: 4K x 9 (Alpha 21264B), 16Kx2 (Ultra III) Max Window Size (OOO): 126 intructions (Pent. 4) Max Pipeline: 22/24 stages (Pentium 4)
EENG449b/Savvides Lec /29/05 Conclusion : 1000X performance –Moore’s Law transistors/chip => Moore’s Law for Performance/MPU Hennessy: industry been following a roadmap of ideas known in 1985 to exploit Instruction Level Parallelism and (real) Moore’s Law to get 1.55X/year –Caches, Pipelining, Superscalar, Branch Prediction, Out-of-order execution, … ILP limits: To make performance progress in future need to have explicit parallelism from programmer vs. implicit parallelism of ILP exploited by compiler, HW? –Otherwise drop to old rate of 1.3X per year? –Less than 1.3X because of processor-memory performance gap? Impact on you: if you care about performance, better think about explicitly parallel algorithms vs. rely on ILP?
EENG449b/Savvides Lec /29/05 Review: Dynamic Branch Prediction Prediction becoming important part of scalar execution Branch History Table: 2 bits for loop accuracy Correlation: Recently executed branches correlated with next branch. –Either different branches –Or different executions of same branches Tournament Predictor: more resources to competitive solutions and pick between them Branch Target Buffer: include branch address & prediction Predicated Execution can reduce number of branches, number of mispredicted branches Return address stack for prediction of indirect jump
EENG449b/Savvides Lec /29/05 Review: Limits of ILP : 1000X performance –Moore’s Law transistors/chip => Moore’s Law for Performance/MPU Hennessy: industry been following a roadmap of ideas known in 1985 to exploit Instruction Level Parallelism to get 1.55X/year –Caches, Pipelining, Superscalar, Branch Prediction, Out-of-order execution, … ILP limits: To make performance progress in future need to have explicit parallelism from programmer vs. implicit parallelism of ILP exploited by compiler, HW? –Otherwise drop to old rate of 1.3X per year? –Less because of processor-memory performance gap? Impact on you: if you care about performance, better think about explicitly parallel algorithms vs. rely on ILP?
EENG449b/Savvides Lec /29/05 Dynamic Scheduling in P6 (Pentium Pro, II, III) Q: How pipeline 1 to 17 byte 80x86 instructions? P6 doesn’t pipeline 80x86 instructions P6 decode unit translates the Intel instructions into 72-bit micro-operations (~ MIPS) Sends micro-operations to reorder buffer & reservation stations Many instructions translate to 1 to 4 micro-operations Complex 80x86 instructions are executed by a conventional microprogram (8K x 72 bits) that issues long sequences of micro- operations 14 clocks in total pipeline (~ 3 state machines)
EENG449b/Savvides Lec /29/05 Dynamic Scheduling in P6 Parameter80x86microops Max. instructions issued/clock36 Max. instr. complete exec./clock5 Max. instr. commited/clock3 Window (Instrs in reorder buffer)40 Number of reservations stations20 Number of rename registers40 No. integer functional units (FUs)2 No. floating point FUs1 No. SIMD Fl. Pt. FUs1 No. memory Fus1 load + 1 store
EENG449b/Savvides Lec /29/05 P6 Pipeline 14 clocks in total (~3 state machines) 8 stages are used for in-order instruction fetch, decode, and issue –Takes 1 clock cycle to determine length of 80x86 instructions + 2 more to create the micro-operations (uops) 3 stages are used for out-of-order execution in one of 5 separate functional units 3 stages are used for instruction commit Instr Fetch 16B /clk Instr Decode 3 Instr /clk Renaming 3 uops /clk Execu- tion units (5) Gradu- ation 3 uops /clk 16B6 uops Reserv. Station Reorder Buffer
EENG449b/Savvides Lec /29/05 IP = PC P6 Block Diagram From: life.com/articles/pentium4/
EENG449b/Savvides Lec /29/05 Pentium III Die Photo EBL/BBL - Bus logic, Front, Back MOB - Memory Order Buffer Packed FPU - MMX Fl. Pt. (SSE) IEU - Integer Execution Unit FAU - Fl. Pt. Arithmetic Unit MIU - Memory Interface Unit DCU - Data Cache Unit PMH - Page Miss Handler DTLB - Data TLB BAC - Branch Address Calculator RAT - Register Alias Table SIMD - Packed Fl. Pt. RS - Reservation Station BTB - Branch Target Buffer IFU - Instruction Fetch Unit (+I$) ID - Instruction Decode ROB - Reorder Buffer MS - Micro-instruction Sequencer 1st Pentium III, Katmai: 9.5 M transistors, 12.3 * 10.4 mm in 0.25-mi. with 5 layers of aluminum
EENG449b/Savvides Lec /29/05 P6 Performance: Stalls at decode stage I$ misses or lack of RS/Reorder buf. entry
EENG449b/Savvides Lec /29/05 P6 Dynamic Benefit? Sum of parts CPI vs. Actual CPI Ratio of sum of parts vs. actual CPI: 1.38X avg. (1.29X integer)
EENG449b/Savvides Lec /29/05 AMD Althon Similar to P6 microarchitecture (Pentium III), but more resources Transistors: PIII 24M v. Althon 37M Die Size: 106 mm 2 v. 117 mm 2 Power: 30W v. 76W Cache: 16K/16K/256K v. 64K/64K/256K Window size: 40 vs. 72 uops Rename registers: 40 v. 36 int +36 Fl. Pt. BTB: 512 x 2 v x 2 Pipeline: stages v stages Clock rate: 1.0 GHz v. 1.2 GHz Memory bandwidth: 1.06 GB/s v GB/s
EENG449b/Savvides Lec /29/05 Pentium 4 Still translate from 80x86 to micro-ops P4 has better branch predictor, more FUs Instruction Cache holds micro-operations vs. 80x86 instructions –no decode stages of 80x86 on cache hit –called “trace cache” (TC) Faster memory bus: 400 MHz v. 133 MHz Caches –Pentium III: L1I 16KB, L1D 16KB, L2 256 KB –Pentium 4: L1I 12K uops, L1D 8 KB, L2 256 KB –Block size: PIII 32B v. P4 128B; 128 v. 256 bits/clock Clock rates: –Pentium III 1 GHz v. Pentium IV 1.5 GHz –14 stage pipeline vs. 24 stage pipeline
EENG449b/Savvides Lec /29/05 Pentium 4 features Multimedia instructions 128 bits wide vs. 64 bits wide => 144 new instructions –When used by programs?? –Faster Floating Point: execute 2 64-bit Fl. Pt. Per clock –Memory FU: bit load, store /clock to MMX regs Using RAMBUS DRAM –Bandwidth faster, latency same as SDRAM –Cost 2X-3X vs. SDRAM ALUs operate at 2X clock rate for many ops Pipeline doesn’t stall at this clock rate: uops replay Rename registers: 40 vs. 128; Window: 40 v. 126 BTB: 512 vs entries (Intel: 1/3 improvement)
EENG449b/Savvides Lec /29/05 Block Diagram of Pentium 4 Microarchitecture BTB = Branch Target Buffer (branch predictor) I-TLB = Instruction TLB, Trace Cache = Instruction cache RF = Register File; AGU = Address Generation Unit "Double pumped ALU" means ALU clock rate 2X => 2X ALU F.U.s From “Pentium 4 (Partially) Previewed,” Microprocessor Report, 8/28/00
EENG449b/Savvides Lec /29/05 Pentium 4 Die Photo 42M Xtors –PIII: 26M 217 mm 2 –PIII: 106 mm 2 L1 Execution Cache –Buffer 12,000 Micro-Ops 8KB data cache 256KB L2$
EENG449b/Savvides Lec /29/05 Benchmarks: Pentium 4 v. PIII v. Althon SPECbase2000 –Int, GHz: 524, 454, AMD –FP, GHz: 549, 329, AMD WorldBench 2000 benchmark (business) PC World magazine, Nov. 20, 2000 (bigger is better) –P4 : 164, PIII : 167, AMD Althon: 180 Quake 3 Arena: P4 172, Althon 151 SYSmark 2000 composite: P4 209, Althon 221 Office productivity: P4 197, Althon 209 S.F. Chronicle 11/20/00: "… the challenge for AMD now will be to argue that frequency is not the most important thing-- precisely the position Intel has argued while its Pentium III lagged behind the Athlon in clock speed."
EENG449b/Savvides Lec /29/05 Why? Instruction count is the same for x86 Clock rates: P4 > Althon > PIII How can P4 be slower? Time = Instruction count x CPI x 1/Clock rate Average Clocks Per Instruction (CPI) of P4 must be worse than Althon, PIII Will CPI ever get < 1.0 for real programs?
EENG449b/Savvides Lec /29/05 Simultaneous Multithreading (SMT) Simultaneous multithreading (SMT): insight that dynamically scheduled processor already has many HW mechanisms to support multithreading –large set of virtual registers that can be used to hold the register sets of independent threads (assuming separate renaming tables are kept for each thread) –out-of-order completion allows the threads to execute out of order, and get better utilization of the HW Source: Micrprocessor Report, December 6, 1999 “Compaq Chooses SMT for Alpha”
EENG449b/Savvides Lec /29/05 SMT is coming Just adding a per thread renaming table and keeping separate PCs –Independent commitment can be supported by logically keeping a separate reorder buffer for each thread Compaq has announced it for future Alpha microprocessor: in 2003; others likely On a multiprogramming workload comprising a mixture of SPECint95 and SPECfp95 benchmarks, Compaq claims the SMT it simulated achieves a 2.25X higher throughput with 4 simultaneous threads than with just 1 thread. For parallel programs, 4 threads 1.75X v. 1 Source: Micrprocessor Report, December 6, 1999 “Compaq Chooses SMT for Alpha”