COSC3330 Computer Architecture Lecture 21. TLP and Multi-core Instructor: Weidong Shi (Larry), PhD Computer Science Department University of Houston

Topic Today Thread Level Parallelism Multi-core Next week Last day of class Student Presentation Final exam

TLP ILP of a single program is hard Large ILP is Far-flung We are human after all, program w/ sequential mind Reality: running multiple threads or programs Thread Level Parallelism Time Multiplexing Throughput computing Multiple program workloads Multiple concurrent threads Helper threads to improve single program performance

Multithreading Difficult to continue to extract instruction-level parallelism (ILP) from a single sequential thread of control Many workloads can make use of thread-level parallelism (TLP) TLP from multiprogramming (run independent sequential jobs) TLP from multithreaded applications (run one job faster using parallel threads) Multithreading uses TLP to improve utilization of a single processor

UNIX Threads A thread is a basic unit of CPU utilization; it consists of: Program counter Register set Stack space A thread shares with its peer threads its: Code segment Data segment Operating-system resources, An OS may supports multiple processes, a process can have multiple threads.

Multi-Tasking Paradigm Virtual memory makes it easy Context switch could be expensive or requires extra HW VIVT cache VIPT cache TLBs FU1 FU2 FU3 FU4 Unused Thread 1 Thread 2 Thread 3 Thread 4 Thread 5 Execution Time Quantum Conventional Superscalar Single Threaded

Conventional Multithreading Zero-overhead context switch Duplicated contexts for threads 0:r0 0:r7 1:r0 CtxtPtr 1:r7 2:r0 2:r7 3:r0 3:r7 Register file Memory (shared by threads)

Cycle Interleaving MT Per-cycle, Per-thread instruction fetching Examples: HEP, Horizon, Tera MTA, MIT M-machine Interesting questions to consider Does it need a sophisticated branch predictor? Or does it need any speculative execution at all? Get rid of “branch prediction”? Does it need any out-of-order execution capability?

Multithreading How can we guarantee no dependencies between instructions in a pipeline? -- One way is to interleave execution of instructions from different program threads on same pipeline F D X M W t0 t1 t2 t3 t4 t5 t6 t7 t8 T1: LW r1, 0(r2) T2: ADD r7, r1, r4 T3: XORI r5, r4, #12 T4: SW 0(r7), r5 T1: LW r5, 12(r1) t9 Interleave 4 threads, T1-T4, on non-bypassed 5-stage pipe Prior instruction in a thread always completes write-back before next instruction in same thread reads register file

CDC 6600 Peripheral Processors (Cray, 1964) First multithreaded hardware 10 “virtual” I/O processors Fixed interleave on simple pipeline Pipeline has 100ns cycle time Each virtual processor executes one instruction every 1000ns Was objective was to cope with long I/O latencies?

Simple Multithreaded Pipeline X PC 1 PC 1 GPR1 I$ IR GPR1 PC 1 GPR1 GPR1 PC 1 D$ Y +1 2 2 Thread select Have to carry thread select down pipeline to ensure correct state bits read/written at each pipe stage Appears to software (including OS) as multiple, albeit slower, CPUs

Thread Scheduling Policies Fixed interleave (CDC 6600 PPUs, 1964) Each of N threads executes one instruction every N cycles If thread not ready to go in its slot, insert pipeline bubble Hardware-controlled thread scheduling (Tera MTA, 1990 - 97) Hardware keeps track of which threads are ready to go Picks next thread to execute based on hardware priority scheme

Tera MTA (1990-97) Multi-Threaded Architecture Up to 256 processors Up to 128 active threads per processor Flat, shared main memory No data cache Sustains one main memory access per cycle per processor

Coarse-Grain Multithreading Tera MTA designed for supercomputing applications with large data sets and low locality No data cache Many parallel threads needed to hide large memory latency Other applications are more cache friendly Few pipeline bubbles if cache mostly has hits Just add a few threads to hide occasional cache miss latencies Swap threads on cache misses

Multithreading on One Processor Unused streams

Superscalar Machine Efficiency Issue width Time Instruction issue Completely idle cycle (vertical waste) Partially filled cycle, i.e., IPC < 4 (horizontal waste)

Vertical Multithreading Issue width Instruction issue Second thread interleaved cycle-by-cycle Time Partially filled cycle, i.e., IPC < 4 (horizontal waste) What is the effect of cycle-by-cycle interleaving? removes vertical waste, but leaves some horizontal waste

IBM Power 4 Single-threaded predecessor to Power 5. out-of-order engine, each may issue an instruction each cycle.

Power 4 Power 5 2 fetch (PC), 2 initial decodes 2 commits (architected register sets) Power 5 2 fetch (PC), 2 initial decodes

Chip Multiprocessing (CMP) Issue width Time What is the effect of splitting into multiple processors? reduces horizontal waste, leaves some vertical waste, and puts upper limit on peak throughput of each thread.

Ideal Superscalar Multithreading [Tullsen, Eggers, Levy, UW, 1995] Issue width Time Interleave multiple threads to multiple issue slots with no restrictions

Simultaneous Multithreading (SMT) Intel’s HyperThreading (2-way SMT) IBM Power7 (4/6/8 cores, 4-way SMT); IBM Power5/6 (2 cores. Each 2-way SMT) Basic ideas: Conventional MT + Simultaneous issue + Sharing common resources Fdiv, unpipe (16 cycles) Fetch Unit RS & ROB plus Physical Register File Decode FMult (4 cycles) Reg File Reg File Reg File Register Renamer FAdd (2 cyc) Reg File Register Renamer Reg File Register Renamer Reg File Register Renamer Reg File PC Register Renamer Reg File PC Register Renamer PC Register Renamer PC Register Renamer PC PC PC PC ALU1 ALU2 I-CACHE D-CACHE Load/Store (variable)

Pentium-4 Hyperthreading (2002) First commercial SMT design (2-way SMT) Hyperthreading == SMT Logical processors share nearly all resources of the physical processor Caches, execution units, branch predictors Die area overhead of hyperthreading ~ 5% When one logical processor is stalled, the other can make progress Hyperthreading dropped on OoO P6 based followons to Pentium-4 (Pentium-M, Core Duo, Core 2 Duo), until revived with Nehalem generation machines in 2008.

Multi-threading Paradigm Unused Execution Time FU1 FU2 FU3 FU4 Conventional Superscalar Single Threaded Fine-grained Multithreading (cycle-by-cycle Interleaving) Thread 2 Thread 3 Thread 4 Thread 5 Coarse-grained Multithreading (Block Interleaving) Chip Multiprocessor (CMP or MultiCore) Simultaneous Multithreading (SMT)

Background: Multicore Multi-processor Computer Computer with multiple CPUs Multi-core: Multiple CPUs on the Same Die

Conventional Processor Designs Run Out of Steam The ILP Wall Limited ILP in Application The Power Wall The Design Wall Time to Market Design Complexity (Verification)

ILP Wall The 1990’s to 2005 was the era of instruction level parallelism.

Uniprocessor Performance Diminishing Gains

Very Deep Pipelined

Power Wall Power = 1/2 C V2 F Power density has increased with transistor scaling! Source: S. Borkar (Intel)

Cooking-Aware (or Colwell’s Charcoal-aware) Computing Heat Dissipation Cooler jet 3D Cooler Pro Pure copper Cooligy’s channel PS3 Grill (http://www.therealps2grill.com Source: K. Skadron Cooking-Aware (or Colwell’s Charcoal-aware) Computing

Costs of Developing the Next Generation Processors Logic Transistor per Chip (M) 1 10 100 1,000 10,000 100,000 1,000,000 10,000,000 2003 1981 1983 1985 1987 1989 1991 1993 1995 1997 1999 2001 2005 2007 2009 100,000,000 Logic Tr./Chip Tr./Staff Month. x 21%/Yr. compound Productivity growth rate 58%/Yr. compounded Complexity growth rate 0.1 0.01 0.001 (K) Trans./Staff - Mo. Productivity Source: Sematech Complexity Fermi GPU – 3 Billion Transistors 8-Core Xeon Nehalem 2.3 Billion Transistors Complexity outpaces design productivity

Replication! Design Scaling With Every Generation Can Integrate 2x More Functions per Chip; Cost of a Function Decreases by 2x But … How to design chips with more and more functions? Design engineering population does not double every two years… Diminishing Returns for Design Dollars Replication!

In the same process technology… Dual Core Power = 1/2 C V2 F In the same process technology… Core Cache Core Cache Voltage = 1 Freq = 1 Area = 1 Power = 1 Perf = 1 Voltage = -15% Freq = -15% Area = 2 Power = 1 Perf = ~1.8

Is a Multicore Really Better Off? Power Large Core Cache 4 Performance = 1/2 3 Performance Small Core 2 2 1 1 1 1 Parallelism is the main technique to improve system performance under a power budget Multicore: Power efficient, better power and thermal management C1 C2 C3 C4 Cache 4 4 3 3 2 2 1 1

Intel TeraFlops Research Prototype 2KB Data Memory 3KB Instruction Memory No coherence support 2 FMACs Next-gen had 3D-integrated memory SRAM first Then DRAM Intel did not report further result

Tilera 100 64-bit processor 40nm 100 cores 32 KB L1I/L1D 256KB L2 cache Integrated DD3 controller On-chip mesh network MIT RAW

Anton, Special-Purpose Machine for Molecular Dynamics Simulation 512-node Anton machine, 2008

Amdahl’s Law + Begins with Simple Software Assumption LogTM: Log-based Transactional Memory 12/16/2017 Amdahl’s Law Begins with Simple Software Assumption Fraction F of execution time perfectly parallelizable Fraction 1 – F Completely Serial Time on 1 core = (1 – F) / 1 + F / 1 = 1 Time on N cores = (1 – F) / 1 + F / N Amdahl’s Speedup = 1 + 1 - F F N UW-Madison Architecture Seminar 39

Multicore Hardware Model LogTM: Log-based Transactional Memory 12/16/2017 Multicore Hardware Model Micro-architects can improve single-core performance using more of the bounded resource A Simple Base Core Consumes 1 Base Core Equivalent (BCE) resources Provides performance normalized to 1 An Enhanced Core (in same process generation) Consumes R BCEs Performance as a function Perf(R) What does function Perf(R) look like? UW-Madison Architecture Seminar 40

LogTM: Log-based Transactional Memory 12/16/2017 More on Enhanced Cores (Performance Perf(R) consuming R BCEs resources) If Perf(R) > R  Always enhance core Cost-effectively speedups both sequential & parallel Therefore, Equations Assume Perf(R) < R UW-Madison Architecture Seminar 41

How Many Cores per Chip? Each Chip Bounded to N BCEs (for all cores) LogTM: Log-based Transactional Memory 12/16/2017 How Many Cores per Chip? Each Chip Bounded to N BCEs (for all cores) Each Core consumes R BCEs Assume Symmetric Multicore = All Cores Identical Therefore, N/R Cores per Chip For an N = 16 BCE Chip: Sixteen 1-BCE cores Four 4-BCE cores One 16-BCE core UW-Madison Architecture Seminar 42

Performance of Multicore Chips LogTM: Log-based Transactional Memory 12/16/2017 Performance of Multicore Chips Serial Fraction 1-F uses 1 core at rate Perf(R) Serial time = (1 – F) / Perf(R) Parallel Fraction uses N/R cores at rate Perf(R) each Parallel time = F / (Perf(R) * (N/R)) = F*R / Perf(R)*N Therefore, w.r.t. one base core: Symmetric Speedup = 1 + 1 - F Perf(R) F * R Perf(R)*N Enhanced Cores speed Serial & Parallel UW-Madison Architecture Seminar 43

LogTM: Log-based Transactional Memory 12/16/2017 Multicore Chip, N = 16 BCEs F=0.5 R=16, Cores=1, Speedup=4 (16 cores) (8 cores) (2 cores) (1 core) (4 cores) F=0.5, Opt. Speedup S = 4 = 1/(0.5/4 + 0.5*16/(4*16)) Need to increase parallelism to make multicore optimal! UW-Madison Architecture Seminar 44

LogTM: Log-based Transactional Memory 12/16/2017 Multicore Chip, N = 16 BCEs F=0.9, R=2, Cores=8, Speedup=6.7 F=0.5 R=16, Cores=1, Speedup=4 At F=0.9, Multicore optimal, but speedup limited Need to obtain even more parallelism! UW-Madison Architecture Seminar 45

LogTM: Log-based Transactional Memory 12/16/2017 Multicore Chip, N = 16 BCEs F1, R=1, Cores=16, Speedup16 F matters: Amdahl’s Law applies to multicore chips UW-Madison Architecture Seminar 46