1. COSC3330 Computer Architecture Lecture 21. TLP and Multi-core
Instructor: Weidong Shi (Larry), PhD
Computer Science Department
University of Houston

2. Topic Today Thread Level Parallelism Multi-core Next week
Last day of class
Student Presentation
Final exam

3. TLP ILP of a single program is hard
Large ILP is Far-flung
We are human after all, program w/ sequential mind
Reality: running multiple threads or programs
Thread Level Parallelism
Time Multiplexing
Throughput computing
Multiple program workloads
Multiple concurrent threads
Helper threads to improve single program performance

4. Multithreading
Difficult to continue to extract instruction-level parallelism (ILP) from a single sequential thread of control
Many workloads can make use of thread-level parallelism (TLP)
TLP from multiprogramming (run independent sequential jobs)
TLP from multithreaded applications (run one job faster using parallel threads)
Multithreading uses TLP to improve utilization of a single processor

5. UNIX Threads
A thread is a basic unit of CPU utilization; it consists of:
Program counter
Register set
Stack space
A thread shares with its peer threads its:
Code segment
Data segment
Operating-system resources,
An OS may supports multiple processes, a process can have multiple threads.

6. Multi-Tasking Paradigm
Virtual memory makes it easy
Context switch could be expensive or requires extra HW
VIVT cache
VIPT cache
TLBs
FU1
FU2
FU3
FU4
Unused
Thread 1
Thread 2
Thread 3
Thread 4
Thread 5
Execution Time Quantum
Conventional
Superscalar
Single
Threaded

7. Conventional Multithreading
Zero-overhead context switch
Duplicated contexts for threads
0:r0
0:r7
1:r0
CtxtPtr
1:r7
2:r0
2:r7
3:r0
3:r7
Register file
Memory (shared by threads)

8. Cycle Interleaving MT Per-cycle, Per-thread instruction fetching
Examples: HEP, Horizon, Tera MTA, MIT M-machine
Interesting questions to consider
Does it need a sophisticated branch predictor?
Or does it need any speculative execution at all?
Get rid of “branch prediction”?
Does it need any out-of-order execution capability?

9. Multithreading
How can we guarantee no dependencies between instructions in a pipeline?
-- One way is to interleave execution of instructions from different program threads on same pipeline F D X M W t0 t1 t2 t3 t4 t5 t6 t7 t8
T1: LW r1, 0(r2)
T2: ADD r7, r1, r4
T3: XORI r5, r4, #12
T4: SW 0(r7), r5
T1: LW r5, 12(r1) t9
Interleave 4 threads, T1-T4, on non-bypassed 5-stage pipe
Prior instruction in a thread always completes write-back before next instruction in same thread reads register file

10. CDC 6600 Peripheral Processors (Cray, 1964)
First multithreaded hardware
10 “virtual” I/O processors
Fixed interleave on simple pipeline
Pipeline has 100ns cycle time
Each virtual processor executes one instruction every 1000ns
Was objective was to cope with long I/O latencies?

11. Simple Multithreaded PipelineX PC 1 PC 1
GPR1 I$ IR
GPR1 PC 1
GPR1
GPR1 PC 1 D$ Y +1 2 2
Thread select
Have to carry thread select down pipeline to ensure correct state bits read/written at each pipe stage
Appears to software (including OS) as multiple, albeit slower, CPUs

12. Thread Scheduling Policies
Fixed interleave (CDC 6600 PPUs, 1964)
Each of N threads executes one instruction every N cycles
If thread not ready to go in its slot, insert pipeline bubble
Hardware-controlled thread scheduling (Tera MTA, )
Hardware keeps track of which threads are ready to go
Picks next thread to execute based on hardware priority scheme

13. Tera MTA (1990-97) Multi-Threaded Architecture
Up to 256 processors
Up to 128 active threads per processor
Flat, shared main memory
No data cache
Sustains one main memory access per cycle per processor

14. Coarse-Grain Multithreading
Tera MTA designed for supercomputing applications with large data sets and low locality
No data cache
Many parallel threads needed to hide large memory latency
Other applications are more cache friendly
Few pipeline bubbles if cache mostly has hits
Just add a few threads to hide occasional cache miss latencies
Swap threads on cache misses

15. Multithreading on One Processor
Unused streams

16. Superscalar Machine Efficiency
Issue width
Time
Instruction issue
Completely idle cycle (vertical waste)
Partially filled cycle, i.e., IPC < 4
(horizontal waste)

17. Vertical Multithreading
Issue width
Instruction issue
Second thread interleaved cycle-by-cycle
Time
Partially filled cycle, i.e., IPC < 4
(horizontal waste)
What is the effect of cycle-by-cycle interleaving?
removes vertical waste, but leaves some horizontal waste

18. IBM Power 4
Single-threaded predecessor to Power 5. out-of-order engine, each may issue an instruction each cycle.

19. Power 4 Power 5 2 fetch (PC), 2 initial decodes
2 commits (architected register sets)
Power 5
2 fetch (PC), 2 initial decodes

20. Chip Multiprocessing (CMP)
Issue width
Time
What is the effect of splitting into multiple processors?
reduces horizontal waste,
leaves some vertical waste, and
puts upper limit on peak throughput of each thread.

21. Ideal Superscalar Multithreading [Tullsen, Eggers, Levy, UW, 1995]
Issue width
Time
Interleave multiple threads to multiple issue slots with no restrictions

22. Simultaneous Multithreading (SMT)
Intel’s HyperThreading (2-way SMT)
IBM Power7 (4/6/8 cores, 4-way SMT); IBM Power5/6 (2 cores. Each 2-way SMT)
Basic ideas: Conventional MT + Simultaneous issue + Sharing common resources
Fdiv, unpipe
(16 cycles)
Fetch
Unit
RS & ROB
plus
Physical
Register
File
Decode
FMult
(4 cycles)
Reg
File
Reg
File
Reg
File
Register
Renamer
FAdd
(2 cyc)
Reg
File
Register
Renamer
Reg
File
Register
Renamer
Reg
File
Register
Renamer
Reg
File PC
Register
Renamer
Reg
File PC
Register
Renamer PC
Register
Renamer PC
Register
Renamer PC PC PC PC
ALU1
ALU2
I-CACHE
D-CACHE
Load/Store
(variable)

23. Pentium-4 Hyperthreading (2002)
First commercial SMT design (2-way SMT)
Hyperthreading == SMT
Logical processors share nearly all resources of the physical processor
Caches, execution units, branch predictors
Die area overhead of hyperthreading ~ 5%
When one logical processor is stalled, the other can make progress
Hyperthreading dropped on OoO P6 based followons to Pentium-4 (Pentium-M, Core Duo, Core 2 Duo), until revived with Nehalem generation machines in 2008.

24. Multi-threading Paradigm
Unused
Execution Time
FU1
FU2
FU3
FU4
Conventional
Superscalar
Single
Threaded
Fine-grained
Multithreading
(cycle-by-cycle
Interleaving)
Thread 2
Thread 3
Thread 4
Thread 5
Coarse-grained
Multithreading
(Block Interleaving)
Chip
Multiprocessor
(CMP or
MultiCore)
Simultaneous
Multithreading
(SMT)

25. Background: Multicore
Multi-processor Computer
Computer with multiple CPUs
Multi-core: Multiple CPUs on the Same Die

26. Conventional Processor Designs Run Out of Steam
The ILP Wall
Limited ILP in Application
The Power Wall
The Design Wall
Time to Market
Design Complexity (Verification)

27. ILP Wall
The 1990’s to 2005 was the era of instruction level parallelism.

28. Uniprocessor Performance
Diminishing Gains

29. Very Deep Pipelined

30. Power Wall
Power = 1/2 C V2 F
Power density has increased with transistor scaling!
Source: S. Borkar (Intel)

31. Cooking-Aware (or Colwell’s Charcoal-aware) Computing
Heat Dissipation
Cooler jet
3D Cooler Pro
Pure copper
Cooligy’s channel
PS3 Grill (
Source: K. Skadron
Cooking-Aware (or Colwell’s Charcoal-aware) Computing

32. Costs of Developing the Next Generation Processors
Logic Transistor per Chip
(M) 1 10
100
1,000
10,000
100,000
1,000,000
10,000,000
2003
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2005
2007
2009
100,000,000
Logic Tr./Chip
Tr./Staff Month. x
21%/Yr. compound
Productivity growth rate
58%/Yr. compounded
Complexity growth rate
0.1
0.01
0.001
(K) Trans./Staff - Mo.
Productivity
Source: Sematech
Complexity
Fermi GPU – 3 Billion Transistors
8-Core Xeon Nehalem 2.3 Billion Transistors
Complexity outpaces design productivity

33. Replication! Design Scaling
With Every Generation Can Integrate 2x More Functions per Chip;
Cost of a Function Decreases by 2x
But …
How to design chips with more and more functions?
Design engineering population does not double every two years…
Diminishing Returns for Design Dollars
Replication!

34. In the same process technology…
Dual Core
Power = 1/2 C V2 F
In the same process technology…
Core
Cache
Core
Cache
Voltage = 1
Freq = 1
Area = 1
Power = 1
Perf = 1
Voltage = -15%
Freq = -15%
Area = 2
Power = 1
Perf = ~1.8

35. Is a Multicore Really Better Off? Power
Large Core
Cache 4
Performance = 1/2 3
Performance
Small
Core 2 2 1 1 1 1
Parallelism is the main technique
to improve system performance
under a power budget
Multicore: Power efficient,
better power and thermal
management C1 C2 C3 C4
Cache 4 4 3 3 2 2 1 1

36. Intel TeraFlops Research Prototype
2KB Data Memory
3KB Instruction Memory
No coherence support
2 FMACs
Next-gen had 3D-integrated memory
SRAM first
Then DRAM
Intel did not report further result

37. Tilera 100 64-bit processor 40nm 100 cores 32 KB L1I/L1D
256KB L2 cache
Integrated DD3 controller
On-chip mesh network
MIT RAW

38. Anton, Special-Purpose Machine for Molecular Dynamics Simulation
512-node Anton machine, 2008

39. Amdahl’s Law + Begins with Simple Software Assumption
LogTM: Log-based Transactional Memory
12/16/2017
Amdahl’s Law
Begins with Simple Software Assumption
Fraction F of execution time perfectly parallelizable
Fraction 1 – F Completely Serial
Time on 1 core = (1 – F) / 1 + F / 1 = 1
Time on N cores = (1 – F) / 1 + F / N
Amdahl’s Speedup = 1 +
1 - F F N
UW-Madison Architecture Seminar 39

40. Multicore Hardware Model
LogTM: Log-based Transactional Memory
12/16/2017
Multicore Hardware Model
Micro-architects can improve single-core performance using more of the bounded resource
A Simple Base Core
Consumes 1 Base Core Equivalent (BCE) resources
Provides performance normalized to 1
An Enhanced Core (in same process generation)
Consumes R BCEs
Performance as a function Perf(R)
What does function Perf(R) look like?
UW-Madison Architecture Seminar 40

41. LogTM: Log-based Transactional Memory
12/16/2017
More on Enhanced Cores
(Performance Perf(R) consuming R BCEs resources)
If Perf(R) > R  Always enhance core
Cost-effectively speedups both sequential & parallel
Therefore, Equations Assume Perf(R) < R
UW-Madison Architecture Seminar 41

42. How Many Cores per Chip? Each Chip Bounded to N BCEs (for all cores)
LogTM: Log-based Transactional Memory
12/16/2017
How Many Cores per Chip?
Each Chip Bounded to N BCEs (for all cores)
Each Core consumes R BCEs
Assume Symmetric Multicore = All Cores Identical
Therefore, N/R Cores per Chip
For an N = 16 BCE Chip:
Sixteen 1-BCE cores
Four 4-BCE cores
One 16-BCE core
UW-Madison Architecture Seminar 42

43. Performance of Multicore Chips
LogTM: Log-based Transactional Memory
12/16/2017
Performance of Multicore Chips
Serial Fraction 1-F uses 1 core at rate Perf(R)
Serial time = (1 – F) / Perf(R)
Parallel Fraction uses N/R cores at rate Perf(R) each
Parallel time = F / (Perf(R) * (N/R)) = F*R / Perf(R)*N
Therefore, w.r.t. one base core:
Symmetric Speedup = 1 +
1 - F
Perf(R)
F * R
Perf(R)*N
Enhanced Cores speed Serial & Parallel
UW-Madison Architecture Seminar 43

44. LogTM: Log-based Transactional Memory
12/16/2017
Multicore Chip, N = 16 BCEs
F=0.5
R=16,
Cores=1,
Speedup=4
(16 cores)
(8 cores)
(2 cores)
(1 core)
(4 cores)
F=0.5, Opt. Speedup S = 4 = 1/(0.5/ *16/(4*16))
Need to increase parallelism to make multicore optimal!
UW-Madison Architecture Seminar 44

45. LogTM: Log-based Transactional Memory
12/16/2017
Multicore Chip, N = 16 BCEs
F=0.9, R=2, Cores=8, Speedup=6.7
F=0.5
R=16,
Cores=1,
Speedup=4
At F=0.9, Multicore optimal, but speedup limited
Need to obtain even more parallelism!
UW-Madison Architecture Seminar 45

46. LogTM: Log-based Transactional Memory
12/16/2017
Multicore Chip, N = 16 BCEs
F1, R=1, Cores=16, Speedup16
F matters: Amdahl’s Law applies to multicore chips
UW-Madison Architecture Seminar 46