1. CPE 631: Multithreading: Thread-Level Parallelism Within a Processor
Electrical and Computer Engineering University of Alabama in Huntsville
Aleksandar Milenkovic

2. Outline Trends in microarchitecture
Exploiting thread-level parallelism
Exploiting TLP within a processor
Resource sharing
Performance implications
Design challenges
Intel’s HT technology

3. Trends in microarchitecture
Higher clock speeds
To achieve high clock frequency make pipeline deeper (superpipelining)
Events that disrupt pipeline (branch mispredictions, cache misses, etc) become very expensive in terms of lost clock cycles
ILP: Instruction Level Parallelism
Extract parallelism in a single program
Superscalar processors have multiple execution units working in parallel
Challenge to find enough instructions that can be executed concurrently
Out-of-order execution => instructions are sent to execution units based on instruction dependencies rather than program order

4. Trends in microarchitecture
Cache hierarchies
Processor-memory speed gap
Use caches to reduce memory latency
Multiple levels of caches: smaller and faster closer to the processor core
Thread-level Parallelism
Multiple programs execute concurrently
Web-servers have an abundance of software threads
Users: surfing the web, listening to music, encoding/decoding video streams, etc.

5. Exploiting thread-level parallelism
CMP – Chip Multiprocessing
Multiple processors, each with a full set of architectural resources, reside on the same die
Processors may share an on-chip cache or each can have its own cache
Examples: HP Mako, IBM Power4
Challenges: Power, Die area (cost)
Time-slice multithreading
Processor switches between software threads after a predefined time slice
Can minimize the effects of long lasting events
Still, some execution slots are wasted

6. Multithreading Within a Processor
Until now, we have executed multiple threads of an application on different processors – can multiple threads execute concurrently on the same processor?
Why is this desireable?
inexpensive – one CPU, no external interconnects
no remote or coherence misses (more capacity misses)
Why does this make sense?
most processors can’t find enough work – peak IPC is 6, average IPC is 1.5!
threads can share resources  we can increase threads without a corresponding linear increase in area

7. What Resources are Shared?
Multiple threads are simultaneously active (in other words, a new thread can start without a context switch)
For correctness, each thread needs its own PC, its own logical regs (and its own mapping from logical to phys regs)
For performance, each thread could have its own ROB (so that a stall in one thread does not stall commit in other threads), I-cache, branch predictor, D-cache, etc. (for low interference), although note that more sharing  better utilization of resources
Each additional thread costs a PC, rename table, and ROB – cheap!

8. Approaches to Multithreading Within a Processor
Fine-grained multithreading: switches threads on every clock cycle
Pro: hide latency of from both short and long stalls
Con: Slows down execution of the individual threads ready to go
Course-grained multithreading: switches threads only on costly stalls (e.g., L2 stalls)
Pros: no switching each clock cycle, no slow down for ready-to-go threads
Con: limitations in hiding shorter stalls
Simultaneous Multithreading: exploits TLP at the same time it exploits ILP

9. How Resources are Shared?
Each box represents an issue slot for a functional unit. Peak thruput is 4 IPC.
Thread 1
Thread 2
Thread 3
Cycles
Thread 4
Idle
Superscalar
Coarse-grained Multithreading
Fine-Grained
Multithreading
Simultaneous
Multithreading
Superscalar processor has high under-utilization – not enough work every cycle, especially when there is a cache miss
Fine-grained multithreading can only issue instructions from a single thread in a cycle – can not find max work every cycle, but cache misses can be tolerated
Simultaneous multithreading can issue instructions from any thread every cycle – has the highest probability of finding work for every issue slot

10. Resource Sharing Thread-1 R1  R1 + R2 R3  R1 + R4 R5  R1 + R3
P73 P1 + P2
P74  P73 + P4
P75  P73 + P74
Instr Fetch
Instr Rename
Issue Queue
Instr Fetch
Instr Rename
P73 P1 + P2
P74  P73 + P4
P75  P73 + P74
P76  P33 + P34
P77  P33 + P76
P78  P77 + P35
R2  R1 + R2
R5  R1 + R2
R3  R5 + R3
P76  P33 + P34
P77  P33 + P76
P78  P77 + P35
Thread-2
Register File FU FU FU FU

11. Performance Implications of SMT
Single thread performance is likely to go down (caches, branch predictors, registers, etc. are shared) – this effect can be mitigated by trying to prioritize one thread
While fetching instructions, thread priority can dramatically influence total throughput – a widely accepted heuristic (ICOUNT): fetch such that each thread has an equal share of processor resources
With eight threads in a processor with many resources, SMT yields throughput improvements of roughly 2-4
Alpha and Intel Pentium 4 are examples of SMT

12. Design Challenges How many threads?
Many to find enough parallelism
However, mixing many threads will compromise execution of individual threads
Processor front-end (instruction fetch)
Fetch as far as possible in a single thread (to maximize thread performance)
However, this limits the number of instructions available for scheduling from other threads
Larger register files (multiple contexts)
Minimize clock cycle time
Cache conflicts

13. Pentium 4: Hyperthreading architecture
One physical processor appears as multiple logical processors
HT implementation on NetBurst microarchitecture has 2 logical processors
Architectural State
Architectural State
Architectural state:
general purpose registers
control registers
APIC: advanced programmable interrupt controller
Processor execution resources

14. Pentium 4: Hyperthreading architecture
Main processor resources are shared
caches, branch predictors, execution units, buses, control logic
Duplicated resources
register alias tables (map the architectural registers to physical rename registers)
next instruction pointer and associated control logic
return stack pointer
instruction streaming buffer and trace cache fill buffers

15. Pentium 4: Die Size and Complexity

16. Pentium 4: Resources sharing schemes
Partition – dedicate equal resources to each logical processors
Good when expect high utilization and somewhat unpredicatable
Threshold – flexible resource sharing with a limit on maximum resource usage
Good for small resources with bursty utilization and when the micro-ops stay in the structure for short predictable periods
Full sharing – flexible with no limits
Good for large structures, with variable working-set sizes

17. Pentium 4: Shared vs. partitioned queues
Cycle 0: 2 light shaded (fast) and 2 dark shaded (slow) microoperations
Cycle 1: send light shaded micro-op 0 down the pipeline
In the shared queue the previous pipeline stage sends dark shaded micro-op 2,
but in the partitioned queue the slow thread is already occupying two entries, so the previous stage will send light shaded micro-op 2.
(c) Cycle 3: Light micro-op 1 is sent down the pipeline.
Shared queue will accept light shaded micro-op 2, while the partitioned will accept the next light shaded micro-op 3.
(d) Cycle 3: light shaded micro-op 2 is sent down the pipeline.
In shared queue, dark shaded micro-op 3 is entering the queue,
but in the partitioned we will have to accept another light shaded micro-op 4.
(e) Finally, the shared queue keeps all dark shaded micro-ops (slow) and will block the other thread.
With the partitioned queue we do not have such problems.
shared
partitioned

18. NetBurst Pipeline
threshold
partitioned

19. Pentium 4: Shared vs. partitioned resources
E.g., major pipeline queues
Threshold
Puts a threshold on the number of resource entries a logical processor can have
E.g., scheduler
Fully shared resources
E.g., caches
Modest interference
Benefit if we have shared code and/or data

20. Pentium 4: Scheduler occupancy

21. Pentium 4: Shared vs. partitioned cache

22. Pentium 4: Performance Improvements

23. Multi-Programmed Speedup