1. Mehmet Altan Açıkgöz Ercan Saraç
CMPE 511 Multithreading
Mehmet Altan Açıkgöz
Ercan Saraç

2. Outline Multithreading Thread scheduling policies Grain multithreading
Design choices
Multi threading architecture
Out-of-order superscalar processor
Simultaneous multithreading
From superscalar to SMT
SMT design issues

3. Pipeline Hazards Without bypassing, need interlocks LW r1, 0(r2)
ADDI r5, r5, #12
SW 12(r1), r5
Each instruction may depend on the next
Without bypassing, need interlocks
Bypassing cannot completely eliminate interlocks or delay slots

4. Multithreading
How can we guarantee no dependencies between instructions in a pipeline?
One way is to interleave execution of instructions from different program threads on same pipeline
Interleave 4 threads, T1-T4, on non-bypassed 5-stage pipe
T1: LW r1, 0(r2)
T2: ADD r7, r1, r4
T3: XORI r5, r4, #12
T4: SW 0(r7), r5
T1: LW r5, 12(r1)

5. CDC 6600 Peripheral Processors (Cray, 1965)
First multithreaded hardware
10 “virtual” I/O processors
fixed interleave on simple pipeline
pipeline has 100ns cycle time
each processor executes one instruction every 1000ns
accumulator-based instruction set to reduce processor state
The 6600 CDC included 10 parallel functional units,
allowing multiple instructions to be worked on at the same time.
Today this is known as a superscalar design, while at the time it was simply "unique".
The system read and decoded instructions from memory as fast as possible,
generally faster than they could be completed, and fed them off to the units for processing.

6. Simple Multithreaded Pipeline
General purpose registers (GPRs) can store both data and addresses, i.e., they are combined Data/Address registers.
Have to carry thread select down pipeline to ensure correct state bits read/written at each pipe stage

7. Multithreading Costs
Appears to software (including OS) as multiple slower CPUs
Each thread requires its own user state
GPRs PC
Also, needs own OS control state
virtual memory page table base register
exception handling registers
Other costs?
General purpose registers (GPRs) can store both data and addresses, i.e., they are combined Data/Address registers.

8. Thread Scheduling Policies
Fixed interleave (CDC 6600 PPUs, 1965)
each of N threads executes one instruction every N cycles
if thread not ready to go in its slot, insert pipeline bubble
Software-controlled interleave (TI ASC PPUs, 1971)
OS allocates S pipeline slots amongst N threads
hardware performs fixed interleave over S slots, executing whichever thread is in that slot
Hardware-controlled thread scheduling (HEP, 1982)
hardware keeps track of which threads are ready to go
picks next thread to execute based on hardware priority
scheme
The Advanced Scientific Computer, or ASC, was a supercomputer architecture designed by Texas Instruments (TI) between 1966 and Key to the ASC's design was a single high-speed shared memory, which was accessed by a number of processors and channel controllers, in a fashion similar to Seymour Cray's groundbreaking CDC One key difference was that the CPU was split in two, the operating system ran on the "peripheral processor", while applications were run on the dedicated, slave, ALU, which they referred to as "the" CPU. The ALU/CPU was one of the first to include vector processing instructions.
The Heterogeneous Element Processor (HEP) was introduced by Denelcor in 1982 as the world's first commercial MIMD computer. A HEP system, as the name implies, was pieced together from many heterogeneous components -- processors, data memory modules, and I/O modules. The components were connected via a switched network.

9. What “Grain” Multithreading?
So far assumed fine-grained multithreading
CPU switches every cycle to a different thread
Coarse-grained multithreading
CPU switches every few cycles to a different thread

10. Multithreading Design Choices
Context switch to another thread every cycle, or on hazard or L1 miss or L2 miss or network request
Per-thread state and context-switch overhead
Interactions between threads in memory hierarchy

11. Denelcor HEP (Burton Smith, 1982)
First commercial machine to use hardware threading in main CPU
120 threads per processor
10 MHz clock rate
Up to 8 processors
precursor to Tera MTA (Multithreaded Architecture)

12. Tera MTA Overview Up to 256 processors
Up to 128 active threads per processor
Processors and memory modules populate a sparse 3D torus interconnection fabric
Flat, shared main memory
No data cache
Sustains one main memory access per cycle per processor
260MHz

13. MTA Instruction Format
Three operations packed into 64-bit instruction word (short VLIW)
One memory operation, one arithmetic operation, plus one arithmetic or branch operation
Memory operations incur ~150 cycles of latency
Explicit 3-bit “lookahead” field in instruction gives number of subsequent instructions (0-7) that are independent of this one
c.f. Instruction grouping in VLIW
allows fewer threads to fill machine pipeline
used for variable- sized branch delay slots
Thread creation and termination instructions

14. MTA Multithreading Each processor supports 128 active hardware threads
128 SSWs, 1024 target registers, 4096 general-purpose registers
Every cycle, one instruction from one active thread is launched into pipeline
Instruction pipeline is 21 cycles long
At best, a single thread can issue one instruction every 21 cycles
Clock rate is 260MHz, effective single thread issue rate is 260/21 = 12.4MHz

15. MTA Pipeline
Although the memory in the MTA is physically distributed, the system is emphatically presented as a shared memory machine (with non-uniform access time). The latency incurred in memory references is hidden by multi-threading, i.e., usually many concurrent program threads (instruction streams) may be active at any time. Therefore, when for instance a load instruction cannot be satisfied because of memory latency the thread requesting this operation is stalled and another thread of which an operation can be done is switched into execution. This switching between program threads only takes 1 cycle. As there may be up to 128 instruction streams and 8 memory references can be issued without waiting for preceding ones, a latency of 1024 cycles can be tolerated. References that are stalled are retried from a retry pool.

16. Coarse-Grain Multithreading
Tera MTA designed for supercomputing applications with large data sets and low locality
No data cache
Many parallel threads needed to hide large memory latency
Other applications are more cache friendly
Few pipeline bubbles when cache getting hits
Just add a few threads to hide occasional cache miss latencies
Swap threads on cache misses

17. MIT Alewife Modified SPARC chips Up to four threads per node
register windows hold different thread contexts
Up to four threads per node
Thread switch on local cache miss
SPARC (Scalable Processor ARChitecture) is a RISC microprocessor instruction set architecture originally designed in 1985 by Sun Microsystems

18. IBM PowerPC RS64-III (Pulsar)
Commercial coarse-grain multithreading CPU
Based on PowerPC with quad-issue in-order fivestage pipeline
Each physical CPU supports two virtual CPUs
On L2 cache miss, pipeline is flushed and execution switches to second thread
short pipeline minimizes flush penalty (4 cycles), small compared to memory access latency
flush pipeline to simplify exception handling
The Pulsar was introduced in 1999 at 450 MHz with an 8 MB DDR SRAM L2 running at 450 MHz effective data rate on a 256 bit bus. On-chip L1 was increased to 256 KB total. Branch prediction was improved and branch misprediction penalty reduced to 0 or 1 cycles.

19. Speculative, Out-of-Order Superscalar Processor
Superscalar processing is the ability to initiate multiple instructions during the same cycle.
It aims at producing ever faster microprocessors.
A typical superscalar processor fetches and decodes several instructions at a time. Instructions are executed in parallel based on the availability of operand data rather than their original program sequence. Upon completion instructions are re-sequenced so that they can be used to update the process state in the correct program order.

20. Out-Of-Order Execution
Why Out-Of-Order Execution?
In-order Processors Stalls
Pipeline may not be full because of
the frequent stalls
Example:
Allow In the Out-Of-Order Processors
- No dependency Move the Instruction for execution
- Means the Instructions that are Ready

21. Out-of-order processors
This new paradigm breaks up the processing of instructions into these steps:
Instruction fetch.
Instruction dispatch to an instruction queue (also called instruction buffer or reservation stations).
The instruction waits in the queue until its input operands are available. The instruction is then allowed to leave the queue before earlier, older instructions.
The instruction is issued to the appropriate functional unit and executed by that unit.
The results are queued.
When all older results have been written back to the register file, then this result is written back to the register file. This is called the graduation or retire stage.
The key concept of OoO processing is to allow the processor to avoid a class of stalls that occur when the data needed to perform an operation is not available. In the outline above, the OoO processor avoids the stall that occurs in step (2) of the in-order processor when the instruction is not completely ready to be processed due to missing data

22. Issue Queues
A list of pending instructions is kept and each cycle these queues select from these instructions as their input data are ready.
Queues issue instructions speculatively and older instructions are given priority over newer in the queue.
An issue queue entry becomes available when the instruction issues or is squashed due to mis-speculation.
Superscalar processing is the ability to initiate multiple instructions during the same cycle.
It aims at producing ever faster microprocessors.
A typical superscalar processor fetches and decodes several instructions at a time. Instructions are executed in parallel based on the availability of operand data rather than their original program sequence. Upon completion instructions are re-sequenced so that they can be used to update the process state in the correct program order.

23. Superscalar Machine Efficiency

24. Vertical Multithreading
Cycle-by-cycle interleaving of second thread removes vertical waste

25. Ideal Multithreading for Superscalar
Interleave multiple threads to multiple issue slots with no restrictions

26. Simultaneous Multithreading
Allows multiple threads to execute different instructions in the same clock cycle, using the execution units that the first thread left spare.
The main additions needed are the ability to fetch instructions from multiple threads in a cycle, and a larger register file to hold data from multiple threads.
Normal multithreading operating systems allow multiple processes and threads to utilize the processor one at a time, giving exclusive ownership to a particular thread for a time slice in the order of milliseconds - this is called Temporal multithreading. Quite often, a process will stall for hundreds of cycles while waiting for some external resource (for example, a RAM load), thus lowering processor efficiency.
A successive improvement is super-threading, where the processor can execute instructions from a different thread each cycle. Thus cycles left unused by a thread can be used by another that is ready to run.

27. Simultaneous Multithreading
Add multiple contexts and fetch engines to wide out-of-order superscalar processor
OOO instruction window already has most of the circuitry required to schedule from multiple threads
Any single thread can utilize whole machine

28. Comparison of Issue Capabilities

29. From Superscalar to SMT
SMT is an out-of-order superscalar extended with hardware to support multiple executing threads

30. From Superscalar to SMT
Extra pipeline stages for accessing thread-shared register files

31. From Superscalar to SMT
Fetch from the two highest throughput threads.

32. From Superscalar to SMT
Small items
per-thread program counters
per-thread return stacks
per-thread bookkeeping for instruction retirement, trap & instruction dispatch queue flush
thread identifiers, e.g., with BTB & TLB entries

33. Simultaneous Multithreaded Processor

34. SMT Design Issues Which thread to fetch from next? Locks
Don’t want to clog instruction window with thread with many stalls  try to fetch from thread that has fewest insts in window
Locks
Virtual CPU spinning on lock executes many instructions but gets nowhere  add ISA support to lower priority of thread spinning on lock

35. Intel Pentium-4 Xeon Processor
Hyperthreading == SMT
Dual physical processors, each 2-way SMT
Logical processors share nearly all resources of the physical processor
Caches, execution units, branch predictors
When one logical processor is stalled, the other can make progress
No logical processor can use all entries in queues when two threads are active
A processor running only one active software thread to run at the same speed with or without hyperthreading

36. References
The anatomy of a modern superscalar processor, Constantinos Kourouyiannis, Madhava Rao Andagunda
Multithreading, L.N. Bhuyan