Caches Vincent H. Berk October 21, 2005 ENGS 116 Lecture 12 Caches Vincent H. Berk October 21, 2005 Reading for Wednesday: Sections 5.1 – 5.4 Reading for Friday: Sections 5.5 – 5.8 (Jouppi article)

Who Cares about the Memory Hierarchy? ENGS 116 Lecture 12 Who Cares about the Memory Hierarchy? So far, have discussed only processor CPU Cost/Performance, ISA, Pipelined Execution, ILP 1980: no cache in microprocessors 1995: 2-level cache, 60% transistors on Alpha 21164 2002: IBM experimenting with Main Memory on die. CPU-DRAM Gap Problem: Memory speed increasing slower Solution: Multiple cache levels, Memory Hyrarchie!!!

The Motivation for Caches ENGS 116 Lecture 12 The Motivation for Caches Memory System Main Memory Processor Cache Motivation: Large memories (DRAM) are slow Small memories (SRAM) are fast Make the average access time small by servicing most accesses from a small, fast memory Reduce the bandwidth required of the large memory

Principle of Locality of Reference ENGS 116 Lecture 12 Principle of Locality of Reference Programs do not access their data or code all at once or with equal probability Rule of thumb: Program spends 90% of its execution time in only 10% of the code Programs access a small portion of the address space at any one time Programs tend to reuse data and instructions that they have recently used Implication of locality: Can predict with reasonable accuracy what instructions and data a program will use in the near future based on its accesses in the recent past

Memory System Illusion Reality Processor Processor Memory Memory ENGS 116 Lecture 12 Memory System Illusion Reality Processor Processor Memory Memory Memory Memory

General Principles Locality Temporal Locality: referenced again soon ENGS 116 Lecture 12 General Principles Locality Temporal Locality: referenced again soon Spatial Locality: nearby items referenced soon Locality + smaller HW is faster  memory hierarchy Levels: each smaller, faster, more expensive/byte than level below Inclusive: data found in top also found in lower levels Definitions Upper is closer to processor Block: minimum, address aligned unit that fits in cache Address = Block frame address + block offset address Hit time: time to access upper level, including hit determination

Cache Measures Hit rate: fraction of accesses found in that level ENGS 116 Lecture 12 Cache Measures Hit rate: fraction of accesses found in that level So high that we usually talk about the miss rate Miss rate fallacy: miss rate induces miss penalty, determines average memory performance Average memory-access time (AMAT) = Hit time + Miss rate  Miss penalty (ns or clocks) • Miss penalty: time to replace a block from lower level, including time to copy to and restart CPU – access time: time to lower level = ƒ(lower level latency) – transfer time: time to transfer block = ƒ(BW upper & lower, block size)

Block Size vs. Cache Measures ENGS 116 Lecture 12 Block Size vs. Cache Measures Increasing block size generally increases the miss penalty Miss Penalty Miss Rate  Avg. Memory Access Time => Miss penalty Transfer time Miss rate Average access time Access time Block Size Block Size Block Size

Key Points of Memory Hierarchy ENGS 116 Lecture 12 Key Points of Memory Hierarchy Need methods to give illusion of large, fast memory Programs exhibit both temporal locality and spatial locality Keep more recently accessed data closer to the processor Keep multiple contiguous words together in memory blocks Use smaller, faster memory close to processor – hits are processed quickly; misses require access to larger, slower memory If hit rate is high, memory hierarchy has access time close to that of highest (fastest) level and size equal to that of lowest (largest) level

ENGS 116 Lecture 12 Implications for CPU Fast hit check since every memory access needs this check Hit is the common case Unpredictable memory access time 10s of clock cycles: wait 1000s of clock cycles: (Operating System) » Interrupt & switch & do something else » Lightweight: multithreaded execution

Four Memory Hierarchy Questions ENGS 116 Lecture 12 Four Memory Hierarchy Questions Q1: Where can a block be placed in the upper level? (Block placement) Q2: How is a block found if it is in the upper level? (Block identification) • Q3: Which block should be replaced on a miss? (Block replacement) • Q4: What happens on a write? (Write strategy)

Q1: Where can a block be placed in the cache? ENGS 116 Lecture 12 Q1: Where can a block be placed in the cache? Block 12 placed in 8 block cache: Fully associative, direct mapped, 2-way set associative S.A. Mapping = Block number modulo number sets Fully associative: block 12 can go anywhere Direct mapped: block 12 can go only into block 4 (12 mod 8) Set associative: block 12 can go anywhere in set 0 (12 mod 4) Block no. 4 5 3 2 1 7 6 Block no. 4 5 3 2 1 7 6 Block no. 4 5 3 2 1 7 6 Cache Set Set 1 Set 2 Set 3 Block frame address Block no. 1 3 2 4 5 3 2 1 7 6 8 9 Memory

ENGS 116 Lecture 12 Direct Mapped Cache Each memory location is mapped to exactly one location in the cache Cache location assigned based on address of word in memory Mapping: (address of block) mod (# of blocks in cache) 000 001 010 011 100 101 110 111 00000 00100 01000 01100 10000 10100 11000 11100

ENGS 116 Lecture 12 Associative Caches Fully Associative: block can go anywhere in the cache N-way Set Associative: block can go in one of N locations in the set

Q2: How is a block found if it is in the cache? ENGS 116 Lecture 12 Q2: How is a block found if it is in the cache? Tag on each block No need to check index or block offset Increasing associativity shrinks index, expands tag Block Address Tag Index Block Offset Fully Associative: No index Direct Mapped: Large index

ENGS 116 Lecture 12 Examples 512-byte cache, 4-way set associative, 16-byte blocks, byte addressable 8-KB cache, 2-way set associative, 32-byte blocks, byte addressable

Q3: Which block should be replaced on a miss? ENGS 116 Lecture 12 Q3: Which block should be replaced on a miss? Easy for direct mapped Set associative or fully associative: Random (large associativities) LRU (smaller associativities) FIFO (large associativities) Associativity: 2-way 4-way Size LRU Random FIFO LRU Random FIFO 16 KB 114.1 117.3 115.5 111.7 115.1 113.3 64 KB 103.4 104.3 103.9 102.4 102.3 103.1 256 KB 92.2 92.1 92.5 92.1 92.1 92.5

Q4: What Happens on a Write? ENGS 116 Lecture 12 Q4: What Happens on a Write? Write through: The information is written to both the block in the cache and to the block in the lower-level memory. Write back: The information is written only to the block in the cache. The modified cache block is written to main memory only when it is replaced. Is block clean or dirty? Pros and Cons of each: WT: read misses cannot result in writes (because of replacements) WB: no writes of repeated writes WT always combined with write buffers so that we don’t wait for lower level memory WB write buffer, giving a read-miss precedence

CPU address Data Data in out ENGS 116 Lecture 12 Example: 21064 Data Cache Index = 8 bits: 256 blocks = 8192/(32  1) Direct Mapped Block address 1 Block offset <5> <21> <8> Index Tag CPU address Data Data in out 4 2 Valid<1> Tag <21> Data <256> (256 Blocks) =? 3 4:1 MUX • • • Write buffer Lower Level Memory

2-way Set Associative, Address to Select Word ENGS 116 Lecture 12 2-way Set Associative, Address to Select Word Block address Block offset <5> <22> <7> Index Tag CPU address Data Data in out Data <64> • • • Valid<1> Tag <21> • • • Two sets of address tags and data RAM 2:1 mux selects data =? 2:1 MU X • • • • • • Write buffer Use address bits to select correct RAM Lower Level Memory

Structural Hazard: Instruction and Data? ENGS 116 Lecture 12 Structural Hazard: Instruction and Data? Size Instruction Cache Data Cache Unified Cache 8 KB 8.16 44.0 63.0 16 KB 3.82 40.9 51.0 32 KB 1.36 38.4 43.3 64 KB 0.61 36.9 39.4 128 KB 0.30 35.3 36.2 256 KB 0.02 32.6 32.9 Misses per 1000 instructions Mix: instructions 74%, data 26%

ENGS 116 Lecture 12 Cache Performance includes hit time CPU time = (CPU execution clock cycles + Memory-stall clock cycles)  Clock cycle time Memory-stall clock cycles = Read-stall cycles + Write-stall cycles =

ENGS 116 Lecture 12 Cache Performance CPU time = IC  (CPIexecution + Mem accesses per instruction  Miss rate  Miss penalty)  Clock cycle time Misses per instruction = Memory accesses per instruction  Miss rate CPU time = IC  (CPIexecution + Misses per instruction  Miss penalty)  Clock cycle time

Summary of Cache Basics ENGS 116 Lecture 12 Summary of Cache Basics Associativity Block size (cache line size) Write Back/Write Through, write buffers, dirty bits AMAT as a basic performance measure Larger block size decreases miss rate but can increase miss penalty Can increase bandwidth of main memory to transfer cache blocks more efficiently Memory system can have significant impact on program execution time, memory stalls can be over 100 cycles Faster processors  memory stalls more costly

Improving Cache Performance ENGS 116 Lecture 12 Improving Cache Performance Average memory-access time (AMAT) = Hit time + Miss rate  Miss penalty (ns or clocks) Improve performance by: 1. Reducing the miss penalty (5.4) 2. Reducing the miss rate (5.5) 3. Reducing through parallelism (5.6) 4. Reducing the time to hit in the cache (5.7)

Reducing Miss Penalty Multilevel Caches ENGS 116 Lecture 12 Reducing Miss Penalty Multilevel Caches Critical Word First and Early Restart Read Misses over Writes Merging Write Buffer Victim Caches Subblock Placement

1. Reduce Miss Penalty: L2 Caches ENGS 116 Lecture 12 1. Reduce Miss Penalty: L2 Caches L2 Equations AMAT = Hit TimeL1 + Miss RateL1  Miss PenaltyL1 Miss PenaltyL1 = Hit TimeL2 + Miss RateL2  Miss PenaltyL2 AMAT = Hit TimeL1 + Miss RateL1  (Hit TimeL2 + Miss RateL2  Miss PenaltyL2) Definitions: Local miss rate — misses in this cache divided by the total number of memory accesses to this cache (Miss rateL2) Global miss rate — misses in the cache divided by the total number of memory accesses generated by the CPU (Miss RateL1  Miss RateL2) Global miss rate is what matters —indicates what fraction of memory accesses from CPU go all the way to main memory

Comparing Local and Global Miss Rates ENGS 116 Lecture 12 Comparing Local and Global Miss Rates 32 KByte 1st level cache; Increasing 2nd level cache Global miss rate close to single level cache rate provided L2 >> L1 Don’t use local miss rate L2 not tied to CPU clock cycle! Cost & A.M.A.T. Generally fast hit times and fewer misses Since hits are few, target miss reduction Linear Log Cache Size

L2 cache block size & A.M.A.T. ENGS 116 Lecture 12 L2 cache block size & A.M.A.T. 32KB L1, 8-byte path to memory

2 . Reduce Miss Penalty: Early Restart and Critical Word First ENGS 116 Lecture 12 2 . Reduce Miss Penalty: Early Restart and Critical Word First Don’t wait for full block to be loaded before restarting CPU Early restart — As soon as the requested word of the block arrives, send it to the CPU and let the CPU continue execution Critical Word First — Request the missed word first from memory and send it to the CPU as soon as it arrives; let the CPU continue execution while filling the rest of the words in the block. Also called wrapped fetch and requested word first. Generally useful only in large blocks, Spatial locality a problem; tend to want next sequential word, so not clear if benefit by early restart block

3. Reduce Miss Penalty: Read Priority over Write on Miss ENGS 116 Lecture 12 3. Reduce Miss Penalty: Read Priority over Write on Miss Write through with write buffers offer RAW conflicts with main memory reads on cache misses If simply wait for write buffer to empty, might increase read miss penalty (old MIPS 1000 by 50%) Check write buffer contents before read; if no conflicts, let the memory access continue Write Back? Read miss replacing dirty block Normal: Write dirty block to memory, and then do the read Instead copy the dirty block to a write buffer, then do the read, and then do the write CPU stalls less frequently since restarts as soon as read finished

4. Reduce Miss Penalty by Merging Write Buffer ENGS 116 Lecture 12 4. Reduce Miss Penalty by Merging Write Buffer Write merging in write buffer Write Address V 100 104 108 112 1 4 entry, 4 word Write Address V 100 1 16 sequential writes in a row

5. Reduce Miss Penalty via a “Victim Cache” ENGS 116 Lecture 12 5. Reduce Miss Penalty via a “Victim Cache” How to combine fast hit time of direct mapped yet still avoid conflict misses? Add buffer to place data discarded from cache Jouppi [1990]: 4-entry victim cache removed 20% to 95% of conflicts for a 4-KB direct-mapped data cache Used in Alpha, HP machines CPU address Data Data in out Tag Data =? Victim cache =? Write buffer Lower Level Memory

6. Reduce Miss Penalty: Subblock Placement ENGS 116 Lecture 12 6. Reduce Miss Penalty: Subblock Placement Don’t have to load full block on a miss Have valid bits per subblock to indicate valid (Originally invented to reduce tag storage) 100 300 200 204 1 1 1 1 1 1 1 1 Valid Bits Subblocks

Reducing Miss Rate Larger Block Size Larger Caches ENGS 116 Lecture 12 Reducing Miss Rate Larger Block Size Larger Caches Higher Associativity Way Prediction and Pseudoassociative Caches Compiler Optimizations: Merging Arrays Loop Interchange Loop Fusion Blocking

Classifying Misses: 3 Cs ENGS 116 Lecture 12 Classifying Misses: 3 Cs Compulsory: The first access to a block is not in the cache, so the block must be brought into the cache. Also called cold start misses or first reference misses. (Misses even in an infinite cache) Capacity: If the cache cannot contain all the blocks needed during execution of a program, capacity misses will occur due to blocks being discarded and later retrieved. (Misses in fully associative, size X cache) Conflict: If block-placement strategy is set associative or direct mapped, conflict misses (in addition to compulsory & capacity misses) will occur because a block can be discarded and later retrieved if too many blocks map to its set. Also called collision misses or interference misses. (Misses in N-way set associative, size X cache) Intuitive Model by Mark Hill

3Cs Absolute Miss Rate (SPEC92) ENGS 116 Lecture 12 3Cs Absolute Miss Rate (SPEC92) Cache Size (KB) Miss Rate per Type 0.02 0.04 0.06 0.08 0.1 0.12 0.14 1 2 4 8 16 32 64 128 1-way 2-way 4-way 8-way Capacity Compulsory Compulsory vanishingly small Conflict

ENGS 116 Lecture 12 2:1 Cache Rule miss rate 1-way associative cache size X = miss rate 2-way associative cache size X/2 Cache Size (KB) Miss Rate per Type 0.02 0.04 0.06 0.08 0.1 0.12 0.14 1 2 4 8 16 32 64 128 1-way 2-way 4-way 8-way Capacity Compulsory Conflict

3Cs Relative Miss Rate Cache Size (KB) Miss Rate per Type 0% 20% 40% ENGS 116 Lecture 12 3Cs Relative Miss Rate Cache Size (KB) Miss Rate per Type 0% 20% 40% 60% 80% 100% 1 2 4 8 16 32 64 128 1-way 2-way 4-way 8-way Capacity Compulsory Conflict Flaws: for fixed block size Good: insight

How Can We Reduce Misses? ENGS 116 Lecture 12 How Can We Reduce Misses? 3 Cs: Compulsory, Capacity, Conflict In all cases, assume total cache size not changed What happens if we: 1) Change Block Size: Which of 3Cs is obviously affected? 2) Change Associativity: Which of 3Cs is obviously affected? 3) Change Compiler: Which of 3Cs is obviously affected? Ask which affected? Block size 1) Compulsory 2) More subtle, will change mapping

1. Reduce Misses via Larger Block Size ENGS 116 Lecture 12 1. Reduce Misses via Larger Block Size

2. Reduce Misses: Larger Cache Size ENGS 116 Lecture 12 2. Reduce Misses: Larger Cache Size Obvious improvement but: Longer hit time Higher cost Each cache size favors a block-size, based on memory bandwidth

3. Reduce Misses via Higher Associativity ENGS 116 Lecture 12 3. Reduce Misses via Higher Associativity 2:1 Cache Rule: Miss Rate DM cache size N ≈ Miss Rate 2-way SA cache size N/2 Beware: Execution time is final measure! Will clock cycle time increase? 8-Way is almost fully associative

Example: Avg. Memory Access Time vs. Miss Rate ENGS 116 Lecture 12 Example: Avg. Memory Access Time vs. Miss Rate Example: assume CCT = 1.10 for 2-way, 1.12 for 4-way, 1.14 for 8-way vs. CCT direct mapped Cache Size Associativity (KB) 1-way 2-way 4-way 8-way 1 2.33 2.15 2.07 2.01 2 1.98 1.86 1.76 1.68 4 1.72 1.67 1.61 1.53 8 1.46 1.48 1.47 1.43 16 1.29 1.32 1.32 1.32 32 1.20 1.24 1.25 1.27 64 1.14 1.20 1.21 1.23 128 1.10 1.17 1.18 1.20 (Red means A.M.A.T. not improved by more associativity)

Reducing Misses via “Pseudo-Associativity” or way prediction ENGS 116 Lecture 12 Reducing Misses via “Pseudo-Associativity” or way prediction How to combine fast hit time of Direct Mapped and have the lower conflict misses of 2-way SA cache? Divide cache: on a miss, check other half of cache to see if there, if so have a pseudo-hit (slow hit) Way Prediction: keep prediction bits to decide what comparison is made first Drawback: CPU pipeline is hard if hit takes 1 or 2 cycles Better for caches not tied directly to processor (L2) Used in MIPS R1000 L2 cache, similar in UltraSPARC Hit Time Pseudo Hit Time Miss Penalty Time