1. Summary 3 Cs: Compulsory, Capacity, Conflict Misses Reducing Miss Rate
1. Larger Block Size
2. Higher Associativity
3. Victim Cache
4. Pseudo-Associativity
5. HW Prefetching Instr, Data
6. SW Prefetching Data
7. Compiler Optimizations
Danger of concentrating on just one parameter when evaluating performance

2. Overview Reducing Miss Penalty Reducing Hit Time
Giving priority to read misses over writes
Sub-block placement
Early restart and critical word first
Nonblocking caches
Second-level caches
Reducing Hit Time
Small and simple caches
Avoiding address translation
Pipelining writes
Small subblocks

3. 1. Giving Priority to Read Misses
Write buffers complicate memory access
RAW hazard in main memory on cache misses
SW 512(R0), R3 (cache index 0)
LW R1, 1024(R0) (cache index 0)
LW R2, 512(R0) (cache index 0)
Wait for write buffer to empty?
Might increase read miss penalty
Check write buffer contents before read
If no conflicts, let the memory access continue
Write Back: Read miss replacing dirty block
Normal: Write dirty block to memory, then do the read
Optimized: copy dirty block to write buffer, then do the read
More optimization: write merging

4. 2. Sub-block Placement Don’t have to load full block on a miss
Valid bits per subblock indicate valid data
Tag
Data 1 1 1 1 1 1 1 1
sub-blocks

5. 3. Early Restart Don’t wait for full block to be loaded
Early restart—As soon as the requested word arrives, send it to the CPU and let the CPU continue execution
Critical Word First—Request the missed word first and send it to the CPU as soon as it arrives; then fill in the rest of the words in the block.
Generally useful only in large blocks
Extremely good spatial locality can reduce impact
Back to back reads on two halves of cache block does not save you much (see example in book)
Need to schedule instructions!

6. 4. Nonblocking Caches
Non-blocking caches continue to supply cache hits during a miss
requires out-of-order execution CPU
“hit under miss” reduces the effective miss penalty by working during miss vs. ignoring CPU requests
“hit under multiple miss” may further lower the effective miss penalty by overlapping multiple misses
Significantly increases the complexity of the cache controller
Requires multiple memory banks (otherwise cannot support)
Pentium Pro allows 4 outstanding memory misses

7. Hit Under Miss FP: AMAT= 0.68 -> 0.52 -> 0.34 -> 0.26
Hit Under i Misses
FP: AMAT= > > > 0.26
Int:: AMAT= > > > 0.19
8 KB Data Cache, Direct Mapped, 32B block, 16 cycle penalty 2
1.8
1.6
1.4
0->1
Avg. Mem. Access Time
1.2
1->2 1
2->64
0.8
Base
0.6
0.4
0.2
ear
wave5
nasa7
ora
xlisp
doduc
su2cor
eqntott
espresso
compress
mdljsp2
fpppp
swm256
mdljdp2
hydro2d
alvinn
spice2g6
tomcatv

8. 5. Second-Level Caches L2 Equations Definitions:
AMAT = Hit TimeL1 + Miss RateL1 x Miss PenaltyL1
Miss PenaltyL1 = Hit TimeL2 + Miss RateL2 x Miss PenaltyL2
AMAT = Hit TimeL1 + Miss RateL1 x (Hit TimeL2 +Miss RateL2 +Miss PenaltyL2)
Definitions:
Local miss rate— misses in this cache divided by the total number of memory accesses to this cache (Miss rateL2)
Global miss rate—misses in this cache divided by the total number of memory accesses generated by the CPU (Miss RateL1 x Miss RateL2)
Global Miss Rate is what matters

9. Local and Global Miss Rates
32 KByte L1 cache;
Global miss rate close to single level cache rate provided L2 >> L1
local miss rate
Do not use to measure impact
Use in equation!
L2 not tied to clock cycle!
Target miss reduction

10. Reducing L2 Miss Rate Reducing Miss Rate
Reduce Misses via Larger Block Size
Inclusion is useful (for consistency)
Can complicate design
 L2 cache miss (need to invalidate L1)
2. Reduce Conflict Misses via Higher Associativity
3. Reducing Conflict Misses via Victim Cache
4. Reducing Conflict Misses via Pseudo-Associativity
5. Reducing Misses by HW Prefetching Instr, Data
6. Reducing Misses by SW Prefetching Data
7. Reducing Capacity/Conf. Misses by Compiler Optimizations

11. Miss Penalty Summary Five techniques
Read priority over write on miss
Subblock placement
Early Restart and Critical Word First on miss
Non-blocking Caches (Hit under Miss)
L2 Cache
Can be applied recursively to Multilevel Caches
Danger: time to DRAM will grow with multiple level

12. Reducing Hit Time Hit time affects the CPU clock rate Techniques
Even for machines that take multiple cycles to access the cache
Techniques
Small and simple caches
Avoiding address translation
Pipelining writes

13. 1. Small and Simple Caches
Small hardware is faster
Fits on the same chip as the processor
Alpha has 8KB Instruction and 8KB data cache + 96KB second level cache?
Small data cache and fast clock rate
Direct Mapped, on chip
Overlap tag check with data transmission
For L2 keep tag check on chip, data off chip  fast tag check, large capacity associated with separate memory chip

14. 2. Avoiding Address Translation
Virtually Addressed Cache (vs. Physical Cache)
Send virtual address to cache.
Every time process is switched must flush the cache;
Cost: time to flush + “compulsory” misses from empty cache
Dealing with aliases (two different virtual addresses map to same physical address)
I/O must interact with cache, so need virtual address
Solution to aliases
HW guarantees that every cache block has unique PA
SW guarantee (page coloring): lower n bits must have same address; as long as covers index field & direct mapped, they must be unique;
Solution to cache flush
PID tag that identifies process and address within process

15. Virtual Addressed Caches
CPU
CPU
CPU VA VA VA VA
Tags TB $ PA
Tags $ TB PA VA PA
L2 $ $ TB
MEM PA PA
MEM
MEM
Overlap $ access
with VA translation:
requires $ index to
remain invariant
across translation
Conventional
Organization
Virtually Addressed Cache
Translate only on miss
Synonym Problem

16. Process ID Impact

17. Index with Physical Portion of Address
If index is physical part of address, can start tag access in parallel with translation so that can compare to physical tag
Limits cache to page size: what if want bigger caches and uses same trick?
Larger page sizes
Higher associativity
Index = log(Cache Size/[block size*associativity])
Page coloring 12 11 31
Page offset
Index Block offset
Page address
Addres tag

18. 3. Pipelined Writes CPU W1 W1 W2 Data R1/W1 R1 Delayed write buffer M
Address
Data Data
in out =? W1
Tag
Delayed write buffer W1 W2
Data M u x =?
Write buffer
R1/W1 R1
Lower level memory