1. An Introduction to Cache Design
2018/11/19
\course\cpeg323-05F\Topic7a

2. \course\cpeg323-05F\Topic7a
Cache
A safe place for hiding and storing things.
Webster Dictionary
2018/11/19
\course\cpeg323-05F\Topic7a

3. \course\cpeg323-05F\Topic7a
Even with the inclusion of cache, almost all CPUs are still mostly strictly limited by the cache access-time:
In most cases, if the cache access time were decreased, the machine would speedup accordingly.
- Alan Smith -
Even more so for MPs!
2018/11/19
\course\cpeg323-05F\Topic7a

4. \course\cpeg323-05F\Topic7a
While one can imagine ref. patterns that can defeat existing cache M designs, it is the author’s experience that cache M improve performance for any program or workload which actually does useful computation.
2018/11/19
\course\cpeg323-05F\Topic7a

5. \course\cpeg323-05F\Topic7a
Optimizing the design of a cache memory generally has four aspects:
Maximizing the probability of finding a memory reference’s target in the cache (the hit ratio),
Minimizing the time to access information that is indeed in the cache (access time),
Minimizing the delay due to a miss, and
Minimizing the overheads of updating main memory, maintaining cache coherence etc.
2018/11/19
\course\cpeg323-05F\Topic7a

6. Key Factor in Design Decision for VM and Cache
Access-timeMainMem
Access-timeCache
Access-timeSecondaryMem
= 4 ~ 20 .
= 104 ~ 106 .
Cache control is usually implemented in hardware.
2018/11/19
\course\cpeg323-05F\Topic7a

7. \course\cpeg323-05F\Topic7a
Technology in 1990s:
Technology in 2000s ?
2018/11/19
\course\cpeg323-05F\Topic7a

8. \course\cpeg323-05F\Topic7a
Technology in 2000s:
2018/11/19
\course\cpeg323-05F\Topic7a

9. \course\cpeg323-05F\Topic7a
Secondary
Memory
Main
Memory
Processor
Cache
Cache in Memory Hierarchy
2018/11/19
\course\cpeg323-05F\Topic7a

10. \course\cpeg323-05F\Topic7a
2018/11/19
\course\cpeg323-05F\Topic7a

11. \course\cpeg323-05F\Topic7a
Four Questions for Classifying Memory Hierarchies:
The fundamental principles that drive all memory hierarchies allow us to use terms that transcend the levels we are talking about. These same principles allow us to pose four questions about any level of the hierarchy:
2018/11/19
\course\cpeg323-05F\Topic7a

12. \course\cpeg323-05F\Topic7a
Four Questions for Classifying Memory Hierarchies
Q1: Where can a block be placed in the upper level? (Block placement)
Q2: How is a block found if it is in the upper
level? (Block identification)
Q3: Which block should be replaced on a miss? (Block replacement)
Q4: What happens on a write? (Write strategy)
2018/11/19
\course\cpeg323-05F\Topic7a

13. \course\cpeg323-05F\Topic7a
These questions will help us gain an understanding of the different tradeoffs demanded by the relationships of memories at different levels of a hierarchy.
2018/11/19
\course\cpeg323-05F\Topic7a

14. \course\cpeg323-05F\Topic7a
TAGS DATA
0117X 35, 72, 55, 30, 64, 23, 16, 14
7620X 11, 31, 26, 22, 55, …
3656X 71, 72, 44, 50, …
1741X 33, 35, 07, 65, ...
Line
01173 30
ADDRESS DATA
Concept of Cache miss and Cache hit
2018/11/19
\course\cpeg323-05F\Topic7a

15. \course\cpeg323-05F\Topic7a
teff : effective cache access time
tcache : cache access time
tmain : main memory access time
h : hit ratio
teff = htcache + (1-h)tmain
2018/11/19
\course\cpeg323-05F\Topic7a

16. \course\cpeg323-05F\Topic7a
Example
Let tcache = 10 ns clock cycles
tmain = 50 ns clock cycles
h = 0.95
teffect = ?
10 x x 0.05 =
2018/11/19
\course\cpeg323-05F\Topic7a

17. \course\cpeg323-05F\Topic7a
Hit Ratio
Need high enough (say > 90%) to obtain desirable level of performance
Amplifying effect of changes
Never a constant even for the same machine
2018/11/19
\course\cpeg323-05F\Topic7a

18. Sensitivity of Performance w.r.t h (hit ratio)
teff = h tcache + (1-h) tmain
= tcache [ h + (1-h) ]
tcache [ 1 + (1-h) ]
since 10, the magnifactor of h changes is 10 times.
Conclusion: very sensitive
tmain
tcache
tmain
tcache
tmain
tcache ~
2018/11/19
\course\cpeg323-05F\Topic7a

19. \course\cpeg323-05F\Topic7a
Remember:
“h 1”
Example:
Let h = 0.90
if h = ( )
then (1 - h) = 0.05
then teff = tcache ( ) ~
2018/11/19
\course\cpeg323-05F\Topic7a

20. \course\cpeg323-05F\Topic7a
Basic Terminology
Cache line (block) - size of a room
1 ~ 16 words
Cache directory - key of rooms
Cache may use associativity to find the “right directory” by matching
“A collection of contiguous
data that are treated as a single
entity of cache storage.”
The portion of a cache that
holds the access keys that
support associative access.
2018/11/19
\course\cpeg323-05F\Topic7a

21. \course\cpeg323-05F\Topic7a
Cache Organization
Fully associative: an element can be in any block
Direct mapping : an element can be in only one block
Set-associative : an element can be in a group of block
2018/11/19
\course\cpeg323-05F\Topic7a

22. \course\cpeg323-05F\Topic7a
An Example
Mem Size = 256 k words x 4B/W = 1 MB
Cache Size = 2 k words = 8 K byte
Block Size = word/block = 64 byte/block So
Main M has = 16 K blocks (16,384)
Cache has = blocks
addr = 18 bit + 2 = (28 x 210) x 22
256K 16 2K 16
(byte) 20
256 K
words
2018/11/19
\course\cpeg323-05F\Topic7a

23. \course\cpeg323-05F\Topic7a
Fully Associative
Feature
any block in M can be in any block-frame in cache
all entries (block frame) are compared simultaneously (by associative search)
2018/11/19
\course\cpeg323-05F\Topic7a

24. \course\cpeg323-05F\Topic7a
A Special Case
simplest example: a block = a word
entire memory word address becomes
“tag”
Address
027560
very “flixible” and higher
probability to reside in cache.
Cache
adv: no trashing (quick reorganizing)
disadv: overhead of associative search:
cost + time
data
2018/11/19
\course\cpeg323-05F\Topic7a

25. \course\cpeg323-05F\Topic7a
Tag
Block 0
Block 1
Block 127
… … … … … ...
Block I
Block 16382
Block 16383
14 bits
Main memory address
tag word
Recall: each block has 16 word – so you need 4 bits
Fully associative cache organization
2018/11/19
\course\cpeg323-05F\Topic7a

26. \course\cpeg323-05F\Topic7a
Direct Mapping
No associative match
From M-addr, “directly” indexed to the block frame in cache where the block should be located. A comparison then is to used to determine if it is a miss or hit.
2018/11/19
\course\cpeg323-05F\Topic7a

27. \course\cpeg323-05F\Topic7a
Direct Mapping
Cont’d
Advantage:
simplest:
Disadvantage: “trashing”
Fast (fewer logic)
Low cost: (only one set comparator is needed
hence can be in the form of standard M
2018/11/19
\course\cpeg323-05F\Topic7a

28. \course\cpeg323-05F\Topic7a
Direct Mapping
Cont’d
Example:
since cache only has 128 block frames so the degree of multiplexing:
Disadr: “trashing”
for addressing
the corresponding
frame or set of
size 1.
Main Memory Size (block)
128 (27)
= = 27 block/frame
the high-order
7 bit is used
as tag.
i.e. 27 blocks “fall” in one block frame.
2018/11/19
\course\cpeg323-05F\Topic7a

29. \course\cpeg323-05F\Topic7a
Main memory
Cache
Block 0
Block 1
Block 2
Block 127
Block 128
Block 129
Block 255
Block 256
Block 257
Block 4095
Block 4096
Block 16383
7 bits
Tag
...
Block 0
Block 1
Block 127
Tag
...
… … … … … ...
Tag
...
Tag
...
Main memory address
Tag Block Word
Direct Mapping
2018/11/19
\course\cpeg323-05F\Topic7a

30. \course\cpeg323-05F\Topic7a
Direct Mapping
Cont’d
Mapping (indexing) block addr mod (# of blocks in cache –
in this case: mod (27))
Adv: low-order log2 (cache size) bit can be used for indexing
2018/11/19
\course\cpeg323-05F\Topic7a

31. \course\cpeg323-05F\Topic7a
Set-Associative
A compromises between direct/full-associative
The cache is divided into S sets
S = 2, 4, 8, …
If the cache has M blocks
than, all together, there are
E = blocks/set
# of buildings available for indexing M S
In our example, S = 128/2 = 64 sets
2018/11/19
\course\cpeg323-05F\Topic7a

32. \course\cpeg323-05F\Topic7a
The 6-bit will index to the right set, then the 8-bit tag will be used for an
associative match.
Main memory
Cache
Block 0
Block 1
Block 63
Block 64
Block 65
Block 4095
Block 16383
8 bits
...
Tag
Block 0
Block 1
Block 2
Block 3
Block 126
Block 127
Set 0
Tag
Set 1
Tag
...
Tag
Set 63
Tag
Tag
Main memory address
Tag Set Word
2018/11/19
\course\cpeg323-05F\Topic7a

33. \course\cpeg323-05F\Topic7a
a 2-way set associative organization:
Set Word
available for indexing
thus or
6 bit used to index
into the right “set”
higher order
214 (16k) 26
= 28 block/set
28 block/per set of 2 blocks
2 way
8 bit used as tag
hence an associative
match of 8 bit with
the tags of the 2
blocks is required
Hence an associative matching of 8 bit
with the tags of the 2 block is required.
2018/11/19
\course\cpeg323-05F\Topic7a

34. \course\cpeg323-05F\Topic7a
Sector Mapping Cache
Sector (IBM 360/85) - 16 sector x 16 block/sector
1 sector = consecutive multiple blocks
Cache miss: sector replacement
Valid bit - one block is moved on demand
Example:
Sector block word
(tag)
A sector in memory can be in any sector in cache
2018/11/19
\course\cpeg323-05F\Topic7a

35. \course\cpeg323-05F\Topic7a
Valid
bit
Block 0
Block 1
Block 15
Block 16
Block 31
Block 16368
Block 16383
Sector 0
Sector 0
...
Tag
Block 0
Block 1
Block 14
Block 15
Block 16
Block 31
Block 112
Block 127
Sector 0
...
Sector 1
Tag
...
Sector 1
Tag
Sector 7
...
...
Sector 1023
Main memory address
Sector Block Word
(tag)
Sector mapping cache
2018/11/19
\course\cpeg323-05F\Topic7a

36. \course\cpeg323-05F\Topic7a
cont’d
128 blocks
16 blocks/sector
Cache has = 8 sector
Main memory has = 1K sectors
16k 16
Sector mapping cache
2018/11/19
\course\cpeg323-05F\Topic7a

37. Address (showing bit positions)……
Byte
offset
Tag 20 10
Hit
Data
Index Valid Tag Data 1 2
. . .
1021
1022
1023 20 32 =
MIPS Example
2018/11/19
\course\cpeg323-05F\Topic7a

38. Total # of Bits in a Cache
(# of bits of a tag +
# of bits of a block +
# of bits in valid field) x Cache size
For a MIPS example :
= (( ) ) x 214
= 214 x 49
= 784 k bits
~ 100 kbytes
= 64 K (bytes)
= blocks with
Assuming a directly-mapped cache.
2018/11/19
\course\cpeg323-05F\Topic7a