1. Computer Organization and Architecture Chapter 7 Large and Fast: Exploiting Memory Hierarchy
Yu-Lun Kuo
Computer Sciences and Information Engineering
University of Tunghai, Taiwan
CS252 S05

2. Major Components of a Computer
Processor
Devices
Control
Input
Memory
Datapath
Output
CS252 S05

3. Processor-Memory Performance Gap
55%/year
(2X/1.5yr)
“Moore’s Law”
Processor-Memory
Performance Gap (grows 50%/year)
DRAM
7%/year
(2X/10yrs)
CS252 S05

4. Introduction The Principle of Locality Two Different Types of Locality
Program access a relatively small portion of the address space at any instant of time.
Two Different Types of Locality
Temporal Locality (Locality in Time)
If an item is referenced, it will tend to be referenced again soon
e.g., loop, subrouting, stack, variable of counting
Spatial Locality (Locality in Space)
If an item is referenced, items whose addresses are close by tend to be referenced soon
e.g., array access, accessed sequentially
CS252 S05

5. Memory Hierarchy Memory Hierarchy Levels Inclusive
A structure that uses multiple levels of memories; as the distance form the CPU increase, the size of the memories and the access time both increase
Locality + smaller HW is faster = memory hierarchy
Levels
each smaller, faster, more expensive/byte than level below
Inclusive
data found in top also found in the bottom
CS252 S05

6. Three Primary Technologies
Building Memory Hierarchies
Main Memory
DRAM (Dynamic random access memory)
Caches (closer to the processor)
SRAM (static random access memory)
DRAM vs. SRAM
Speed : DRAM < SRAM
Cost: DRAM < SRAM
CS252 S05

7. Introduction Cache memory Made by SRAM (Static RAM)
Small amount of fast and high speed memory
Sits between normal main memory and CPU
May be located on CPU chip or module
CS252 S05

8. Introduction
Cache memory
CS252 S05

9. A Typical Memory Hierarchy c.2008
Split instruction & data primary caches
(on-chip SRAM)
Multiple interleaved memory banks
(off-chip DRAM)
L1 Instruction Cache
Unified L2 Cache
Memory
CPU
Memory
Memory
L1 Data Cache RF
Memory
Multiported register file (part of CPU)
Large unified secondary cache (on-chip SRAM)
CS252 S05

10. A Typical Memory Hierarchy
By taking advantage of the principle of locality
Can present the user with as much memory as is available in the cheapest technology
at the speed offered by the fastest technology
On-Chip Components
Control
eDRAM
Secondary
Memory
(Disk)
Cache
Instr
Second
Level
Cache
(SRAM)
ITLB
Main
Memory
(DRAM)
Datapath
Cache
Data
RegFile
DTLB
Speed (%cycles): ½’s ’s ’s ’s ,000’s
Size (bytes): ’s K’s K’s M’s G’s to T’s
Cost: highest lowest
CS252 S05

11. Characteristics of Memory Hierarchy
Processor
4-8 bytes (word)
1 to 4 blocks
1,024+ bytes (disk sector = page)
8-32 bytes (block)
Inclusive– what is in L1$ is a subset of what is in L2$ is a subset of what is in MM that is a subset of is in SM
Increasing distance from the processor in access time
L1$
L2$
Main Memory
Secondary Memory
(Relative) size of the memory at each level
CS252 S05

12. Memory Hierarchy List Registers L1 Cache L2 Cache L3 cache Main memory
Disk cache
Disk (RAID)
Optical (DVD)
Tape
CS252 S05

13. Why IC and DC need?
CS252 S05

14. The Memory Hierarchy: Terminology
Hit: data is in some block in the upper level (Blk X)
Hit Rate: the fraction of memory accesses found in the upper level
Hit Time: Time to access the upper level which consists of
RAM access time + Time to determine hit/miss
Lower Level
Memory
Upper Level
To Processor
From Processor
Blk X
Blk Y
CS252 S05

15. The Memory Hierarchy: Terminology
Miss: data is not in the upper level so needs to be retrieve from a block in the lower level (Blk Y)
Miss Rate = 1 - (Hit Rate)
Miss Penalty
Time to replace a block in the upper level + Time to deliver the block the processor
Hit Time << Miss Penalty
Lower Level
Memory
Upper Level
To Processor
From Processor
Blk X
Blk Y
CS252 S05

16. How is the Hierarchy Managed?
registers  memory
by compiler (programmer?)
cache  main memory
by the cache controller hardware
main memory  disks
by the operating system (virtual memory)
virtual to physical address mapping assisted by the hardware (TLB)
by the programmer (files)
CS252 S05

17. 7.2 The basics of Caches Simple cache
The processor requests are each one word
The block size is one word of data
Two questions to answer (in hardware):
Q1: How do we know if a data item is in the cache?
Q2: If it is, how do we find it?
CS252 S05

18. Caches
Direct Mapped
Assign the cache location based on the address of the word in memory
Address mapping:
(block address) modulo (# of blocks in the cache)
First consider block sizes of one word
CS252 S05

19. Direct Mapped (Mapping) Cache
CS252 S05

20. Caches
Tag
Contain the address information required to identify whether a word in the cache corresponds to the requested word
Valid bit
After executing many instructions, some of the cache entries may still be empty
Indicate whether an entry contains a valid address
If valid bit = 0, there cannot be a match for this block
CS252 S05

21. Direct Mapped Cache Consider the main memory word reference string
Start with an empty cache - all blocks initially marked as not valid
miss 1
miss 2
miss 3
miss
00 Mem(0)
00 Mem(1)
00 Mem(2)
00 Mem(0)
00 Mem(0)
00 Mem(0)
00 Mem(1)
00 Mem(1)
00 Mem(2)
00 Mem(3) 4
miss 3
hit 4
hit 15
miss 01 4
00 Mem(0)
00 Mem(1)
00 Mem(2)
00 Mem(3)
01 Mem(4)
00 Mem(1)
00 Mem(2)
00 Mem(3)
01 Mem(4)
00 Mem(1)
00 Mem(2)
00 Mem(3)
01 Mem(4)
00 Mem(1)
00 Mem(2)
00 Mem(3) 11 15
8 requests, 6 misses
CS252 S05

22. Hits vs. Misses Read hits Read misses Write hits Write misses
this is what we want!
Read misses
stall the CPU, fetch block from memory, deliver to cache, restart
Write hits
can replace data in cache and memory (write-through)
write the data only into the cache (write-back the cache later)
Write misses
read the entire block into the cache, then write the word
CS252 S05

23. What happens on a write? Write work somewhat differently
Suppose on a store instruction
Write the data into only the data cache
Memory would have different value
The cache & memory are “inconsistent”
Keep the main memory & cache
Always write the data into both the memory and the cache
Called write-through (直接寫入)
CS252 S05

24. What happens on a write? Although this design handles writes simple
Not provide very good performance
Every write causes the data to be written to main memory
Take a long time
Ex. 10% of the instructions are stores
CPI without cache miss: 1.0
spending 100 extra cycles on every write
CPI = x 10% = 11
reducing performance
CS252 S05

25. Write Buffer for Write Through
A Write Buffer is needed between the Cache and Memory (TLB: Translation Lookaside Buffer 轉譯旁觀緩衝區)
A queue that holds data while the data are waiting to be written to memory
Processor:
writes data into the cache and the write buffer
Memory controller:
write contents of the buffer to memory
Processor
Cache
Write Buffer
DRAM
CS252 S05

26. What happens on a write? Write back (間接寫入)
New value only written only to the block in the cache
The modified block is written to the lower level of the hierarchy when it is replaced
CS252 S05

27. What happens on a write? Write Through Write Back
All writes go to main memory as well as cache
Multiple CPUs can monitor main memory traffic to keep local (to CPU) cache up to date
Lots of traffic
Slows down writes
Write Back
Updates initially made in cache only
Update bit for cache slot is set when update occurs
If block is to be replaced, write to main memory only if update bit is set
Other caches get out of sync
CS252 S05

28. Memory System to Support Caches
It is difficult to reduce the latency to fetch the first word from memory
We can reduce the miss penalty if increase the bandwidth from the memory to the cache
CPU
CPU
CPU
Multiplexor
Cache
Cache
Cache
bus
bus
bus
Memory
Memory
bank 0
Memory
bank 1
Memory
bank 2
Memory
bank 3
Memory
CS252 S05

29. One-word-wide memory organization
Assume
A cache block for 4 words
1 memory bus clock cycle to send the address
15 clock cycles for DRAM access initiated
1 memory bus clock cycle to return a word of data
Miss penalty: 1+ 4x15 + 4x1 = 65 clock cycles
Number of bytes transferred per bus clock cycle for a single miss
4 x 4 / 65 = 0.25
CPU
Cache
bus
Memory
CS252 S05

30. Wide memory organization
Assume
A cache block for 4 words
1 memory bus clock cycle to send the address
15 clock cycles for DRAM access initiated
1 memory bus clock cycle to return a word of data
Two word wide
1 + 2 x x 1 = 33 clock cycles
4 x 4 / 33 = 0.48
Four word wide
1 + 1 x x 1 = 17 clock cycles
4 x 4 / 17 = 0.94
CPU
Multiplexor
Cache
bus
Memory
CS252 S05

31. Interleaved memory organization
Assume
A cache block for 4 words
1 memory bus clock cycle to send the address
15 clock cycles for DRAM access initiated
1 memory bus clock cycle to return a word of data
Each memory bank: 1 word wide
Advance: One latency time
1 + 1 x x 1 = 20 clock cycle
4 x 4 / 20 = 0.8 byte/clock
3 times for one-word-wide
CPU
Cache
bus
Memory
bank 0
Memory
bank 1
Memory
bank 2
Memory
bank 3
CS252 S05

32. Q & A
CS252 S05