1. Morgan Kaufmann Publishers Memory Hierarchy: Introduction
17 January, 2019
Lecture 4.1
Memory Hierarchy:
Introduction
Chapter 5 — Large and Fast: Exploiting Memory Hierarchy

2. Learning Objectives Outline the memory hierarchy
Explain the principle of locality
Spatial locality
Temporal locality
Understand the abstract view of cache in computer organization
Cache is transparent to processor
Calculate hit rate and miss rate 2

3. Coverage
Textbook Chapter 5.1 3

4. Processor-Memory Performance Gap
55%/year
(2X/1.5yr)
“Moore’s Law”
Processor-Memory
Performance Gap (grows 50%/year)
DRAM
7%/year
(2X/10yrs)
HIDDEN SLIDE – KEEP?
Memory baseline is a 64KB DRAM in 1980, with three years to the next generation until 1996 and then two years thereafter with a 7% per year performance improvement in latency.
Processor assumes a 35% improvement per year until 1986, then a 55% until 2003, then 5%
Need to supply an instruction and a data every clock cycle
In 1980 there were no caches (and no need for them), by 1995 most systems had 2 level caches (e.g., 60% of the transistors on the Alpha were in the cache)

5. The “Memory Wall” Processor vs DRAM speed disparity continues to grow
Clocks per DRAM access
Clocks per instruction
Good memory hierarchy (cache) design is increasingly important to overall performance

6. Morgan Kaufmann Publishers
17 January, 2019
Memory Technology
Static RAM (SRAM)
0.5ns – 2.5ns, $2000 – $5000 per GB
Dynamic RAM (DRAM)
50ns – 70ns, $20 – $75 per GB
Magnetic disk
5ms – 20ms, $0.20 – $2 per GB
Ideal memory
Access time of SRAM
Capacity and cost/GB of disk 6
Chapter 5 — Large and Fast: Exploiting Memory Hierarchy

7. Memory Hierarchy
The Pyramid 7

8. A Typical Memory Hierarchy
Take advantage of the principle of locality to present the user with as much memory as is available in the cheapest technology while at the speed offered by the fastest technology
On-Chip Components
Control
Secondary
Memory
(Disk)
Cache
Instr
Second
Level
Cache
(SRAM)
ITLB
Main
Memory
(DRAM)
Datapath
Cache
Data
RegFile
DTLB
Instead, the memory system of a modern computer consists of a series of black boxes ranging from the fastest to the slowest.
Besides variation in speed, these boxes also varies in size (smallest to biggest) and cost.
What makes this kind of arrangement work is one of the most important principle in computer design. The principle of locality. The principle of locality states that programs access a relatively small portion of the address space at any instant of time.
The design goal is to present the user with as much memory as is available in the cheapest technology (points to the disk).
While by taking advantage of the principle of locality, we like to provide the user an average access speed that is very close to the speed that is offered by the fastest technology.
(We will go over this slide in detail in the next lectures on caches).
Speed (cycles): ½’s ’s ’s ’s ,000’s
Size (bytes): ’s K’s M’s G’s T’s
Cost: highest lowest

9. Morgan Kaufmann Publishers
January 17, 2019
Inside the Processor
AMD Barcelona: 4 processor cores
Chapter 1 — Computer Abstractions and Technology

10. Morgan Kaufmann Publishers
17 January, 2019
Principle of Locality
Programs access a small proportion of their address space at any time
Temporal locality (locality in time)
Items accessed recently are likely to be accessed again soon
e.g., instructions in a loop, induction variables
Keep most recently accessed items into the cache
Spatial locality (locality in space)
Items near those accessed recently are likely to be accessed soon
E.g., sequential instruction access, array data
Move blocks consisting of contiguous words closer to the processor 10
Chapter 5 — Large and Fast: Exploiting Memory Hierarchy

11. Taking Advantage of Locality
Morgan Kaufmann Publishers
17 January, 2019
Taking Advantage of Locality
Memory hierarchy
Store everything on disk
Copy recently accessed (and nearby) items from disk to smaller DRAM memory
Main memory
Copy more recently accessed (and nearby) items from DRAM to smaller SRAM memory
Cache memory attached to CPU
Chapter 5 — Large and Fast: Exploiting Memory Hierarchy — 11
Chapter 5 — Large and Fast: Exploiting Memory Hierarchy

12. Memory Hierarchy Levels
Morgan Kaufmann Publishers
17 January, 2019
Memory Hierarchy Levels
Block (aka cache line)
Unit of copying
May be multiple words
If accessed data is present in upper level
Hit: access satisfied by upper level
Hit ratio: hits/accesses
Hit time:
If accessed data is absent
Miss: data not in upper level
Miss ratio: misses/accesses = 1 – hit ratio
Miss penalty:
Time to access the block + Time to determine hit/miss
Time to access the block in the lower level +
Time to transmit that block to the level that experienced
the miss +
Time to insert the block in that level +
Time to pass the block to the requestor
Chapter 5 — Large and Fast: Exploiting Memory Hierarchy — 12
Chapter 5 — Large and Fast: Exploiting Memory Hierarchy