Presentation is loading. Please wait.

Presentation is loading. Please wait.

Cache Memory Organization

Similar presentations

Presentation on theme: "Cache Memory Organization"— Presentation transcript:

1 Cache Memory Organization
Today's Agends Memory Hierarchy Locality of Reference Cache Memory Organization Virtual Memory

2 Bus Structure for a computer system
CPU chip Register file ALU Main memory Bus interface I/O bus USB controller Graphics adapter Disk controller Mouse Keyboard Monitor Disk

3 The CPU-Memory Gap The gap widens between DRAM, disk, and CPU speeds.

4 Locality to the Rescue! The key to bridging this CPU-Memory gap is a fundamental property of computer programs known as locality

5 Locality of Reference Principle of Locality:Programs tend to use data and instructions with addresses near or equal to those they have used recently Temporal locality: Recently referenced items are likely to be referenced again in the near future Spatial locality: Items with nearby addresses tend to be referenced close together in time

6 Locality Example Data references Instruction references
sum = 0; for (i = 0; i<n; i++) sum += a[i]; return sum; Data references Reference array elements in succession (stride-1 reference pattern). Reference variable sum each iteration. Instruction references Reference instructions in sequence. Cycle through loop repeatedly. Spatial locality Temporal locality Spatial locality Temporal locality

7 Memory Hierarchies Some fundamental and enduring properties of hardware and software: Fast storage technologies cost more per byte, have less capacity, and require more power (heat!). The gap between CPU and main memory speed is widening. Well-written programs tend to exhibit good locality. They suggest an approach for organizing memory and storage systems known as a memory hierarchy.

8 An Example Memory Hierarchy
CPU registers hold words retrieved from L1 cache Registers L1: L1 cache (SRAM) Smaller, faster, costlier per byte L1 cache holds cache lines retrieved from L2 cache L2: L2 cache (SRAM) L2 cache holds cache lines retrieved from main memory L3: Main memory (DRAM) Larger, slower, cheaper per byte Main memory holds disk blocks retrieved from local disks L4: Local secondary storage (local disks) Local disks hold files retrieved from disks on remote network servers Remote secondary storage (tapes, distributed file systems, Web servers) L5:

9 Caches Cache:A smaller, faster storage device that acts as a staging area for a subset of the data in a larger, slower device. Fundamental idea of a memory hierarchy: For each k, the faster, smaller device at level k serves as a cache for the larger, slower device at level k+1. Why do memory hierarchies work? Because of locality, programs tend to access the data at level k more often than they access the data at level k+1. Thus, the storage at level k+1 can be slower, and thus larger and cheaper per bit. Big Idea: The memory hierarchy creates a large pool of storage that costs as much as the cheap storage near the bottom, but that serves data to programs at the rate of the fast storage near the top.

10 General Cache Concepts
Smaller, faster, more expensive memory caches a subset of the blocks Cache 8 4 9 14 10 3 Data is copied in block-sized transfer units 4 10 Larger, slower, cheaper memory viewed as partitioned into “blocks” Memory 1 2 3 4 4 5 6 7 8 9 10 10 11 12 13 14 15

11 General Cache Concepts: Hit
Request: 14 Data in block b is needed Block b is in cache: Hit! Cache 8 9 14 14 3 Memory 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

12 General Cache Concepts: Miss
Request: 12 Data in block b is needed Block b is not in cache: Miss! Cache 8 9 12 14 3 Block b is fetched from memory 12 Request: 12 Block b is stored in cache Placement policy: determines where b goes Replacement policy: determines which block gets evicted (victim) Memory 1 2 3 4 5 6 7 8 9 10 11 12 12 13 14 15

13 General Caching Concepts: Types of Cache Misses
Cold (compulsory) miss Cold misses occur because the cache is empty. Conflict miss Most caches limit blocks at level k+1 to a small subset (sometimes a singleton) of the block positions at level k. E.g. Block i at level k+1 must be placed in block (i mod 4) at level k. Conflict misses occur when the level k cache is large enough, but multiple data objects all map to the same level k block. E.g. Referencing blocks 0, 8, 0, 8, 0, 8, ... would miss every time. Capacity miss Occurs when the set of active cache blocks (working set) is larger than the cache.

14 Examples of Caching in the Hierarchy
Cache Type What is Cached? Where is it Cached? Latency (cycles) Managed By Registers 4-8 bytes words CPU core Compiler TLB Address translations On-Chip TLB Hardware L1 cache 64-bytes block On-Chip L1 1 Hardware L2 cache 64-bytes block On/Off-Chip L2 10 Hardware Virtual Memory 4-KB page Main memory 100 Hardware + OS Buffer cache Parts of files Main memory 100 OS Disk cache Disk sectors Disk controller 100,000 Disk firmware Network buffer cache Parts of files Local disk 10,000,000 AFS/NFS client Browser cache Web pages Local disk 10,000,000 Web browser Web cache Web pages Remote server disks 1,000,000,000 Web proxy server

15 Put it together The speed gap between CPU, memory and mass storage continues to widen. Well-written programs exhibit a property called locality. Memory hierarchies based on caching close the gap by exploiting locality.

16 Cache/Main Memory Structure
CPU requests data from memory location Check cache for this data If present, get from cache (fast) If not present, read required block from main memory to cache Then deliver from cache to CPU Cache includes tags to identify which block of main memory is in each cache slot

17 Cache Design Issues Size does matter Size Mapping Function
Replacement Algorithm Write Policy Block Size Number of Caches Size does matter Cost More cache is expensive Speed More cache is faster (up to a point) Checking cache for data takes time

18 Mapping Function i.e. cache is 16k (214) lines of 4 bytes each
Cache of 64kByte Cache block of 4 bytes i.e. cache is 16k (214) lines of 4 bytes each 16MBytes main memory 24 bit address (224=16M) Direct Mapping Each block of main memory maps to only one cache line i.e. if a block is in cache, it must be in one specific place Address is in two parts Least Significant w bits identify unique word. Most Significant s bits specify one memory block The MSBs are split into a cache line field r and a tag of s-r (most significant)

19 Direct Mapping: Address Structure
Tag s-r Line or Slot r Word w 8 14 2 24 bit address 2 bit word identifier (4 byte block) 22 bit block identifier 8 bit tag (=22-14) 14 bit slot or line No two blocks in the same line have the same Tag field Check contents of cache by finding line and checking Tag

20 Direct Mapping Cache Organization
Cache line Main Memory blocks held 0, m, 2m, 3m…2s-m 1 1,m+1, 2m+1…2s-m+1 m-1 m-1, 2m-1,3m-1…2s-1

21 Direct Mapped Cache Simple Inexpensive Fixed location for given block
If a program accesses 2 blocks that map to the same line repeatedly, cache misses are very high Address length = (s + w) bits Number of addressable units = 2s+w words or bytes Block size = line size = 2w words or bytes Number of blocks in main memory 2s+w/2w = 2s Number of lines in cache = m = 2r Size of tag = (s – r) bits

22 Fully Associative Mapping
A main memory block can load into any line of cache Memory address is interpreted as tag and word Tag uniquely identifies block of memory Every line’s tag is examined for a match Cache searching gets expensive

23 Fully Associative Mapping

24 Fully Associative Mapping: Address Structure
Word 2 bit Tag 22 bit 22 bit tag stored with each 32 bit block of data Compare tag field with tag entry in cache to check for hit Least significant 2 bits of address identify which 16 bit word is required from 32 bit data block

25 Set Associative Mapping
Cache is divided into a number of sets Each set contains a number of lines A given block maps to any line in a given set e.g. Block B can be in any line of set i e.g. 2 lines per set 2 way associative mapping A given block can be in one of 2 lines in only one set

26 Set Associative Mapping

27 Two-Way Set Associative Mapping Address Structure
Tag 9 bit Set 13 bit Word 2 bit Use set field to determine cache set to look in Compare tag field to see if we have a hit

28 Set Associative Mapping Summary
Address length = (s + w) bits Number of addressable units = 2s+w words or bytes Block size = line size = 2w words or bytes Number of blocks in main memory = 2s Number of lines in set = k Number of sets = v = 2d Number of lines in cache = kv = k * 2d Size of tag = (s – d) bits

29 Replacement Algorithms
No choice in Direct mapping because each block only maps to one line!!! So it is applicable for other mapping functions Hardware implemented algorithm (speed) Least Recently used (LRU) e.g. in 2 way set associative Which of the 2 block is LRU? First in first out (FIFO) Replace block that has been in cache longest Least frequently used Replace block which has had fewest hits Random

30 Write Policy What is the need of Write Policy? Write through
Must not overwrite a cache block unless main memory is up to date Multiple CPUs may have individual caches I/O may address main memory directly Write through All writes go to main memory as well as cache Multiple CPUs can monitor main memory traffic to keep local (to CPU) cache up to date Lots of traffic and Slows down writes Write back Updates initially made in cache only Update bit for cache slot is set when update occurs If block is to be replaced, write to main memory only if update bit is set Other caches get out of sync I/O must access main memory through cache

31 What about writes? Multiple copies of data exist:
L1, L2, Main Memory, Disk What to do on a write-hit? Write-through (write immediately to memory) Write-back (defer write to memory until replacement of line) Need a dirty bit (line different from memory or not) What to do on a write-miss? Write-allocate (load into cache, update line in cache) Good if more writes to the location follow No-write-allocate (writes immediately to memory) Typical Write-through + No-write-allocate Write-back + Write-allocate

32 Intel Core i7 Cache Hierarchy
Processor package Core 0 Core 3 L1 i-cache and d-cache: 32 KB, 8-way, Access: 4 cycles L2 unified cache: 256 KB, 8-way, Access: 11 cycles L3 unified cache: 8 MB, 16-way, Access: cycles Block size: 64 bytes for all caches. Regs Regs L1 d-cache L1 i-cache L1 d-cache L1 i-cache L2 unified cache L2 unified cache L3 unified cache (shared by all cores) Main memory

33 Cache Performance Metrics
Miss Rate Fraction of memory references not found in cache (misses / accesses) = 1 – hit rate Typical numbers (in percentages): 3-10% for L1 can be quite small (e.g., < 1%) for L2, depending on size, etc. Hit Time Time to deliver a line in the cache to the processor includes time to determine whether the line is in the cache Typical numbers: 1-2 clock cycle for L1 5-20 clock cycles for L2 Miss Penalty Additional time required because of a miss typically cycles for main memory (Trend: increasing!)

34 Writing Cache Friendly Code
Make the common case go fast Focus on the inner loops of the core functions Minimize the misses in the inner loops Repeated references to variables are good (temporal locality) Stride-1 reference patterns are good (spatial locality) Key idea: Our qualitative notion of locality is quantified through our understanding of cache memories.

35 Lecture on Friday (Extra Class)
10 to 11:am in room no 1220

36 Memory Management In multi-tasking environment, CPU is shared among multiple processes and CPU scheduling takes care of assigning CPU to each ready process. In-order to realize multi-tasking multiple processes should be kept in memory or memory should be shared between multiple processes. This sharing of memory is taken care by memory management techniques. These techniques are usually implemented in hardware.

37 Main Memory Main memory is divided into two parts
One for OS Other for multiple processes User processes should be restricted from accessing OS memory area and also other user processes This protection is provided by the hardware in the form of base and limit register. (8.1,pno.316) Base register holds starting physical address and limit register holds the size of the process. Both registers are loaded by OS by using privileged instructions.

38 8.1, 316

39 8.2, 317

40 Address Binding Address in user program is different from physical address. Needs address binding. Compiler usually bind symbolic address in an executable to relocatable address. Loader in turn bind relocatable address to absolute physical address. Binding can be done at Compile time – generates absolute code Load time – compiler generates relocatable code. Binding is delayed until load time Execution time – binding is delayed until run time. That is process is brought into the memory at execution time.

41 Logical versus Physical Address Space
CPU generates logical address Address seen by memory is Physical address Logical Address Space : it is set of all logical addresses generated by a program Physical Address Space: It is a set of physical addresses corresponding to logical addresses. Memory Management Unit: Runtime mapping from logical to physical address is done by MMU (hardware)

42 Simple MMU scheme Using base register (a.k.a Relocation Register)
User program deals with logical address MMU converts logical address to physical address Physical address is used to access main or physical memory

43 Dynamic Loading Do not load entire program into memory before execution It limits the size of program to the size of physical memory Since programs divided into modules or routines Dynamically load a routine when it called. Advantage: Unused routine is never loaded

44 Static Linking and Dynamic Linking
Shared libraries are linked statically and are combined into the binary program image before execution Dynamic Linking Linking shared libraries is postponed until execution time

45 Memory Allocation Memory can be allocated in two ways
Contiguous Memory Allocation A process resides in a single contiguous section of memory Fixed sized partition Multi-programming is bounded to number of partitions Internal Fragmentation Variable sized partition OS maintains a table to keep track of which memory is available and which is occupied External Fragmentation Non- Contiguous Memory Allocation Process is not stored in contiguous memory segments Paging

46 Paging It’s a memory management scheme that allows memory allocation to be non-contiguous Paging avoids external fragmentation and need of compaction Traditional systems provide hardware support for paging Modern systems implement paging by closely integrating hardware and OS (in 64-microprocessors)

47 Basic method of Paging Logical memory is divided into fixed-sized blocks called pages Physical memory is divided into fixed-sized blocks called frames Backing store (disk) is also divide into fixed sized blocks called memory frames CPU generates logical address which is divided into two parts: page number and page offset Page size is defined by hardware and usually is power of 2 varying between512 bytes to 16MB page size. Page size is growing over time as processes, data sets and main memory have become large Internal fragmentation may exist

48 Page Table It is used to map a virtual page to physical frame
Page table contains information for this mapping For each page number there is a corresponding frame number stored in page table. Page table is implemented in hardware.

49 Paging Hardware

50 Paging, memory mapping using page table

51 Page table – hardware implementation
Method 1 Page table as a set of dedicated set of registers These registers are made up of high speed logic to make paging address translation efficient CPU dispatcher reloads these registers just as it reloads other registers This method is okay if page table is small but for large processes this method is not feasible

52 Page table – hardware implementation
Method 2 Store page table in main memory Page Table Base Register (PTBR) is used to point to page table in memory But with this method, accessing memory require two memory access. One for page table and another for actual access Very slow Need another technique

53 Translation Lookaside Buffer (TLB)
Since the page tables vary in size. Require Larger page table for larger size processes Its not possible to store it in registers. Need to store it in main memory Every virtual memory reference causes two physical memory access Fetch page table entry Fetch data Slows down the system To overcome this problem special cache memory can be used to store page table entries TLB

54 TLB

55 TLB and Cache Operation

56 Structure of page table
Hierarchical Paging Here Logical address is divided into three parts: Page number 1 (p1) Page number 2 (p2) Page offset 8.14 and 8.15 figures pno. 338



59 Virtual Memory Demand paging Page fault
Do not require all pages of a process in memory Bring in pages as required Page fault Required page is not in memory Operating System must swap in required page May need to swap out a page to make space Select page to throw out based on recent history

60 Page Fault

61 Thrashing Too many processes in too little memory
Operating System spends all its time swapping Little or no real work is done Disk light is on all the time Solutions Good page replacement algorithms Reduce number of processes running Fit more memory

62 Advantage We do not need all of a process in memory for it to run
We can swap in pages as required So - we can now run processes that are bigger than total memory available! Main memory is called real memory User/programmer sees much bigger memory - virtual memory

Download ppt "Cache Memory Organization"

Similar presentations

Ads by Google