CACHE MEMORY CS 147 October 2, 2008 Sampriya Chandra

LOCALITY PRINCIPAL OF LOCALITY is the tendency to reference data items that are near other recently referenced data items, or that were recently referenced themselves. TEMPORAL LOCALITY : memory location that is referenced once is likely to be referenced multiple times in near future. SPATIAL LOCALITY : memory location that is referenced once, then the program is likely to be reference a nearby memory location in near future.

CACHE MEMORY Principle of locality helped to speed up main memory access by introducing small fast memories known as CACHE MEMORIES that hold blocks of the most recently referenced instructions and data items. Cache is a small fast storage device that holds the operands and instructions most likely to be used by the CPU.

Memory Hierarchy of early computers: 3 levels – CPU registers – DRAM Memory – Disk storage

Due to increasing gap between CPU and main Memory, small SRAM memory called L1 cache inserted. L1 caches can be accessed almost as fast as the registers, typically in 1 or 2 clock cycle Due to even more increasing gap between CPU and main memory, Additional cache: L2 cache inserted between L1 cache and main memory : accessed in fewer clock cycles.

L2 cache attached to the memory bus or to its own cache bus Some high performance systems also include additional L3 cache which sits between L2 and main memory. It has different arrangement but principle same. The cache is placed both physically closer and logically closer to the CPU than the main memory.

CACHE LINES / BLOCKS Cache memory is subdivided into cache lines Cache Lines / Blocks: The smallest unit of memory than can be transferred between the main memory and the cache.

TAG / INDEX Every address field consists of two primary parts: a dynamic (tag) which contains the higher address bits, and a static (index) which contains the lower address bits The first one may be modified during run-time while the second one is fixed.

VALID BIT / DIRTY BIT When a program is first loaded into main memory, the cache is cleared, and so while a program is executing, a valid bit is needed to indicate whether or not the slot holds a line that belongs to the program being executed. There is also a dirty bit that keeps track of whether or not a line has been modified while it is in the cache. A slot that is modified must be written back to the main memory before the slot is reused for another line.

Example: Memory segments and cache segments are exactly of the same size. Every memory segment contains equally sized N memory lines. Memory lines and cache lines are exactly of the same size. Therefore, to obtain an address of a memory line, it needs to determine the number of its memory segment first and the number of the memory line inside of that segment second, then to merge both numbers. Substitute the segment number with the tag and the line number with the index, and you should have realized the idea in general.

Therefore, cache line's tag size depends on 3 factors: Size of cache memory; Associativity of cache memory; Cacheable range of operating memory. Stag — size of cache tag, in bits; Smemory — cacheable range of operating memory, in bytes; Scache — size of cache memory, in bytes; A — associativity of cache memory, in ways. Here,

CACHE HITS / MISSES Cache Hit: a request to read from memory, which can satisfy from the cache without using the main memory. Cache Miss: A request to read from memory, which cannot be satisfied from the cache, for which the main memory has to be consulted.

CACHE MEMORY : PLACEMENT POLICY There are three commonly used methods to translate main memory addresses to cache memory addresses. Associative Mapped Cache Direct-Mapped Cache Set-Associative Mapped Cache The choice of cache mapping scheme affects cost and performance, and there is no single best method that is appropriate for all situations

Associative Mapping a block in the Main Memory can be mapped to any block in the Cache Memory available (not already occupied) Advantage: Flexibility. An Main Memory block can be mapped anywhere in Cache Memory. Disadvantage: Slow or expensive. A search through all the Cache Memory blocks is needed to check whether the address can be matched to any of the tags.

Direct Mapping To avoid the search through all CM blocks needed by associative mapping, this method only allows # blocks in main memory. # blocks in cache memory blocks to be mapped to each Cache Memory block.

Advantage: Direct mapping is faster than the associative mapping as it avoids searching through all the CM tags for a match. Disadvantage: But it lacks mapping flexibility. For example, if two MM blocks mapped to same CM block are needed repeatedly (e.g., in a loop), they will keep replacing each other, even though all other CM blocks may be available.

Set-Associative Mapping This is a trade-off between associative and direct mappings where each address is mapped to a certain set of cache locations. The cache is broken into sets where each set contains "N" cache lines, let's say 4. Then, each memory address is assigned a set, and can be cached in any one of those 4 locations within the set that it is assigned to. In other words, within each set the cache is associative, and thus the name.

DIFFERENCE BETWEEN LINES, SETS AND BLOCKS In direct-mapped caches, sets and lines are equivalent. However in associative caches, sets and lines are very different things and the terms cannot be interchanged.

BLOCK: fixed sized packet of information that moves back and forth between a cache and main memory. LINE: container in a cache that stores a block as well as other information such as the valid bit and tag bits. SET: collection of one or more lines. Sets in direct-mapped caches consist of a single line. Set in fully associative and set associative caches consists of multiple lines.

ASSOCIATIVITY Associativity : N-way set associative cache memory means that information stored at some address in operating memory could be placed (cached) in N locations (lines) of this cache memory. The basic principle of logical segmentation says that there is only one line within any particular segment to be capable of caching information located at some memory address.

REPLACEMENT ALGORITHM Optimal Replacement: replace the block which is no longer needed in the future. If all blocks currently in Cache Memory will be used again, replace the one which will not be used in the future for the longest time. Random selection: replace a randomly selected block among all blocks currently in Cache Memory.

FIFO (first-in first-out): replace the block that has been in Cache Memory for the longest time. LRU (Least recently used): replace the block in Cache Memory that has not been used for the longest time. LFU (Least frequently used): replace the block in Cache Memory that has been used for the least number of times.

The optimal replacement : the best but is not realistic because when a block will be needed in the future is usually not known ahead of time. The LRU is suboptimal based on the temporal locality of reference, i.e., memory items that are recently referenced are more likely to be referenced soon than those which have not been referenced for a longer time. FIFO is not necessarily consistent with LRU therefore is usually not as good. The random selection, surprisingly, is not necessarily bad.

HIT RATIO and EFFECTIVE ACCESS TIMES Hit Ratio : The fraction of all memory reads which are satisfied from the cache

LOAD-THROUGH STORE-THROUGH Load-Through : When the CPU needs to read a word from the memory, the block containing the word is brought from MM to CM, while at the same time the word is forwarded to the CPU. Store-Through : If store-through is used, a word to be stored from CPU to memory is written to both CM (if the word is in there) and MM. By doing so, a CM block to be replaced can be overwritten by an in-coming block without being saved to MM.

WRITE METHODS Note: Words in a cache have been viewed simply as copies of words from main memory that are read from the cache to provide faster access. However this view point changes. There are 3 possible write actions: –Write the result into the main memory –Write the result into the cache –Write the result into both main memory and cache memory

Write Through: A cache architecture in which data is written to main memory at the same time as it is cached. Write Back / Copy Back: CPU performs write only to the cache in case of a cache hit. If there is a cache miss, CPU performs a write to main memory.

When the cache is missed : Write Allocate: loads the memory block into cache and updates the cache block No-Write allocation: this bypasses the cache and writes the word directly into the memory.

CACHE CONFLICT A sequence of accesses to memory repeatedly overwriting the same cache entry. This can happen if two blocks of data, which are mapped to the same set of cache locations, are needed simultaneously.

EXAMPLE: In the case of a direct mapped cache, if arrays A, B, and C map to the same range of cache locations, thrashing will occur when the following loop is executed: for (i=1; i<n; i++) C[i] = A[i] + B[i]; Cache conflict can also occur between a program loop and the data it is accessing.

CACHE COHERENCY The synchronization of data in multiple caches such that reading a memory location via any cache will return the most recent data written to that location via any (other) cache. Some parallel processors do not cache accesses to shared memory to avoid the issue of cache coherency.

If caches are used with shared memory then some system is required to detect when data in one processor's cache should be discarded or replaced because another processor has updated that memory location. Several such schemes have been devised.

REFERENCES Computer Architecture and Organization: An Integrated Approach by Miles J. Murdocca and Vincent P. Heuring Logic and Computer Design Fundamentals by M.Morris Mano and Charles R. Kime Computer Systems : A Programmers Perspective by Randal E. Bryant and David R. O’Hallaron Websites :