Fine-Grain CAM-Tag Cache Resizing Using Miss Tags Michael Zhang Krste Asanovic {rzhang|krste}@lcs.mit.edu ISLPED ’02, August 12-14, Monterey, CA

Motivation Cache uses 30-60% processor energy in embedded systems 43% for StrongArm-1 [vs. 16% for Alpha 21264] Working set of many applications smaller than cache size Can reduce cache size adaptively to match the current working set Deactivate unused portions of the cache circuitry to save power Active power Leakage power

Related Work Off-Line Techniques – Selective Ways Statically deactivate cache ways according to profiling information before application execution. [Albonesi ’99] On-Line Techniques – DRI-Cache Dynamically keeps the instruction cache miss rate under a preset bound. [Powell et. al. ’01] Line Deactivation – Cache Decay Per cache line counter used to track and turn off not recently used cache lines to reduce leakage. [Kaxiras et. al.’01] Line Deactivation – Adaptive Mode Control Only deactivate lines, not tags. [Zhou et. al. ’01] A hit in the tag of a deactivated line is a sleep miss A miss in the tag is a real miss The ratio of sleep miss to real miss used to adjust resizing intervals. Limit Study Various design choices for RAM-tag caches are studied and compared in [Yang et. al. ’02.]

Miss Tags – The Idea Data Array Tag Miss Tag Array Non- Resizable Resizable Data Array Tag Miss Tag Array Miss Tags : A second set of tags used as predictors Fixed-Size, acts as the tag array of the full-sized cache Checked only during cache miss to see if full sized cache would have avoided the miss Live in the non-critical miss path, can be implemented with smaller, slower, non-leaky transistors

Miss Tag – The Idea Data Array Tag Miss Tag Array Non- Resizable Resizable Data Array Tag Miss Tag Array hit Downsize cache If many hits in regular tags, few accesses to miss tags Application has small working set Smaller cache probably okay Hint to downsize the cache

Miss Tag – The Idea Data Array Tag Miss Resizable Miss Non- miss Downsize cache If miss in regular tags and miss in miss tags Larger cache unlikely to help Example - streaming applications with no temporal locality Hint to downsize the cache

Miss Tag – The Idea Data Array Tag Miss Tag Array Non- Resizable Resizable Data Array Tag Miss Tag Array miss Upsize cache hit If miss in regular tags but hit in miss tags Larger cache likely to help Upsize the cache

Resizing Illustration Cache Size Execution time Initial Period Resizing Interval Resizing Point 1 Downsizing # of Miss Tag Hits < Lower Bound Resizing Point 2 No Action Upper Bound > # of Miss Tag Hits Lower Bound Resizing Point 3 Upsizing # of Miss Tag Hits > Upper Bound Measurement # of Miss Tag Hits Parameters Resizing Interval Upper Bound Lower Bound Algorithm: Adjust cache size such that # of miss tag hits is within upper and lower bounds per resizing interval

CAM-Tag Cache Popular among low-power processors Generally Sub-banked Hit? Data Array Tag Popular among low-power processors ARM3 [’89] StrongArm [’98] Xscale [’01] Generally Sub-banked One sub-bank activated Each bank is a set Each line within a sub-bank is a way All tags searched in parallel Matched tag asserts appropriate word line for data read/write. Tag Bank Offset

MTR with CAM-Tag Cache Tag Array Data Array Miss Tag Array Non- resizable resizable Sub-bank Structure Tag Array Data Array Miss Tag Array MTR cache sub-bank configuration Each sub-bank has 8 equal partitions For upsize, turn on entire partition For downsize, turn off last active line Conservative downsizing Sub-bank resizing Each sub-bank resized individually Resizing spaced out evenly in time No burst of dirty write backs

Hardware Modifications The Miss Tags Only accessed during miss – not on critical path Can be implemented using slow non-leaky transistors, using alternative serial/parallel RAM/CAM structures Turning off cache lines – Leakage Reduction Gated-Vdd: Adding a stacked N-type transistor to reduce leakage energy. [Powell et. al. ’01] Leakage-Biased-Bitlines – Leakage Reduction Turning off the precharge of CAM/RAM bitlines and CAM match lines. Automatically biases the voltage to minimize leakage. [Heo et. al. ’02] Hierarchical Bitlines – Active Energy Reduction Used to turn off portions of the cache block to reduce active energy. [Ghose & Kamble ’99] Minimal cycle time impact < 1.5% (from Gated-Vdd) No cycle time penalty without Gated-Vdd Small area Impact at ~ 10% depending on implementation

Hardware Modification Cont’d Leakage-Biased Bitlines Gated-Vdd Hierarchical Bitlines

Experimental Setup Modified SimpleScalar 3.0 simulator Single-issue in-order processor Baseline cache similar to Intel XScale 32KB implemented in 32 1KB sub-banks 32-way set-associative with 32-Byte cache lines FIFO replacement policy per sub-bank Benchmarks: SpecINT2000 and SpecFP2000 1.5 Billion cycles of reference inputs Baseline resizing scheme implemented for performance comparison Compares to a fixed miss rate If current miss rate > fixed threshold, upsize, otherwise, downsize Similar to DRI-Cache

Dynamic Resizing Illustration D-cache Miss Rate vs. Time D-cache Active Size vs. Time Miss rate and average active cache size obtained by applying MTR to our D-cache

Small Working Set Example Working set is very small small cache sufficient!

Low Temporal Locality Examples High miss rate, no temporal locality Small cache size has similar performance as large cache size

Adaptive Examples

CPI Comparison Each MTR data point, (active cache size, CPI) is obtained by varying resizing interval, upper and lower bound Each Baseline data point, (active cache size, CPI) is obtained by varying the preset miss rate threshold For optimal results in MTR cache, parameters picked to yield average active I-cache size of 12KB and D-Cache size of 8KB

Energy Savings Sensitivity Analysis with two factors L2 = 16xL1 Leakage = 50% of total L2 = 16xL1 Leakage = 50% of total L2 = 128xL1 Leakage = 0% of total L2 = 128xL1 Leakage = 0% of total Sensitivity Analysis with two factors Percentage of leakage energy to total energy (0% to 50%) L2 refill energy in multiples of L1 access energy (16x to 128x) Energy figures from simulation of extracted layout TSMC 0.25 mm technology Writeback energy included

Performance Performance is affected by ratio of upper bound and lower bound to resizing interval [5, 10] / 32k ~= [10, 20] / 64k length of resizing interval Large resizing interval yields less writebacks If resizing interval is too large, lose resizing ability If resizing interval is too small, thrashing Performance consistent across all benchmarks Duplicated tags acts as predictor for each benchmark Easy parameter tuning

MTR – A Summary CAM-Tag cache offers very fine-grained resizing One cache line at a time Avoids writeback bursts Miss Tag Resizing Algorithm Resizes dynamically No delay overhead – resizing operations completely fall into the miss path Resizing determined by the difference between actual miss rate and the predicted miss rate of the system Resizing parameters can be tuned to work well for all benchmarks - no need for application-specific parameter tuning. Reduces both active and leakage energy

Conclusion Proposed MTR, a dynamic cache resizing technique for CAM-Tag caches. Uses a fixed-sized duplicate tag array to keep track of miss rate of full-sized cache. Negligible delay overhead (accessed only on miss) Negligible energy overhead (non-leaky slow transistors used) Achieves 28% to 56% energy saving for D-Cache depending on operating point Achieves 34% to 49% energy saving for I-Cache depending on operating point Minimal cycle time impact at < 1.5% Small area impact at ~ 10%