1. Stall-Time Fair Memory Access Scheduling Onur Mutlu and Thomas Moscibroda Computer Architecture Group Microsoft Research

2. 2 Multi-Core Systems CORE 0CORE 1CORE 2CORE 3 L2 CACHE L2 CACHE L2 CACHE L2 CACHE DRAM MEMORY CONTROLLER DRAM Bank 0 DRAM Bank 1 DRAM Bank 2 DRAM Bank 7... Shared DRAM Memory System Multi-Core Chip unfairness

3. 3 DRAM Bank Operation Row Buffer Access Address (Row 0, Column 0) Row decoder Column decoder Row address 0 Column address 0 Data Row 0Empty Access Address (Row 0, Column 1) Column address 1 Access Address (Row 0, Column 9) Column address 9 Access Address (Row 1, Column 0) HIT Row address 1 Row 1 Column address 0 CONFLICT ! Columns Rows

4. 4 DRAM Controllers A row-conflict memory access takes significantly longer than a row-hit access Current controllers take advantage of the row buffer Commonly used scheduling policy (FR-FCFS) [Rixner, ISCA’00] (1) Row-hit (column) first: Service row-hit memory accesses first (2) Oldest-first: Then service older accesses first This scheduling policy aims to maximize DRAM throughput  But, it is unfair when multiple threads share the DRAM system

5. 5 Outline The Problem  Unfair DRAM Scheduling Stall-Time Fair Memory Scheduling  Fairness definition  Algorithm  Implementation  System software support Experimental Evaluation Conclusions

6. 6 The Problem Multiple threads share the DRAM controller DRAM controllers are designed to maximize DRAM throughput DRAM scheduling policies are thread-unaware and unfair  Row-hit first: unfairly prioritizes threads with high row buffer locality Streaming threads Threads that keep on accessing the same row  Oldest-first: unfairly prioritizes memory-intensive threads

7. 7 The Problem Row Buffer Row decoder Column decoder Data Row 0 T0: Row 0 Row 0 T1: Row 16 T0: Row 0T1: Row 111 T0: Row 0 T1: Row 5 T0: Row 0 Request Buffer T0: streaming thread T1: non-streaming thread Row size: 8KB, cache block size: 64B 128 requests of T0 serviced before T1

8. 8 DRAM is the only shared resource Consequences of Unfairness in DRAM Vulnerability to denial of service [Moscibroda & Mutlu, Usenix Security’07] System throughput loss Priority inversion at the system/OS level Poor performance predictability 1.05 1.85 4.72 7.74

9. 9 Outline The Problem  Unfair DRAM Scheduling Stall-Time Fair Memory Scheduling  Fairness definition  Algorithm  Implementation  System software support Experimental Evaluation Conclusions

10. 10 Fairness in Shared DRAM Systems A thread’s DRAM performance dependent on its inherent  Row-buffer locality  Bank parallelism Interference between threads can destroy either or both A fair DRAM scheduler should take into account all factors affecting each thread’s DRAM performance  Not solely bandwidth or solely request latency Observation: A thread’s performance degradation due to interference in DRAM mainly characterized by the extra memory-related stall-time due to contention with other threads

11. 11 Stall-Time Fairness in Shared DRAM Systems A DRAM system is fair if it slows down equal-priority threads equally  Compared to when each thread is run alone on the same system  Fairness notion similar to SMT [Cazorla, IEEE Micro’04][Luo, ISPASS’01], SoEMT [Gabor, Micro’06], and shared caches [Kim, PACT’04] T shared : DRAM-related stall-time when the thread is running with other threads T alone : DRAM-related stall-time when the thread is running alone Memory-slowdown = T shared /T alone The goal of the Stall-Time Fair Memory scheduler (STFM) is to equalize Memory-slowdown for all threads, without sacrificing performance  Considers inherent DRAM performance of each thread

12. 12 Outline The Problem  Unfair DRAM Scheduling Stall-Time Fair Memory Scheduling  Fairness definition  Algorithm  Implementation  System software support Experimental Evaluation Conclusions

13. 13 STFM Scheduling Algorithm (1) During each time interval, for each thread, DRAM controller  Tracks T shared  Estimates T alone At the beginning of a scheduling cycle, DRAM controller  Computes Slowdown = T shared /T alone for each thread with an outstanding legal request  Computes unfairness = MAX Slowdown / MIN Slowdown If unfairness <   Use DRAM throughput oriented baseline scheduling policy (1) row-hit first (2) oldest-first

14. 14 STFM Scheduling Algorithm (2) If unfairness ≥   Use fairness-oriented scheduling policy (1) requests from thread with MAX Slowdown first (2) row-hit first (3) oldest-first Maximizes DRAM throughput if it cannot improve fairness Does NOT waste useful bandwidth to improve fairness  If a request does not interfere with any other, it is scheduled

15. 15 How Does STFM Prevent Unfairness? Row Buffer Data Row 0 T0: Row 0 Row 0 T1: Row 16 T0: Row 0 T1: Row 111 T0: Row 0 T1: Row 5 T0: Row 0 T0 Slowdown T1 Slowdown 1.00 Unfairness 1.03 1.06  1.05 1.03 1.06 1.031.04 1.08 1.04 1.11 1.06 1.07 1.04 1.10 1.14 1.03 Row 16Row 111

16. 16 Outline The Problem  Unfair DRAM Scheduling Stall-Time Fair Memory Scheduling  Fairness definition  Algorithm  Implementation  System software support Experimental Evaluation Conclusions

17. 17 Implementation Tracking T shared  Relatively easy  The processor increases a counter if the thread cannot commit instructions because the oldest instruction requires DRAM access Estimating T alone  More involved because thread is not running alone  Difficult to estimate directly  Observation: T alone = T shared - T interference  Estimate T interference : Extra stall-time due to interference

18. 18 Estimating T interference (1) When a DRAM request from thread C is scheduled  Thread C can incur extra stall time: The request’s row buffer hit status might be affected by interference  Estimate the row that would have been in the row buffer if the thread were running alone  Estimate the extra bank access latency the request incurs Extra Bank Access Latency T interference (C) += # Banks Servicing C’s Requests Extra latency amortized across outstanding accesses of thread C (memory level parallelism)

19. 19 Estimating T interference (2) When a DRAM request from thread C is scheduled  Any other thread C’ with outstanding requests incurs extra stall time  Interference in the DRAM data bus  Interference in the DRAM bank (see paper) Bus Transfer Latency of Scheduled Request T interference (C’) += Bank Access Latency of Scheduled Request T interference (C’) += # Banks Needed by C’ Requests * K

20. 20 Hardware Cost <2KB storage cost for  8-core system with 128-entry memory request buffer Arithmetic operations approximated  Fixed point arithmetic  Divisions using lookup tables Not on the critical path  Scheduler makes a decision only every DRAM cycle More details in paper

21. 21 Outline The Problem  Unfair DRAM Scheduling Stall-Time Fair Memory Scheduling  Fairness definition  Algorithm  Implementation  System software support Experimental Evaluation Conclusions

22. 22 Support for System Software Supporting system-level thread weights/priorities  Thread weights communicated to the memory controller  Larger-weight threads should be slowed down less  Each thread’s slowdown is scaled by its weight  Weighted slowdown used for scheduling Favors threads with larger weights  OS can choose thread weights to satisfy QoS requirements  : Maximum tolerable unfairness set by system software  Don’t need fairness? Set  large.  Need strict fairness? Set  close to 1.  Other values of  : trade-off fairness and throughput

23. 23 Outline The Problem  Unfair DRAM Scheduling Stall-Time Fair Memory Scheduling  Fairness definition  Algorithm  Implementation  System software support Experimental Evaluation Conclusions

24. 24 Evaluation Methodology 2-, 4-, 8-, 16-core systems  x86 processor model based on Intel Pentium M  4 GHz processor, 128-entry instruction window  512 Kbyte per core private L2 caches Detailed DRAM model based on Micron DDR2-800  128-entry memory request buffer  8 banks, 2Kbyte row buffer  Row-hit round-trip latency: 35ns (140 cycles)  Row-conflict latency: 70ns (280 cycles) Benchmarks  SPEC CPU2006 and some Windows Desktop applications  256, 32, 3 benchmark combinations for 4-, 8-, 16-core experiments

25. 25 Comparison with Related Work Baseline FR-FCFS [Rixner et al., ISCA’00]  Unfairly penalizes non-intensive threads with low-row-buffer locality FCFS  Low DRAM throughput  Unfairly penalizes non-intensive threads FR-FCFS+Cap  Static cap on how many younger row-hits can bypass older accesses  Unfairly penalizes non-intensive threads Network Fair Queueing (NFQ) [Nesbit et al., Micro’06]  Per-thread virtual-time based scheduling A thread’s private virtual-time increases when its request is scheduled Prioritizes requests from thread with the earliest virtual-time Equalizes bandwidth across equal-priority threads Does not consider inherent performance of each thread  Unfairly prioritizes threads with bursty access patterns (idleness problem)  Unfairly penalizes threads with unbalanced bank usage (in paper)

26. 26 Idleness/Burstiness Problem in Fair Queueing Thread 1’s virtual time increases even though no other thread needs DRAMOnly Thread 2 serviced in interval [t1,t2] since its virtual time is smaller than Thread 1’sOnly Thread 3 serviced in interval [t2,t3] since its virtual time is smaller than Thread 1’sOnly Thread 4 serviced in interval [t3,t4] since its virtual time is smaller than Thread 1’s Non-bursty thread suffers large performance loss even though it fairly utilized DRAM when no other thread needed it Serviced

27. 27 Unfairness on 4-, 8-, 16-core Systems Unfairness = MAX Memory Slowdown / MIN Memory Slowdown 1.27X1.81X 1.26X

28. 28 System Performance 5.8%4.1%4.6%

29. 29 Hmean-speedup (Throughput-Fairness Balance) 10.8%9.5%11.2%

30. 30 Outline The Problem  Unfair DRAM Scheduling Stall-Time Fair Memory Scheduling  Fairness definition  Algorithm  Implementation  System software support Experimental Evaluation Conclusions

31. 31 Conclusions A new definition of DRAM fairness: stall-time fairness  Equal-priority threads should experience equal memory-related slowdowns  Takes into account inherent memory performance of threads New DRAM scheduling algorithm enforces this definition  Flexible and configurable fairness substrate  Supports system-level thread priorities/weights  QoS policies Results across a wide range of workloads and systems show:  Improving DRAM fairness also improves system throughput  STFM provides better fairness and system performance than previously-proposed DRAM schedulers

32. Thank you. Questions?

33. Stall-Time Fair Memory Access Scheduling Onur Mutlu and Thomas Moscibroda Computer Architecture Group Microsoft Research

34. Backup

35. 35 Structure of the STFM Controller

36. 36 Comparison using NFQ QoS Metrics Nesbit et al. [MICRO’06] proposed the following target for quality of service:  A thread that is allocated 1/N th of the memory system bandwidth will run no slower than the same thread on a private memory system running at 1/N th of the frequency of the shared physical memory system  Baseline with memory bandwidth scaled down by N We compared different DRAM schedulers’ effectiveness using this metric  Number of violations of the above QoS target  Harmonic mean of IPC normalized to the above baseline

37. 37 Violations of the NFQ QoS Target

38. 38 Hmean Normalized IPC using NFQ Baseline 10.3% 9.1% 7.8% 7.3%5.9%5.1%

39. 39 Shortcomings of the NFQ QoS Target Low baseline (easily achievable target) for equal-priority threads  N equal-priority threads  a thread should do better than on a system with 1/N th of the memory bandwidth  This target is usually very easy to achieve Especially when N is large Unachievable target in some cases  Consider two threads always accessing the same bank in an interleaved fashion  too much interference Baseline performance very difficult to determine in a real system  Cannot scale memory frequency arbitrarily  Not knowing baseline performance makes it difficult to set thread priorities (how much bandwidth to assign to each thread)

40. 40 A Case Study Unfairness: 7.28 2.07 2.08 1.87 1.27 Memory Slowdown

41. 41 Windows Desktop Workloads

42. 42 Enforcing Thread Weights

43. 43 Effect of 

44. 44 Effect of Banks and Row Buffer Size