Using Uncacheable Memory to Improve Unity Linux Performance Ning Qu Xiaogang Gou Xu Cheng Microprocessor Research and Development Center Peking University

Unity SoC architecture Issues Unity SoC architecture No snooping Cache coherency problem everywhere !! Peking University

Issues cont. poor temporal locality! DMA DMA User Process User Process process I/O buffer process I/O buffer poor temporal locality! Linux Kernel Linux Kernel kernel I/O buffer kernel I/O buffer DMA DMA I/O device buffer I/O device buffer I/O Device I/O Device Peking University

Motivation How to avoid the disadvantages? Heavy cost of Cache coherency operations Many high-end embedded processors have Cache, But many of them have very limited support to guarantee cache coherency Poor locality leads to more data Cache pollution Cache is based on property of locality Some programs have poor locality, for example TCP/IP processing How to avoid the disadvantages? Uncacheable memory may be a solution! 1) For example, Unity I and some ARM processors doesn’t support single cache line operation 2) Some programs have poor locality, hence the existing cache policies do not help Peking University

Contributions Analyze the scenarios in which Cache doesn’t perform well, propose uncacheable memory has two advantages Eliminate most of Cache coherency operations Avoid Cache pollution Apply uncacheable memory in Unity Linux to improve the I/O performance. Some important aspects improves from 5% - 29% Won’t hurt the system performance with carefully design Peking University

Outline Issues Motivation Contribution Uncacheable Memory Evaluation Related Work Conclusions Peking University

using uncacheable memory Recv Packet Flow step 1 step 2 step 3 step 4 using uncacheable memory User Space Simple data processing flush cache User Buffer Kernel Space Buffer Buffer Buffer Buffer I/O Device CPU copy Step3: 1) Uncacheable memory access is slow 2) Simple data processing means low cache pollution Step4: 1) Uncacheable memory has no cache pollution, but access is slow 2) Cacheable memory has much cache pollution, but access is fast DMA copy Peking University

using uncacheable memory Send Packet Flow step 1 step 2 step 3 step 4 using uncacheable memory User Space User Buffer clean cache DMA copy Kernel Space Buffer Buffer Buffer Buffer CPU copy Simple data processing I/O Device Step1: 1) Uncacheable memory write is slow, but has no cache pollution 2) Cacheable memory has to do write allocates and pollute cache, but has access acceleration Peking University

Cacheable vs. Uncacheable Send Receive CH processing 1. copy from U to K 2. clean data cache 1. clean&invalidate data cache 2. copy from K to U NC 1. copy from U to K(N) 1. copy from K(N) to U side effect 1. accessing uncacheable memory is slower 2. no data cache pollution 3. no cache clean operation 3. no cache flush operation DMA send and receive cost analysis Peking University

Cacheable vs. Uncacheable cont. Cache clean cost DMA Send: load U to Cache load U into Cache load K to Cache store to K load K to Cache load K Cache flush cost load U into Cache and store DMA Recv: Peking University

Cacheable vs. Uncacheable cont. Single write cost is small, but single read cost is great! Send Uncacheable approach cost = write K once in single write mode. Cacheable approach cost >= read K once in burst mode and write K once in burst mode. Receive Uncacheable approach cost = read K once in single read mode. Cacheable approach cost = read K once in burst mode. To evaluate the cost and the TCP/IP processing performance improvement by uncacheable method compared with hardware cacheable method, we design a simplified experiment according to the TCP/IP processing described in Table There are two factors in the experiment, one is the data length which determines the cost of data copy operations and the other is the data cache dirty ratio which determines the cost of data cache’s clean&invalidate operations We implement a kernel module to measure the cost of sending and receiving packets by these two methods In the diagrams Cache(6.25%) means cacheable method with 6.25% data cache dirty ratio and NonC means uncacheable. The results convince us that it’s likely to gain benefits from uncacheable method when sending packets. But for receiving packets, uncacheable method costs too much because of copy and TCP/IP processing. Recv and Send Performance CH vs NC Peking University

Using Uncacheable Memory Implemented in Unity Linux ported from Linux 2.4.17 Uncacheable page table eliminate Cache coherency operations when modifying the page tables Uncacheable socket buffer for sending eliminate Cache coherency operations avoid data Cache pollution Peking University

Outline Motivation Issues Contribution Uncacheable Memory? Evaluation Related Work Conclusions Peking University

Methodology Benchmarks: Netperf, Lmbench and Modified Andrew benchmark. Experiments environment 160 MHz Unity network computer with 256 MB DRAM, a SoC build-in 10M/100M Ethernet card Dell 4600 server, two Intel Xeon PIII 700 MHz processors with 4 GB DRAM and 1000M/100M Ethernet card All benchmarks are executed in single-user mode on NFS. Peking University

Netperf Benchmark Results Netperf TCP_STREAM Send Performance Peking University

Netperf Benchmark Results cont. Q: When receive size is increasing, transactions throughput improvement is decreasing. ? using uncacheable memory in recv socket buffer Netperf TCP_RR Performance Peking University

Lmbench Benchmark Results Lmbench Performance Peking University

Modified Andrew Benchmark Results As expected, execution time reduces from 6%-12% for the first four phases. For Phase V, the time reduction is less than 1%. This is reasonable because only Phase V heavily depends on computation instead of I/O. Summary: Based on these results, We believe by using uncacheable memory with careful design, overall performance of Unity system will outperform the implementation by simply using cacheable memory and cache hardware operations. Modified Andrew Benchmark Peking University

Related Work Future work: new memory type support Related work: accelerate uncacheable memory performance New memory type Intel write-combining MIPS R10000: uncached-accelerated page New instructions SPARC V9, ARM, Unity II: block move instructions Future work: new memory type support Read like common cache with low pollution Write like Write-Combining without write-allocate Peking University

Conclusions This paper focuses on the uncacheable memory usage. Pros: eliminating coherency operations and avoiding data Cache pollution. Cons: slow accessing time Uncacheable memory can perform well with a carefully design when considering system specialties Many embedded architectures have a lot of their own design specialties due to the limitation on energy cost, area size and design complexity There may be more fields the uncacheable memory can be applied in Peking University

Thank You! Questions? Peking University