Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 1 Ditto : Eavesdropping for the Common Good in Multi-hop Wireless Networks Amar Phanishayee Fahad Dogar, Himabindu Pucha, Olatunji Ruwase, Dave Andersen Carnegie Mellon University CMU Speaking Skills Talk

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 2 Wireless Networks Wired Link Wireless Router Wireless Link Cable Modem Access-Point Based (Single Wireless Hop)

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 3 No Infrastructure Cost Effective, Greater Coverage Poor throughput Wireless Networks Multi-hop Wireless (Mesh) Networks Gateway

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 4 Hop Count Affects Throughput Pairwise transfers in a 28 node campus testbed More hops  slower transfers (low throughput) Median 3.2Mbps 0.5Mbps 0.2Mbps

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 5 Ditto: Key Contributions Ditto improves throughput by reducing hop count to acquire commonly requested data Caching of overheard data Also cache data on the path of a transfer Application independent caching Ditto improves throughput (on 2 testbeds) Up to 7x better than on-path caching Up to 10x better than no caching

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 6 Caching In Ditto Helps Speed Transfers On-Path + Opportunistic Caching  Ditto Path of the transfer: Alice--A1--A2--Gateway A1 and A2 -- on-path caching A3 -- opportunistic caching A2 A3 Gateway A1 Request Response Cached copy

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 7 Outline Overview Why poor throughput in mesh networks? Ditto’s Design Evaluation Future Work

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 8 Transmit Range: Range in which router’s transmission can be fully decoded Interference Range: Range in which router’s transmission can interfere but may not be fully decoded Interference Interference: Receivers in Interference Range cannot decode simultaneous transmissions

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 9 Avoiding Interference: Don’t Transmit If You Know Others Are Transmitting one at a time avoids interference

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 10 Gateway A3 A1A2 Subsequent Hops Interfere D1’s transmission from A2 to A3 stalled Cause for poor throughput in mesh networks Subsequent Wireless Hops Interfere D1 D2

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 11 Mesh Networks Have Poor Throughput Some links have a high loss rate Subsequent hops interfere with each other More hops  slower transfers (low throughput) Gateway is a bottleneck A1 A2 A3 Gateway

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 12 Outline Overview Why poor throughput in mesh networks? Ditto’s Design Opportunities to improve throughput How does Ditto utilize these opportunities? Evaluation Future Work

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 13 Opportunities To Improve Throughput Locality in access patterns Many clients from different parts of the network request the same data over time e.g. Operating System Updates, Streaming video Wireless routers can overhear other transfers The positive aspect of broadcast Looks like a job for … Opportunistic Caching

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 14 Challenge: Lossy Overhearing Wireless networks have high loss rates Loss recovery: link layer retransmissions Overhearing node also experiences losses Losses independent of that at Receiver Cannot ask for retransmissions Successfully overhearing a large file is unlikely A3 A1 A A1 A2

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 15 Chunk Based Transfers Lossy overhearing  cannot overhear entire file Smaller caching granularity Divide file into smaller chunks (8 – 32 KB) Use chunk as a unit of transfer Ditto uses Data Oriented Transfer (DOT) 1 for chunk based transfers 1 Tolia et al, An Architecture for Internet Data Transfer. NSDI An Architecture for Internet Data Transfer

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 16 A Ditto Transfer chunkID3 chunkID1 chunkID2 foo.txtfoo.txt Cryptographic Hash Chunking App DITTO chunk ids Chunk request DITTO Chunk response request response request response ReceiverSender Ditto Proxy Request – foo.txt Response: chunk ids{1,2,3} chunk ids

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 17 Proxy Serves Chunk Request Directly If Chunk In Local Cache Proxy Chunk Request Chunk Response Cache

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 18 Proxy Routes Request Towards Gateway If Chunk Not In Cache Next-Hop based on routing table information Separate TCP connection on each hop On-Path Caching Proxy Chunk Request Chunk Response Cache Chunk Request Chunk Response GWNext Hop

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 19 Sniffer Module Caches Overheard Chunks Opportunistic Caching Proxy Sniffer Cache Overhear and reassemble streams Identify chunks in streams Handoff entire chunk to proxy for caching

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 20 Outline Overview Why poor throughput in mesh networks? Ditto’s Design Why caching, overhearing? Chunk based transfers and caching How does Ditto overhear and reconstruct chunks? Evaluation Future Work

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 21 Sniffer Module Reassembles Overheard Streams A3 (Overhearing) Path of the transfer: Alice – A1 – A2 … Stream identification & placement within the stream Next Steps: identify chunk A1 A2

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 22 Chunk Identification A3 (Overhearing) Path of the transfer: Alice – A1 – A2 … A1 A2 Chunk Boundaries Look for Ditto header What if there were losses within the overheard chunk? C1

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 23 Optimization: Inter-Stream Chunk Reassembly Look for Ditto header Chunk Boundaries Sniffer identifies the same chunk across streams Correlates and fills in the blanks A1 A2 C1

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 24 Outline Overview Why poor throughput in mesh networks? Ditto’s Design Evaluation Reconstruction Efficiency Transfer Throughput Future Work

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 25 Emulab Indoor Wireless Testbed

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 26 MAP Campus Testbed (Purdue Univ.) Gateway node Indoor/Outdoor Campus-wide Mesh Network

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 27 Experiment Can we overhear complete chunks? Receiver Observer A receiver initiates transfer 1MB file, 8KB chunks Observers report % of chunks reconstructed Receiver reports throughput Caches are cleared & next node becomes a receiver If n nodes in network, [n* (n - 1)] observers in all

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 28 Overhearing Complete Chunks Is Feasible Overhearing Complete Chunks Is Feasible Around 30% of the observers reconstruct at least 50% chunks For any transfer, ~30% of the nodes can overhear a large fraction of the chunks Complete chunks overheard (%)

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 29 Nodes farthest from the gateway & transfer paths overhear nothing Complete chunks overheard (%)

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 30 Gateway

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 31 Nodes closer to the gateway & transfer paths overhear effectively Complete chunks overheard (%)

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 32 Gateway Shield the gateway from becoming a bottleneck

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 33 Factors Affecting Chunk Reconstruction FactorInsight ProximityNodes closer to gateway overhear more chunks. Can shield the gateway from becoming a hotspot. Chunk Size Smaller Chunk Size  Better Reconstruction 8 KB provides good reconstruction efficiency with low overhead.

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 34 Experiment: Throughput Evaluation Receiver Observer Leaf nodes are receivers A receiver initiates transfer reports throughput Next receiver downloads the same file

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 35 Opportunistic Caching >> On-path Caching Receiver On-path Caching Overhearing & Caching

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 36 Throughput Improvement Using Ditto Median Log Scale! ~900 Kbps No Caching

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 37 Throughput Improvement Using Ditto Median Log Scale! ~1200Kbps ~900 Kbps No Caching

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 38 Throughput Improvement Using Ditto Opportunistic caching >> On-path caching > No caching Median Log Scale! ~1200Kbps ~900 Kbps 9000 Kbps No Caching

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 39 Future Work Support for mobility Alternate proxy selection techniques Application specific Ditto gateways

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 40 Conclusion Ditto improves throughput by reducing hop count to acquire commonly requested data Caching of overheard data Also cache data on the path of a transfer Chunk based transfer, inter-stream chunk reconstruction Ditto improves throughput (on 2 testbeds) Up to 7x better than on-path caching Up to 10x better than no caching

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 41 Thank you!

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 42 Extra Slides

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 43 Experimental Setup Mode b RateAuto Other ParametersDefault RoutingStatic OLSR (Setup) Cross TrafficNo Cache EvictionNo File Sizes1MB - 5 MB Chunk Sizes8KB, 16KB, 32KB

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 44 Hop Count Affects Throughput 85% > 1Mbps 35% > 1Mbps 10% > 1Mbps Pairwise transfers in a 28 node campus testbed More hops  slower transfers (low throughput)

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 45 Evaluation Summary Metric/FactorInsight ProximityNodes closer to gateway have a very high reconstruction efficiency. Can shield the gateway from becoming a hotspot. Chunk SizeSmaller the chunk size better the reconstruction efficiency. 8 KB provides good reconstruction efficiency with low overhead. ThroughputUp to an order of magnitude improvement with Ditto. Inter-Stream Reassembly A few opportunities for multiple overhearing; 10% improvement where applicable

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 46 Related Work Hierarchical Caching [Fan98, Das07,..] Caching more effective on lossy wireless links Ditto’s overhearing feature is unique Packet Level Caching [Spring00, Afanasyev08] Ditto is purely opportunistic Ditto exploits similarity at inter-request timescale Making the best of broadcast [MORE, ExOR,..] Largely orthogonal

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 47 Packet Format

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 48 (Chunk) Size Matters Average Reconstruction Effectiveness (% chunks) 8KB 70% of the observers reconstructed at most 50% of total chunks Complete chunks overheard (%)

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 49 (Chunk) Size Matters 32KB 70% of the observers reconstructed at most 10% of total chunks 8KB 70% of the observers reconstructed at most 50% of total chunks Smaller Chunk Size  Better Reconstruction Effectiveness Complete chunks overheard (%)

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 50 Inter-Stream Reassembly Reconstruction Efficiency

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 51 Useless Bytes --- Different Chunk Sizes

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 52 Useless Bytes Reduction Due to Inter- Stream Reassembly

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 53 Rabin Fingerprinting File Data Rabin Fingerprints Given Value - 8 Natural Boundary Hash 1 Hash 2

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 54 Rabin Fingerprinting: Examples of Edits 1. Original File 2. Addition in chunk Changes only one hash 3. Addition creating a new breakpoint 4. Deletion changing size of chunk Figure from “A Low-bandwidth Network File System”

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 55 Does Ditto work with encrypted data? If the data is only meant for one user Then opportunistic caching doesn’t work and it need not work If the data is shared by many users all of them can have a shared key and data can be encrypted using this shared key Convergent Encryption Uses content hash as the encryption key Middle ground between strong encryption and low overheard

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 56

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 57 How much traffic locality is there in mesh network? No accurate estimate as yet! Reasonable to assume that locality is present Similar users in mesh networks --- e.g: software updates, video viewing, DARPA’s military apps Internet access exhibits locality Packet Level (roughly % traffic is redundant) HTTP caching (lower than packet level caching)

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 58 Overhead: Cache Size Practically need some cache eviction policy Cheap Storage -> bigger caches can be maintained Worst case analysis using numbers from Meraki’s mesh deployments 2 Deployments and 10 Meraki minis respectively Assuming a mesh node overhears ALL traffic passing through other mesh nodes Most software updates take place within 24 hours 3 GB cache size is enough to serve software updates

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 59 Overhead Overhead of Sniffer Sniffer overhears at line speed Sniffer runs as a separate process and doesn’t affect performance Didn’t miss a picket at g 54 Mbps transfer We filter out non-Ditto traffic Overhead of Proxying Going up the stack can be costly Separate TCP connection can be beneficial (lower RTT) We observe that things balance out in practice i.e., performance with/without proxying is similar

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 60 FAQ Should one design multi-hop networks with overhearing in mind? Our approach --- No --- we didn’t want the request that doesn’t benefit from caching to suffer Other approach can be adopted based on the goals (for example: goal is to minimize the wired bw used by the gateway) What if routes change? System wouldn’t break But you would have longer/sub-optimal paths (next-hop Ditto proxy remains the same) Recent study(PAM 2007) shows that in mesh networks you only have 1-2 very good paths to the gw, so better route selection can minimize this problem

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 61 FAQ Is Ditto’s evaluation the best-case scenario? Yes, in terms of cache hit No, in terms of request pattern (we consider a random order request pattern) Cost of missing the chunk header? Offline analysis shows that we could’ve constructed approx. 5% more chunks had we received the chunk headers

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 62 FAQ Impact of different things? Multi-Radio: Less interference but still reducing number of transmissions and hence interference is important Different mode (a/g): Opportunistic caching still possible (we tested that on a smaller set of nodes) --- we don’t know how #s will compare to our current eval. Directional antennas: less overhearing -> opportunistic caching less useful Rate selection: we only tried auto-rate, shouldn’t fundamentally change results with other mechanisms Cross Traffic: Reconstruction #s shouldn’t be too different, throughput #s may be lower for all schemes

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 63 FAQ Do we necessarily need Ditto on the server side? No --- we can have an application level proxy at the gateway that translates between the application and Ditto’s chunk based transfer Do we need Ditto on all the nodes? No, some nodes in the mesh could just forward traffic (simple mesh routers) Do we need Ditto on the client? Yes! You can’t skip that.

Carnegie Mellon University Computer Science Department Carnegie Mellon University Computer Science Department 64 icons