p2p, Fall 06 1 Querying the Internet with PIER (PIER = Peer-to-peer Information Exchange and Retrieval) VLDB 2003 Ryan Huebsch, Joe Hellerstein, Nick Lanham, Boon Thau Loo, Timothy Roscoe, Scott Shenker, Ion Stoica

p2p, Fall 06 2 What is PIER? A query engine that scales up to thousands of participating nodes = relational queries + DHT Built on top of a DHT

p2p, Fall 06 3 Motivation, Why? In situ distributed querying (as opposed to warehousing) Network monitoring network intrusion detection: sharing and querying fingerprint information Fingerprints (sequences of port accesses, port numbers, packet contents etc) Where: network servers: mail servers, web servers, remote shells) & applications (mail clients, web browsers) Publish the fingerprint of an attack in PIER for a short time Query it – similar reports, statistics, intrusions from the same domain, etc

p2p, Fall 06 4 Motivation, Why? Besides security: network statistics (eg bandwidth utilization, etc) In-network processing – streams Text search Deep web crawling

p2p, Fall 06 5 Architecture DHT is divided into 3 modules:  Routing Layer  Storage Manager  Provider Goal is to make each simple and replaceable An instance of each DHT and PIER component runs at each participating node PIER is build on top of a DHT (in the paper, the DHT is CAN with d = 4) The DHT is used both for indexing and storage

p2p, Fall 06 6 Architecture Routing Layer Maps a key to the IP address of the node currently responsible for it API lookup(key) -> ippaddr join(landmark) leave() LocationMapChange() Callback used to notify higher levels asynchronously when a set of keywords mapped locally has changed Core Relational Execution Engine ProviderStorage Manager Overlay Routing Catalog Manager Query Optimizer Various User Applications PIER DHT Apps

p2p, Fall 06 7 Architecture Storage Manager Temporary storage of DHT-based data Local database at each DHT node a simple in memory storage- system or a disk-based package like Berkeley DB In this paper, a main memory storage manager API store(key, item) retrieve(key) -> item remove(key) Core Relational Execution Engine Provider Storage Manager Overlay Routing Catalog Manager Query Optimizer Various User Applications PIER DHT Apps

p2p, Fall 06 8 Architecture Provider What PIER sees What are the data items (relations) handled by PIER? Core Relational Execution Engine Provider Storage Manager Overlay Routing Catalog Manager Query Optimizer Various User Applications PIER DHT Apps

p2p, Fall 06 9 Naming Scheme Each object: (namespace, resourceID, instanceID) DHT key: hash on namespace, resourceID  Namespace: group, application the object belongs to In PIER, the Relation Name No need to be predefined, can be created implicitly when the first item is put and destroyed when the last item expires  ResourceID: some semantic meaning In PIER, the value of the primary key for base tuples  InstanceID: an integer randomly assigned by the user application Used by the storage manager to separate items (when they are not stored based on the primary key) What are the data items (relations) handled by PIER?

p2p, Fall Soft State Each object associated with a lifetime: how long should the DHT store the object To extend it, must use periodical RENEW calls

p2p, Fall Architecture Provider API get(namespace, resourceID) -> item put(namespace, resourceID, instanceID, item, lifetime) renew(namespace, resourceID, instanceID, item, lifetime) -> bool multicast(namespace, resourceID, item) Contacts all nodes that hold data in a particular namespace lscan(namespace) -> iterator Scans over all data stored locally newData(namespace) -> item Callback to the application to inform it that new data has arrived in a particular namespace

p2p, Fall Architecture  PIER currently only one primary module: the relational execution engine  Executes a pre-optimized query plan  Traditional DBs use an optimizer + catalog to take SQL and generate the query plan, those are “just” add-ons to PIER

p2p, Fall Architecture Query plan is a box-and-arrow description of how to connect basic operators together  selection, projection, join, group-by/aggregation, and some DHT specific operators such as rehash No iterator model Instead, operators produced results as quickly as possible and enqueue the data for the next operator

p2p, Fall Joins: The Core of Query Processing R Join S, relations R and S stored in separate namespaces N R and N S How: –Get tuples that have the same value for a particular attribute(s) (the join attribute(s)) to the same site, then append tuples together Why Joins? A relational join can be used to calculate: –The intersection of two sets –Correlate information –Find matching data Algorithms come from existing database literature, minor adaptations to use DHT.

p2p, Fall Symmetric Hash Join (SHJ) Algorithm for each site –(Scan – Retrieve local data) Use two lscan calls to retrieve all data stored locally from the source tables –(Rehash based on the join attribute) put a copy of each eligible tuple with the hash key based on the value of the join attribute (new unique namespace N Q ) –(Listen) use newData and get to N Q to see the rehashed tuples –(Compute) Run standard one-site join algorithm on the tuples as they arrive Scan/Rehash steps must be run on all sites that store source data Listen/Compute steps can be run on fewer nodes by choosing the hash key differently

p2p, Fall Fetch Matches (FM) Algorithm for each site –(Scan) Use lscan to retrieve all data from ONE table N R –(Get) Based on the value for the join attribute, for each R tuple issue a get for the possible matching tuples from the S table Big picture: –SHJ is put based –FM is get based When one of the tables, say S is already hashed on the join attribute

p2p, Fall Joins: Additional Strategies Symmetric Semi-Join –Locally at each site, run a SHJ on the source data projected to only have the hash key and join attributes. –Use the results of this mini-join as source for two FM joins to retrieve the other attributes for tuples that are likely to be in the answer set Bloom Filters –Create Bloom filters for each local R and S data and publish them at a temporary DHT, OR all local R and multicast to nodes hosting S – use lscan but rehash only those that match Big Picture: –Tradeoff bandwidth (extra rehashing) for latency (time to exchange filters)

p2p, Fall Naïve Group-By/Aggregation A group-by/aggregation can be used to calculate: –Split data into groups based on value –Max, Min, Sum, Count, etc. Goal: –Get tuples that have the same value for a particular attribute(s) (group-by attribute(s)) at the same site, then summarize data (aggregation).

p2p, Fall Naïve Group-By/Aggregation At each site –(Scan) lscan the source table Determine group tuple belongs in Add tuple’s data to that group’s partial summary –(Rehash) for each group represented at the site, rehash the summary tuple with hash key based on group-by attribute –(Combine) use newData to get partial summaries, combine and produce final result after specified time, number of partial results, or rate of input Hierarchical Aggregation: Can add multiple layers of rehash/combine to reduce fan-in. –Subdivide groups in subgroups by randomly appending a number to the group’s key

p2p, Fall Naïve Group-By/Aggregation Sources Root Application Overlay Sources Root … Each message may take multiple hops Each level fewer nodes participate

p2p, Fall Codebase Approximately 17,600 lines of NCSS Java Code Same code (overlay components/pier) run on the simulator or over a real network without changes Runs simple simulations with up to 10k nodes –Limiting factor: 2GB addressable memory for the JVM (in Linux) Runs on Millennium and Planet Lab up to 64 nodes –Limiting factor: Available/working nodes & setup time Code: –Basic implementations of Chord and CAN –Selection, projection, joins (4 methods), and naïve aggregation. –Non-continuous queries

p2p, Fall Seems to scale Simulations of 1 SHJ Join Warehousing Full Parallelization

p2p, Fall Some real-world results 1 SHJ Join on Millennium Cluster

p2p, Fall Other applications Recursive queries on network graphs set of nodes reachable within k hops of each node Hierarchical aggregation Range Predicates locality-preserving hash functions