ARGUS: Rete + DBMS = Efficient Persistent Profile Matching on Large-Volume Data Streams Chun Jin Language Technologies Institute School of Computer Science Carnegie Mellon University

Chun Jin Carnegie Mellon 2 Stream Processing Model Stream Processing becomes demanding and prevalent. Storage Data Streams Output

Chun Jin Carnegie Mellon 3 Stream Databases Stream Database Applications Network Traffic Analysis and Router Configuration Dynamic Internet Services Sensor Data Analysis Anomaly Detection Stream Database Projects STREAM, TelegraphCQ, Aurora NiagaraCQ, OpenCQ, WebCQ Gigascope, Tribeca Tapestry, Alert, Tukwila, etc. ARGUS

Chun Jin Carnegie Mellon 4 Stream Anomaly Monitoring Systems (SAMS) SAMS monitors structured data streams for anomalies or potential hazards. Matches of queries may be high urgency alerts. Prompt detections are desirable. Satisfaction of a SAMS query is often rare (very-high-selectivity).

Chun Jin Carnegie Mellon 5 SAMS Dataflow Analyst Stream Anomaly Monitoring System Storage Queries Alerts Data Streams FedWire Money Transfers Patient Records

Chun Jin Carnegie Mellon 6 Challenges to SAMS Persistent queries may number in thousands or tens of thousands. Daily stream volumes may exceed millions of records. Prompt detections are desirable. Very-high-selectivity Query Property.

Chun Jin Carnegie Mellon 7 Proposed ARGUS Approach Basic Framework: Incremental evaluation schemes (Adapted Rete algorithm) Rete (Forgy 1982): Incremental Evaluation based on Materialized Intermediate Results. Upon a traditional DBMS platform Exploiting Very-High-Selectivity Query Property: Transitivity Inference Conditional Materialization Optimizing Join Order Computation Sharing Related to Other Applications Stream Databases Modern DBMS Query Optimization

Chun Jin Carnegie Mellon 8 Query Example 4 Suppose for every big transaction of type code 1000, the analyst wants to check if the money stayed in the bank or left within ten days. An additional sign of possible fraud is that transactions involve at least one intermediate bank. The query generates an alarm whenever the receiver of a large transaction (over $1,000,000) transfers at least half of the money further within ten days of this transaction using an intermediate bank.

Chun Jin Carnegie Mellon 9 SQL Query for Example 4 FROM transaction r1, transaction r2, transaction r3 WHERE r2.type_code = 1000 AND r3.type_code = 1000 AND r1.type_code = 1000 AND r1.amount > AND r1.rbank_aba = r2.sbank_aba AND r1.benef_account = r2.orig_account AND r2.amount > 0.5 * r1.amount AND r1.tran_date <= r2.tran_date AND r2.tran_date <= r1.tran_date + 10 AND r2.rbank_aba = r3.sbank_aba AND r2.benef_account = r3.orig_account AND r2.amount = r3.amount AND r2.tran_date <= r3.tran_date AND r3.tran_date <= r2.tran_date + 10;

Chun Jin Carnegie Mellon 10 ARGUS System Architecture Rete Network Generator Query Rete Networks Data Tables Analyst Identified Threats Intermediate Tables Data Streams Query Table Stream Anomaly Monitoring Do_queries Scheduler

Chun Jin Carnegie Mellon 11 ReteGenerator Architecture System Catalog Transitivity Inference SQL Queries ReteGenerator Sharing Module Join Order Conditional Materialization Optimizer Common Computation Identification Predicate Indexing Extended Predicate Set Operations Choose what and how to share Recording and Manipulating Network Topology Estimating Sharing Costs

Chun Jin Carnegie Mellon 12 Adapted Rete Algorithm (Selection) n and m are old data sets Δn and Δm are the new much smaller incremental data sets. Selection ơ ơ(n+ Δn) ơ(n) ơ(Δn)= +

Chun Jin Carnegie Mellon 13 Adapted Rete Algorithm (Join) Join (n+Δn) (m+Δm) = n m + Δn m + n Δm + Δn Δm When Δn and Δm are very small compared to n and m, time complexity of incremental join is O(n+m) Old Results New Incremental Results

Chun Jin Carnegie Mellon 14 Incremental Evaluation in Rete Example 4 DataTable r1, r2, r3 Type_code=1000 Amount> Type_code=1000 r1.rbank_aba = r2.sbank_aba r1.benef_account = r2.orig_account r2.amount > r1.amount*0.5 r1.tran_date <= r2.tran_date r2.tran_date >= r1.tran_date+10 r2.rbank_aba = r3.sbank_aba r2.benef_account = r3.orig_account r2.amount = r3.amount r2.tran_date <= r3.tran_date r3.tran_date >= r2.tran_date+10

Chun Jin Carnegie Mellon 15 Complex Queries A persistent query may contain multiple SQL statements, and a single SQL statement may contain unions of multiple SQL terms. Each SQL term is mapped to a sub-Rete network. These sub-Rete networks are then connected to form the statement-level sub-networks. And the statement-level subnetworks are further connected based on the view references to form the final query-level Rete network.

Chun Jin Carnegie Mellon 16 Transitivity Inference Exploring transitivity properties of comparison operators To derive hidden high-selective selection predicates High-selective selection predicates can significantly improve performance as they may produce very small intermediate results. Subsequent join could be performed very fast on the materialized intermediate results. Ono/Lohman VLDB90, Pirahesh/Leung/Hasan ICDE97

Chun Jin Carnegie Mellon 17 Transitivity Inference Example Given r1.amount > and r2.amount > r1.amount * 0.5 and r3.amount = r2.amount r1.amount > is very high- selective on r1 We can infer high-selective predicates: r2.amount > r3.amount >

Chun Jin Carnegie Mellon 18 Conditional Materialization r2 r1 r2 r1 Unconditional Materialization Conditional Materialization: Choose materialization or not based on cost estimates

Chun Jin Carnegie Mellon 19 Preliminary Evaluation: Queries and Data 7 queries on synthesized FedWire money transfer database records. Two Data Conditions: Data1: Old: first records New: remaining records ALERT Data2: Old: first records New: next records NOT alert

Chun Jin Carnegie Mellon 20 Preliminary Results Rete with Transitivity Inference Q1Q2Q3Q4Q5Q6Q7 Execution Time(s) Rete Data1SQL Data1Rete Data2SQL Data2

Chun Jin Carnegie Mellon 21 Transitivity Inference Q2 Q Data1Data2 Execution Time(s) Data1Data2 Execution Time(s) Rete TIRete Non-TISQL Non-TISQL TI

Chun Jin Carnegie Mellon 22 Conditional Materialization Q4 assumes Transitivity Inference not applicable Data1Data2 Execution Time(s) Conditional Rete SQL

Chun Jin Carnegie Mellon 23 ARGUS Summary Adapted Rete Algorithm upon a traditional DBMS platform Exploit the very-high-selectivity query property for optimization: Transitivity Inference Conditional Materialization Current and Future Work: Optimizing Join Order Computation Sharing

Chun Jin Carnegie Mellon 24 Thank you! Questions and Comments?