Moirae: History-Enhanced Monitoring Magdalena Balazinska, YongChul Kwon, Nathan Kuchta, and Dennis Lee University of Washington and Marchex Inc.

Monitoring Applications Continuously observe current state Produce near real-time information and alerts Event: (eid, timestamp, a1, ..., an) Examples Sensor-based environment monitoring Computer system monitoring Network intrusion detection load level time

Problem: Exploit History Monitoring applications accumulate history Exploiting history can improve monitoring apps Refine event detection Explain newly detected events What types of queries to support? How to support them?

Types of Queries Standard hybrid queries Contextual hybrid queries Standard SQL query over the data archive executes as part of the continuous query “For each network intrusion, show historical activity of the intruder on the network” Contextual hybrid queries For each newly detected event, produce approximate set of k most similar past events “If a server fails, show similar alerts that occurred in the past”

historical information Standard Hybrid Query Query model based on Borealis Input Streams Stream Proc. Operators Recall Other Stream Proc. Ops. Continuous stream processing (event query) Archive Look up historical information (historical query)

Contextual Hybrid Query Similar past events Input Stream Event detection Event Similarity Recall Input Stream Window Join Past contexts Context Input Sream Window Join Find similar past events Use TF-IDF for similarity computation Archive

Framework Three key issues Three goals History size, near real-time, concurrent events Three goals Responsiveness and fairness Incremental processing integrated with stream processing Retrieve at least some historical data for all new events Relevance: favor recent history over older history Similarity: find similar past events Exploit context similarity in other ways as well

Moirae’s Design Based on Borealis Based on PostgreSQL Approximate & incrementally improving results Stop Improving Query Contextual & Standard hybrid queries Application MOIRAE SPE Stream Processor Deploy Manager Based on Borealis Raw Streams Recall Manager RDBMS Storage Manager Based on PostgreSQL Archiver Materialized Events & Context Raw Stream Archive Other Materialized Views Present Chunk

Design Components Archiver: partitioned stream archive Archive raw and intermediate streams Present chunks in memory Recent chunks on disk Materialize & index necessary streams and contexts Old chunks on disk Recall Manager: partitioned, incremental queries Execute queries one chunk at the time (present ->past) Schedule concurrent queries to ensure fairness Incorporate user feedback to drop events

Related Work Queries over live data & data archives [chandrasekaran:04,chandrasekaran:05, franklin:05] Log-structured access method [muth:00] Multi-level storage manager [stonebraker:91] Materialized views (e.g., [goldstein:01]) Partial indexes [stonebraker:89,sartori:94,seshadri:95] Online processing [hellerstein:97,hellerstein:00, raman:02,shanmugasundaram:01,tan:99] top-K, kNN, IR+RDBMS [e.g.,carey:97,fagin:03,chaudhuri:05,li:05]

Conclusion Monitoring applications accumulate history How to leverage history ? By supporting queries for specific historical data Through new types of queries How to support all these different queries ? Can reuse several database/IR techniques Need to integrate and extend these techniques More information about Moirae http://data.cs.washington.edu/moirae/moirae.shtml