1. Moirae: History-Enhanced Monitoring
Magdalena Balazinska, YongChul Kwon, Nathan Kuchta, and Dennis Lee
University of Washington
and Marchex Inc.

2. Monitoring Applications
Continuously observe current state
Produce near real-time information and alerts
Event: (eid, timestamp, a1, ..., an)
Examples
Sensor-based environment monitoring
Computer system monitoring
Network intrusion detection
load level
time

3. Problem: Exploit History
Monitoring applications accumulate history
Exploiting history can improve monitoring apps
Refine event detection
Explain newly detected events
What types of queries to support?
How to support them?

4. Types of Queries Standard hybrid queries Contextual hybrid queries
Standard SQL query over the data archive executes as part of the continuous query
“For each network intrusion, show historical activity of the intruder on the network”
Contextual hybrid queries
For each newly detected event, produce approximate set of k most similar past events
“If a server fails, show similar alerts that occurred in the past”

5. historical information
Standard Hybrid Query
Query model based on Borealis
Input
Streams
Stream Proc.
Operators
Recall
Other Stream
Proc. Ops.
Continuous
stream processing
(event query)
Archive
Look up
historical information
(historical query)

6. Contextual Hybrid Query
Similar past
events
Input
Stream
Event
detection
Event
Similarity
Recall
Input
Stream
Window
Join
Past
contexts
Context
Input
Sream
Window
Join
Find similar past events
Use TF-IDF for similarity computation
Archive

7. Framework Three key issues Three goals
History size, near real-time, concurrent events
Three goals
Responsiveness and fairness
Incremental processing integrated with stream processing
Retrieve at least some historical data for all new events
Relevance: favor recent history over older history
Similarity: find similar past events
Exploit context similarity in other ways as well

8. Moirae’s Design Based on Borealis Based on PostgreSQL Approximate
& incrementally
improving results
Stop
Improving
Query
Contextual
& Standard
hybrid queries
Application
MOIRAE
SPE
Stream Processor
Deploy Manager
Based on Borealis
Raw
Streams
Recall Manager
RDBMS
Storage Manager
Based on PostgreSQL
Archiver
Materialized
Events
& Context
Raw
Stream
Archive
Other
Materialized
Views
Present
Chunk

9. Design Components Archiver: partitioned stream archive
Archive raw and intermediate streams
Present chunks in memory
Recent chunks on disk
Materialize & index necessary streams and contexts
Old chunks on disk
Recall Manager: partitioned, incremental queries
Execute queries one chunk at the time (present ->past)
Schedule concurrent queries to ensure fairness
Incorporate user feedback to drop events

10. Related Work
Queries over live data & data archives [chandrasekaran:04,chandrasekaran:05, franklin:05]
Log-structured access method [muth:00]
Multi-level storage manager [stonebraker:91]
Materialized views (e.g., [goldstein:01])
Partial indexes [stonebraker:89,sartori:94,seshadri:95]
Online processing [hellerstein:97,hellerstein:00, raman:02,shanmugasundaram:01,tan:99]
top-K, kNN, IR+RDBMS [e.g.,carey:97,fagin:03,chaudhuri:05,li:05]

11. Conclusion Monitoring applications accumulate history
How to leverage history ?
By supporting queries for specific historical data
Through new types of queries
How to support all these different queries ?
Can reuse several database/IR techniques
Need to integrate and extend these techniques
More information about Moirae