1. Telegraph: An Adaptive Global-Scale Query Engine
Joe Hellerstein

2. Scenarios Ubiquitous computing: more than clients!
sensors and their data feeds are key
smart dust, biomedical (MEMS sensors)
each consumer good records (mis)use
disposable computing
video from surveillance cameras, broadcasts, etc.
Global Data Federation
all the data is online – what are we waiting for?
The plumbing is coming
XML/HTTP, etc. give LCD communication
but how do you query robustly over many sites in the wide area?

3. There’s a Data Flood Coming

4. There’s a Data Flood Coming
What does it look like?
Never ends: interactivity required
Big: data reduction/aggregation is key
Unpredictable: this scale of devices and nets will not behave nicely

5. The Telegraph Query Engine
Key technologies
Interactive Control
interactivity with early answers
online aggregation for data reduction
Continuously adaptive flow optimization
massively parallel, adaptive dataflow via Rivers and Eddies

6. CONTROL Continuous Output, Navigation & Transformation with Refinement On Line
Data-intensive jobs are long-running. How to give early answers and interactivity?
online interactivity over feeds: data “juggle”
online query processing algs: ripple joins
statistical estimators, and their performance implications
Appreciate interplay of massive data processing, stats, and UIs

7. CONTROL Continuous Output and Navigation Technology with Refinement On Line

8. CONTROL Continuous Output and Navigation Technology with Refinement On Line

9. River Q We built the world’s fastest sorting machine
On the “NOW”: 100 Sun workstations + SAN
But it only beat the record under ideal conditions!
River: performance adaptivity for data flows on clusters
simplifies management and programming
perfect for sensor-based streams

10. Eddy Eddy How to order and reorder operators over time
based on performance, economic/admin feedback
Vs.River:
River optimizes each operator “horizontally”
Eddies optimize a pipeline “vertically”

11. Telegraph: Putting it Together
Scalable, adaptive dataflow infrastructure. Apps include…
sensor nets
massively parallel and wide-area query engines
net appliances: chaining xform8n/aggreg8n/etc. proxies
any unpredictable dataflow scenario
Technology: a marriage of…
CONTROL, River & Eddy
Many research questions here
E.g. how to combine River and Eddy adaptivity
E.g. how to tune Eddies for statistical performance goals
Combinations of browse/query/mine at UI
Storage management to handle new hardware realities

12. Integration with Endeavour
Give
Be data-intensive backbone to diverse clients
Be replication dataflow engine for OceanStore
Telegraph Storage Manager provides storage (xactional/otherwise) for OceanStore
Provide platform for data-intensive “tacit info mining”
Take
Leverage OceanStore to manager distributed metadata, security
Leverage protocols out of TinyOS for sensors

13. Additional Slides
For use in questions, etc.

14. Connectivity & Heterogeneity
Lots of folks working on data format translation, parsing
we will borrow, not build
currently using JDBC & Cohera Net Query
commercial tool, donated by Cohera Corp.
gateways XML/HTML (via http) to ODBC/JDBC
we may write “Teletalk” gateways from sensors
Heterogeneity
never a simple problem
Control project developed interactive, online data transformation tool: Potter’s Wheel

16. Potter’s Wheel Anomaly Detection