Distributed Query Processing over Streaming and Stored Data Alasdair J G Gray Information Management Group University of Manchester Dagstuhl Seminar – Semantic Challenges in Sensor Networks 25 – 29 January 2010

Acknowledgements RAs/PhDs Christian Y. A. Brenninkmeijer Ixent Galpin Alasdair J. G. Gray Farhana Jabeen Academics Alvaro A. A. Fernandes Norman W. Paton MSc Students Jamil Naja Varadarajan Rajagopalan 26 January 20102Semantic Challenges in Sensor Networks

Overview of the Talk Motivation Data source characteristics Query language: SNEEql Query processor: SNEE-DQP 26 January 20103Semantic Challenges in Sensor Networks

Motivating Scenario Discover relevant data sources (see Manolis Koubarakis’s talk) Unify data models (see Oscar Corcho’s talk) Extract, combine, and process relevant data. This talk and Alvaro’s 26 January 20104Semantic Challenges in Sensor Networks Stored data Sensor Network

Motivating Scenario Stored data Sensor Network 26 January 20105Semantic Challenges in Sensor Networks Sensor Network Stored data service Streaming data service

Data Source Characteristics Traditional stored data –Data stored in a database –User observes a static data set –One-off query execution Streaming data –Data processed on-the-fly (may also be stored for later access) –User observes changes in data set –Continuous or snap-shot query execution 26 January 20106Semantic Challenges in Sensor Networks

Types of Data Stream Pull StreamPush Stream Stream Processor Source GetData()Data Stream Processor Source Data 26 January 20107Semantic Challenges in Sensor Networks

Query Processing Challenges Variety of data sources –Stored –Push-stream –Pull-stream No common query semantics –Streaming data languages –Stored data languages Distributed data sources 26 January 2010Semantic Challenges in Sensor Networks8

SNEE-DQP Stored data Sensor Network SNEE-DQP 26 January 20109Semantic Challenges in Sensor Networks Stored data service Streaming data service Sensor Network Streaming data service

Query Language: SNEEql Aimed at in-WSN query processing –Pull streams –Reactive/periodic operators –Controls network behaviour Also capable of querying –Push streams –Stored sources Well defined semantics –Independent of system 26 January Semantic Challenges in Sensor Networks

SNEEql Query Syntax SELECT {RSTREAM | DSTREAM | ISTREAM} + attribute list FROM extent list WHERE expression *STREAM optional –Converts a window to a stream Extent list: –Streams with windows of the form [FROM t1 TO t2 SLIDE int unit] –Relations with windows of the form [SCAN EVERY t1 unit] 26 January Semantic Challenges in Sensor Networks

Example Query Every 15 minutes, and within 24 hours of their being taken, we wish to obtain time-correlated measurements of the river depth now and the rainfall at the top of the hill 15 minutes before, provided that it is now raining less in the river than it was in the hill top, that the rainfall in the hill top was above 5mm and greater than average rainfall. SELECT RSTREAM r.time, h.rain, r.depth FROM River[NOW] r, Hilltop[AT NOW-15 MINUTES] h, WHERE h.rain > 5 AND r.rain < h.rain AND h.rain >= (SELECT AVG(weather.rain) FROM Weather [rescan every day] WHERE weather.region = 'Peak District'); 26 January 2010Semantic Challenges in Sensor Networks12

SNEE DQP Query Stack Metadata –Logical schema –Physical schema Source Allocation –Splitting the query into parts for each data source Source Planning –Physical operator selection –Generate plan for source 26 January 2010Semantic Challenges in Sensor Networks13 Metadata SNEEql query + QoS Query Execution Plan Parsing Logical Planning Source Allocation Source Planning More details on in-WSN planning in Alvaro’s talk

Stream Data Query Processing Sensor Network Data Service Stream In-Network SNEE WSDL Stream Access Service WSDL Stream Access Service 26 January Semantic Challenges in Sensor Networks Data Service Stream Event Stream SNEE WSDL Stream Access Service WSDL Stream Access Service Sensor Network Acquisitional Stream Processing Event Stream Processing

Worked Example SELECT RSTREAM r.time, h.rain, r.depth FROM River[NOW] r, Hilltop[AT NOW-15 MINUTES] h, WHERE h.rain > 5 AND r.rain < h.rain AND h.rain >= (SELECT AVG(weather.rain) FROM Weather [rescan every day] WHERE weather.region = 'Peak District'); 26 January 2010Semantic Challenges in Sensor Networks15 EXCHANGE JOIN river.rain<hilltop.rain ACQUIRE [time,rain] rain > 5 hilltop EVERY 15 min ACQUIRE [time,rain] rain > 5 hilltop EVERY 15 min ACQUIRE [time,rain, depth] true river EVERY 15 min ACQUIRE [time,rain, depth] true river EVERY 15 min TIME_WINDOW [t-15, t-15, 15] DELIVER EXCHANGE AVERAGE (rain) AVERAGE (rain) SCAN [rain] region = ‘Peak District’ weather EVERY HOUR SCAN [rain] region = ‘Peak District’ weather EVERY HOUR JOIN h.rain >= AVG(weather.rain) JOIN h.rain >= AVG(weather.rain)

Conclusions Query-based access to distributed data sources, both streaming and stored SNEEql provides well defined, unified semantics for streaming and stored data SNEE-DQP provides execution environment 26 January 2010Semantic Challenges in Sensor Networks16

Motivating Scenario Stored data Sensor Network 26 January Semantic Challenges in Sensor Networks Stored data service Streaming data service Sensor Network Streaming data service