IBM TJ Watson Research Center © 2010 IBM Corporation – All Rights Reserved AFRL 2010 Anand Ranganathan Role of Stream Processing in Ad-Hoc Networks Where and How should Streaming Sensor Data be Processed?

IBM TJ Watson Research Center © 2010 IBM Corporation – All Rights Reserved CIKM 2009, Hong Kong Sensors produce huge amounts of real-time data How and where should this data be processed? On the sensor itself? Within a sensor network? At the sensor / sensor network gateway? In a centralized data processing center? Some factors that influence the decision: Volume and rate of data produced by the sensors Data processing latency requirements Complexity of processing Security and privacy considerations Administrative and Organizational Boundaries Is there a need to access large amounts of (static or dynamic) data

IBM TJ Watson Research Center © 2010 IBM Corporation – All Rights Reserved One example in the domain of Intelligent Transportation Systems (ITS) ITS encompasses sensor, communications and computing technologies to manage existing infrastructure and transportation systems more efficiently. Incorporates both Floating Car Data (FCD) and Fixed Sensor Data FCD represents the location of vehicles collected by mobile sources, such as GPS devices installed in vehicles or cellular phones. Fixed sensor data includes data from video cameras, loop detectors, toll booths, etc. CIKM 2009, Hong Kong

IBM TJ Watson Research Center © 2010 IBM Corporation – All Rights Reserved Lets look at one application of FCD data to derive traffic information and then use that to provide various user- added services CIKM 2009, Hong Kong

IBM TJ Watson Research Center © 2010 IBM Corporation – All Rights Reserved Interactive visualization Data Warehouse Offline statistical analysis Storage adapters GPS Data Streams Real Time Transformation Logic Real Time Geo Mapping Real Time Speed & Heading Estimation Real-time GPS data processing Cleaning Map-Matching Per-Vehicle Speed Estimation Real Time Aggregates & Statistics Aggregated Statistics Per-link / per-region Web Server Google Earth Stochastic Link Travel Time Calculation Stochastic Path Travel Time Calculation Splitter Shortest Path 1 Shortest Path n User-driven computations Travel times, Shortest Paths

IBM TJ Watson Research Center © 2010 IBM Corporation – All Rights Reserved GPS Data Streams Real Time Transformation Logic (Cleaning, Denoising) Real Time Geo Mapping Real Time Speed & Heading Estimation Real Time Aggregates & Statistics Maybe Performed On Device Maybe Performed On Device Maybe Performed On Device (if device had access to map) Maybe Performed On Device (if device had access to map) Best performed on centralized data processing facility Best performed on centralized data processing facility Best performed on centralized data processing facility Best performed on centralized data processing facility

IBM TJ Watson Research Center © 2010 IBM Corporation – All Rights Reserved In case GPS data is obtained from cellphones Some processing can also occur at the gateway or mobile switching center Performed by cellphone service provider or an authorized party. The processing performed here may include aggregation and anonymization of the raw GPS data To protect privacy CIKM 2009, Hong Kong

IBM TJ Watson Research Center © 2010 IBM Corporation – All Rights Reserved In previous work… We had performed all 4 steps at a centralized location Using the IBM InfoSphere Streams stream processing system Able to process over 100,000 GPS points per second on a 5 machine cluster Estimates current traffic conditions for the city of Stockholm (using real-time data from taxis and other vehicles) Traffic Information then used for shortest path queries, traffic monitoring, etc. CIKM 2009, Hong Kong

IBM TJ Watson Research Center © 2010 IBM Corporation – All Rights Reserved IBM InfoSphere Streams Supports high performance stream processing. Offers both language and runtime support for improving the performance of streaming applications via a combination of optimized code generation, pipelining and parallelization. Supports a component-based programming model Reduces the computational requirements on the sensors. Since all the data processing happens in one place, it is easier to adapt or extend the processing to meet changing requirements. CIKM 2009, Hong Kong

IBM TJ Watson Research Center © 2010 IBM Corporation – All Rights Reserved In summary… There are different places where sensor data can be processed to produce useful information. So far, however, the partitioning of the processing has tended to be dictated by organizational or administrative boundaries. More research needs to be done to determine what is the optimal way of splitting the processing to meet performance, privacy, functionality and other requirements. CIKM 2009, Hong Kong