1. Semantic Streams: a Framework for Composable Semantic Interpretation of Sensor Data Kamin Whitehouse UC Berkeley EWSN, Feb 13, 2006 Joint with Feng Zhao and Jie Liu, Microsoft Research

2. fact Data Processing Data Processing fact Data Processing Motivation Imaging that sensors cover the globe Can we automatically query for world facts? No: each fact is an interpretation of data Processing must consider –Sensor context –Sensor fusion –Data formats –Calibration –Signal transforms –Etc. World Sensor DB fact Data Processing fact Data Processing

3. Semantic Streams Goal: to allow direct querying of facts Example query: “Is a vehicle in the parking lot” Instantiates a graph of inference units to derive the desired fact Output is a stream of all vehicles detected Fact: vehicle Size Inference Vehicle Inference Space Inference Fact: empty space Sensor.data=72Sensor.data=56Sensor.data=98obj.size=23 Sensor.data=49 obj.type=vehicle

4. Usage Model Fixed Sensor infrastructure –Many simultaneous, unrelated users –Short-term queries repeatedly over long periods –Queries are often similar (though not identical) When programmer poses new query, system will use existing sensors and inference units, if possible. Otherwise, system may give actionable error messages, –need to add 2 more sensors in area XYZ –need to add new inference unit to the system Semantic values produced by infrastructure grows organically as it is used for new purposes

5. Complete System Mobile Application Query Processor Inference Graph Embedding Engine MSTML Execution Engine Sensors Planning time User Output Stream Inference Unit Instances Run time

6. Background: Prolog and FOL george, jane, carA X, Y, Z, StudentA female(jane), parent(X,Y) parent(george,jane). male(george). father(X) :- parent(X,Y),male(X). male(X). > X = george father(X). > X = george father(jane). > false Constants Variables Predicates Facts Rules Queries Query processing

7. Markup Language sensor(magnetometer, [[60,0,0][70,10,10]]). sensor(camera, [[40,0,0][55,15,15]]). sensor(breakBeam, [[10,0,0][12,10,2]]). inference( speedDetector, needs( sensor(magnetometer, R) ), creates( stream(X), isa(X,object), property(X,T,time), property(X,R,region), property(X,S,speed) ) ). Sensors are declared as logical facts –All regions are appx’d by 3D cubes Inference units are logical rules –Antecedents: the fact streams that it needs –Consequents: the fact streams that it creates –Spatial relationship encoded in variable name

8. Stream Relationships subregion( A, B ) inference(speedDetector, needs( sensor(magnetometer, R) ), creates( stream(X), isa(X,object), property(X,T,time), property(X,R2,region), subregion(R2,R), property(X,S,speed) ) ). More sophisticated spatial relationships –Subregions Sensor coverage (R) Queried area (R2)

9. Stream Relationships inference( speedDetector, needs( sensor(breakBeam, R1), sensor(breakBeam, R2), sensor(breakBeam, R3), subregion(R1,R), subregion(R2,R), subregion(R3,R), \+ intersecttion(_,R1,R2), \+ intersecttion(_,R1,R3), \+ intersecttion(_,R2,R3) ), creates( stream(X), isa(X,object), property(X,T,time), property(X,R,region) ) ). More sophisticated spatial relationships –Subregions –Non-intersection Queried area Sensor coverage

10. Stream Relationships inference( vehicleDetector, needs( sensor(magnetometer, R), stream(X), isa(X,object), property(X,S,speed) ), creates( stream(X), isa(X,vehicle) ) ). More sophisticated spatial relationships –Subregions –Non-intersection Stream identity

11. Query processing 1.Choose first predicate in query 2.Search KB for sensors that match 3.Search output of inference units for streams that match 4.Add new input streams to query 5.Back to step 1 Region(A,[[10…]])isa(A,vehicle) Stream(X), Isa(X,vehicle) stream(X), property(X,speed) Sensor(mag,…) Vehicle Inference stream(Y), property(Y,speed) Sensor1Sensor2Sensor3 Speed Inference

12. Query processing Actionable Error Messages –When missing inference unit –When missing sensor Region(A,[[10…]])isa(A,vehicle) Stream(X), Isa(vehicle,X) stream(X), property(X,speed) Sensor(mag,…) stream(Y), property(Y,speed) Sensor1Sensor2Sensor3 Vehicle Inference Speed Inference

13. Query processing When too many choices –Multiple graphs provide logically equivalent results Region(A,[[10…]])isa(A,vehicle) Stream(X), Isa(vehicle,X) stream(X), property(X,speed) Sensor(mag,…) stream(Y), property(Y,speed) Sensor1Sensor2Sensor3 Vehicle Inference Speed Inference stream(Y), property(Y,speed) Camera Speed Inference

14. Quality of Service Constraints subregion( A, B ) inference(speedDetector, needs( sensor(camera, R) ), creates( stream(X), isa(X,object), property(X,T,time), property(X,R2,region), property(X,S,speed), property(X,C,confidence), {C > 99}, property(X,L,latency), {L < 450} ) ). Inference unit may specify QoS of output –Confidence –Latency

15. Quality of Service Constraints subregion( A, B ) inference(speedDetector, needs( sensor(breakBeam, R1) ), sensor(breakBeam, R2) ), sensor(breakBeam, R3) ), creates( stream(X), isa(X,object), property(X,T,time), property(X,R2,region), property(X,S,speed) ) ). property(X,C,confidence), {C > 80}, property(X,L,latency), {L < 75} ) ). Inference unit may specify QoS of output –Confidence –Latency

16. Quality of Service Constraints subregion( A, B ) inference(vehicleDetector, needs( sensor(magnetometer, R1), stream(X), property(X,S,speed), property(X,C2,confidence), property(X,L2,latency), creates( stream(X), isa(X,vehicle), {C > 0.9 * C2}, {L = L2 + 125} ) ). Inference unit may specify QoS of output –Confidence –Latency May be function of input QoS

17. Query processing Query can declare QoS parameters to create ordering of inference graphs Region(A,[[10…]])isa(A,vehicle) Stream(X), Isa(vehicle,X) stream(X), property(X,speed) Sensor(mag,…) stream(Y), property(Y,speed) Sensor1Sensor2Sensor3 Vehicle Inference Size Inference stream(Y), property(Y,speed) Camera Size Inference {C>70, L <300}

18. Example Infrastructure

19. Goal Application: Vehicle Detection

20. Query and Inference Graph isa(Y,histogram),property(Y,T,value),property(X,T,time), isa(X,vehicle),region(X,R,[[10,10,0][40,30,12]]), Isa(Z,photo),property(Z,X,triggerStream) Vehicle Inference Speed Inference Direction Inference Size Inference Histogram Service Photo Service Break Beams Magnetometers Camera

21. Screenshot

22. Final Application Camera image Report (histogram) Break beam plot Micro ServerCamera Server Magnetometer plot

23. Conclusion Semantic Streams –Infers facts from data –Intended for sensor infrastructure –Can select inference graph based on QoS constraints Limitations –Must be a large supply of inference units –Stream operators are difficult to program Different programming language/execution model Programmer must deal with temporal values, implement appropriate buffering, etc.

24. Future Work: Streaming KB’s Relational Database : Knowledge Base :: Streaming Database : Streaming KB SQL operators: Select, Join, Max Operate on Tables Logical rules: c(X) :- a(x),b(x). Operate on Facts Infinite tablesInfinite facts

25. Video: Vehicle Detection

26. Video: Human Filtering

27. Traffic Histogram