1. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Project IST-2000-26031 Middleware Platform for Sentient Computing Applications Computing Department, Lancaster University, UK Thirunavukkarasu Sivaharan, Maomao Wu, Gordon Blair, Adrian Friday, Paul Okanda. 2nd MiNEMA Closed Workshop@ Lancaster, 1 st Dec 2004

2. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 2 Overview of Presentation Introduction Sentient Objects Research Challenges & Component Frameworks Middleware Architecture Sentient Vehicle Demonstrator Conclusions

3. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 3 Introduction(2) EU FET Project : CORTEX –Universidade de Lisboa (Portugal) –Lancaster University (United Kingdom) –Trinity College (Ireland) –Universität Ulm (Germany) Aims –Middleware support for constructing distributed mobile proactive applications based on real-time sentient objects –Proposes sentient object model to support the construction of mobile, context aware, decentralised,autonomus,proactive and collaborative applications such as intelligent vehicles and smart buildings. –A middleware for networked embedded systems

4. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 4 Sentient Object Model(1) Sentient Object Model –System consists of environment and a set of sentient objects –Sentient objects are capable of independently sensing the environment, derive context and infer autonomous actions –Sentinet objects communicate using event channels to establish higher level context and thus cooperate with each other

5. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 5 Sentient Object(2)

6. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 6 Autonomous sentient vehicle application in MANET Autonomous navigation of vehicles from a source to destinations Cooperating vehicles in MANET Context aware vehicles

7. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 7 Some of the research challenges addressed Suitable Communication Model for MANET Routing in mobile ad-hoc environment Context-awareness End-to-End QoS and Fail safety Run time and deployment time reconfigurations

8. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 8 Component Framework based Reflective Middleware Publish-Subscribe Component framework (CF) Multicast CF Context CF Resource Management CF

9. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 9 Why Component Framework based Middleware Platform? Middleware is engineered as family of Component frameworks (CF) using Reflection and component technology Each CF addresses specific research areas Component Frameworks are highly configurable and dynamically reconfigurable (with the granularity of a component) Clear separation of concerns Adaptable to diversity of CORTEX applications Reduction of memory footprint CFs are implemented using Lancaster’s OpenCOM reflective component technology

10. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 10 Middleware Architecture WLAN 802.11b (ad-hoc), Windows CE Payload ChannelTCB control channel Group Communication CF-( Ad-hoc Multicast ) Publish-Subscribe CF- (for MANET) Programming Interfaces MIDDLEWAREMIDDLEWARE Sentient Objects Sentient Objects Context CF- Sensor Fusion Inference Engine Timely Computing Base Middleware Configuration for MANET

11. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 11 Publish-Subscribe CF(1) Communication model inspired by STEAM Implicit event model Sender & receiver based event filtering Subscription Language supports subject, content & context based event filtering Supports distance based context filtering & extensible to other contexts XML based generic events Events transported via selectable Multicast protocol

12. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 12 Publish-Subscribe CF(2) Subscrib er SOAP Messagi ng Filter SOAPtoMult icast Multicast IFilter IMulticast ISOAPMessaging ISubscribe ISOAPTransport Publisher Notifier IApplicationNotify Filter IFilter Receptacle Interface IPublish Dispatc her IDispatch

13. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 13 Multicast CF Underlying event Routing Protocol is based on multicast The multicast protocol for ad-hoc networks is a probabilistic, stateless and multi-hop protocol We offer this service in the form of a component framework. Shared memory based IP Multicast Probabilistic Multicast

14. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 14 Context CF (1) Sensor capture and fusion –Multivariate Gaussian modelling –Bayesian networks –Dead-reckoning Inference engine –A program that reasons about a set of rules (a knowledge base) in order to derive an output. –The knowledge is encoded as a set of production rules, contexts are represented as “fact”. –CLIPS – C Language Integrated Production System, its internal implementation is based on RETE net.

15. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 15 Context CF (2) CLIPS rule sample The paradigm facilitates uniform treatment of both context and QoS –Rules to trigger adaptations and actuations based on changes in measure of QoS data CLIPS DLL and OpenCOM component for WinXP and WinCE (defrule rule-obstacle-near "CLIPS rule for obstacle near" (car-id (id ?id)) ?f1 <- (obstacle (distance near)) => (retract ?f1) (publish ?id stop) )

16. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 16 End-to-End QoS Management and Fail Safety- Timeliness requirement How can this be achieved? –Enforcing timely perceptions of the environment and timely actuations on it. – Which means timely event delivery and awareness of QoS of the event channels used for inter-sentient object communication The key issue in uncertain and highly dynamic environments is that timing bounds for distributed actions may be violated because of timing failure

17. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 17 End-to-End QoS Management and Fail Safety-Timeliness Requirements We model the uncertainty of timely event dissemination via event channels using a dependable timing failure detection service. This service is provided by University of Lisboa’s Timely Computing Base (TCB) TCB facilitates to construct distributed event channels with timing bound specification This enables publisher or subscriber to be aware of the timing failures of event channels Thus providing awareness of timing failure probability for a given required coverage Fail safety is achieved by switching to fail-safe state as soon as QoS specifications are violated.

18. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 18 Autonomous Sentient Vehicles Demonstrator Two Sub problems –Cooperative behaviour without human control –Autonomous vehicle navigation from a given source to pre-determined destination Vehicles Objectives –Travel along a given path( virtual circuit-VC) defined by set of GPS waypoints and bearings. –Every vehicle that travels on the VC cooperate with other vehicles to avoid collisions and travel safely –Obey external roadside traffic lights. –Give way to pedestrians who cross the road.

19. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 19 OC CLOSE( 4m)OC BEHINDOC FAR(4- 10m) Location aware Cooperating Sentient Vehicles Car publishes on Carcontrol channel: Event Packet: Car subscribes to: CarControlCh annel & Receives events from other cars Car subscribes to: CarControlCh annel & Receives events from other cars Car publishes on Carcontrol channel: Event Packet: IEEE 802.11b(ad-hoc) ---Event Channel--- CarControlChannel Satellites Car ACar B OC – Other car OC VERY FAR Other car’s location context w.r.t car A OC BEHIND Other Car’s location context w.r.t Car B 4m OC CLOSE

20. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 20 Pedestrian detection Obstacle Sensing Service: Consumes raw ultrasonic sensor data and fuses using a suitable algorithm (reliable, timely-unreliable, Gaussian, …) to derive higher level obstacle distance context such as NEAR, FAR, NOOBJECT. Ultra sound waves Ultrasonic sensors Non event publishing obstacle

21. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 21 Component Example: The Car Sentient Object & Context CF Component LocationSensingService GPSFusion 1 GPSFusion 2 ObstacleSensingService UltrasonicFusion 1 UltrasonicFusion 2 DirectionSensingService CompassFusion 1 Inference Service CLIPS Inference Engine Ultrasonic sensor GPS sensor Sentient object Digital Compass sensor Consume Produce Speed Actuator Steer Actuator Sentient object receptacle Interface

22. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 22 Sentient Vehicle Test Bed

23. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 23 Cont’d

24. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 24 Cont’d

25. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 25 Demo Settings

26. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 26 Waypoint 1 Waypoint 2 Waypoint 3 Waypoint 4 Traffic Light Virtual Circuit

27. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 27 Demo Video

28. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 28 Concluding Remarks The sentient object model –has proved to be valuable programming abstraction for the development of real-time, cooperative, context-aware applications. The component-Framework based Middleware approach –offers benefits of flexible configuration and reconfiguration of the middleware components The middleware architecture –also provides the management of non-functional concerns such as timeliness and reliability properties. Our middleware is reusable –we are keen to investigate the generality of our approach by applying our middleware to other application domains involving embedded autonomous components.

29. CO-operating Real-time senTient objects: architecture and EXperimental evaluation Lancaster University 29 Thank You Questions