1. University of Milan ORESTEIA -MODULAR HYBRID ARTEFACTS WITH ADAPTIVE FUNCTIONALITY http://www.image.ntua.gr/oresteia/

2. PARTNERSProject Technical Committee Technical Manager: John Taylor (KCL) ALTEC SA Sotiris Pavlopoulos, George Malinescos IVML-NTUA Stefanos Kollias, Nicolas Tsapatsoulis University of Milan Bruno Apolloni, Dario Malchiodi Kings College of London Stathis Kasderidis Imperial College Eric Yeatman, Tim Green

3. Contents

4.  Executive Summary Executive Summary Executive Summary  Problems to be Solved Problems to be Solved Problems to be Solved –Attention Attention –Data Fusion Data FusionData Fusion –Emergence Emergence  Architecture and partners' tasks Architecturepartners' tasks Architecturepartners' tasks –Diagram Diagram –Module Explanation Module ExplanationModule Explanation  Feature Extraction through Signal Modelling  Methods for State mappings Methods for State mappings Methods for State mappings –CAM/SPM CAM/SPM –PAC Meditation / Fuzzy Relaxation PAC Meditation / Fuzzy RelaxationPAC Meditation / Fuzzy Relaxation  Demo 1 Demo 1 Demo 1  Demo 2 Demo 2 Demo 2  Data Collection Data Collection Data Collection –Laboratory Studies –Car Driving Simulator –Linguistic Rules  MicroPower Generation – Wireless Communication MicroPower Generation – Wireless Communication MicroPower Generation – Wireless Communication

5. Executive Summary

6. Scope/Aims  Create a guidance system for humans, for more efficient and less hazardous living and interacting with their environment, through a set of decision-making facilities embedded in the environment and suitably adapted to the particular user  Investigate enabling technologies for DC in the form of energy harvesting and low power wireless communications

7. Inputs  Low level sensorial data –Physiological class of sensors –Environmental class of sensors –Other  Symbolic knowledge, a priori available (linguistic rules)  Subsymbolic knowledge, constructed based on numerical data (Input/Output pairs)  Attention-based functionality, inspired from brain operation

8. Outputs  Decisions  Actions on behalf of the user for: –Managing repetitive and trivial jobs, –Providing indication of abnormal user activity and state, –Providing planning facilities, –Providing information filtering facilities,  Maintaining good user state (physiological, psychological, etc)

9. Key Properties  Autonomy  Responsiveness  Robustness

10. Approach  Develop a multi-level attention-based agent architecture adapted to solve decision /guidance problems arising from sensors of various types, some worn by humans, others in devices (such as cars) being used by the humans. The decision/ guidance response of an agent is as to what is the state of the human, given the sensor data, or what is optimal continued use of the device on the basis of joint sensor data arriving at the agent for a given user from all sources  Develop multi-agent systems that handle data available, also, from a set of agents (from interacting users), providing for decision/guidance as to overall optimal (best) use, and ranking of the users as to which needs further analysis

11. Problems to be solved

12.  Attention  Data Fusion  Emergence

13. Problems to be solved: Attention

14. OUTPUTS Artefact INPUTS Attention Controller The input signals have irregular patterns… Data History My batter y is low! Self Evaluator Attention Signal I shouldn’t produce these outputs, with these inputs… Irregular inputs Hardware failure Self-evaluation error

15. Problems to be solved: Data Fusion

16. “Clever Space” Artefact data Which of these data shall I need? Intelligent Artefact You can make use of the two bottom Artefacts “WORLD MODEL” AGENT

17. Problems to be solved: Emergence

18. AGENT Artefact “Clever Space” Artefact I need to use these! And I need to use these! I’ll take these! HEY,I SAW THEM FIRST! WHY NOT BE MINE? THESE ARE MINE! How much are you willing to pay for the services?

19. Architecture of a single Agent

20. Overall View

21. Level 1: Sensors  Data Content: Classes of signals used by higher levels (Level 2-4) –Data collection (KCL-QUB, ALTEC) –Synthetic data generators (NTUA, UM, KCL)  Sensor autonomy –Efficient energy harvesting (ICSTM)  Communication links –Low power consumption (ICSTM)

22. Level 2: Preprocessing  Signal preprocessing (NTUA, KCL) –Noise Reduction –Buffering –Transforms  Feature extraction –Which features? (ALTEC, KCL-QUB) –How? (NTUA, KCL)  Modeling signals –Extraction of hidden parameters (UM)

23. Level 3: Domain Experts – State Representation Architecture of a single Agent

24. State Mappings  ANN Hourglass –Subsymbolic state representation (UM, NTUA)  Neurofuzzy –Symbolic state representation (NTUA, UM)

25. Action Module  Stores the ‘response’ of the system. Three levels of sophistication: –No real action. –Simple suggestive actions/messages. –Simple action sequences.  Responsible Partners: KCL, NTUA, UM

26. Rules Module  Consists of three components: –World Model. Contains all the information needed for forming useful functionalities and maintaining a set of artefacts. –Autonomic. Maintains rules that are necessary for the robust run-time behaviour of the system. –Other. Aids the implementation of alternative (to the ones implemented in the State module) decision-making systems.  Responsible Partners: KCL, NTUA, UM

27. Goals Module  This module includes three parts: –Values. Is closely associated with the World Model (in the Rules module) to provide default (universal) values for the various thresholds and triggers present in the architecture. –User Profile. Provides specific user deltas (i.e. deviations from the default values defined in the Values part). –Services. Includes a catalogue of services that are offered by the artefact.  Responsible Partners: KCL, NTUA, UM

28. Monitor Module  Creates an error signal level after comparing the current State with a Historic State. It fulfils two basic requirements: –Universal definition of an error function. Independent of the output of State Module (UM, NTUA, KCL) –Standard definition of an error function. Context sensitive, seamless knowledge of state representation (UM, NTUA, KCL)

29. Attention Controller  This module is inspired from motor control systems in the brain as well as from engineering control ideas. It operates in two modes: –Feedforward mode. The controller sends a signal, governed directly by the Goal module, to produce a desired response from the action module (KCL) –Feedback mode. Feedback information from the Monitor module is used as a feedback component (KCL)

30. Level 4: Agent Construction  Combination of conceptual blocks  Agent Formation  Data Fusion  Overall system training  Reinforcement signal production/handling  Attention Control Responsible partner: KCL

31. Feature Extraction through Signal Modelling

32.  E(w) =  e(y1,..., yT) Integrated by a fourth order Runge-Kutta method DYNAMICS HYBRID TRAINING Symbolic health diagnosys SIGNALS FROM BODY ECG

33. Methods for State Mappings: CAM/SPM Module

34. Overall View

35. CAM Module

36. Scope  The Connectionist Association Module (CAM) provides the system with the ability of grounding the symbolic predicates  Using the CAM, the set of features is associated with the set of evaluated symbolic predicates (partitioning the input space)

37. Why Neural Network?  Generally the internal state defined by the neural network output is not so simple to be considered as a simple fuzzy partitioning;  Instead the neural network performs the appropriate data clustering to provide the evaluation of the required symbolic predicates based on numerical data

38. Diastolic Pressure Example

39. Attention Signal Handling  To which input elements have to be concentrated on?

40. SPM Module

41. Scope  It implements a semantically rich reasoning process. It takes as inputs a set of features and gives a set of recognised situations.  It performs the conceptual reasoning process that finally results to the degree of which the output situations are recognised

42. Why Neurofuzzy?  Fuzzy relational systems represent symbolic knowledge in a formal, numerical framework.  On the other hand, neural networks are typical learning systems that work in a numeric framework.

43. Rule Insertion  Rules describing situations are based on linguistic terms and are generally of the form If fuzzy_predicate(1) and fuzzy_predicate(2) then output(3) ”  Each rule consists of an antecedent (the if part of the rule) and a consequence (the then part of the rule)

44. Rule Insertion  The antecedent part of the rule is used to create the weight matrix of the first layer  The consequence part of the rule is used to create the rule matrix of the second layer  The antecedent of all the rules existed is the set of the fuzzy predicates describing the system  The consequence of all the rules is the set of the recognised situations of the system

45. Rule Insertion (Example) Layer 1 Layer 2

46. Methods for State Mappings: PAC Meditation / Fuzzy Relaxation

47. PAC Meditation mapping formula fitness ok? n y fitness formula fuzzy relaxation end formula final formula n y prejudice

48. PAC Meditation 0-level inner border 0-level outer border 1-level inner border 1-level outer border

49. Fuzzy relaxation  SA Algorithm CurrentState := InitialState CurrentTemperature := InitialTemperature Repeat GetTemperature(CoolingSchedule) ProposedState := SelectNeighborState ProposedCost := EvaluateCost(ProposedState) If (Accepted(ProposedState, ProposedCost)) Then CurrentState := ProposedState Until StoppingRule Return(CurrentState)

50. DEMO 1: Health Monitoring

52. DEMO 2: Car Hazard Avoidance

53. Description  This demo includes the ability to generate a set of events in the environment, driver, or car. Environmental effects include shocks in the visibility (fog) and the temperature. In this category also the appearance of ice patches in road segments and existence of other cars in the same or opposite lane is included as well

54. Aims  To validate the ORESTEIA Architecture  To test the integration of the work of the partners  To offer another context for testing the adequacy of the State mapping methods  To search for the existence of common design principles in the various contexts

56. Data Collection

57.  Laboratory Studies  Car Driving Simulator  Linguistic Rules

58. Data Collection: Laboratory Studies

59. Aim  Experiment design for collecting ECG, Respiration Rate (RSP), Galvanic Skin Response (GSR), and Skin Temperature (SKT) in laboratory conditions for abnormal physiological and psychological state prediction

60. ECG Signal

61. RSP Signal

62. GSR Signal

63. SKT Signal

64. Data Collection: Car Driving Simulator

65. Aim  Experiment design for collecting ECG, Respiration Rate (RSP), Galvanic Skin Response (GSR), and Skin Temperature (SKT) while driving a car simulator (for abnormal physiological and psychological state prediction)

66. Car Driving Simulator

67. Data Collection: Linguistic Rules

68. Physiological Features  The system takes as inputs a set of medical features and gives a set of recognised situations.  The input features are the values of RSP, BT, HR, PS, PD, and the derived features from the ECG

69. Inputs  RSP: Respiration  BT: Body Temperature  HR: Heart Rate  PS: Systolic Blood Pressure  PD: Diastolic Blood Pressure  ECG: Electrocardiogram

70. Outputs  Normal: some features are not perfect  Warning: stop and rest until you get normal  Urgent: stop and call medical centre  Emergency: very urgent

71. Definition of Symbolic Predicates  The term predicate refers to the partition of the input features  Predicates are characterised as Very Low (VL), Low (L), Medium Low (ML), Medium High (MH) High (H), and Very High (VH)  Some features are characterised as Normal (N) and Abnormal (A)

72. Rule Extraction  Rules describing situations are based on linguistic terms and are generally of the form “if predicate(1) and predicate(2) then output(3)”  In order to detect the recognised situations, we must first define the rules that describe these situations

73. A Subset of the Rules

74. MicroPower Generation and Wireless Communication

75. Aims  Development of artefacts that –Are autonomous –Have a long lifetime without maintenance  Development of sources that can scavenge energy from the local environment  Electromechanical energy conversion using MEMS

76. Analysis of micro-power generator topologies  Detailed investigation of the operation of various different inertial generator topologies  The parametric generator is optimal when the input movement is greater than the dimensions of the device by a factor of ~10 or more

77. Fabrication and test of an initial prototype generator  Cross-section of prototype generator  Photograph of fabricated device

78. Fabrication and test of an initial prototype generator  Experimental setup for charge transfer experiments  Typical discharge transient

79. Analysis of wireless communication schemes  Comparison of near- field transmission (at 50 MHz) with far-field transmission (at 470 MHz) for a distance of 50cm, varying the near and far-field antenna dimensions (shortened to NF and FF respectively on the axis labels)