1. 10th Kovacs Colloquium UNESCO Water Resource Planning and Management using Motivated Machine Learning Janusz Starzyk School of Electrical Engineering and Computer Science, Ohio University, USA www.ent.ohiou.edu/~starzyk 10th Kovacs Colloquium UNESCO, Hydrocomplexity: New Tools for Solving Wicked Water

2. 10th Kovacs Colloquium UNESCO  Challenges in Water Management  Embodied Intelligence (EI)  Embodiment of Mind  EI Interaction with Environment  How to Motivate a Machine  Motivated Learning  ML Experiment  Abstract Motivations and Goal Hierarchy  Promises of EI Outline Flood in Poland

3. 10th Kovacs Colloquium UNESCO Water management is challenging since:  Strategies are developed mostly on national level  There is a competition between countries for water  Water policy plans effects environment and society, health and development, and economy  Growing demands for water  Need to integrate water management and policy making  There is an acute need for legitimate scientific data  Decision making in water-related health, food and energy systems are critical to economical development and national security Challenges in Water Management South–North Water Transfer Project China

4. 10th Kovacs Colloquium UNESCO Decision makers must consider important questions:  How do we make water use sustainable?  How to protect water resources from overuse and contamination?  Water problems are interconnected and too complex to be handled by a single institution or a single group of people  It is a challenge to evolve strategies and institutional frameworks for quick policy changes towards an acceptable water use  It is necessary to create assessment and modeling tools to improve policy making resolve conflicting issues and facilitate interaction. Challenges in Water Management

5. 10th Kovacs Colloquium UNESCO Why accurate integrated models to support decision making are important ?  Computerized models were used for many years to support water related decisions.  Models often simplify dynamics of economic, social and environmental interactions and lead to inappropriate policy making and management decisions.  This work proposes models that emerge from interaction with real dynamically changing environments with all of their complexities and societal dependencies.  The main objective is to suggest an integrated modeling framework that may assist with the tasks of water related decision making. Challenges in Water Management

6. 10th Kovacs Colloquium UNESCO What are deficiencies of machine created models?  Artificial neural networks, fuzzy logic, and genetic algorithms have been used to model resource planning and water management  It is difficult to apply these tools in real-life decisions as the related parameters are not explicitly known  This work presents a machine learning approach that motivates machine to develop into a useful toll.  Motivated machine learning can characterize data and make predictions about their dynamic changes  It could support intelligent decision making in dynamically changing environment  It could observe impacts of alternative water management policies  It may consider social, cultural, political, economic and institutional elements of decision making Challenges in Water Management

7. 10th Kovacs Colloquium UNESCO Embodied intelligence may support decision making:  EI mimics biological intelligent systems, extracting general principles of intelligent behaviour  It uses emerging, self-organizing, goal creation (GC) system that motivates EI to learn how to interact with the environment  Knowledge is not entered into such systems, but is a result of useful actions in a given environment.  This knowledge is validated through active interaction with the environment.  Motivated intelligent systems adapt to unpredictable and dynamic situations in the environment by learning, which gives them a high degree of autonomy  Learning in such systems is incremental, with continuous prediction of the input associations based on the emerging models - only new information is registered in the memory Challenges in Water Management

8. 10th Kovacs Colloquium UNESCO How to use the motivated learning scheme to integrate modelling and decision making?  It is suggested to apply ML approach to water management in changing environments where the existing methods fail or work with difficulty.  For instance, using classical machine learning to represent physical processes works only under the assumption that the same processes will repeat.  However, if a process changes beyond certain limits, the prediction will not be correct.  ML systems may overcome this difficulty and such systems can be trained to advice humans.  Design concepts, computational mechanisms, and architectural organization of embodied intelligence are presented in this talk  The talk will explain internal motivation mechanism that leads to effective goal oriented learning, abstract goal creation and goal management Challenges in Water Management

9. 10th Kovacs Colloquium UNESCO Intelligence http://www.home-business-smarts.net/ Mainstream Science on Intelligence December 13, 1994: An Editorial With 52 Signatories, by Linda S. Gottfredson, University of Delaware Intelligence is a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience.

10. 10th Kovacs Colloquium UNESCO Animals’ Intelligence  Defining intelligence through humans is not appropriate to design intelligent machines: –Animals are intelligent too  Dog IQ test:  Dogs can learn 165 words (similar to 2 year olds)  Average dog has the mental abilities of a 2-year-old child (or better)  They would beat a 3- or 4-year-old in basic arithmetic,  Dogs show some basic emotions, such as happiness, anger and disgust  “The social life of dogs is very complex - more like human teenagers - interested in who is moving up in the pack, who is sleeping with who etc,“ says professor Stanleay Coren from University of British Columbia

11. 10th Kovacs Colloquium UNESCO Computational Models of Intelligence  Five paradigms of Computational intelligence http://rtpis.mst.edu/images/paradigms_of_CI.jpg  How to define and compute intelligence?

12. 10th Kovacs Colloquium UNESCO Traditional AI Embodied Intelligence  Abstract intelligence  attempt to simulate “highest” human faculties: –language, discursive reason, mathematics, abstract problem solving  Environment model  Condition for problem solving in abstract way  “brain in a vat”  Embodiment  knowledge is implicit in the fact that we have a body –embodiment supports brain development  Intelligence develops through interaction with environment  Situated in environment  Environment is its best model

13. 10th Kovacs Colloquium UNESCO Design principles of intelligent systems from Rolf Pfeifer “Understanding of Intelligence”, 1999  Interaction with complex environment  cheap design  ecological balance  redundancy principle  parallel, loosely coupled processes  asynchronous  sensory-motor coordination  value principle Agent Drawing by Ciarán O’Leary- Dublin Institute of Technology

14. 10th Kovacs Colloquium UNESCO Embodied Intelligence –Mechanism: biological, mechanical or virtual agent with embodied sensors and actuators –EI acts on environment and perceives its actions –Environment hostility is persistent and stimulates EI to act –Hostility: direct aggression, pain, scarce resources, etc –EI learns so it must have associative self-organizing memory –Knowledge is acquired by EI Definition  Embodied Intelligence (EI) is a mechanism that learns how to survive in a hostile environment

15. 10th Kovacs Colloquium UNESCO Embodied Intelligence  For water resource planning and management hostility of the environment means  Insufficient water resources  Poor water quality  Growing demand of industry for water  Conflicts between stakeholders, etc  These hostile signals represent the primitive pains that grow unless they are addressed by proper actions  Surviving in this environment (politically) is to keep these signals below specified level, otherwise economical crises, social unrest, drought or famine will follow

16. 10th Kovacs Colloquium UNESCO Embodiment of a Mind  Embodiment is a part of the environment that EI controls to interact with the rest of the environment  It contains intelligence core and sensory motor interfaces under its control  Necessary for development of intelligence  Not necessarily constant or in the form of a physical body  Boundary transforms modifying brain’s self- determination

17. 10th Kovacs Colloquium UNESCO  Brain learns own body’s dynamic  Self-awareness is a result of identification with own embodiment  Embodiment can be extended by using tools and machines  Successful operation is a function of correct perception of environment and own embodiment Embodiment of a Mind

18. 10th Kovacs Colloquium UNESCO INPUTOUTPUT Simulation or Real-World System Task Environment Agent Architecture Long-term Memory Short-term Memory Reason Act Perceive RETRIEVALLEARNING EI Interaction with Environment From Randolph M. Jones, P : www.soartech.com

19. 10th Kovacs Colloquium UNESCO How to Motivate a Machine ? The fundamental question is how to motivate a machine to do anything, in particular to increase its “brain” complexity? How to motivate it to explore the environment and learn how to effectively work in this environment? Can a machine that only implements externally given goals be intelligent? If not how these goals can be created?

20. 10th Kovacs Colloquium UNESCO  I suggest that hostility of environment motivates us.  It is the pain that moves us.  Our intelligence that tries to minimize this pain motivates our actions, learning and development  We need both the environment hostility and the mechanism that learns how to reduce inflicted by the environment pain How to Motivate a Machine ?  In this work I propose, based on the pain, mechanism that motivates the machine to act, learn and develop.  Without the pain there will be no motivation to develop.

21. 10th Kovacs Colloquium UNESCO Motivated Learning  I suggest a goal-driven mechanism to motivate a machine to act, learn, and develop.  A simple pain based goal creation system.  It uses externally defined pain signals that are associated with primitive pains.  Machine is rewarded for minimizing the primitive pain signals.  Definition: Motivated learning (ML) is learning based on the self-organizing system of goal creation in embodied agent.  Machine creates abstract goals based on the primitive pain signals.  It receives internal rewards for satisfying its goals (both primitive and abstract).  ML applies to EI working in a hostile environment.

22. 10th Kovacs Colloquium UNESCO Pain-center and Goal Creation  Simple Mechanism  Creates hierarchy of values  Motivation is to reduce the primitive pain level  Leads to formulation of complex goals  Reinforcement : Pain increase Pain decrease  Forces exploration + - Environment Sensor Motor Pain level Dual pain level Pain increase Pain decrease (-) (+) Motivation (-) (+) Wall-E’s goal is to keep his plants from dying (+) (-) Goal

23. 10th Kovacs Colloquium UNESCO Primitive Goal Creation -+ Pain Dry soil Primitive level open tank sit on garbage refill faucet w. can water Dual pain  Reinforcing a proper action

24. 10th Kovacs Colloquium UNESCO Abstract Goal Hierarchy  Abstract goals are created to reduce abstract pains in order to satisfy the primitive goals  A hierarchy of abstract goals is created - they satisfy the lower level goals Activation Stimulation Inhibition Reinforcement Echo Need Expectation -+ + Dry soil Primitive Level Level I Level II faucet - w. can open water + Sensory pathway (perception, sense) Motor pathway (action, reaction) Level III tank - refill

25. 10th Kovacs Colloquium UNESCO Motivated Learning Experiment Sensory-motor pairs and their effect on the environment SensoryMotorIncreasesDecreases Dry soilWater from CanMoistureWater in Can No Water in CanWater from TankWater in CanWater in Tank No Water in TankWater from ReservoirWater in TankWater in Reservoir No Water in ReservoirWater from LakeWater in ReservoirWater in Lake No Water in LakeRegulate UsageWater in Lake- Case study: “How can Wall-E water his plants if the water resources are limited and hard to find?”

26. 10th Kovacs Colloquium UNESCO Action scatters in 5 ML simulations Motivated Learning Experiment

27. 10th Kovacs Colloquium UNESCO The average pain signals in 100 ML simulations 0100200300400500600 0 0.5 Primitive pain – dry soil Pain 0100200300400500600 0 0.1 0.2 Lack of water in can Pain 0100200300400500600 0 0.1 0.2 Lack of water in tank Pain 0100200300400500600 0 0.1 0.2 Lack of water in reservoir Pain 0100200300400500600 0 0.05 0.1 Lack of water in lake Pain Discrete time Motivated Learning Experiment

28. 10th Kovacs Colloquium UNESCO Averaged performance over 10 trials: ML RL Machine using ML learns to control all abstract pains and maintains the primitive pain signal on a low level. ML vs. Reinforcement Learning

29. 10th Kovacs Colloquium UNESCO Multiple dependencies: - two resources that can provide money Motivated Learning Experiment II

30. 10th Kovacs Colloquium UNESCO  When the environment is abundant in both resources  Competition between Work and Sell valuables  The loser and its associated further goals will be ignored by the system Motivated Learning Experiment II

31. 10th Kovacs Colloquium UNESCO ML Abstract Goal Hierarchy

32. 10th Kovacs Colloquium UNESCO Compare ML and RL Mean primitive pain P p value as a function of the number of iterations. >10 levels of hierarchy >complex environment - green line for RL - blue line for ML.

33. 10th Kovacs Colloquium UNESCO Reinforcement Learning Motivated Learning  Single value function  Measurable rewards  Can be optimized  Predictable  Objectives set by designer  Maximizes the reward  Potentially unstable  Learning effort increases with complexity  Always active  Multiple value functions  One for each goal  Internal rewards  Cannot be optimized  Unpredictable  Sets its own objectives  Solves minimax problem  Always stable  Learns better in complex environment than RL  Acts when needed http://www.bradfordvts.co.uk/images/goal.jpg

34. 10th Kovacs Colloquium UNESCO Machine Working for Humanity?  If you’re trying to look far ahead, and what you see seems like science fiction, it might be wrong.  But if it doesn’t seem like science fiction, it’s definitely wrong. From presentation by Foresight Institute

35. 10th Kovacs Colloquium UNESCOQuestions?