1. EE141 Motivated Learning based on Goal Creation Janusz Starzyk School of Electrical Engineering and Computer Science, Ohio University, USA www.ent.ohiou.edu/~starzyk Istituto Dalle Molle di Studi sull'Intelligenza Artificiale, 4 December 2009.

2. EE141  Embodied Intelligence (EI)  Embodiment of Mind  How to Motivate a Machine  Goal Creation Hierarchy  GCS Experiment  Motivated Learning Outline

3. EE141 Design principles of intelligent systems from Rolf Pfeifer “Understanding of Intelligence”, 1999  Interaction with complex environment  cheap design  ecological balance  redundancy principle  parallel, loosely coupled processes  asynchronous  sensory-motor coordination  value principle Agent Drawing by Ciarán O’Leary- Dublin Institute of Technology

4. EE141 Embodied Intelligence –Mechanism: biological, mechanical or virtual agent with embodied sensors and actuators –EI acts on environment and perceives its actions –Environment hostility is persistent and stimulates EI to act –Hostility: direct aggression, pain, scarce resources, etc –EI learns so it must have associative self-organizing memory –Knowledge is acquired by EI  Definition Embodied Intelligence (EI) is a mechanism that learns how to survive in a hostile environment

5. EE141 Embodiment of a Mind  Embodiment is a part of environment under control of the mind  It contains intelligence core and sensory motor interfaces to interact with environment  It is necessary for development of intelligence  It is not necessarily constant

6. EE141  Changes in embodiment modify brain’s self-determination  Brain learns its own body’s dynamics  Self-awareness is a result of identification with own embodiment  Embodiment can be extended by using tools and machines  Successful operation is a function of correct perception of environment and own embodiment Embodiment of Mind

7. EE141 How to Motivate a Machine ? A fundamental question is what motivates an agent to do anything, and in particular, to enhance its own complexity? What drives an agent to explore the environment and learn ways to effectively interact with it?

8. EE141 How to Motivate a Machine ?  Pfeifer claims that an agent’s motivation should emerge from the developmental process.  He called this the “motivated complexity” principle.  Chicken and egg problem? An agent must have a motivation to develop while his motivation comes from development?  Steels suggested equipping an agent with self-motivation.  “Flow” experienced when people perform their expert activity well would motivate to accomplish even more complex tasks.  But what is the mechanism of “flow”?  Oudeyer proposed an intrinsic motivation system.  Motivation comes from a desire to minimize the prediction error.  Similar to “artificial curiosity” presented by Schmidhuber.

9. EE141 How to Motivate a Machine ?  Exploration is needed in order to learn and to model the environment.  But is exploration the only motivation we need to develop EI?  Can we find a more efficient mechanism for learning?  I suggest a simpler mechanism to motivate a machine.  Although artificial curiosity helps to explore the environment, it leads to learning without a specific purpose.  It may be compared to exploration in reinforcement learning.

10. EE141 How to Motivate a Machine ?  I suggest that it is the hostility of the environment, in the definition of EI that is the most effective motivational factor.  It is the pain we receive that moves us.  It is our intelligence determined to reduce this pain that motivates us to act, learn, and develop.  Both are needed - hostility of the environment and intelligence that learns how to reduce the pain.  Thus pain is good.  Without pain we would not be motivated to develop. Fig. englishteachermexico.wordpress.com/

11. EE141 Motivated Learning  I suggest a goal-driven mechanism to motivate a machine to act, learn, and develop.  A simple pain based goal creation system.  It uses externally defined pain signals that are associated with primitive pains.  Machine is rewarded for minimizing the primitive pain signals.  Definition: Motivated learning (ML) is learning based on the self-organizing system of goal creation in embodied agent.  Machine creates abstract goals based on the primitive pain signals.  It receives internal rewards for satisfying its goals (both primitive and abstract).  ML applies to EI working in a hostile environment.

12. EE141 Pain-center and Goal Creation  Simple Mechanism  Creates hierarchy of values  Leads to formulation of complex goals  Reinforcement Pain increase Pain decrease  Forces exploration

13. EE141 Abstract Goal Creation for ML  The goal is to reduce the primitive pain level  Abstract goals are created if they satisfy the primitive goals Expectation Association Inhibition Reinforcement Connection Planning -+ PainDual pain Food refrigerator -+ Stomach Abstract pain (Delayed memory of pain) “food”becomes a sensory input to abstract pain center Sensory pathway (perception, sense) Motor pathway (action, reaction) Primitive Level Level I Level II Eat Open

14. EE141 Goal Creation Experiment Sensory-motor pairs and their effect on the environment PAIR # SENSORYMOTORINCREASESDECREASES 1FoodEatsugar levelfood supplies 8GroceryBuyfood suppliesmoney at hand 15BankWithdrawmoney at handspending limits 22OfficeWorkspending limitsjob opportunities 29SchoolStudyjob opportunities -

15. EE141 Goal Creation Experiment in ML Pain signals in GCS simulation

16. EE141 Goal Creation Experiment in ML Action scatters in 5 GCS simulations

17. EE141 Goal Creation Experiment in ML The average pain signals in 100 GCS simulations

18. EE141 Compare RL (TDF) and ML (GCS) Mean primitive pain P p value as a function of the number of iterations: - green line for TDF - blue line for GCS. Primitive pain ratio with pain threshold 0.1

19. EE141  Comparison of execution time on log-log scale  TD-Falcon green  GCS blue  Combined efficiency of GCS 1000 better than TDF Compare RL (TDF) and ML (GCS) Problem solved Conclusion: embodied intelligence, with motivated learning based on goal creation is an effective learning and decision making system for dynamic environments.

20. EE141 Reinforcement Learning Motivated Learning  Single value function  Measurable rewards  Can be optimized  Predictable  Objectives set by designer  Maximizes the reward  Potentially unstable  Learning effort increases with complexity  Always active  Multiple value functions  One for each goal  Internal rewards  Cannot be optimized  Unpredictable  Sets its own objectives  Solves minimax problem  Always stable  Learns better in complex environment than RL  Acts when needed

21. EE141 Sounds like science fiction  If you’re trying to look far ahead, and what you see seems like science fiction, it might be wrong.  But if it doesn’t seem like science fiction, it’s definitely wrong. From presentation by Feresight Institute

22. EE141 Questions?

23. From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006 Resources – Evolution of Electronics

24. EE141 By Gordon E. Moore

25. EE141

26. From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006 Clock Speed (doubles every 2.7 years)

27. EE141 Doubling (or Halving) times  Dynamic RAM Memory “Half Pitch” Feature Size5.4 years  Dynamic RAM Memory (bits per dollar)1.5 years  Average Transistor Price1.6 years  Microprocessor Cost per Transistor Cycle1.1 years  Total Bits Shipped1.1 years  Processor Performance in MIPS1.8 years  Transistors in Intel Microprocessors2.0 years  Microprocessor Clock Speed2.7 years From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006

28. EE141 From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006

29. EE141 From Hans Moravec, Robot, 1999

30. EE141 Software or hardware?  Sequential  Error prone  Require programming  Low cost  Well developed programming methods  Concurrent  Robust  Require design  Significant cost  Hardware prototypes hard to build SoftwareHardware

31. EE141 Future software/hardware capabilities Human brain complexity

32. EE141 Why should we care? Source: SEMATECH

33. EE141 Percent of die area that must be occupied by memory to maintain SOC design productivity Design Productivity Gap  Low-Value Designs? Source = Japanese system-LSI industry

34. EE141 Self-Organizing Learning Arrays SOLAR  Integrated circuits connect transistors into a system -millions of transistors easily assembled -first 50 years of microelectronic revolution  Self-organizing arrays connect processors into a system -millions of processors easily assembled -next 50 years of microelectronic revolution  * Self-organization  * Sparse and local interconnections  * Dynamically reconfigurable  * Online data-driven learning