1. Fuzzy Reinforcement Learning Agents By Ritesh Kanetkar Systems and Industrial Engineering Lab Presentation May 23, 2003

2. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering What is a agent? An agent is a computer system situated in some environment, and that is capable of autonomous action in this environment in order to meet its design objectives. An autonomous agent should be able to act without the direct intervention of humans or other agents, and should have control over its own actions and internal state.

3. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Why Agents? Ability to act autonomously Flexibility, scalability and modularity characteristics Real-time performance Suitability for distributed applications Ability to work co-operatively in teams

4. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Learning in Agents Supervised Learning  Neural Network Unsupervised Learning  Reinforcement Learning

5. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Supervised vs. Unsupervised Supervised Learning  Learning under a skilled teacher  Learning through presentation of input-output pairs  Given a set of inputs attempts to predict the output values Unsupervised Learning  No supervisor present  Only data available is through feedback  Learning through evaluation of actions

6. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Reinforcement Learning Maps states to actions Input is current state S1 Output is selected action Action change the state to S2 After evaluating the mapping a reinforcement signal is given to the agent

7. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Reinforcement Learning Advantages  Less environment oriented programming  Works in changing environment Problems  Large number of possible states  Consider only discrete events ( Real world problems are continuous)

8. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering b1b1 b3 b2 a1a1 a3 a2 R=0. 5 S1S2 T=30 M1 T=30 M1 T=20 M2 T=10 M3 S3 How RL works?

9. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Continued Aim 1 : To find the shortest processing time. Ideal Actions : a3 – b3. Assumptions :  Action with highest utility is chosen  Each machine bids for the part as per its utility value (initially all 0. 5).  The winning machine gives a part of its utility to the previous winning agent for successfully creating the state for him.

10. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Continued ( Rule for reward) Rule for giving reward to previous winning agent t (min) r 10 0.3 20 0.2 30 0.1 Reward from state S0 and S1, say 0.25 for our model.

11. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Continued (Calculations of utility value) IterationStateMachine selectedUtility value 1S1M2( 0.5 -0.25) + 0.2 = 0.45 2S1M1( 0.5 -0.25) + 0.1 = 0.35 3S1M3( 0.5 -0.25) + 0.3 = 0.55

12. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Continued (Changes in utility value) IterationsM1M2M3M1M2M3 00.5 1 0.450.5 20.350.450.5 30.350.450.55

13. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Use of Fuzzy Logic Fuzzy logic to map states (environment) to actions. Problem tackled is of the elimination of discrete events by use of fuzzy logic. Fuzzy logic to integrate the multiple rewards into a single feedback signal. Due to large action space we cannot use traditional lookup tables. So generalization of mapping is required. Incorporation of human language.

14. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Problems Agents as dynamical systems interacting with the environment Network of agents (Multi-agent system) Multiple reward system Multiple criteria systems Continues events system Large state space in real world problems Bargaining problems

15. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Fuzzy Inference System (FIS)

16. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering FIS FIS rule base is made of N rules : R i : If s1 = L 1i and ……and sn = L N1i then y1 = O 1o and ……and yn = O N1o Where, Si = input vector R i = i’th rule L ji = Fuzzy label Yi = Output vector

17. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Fuzzy Inference System (FIS) Layer 1: Input layer  Defines the input variables needed to describe the states completely. Layer 2: Linguistic Labels  This layer does the fuzzification process. Layer 3: Rules  This layer defines the if-else rules giving rule truth values. Layer 4: Output layer  Gives the FIS output.

18. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Assumptions Number of input variables and fuzzy labels are selected depending on problem Number of rules is determined by numbers of elements in first two layers. (Product of labels for each input variable) Each have a predefined number of outputs So only most difficult part left is the conclusion of all possible combinations (Rule conclusion)

19. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering What it does? Maps states to actions. Rules can be formulated in human language. Each rule contains:  Value V i to approx. optimal evaluation function.  Action set U i  Parameter vector w i giving the weight of different action in a rule to approximate policy. Final output is the weighted average of all the actions.

20. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering FIS Output (Primary reinforcement) (Internal reinforcement through critic)

21. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Procedure Estimate the evaluation function corresponding to current state. V t (S t+1 )= v t.Ф t+1 Compute the TD error є t+1. Tune the parameters v and w. Estimate the new evaluation function with new conclusion vector v t+1. Learning rate updating. Computing and triggering of global action U t+1

22. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Problem Single machine scheduling problem 3 parts Each part with individual earliness-tardiness penalties, due dates and processing times 19 time slots on machine Minimize the deviation from due dates reducing the penalties

23. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering Work in progress Currently working with the single machine scheduling problem with earliness/tardiness penalty and due dates. Identifying the various parameters. Understanding the mathematics behind the FIS. Incorporating bargaining model in FIS.

24. COMPUTER INTEGRATED MANUFACTURING LAB Department of Systems and Industrial Engineering References Fuzzy Inference System Learning by Reinforcement Methods – Lionel Jouffe (IEEE) Dynamic single machine scheduling under distributed decision making – Pooja Dewan, Sanjay Joshi (IJPR) Evolutionary Learning agents for shop floor control- Bruno Maione, David Naso (IEEE) A fuzzy logic based methodology to rank shop floor dispatching rules – Albert Petroni (IJPE) Multi Agent Reinforcement Learning with bidding for automatic segmentation of action sequence – Ron Sun (IEEE) AI depot - http://ai-depot.com/ (for RL)http://ai-depot.com/ RL – An Introduction (Suttons and Barto) Matlab fuzzy logic toolbox tutorials

25. Thank You