1. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 1 Reinforcement Learning (RL) Consider an “agent” embedded in an environmentConsider an “agent” embedded in an environment Task of the agentTask of the agent Repeat forever: 1)sense world 2)reason 3)choose an action to perform

2. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 2 Definition of RL Assume the world (ie, environment) periodically provides rewards or punishments (“reinforcements”)Assume the world (ie, environment) periodically provides rewards or punishments (“reinforcements”) Based on reinforcements received, learn how to better choose actionsBased on reinforcements received, learn how to better choose actions

3. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 3 Sequential Decision Problems Courtesy of A.G. Barto, April 2000 Decisions are made in stagesDecisions are made in stages The outcome of each decision is not fully predictable but can be observed before the next decision is madeThe outcome of each decision is not fully predictable but can be observed before the next decision is made The objective is to maximize a numerical measure of total reward (or equivalently, to minimize a measure of total cost)The objective is to maximize a numerical measure of total reward (or equivalently, to minimize a measure of total cost) Decisions cannot be viewed in isolation:Decisions cannot be viewed in isolation: need to balance desire for immediate reward with possibility of high future reward

4. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 4 Reinforcement Learning vs Supervised Learning How would we use SL to train an agent in an environment?How would we use SL to train an agent in an environment? Show action to choose in sample of world states – “I/O pairs”Show action to choose in sample of world states – “I/O pairs” RL requires much less of teacherRL requires much less of teacher Must set up “reward structure”Must set up “reward structure” Learner “works out the details” – i.e. writes a program to maximize rewards receivedLearner “works out the details” – i.e. writes a program to maximize rewards received

5. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 5 Embedded Learning Systems: Formalization S E = the set of states of the worldS E = the set of states of the world e.g., an N -dimensional vectore.g., an N -dimensional vector “sensors”“sensors” A E = the set of possible actions an agent can performA E = the set of possible actions an agent can perform “effectors”“effectors” W = the worldW = the world R = the immediate reward structureR = the immediate reward structure W and R are the environment, can be probabilistic functions

6. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 6 Embedded learning Systems (formalization) W: S E x A E  S E The world maps a state and an action and produces a new state R: S E x A E  “reals” Provides rewards (a number) as a function of state and action (as in textbook). Can equivalently formalize as a function of state (next state) alone.

7. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 7 A Graphical View of RL Note that both the world and the agent can be probabilistic, so W and R could produce probability distributions.Note that both the world and the agent can be probabilistic, so W and R could produce probability distributions. For now, assume deterministic problemsFor now, assume deterministic problems The real world, W The Agent an action sensory info R, reward (a scalar) - indirect teacher

8. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 8 Common Confusion State need not be solely the current sensor readings Markov AssumptionMarkov Assumption Value of state is independent of path taken to reach that state Can have memory of the pastCan have memory of the past Can always create Markovian task by remembering entire past history

9. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 9 Need for Memory: Simple Example “out of sight, but not out of mind” T=1 learning agent W A L L opponent T=2 learning agent W A L L opponent Seems reasonable to remember opponent recently seen

10. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 10 State vs. Current Sensor Readings Remember state is what is in one’s head (past memories, etc) not ONLY what one currently sees/hears/smells/etc

11. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 11 Policies The agent needs to learn a policy  E : S E  A E Given a world state, S E, which action, A E, should be chosen? The policy,  E, function Remember: The agent’s task is to maximize the total reward received during its lifetime

12. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 12 Policies (cont.) To construct  E, we will assign a utility (U) (a number) to each state. -  is a positive constant < 1 -R(s,  E, t) is the reward received at time t, assuming the agent follows policy  E and starts in state s at t=0 -Note: future rewards are discounted by  t-1

13. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 13 The Action-Value Function We want to choose the “best” action in the current state So, pick the one that leads to the best next state (and include any immediate reward) Let immediate reward received for going to state W(s,a) Future reward from further actions (discounted due to 1- step delay)

14. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 14 The Action-Value Function (cont. ) If we can accurately learn Q (the action- value function), choosing actions is easy Choose a, where

15. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 15 Q vs. U Visually U(1) U(3) U(2) U(4) U(5) U(6) state Q(1,i) Q(1,ii) Q(1,iii) state action Key states actions U’s “stored” on states Q’s “stored” on arcs

16. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 16 Q-Learning (Watkins PhD, 1989) Let Q t be our current estimate of the optimal Q Our current policy is such that Our current utility-function estimate is - hence, the U table is embedded in the Q table and we don’t need to store both

17. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 17 Q-Learning (cont.) Assume we are in state S t “Run the program” (1) for awhile (n steps) Determine actual reward and compare to predicted reward Adjust prediction to reduce error (1 ) I.e., follow the current policy

18. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 18 How Many Actions Should We Take Before Updating Q ? Why not do so after each action? “1 – Step Q learning”“1 – Step Q learning” Most common approachMost common approach

19. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 19 Exploration vs. Exploitation In order to learn about better alternatives, we can’t always follow the current policy (“exploitation”) Sometimes, need to try “random” moves (“exploration”)

20. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 20 Exploration vs. Exploitation (cont) Approaches 1) p percent of the time, make a random move; could let 2) Prob(picking action A in state S ) A in state S ) Exponentia- ting gets rid of negative values

21. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 21 0.S  initial state 1. If random #  P then a = random choice Else a =  t (S) 2. S new  W(S, a) R immed  R(S new ) 3.Q(S, a)  R immed +  max a’ Q(S new, a’) 4. S  S new Go to 1Go to 1 One-Step Q-Learning Algo Act on world and get reward

22. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 22 A Simple Example (of Q-learning - with updates after each step, ie N =1) Repeat (deterministic world, so α =1) Algo: Pick State +Action S 0 R = 0 S 1 R = 1 S 3 R = 0 S 2 R = -1 S 4 R = 3 Let  = 2/3 Q = 0

23. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 23 A Simple Example (Step 1) S 0  S 2 Repeat (deterministic world, so α =1) Algo: Pick State +Action S 0 R = 0 S 1 R = 1 S 3 R = 0 S 2 R = -1 S 4 R = 3 Let  = 2/3 Q = 0 Q = -1

24. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 24 A Simple Example (Step 2) S 2  S 4 Repeat (deterministic world, so α =1) Algo: Pick State +Action S 0 R = 0 S 1 R = 1 S 3 R = 0 S 2 R = -1 S 4 R = 3 Let  = 2/3 Q = 0 Q = 3 Q = 0 Q = -1

25. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 25 A Simple Example (Step  ) Repeat (deterministic world, so α =1) Algo: Pick State +Action S 0 R = 0 S 1 R = 1 S 3 R = 0 S 2 R = -1 S 4 R = 3 Let  = 2/3 Q = 1 Q = 3 Q = 0 Q = 1

26. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 26 Q-Learning: Implementation Details Remember, conceptually we are filling in a huge table S0 S1 S2... Sn abc...zabc...z... Q(S2, c)...... Tables are a very verbose representation of a function States Actions Actions

27. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 27 Q-Learning: Convergence Proof Applies to Q tables and deterministic, Markovian worlds. Initialize Q’s 0 or random finite.Applies to Q tables and deterministic, Markovian worlds. Initialize Q’s 0 or random finite. Theorem: if every state-action pair visited infinitely often, 0≤  <1, and |rewards| ≤ C (some constant), thenTheorem: if every state-action pair visited infinitely often, 0≤  <1, and |rewards| ≤ C (some constant), then  s, a the approx. Q table (Q) the true Q table (Q) ^

28. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 28 Q-Learning Convergence Proof (cont.) Consider the max error in the approx. Q-table at step t :Consider the max error in the approx. Q-table at step t : The max is finite since |r| ≤ C, so max | |The max is finite since |r| ≤ C, so max | | Since finite, we have finite, i.e. initial max error is finiteSince finite, we have finite, i.e. initial max error is finite

29. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 29 Q-Learning Convergence Proof (cont.) Let s’ be the state that results from doing action a in state s. Consider what happens when we visit s and do a at step t + 1: Current state Next state By Q-learning rule (one step) By def’n of Q (notice best a in s’ might be different)

30. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 30 Q-Learning Convergence Proof (cont.) By algebra Since Max at s’ ≤ max at any s Plugging in defn of Δ t Trickiest step, can prove by contradiction =  | max a’ Q t (s’, a’) – max a’’ Q(s’, a’’) | =  Δ t ≤  max a’’’ | Q t (s’, a’’’) – Q(s’, a’’’) | ≤  max s’’,a’’’ | Q t (s’’, a’’’) – Q(s’’, a’’’) | ^ ^ ^

31. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 31 Q-Learning Convergence Proof (cont.) Hence, every time, after t, we visit an, its Q value differs from the correct answer by no more than  Δ tHence, every time, after t, we visit an, its Q value differs from the correct answer by no more than  Δ t Let T o =t o (i.e. the start) and T N be the first time since T N-1 where every visited at least onceLet T o =t o (i.e. the start) and T N be the first time since T N-1 where every visited at least once Call the time between T N-1 and T N, a complete intervalCall the time between T N-1 and T N, a complete interval Clearly Δ T N Δ T N-1 Clearly Δ T N ≤  Δ T N-1

32. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 32 Q-Learning Convergence Proof (concluded) That is, every complete interval, Δ t is reduced by at least That is, every complete interval, Δ t is reduced by at least  Since we assumed every pair visited infinitely often, we will have an infinite number of complete intervalsSince we assumed every pair visited infinitely often, we will have an infinite number of complete intervals Hence, lim Δ t = 0 t   t  

33. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 33......... Q (S, a) Q (S, b) Q (S, z) Representing Q Functions More Compactly We can use some other function representation (eg, neural net) to compactly encode this big table An encoding of the state (S) Second argument is a constant Or could have one net for each possible action Each input unit encodes a property of the state (eg, a sensor value)

34. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 34 Q Tables vs Q Nets Given: 100 Boolean-valued features 10 possible actions 10 possible actions Size of Q table 10 * 2 to the power of 100 Size of Q net (100 HU’s) 100 * 100 + 100 * 10 = 11,000 # of possible states Weights between inputs and HU’s Weights between HU’s and outputs

35. © Jude Shavlik 2006, David Page 2007 CS 760 – Machine Learning (UW-Madison)RL Lecture, Slide 35 Why Use a Compact Q-Function? 1.Full Q table may not fit in memory for realistic problems 2.Can generalize across states, thereby speeding up convergence i.e., one example “fills” many cells in the Q table