Presentation is loading. Please wait.

Presentation is loading. Please wait.

Reinforcement Learning: Learning from Interaction

Similar presentations

Presentation on theme: "Reinforcement Learning: Learning from Interaction"— Presentation transcript:

1 Reinforcement Learning: Learning from Interaction
Winter School on Machine Learning and Vision, 2010 B. Ravindran Many slides adapted from Sutton and Barto

2 Learning to Control So far looked at two models of learning
Supervised: Classification, Regression, etc. Unsupervised: Clustering, etc. How did you learn to cycle? Neither of the above Trial and error! Falling down hurts! Intro to RL

3 Can You hear me now? Can You hear me now? Can You hear me now?

4 Reinforcement Learning
A trial-and-error learning paradigm Rewards and Punishments Not just an algorithm but a new paradigm in itself Learn about a system – behaviour control from minimal feed back Inspired by behavioural psychology Intro to RL

5 RL Framework Learn from close interaction Stochastic environment
State Action evaluation Agent Benefits of RL.Applications. Future rewards. Situate in literature.. AI. Stochastic systems. Connections to OR. Learn from close interaction Stochastic environment Noisy delayed scalar evaluation Maximize a measure of long term performance

6 Not Supervised Learning!
Input Output Agent Target Error Very sparse “supervision” No target output provided No error gradient information available Action chooses next state Explore to estimate gradient – Trail and error learning Intro to RL

7 Not Unsupervised Learning
Input Activation Agent Evaluation Sparse “supervision” available Pattern detection not primary goal Intro to RL

8 TD Gammon Tesauro 1992, 1994, 1995, ... White has just rolled a 5 and a 2 so can move one of his pieces 5 and one (possibly the same) 2 steps Objective is to advance all pieces to points 19-24 Hitting 30 pieces, 24 locations implies enormous number of configurations Effective branching factor of 400 Intro to RL

9 The Agent-Environment Interface
. . . s a r t +1 t +2 t +3

10 The Agent Learns a Policy
Reinforcement learning methods specify how the agent changes its policy as a result of experience. Roughly, the agent’s goal is to get as much reward as it can over the long run. Intro to RL

11 Goals and Rewards Is a scalar reward signal an adequate notion of a goal?—maybe not, but it is surprisingly flexible. A goal should specify what we want to achieve, not how we want to achieve it. A goal must be outside the agent’s direct control—thus outside the agent. The agent must be able to measure success: explicitly; frequently during its lifespan. Intro to RL

12 Returns Episodic tasks: interaction breaks naturally into episodes, e.g., plays of a game, trips through a maze. where T is a final time step at which a terminal state is reached, ending an episode. Intro to RL

13 The Markov Property “the state” at step t, means whatever information is available to the agent at step t about its environment. The state can include immediate “sensations”, highly processed sensations, and structures built up over time from sequences of sensations. Ideally, a state should summarize past sensations so as to retain all “essential” information, i.e., it should have the Markov Property: Intro to RL

14 Markov Decision Processes
If a reinforcement learning task has the Markov Property, it is basically a Markov Decision Process (MDP). If state and action sets are finite, it is a finite MDP. To define a finite MDP, you need to give: state and action sets one-step “dynamics” defined by transition probabilities: reward expectations: Intro to RL

15 An Example Finite MDP Recycling Robot
At each step, robot has to decide whether it should (1) actively search for a can, (2) wait for someone to bring it a can, or (3) go to home base and recharge. Searching is better but runs down the battery; if runs out of power while searching, has to be rescued (which is bad). Decisions made on basis of current energy level: high, low. Reward = number of cans collected Intro to RL

16 Recycling Robot MDP Intro to RL

17 Value Functions The value of a state is the expected return starting from that state; depends on the agent’s policy: The value of a state-action pair is the expected return starting from that state, taking that action, and thereafter following p : Intro to RL

18 Bellman Equation for a Policy p
The basic idea: So: Or, without the expectation operator: Intro to RL

19 Optimal Value Functions
For finite MDPs, policies can be partially ordered: There is always at least one (and possibly many) policies that is better than or equal to all the others. This is an optimal policy. We denote them all p *. Optimal policies share the same optimal state-value function: Intro to RL

20 Bellman Optimality Equation
The value of a state under an optimal policy must equal the expected return for the best action from that state: is the unique solution of this system of nonlinear equations. Intro to RL

21 Bellman Optimality Equation
Similarly, the optimal value of a state-action pair is the expected return for taking that action and thereafter following the optimal policy is the unique solution of this system of nonlinear equations. Intro to RL

22 Dynamic Programming DP is the solution method of choice for MDPs
Require complete knowledge of system dynamics (P and R) Expensive and often not practical Curse of dimensionality Guaranteed to converge! RL methods: online approximate dynamic programming No knowledge of P and R Sample trajectories through state space Some theoretical convergence analysis available Intro to RL

23 Policy Evaluation Policy Evaluation: for a given policy p, compute the
state-value function Recall: Intro to RL

24 Policy Improvement Suppose we have computed for a deterministic policy p. For a given state s, would it be better to do an action ? Intro to RL

25 Policy Improvement Cont.
Intro to RL

26 Policy Improvement Cont.
Intro to RL

27 Policy Iteration policy evaluation policy improvement “greedification”
Intro to RL

28 Value Iteration Recall the Bellman optimality equation:
We can convert it to an full value iteration backup: Iterate until “convergence” Intro to RL

29 Generalized Policy Iteration
Generalized Policy Iteration (GPI): any interaction of policy evaluation and policy improvement, independent of their granularity. A geometric metaphor for convergence of GPI: Intro to RL

30 Dynamic Programming T T T T T T T T T T T T T Intro to RL

31 Simplest TD Method T T T T T T T T T T T Intro to RL

32 RL Algorithms – Prediction
Policy Evaluation (the prediction problem): for a given policy, compute the state-value function. No knowledge of P and R, but access to the real system, or a “sample” model assumed. Uses “bootstrapping” and sampling Intro to RL

33 Advantages of TD TD methods do not require a model of the environment, only experience TD methods can be fully incremental You can learn before knowing the final outcome Less memory Less peak computation You can learn without the final outcome From incomplete sequences Intro to RL

34 RL Algorithms – Control
SARSA Q-learning Intro to RL

35 Cliffwalking e-greedy, e = 0.1 Intro to RL

36 Actor-Critic Methods Explicit representation of policy as well as value function Minimal computation to select actions Can learn an explicit stochastic policy Can put constraints on policies Appealing as psychological and neural models Intro to RL

37 Actor-Critic Details Intro to RL

38 Applications of RL Robot navigation Adaptive control
Helicopter pilot! Combinatorial optimization VLSI placement and routing , elevator dispatching Game playing Backgammon – world’s best player! Computational Neuroscience Modeling of reward processes Intro to RL

39 Other Topics Different measures of return Policy gradient approaches
Average rewards, Discounted Returns Policy gradient approaches Directly perturb policies Generalization [Saturday] Function approximation Temporal abstraction Least square methods Better use of data More suited for “off-line” RL Intro to RL

40 References Mitchell, T. Machine Learning. McGraw Hill. 1992
Russell, S. J. and Norvig, P. Artificial Intelligence – A modern approach. Pearson Educational Sutton, R.S. and Barto, A. G. Reinforcement Learning: An Introduction. MIT Press Dayan, P. and Abbott, L. F. Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems. MIT Press Bertsikas, D. P. and Tsitsiklis, J. N. Neuro-Dynamic Programming. Athena Scientific Intro to RL

Download ppt "Reinforcement Learning: Learning from Interaction"

Similar presentations

Ads by Google