Download presentation
Presentation is loading. Please wait.
Published bySolomon Stuart McBride Modified over 9 years ago
1
Reinforcement Learning & Apprenticeship Learning Chenyi Chen
2
Markov Decision Process (MDP) What’s MDP? A sequential decision problem Fully observable, stochastic environment Markovian transition model: the n th state is only determined by (n-1) th state and (n- 1) th action Each state has a reward, and the reward is additive
3
Markov Decision Process (MDP) State s: a representation of current environment;
4
Markov Decision Process (MDP) Example: Tom and Jerry, control Jerry (Jerry’s perspective) State: the position of Tom and Jerry, 25*25=625 in total; One of the states
5
Markov Decision Process (MDP) State s: a representation of current environment; Action a: the action can be taken by the agent in state s;
6
Markov Decision Process (MDP) Example: Tom and Jerry, control Jerry (Jerry’s perspective) State: the position of Tom and Jerry, 25*25=625 in total; Action: both can move to the neighboring 8 squares or stay; One of the states
7
Markov Decision Process (MDP) State s: a representation of current environment; Action a: the action can be taken by the agent in state s; Reward R(s): the reward of current state s (+,-,0); Value (aka utility) of state s: different from reward, related with future optimal actions;
8
An Straightforward Example 100 bucks if you came to class Reward of “come to class” is 100 You can use the money to: Eat food (you only have 50 bucks left) Stock market (you earn 1000 bucks, including the invested 100 bucks) The value (utility) of “come to class” is 1000
9
Markov Decision Process (MDP) Example: Tom and Jerry, control Jerry (Jerry’s perspective) State: the position of Tom and Jerry, 25*25=625 in total; Action: both can move to the neighboring 8 squares or stay; Reward: 1) Jerry and cheese at the same square, +5; 2) Tom and Jerry at the same square, -20; 3) otherwise 0; One of the states
10
Markov Decision Process (MDP) State s: a representation of current environment; Action a: the action can be taken by the agent in state s; Reward R(s): the reward of current state s (+,-,0); Value (aka utility) of state s: different from reward, related with future optimal actions; Transition probability P(s’|s,a): given the agent is in state s and taking action a, the probability of reaching state s’ in the next step;
11
Markov Decision Process (MDP) Example: Tom and Jerry, control Jerry (Jerry’s perspective) State: the position of Tom and Jerry, 25*25=625 in total; Action: both can move to the neighboring 8 squares or stay; Reward: 1) Jerry and cheese at the same square, +5; 2) Tom and Jerry at the same square, -20; 3) otherwise 0; Transition probability: about Tom’s moving pattern. One of the states
12
Markov Decision Process (MDP) Example: Tom and Jerry, control Jerry (Jerry’s perspective) …
13
Markov Decision Process (MDP) State s: a representation of current environment; Action a: the action can be taken by the agent in state s; Reward R(s): the reward of current state s (+,-,0); Value (aka utility) of state s: different from reward, related with future optimal actions; Transition probability P(s’|s,a): given the agent is in state s and taking action a, the probability of reaching state s’ in the next step; Policy π(s)->a: a table of state-action pairs, given state s, output action a that should be taken.
14
Bellman Equation
16
Value Iteration
18
Reinforcement Learning Similar to MDPs But we assume the environment model (transition probability P(s’|s,a) ) is unknown
19
Reinforcement Learning How to solve it? Solution #1: Use Monte Carlo method to sample the transition probability, then implement Value Iteration limitation: too slow for problems with many possible states because it ignores frequencies of states
20
Monte Carlo Method A broad class of computational algorithms that rely on repeated random sampling to obtain numerical results; Typically one runs simulations many times in order to obtain the distribution of an unknown probabilistic entity. From Wikipedia
21
Monte Carlo Example
22
Reinforcement Learning How to solve it? Solution #1: Use Monte Carlo method to sample the transition probability, then implement Value Iteration limitation: too slow for problems with many possible states because it ignores frequencies of states Solution #2: Q-learning the major algorithm for reinforcement learning
23
Q-learning
25
Playing Atari with Deep Reinforcement Learning The Atari 2600 is a video game console released in September 1977 by Atari, Inc. Atari emulator: Arcade Learning Environment (ALE)
26
What did they do? Train a deep learning convolutional neural network Input is current state (raw image sequence) Output is all the legal action and corresponding Q(s,a) value Let the CNN play Atari games
27
What’s Special? Input is raw image! Output is the action! Game independent, same convolutional neural network for all games Outperform human expert players in some games
28
Problem Definition
29
A Variant of Q-learning In the paper:
30
Deep Learning Approach Approach the Q-value with a convolutional neural network Q(s,a;θ) VS Straightforward structure The structure used in the paper
31
How to Train the Convolutional Neural Network? Loss function: Q-value is defined as: Where:
32
Some Details The distribution of action a (ε-greedy policy): choose a “best” action with probability 1- ε, and selects a random action with probability ε, ε annealed linearly from 1 to 0.1 Input image preprocessing function φ(s t ) Build a huge database to store historical samples
33
During Training… Play the game for one step Do mini-batch gradient descent on parameter θ for one step Add new data sample to database
34
CNN Training Pipeline
35
After Training… Play the game
36
Results Screen shots from five Atari 2600 games: (Left-to-right) Beam Rider, Breakout, Pong, Seaquest, Space Invaders Comparison of average total reward for various learning methods by running an ε-greedy policy with ε = 0.05 for a fixed number of steps
37
Results The leftmost plot shows the predicted value function for a 30 frame segment of the game Seaquest. The three screenshots correspond to the frames labeled by A, B, and C respectively
38
Apprenticeship Learning via Inverse Reinforcement Learning Teach the computer to do something by demonstration, rather than by telling it the rules or reward Reinforcement Learning: tell computer the reward, let it learn by itself using the reward Apprenticeship Learning: demonstrate to the computer, let it mimic the performance
39
Why Apprenticeship Learning? For standard MDPs, a reward for each state needs to be specified Specify a reward some time is not easy, what’s the reward for driving? When teaching people to do something (e.g. driving), usually we prefer to demonstrate rather than tell them the reward function
40
How Does It Work? Reward is unknown, but we assume it’s a linear function of features, is a function mapping state s to features, so:
41
Example of Feature State s t of the red car is defined as: s t ==1 left lane, s t ==2 middle lane, s t ==3 right lane Feature φ(s t ) is defind as: [1 0 0] left lane, [0 1 0] middle lane, [0 0 1] right lane w is defined as: w=[0.1 0.5 0.3] R(left lane)=0.1, R(middle lane)=0.5, R(right lane)=0.3 So in this case staying in the middle lane is preferred
42
How Does It Work? Reward is unknown, but we assume it’s a linear function of features, is a function mapping state s to features, so: The value (utility) of policy π is:
43
How Does It Work? Define feature expectation as: Then: Assume the expert’s demonstration defines the optimal policy: We need to sample the expert’s feature expectation by (sample m times):
44
What Does Feature Expectation Look Like? State s t of the red car is defined as: s t ==1 left lane, s t ==2 middle lane, s t ==3 right lane Feature φ(s t ) is defind as: [1 0 0] left lane, [0 1 0] middle lane, [0 0 1] right lane During sampling, assume γ=0.9 Step 1, red car in middle lane μ=0.9^0*[0 1 0]=[0 1 0] Step 2, red car still in middle lane μ= [0 1 0]+0.9^1*[0 1 0]=[0 1.9 0] Step 3, red car move to left lane μ= [0 1.9 0]+0.9^2*[1 0 0]=[0.81 1.9 0] …
45
How Does It Work? We want to mimic the expert’s performance by minimize the difference between and If we have, and assuming Then
46
Pipeline
47
Supporting Vector Machine (SVM) The 2 nd step of the pipeline is a SVM problem Which can be rewritten as:
48
Pipeline
49
Their Testing System
50
Demo Videos Driving StyleExpertLearned Controller Both (Expert left, Learned right) 1: Niceexpert1.avilearnedcontroller1.avijoined1.avi 2: Nastyexpert2.avilearnedcontroller2.avijoined2.avi 3: Right lane niceexpert3.avilearnedcontroller3.avijoined3.avi 4: Right lane nastyexpert4.avilearnedcontroller4.avijoined4.avi 5: Middle laneexpert5.avilearnedcontroller5.avijoined5.avi http://ai.stanford.edu/~pabbeel/irl/
51
Their Results
52
Questions?
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.