Presentation on theme: "Introduction to Reinforcement Learning"— Presentation transcript:
1 Introduction to Reinforcement Learning Gerry TesauroIBM T.J.Watson Research Center
2 Outline Statement of the problem: What RL is all about How it’s different from supervised learningMathematical FoundationsMarkov Decision Problem (MDP) frameworkDynamic Programming: Value Iteration, ...Temporal Difference (TD) and Q LearningApplications: Combining RL and function approximation
3 AcknowledgementLecture material shamelessly adapted from: R. S. Sutton and A. G. Barto, “Reinforcement Learning”Book published by MIT Press, 1998Available on the web at: RichSutton.comMany slides shamelessly stolen from web site
4 Basic RL Framework 1. Learning with evaluative feedback Learner’s output is “scored” by a scalar signal (“Reward” or “Payoff” function) saying how well it didSupervised learning: Learner is told the correct answer!May need to try different outputs just to see how well they score (exploration …)
9 Basic RL FrameworkLearner has to figure out which action is best, and which actions lead to which states. Might have to try all actions! Exploration vs. Exploitation: when to try a “wrong” action vs. sticking to the “best” action
10 Basic RL Framework 3. Learning Through Time: Reward is delayed (Act now, reap the reward later)Agent may take long sequence of actions before receiving reward“Temporal Credit Assignment” Problem: Given sequence of actions and rewards, how to assign credit/blame for each action?
14 Agent’s objective is to maximize expected value of “return” R: sum of future rewards: is a “discount parameter” (0 1)Example: Cart-Pole Balancing Problem:reward = -1 at failure, else 0expected return = -k for k steps to failurereward maximized by making k
15 We consider non-deterministic environments: Action at in state st Probability distribution of rewards rt+1Probability distribution of new states st+1Some environments have nice property: distributions are history-independent and stationary. These are called Markov environments and the agent’s task is a Markov Decision Problem (MDP)
16 An MDP specification consists of: list of states s Slist of legal action set A(s) for every sset of transition probabilities for every s,a,s’:set of expected rewards for every s,a,s’:
17 Given an MDP specification: Agent learns a policy : deterministic policy (s) = action to take in state snon-deterministic policy (s,a) = probability of choosing action a in state sAgent’s objective is to learn the policy that maximizes expected value of return Rt“Value Function” associated with a policy tells us how good the policy is. Two types of value functions ...
18 State-Value Function V (s) = Expected return starting in state s and following policy : Action-Value Function Q (s,a) = Expected return starting from action a in state s, and then following policy :
19 Bellman Equation for a Policy The basic idea:Apply expectation for state s under policy :A linear system of equations for V ; unique solution
22 Why V*, Q* are usefulAny policy that is greedy w.r.t. V* or Q* is an optimal policy *.One-step lookahead using V*:Zero-step lookahead using Q*:
23 Two methods to solve for V*, Q* Policy improvement: given a policy , find a better policy ’.Policy Iteration: Keep repeating above and ultimately you will get to *.Value Iteration: Directly solve Bellman’s optimality equation, without explicitly writing down the policy.
24 Policy ImprovementEvaluate the policy: given , compute V (s) and Q (s,a) (from linear Bellman equations).For every state s, construct new policy: do the best initial action, and then follow policy thereafter.The new policy is greedy w.r.t. Q (s,a) and V (s) V’ (s) V (s) ’ in our partial ordering.
25 Policy Improvement, contd. What if the new policy has the same value as the old policy? ( V’ (s) = V (s) for all s)But this is the Bellman Optimality equation: if V solves it, then it must be the optimal value function V*.
27 Value Iteration Use the Bellman Optimality equation to define an iterative “bootstrap” calculation:This is guaranteed to converge to a unique V* (backup is a contraction mapping)
28 Summary of DP methodsGuaranteed to converge to * in polynomial time (in size of state space); in practice often faster than linearThe method of choice if you can do it.Why it might not be doable:your problem is not an MDPthe transition probs and rewards are unknown or too hard to specifyBellman’s “curse of dimensionality:” the state space is too big (>> O(106) states)RL may be useful in these cases
29 Monte Carlo Methods Estimate V (s) by sampling perform a trial: run the policy starting from s until termination state reached; measure actual return RtN trials: average Rt accurate to ~ 1/sqrt(N)no “bootstrapping:” not using V(s’) to estimate V(s)Two important advantages of Monte Carlo:Can learn online without a model of the environmentCan learn in a simulated environment
31 Temporal Difference Learning Error signal: difference between current estimate and improved estimate; drives change of current estimateSupervised learning error:error(x) = target_output(x) - learner_output(x)Bellman error (DP):“1-step full-width lookahead” - “0-step lookahead”Monte Carlo error:error(s) = <Rt > - V(s)“many-step sample lookahead” - “0-step lookahead”
32 TD error signalTemporal Difference Error Signal: take one step using current policy, observe r and s’, then:“1-step sample lookahead” - “0-step lookahead”In particular, for undiscounted sequences with no intermediate rewards, we have simply:Self-consistent prediction goal: predicted returns should be self-consistent from one time step to the next (true of both TD and DP)
33 Learning using the Error Signal: we could just do a reassignment: But it’s often a good idea to learn incrementally:where is a small “learning rate” parameter (either constant, or decreases with time)the above algorithm is known as “TD(0)” ; convergence to be discussed later...
34 Advantages of TD Learning Combines the “bootstrapping” (1-step self-consistency) idea of DP with the “sampling” idea of MC; maybe the best of both worldsLike MC, doesn’t need a model of the environment, only experienceTD, but not MC, can be fully incrementalyou can learn before knowing the final outcomeyou can learn without the final outcome (from incomplete sequences)Bootstrapping TD has reduced variance compared to Monte Carlo, but possibly greater bias
39 The point of the parameter (My view): in TD() is a knob to twiddle: provides a smooth interpolation between =0 (pure TD) and =1 (pure MC)For many toy grid-world type problems, can show that intermediate values of work best.For real-world problems, best will be highly problem-dependent.
40 Convergence of TD ()TD() converges to the correct value function V (s) with probability 1 for all . Requires:lookup table representation (V(s) is a table),must visit all states an infinite # of times,a certain schedule for decreasing (t) (Usually (t) ~ 1/t)BUT: TD() converges only for a fixed policy. What if we want to learn as well as V? We still have more work to do ...
41 Q-Learning: TD Idea to Learn * Q-Learning (Watkins, 1989): one-step sample backup to learn action-value function Q(s,a). The most important RL algorithm in use today. Uses one-step error:to define an incremental learning algorithm:where (t) follows same schedule as in TD algorithm.
42 Nice properties of Q-learning Q guaranteed to converge to Q* w/probability 1.Greedy guaranteed to converge to *.But (amazingly), don’t need to follow a fixed policy, or the greedy policy, during learning! Virtually any policy will do, as long as all (s,a) pairs visited infinitely often.As with TD, don’t need a model, can learn online, both bootstraps and samples.
43 RL and Function Approximation DP infeasible for many real applications due to curse of dimensionality: |S| too big.FA may provide a way to “lift the curse:”complexity D of FA needed to capture regularity in environment may be << |S|.no need to sweep thru entire state space: train on N “plausible” samples and then generalize to similar samples drawn from the same distribution.PAC learning tells us generalization error ~D/N; N need only scale linearly with D.
44 RL + Gradient Parameter Training Recall incremental training of lookup tables:If instead V(s) = V (s), adjust to reduce MSE (R-V(s))2 by gradient descent:
45 Example: TD() training of neural networks (episodic; =1 and intermediate r = 0):
46 Case-Study Applications Several commonalities:Problems are more-or-less MDPs|S| is enormous can’t do DPState-space representation critical: use of “features” based on domain knowledgeFA is reasonably simple (linear or NN)Train in a simulator! Need lots of experience, but still << |S|Only visit plausible states; only generalize to plausible states
48 Learning backgammon using TD() Neural net observes a sequence of input patterns x1, x2, x3, …, xf : sequence of board positions occurring during a gameRepresentation: Raw board description (# of White or Black checkers at each location) using simple truncated unary encoding. (“hand-crafted features” added in later versions)At final position xf, reward signal z given:z = 1 if White wins;z = 0 if Black winsTrain neural net using gradient version of TD()Trained NN output Vt = V (xt , w) should estimate prob (White wins | xt )
50 Q: Who makes the moves??A: Let neural net make the moves itself, using its current evaluator: score all legal moves, and pick max Vt for White, or min Vt for Black.Hopelessly non-theoretical and crazy:Training V using non-stationary (no convergence proof)Training V using nonlinear func. approx. (no cvg. proof)Random initial weights Random initial play! Extremely long sequence of random moves and random outcome Learning seems hopeless to a human observerBut what the heck, let’s just try and see what happens...
51 “TD-Leaf:” n-step TD backups in 2-player TD-Gammon can teach itself by playing games against itself and learning from the outcomeWorks even starting from random initial play and zero initial expert knowledge (surprising!) achieves strong intermediate playadd hand-crafted features: advanced level of play (1991)2-ply search: strong master play (1993)3-ply search: superhuman play (1998)“TD-Leaf:” n-step TD backups in 2-playergames (Beal; Baxter et al.): great resultsfor checkers and chess
52 RL Success Stories/Videos U. Michigan RL wiki page:“keep-away” in Robocup simulatorAibo fast walk gate; ball acquisitionHumanoid robot Air hockeyHelicopter aerobatics (Ng et al.)Human flies helicopter for minsPerform System Identification: learn model of helicopter dynamicsUsing model, train RL policy in simulator
53 Cell-phone channel allocation S. Singh and D. Bertsekas, NIPS-96Dynamic resource allocation: assign channels to calls in a cell; can’t interfere with neighboring cellProblem is a real-time discrete-event MDP with huge state space ~ 7049 statesObjective: maximize:
54 Modified Bellman optimality equation Modify equation to handle continuous time, discrete events:where: s = configuration, e=random event (arrival, handoff, departure) a=action, t=random time to next event, c(s,a, t) = effective immediate payoff
55 represent sx using 2 features for each cell: Availability: # of free channels in a cellCell-channel packing: # of times channel is used in 4-cell radiusrepresent V using linear FA: V = xtrain in simulator using gradient version of TD(0)54
66 RL for Spoken Dialogue Systems Singh, Litman, Kearns, Walker (JAIR 2002)Sequence of human-computer speech interactionsUse in DB-query system “NJFun:” database of leisure activities in NJ, organized by (type, location, time)Humans aren’t MDPs, but pretend they are: devise MDP representation of system-human interaction:
67 Severely restrict state space: 7 state variables and 42 “choice-state” combinations
68 Actions are spoken requests to the user, classified as: Severely restrict the policy: 2 actions possible in each choice-state: possible policies; train using random explorationActions are spoken requests to the user, classified as:system initiative: “Please state the type of activity you are interested in”user initative: “How may I help you?”mixed initiative: “Please say the location you are interested in. You can also tell me the time.”confirmation of an attribute: “Did you say you are interested in going to a museum?”Train on a corpus of 311 dialogues (using AT&T volunteers); test trained system on 124 test dialogues. “Reward” after each dialogue is both objective (was the specific task completed exactly or partially) as well as subjective (“good,” “bad,” or “so-so” performance) from the humanSmall MDP but don’t have a model! Do Q-Learning using sample trajectories with the above random-exploration policy
69 Results: Learned policy much better than random exploration
70 Results: Learned policy much better than standard policies
72 RL Mashups RL + semi-supervised learning RL + active learning RL + metric learningRL + dimensionality reductionBayesian RLRL + SVMs/kernel methodsRL + semi-definite programmingRL + Gaussian process modelsetc. etc.NIPS 2006 workshop “Towards A New Reinforcement Learning:”
73 Final remarks on RLCan solve MDPs on-line, in real environment, without knowing underlying MDPFunction Approximators can avoid the “curse of dimensionality”Beyond MDPs: active research in RL for:high-level planning,structured (e.g. factored, hierarchical) MDPs,partially observable MDPs (POMDPs),history dependent problems,non-stationary problems,multi-agent problemsFor more info, go to: RichSutton.com
75 Outline Description of the problem Tools and concepts from RL & game theory“Naïve” approaches to multi-agent learningordinary single-agent RLevolutionary game theory“Sophisticated” approachesminimax-Q, FriendOrFoe-Q (Littman),tinkering with learning rates: WoLF (Bowling), “strategic teaching” (Camerer)Challenges and Opportunities
76 Normal single-agent learning Assume that environment has observable states, characterizable expected rewards and state transitions, and all of the above is stationary (MDP-ish)Non-learning, theoretical solution to fully specified problem: DP formalismLearning: solve by trial and error without a full specification: RL + exploration, Monte Carlo, ...
77 Multi-Agent Learning Problem: Agent tries to solve its learning problem, while other agents in the environment also are trying to solve their own learning problems. challenging non-stationarity.Main scenarios: (1) cooperative; (2) self-interest (many deep issues swept under the rug)Agent may know very little about other agents:payoffs may be unknownlearning algorithms unknownTraditional method of solution: game theory (uses several questionable assumptions)
78 MAL needs foundational principles! A precise problem formulation is still lacking! See: “If Multi-Agent Learning is the Answer, What is the Question?” Shoham et al, 2006Some (debatable) MAL objectives:Learning should converge to a stationary strategyIn “self-play” learning (all agents use same learning algorithm), learners should jointly converge to an equilibrium strategyLearning should achieve payoffs as good as a best-response to other agents’ strategies(Worst case bound) Learning should guarantee a minimum payoff (“security payment,” “no-regret” property)
79 Game TheoryProvides essential theoretical/conceptual background for tackling multi-agent learningWikipedia definition:Game theory is most often described as a branch of applied mathematics and economics that studies situations where players choose different actions in an attempt to maximize their returns. The essential feature, however, is that it provides a formal modelling approach to social situations in which decision makers interact with other minds.Today, widely used in politics, business, economics, biology, psychology, computer science etc.
80 Fundamental Postulate of Game Theory: “Rationality” A rational player/agent will make decisions that maximize her individual expected utility (= expected payoff for simplicity) given her understanding/beliefs about the problem. Also, perfectly indifferent to payoffs received by other players.
81 Basics of game theoryA game is specified by: players (1…N), actions, and (expected) payoff matrices (functions of joint actions)B’s actionA’s actionA’s payoff B’s payoffIf payoff matrices are identical, A and B are cooperative, else non-cooperative (zero-sum = purely competitive)
82 Basic lingo…(2) Games with no states: (bi)-matrix games Games with states: stochastic games, Markov games; (state transitions are functions of joint actions)Games with simultaneous moves: normal formGames with alternating turns: extensive formNo. of rounds = 1: one-shot gameNo. of rounds > 1: repeated gamedeterministic action policy: pure strategynon-deterministic action policy: mixed strategy e.g. Prob(R,P,S) = (½,¼,¼)
83 Stochastic vs. Matrix Games A stochastic game (a.k.a. “Markov game” ) generalizes MDPs to multiple agentsfinite state space Sjoint action setstationary reward distributionstationary transition probabilitiesA matrix game has no state information, only joint actions and payoffs (|S| = 1)
84 Basic AnalysisAgent i’s mixed strategy xi is a best-response to others’ x-i if it maximizes payoff given x-ixi is a dominant strategy if it maximizes payoff regardless of what others doA joint strategy x is an equilibrium if each agent’s strategy is simultaneously a best-response to everyone else’s strategy, i.e. no incentive to deviate. Nash equilibrium is the main one, but there are others (e.g. correlated equilibrium)A Nash equilibrium always exists, but may be exponentially many of them, and very hard to computeequilibrium coordination (players agree on which eqm to choose) is a big problem
85 What about imperfect information games? Nash eqm. requires full observability of all game info. For imperfect info. games (e.g. each player has private info), corresponding concept is Bayes-Nash equilibrium (Nash plus Bayesian inference over hidden information). Even more intractable than regular Nash.
86 Pros and Cons of game theory Game theory provides a basic conceptual/theoretical framework for thinking about multi-agent learning.Game theory is appropriate provided that:Game is stationary and fully specified; XEnough computer power to compute equilibrium; XCan assume other agents are also game theorists; XCan solve equilibrium coordination problem XAbove conditions rarely hold in real applicationsMulti-agent learning is not only a fascinating problem, it may be the only viable option.
87 Real-Life vs. Game Theory games NFL playoffsWorld Series of PokerWorld of WarcraftBuying a houseSalary negotiationsCompetitive pricing:Best Buy vs. Circuit CityAirline fare warsOPEC production cutsNASDAQ, NYSE, …FCC spectrum auctionsMatching PenniesRock-Paper-ScissorsPrisoners’ DilemmaBattle-of-the-SexesChickenUltimatumRock-Paper-Scissor is a real life kids game!I would say that hose real life games could all be potentially analyzed by game-theory but that they are typically very complex, and have many external influences so that in game-theory we often abstract, focus on the essential, and that in this course we shall look that very simple abstract games that can have important implications for real life situations.
88 Assumptions in Normal-Form Games Game specification is fully known; actions and payoffs are fully observable by all playersPlayers act “simultaneously”, i.e. without observing actions of others (not scalable!)Assume no communication between players, or it doesn’t affect play (communication is “cheap talk”)Basic analysis assumes the game is only played once (called one-shot)A coo
89 Presentation of Rock Paper Scissors Payoffs in a Bimatrix This is a zero-sum game since for each pair of joint actions, the players’ payoffs add up to zero.This is a symmetric game: invariant under swapping of player labelsThis game has a unique mixed strategy Nash equilibrium: both players play uniform random strategies: prob(R,P,S)=(1/3,1/3,1/3)Column playerRPS-1+1Row player
90 Prisoners’ Dilemma Game Confess (Defect)Hold out (Cooperate)Prisoner 1Confess(Defect)-8-10Hold out(Cooperate)-1
91 Prisoners’ Dilemma Game Confess (Defect)Hold out (Cooperate)Prisoner 1Confess(Defect)-8-10Hold out(Cooperate)-1Whatever Prisoner 2 does, the best that Prisoner 1 can do is Confess
92 Prisoners’ Dilemma Game Confess (Defect)Hold out (Cooperate)Prisoner 1Confess(Defect)-8-10Hold out(Cooperate)-1Whatever Prisoner 1 does, the best that Prisoner 2 can do is Confess.
93 Prisoners’ Dilemma Game Confess (Defect)Hold out (Cooperate)Prisoner 1Confess(Defect)-8-10Hold out(Cooperate)-1A strategy is a dominant strategy if it is a player’s strictly best response to any strategies the other players might pick.A dominant strategy equilibrium is a strategy combination consisting of each players dominant strategy.Each player has a dominant strategy to Confess.The dominant strategy equilibrium is (Confess,Confess)
94 Prisoners’ Dilemma Game Confess (Defect)Hold out (Cooperate)Prisoner 1Confess(Defect)-8-10Hold out(Cooperate)-1The payoff in the dominant strategy equilibrium (-8,-8) is worse for both players than (-1,-1), the payoff in the case that both players hold out. Thus, the Prisoners’ Dilemma Game is a game of social conflict.Opportunity for multi-agent learning: by learning during repeated play, the Pareto optimal solution (-1,-1) can emerge as a result of learning (also can arise in evolutionary game theory).
95 Battle of the SexesBobPrize FightBalletAlicePrize fight21-1-5
96 Battle of the Sexes Bob Prize Fight Ballet Alice Prize fight 2 1 -1 -5 This game hasno (iterated) dominant strategy equilibrium
97 Battle of the Sexes Bob Prize Fight Ballet Alice Prize fight 2 1 -1 -5 This game hasno (iterated) dominant strategy equilibrium
98 Battle of the Sexes Bob Prize Fight Ballet Alice Prize fight 2 1 -1 -5 This game hasno (iterated) dominant strategy equilibriumtwo Nash equilibria (Prize Fight, Prize Fight) and (Ballet, Ballet)
99 Battle of the Sexes Bob Prize Fight Ballet Alice Prize fight 2 1 -1 -5 This game has two Nash equilibriaHow can these two players coordinate ?
100 Multiagent Q-learning desiderata “performs well” vs. arbitrarily adapting other agentsbest-response probably impossibleDoesn’t need correct model of other agents’ learning algorithmsBut modeling is fair gameDoesn’t need to know other agents’ payoffsEstimate other agents’ strategies from observationdo not assume game-theoretic playNo assumption of stationary outcome: population may never reach eqm, agents may never stop adaptingSelf-play: convergence to repeated Nash would be nice but not necessary. (unreasonable to seek convergence to a one-shot Nash)
101 Naïve Approaches to Multi-Agent Learning Basic idea: agent adapts, ignoring non-stationarity of other agents’ strategies1. Evolutionary game theory: “Replicator Dynamics” models: large population of agents using different strategies, fittest agents breed more copies.Let x= population strategy vector, and xk = fraction of population playing strategy k. Growth rate then:Above eqn also derived from an “imitation” modelNE are fixed points of above equation, but not necessarily attractors (unstable or neutral stable)
102 Many possible dynamic behaviors... limit cycles attractors unstable f.p.Also saddle points, chaotic orbits, ...
104 More Naïve Approaches… 2. Iterated Gradient Ascent: (Singh, Kearns and Mansour): Again does a myopic adaptation to other players’ current strategy.Coupled system of linear equations: u is linear in xi and x-iAnalysis for two-player, two-action games: either converges to a Nash fixed point on the boundary (at least one pure strategy), or get limit cycles
105 Further Naïve Approaches… 3. Dumb Single-Agent Learning: Use a single-agent algorithm in a multi-agent problem & hope that it worksNo-regret learning by pricebots (Greenwald & Kephart)Simultaneous Q-learning by pricebots (Tesauro & Kephart)In many cases, this actually works: learners converge either exactly or approximately to self-consistent optimal strategiesNaïve approaches are “rational” i.e. they converge to a best response against a stationary opponentbut they generally don’t converge to Nash equilibrium
106 A Fancier Approach4. No-regret learning: (Hart & Mas-Colell, Freund & Schapire, many others): Define regret for playing a sequence si instead of constant action aj for t time steps:Then choose next action with probability proportional to:prob (action j) ~This has a worst-case guarantee that asymptotic regret per time step 0, i.e., will be as good as best (constant) action choice
107 “Sophisticated” approaches Takes into account the possibility that other agents’ strategies might change.4. Equilibrium Q-learners:Minimax-Q (Littman): converges to Nash equilibrium for two-player zero-sum stochastic gamesFriendOrFoe-Q (Littman): convergent algorithm for games where every other player can be identified as “friend” (same payoffs as me) or “foe” (payoffs are zero-sum)These algorithms converge to Nash equilibrium but aren’t “rational” since they don’t best-respond to a fixed opponent
108 More sophisticated approaches... 5. Varying learning ratesWoLF: “Win or Learn Fast” (Bowling): agent reduces its learning rate when performing well, and increases when doing badly. Improves convergence of IGA and policy hill-climbingGIGA-WoLF (Bowling): Combines the IGA algorithm with WoLF idea. Provably convergent + no-regret.
109 More sophisticated approaches... 6. “Strategic Teaching:” recognizes that other players’ strategy are adaptive“A strategic teacher may play a strategy which is not myopically optimal (such as cooperating in Prisoner’s Dilemma) in the hope that it induces adaptive players to expect that strategy in the future, which triggers a best-response that benefits the teacher.” (Camerer, Ho and Chong)
110 Theoretical Research Challenges Proper theoretical formulation?“No short-cut” hypothesis: Massive on-line search a la Deep Blue to maximize expected long-term reward(Bayesian) Model and predict behavior of other players, including how they learn based on my actions (beware of infinite model recursion)trial-and-error explorationcontinual Bayesian inference using all evidence over all uncertainties (Boutilier: Bayesian exploration)When can you get away with simpler methods?
111 Real-World Opportunities Multi-agent systems where you can’t do game theory (covers everything :-))Electronic marketplacesMobile networksSelf-managing computer systemsTeams of robotsVideo gamesMilitary/counter-terrorism applications