1. Randomized Strategies and Temporal Difference Learning in Poker Michael Oder April 4, 2002 Advisor: Dr. David Mutchler

2. Overview Perfect vs. Imperfect Information Games Poker as Imperfect Information Game Randomization Neural Nets and Temporal Difference Experiments Conclusions Ideas for Further Study

3. Perfect vs. Imperfect Information World-class AI agents exist for many popular games –Checkers –Chess –Othello These are games of perfect information All relevant information is available to each player Good understanding of imperfect information games would be a breakthrough

4. Poker as an Imperfect Information Game Other players’ hands affect how much will be won or lost. However, each player is not aware of this vital information. Non-deterministic aspects as well

5. Enter Loki One of the most successful computer poker players created Produced at University of Alberta by Jonathan Schaeffer et al Employs randomized strategy –Makes player less predictable –Allows for bluffing

6. Probability Triples At any point in a poker game, player has 3 choices –Bet/Raise –Check/Call –Fold Assign a probability to each possible move Single move is now a probability triple Problem: Associate payoff with hand, betting history, and triple (move selected)

7. Neural Nets One promising way to learn such functions is with a neural network Neural Networks consist of connected neurons Each connection has a weight Input game state, output a prediction of payoff Train by modifying weights Weights are modified by an amount proportional to learning rate

8. Neural Net Example hand history triple P(2) P(1) P(-1) P(-2)

9. Temporal Difference Most common way to train multiple layer neural net is with backpropagation Relies on simple input-output pairs. Problem: need to know correct answer right away in order to train nets Solution: Temporal Difference (TD) learning. TD(λ) algorithm developed by Richard Sutton

10. Temporal Difference (cont’d) Trains responses over the course of a game over many time steps Tries to make each prediction closer to the prediction in the next time step P 1 P 2 P 3 P 4 P 5

11. University of Mauritius Group TD Poker program produced by group supervised by Dr. Mutchler Provides environment for playing poker variants and testing agents

12. Simple Poker Game Experiments were conducted on extremely simple variant of Poker Deck consists of 2, 3, and 4 of Hearts Each player gets one card One round of betting Player with highest card wins the pot Goal: Get the net to produce accurate payoff values as outputs

13. Early Results Started by pitting a neural net player against a random one Results were inconsistant Problem: Innappropriate value for learning rate Too low: Outputs never approach true payoffs Too high: Outputs fluctuate between too high and too low

14. Experiment Set I Conjecture: Learning should occur with very small learning rate over many games Learning Rate = 0.01 Train for 50,000 games Only set to train when card is a 4 First player always bets, second player tested Two Choices –call 80%, fold 20% -> avg. payoff = 1.4 –call 20%, fold 80% -> avg. payoff = -0.4 Want payoffs to settle in on average values

15. Results 3 out of 10 trials came within 0.1 of the correct result for the highest payoff 2 out of 10 trials came within 0.1 of the correct result for the lowest payoff None of the trials came within 0.1 of the correct result for both The results were in the correct order in only half of the trials

16. More Distributions Repeated experiment with six choices instead of two –call 100% -> avg. payoff = 2.0 –call 80%, fold 20% -> avg. payoff = 1.4 –call 60%, fold 40% -> avg. payoff = 0.8 –call 40%, fold 60% -> avg. payoff = 0.2 –call 20% fold 80% -> avg. payoff = -0.4 –fold 100% -> avg. payoff = -1.0 Using more distributions did help the program learn to order value of the distributions correctly All six distributions were ranked correctly 7 out of 10 times (0.14% chance for any one trial)

17. Output Encoding Distributions are ranked correctly, but many output values are still inaccurate. Seems to be largely caused by the encoding of outputs Network has four outputs, each representing probability of a specific payoff This encoding is not expandable, and four outputs must all be correct for good payoff prediction.

18. Relative Payoff Encoding Replace four outputs with single number The number represents the payoff relative to highest payoff possible P = 0.5 + (winnings/total possible) Total possible winnings determined at beginning of game (sum of other players’ holdings) Repeated previous experiments using this encoding

19. Results (Experiment Set 2) Payoff predictions were generally more accurate using this encoding 5 out of 10 trials got exact payoff (0.502) for best distribution choice with six choices available Most trials had very close value for payoff associated with one of the distributions However, no trial was significantly close on multiple probability distributions

20. Observations/Conclusions Neural Net player can learn strategies based on probability Payoff is successfully learned as a function of betting action Consistency is still a problem Trouble learning correct payoffs for more than one distribution

21. Further Study Issues of expandability –Coding for multiple-round history –Can previous learning be extended? Variable learning rate Study distribution choices Sample some bad distribution choices Test against a variety of other players

22. Questions?