1. Artificial Intelligence in Game Design Board Games and the MinMax Algorithm

2. Board Games Examples: –Chess, checkers, tic tac toe, backgammon, etc. Characteristics: –Alternating moves: player  AI  player  AI  … –Limited (but usually not small) set of possible moves Example: legal moves in chess –Set of winning and losing board configurations Examples: checkmate in chess, all pieces gone in checkers Goal of AI: –Choose next move so that will “likely” lead to victory Key problem: What exactly does this mean?

3. Planning Against an Opponent Idea: Create sequence of steps that lead from initial board state to winning state Problem: Unlike normal planning, AI does not get to choose every step of path –Player gets to choose every other step in plan –Will choose steps that defeat the AI X XOXO X XO XO XO XO X XXOXO X XO X O

4. Tree Representation Can represent possible moves as tree –Node = board state –Branch = possible moves from that state –Opponent controls every other branching of tree! Example: Tic Tac Toe –Simplified to remove duplicate branches for reflection, rotation Root node = initial board state X X X Possible X moves XOXOX O X O X O XOX O XOX O X O X O X O Possible O moves

5. Choosing Next Move Simple idea: Choose next move so guaranteed win no matter which moves opponent makes –No real game is this simple! Example: I winI lose I winI lose I win AB CDE F G H

6. Choosing Next Move Reasoning: If choose move A, then opponent will choose move D and I lose. If choose move B, I win, since: –If opponent chooses move F, I win –If opponent chooses move E, I choose move G and win Therefore, choose move B –Guaranteed win regardless of opponent moves

7. Lookahead Problem Main Problem: No nontrivial game has tree which can be completely explored –m possible moves each turn –n turns until game ends –m n nodes in game tree Exponential growth –Example: Chess ~20 moves per turn ~50 turns per game ~20 50 possible moves –Close to number of particles in universe!

8. Heuristic Evaluation of Boards How many levels can be explored in tree? Can process p nodes per second Have maximum of s seconds per move –Can process ps nodes –m n nodes looking ahead n moves –Can explore n = log m (ps) moves ahead What if game not over at deepest lookahead level? –Should AI try to force this branch or avoid it? No time to explore moves past this point

9. Heuristic Evaluation of Boards Key idea: Create heuristic measure of how “good” a board configuration is H(board) –Positive or negative number: Higher = better, lower = worse –Win = MAXINT, Loss = - MAXINT Goal: Choose current move that guarantees reaching board position with highest possible heuristic measure MAXINT9317-100- MAXINT

10. Heuristic Evaluation of Boards TicTacToe Example (with AI playing X): H(board) = 2 × # of possible rows/columns/diagonals where X could win in one move + + 1 × # of possible rows/columns/diagonals where X could win in two moves + - 2 × # of possible rows/columns/diagonals where O could win in one move + - 1 × # of possible rows/columns/diagonals where O could win in two moves Example: X OX XO H = 2 + 1 + 1 - 1 = 3

11. MINMAX Algorithm Explore possible moves from current board position –Continue until reach maximum lookahead level –Forms tree of possible moves and countermoves Apply heuristic measure to all boards at leafs of tree Work from leafs to root of tree –Measure of each node based on measure of it child nodes Choose current move that results in highest heuristic value AI makes move Player makes move (may not be one expected, but can’t be worse)

12. MINMAX Algorithm AI’s turn = MAX level –Choose move that gives highest heuristic AI should choose move that guarantees best result! –Value of board position at that level = maximum of its children –Example: TicTacToe with AI playing X X OX O X OX XO XX OX O XX OX O X OXX O X OX XO 33341 Best move Bad move, but not relevant Will choose move with highest heuristic, so value of reaching this board = 4 4

13. MINMAX Algorithm Player’s turn = MIN level –Assume they choose move which is best for them Assume good for player = bad for AI Assume move that gives lowest heuristic –Value of board position at that level = minimum of its children X X OXO -2 Worst move for AI Bad moves for player, but can’t assume they will do this Player will choose move with lowest heuristic, so value of reaching this board = -2 -2 X OX OXO X OX OXO OX X OXO OX X OXO

14. Simple MinMax Example AI move 3 levels of lookahead Heuristic Measures No time to explore moves past this point 8 -2 3 5 -5 -1 4 12 AI move player move

15. Simple MinMax Example AI move 3 levels of lookahead Heuristic Measures No time to explore moves past this point 8 -2 3 5 -5 -1 4 12 AI move player move Max(8, -2) = 8Max(-5, -1) = -1Max(3, 5) = 5 Max(4, 12) = 12

16. Simple MinMax Example AI move 3 levels of lookahead Heuristic Measures No time to explore moves past this point 8 -2 3 5 -5 -1 4 12 AI move player move Max(8, -2) = 8Max(-5, -1) = -1Max(3, 5) = 5 Max(4, 12) = 12 Min(8, 5) = 5Min(-1, 12) = -1

17. Simple MinMax Example Heuristic Measures No time to explore moves past this point 8 -2 3 5 -5 -1 4 12 AI move player move Max(8, -2) = 8Max(-5, -1) = -1Max(3, 5) = 5 Max(4, 12) = 12 Min(8, 5) = 5Min(-1, 12) = -1 AI move Max(5, -1) = 5 Best possible outcome = 5 Follow this branch

18. MinMax Example Max(-1, 1, -2) = 1 X X X Min(1, -1, 0, 1, 0) = -1 XOXOX O X O X O XOX O XOX O X O X O X O Heuristic Measures No time to explore moves past this point 1 -1 0 1 0 1 2 -1 0 -2 -1 0 Min(-1, 0, -2, -1, 0) = -2 Min(1, 2) = 1 Choose this move Assume player will then do this AI playing X Maximum lookahead of 2 moves Initial move choice for AI

19. Efficiency in MinMax Goal: Speed up MinMax algorithm –Negamaxing –Depth-first search –Alpha-beta pruning –Move ordering Idea: Faster evaluation  Able to explore more levels in game tree  Better decision making –Can speed up process by factor of 10 – 100

20. Negamaxing MinMax uses different function at each level –AI move: maximum –Player move: minimum –More efficient if used single function Value(node n) = maximum(–Value(child nodes of n)) –Maximizing the negative has same effect as minimizing –Signs changed at each level –Also need to invert heuristic function if player’s move at bottom of tree

21. Negamaxing Example: Heuristic Measures (inverted since player’s move) - 8 2 -3 -5 5 1 -4 -12 AI move player move Max(8, -2) = 8Max(-5, -1) = -1Max(3, 5) = 5 Max(4, 12) = 12 Max(-8, -5) = -5Max(1, -12) = 1 AI move Max(5, -1) = 5

22. Depth-first Search Don’t need to store entire evaluation tree in memory –Less memory = faster execution Just need: –Current branch being explored –Best score so far at each node along that branch Recursive algorithm: float DFS(node N) { if N is Leaf return H(N) bestScoreSoFar = -MAXINT for child C of N { float childScore = DFS(C) if childScore > bestScoreSoFar bestScoreSoFar = childScore } return bestScoreSoFar }

23. Depth-first Search Example - 8 2 bestScoreSoFar = 8 bestScoreSoFar = -8 bestScoreSoFar = -MAXINT

24. Depth-first Search Example bestScoreSoFar = 5 bestScoreSoFar = -8 -5 bestScoreSoFar = 5 -3 -5

25. Depth-first Search Example bestScoreSoFar = -1 bestScoreSoFar = 5 5 1 bestScoreSoFar = 1

26. Depth-first Search Example bestScoreSoFar = -4 bestScoreSoFar = 5 bestScoreSoFar = 1 -4 -12

27. Alpha-Beta Pruning Idea: Many branches cannot possibly improve score –Once this is known, stop exploring them –Can typically lead to speedup by factor of 10 Example: BestSoFar = max(-1, 1, -?) = 1 X X X BestSoFar = 1 XO BestSoFar = max (1, -?, -?, -?, -?) BestSoFar = -1 1) Maximum will never get lower than -1 2) So this can never be greater than -1, which means it cannot increase BestSoFar 3) Which means that there is no point in exploring any of these branches

28. Alpha-Beta Pruning In general: Define -α as the value found down a previously explored branch –Can do no worse than α at this point Define β as a value found below the current branch –Can do no better than – β down this branch If α ≥ β, there is no point in exploring this branch any further -α-α BestSoFar = max (α, x) where x ≤ β = α if α ≥ β β BestSoFar = max (-β, …) ≥ -β

29. Move Ordering Problem: Alpha-Beta pruning only works well if best branches explored first –Only know branch is not worth exploring if have already seen a better one Goal: Order branches so moves “most likely” to be good are explored first –This is very difficult! XX X Probably no branches eliminated if explored in this order

30. Move Ordering Idea: Use results of last evaluation to order moves this time –Tree reevaluated each move A player move chosen CB EDGF Tree created in previous move AI move chosen Attempt to “reuse” this part of tree 7 2 4 8 -2-4 4

31. Move Ordering Each board in subtree already has an evaluation –May no longer be completely accurate –Should still be somewhat accurate Use those to determine order in which next evaluation done A CB EDGF 7 2 4 8 -2-4 4 A CB DEGF 4 8 2 7 -2-4 4

32. Games with Random Component Example: Backgammon –Roll dice (random event) –Determine which piece or pieces to move that many steps One piece 11 steps Two pieces 5 and 6 steps Which piece? Possible actions depend on random component

33. Games with Random Component Can treat random event like extra level of game tree Dice throw 12111098765432 Possible moves based on dice throw (8 in this case)

34. MinMax in Random Games Algorithm: Generate tree of possible outcomes up to lookahead limit –Will alternate possible moves and possible outcomes of random event –Start with possible AI moves after random event Point at which AI has to make decision about what to do AI dice throw at current board state Possible AI moves from current board for that dice throw Possible player dice throws Possible player moves based on board and their dice throw Possible player dice throws Possible AI moves from this board for that dice throw

35. MinMax in Random Games Use Negamax to compute values at decision levels –AI and player make best possible move based on their dice throw Weight each node at random event level based on probability of the random event 1) Value of board for each possible move after dice throw of 8 12111098765432 -5 8 7 -18 11 -1 2) Negamax uses this value 3) Probability of dice throw of 8 = 5/36 4) Weight this state = 5/36 * 18 = 2.5

36. MinMax in Random Games 12111098765432 Probability: 1/36 1/18 1/12 1/9 5/36 1/6 5/36 1/9 1/12 1/18 1/36 Negamax value from child nodes: -18 -27 -24 0 18 18 18 27 24 36 36 Weighted value: -0.5 -1.5 -2 0 2.5 3 2.5 3 2 2 1 Total expected value = 12 Compute value of board before random event based on expected value of boards after random event

37. MinMax in Random Games Random component greatly increases size of tree Example: –11 possibilities for dice throw –n moves lookahead –Increases nodes to explore by factor of 11 n Will probably not be able to look ahead more than 2 or 3 moves TD-Gammon –Plays at world championship level –Also uses learning