2. CS 416 Artificial Intelligence Lecture 8 Adversarial Search Chapter 6 Lecture 8 Adversarial Search Chapter 6

3. Chess Match – Spring 2003 Ends in a 3-3 Draw

4. Adversarial Search Problems involving Multiple agentsMultiple agents Competitive environmentsCompetitive environments Agents have conflicting goalsAgents have conflicting goals Also called games Problems involving Multiple agentsMultiple agents Competitive environmentsCompetitive environments Agents have conflicting goalsAgents have conflicting goals Also called games

5. Since the dawn of time? Oldest known written fair-division problem Talmud – Jewish Oral Law dating to first century A Bankruptcy CaseA Bankruptcy Case –A man married three wives and in each marriage contract he promised each of them different amounts of money upon his death:  one of them gets $100  another gets $200  the third gets $300 –When he died, he had fewer than $600 units of money What do you do? Oldest known written fair-division problem Talmud – Jewish Oral Law dating to first century A Bankruptcy CaseA Bankruptcy Case –A man married three wives and in each marriage contract he promised each of them different amounts of money upon his death:  one of them gets $100  another gets $200  the third gets $300 –When he died, he had fewer than $600 units of money What do you do?

6. Bankruptcy law Modern bankruptcy provides shares of the estate proportional to their individual claims, no matter what size of the estateModern bankruptcy provides shares of the estate proportional to their individual claims, no matter what size of the estate –A receives 100/600 * estate_holdings –B receives 200/600 * estate_holdings –C receives 300/600 * estate_holdings Modern bankruptcy provides shares of the estate proportional to their individual claims, no matter what size of the estateModern bankruptcy provides shares of the estate proportional to their individual claims, no matter what size of the estate –A receives 100/600 * estate_holdings –B receives 200/600 * estate_holdings –C receives 300/600 * estate_holdings

7. Bankruptcy law Rabbi Nathan in Mishnah section of Talmud This allocation not understood until recently Rabbi Nathan in Mishnah section of Talmud This allocation not understood until recently Estate→ Claims↓ 100200300 10033.35050 20033.375100 30033.375150

8. Unexplained until 1984 Aumann and Maschler (Israeli Mathematicians) Realistically, when you die, people could come out of the woodwork saying you owe them money. Some could coalesce into deceptive groups. How can we reduce the incentives (rewards) of forming such groups?Realistically, when you die, people could come out of the woodwork saying you owe them money. Some could coalesce into deceptive groups. How can we reduce the incentives (rewards) of forming such groups? Minimize largest dissatisfaction among all possible coalitionsMinimize largest dissatisfaction among all possible coalitions A common fair-division problemA common fair-division problem –http://www.math.gatech.edu/~hill/publications/cv.dir/madevice.pdf Aumann and Maschler (Israeli Mathematicians) Realistically, when you die, people could come out of the woodwork saying you owe them money. Some could coalesce into deceptive groups. How can we reduce the incentives (rewards) of forming such groups?Realistically, when you die, people could come out of the woodwork saying you owe them money. Some could coalesce into deceptive groups. How can we reduce the incentives (rewards) of forming such groups? Minimize largest dissatisfaction among all possible coalitionsMinimize largest dissatisfaction among all possible coalitions A common fair-division problemA common fair-division problem –http://www.math.gatech.edu/~hill/publications/cv.dir/madevice.pdf

9. Garment Principle Two people claim a garment worth $100 One claims the entire garment belongs to himOne claims the entire garment belongs to him The other claims half the garment is hisThe other claims half the garment is his The one claiming the full garment gets $75 The one claiming half gets $25 Why? Two people claim a garment worth $100 One claims the entire garment belongs to himOne claims the entire garment belongs to him The other claims half the garment is hisThe other claims half the garment is his The one claiming the full garment gets $75 The one claiming half gets $25 Why?

10. Minimizing maximum dissatisfaction The one who wants the entire garment cedes nothing to the other and thus wants $100.The one who wants the entire garment cedes nothing to the other and thus wants $100. The one who wants half the garment would be perfectly happy to cede $50 to the other.The one who wants half the garment would be perfectly happy to cede $50 to the other. –But a split of 50/50 would make one person unhappy and the other perfectly happy  How to make them equally unhappy? The one who wants the entire garment cedes nothing to the other and thus wants $100.The one who wants the entire garment cedes nothing to the other and thus wants $100. The one who wants half the garment would be perfectly happy to cede $50 to the other.The one who wants half the garment would be perfectly happy to cede $50 to the other. –But a split of 50/50 would make one person unhappy and the other perfectly happy  How to make them equally unhappy?

11. A $100 Garment Person 1 Person 2 Requested Amount 10050 Ceded from competitor 500 Split what remains 2525 Sum of ceded and split 7525

12. Game Theory Studied by mathematicians, economists, finance In AI we limit games to: deterministicdeterministic turn-takingturn-taking two-playertwo-player zero-sumzero-sum perfect informationperfect information Studied by mathematicians, economists, finance In AI we limit games to: deterministicdeterministic turn-takingturn-taking two-playertwo-player zero-sumzero-sum perfect informationperfect information

13. Games “Shall we play a game?” Let’s play tic-tac-toe “Shall we play a game?” Let’s play tic-tac-toe

14. Tic-Tac-Toe game tree MAX’s first move MIN’s first move Each layer is a ply

15. What data do we need to play? Initial State How does the game start?How does the game start? Successor Function A list of legal (move, state) pairs for each stateA list of legal (move, state) pairs for each state Terminal Test Determines when game is overDetermines when game is over Utility Function Provides numeric value for all terminal statesProvides numeric value for all terminal states Initial State How does the game start?How does the game start? Successor Function A list of legal (move, state) pairs for each stateA list of legal (move, state) pairs for each state Terminal Test Determines when game is overDetermines when game is over Utility Function Provides numeric value for all terminal statesProvides numeric value for all terminal states

16. Minimax strategy Optimal Strategy Leads to outcomes at least as good as any other strategy when playing an infallible opponentLeads to outcomes at least as good as any other strategy when playing an infallible opponent Pick the option that minimizes the maximum damage your opponent can doPick the option that minimizes the maximum damage your opponent can do –minimize the worst-case outcome –because your skillful opponent will certainly find the most damaging move Optimal Strategy Leads to outcomes at least as good as any other strategy when playing an infallible opponentLeads to outcomes at least as good as any other strategy when playing an infallible opponent Pick the option that minimizes the maximum damage your opponent can doPick the option that minimizes the maximum damage your opponent can do –minimize the worst-case outcome –because your skillful opponent will certainly find the most damaging move

17. Minimax Algorithm MinimaxValue(n) =MinimaxValue(n) = Utility (n) if n is a terminal state max MinimaxValue(s) of all successors, s if n is a MAX node min MinimaxValue(s) of all successors, s if n is a MIN node Algorithm MinimaxValue(n) =MinimaxValue(n) = Utility (n) if n is a terminal state max MinimaxValue(s) of all successors, s if n is a MAX node min MinimaxValue(s) of all successors, s if n is a MIN node This is optimal strategy assuming both players play optimally from there until end of game

18. A two-ply example MIN considers minimizing how much it loses…

19. MAX considers minimizing how much it loses… A two-ply example

20. Minimax Algorithm We wish to identify minimax decision at the root Recursive evaluation of all nodes in game treeRecursive evaluation of all nodes in game tree Time complexity = O (b m )Time complexity = O (b m ) We wish to identify minimax decision at the root Recursive evaluation of all nodes in game treeRecursive evaluation of all nodes in game tree Time complexity = O (b m )Time complexity = O (b m )

21. Feasibility of minimax? How about a nice game of chess? Avg branching = 35 and avg # moves = 50 for each playerAvg branching = 35 and avg # moves = 50 for each player –O(35 100 ) time complexity = 10 154 nodes  10 40 distinct nodes Minimax is impractical if directly applied to chess How about a nice game of chess? Avg branching = 35 and avg # moves = 50 for each playerAvg branching = 35 and avg # moves = 50 for each player –O(35 100 ) time complexity = 10 154 nodes  10 40 distinct nodes Minimax is impractical if directly applied to chess

22. Pruning minimax tree Are there times when you know you need not explore a particular move? When the move is poor?When the move is poor? Poor compared to what?Poor compared to what? Poor compared to what you have explored so farPoor compared to what you have explored so far Are there times when you know you need not explore a particular move? When the move is poor?When the move is poor? Poor compared to what?Poor compared to what? Poor compared to what you have explored so farPoor compared to what you have explored so far

23. Alpha-beta pruning  –the value of the best (highest) choice so far in search of MAX  –the value of the best (lowest) choice so far in search of MIN Order of considering successors mattersOrder of considering successors matters –If possible, consider best successors first  –the value of the best (highest) choice so far in search of MAX  –the value of the best (lowest) choice so far in search of MIN Order of considering successors mattersOrder of considering successors matters –If possible, consider best successors first

24. Alpha-beta pruning MIN knows it will at least score a 3. MAX worries that –inf is still possible MAX knows that 3 is worst case for this node. MAX knows that it can accomplish a score of at least 3. Discovery could find a higher value MIN knows player MAX has an option of going to node B with a min payoff of 3. MAX will never take action C and culling is possible.

25. Alpha-beta pruning Without pruningWithout pruning –O(b d ) nodes to explore With a good heuristic pruner (consider part (f) of figure)With a good heuristic pruner (consider part (f) of figure) – O(b d/2 )  Chess can drop from O(35 100 ) to O(6 100 ) With a random heuristicWith a random heuristic –O(b 3d/4 ) Without pruningWithout pruning –O(b d ) nodes to explore With a good heuristic pruner (consider part (f) of figure)With a good heuristic pruner (consider part (f) of figure) – O(b d/2 )  Chess can drop from O(35 100 ) to O(6 100 ) With a random heuristicWith a random heuristic –O(b 3d/4 )

26. Real-time decisions What if you don’t have enough time to explore entire search tree? We cannot search all the way down to terminal state for all decision sequencesWe cannot search all the way down to terminal state for all decision sequences Use a heuristic to approximate (guess) eventual terminal stateUse a heuristic to approximate (guess) eventual terminal state What if you don’t have enough time to explore entire search tree? We cannot search all the way down to terminal state for all decision sequencesWe cannot search all the way down to terminal state for all decision sequences Use a heuristic to approximate (guess) eventual terminal stateUse a heuristic to approximate (guess) eventual terminal state

27. Evaluation Function (Estimator) The heuristic that estimates expected utility Cannot take too long (otherwise recurse to get answer)Cannot take too long (otherwise recurse to get answer) It should preserve the ordering among terminal statesIt should preserve the ordering among terminal states –otherwise it can cause bad decision making Define features of game state that assist in evaluationDefine features of game state that assist in evaluation –what are features of chess? The heuristic that estimates expected utility Cannot take too long (otherwise recurse to get answer)Cannot take too long (otherwise recurse to get answer) It should preserve the ordering among terminal statesIt should preserve the ordering among terminal states –otherwise it can cause bad decision making Define features of game state that assist in evaluationDefine features of game state that assist in evaluation –what are features of chess?

28. Truncating minimax search When do you recurse or use evaluation function? Cutoff-Test (state, depth) returns 1 or 0Cutoff-Test (state, depth) returns 1 or 0 –When 1 is returned, use evaluation function When do you recurse or use evaluation function? Cutoff-Test (state, depth) returns 1 or 0Cutoff-Test (state, depth) returns 1 or 0 –When 1 is returned, use evaluation function

29. When do you cut off? When exploring beyond a certain depthWhen exploring beyond a certain depth –The horizon effect When exploring beyond a certain depthWhen exploring beyond a certain depth –The horizon effect

30. When do you cut off? Cutoff if state is stable or quiescient (more predictable)Cutoff if state is stable or quiescient (more predictable)

31. When do you cut off? Cutoff moves you know are bad (forward pruning)

32. Benefits of truncation Comparing Chess Number of plys that can considered per unit time Using minimax5 plyUsing minimax5 ply Average Human6-8 plyAverage Human6-8 ply Using alpha-beta10 plyUsing alpha-beta10 ply Intelligent pruning14 plyIntelligent pruning14 ply Comparing Chess Number of plys that can considered per unit time Using minimax5 plyUsing minimax5 ply Average Human6-8 plyAverage Human6-8 ply Using alpha-beta10 plyUsing alpha-beta10 ply Intelligent pruning14 plyIntelligent pruning14 ply

33. Games with chance How to include chance in game tree? Add chance nodesAdd chance nodes How to include chance in game tree? Add chance nodesAdd chance nodes

34. Expectiminimax Expectiminimax (n) = utility(n) if n is a terminal stateutility(n) if n is a terminal state if n is a MAX node if n is a MAX node if n is a MIN node if n is a MIN node if n is a chance node if n is a chance node Expectiminimax (n) = utility(n) if n is a terminal stateutility(n) if n is a terminal state if n is a MAX node if n is a MAX node if n is a MIN node if n is a MIN node if n is a chance node if n is a chance node

35. Pruning Can we prune search in games of chance? Think about alpha-beta pruningThink about alpha-beta pruning –With alpha-beat, we don’t explore nodes that we know are worse than what we know we can accomplish –With randomness, we never really what we will accomplish  chance node values are average of successors Thus it is hard to use alpha-beta Can we prune search in games of chance? Think about alpha-beta pruningThink about alpha-beta pruning –With alpha-beat, we don’t explore nodes that we know are worse than what we know we can accomplish –With randomness, we never really what we will accomplish  chance node values are average of successors Thus it is hard to use alpha-beta

36. History of Games Chess, Deep Blue IBM: 30 RS/6000 comps with 480 custom VLSI chess chipsIBM: 30 RS/6000 comps with 480 custom VLSI chess chips Deep Thought design came from Campbell and Hsu at CMUDeep Thought design came from Campbell and Hsu at CMU 126 mil nodes / s126 mil nodes / s 30 bil positions per move30 bil positions per move routine reaching depth of 14routine reaching depth of 14 iterative deepening alpha-beta searchiterative deepening alpha-beta search Chess, Deep Blue IBM: 30 RS/6000 comps with 480 custom VLSI chess chipsIBM: 30 RS/6000 comps with 480 custom VLSI chess chips Deep Thought design came from Campbell and Hsu at CMUDeep Thought design came from Campbell and Hsu at CMU 126 mil nodes / s126 mil nodes / s 30 bil positions per move30 bil positions per move routine reaching depth of 14routine reaching depth of 14 iterative deepening alpha-beta searchiterative deepening alpha-beta search

37. Deep Blue evaluation function had 8000 featuresevaluation function had 8000 features 4000 opening moves in memory4000 opening moves in memory 700,000 grandmaster games from which recommendations extracted700,000 grandmaster games from which recommendations extracted many endgames solved for all five piece combosmany endgames solved for all five piece combos evaluation function had 8000 featuresevaluation function had 8000 features 4000 opening moves in memory4000 opening moves in memory 700,000 grandmaster games from which recommendations extracted700,000 grandmaster games from which recommendations extracted many endgames solved for all five piece combosmany endgames solved for all five piece combos

38. Checkers Arthur Samuel of IBM, 1952 program learned by playing against itselfprogram learned by playing against itself beat a human in 1962 (but human clearly made error)beat a human in 1962 (but human clearly made error) 19 KB of memory19 KB of memory 0.000001 Ghz processor0.000001 Ghz processor Arthur Samuel of IBM, 1952 program learned by playing against itselfprogram learned by playing against itself beat a human in 1962 (but human clearly made error)beat a human in 1962 (but human clearly made error) 19 KB of memory19 KB of memory 0.000001 Ghz processor0.000001 Ghz processor

39. Checkers Chinook, Jonathan Schaeffer, 1990 Alpha-beta search on regular PCsAlpha-beta search on regular PCs database of all 444 billion endgame positions with 8 piecesdatabase of all 444 billion endgame positions with 8 pieces Played against Marion TinsleyPlayed against Marion Tinsley –world champion for over 40 years –lost only 3 games in 40 years –Chinook won two games, but lost match Rematch with Tinsley was incomplete for health reasonsRematch with Tinsley was incomplete for health reasons –Chinook became world champion Chinook, Jonathan Schaeffer, 1990 Alpha-beta search on regular PCsAlpha-beta search on regular PCs database of all 444 billion endgame positions with 8 piecesdatabase of all 444 billion endgame positions with 8 pieces Played against Marion TinsleyPlayed against Marion Tinsley –world champion for over 40 years –lost only 3 games in 40 years –Chinook won two games, but lost match Rematch with Tinsley was incomplete for health reasonsRematch with Tinsley was incomplete for health reasons –Chinook became world champion

40. Othello Smaller search space (5 to 15 legal moves) Humans are no match for computers Smaller search space (5 to 15 legal moves) Humans are no match for computers

41. Backgammon Garry Tesauro, TD-Gammon, 1992 Reliably ranked in top-three players of worldReliably ranked in top-three players of world Learned to play through playing against itselfLearned to play through playing against itself –Reinforcement Learning Garry Tesauro, TD-Gammon, 1992 Reliably ranked in top-three players of worldReliably ranked in top-three players of world Learned to play through playing against itselfLearned to play through playing against itself –Reinforcement Learning

42. Discussion How reasonable is minimax? perfectly performing opponentperfectly performing opponent perfect knowledge of leaf node evaluationsperfect knowledge of leaf node evaluations strong assumptionsstrong assumptions How reasonable is minimax? perfectly performing opponentperfectly performing opponent perfect knowledge of leaf node evaluationsperfect knowledge of leaf node evaluations strong assumptionsstrong assumptions

43. Building alpha-beta tree Can we restrict the size of game tree? alpha-beta will blindly explore tree in depth-first fashion even if only one move is possible from rootalpha-beta will blindly explore tree in depth-first fashion even if only one move is possible from root even if multiple moves are possible, can we use a quick search to eliminate some entirely?even if multiple moves are possible, can we use a quick search to eliminate some entirely? utility vs. time tradeoff to decide when to explore new branches or to stay with what you haveutility vs. time tradeoff to decide when to explore new branches or to stay with what you have Can we restrict the size of game tree? alpha-beta will blindly explore tree in depth-first fashion even if only one move is possible from rootalpha-beta will blindly explore tree in depth-first fashion even if only one move is possible from root even if multiple moves are possible, can we use a quick search to eliminate some entirely?even if multiple moves are possible, can we use a quick search to eliminate some entirely? utility vs. time tradeoff to decide when to explore new branches or to stay with what you haveutility vs. time tradeoff to decide when to explore new branches or to stay with what you have

44. Metareasoning Reasoning about reasoning alpha-beta is one examplealpha-beta is one example –think before you think –think about utility of thinking about something before you think about it –don’t think about choices you don’t have to think about Reasoning about reasoning alpha-beta is one examplealpha-beta is one example –think before you think –think about utility of thinking about something before you think about it –don’t think about choices you don’t have to think about

45. Goal-directed reasoning / planning Minimax starts from root and moves forward using combinatorial search What about starting at goal and working backward We talked about difficulty of identifying goal states in bidirectional searchWe talked about difficulty of identifying goal states in bidirectional search We do not know how to combine the two in practical wayWe do not know how to combine the two in practical way Minimax starts from root and moves forward using combinatorial search What about starting at goal and working backward We talked about difficulty of identifying goal states in bidirectional searchWe talked about difficulty of identifying goal states in bidirectional search We do not know how to combine the two in practical wayWe do not know how to combine the two in practical way