1. G AME P LAYING 2

2. T HIS L ECTURE Alpha-beta pruning Games with chance

3. N ONDETERMINISM Uncertainty is caused by the actions of another agent (MIN), who competes with our agent (MAX) MAX’s play MAX cannot tell what move will be played MIN’s play

4. N ONDETERMINISM Uncertainty is caused by the actions of another agent (MIN), who competes with our agent (MAX) MAX’s play MAX must decide what to play for BOTH these outcomes MIN’s play Instead of a single path, the agent must construct an entire plan

5. M INIMAX B ACKUP MIN’s turn MAX’s turn +1 0 MAX’s turn 0 +100 0 +1

6. D EPTH -F IRST M INIMAX A LGORITHM MAX-Value(S) 1. If Terminal?(S) return Result(S) 2. Return max S’  SUCC(S) MIN-Value(S’) MIN-Value(S) 1. If Terminal?(S) return Result(S) 2. Return min S’  SUCC(S) MAX-Value(S’) MINIMAX-Decision(S) Return action leading to state S’  SUCC(S) that maximizes MIN-Value(S’)

7. R EAL -T IME G AME P LAYING WITH E VALUATION F UNCTION e(s): function indicating estimated favorability of a state to MAX Keep track of depth, and add line: If(depth(s) = cutoff) return e(s) After terminal test

8. C AN WE DO BETTER ? Yes ! Much better ! 3  Pruning  -1  3 3 This part of the tree can’t have any effect on the value that will be backed up to the root

9. E XAMPLE

10.  = 2 2 The beta value of a MIN node is an upper bound on the final backed-up value. It can never increase

11. E XAMPLE The beta value of a MIN node is an upper bound on the final backed-up value. It can never increase 1  = 1 2

12. E XAMPLE  = 1 The alpha value of a MAX node is a lower bound on the final backed-up value. It can never decrease 1  = 1 2

13. E XAMPLE  = 1 1  = 1 2  = -1

14. E XAMPLE  = 1 1  = 1 2  = -1 Search can be discontinued below any MIN node whose beta value is less than or equal to the alpha value of one of its MAX ancestors Search can be discontinued below any MIN node whose beta value is less than or equal to the alpha value of one of its MAX ancestors

15. A LPHA -B ETA P RUNING Explore the game tree to depth h in depth-first manner Back up alpha and beta values whenever possible Prune branches that can’t lead to changing the final decision

16. A LPHA -B ETA A LGORITHM Update the alpha/beta value of the parent of a node N when the search below N has been completed or discontinued Discontinue the search below a MAX node N if its alpha value is  the beta value of a MIN ancestor of N Discontinue the search below a MIN node N if its beta value is  the alpha value of a MAX ancestor of N

17. E XAMPLE 05-325-232-3033-501-3501-5532-35 MAX MIN MAX MIN MAX MIN

18. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 MAX MIN MAX MIN MAX MIN

19. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 MAX MIN MAX MIN MAX MIN

20. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 MAX MIN MAX MIN MAX MIN

21. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 MAX MIN MAX MIN MAX MIN

22. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 MAX MIN MAX MIN MAX MIN

23. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 3 3 MAX MIN MAX MIN MAX MIN

24. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 3 3 MAX MIN MAX MIN MAX MIN

25. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 MAX MIN MAX MIN MAX MIN

26. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 5 MAX MIN MAX MIN MAX MIN

27. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2

28. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 MAX MIN MAX MIN MAX MIN

29. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 MAX MIN MAX MIN MAX MIN

30. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 MAX MIN MAX MIN MAX MIN

31. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 0 MAX MIN MAX MIN MAX MIN

32. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 5 0 MAX MIN MAX MIN MAX MIN

33. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 1 1 0 MAX MIN MAX MIN MAX MIN

34. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 1 1 0 MAX MIN MAX MIN MAX MIN

35. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 1 1 0 MAX MIN MAX MIN MAX MIN

36. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 1 1 1 1 0 MAX MIN MAX MIN MAX MIN

37. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 1 1 1 1 -5 0 MAX MIN MAX MIN MAX MIN

38. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 1 1 1 1 -5 0 MAX MIN MAX MIN MAX MIN

39. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 1 1 1 1 -5 0 MAX MIN MAX MIN MAX MIN

40. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 1 1 1 1 -5 0

41. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 1 1 1 1 -5 0 1 MAX MIN MAX MIN MAX MIN

42. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 1 1 1 1 -5 2 2 2 2 1 1 MAX MIN MAX MIN MAX MIN

43. E XAMPLE 05-325-232-3033-501-3501-5532-35 0 0 0 0 3 3 0 2 2 2 2 1 1 1 1 -5 1 2 2 2 2 1 MAX MIN MAX MIN MAX MIN

44. H OW MUCH DO WE GAIN ? Consider these two cases: 3  = 3  =-1 (4) 3  = 3 4  =4

45. H OW MUCH DO WE GAIN ? Assume a game tree of uniform branching factor b Minimax examines O(b h ) nodes, so does alpha-beta in the worst-case The gain for alpha-beta is maximum when: The children of a MAX node are ordered in decreasing backed up values The children of a MIN node are ordered in increasing backed up values Then alpha-beta examines O(b h/2 ) nodes [Knuth and Moore, 1975] But this requires an oracle (if we knew how to order nodes perfectly, we would not need to search the game tree) If nodes are ordered at random, then the average number of nodes examined by alpha-beta is ~O(b 3h/4 )

46. A LPHA -B ETA I MPLEMENTATION MAX-Value(S, ,  ) 1. If Terminal?(S) return Result(S) 2. For all S’  SUCC(S) 3.   max( ,MIN-Value(S’, ,  )) 4. If   , then return  5. Return  MIN-Value(S, ,  ) 1. If Terminal?(S) return Result(S) 2. For all S’  SUCC(S) 3.   min( ,MAX-Value(S’, ,  )) 4. If   , then return  5. Return  Alpha-Beta-Decision(S) Return action leading to state S’  SUCC(S) that maximizes MIN-Value(S’,- ,+  )

47. H EURISTIC O RDERING OF N ODES Order the nodes below the root according to the values backed-up at the previous iteration

48. O THER I MPROVEMENTS Adaptive horizon + iterative deepening Extended search: Retain k>1 best paths, instead of just one, and extend the tree at greater depth below their leaf nodes (to help dealing with the “horizon effect”) Singular extension: If a move is obviously better than the others in a node at horizon h, then expand this node along this move Use transposition tables to deal with repeated states Null-move search

49. G AMES OF C HANCE

50. Dice games: backgammon, Yahtzee, craps, … Card games: poker, blackjack, … Is there a fundamental difference between the nondeterminism in chess-playing vs. the nondeterminism in a dice roll?

51. MAX CHANCE MIN CHANCE MAX

52. E XPECTED V ALUES The utility of a MAX/MIN node in the game tree is the max/min of the utility values of its successors The expected utility of a CHANCE node in the game tree is the average of the utility values of its successors ExpectedValue(s) =  s’  SUCC(s) ExpectedValue(s’) P(s’) MinimaxValue(s) = max s’  SUCC(s) MinimaxValue(s’) Compare to MinimaxValue(s) = min s’  SUCC(s) MinimaxValue(s’) CHANCE nodes MAX nodes MIN nodes

53. A DVERSARIAL G AMES OF C HANCE E.g., Backgammon MAX nodes, MIN nodes, CHANCE nodes Expectiminimax search Backup step: MAX = maximum of children CHANCE = average of children MIN = minimum of children CHANCE = average of children 4 levels of the game tree separate each of MAX’s turns! Evaluation function? Pruning?

54. G ENERALIZING M INIMAX V ALUES Utilities can be continuous numerical values, rather than +1,0,-1 Allows maximizing the amount of “points” (e.g., $) rewarded instead of just achieving a win Rewards associated with terminal states Costs can be associated with certain decisions at non-terminal states (e.g., placing a bet)

55. R OULETTE “Game tree” only has depth 2 Place a bet Observe the roulette wheel No bet Bet: Red, $5 RedNot red Chance node 18/3820/38Probabilities +100

56. C HANCE N ODE B ACKUP Expected value: For k children, with backed up values v 1,…,v k Chance node value = p 1 * v 1 + p 2 * v 2 + … + p k * v k RedNot red Chance node 18/3820/38Probabilities +100 Bet: Red, $5 Value: 18/38 * 10 + 20/38 * 0 = 4.74

57. MAX/C HANCE N ODES RedNot red 18/3820/38 +100 Bet: Red, $5 4.74 MAX Chance Bet: 17, $5 3.95 = 150/38 17Not 17 1/3837/38 +1500 Max should pick the action leading to the node with the highest value

58. A SLIGHTLY MORE COMPLEX EXAMPLE Two fair coins Pay $1 to start, at which point both are flipped Can flip up to two coins again, at a cost of $1 each Payout: $5 for HH, $1 for HT or TH, $0 for TT HT HH 1/2 TTHT 1/2 HT Flip TFlip H Done HTHHTTHT 1/2 Flip T Flip H HT Done TT Done Flip T HTTT 1/2

59. A SLIGHTLY MORE COMPLEX EXAMPLE Two fair coins Pay $1 to start, at which point both are flipped Can flip up to two coins again, at a cost of $1 each Payout: $5 for HH, $1 for HT or TH, $0 for TT HT HH 1/2 TTHT 1/2 HT Flip TFlip H Done HTHHTTHT 1/2 Flip T Flip H HT Done TT Done Flip T HTTT 1/2 1 5-1=4 1-1=0 1-2=-15-2=3-2-2

60. A SLIGHTLY MORE COMPLEX EXAMPLE Two fair coins Pay $1 to start, at which point both are flipped Can flip up to two coins again, at a cost of $1 each Payout: $5 for HH, $1 for HT or TH, $0 for TT HT HH 1/2 TTHT 1/2 HT Flip TFlip H Done HTHHTTHT 1/2 Flip T Flip H HT Done TT Done Flip T HTTT 1/2 1 4 0 3 -2-2 1 -3/2

61. A SLIGHTLY MORE COMPLEX EXAMPLE Two fair coins Pay $1 to start, at which point both are flipped Can flip up to two coins again, at a cost of $1 each Payout: $5 for HH, $1 for HT or TH, $0 for TT HT HH 1/2 TTHT 1/2 HT Flip TFlip H Done HTHHTTHT 1/2 Flip T Flip H HT Done TT Done Flip T HTTT 1/2 1 -2-2 1 -3/2 1 4 0 3

62. A SLIGHTLY MORE COMPLEX EXAMPLE Two fair coins Pay $1 to start, at which point both are flipped Can flip up to two coins again, at a cost of $1 each Payout: $5 for HH, $1 for HT or TH, $0 for TT HT HH 1/2 TTHT 1/2 HT Flip TFlip H Done HTHHTTHT 1/2 Flip T Flip H HT Done TT Done Flip T HTTT 1/2 1 -2-2 2 -3/2 2 3 4 0 3

63. A SLIGHTLY MORE COMPLEX EXAMPLE Two fair coins Pay $1 to start, at which point both are flipped Can flip up to two coins again, at a cost of $1 each Payout: $5 for HH, $1 for HT or TH, $0 for TT HT HH 1/2 TTHT 1/2 HT Flip TFlip H Done HTHHTTHT 1/2 Flip T Flip H HT Done TT Done Flip T HTTT 1/2 1 -2-2 2 -3/2 2 3 4 0 3

64. TTHT 1/2 A SLIGHTLY MORE COMPLEX EXAMPLE Two fair coins Pay $1 to start, at which point both are flipped Can flip up to two coins again, at a cost of $1 each Payout: $5 for HH, $1 for HT or TH, $0 for TT HT HH 1/2 HT Flip TFlip H Done HTHHTTHT 1/2 Flip T Flip H HT Done TT Done Flip T HTTT 1/2 1 -2-2 2 -3/2 2 3 2 1/2 4 0 3

65. A SLIGHTLY MORE COMPLEX EXAMPLE Two fair coins Pay $1 to start, at which point both are flipped Can flip up to two coins again, at a cost of $1 each Payout: $5 for HH, $1 for HT or TH, $0 for TT HT HH 1/2 TTHT 1/2 HT Flip TFlip H Done HTHHTTHT 1/2 Flip T Flip H HT Done TT Done Flip T HTTT 1/2 1 -2-2 2 -3/2 2 3 2 1/2 3 4 0 3

66. C ARD G AMES Blackjack (6-deck), video poker: similar to coin- flipping game But in many card games, need to keep track of history of dealt cards in state because it affects future probabilities One-deck blackjack Bridge Poker (We won’t even get started on betting strategies)

67. P ARTIALLY O BSERVABLE G AMES Partial observability Don’t see entire state (e.g., other players’ hands) “Fog of war” Examples: Kriegspiel (see R&N) Battleship Stratego

68. P ARTIALLY -O BSERVABLE C ARD G AMES One possible strategy: Consider all possible deals Solve each deal as a fully-observable problem Choose the move that has the best average minimax value “Averaging over clairvoyance” [Why doesn’t this always work?]

69. O BSERVATION OF THE R EAL W ORLD 69 Real world in some state Percepts On(A,B) On(B,Table) Handempty Interpretation of the percepts in the representation language Percepts can be user’s inputs, sensory data (e.g., image pixels), information received from other agents,...

70. S ECOND SOURCE OF U NCERTAINTY : I MPERFECT O BSERVATION OF THE W ORLD Observation of the world can be: Partial, e.g., a vision sensor can’t see through obstacles (lack of percepts) 70 R1R1 R2R2 The robot may not know whether there is dust in room R2

71. S ECOND SOURCE OF U NCERTAINTY : I MPERFECT O BSERVATION OF THE W ORLD Observation of the world can be: Partial, e.g., a vision sensor can’t see through obstacles Ambiguous, e.g., percepts have multiple possible interpretations 71 A B C On(A,B)  On(A,C)

72. S ECOND SOURCE OF U NCERTAINTY : I MPERFECT O BSERVATION OF THE W ORLD Observation of the world can be: Partial, e.g., a vision sensor can’t see through obstacles Ambiguous, e.g., percepts have multiple possible interpretations Incorrect 72

73. E XAMPLE : B ELIEF S TATE In the presence of non-deterministic sensory uncertainty, an agent belief state represents all the states of the world that it thinks are possible at a given time or at a given stage of reasoning In the probabilistic model of uncertainty, a probability is associated with each state to measure its likelihood to be the actual state 73 0.20.30.40.1

74. B ELIEF S TATE A belief state is the set of all states that an agent think are possible at any given time or at any stage of planning a course of actions, e.g.: To plan a course of actions, the agent searches a space of belief states, instead of a space of states

75. S ENSOR M ODEL State space S The sensor model is a function SENSE: S  2 S that maps each state s  S to a belief state (the set of all states that the agent would think possible if it were actually observing state s) Example: Assume our vacuum robot can perfectly sense the room it is in and if there is dust in it. But it can’t sense if there is dust in the other room SENSE( ) =

76. V ACUUM R OBOT A CTION M ODEL Right either moves the robot right, or does nothing Left always moves the robot to the left, but it may occasionally deposit dust in the right room Suck picks up the dirt in the room, if any, and always does the right thing The robot perfectly senses the room it is in and whether there is dust in it But it can’t sense if there is dust in the other room

77. T RANSITION B ETWEEN B ELIEF S TATES Suppose the robot is initially in state: After sensing this state, its belief state is: Just after executing Left, its belief state will be: After sensing the new state, its belief state will be: or if there is no dust if there is dust in R 1 in R 1

78. T RANSITION B ETWEEN B ELIEF S TATES Playing a “game against nature” Left Clean(R 1 )  Clean(R 1 ) After receiving an observation, the robot will have one of these two belief states

79. AND/OR T REE OF B ELIEF S TATES Left Suck goal A goal belief state is one in which all states are goal states An action is applicable to a belief state B if its preconditions are achieved in all states in B Right loop goal

80. R ECAP Alpha-beta pruning: reduce complexity of minimax to O(b h/2 ) ideally Games with chance Expected values: averaging over probabilities Partial observability Reason about sets of states: belief state Much more on latter 2 topics later

81. P ROJECT P ROPOSAL (O PTIONAL ) Mandatory: instructor’s advance approval Out of town 9/24-10/1, can discuss via email Project title, team members 1/2 to 1 page description Specific topic (problem you are trying to solve, topic of survey, etc) Why did you choose this topic? Methods (researched in advance, sources of references) Expected results Email to me by 10/2

82. H OMEWORK Reading: R&N 4.3-4,13.1-2