1. Games: Expectimax MAX MIN MAX Prune if α ≥ β

2. Games: Expectimax MAX MIN MAX <=4 Prune if α ≥ β

3. Games: Expectimax MAX MIN MAX 4 Prune if α ≥ β

4. Games: Expectimax MAX MIN MAX 4 >=4 Prune if α ≥ β

5. Games: Expectimax MAX MIN MAX 4 >=4 <=6 Prune if α ≥ β

6. Games: Expectimax MAX MIN MAX 4 >=4 <=6 >=3 Prune if α ≥ β

7. Games: Expectimax MAX MIN MAX 4 >=4 <=6 >=4 Prune if α ≥ β

8. Games: Expectimax MAX MIN MAX 4 >=4 <=6 4 Prune if α ≥ β

9. Games: Expectimax MAX MIN MAX 4 >=4 <=4 4 Prune if α ≥ β

10. Games: Expectimax MAX MIN MAX 4 >=4 <=4 4 Prune if α ≥ β

11. Games: Expectimax MAX MIN MAX 4 >=4 <=4 4 Doesn’t change Prune if α ≥ β

12. Games: Expectimax MAX MIN MAX 4 >=4 <=4 4 <=3 Prune if α ≥ β

13. Games: Expectimax MAX MIN MAX 4 >=4 <=4 4 <=3 Prune if α ≥ β

14. Games: Expectimax MAX MIN MAX 4 [4] <=4 4 <=3 Prune if α ≥ β

15. function αβSEARCH(state) v = MAX-VALUE(state, -infinity, +infinity) return v function MAX-VALUE(state, α, β) returns a utility value v if isLeaf(state) then return UTILITY(state) v = -infinity for each sucessor c of state: do v = MAX(v, MIN-VALUE(c), α, β)) if v ≥ β then return v α = MAX(α, v) return v function MIN-VALUE(state, α, β) returns a utility value v if isLeaf(state) then return UTILITY(state) v = infinity for each sucessor c of state: do v = MIN(v, MAX-VALUE(c, α, β)) if v ≤ α then return v β = MIN(β, v) return v

16. Stochastic Games

17. Modify Minimax Tree? MAX MIN MAX knows his/her possible next moves, but not MIN’s Why no randomness for MAX’s moves?

18. Modify Minimax Tree? 2 2 4 4 7 7 4 4 6 6 0 0 5 5 -2 0.5 ? ? MAX MIN

19. Modify Minimax Tree chance nodes 2 2 4 4 7 7 4 4 6 6 0 0 5 5 -2 0.5 MAX MIN

20. Modify Minimax Tree chance nodes 2 2 4 4 7 7 4 4 6 6 0 0 5 5 -2 0.5 MAX MIN Now MAX maximizes expected minimax value

21. Quick Review of Expected Value

22. Expectimax 2 2 4 4 7 7 4 4 6 6 0 0 5 5 -2 240 0.5 MAX MIN CHANCE

23. Expectimax 2 2 4 4 7 7 4 4 6 6 0 0 5 5 -2 240 0.5 3 MAX MIN CHANCE

24. Expectimax 2 2 4 4 7 7 4 4 6 6 0 0 5 5 -2 240 0.5 3 MAX MIN CHANCE

25. Pruning Expectimax Trees? 2 2 4 4 2 MAX MIN >=2? CHANCE

26. Pruning Expectimax Trees? 2 2 4 4 1 1 1 1 2 MAX MIN >=2? No!.00010.9999 1 1.0001 CHANCE

27. Pruning Expectimax Trees Pruning is much more difficult – Possible if there is a bound on the utility values (not always very realistic) – Instead, use evaluation functions and depth- limited search

28. Searching in General

29. Recall N-Queens problem

30. N-Queens problem What is the depth? –8–8 What is the branching factor? – ≤ 8 How many nodes? – 8 8 = 17 million nodes Do we care about the path? What do we really care about?

31. Local search Key difference: we don’t care about the path to the solution, only the solution itself! Other similar problems? – Sudoku – Crossword puzzles – Scheduling –…–…

32. Local Search Approach Start with a random configuration Repeat until goal: generate a set of “local” next states (successors) move to one of these next states

33. Local Search Approach Start with a random configuration Repeat until goal: generate a set of “local” next states (successors) move to one of these next states Requirements: – ability to generate an initial, random state – generate the set of next states – criterion for evaluating which state to choose next

34. Example: 4 Queens Generating random state: – One queen per column Generate new states: – move queen in column Goal test: – no attacks Evaluation? (assume a cost, so lower the better)

35. Example: 4 Queens Generating random state: – One queen per column Generate new states: – move queen in column Goal test: – no attacks Evaluation? (assume a cost, so lower the better) – eval(state) = number of attacked queens

36. Local Search Approach Start with a random configuration Repeat until goal: generate a set of “local” next states (successors) move to one of these next states  Which one? Requirements: – ability to generate an initial, random state – generate the set of next states – criterion for evaluating which state to choose next

37. Hill-Climbing hillClimbing(problem): currentNode = makeRandomNode(problem) while True: nextNode = getLowestSuccessor(currentNode) if eval(currentNode) <= eval(nextNode): return currentNode //else currentNode = nextNode

38. Example: n-queens 3 steps! eval(state) = 4eval(state) = 3 eval(state) = 2 Goal!

39. Hill-Climbing Problems? hillClimbing(problem): currentNode = makeRandomNode(problem) while True: nextNode = getLowestSuccessor(currentNode) if eval(currentNode) <= eval(nextNode): return currentNode //else currentNode = nextNode

40. Hill-Climbing Problems? Does not look beyond successors of current state Does not remember previous states “Like climbing Everest in dense fog while suffering amnesia.” hillClimbing(problem): currentNode = makeRandomNode(problem) while True: nextNode = getLowestSuccessor(currentNode) if eval(currentNode) <= eval(nextNode): return currentNode //else currentNode = nextNode

41. Hill-climbing search: 8-queens problem After 5 moves eval(state) = 1 Initial State (ignore numbers) Problem with this state?

42. Problems with hill-climbing Current State Local minimum (where we end up) Global minimum (where we want to end up)

43. Idea 1: restart! Random-restart hill climbing – if we find a local minima start over again at a new random location Pros: Cons:

44. Idea 1: restart! Random-restart hill climbing – if we find a local minima start over again at a new random location Pros: – simple – no memory increase – for n-queens, usually a few restarts gets us there the 3 million queens problem can be solve in < 1 min! Cons: – if space has a lot of local minima, will have to restart a lot – loses any information we learned in the first search – sometimes we may not know we’re in a local minima/maxima

45. Idea 2: introduce randomness Rather than always selecting the best, pick a random move with some probability sometimes pick best, sometimes random make better states more likely, worse states less likely book just gives one… many ways of introducing randomness! hillClimbing(problem): currentNode = makeRandomNode(problem) while True: nextNode = getLowestSuccessor(currentNode) if eval(currentNode) <= eval(nextNode): return currentNode //else currentNode = nextNode

46. Randomness with simulated annealing What the does the term annealing mean? “When I proposed to my wife I was annealing down on one knee”?

47. Idea 3: simulated annealing What the does the term annealing mean?

48. Simulated annealing Early on, lots of randomness – avoids getting stuck in local minima – avoids getting lost on a plateau As time progresses, allow less and less randomness randomness time

49. Simulated Annealing Start with a random configuration Repeat until goal: generate a set of “local” next states (successors) move to a random next state

50. Simulated Annealing Start with a random configuration Repeat until goal: generate a set of “local” next states (successors) move to a random next state

51. Idea 3: why just 1 state? Local beam search: keep track of k states – Start with k randomly generated states – At each iteration, all the successors of all k states are generated If any one is a goal state stop – else select the k best successors from the complete list and repeat

52. Idea 3: why just 1 state? Local beam search: keep track of k states – Start with k randomly generated states – At each iteration, all the successors of all k states are generated If any one is a goal state stop – else select the k best successors from the complete list and repeat Cons? uses more memory over time, set of states can become very similar

53. Idea 5: genetic algorithms Start with a set of k states Rather than pick from these, create new states by combining states Maintain a “population” of states

54. Idea 5: genetic algorithms Start with a set of k states Rather than pick from these, create new states by combining states Maintain a “population” of states Inspired by the biological evolution process Uses concepts of “Natural Selection” and “Genetic Inheritance” (Darwin 1859) Originally developed by John Holland (1975)

55. The Algorithm Randomly generate an initial population. Repeat the following: 1.Select parents and “reproduce” the next generation 2.Randomly mutate some 3.Evaluate the fitness of the new generation 4.Discard old generation and keep some of the best from the new generation

56. 1 0 1 0 1 1 1 1 1 0 0 0 1 1 Parent 1 Parent 2 1 0 1 0 0 1 1 1 1 0 0 1 1 1 Child 1 Child 2 Genetic Algorithm Crossover

57. 1 0 1 0 1 1 1 1 1 0 0 0 1 1 Parent 1 Parent 2 1 0 1 0 0 1 1 1 1 0 0 1 1 0 Child 1 Child 2 Mutation Genetic Algorithm Mutation

58. Genetic algorithms on 8-Queens eval(state) = 7 eval(state) = 6

59. Genetic algorithms eval(state) = 7 eval(state) = 6 eval(state) = 5