1. Monte-Carlo Tree Search
Alan Fern

2. Introduction Rollout does not guarantee optimality or near optimality
It only guarantees policy improvement (under certain conditions)
Theoretical Question: Can we develop Monte-Carlo methods that give us near optimal policies?
With computation that does NOT depend on number of states!
This was an open theoretical question until late 90’s.
Practical Question: Can we develop Monte-Carlo methods that improve smoothly and quickly with more computation time?

3. Look-Ahead Trees Sparse Sampling is one such algorithm
Rollout can be viewed as performing one level of search on top of the base policy
In deterministic games and search problems it is common to build a look-ahead tree at a state to select best action
Can we generalize this to general stochastic MDPs?
Sparse Sampling is one such algorithm
Strong theoretical guarantees of near optimality a1 a2 ak …
Maybe we should search for multiple levels.

4. Online Planning with Look-Ahead Trees
At each state we encounter in the environment we build a look-ahead tree of depth h and use it to estimate optimal Q-values of each action
Select action with highest Q-value estimate
s = current state
Repeat
T = BuildLookAheadTree(s) ;; sparse sampling or UCT
;; tree provides Q-value estimates for root action
a = BestRootAction(T) ;; action with best Q-value
Execute action a in environment
s is the resulting state

5. Planning with Look-Ahead Trees
Real world state/action sequence s … …
Build look-ahead tree
Build look-ahead tree s a1 a2 …
s11
s1w
s21
s2w
R(s11,a1)
R(s11,a2)
R(s1w,a1)
R(s1w,a2)
R(s21,a1)
R(s21,a2)
R(s2w,a1)
R(s2w,a2) s a1 a2 …
s11
s1w
s21
s2w
R(s11,a1)
R(s11,a2)
R(s1w,a1)
R(s1w,a2)
R(s21,a1)
R(s21,a2)
R(s2w,a1)
R(s2w,a2)
…………
…………

6. Expectimax Tree (depth 1)
Alternate max & expectation
max node (max over actions) s a b
expectation node (weighted average over states) … …
s1 s sn-1 sn
s1 s sn-1 sn
States -- weighted by
probability of occuring
after taking a in s
After taking each action there is a distribution over next states (nature’s moves)
The value of an action depends on the immediate reward and weighted value of those states

7. Expectimax Tree (depth 2)
Alternate max & expectation
max node (max over actions) s a b
expectation node (max over actions) … … a a b b
…...... … … … …
Max Nodes: value equal to max of values of child expectation nodes
Expectation Nodes: value is weighted average of values of its children
Compute values from bottom up (leaf values = 0). Select root action with best value.

8. Exact Expectimax Tree V*(s,H) Alternate max & expectation Q*(s,a,H)
In general can grow tree to any horizon H
Size depends on size of the state-space. Bad!

9. Sparse Sampling Tree V*(s,H) Q*(s,a,H)
Sampling width w
(kw)H leaves
Replace expectation with average over w samples
w will typically be much smaller than n.

10. Sparse Sampling [Kearns et. al. 2002]
The Sparse Sampling algorithm computes root value via depth first expansion
Return value estimate V*(s,h) of state s and estimated optimal action a*
SparseSampleTree(s,h,w)
If h == 0 Then Return [0, null]
For each action a in s
Q*(s,a,h) = 0
For i = 1 to w
Simulate taking a in s resulting in si and reward ri [V*(si,h),a*] = SparseSample(si,h-1,w) Q*(s,a,h) = Q*(s,a,h) + ri + β V*(si,h)
Q*(s,a,h) = Q*(s,a,h) / w ;; estimate of Q*(s,a,h)
V*(s,h) = maxa Q*(s,a,h) ;; estimate of V*(s,h)
a* = argmaxa Q*(s,a,h)
Return [V*(s,h), a*]

11. SparseSample(h=2,w=2) Alternate max & averaging s a
Max Nodes: value equal to max of values of child expectation nodes
Average Nodes: value is average of values of w sampled children
Compute values from bottom up (leaf values = 0). Select root action with best value.

12. SparseSample(h=2,w=2) Alternate max & averaging s a 10
Max Nodes: value equal to max of values of child expectation nodes
Average Nodes: value is average of values of w sampled children
Compute values from bottom up (leaf values = 0). Select root action with best value.

13. SparseSample(h=2,w=2) Alternate max & averaging s a 10
SparseSample(h=1,w=2)
SparseSample(h=1,w=2)
Max Nodes: value equal to max of values of child expectation nodes
Average Nodes: value is average of values of w sampled children
Compute values from bottom up (leaf values = 0). Select root action with best value.

14. SparseSample(h=2,w=2) Alternate max & averaging s a 5 10
SparseSample(h=1,w=2)
SparseSample(h=1,w=2)
Max Nodes: value equal to max of values of child expectation nodes
Average Nodes: value is average of values of w sampled children
Compute values from bottom up (leaf values = 0). Select root action with best value.

15. SparseSample(h=2,w=2) Alternate max & averaging s a b 3 5 2 4
Select action ‘a’ since its value is larger than b’s.
Max Nodes: value equal to max of values of child expectation nodes
Average Nodes: value is average of values of w sampled children
Compute values from bottom up (leaf values = 0). Select root action with best value.

16. Sparse Sampling (Cont’d)
For a given desired accuracy, how large should sampling width and depth be?
Answered: Kearns, Mansour, and Ng (1999)
Good news: gives values for w and H to achieve policy arbitrarily close to optimal
Values are independent of state-space size!
First near-optimal general MDP planning algorithm whose runtime didn’t depend on size of state-space
Bad news: the theoretical values are typically still intractably large---also exponential in H
Exponential in H is the best we can do in general
In practice: use small H and use heuristic at leaves

17. Sparse Sampling (Cont’d)
In practice: use small H & evaluate leaves with heuristic
For example, if we have a policy, leaves could be evaluated by estimating the policy value (i.e. average reward across runs of policy) a1 a2 … …
…...... … … … …
policy sims

18. Uniform vs. Adaptive Tree Search
Sparse sampling wastes time on bad parts of tree
Devotes equal resources to each state encountered in the tree
Would like to focus on most promising parts of tree
But how to control exploration of new parts of tree vs. exploiting promising parts?
Need adaptive bandit algorithm that explores more effectively

19. What now? Adaptive Monte-Carlo Tree Search UCB Bandit Algorithm
UCT Monte-Carlo Tree Search

20. Bandits: Cumulative Regret Objective
Problem: find arm-pulling strategy such that the expected total reward at time n is close to the best possible (one pull per time step)
Optimal (in expectation) is to pull optimal arm n times
UniformBandit is poor choice --- waste time on bad arms
Must balance exploring machines to find good payoffs and exploiting current knowledge s a1 a2 ak …

21. Cumulative Regret Objective
Theoretical results are often about “expected cumulative regret” of an arm pulling strategy.
Protocol: At time step n the algorithm picks an arm 𝑎 𝑛 based on what it has seen so far and receives reward 𝑟 𝑛 ( 𝑎 𝑛 and 𝑟 𝑛 are random variables).
Expected Cumulative Regret ( 𝑬[𝑹𝒆𝒈 𝒏 ]): difference between optimal expected cumulative reward and expected cumulative reward of our strategy at time n
𝐸[𝑅𝑒𝑔 𝑛 ]=𝑛⋅ 𝑅 ∗ − 𝑖=1 𝑛 𝐸[𝑟 𝑛 ]

22. UCB Algorithm for Minimizing Cumulative Regret
Q(a) : average reward for trying action a (in our single state s) so far
n(a) : number of pulls of arm a so far
Action choice by UCB after n pulls:
Assumes rewards in [0,1]. We can always normalize if we know max value.
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2),

23. UCB: Bounded Sub-Optimality
Value Term: favors actions that looked good historically
Exploration Term: actions get an exploration bonus that grows with ln(n)
Expected number of pulls of sub-optimal arm a is bounded by:
where is the sub-optimality of arm a
Doesn’t waste much time on sub-optimal arms, unlike uniform!

24. UCB Performance Guarantee [Auer, Cesa-Bianchi, & Fischer, 2002]
Theorem: The expected cumulative regret of UCB 𝑬[𝑹𝒆𝒈 𝒏 ] after n arm pulls is bounded by O(log n)
Is this good?
Yes. The average per-step regret is O log 𝑛 𝑛
Theorem: No algorithm can achieve a better expected regret (up to constant factors)

25. What now? Adaptive Monte-Carlo Tree Search UCB Bandit Algorithm
UCT Monte-Carlo Tree Search

26. UCT Algorithm [Kocsis & Szepesvari, 2006]
UCT is an instance of Monte-Carlo Tree Search
Applies principle of UCB
Similar theoretical properties to sparse sampling
Much better anytime behavior than sparse sampling
Famous for yielding a major advance in computer Go
A growing number of success stories

27. Monte-Carlo Tree Search
Builds a sparse look-ahead tree rooted at current state by repeated Monte-Carlo simulation of a “rollout policy”
During construction each tree node s stores:
state-visitation count n(s)
action counts n(s,a)
action values Q(s,a)
Repeat until time is up
Execute rollout policy starting from root until horizon (generates a state-action-reward trajectory)
Add first node not in current tree to the tree
Update statistics of each tree node s on trajectory
Increment n(s) and n(s,a) for selected action a
Update Q(s,a) by total reward observed after the node
What is the rollout policy?

28. Rollout Policies
Monte-Carlo Tree Search algorithms mainly differ on their choice of rollout policy
Rollout policies have two distinct phases
Tree policy: selects actions at nodes already in tree (each action must be selected at least once)
Default policy: selects actions after leaving tree
Key Idea: the tree policy can use statistics collected from previous trajectories to intelligently expand tree in most promising direction
Rather than uniformly explore actions at each node

29. Example MCTS For illustration purposes we will assume MDP is:
Deterministic
Only non-zero rewards are at terminal/leaf nodes
Algorithm is well-defined without these assumpitons

30. Assume all non-zero reward occurs at terminal nodes.
At a leaf node tree policy selects a random action then executes default
Iteration 1
Current World State 1
Initially tree is single leaf 1
new tree node
Default Policy 1
Terminal (reward = 1)
Assume all non-zero reward occurs at terminal nodes.

31. Iteration 2 new tree node Default Policy
Must select each action at a node at least once
Iteration 2
Current World State 1 1
new tree node
Default Policy
Terminal (reward = 0)

32. Iteration 3 Must select each action at a node at least once
Current World State
1/2 1

33. When all node actions tried once, select action according to tree policy
Iteration 3
Current World State
1/2
Tree Policy 1

34. Iteration 3 Tree Policy new tree node Default Policy
When all node actions tried once, select action according to tree policy
Iteration 3
Current World State
1/2
Tree Policy 1
Default Policy
new tree node

35. When all node actions tried once, select action according to tree policy
Iteration 4
Current World State
1/3
Tree Policy
1/2

36. When all node actions tried once, select action according to tree policy
Iteration 4
Current World State
1/3
1/2 1

37. What is an appropriate tree policy? Default policy?
When all node actions tried once, select action according to tree policy
Current World State
2/3
Tree Policy
2/3 1
What is an appropriate tree policy? Default policy?

38. UCT Algorithm [Kocsis & Szepesvari, 2006]
Basic UCT uses random default policy
In practice often use hand-coded or learned policy
Tree policy is based on UCB:
Q(s,a) : average reward received in trajectories so far after taking action a in state s
n(s,a) : number of times action a taken in s
n(s) : number of times state s encountered
Theoretical constant that must be selected empirically in practice

39. When all node actions tried once, select action according to tree policy
Current World State
1/3 a1 a2
Tree Policy
1/2 1

40. When all node actions tried once, select action according to tree policy
Current World State
1/3
Tree Policy
1/2 1

41. When all node actions tried once, select action according to tree policy
Current World State
1/3
Tree Policy
1/2 1

42. When all node actions tried once, select action according to tree policy
Current World State
1/4
Tree Policy
1/3
0/1 1

43. When all node actions tried once, select action according to tree policy
Current World State
1/4
Tree Policy
1/3
0/1 1 1

44. When all node actions tried once, select action according to tree policy
Current World State
2/5
Tree Policy
1/3
1/2
0/1 1 1

45. UCT Recap To select an action at a state s
Build a tree using N iterations of monte-carlo tree search
Default policy is uniform random
Tree policy is based on UCB rule
Select action that maximizes Q(s,a) (note that this final action selection does not take the exploration term into account, just the Q-value estimate)
The more simulations the more accurate

46. Some Successes Computer Go Klondike Solitaire (wins 40% of games)
General Game Playing Competition
Real-Time Strategy Games
Combinatorial Optimization
Crowd Sourcing
List is growing
Usually extend UCT is some ways

47. Some Improvements
Use domain knowledge to handcraft a more intelligent default policy than random
E.g. don’t choose obviously stupid actions
In Go a hand-coded default policy is used
Learn a heuristic function to evaluate positions
Use the heuristic function to initialize leaf nodes (otherwise initialized to zero)

48. Practical Issues Selecting K
There is no fixed rule
Experiment with different values in your domain
Rule of thumb – try values of K that are of the same order of magnitude as the reward signal you expect
The best value of K may depend on the number of iterations N
Usually a single value of K will work well for values of N that are large enough

49. Practical Issues
UCT can have trouble building deep trees when actions can result in a large # of possible next states
Each time we try action ‘a’ we get a new state (new leaf) and very rarely resample an existing leaf s a b

50. Practical Issues
UCT can have trouble building deep trees when actions can result in a large # of possible next states
Each time we try action ‘a’ we get a new state (new leaf) and very rarely resample an existing leaf s a b

51. Practical Issues Degenerates into a depth 1 tree search
Solution: We have a “sparseness parameter” that controls how many new children of an action will be generated
Set sparseness to something like 5 or 10 s a b
…..
…..