1. Stochastic and non-Stochastic Games – a survey
Uri Zwick Tel Aviv University
Based on joint work with Oliver Friedmann, Thomas Dueholm Hansen, Marcin Jurdzinski, Peter Bro Miltersen, Mike Paterson

2. More precisely… 2-player, zero-sum, finite state, full-information,
turn-based,
infinite-horizon,
total payoff,
stochastic games
For short, 2TBSG

3. Overview of talk What are stochastic games? Why are they interesting?
How can we solve stochastic games?
Relation to other problems
Many intriguing open problems

4. Turn-based Stochastic Games [Shapley ’53] [Gillette ’57] … [Condon ’92]
Total reward version
Discounted version
Limiting average version
Both players have optimal positional strategies
Can optimal strategies be found in polynomial time?

5. A positional strategy is a deterministic memoryless strategy
Strategies
A deterministic strategy specifies which action to take given every possible history
A mixed strategy is a probability distribution over deterministic strategies
A memoryless strategy is a strategy that depends only on the current state
A positional strategy is a deterministic memoryless strategy

6. Both players have positional optimal strategies
Values
positional
general
positional
general
When the strategies are positional we get a Markov chain and y_i(sigma,tau) is easy to compute.
Both players have positional optimal strategies
There are positional strategies that are optimal for every starting position

7. Markov Decision Processes [Bellman ’57] [Howard ’60] …
Total reward version
Discounted version
Limiting average version
Optimal positional strategies can be found using LP
Is there a strongly polynomial time algorithm?

8. Stochastic shortest paths (SSPs)
Minimize the expected cost of getting to the target

9. To show that value  v , guess an optimal strategy  for MAX
SG  NP  co-NP
Deciding whether the value of a state is at least (at most) v is in NP  co-NP
To show that value  v , guess an optimal strategy  for MAX
Find an optimal counter-strategy  for min by solving the resulting MDP.
Is the problem in P ?

10. Turn-based non-Stochastic Games [Ehrenfeucht-Mycielski (1979)]…
Total reward version
Discounted version
Limiting average version
Both players have optimal positional strategies
Still no polynomial time algorithms known!

11. Why are stochastic games interesting?
Intrinsically interesting…
Model long term planning in adversarial and stochastic environments. Applications in AI and OR
Applications in automatic verification and automata theory
Competitive version of well-known optimization problems
Perhaps the simplest combionatorial problem in NP  co-NP not known to be in P
Many intriguing open problems

12. Algorithms for stochastic games
Value Iteration:
Solve a finite-horizon game using dynamic programming.
Let the horizon tend to infinity.
Polynomial, but not strongly polynomial, in game description and in 1/(1).

13. Algorithms for stochastic games
Policy/Strategy Iteration:
Construct a sequence of strategies for one of the players, each improving on the previous one.
Strategies improved by performing local improving steps.
Guaranteed to terminate in a finite number of steps with an optimal strategy.
How exactly do we improve a strategy?
Is number of steps polynomial?
[Howard (1960)]

14. Evaluating an MDP strategy/policy
MDP + policy  Markov Chain
Values of a fixed policy can be found by solving a system of linear equations

15. Improving switches

16. Best improving switches

17. Dual LP formulation for stochastic shortest paths
[d’Epenoux (1964)]

18. Primal LP formulation for stochastic shortest paths
[d’Epenoux (1964)]

19. Multiple improving switches

20. Policy iteration for MDPs [Howard ’60]

21. Policy iteration variants

22. Policy iteration for 2-player SGs

23. Policy iteration for 2-player SGs Correctness
Let (1,2) be the policies of the two players at the start of some iteration
Let (’1,’2) be the policies of the two players at the end of some iteration
Need to show that (1,2) <1 (’1,’2)
Since 2 is an optimal counter-strategy against 1, all switches of 2 are improving for player 1
Only improving switches are performed on 1

24. The RANDOM FACET algorithm [Kalai (1992)] [Matousek-Sharir-Welzl (1992)] [Ludwig (1995)]
A randomized strategy improvement algorithm
Initially devised for LP and LP-type problems
Applies to all turn-based games
Sub-exponential complexity
Fastest known for non-discounted 2-player games
Performs only one improving switch at a time

25. The RANDOM FACET algorithm

26. Binary game – two actions per state
RANDOM FACET for binary games [Kalai (1992)] [Matousek-Sharir-Welzl (1992)]
Binary game – two actions per state
Policy – n-bit vector
In the binary case, removing an action is equivalent to fixing the other action from the corresponding state
Sort the states so that the best policy with the i-th bit fixed is better than the best policy with the (i1)-th bit fixed

27. Would never be switched !
RANDOM FACET for binary games [Kalai (1992)] [Matousek-Sharir-Welzl (1992)]
All correct !
Would never be switched !

28. Complexity of Policy Iteration?
Essentially no answer for almost 50-years!
Algorithm works very well in practice
Is there a variant of PI that solves 2-player games in polynomial time?
Some answers obtained in recent years

29. Upper bounds for Policy Iteration
Dantzig’s pivoting rule, and the Howard’s policy iteration algorithm, Switch-All, require only O*(mn/(1)) iterations for discounted MDPs [Ye ’10]
Howard’s policy iteration algorithm, Switch-All requires only O*(m/(1)) iterations for discounted turn-based 2-player Stochastic Games [Hansen-Miltersen-Z ’11]

30. Lower bounds for Policy Iteration
Switch-All for Parity Games is exponential [Friedmann ’09]
Switch-All for MDPs is exponential [Fearnley ’10]
Random-Facet for Parity Games is sub-exponential [Friedmann-Hansen-Z ’11]
Random-Facet and Random-Edge for MDPs and hence for LPs are sub-exponential [FHZ ’11]

31. Parity Games (PGs) A simple example
Priorities 2 3 2 1 4 1
EVEN wins if largest priority seen infinitely often is even

32. Non-emptyness of -tree automata modal -calculus model checking
Parity Games (PGs)
ODD 8
EVEN 3
EVEN wins if largest priority seen infinitely often is even
Equivalent to many interesting problems in automata and verification:
Non-emptyness of -tree automata
modal -calculus model checking

33. Mean Payoff Games (MPGs)
Parity Games (PGs)
Mean Payoff Games (MPGs)
[Stirling (1993)] [Puri (1995)]
ODD 8
EVEN 3
Replace priority k by payoff (n)k
Move payoffs to outgoing edges

34. Deterministic Algorithms
Non- discounted
Discounted
Algorithm
SWITCH BEST
SWITCH ALL
[Condon ’93]
[Ye ’10]
[Friedmann ’09]
[Hansen-Miltersen-Z ’11]
[Fearnley ’10]

35. Randomized Algorithms
Upper bound
Lower bound
Algorithm
RANDOM EDGE
RANDOM FACET
[Kalai ’92] [Matousek-Sharir-Welzl ’92]
[Friedmann-Hansen-Z ’11]

36. 2TBSG  CLS  PLS  PPAD, but unlikely to be CLS-complete
Related problems
2TBSGs can be reduced to Generalized P-matrix Linear Complementarity Problems
2TBSG  CLS  PLS  PPAD, but unlikely to be CLS-complete
2TBSG  LP-type
Another LP-type problem, not known to be in P, is Smallest Enclosing Ball
Nice abstract formulations that capture many of these problems are USOs and AUSOs

37. Summary
The hope that simple deterministic or randomized versions of the policy iteration algorithm would solve 2-player stochastic games in polynomial time failed to materialize.
New techniques are probably needed.
Interior-point methods?
Hardness results?

38. Many intriguing and important open problems
Polynomial Policy Iteration Algorithms ???
Polynomial algorithms for
SGs ?
nSGs ?
PGs ?
Strongly Polynomial algorithms for
LPs ?
MDPs ?

39. THE END

40. 3-bit counter
(−N)15

41. 3-bit counter 1

42. Observations Five “decisions” in each level
Process always reaches the sink
Values are expected lengths of paths to sink

43. 3-bit counter – Improving switches
Random-Edge can choose either one of these improving switches… 1

44. Cycle gadgets Cycles close one edge at a time
Shorter cycles close faster

45. Cycles open “simultaneously”
Cycle gadgets
Cycles open “simultaneously”

46. 3-bit counter 23 1 1

47. From b to b+1 in seven phases
Bk-cycle closes
Ck-cycle closes
U-lane realigns
Ai-cycles and Bi-cycles for i<k open
Ak-cycle closes
W-lane realigns
Ci-cycles of 0-bits open

48. 3-bit counter 34 1 1

49. Size of cycles Various cycles and lanes compete with each other
Some are trying to open while some are trying to close
We need to make sure that our candidates win!
Length of all A-cycles = 8n
Length of all C-cycles = 22n
Length of Bi-cycles = 25i2n
O(n4) vertices for an n-bit counter
Can be improved using a more complicated construction and an improved analysis (work in progress)

50. Lower bound for Random-Facet
Implement a randomized counter

51. Optimal Values in 2½-player games
positional
general
positional
general
Both players have positional optimal strategies
There are strategies that are optimal for every starting position

52. To show that value  v , guess an optimal strategy  for MAX
2½G  NP  co-NP
Deciding whether the value of a state is at least (at most) v is in NP  co-NP
To show that value  v , guess an optimal strategy  for MAX
Find an optimal counter-strategy  for min by solving the resulting MDP.
Is the problem in P ?

53. Discounted 2½-player games
Strategy improvement correct version

54. Upper bounds for policy iteration algorithm
Day 1 Monday, October 31
Uri Zwick (武熠) (Tel Aviv University) Perfect Information Stochastic Games
Lecture 1½
Upper bounds for policy iteration algorithm

55. Complexity of Strategy Improvement
Greedy strategy improvement for non-discounted 2-player and 1½-player games is exponential !
[Friedmann ’09] [Fearnley ’10]
A randomized strategy improvement algorithm for 2½-player games runs in sub-exponential time
[Kalai (1992)] [Matousek-Sharir-Welzl (1992)]
Greedy strategy improvement for 2½-player games with a fixed discount factor is strongly polynomial
[Ye ’10] [Hansen-Miltersen-Z ’11]

56. The RANDOM FACET algorithm
Analysis

57. The RANDOM FACET algorithm
Analysis

58. The RANDOM FACET algorithm
Analysis

59. The RANDOM FACET algorithm
Analysis

60. Lower bounds for policy iteration algorithm
Day 1 Monday, October 31
Uri Zwick (武熠) (Tel Aviv University) Perfect Information Stochastic Games
Lecture 3
Lower bounds for policy iteration algorithm

61. 单纯形算法中随机 主元旋转规则的次指数级下界
Subexponential lower bounds for randomized pivoting rules for the simplex algorithm
单纯形算法中随机 主元旋转规则的次指数级下界
Oliver Friedmann – Univ. of Munich Thomas Dueholm Hansen – Aarhus Univ. Uri Zwick (武熠) – Tel Aviv Univ.
TexPoint fonts used in EMF.
Read the TexPoint manual before you delete this box.: AAAAAA

62. Linear Programming
Maximize a linear objective function subject to a set of linear equalities and inequalities

63. Linear Programming Simplex algorithm (Dantzig 1947)
Ellipsoid algorithm (Khachiyan 1979)
Interior-point algorithm (Karmakar 1984)

64. The Simplex Algorithm Dantzig (1947)
Move up, along an edge to a neighboring vertex, until reaching the top

65. Pivoting Rules – Where should we go?
Largest improvement?
Largest slope? …

66. Deterministic pivoting rules
Largest improvement
Largest slope
Dantzig’s rule – Largest modified cost
Bland’s rule – avoids cycling
Lexicographic rule – also avoids cycling
All known to require an exponential number of steps, in the worst-case
Klee-Minty (1972)
Jeroslow (1973), Avis-Chvátal (1978),
Goldfarb-Sit (1979), … , Amenta-Ziegler (1996)

67. Taken from a paper by Gärtner-Henk-Ziegler
Klee-Minty cubes (1972)
Taken from a paper by Gärtner-Henk-Ziegler

68. Is there a polynomial pivoting rule?
Is the diameter polynomial?
Hirsch conjecture (1957): The diameter of a d-dimensional, n-faceted polytope is at most n−d
Refuted recently by Santos (2010)!
Diameter is still believed to be polynomial (See, e.g., the polymath3 project)
Quasi-polynomial (nlog d+1) upper bound Kalai-Kleitman (1992)

69. Randomized pivoting rules
Random-Edge
Choose a random improving edge
Random-Facet
If there is only one improving edge, take it. Otherwise, choose a random facet containing the current vertex and recursively find the optimum within that facet.
[Kalai (1992)] [Matoušek-Sharir-Welzl (1996)]
Random-Facet is sub-exponential!
Are Random-Edge and Random-Facet polynomial ???

70. Abstract objective functions (AOFs)
Acyclic Unique Sink Orientations (AUSOs)
Every face should have a unique sink

71. AUSOs of n-cubes 2n facets 2n vertices USOs and AUSOs
Stickney, Watson (1978)
Morris (2001)
Szabó, Welzl (2001)
Gärtner (2002)
The diameter is exactly n
Bypassing the diameter issue!

72. AUSO results
Random-Facet is sub-exponential [Kalai (1992)] [Matoušek-Sharir-Welzl (1996)]
Sub-exponential lower bound for Random-Facet [Matoušek (1994)]
Sub-exponential lower bound for Random-Edge [Matoušek-Szabó (2006)]
Lower bounds do not correspond to actual linear programs
Can geometry help?

73. Consider LPs that correspond to Markov Decision Processes (MDPs)
LP results
Explicit LPs on which Random-Facet and Random-Edge make an expected sub-exponential number of iterations
Technique
Consider LPs that correspond to Markov Decision Processes (MDPs)
Observe that the simplex algorithm on these LPs corresponds to the Policy Iteration algorithm for MDPs
Obtain sub-exponential lower bounds for the Random-Facet and Random-Edge variants of the Policy Iteration algorithm for MDPs, relying on similar lower bounds for Parity Games (PGs)

74. Dual LP formulation for MDPs

75. Primal LP formulation for MDPs

76. Vertices and policies