1. 1 A Semiparametric Statistics Approach to Model-Free Policy Evaluation Tsuyoshi UENO (1), Motoaki KAWANABE (2), Takeshi MORI (1), Shin-ich MAEDA (1), Shin ISHII (1),(3) (1) Kyoto University (2) Fraunhofer FIRST

2. 2 Summary of This Talk We discussed LSTD-based policy evaluation from the viewpoint of semiparametric statistics and estimating function. 1.How good is LSTD? 2.Can we improve LSTD ? LSTD is a type of estimating function method, and evaluate the asymptotic estimation variance of LSTD. We derive an optimal estimating function with the minimum asymptotic estimation variance. We propose a new policy evaluation algorithm (gLSTD)

3. Model-Free Reinforcement Learning 3 Goal: Obtain an optimal policy which maximizes the sum of future rewards Environment Action State Reward Policy

4. 4 Policy Iteration [Sutton & Barto, 1998] Policy Evaluation （ Estimate the value function ） Policy Improvement (Update the policy) Value function estimation is a key of policy iteration !! If the value function can be correctly estimated, policy iteration converges the optimal policy

5. 5 Policy Evaluation Method: LSTD [Bratke & Barto, 1996] Least Squares Temporal Difference (LSTD) –LSTD-based policy iteration algorithms have shown good practical performance. Least Squares Policy Iteration (LSPI) [Lagoudakis & Parr, 2003] Natural Actor-Critic (NAC) [Peters et.al., 2003, 2005] Representation Policy Iteration (RPI) [Mahadevan & Maggino, 2007] LSTD is one of the important algorithms in RL field

6. 6 Least Square Temporal Difference (LSTD) Bellman equation [Bellman, 1966 ] Feature Parameter Assumption We assume that the linear function ‘completely’ represents the value function. (There are no bias.)

7. 7 Linearly approximated bellman equation Parameter Noise Just a linear regression problem (Error in (input) variable problem [Young,1984] ) Input: Output: Least Square Temporal Difference (LSTD) the input and observation noise are mutually dependent!!

8. 8 Linear Regression with Error in Variables x y OLS estimator is biased. Ordinary least squares method (OLS): OLS the observation noise depends on the input variable,

9. 9 Instrumental Variable Method [Soderstrom and Stoica, 2002] Introduce the instrumental variable: is an unbiased estimator Input: x Output: y The instrumental variable is correlated with the input but uncorrelated with the noise

10. 10 LSTD = Instrumenatal variable method. –Instrumental variable : Least Square Temporal Difference (LSTD) (for example) are also instrumental variables It is important to choose an appropriate instrumental variable.

11. Our Approach How good is LSTD ? Can we improve LSTD? 11 We analysis the asymptotic estimation variance of instrumental variable method. We optimize the instrumental variable so as to minimize the asymptotic estimation variance. We introduce a viewpoint of semiparametric statistical inference

12. 12 Semiparametric model: – is target parameter – are nuisance parameter (infinite degree of freedom ) Semiparametric Statistics Approach We need to estimate only the target parameter regardless of the nuisance parameters Linearly approximated Bellman equation We don’t know the noise distribution.

13. Estimating function [Godambe, 1985] [Conditions] Estimating equation 13 Inference of Semiparametric Model converges to the true parameter regardless of nuisance parameter. For any nuisance parameter

14. 14 Estimating Functions Estimating function = LSTD Estimating function = Instrumental variable method Are there any other estimating functions ? Instrumental Variable

15. 15 Are There Any Other Estimating Functions ? Proposition 1 Every admissible estimating functions must have the form of No !! “Inadmissible” estimating function means there are superior estimating functions to it.

16. 16 Asymptotic Variance of LSTD-Based Estimators Lemma 2. The asymptotic estimation variance of estimating function for value functions is given by where and Which instrumental variable performs the minimum asymptotic variance ?

17. 17 The Optimal Estimating Function Theorem 1. The optimal instrumental variable with the minimum asymptotic variance is given by where True parameter (unknown) Unknown conditional expectations gLSTD Approximation is necessary

18. gLSTD The residual of true parameter Unknown conditional expectations 18 The optimal instrumental variable Replace the regression residual of true parameter with that of LSTD estimator. Approximate these conditional expectations by using a sample-based function approximation technique. (Unknown)

19. 19 Summary of gLSTD 1)Calculate the initial estimator and replace the true residual 2)Approximate the conditional expectations 3)Construct the instrumental variable 4)Calculate the gLSTD estimator

20. 20 Simulation (Markov Random Walk) Conditions of the simulation experiment –Policy: Random –The number of steps: 100 –The number of episodes: 100 –Discounted factor: 0.9 Basis function : –We generated three basis functions by the diffusion model. [Mahadevan & Maggino, 2007] 1 32 4 5 R=0 R=1.0R=0.5

21. 21 Simulation Result. The estimator of gLSTD achieved 20% smaller MSE than that of the LSTD Median The upper and lower quartiles 20%

22. 22 Conclusion We discussed LSTD-based policy evaluation in the framework of semiparametric statistics approach. –We evaluated the asymptotic variance of LSTD-based estimator. –We derived the optimal estimating function with the minimum asymptotic variance and proposed its practical implementation method: gLSTD. –Through an simple Markov chain problem, we demonstrated that gLSTD reduces the estimation variance of LSTD.

23. 23 Future Work A Semiparametric Approach to Model-Free Policy Evaluation A Semiparametric Approach to Model-Free Reinforcement Learning Application to the policy improvement - Least Squares Policy Iteration (LSPI) - Natural Actor Critic (NAC) etc.

24. 24 End Thank you for your attention!!

25. Cost Function 25

26. Simulation Result 26 1 2345

27. 27

28. 28

29. 29 Questions 1.How good is the LSTD ? 2.Can we improve the LSTD ?

30. 30 Questions 1.How good is the LSTD ? 2.Can we improve the LSTD ?

31. 31 Questions 1.How good is the LSTD? 2.Can we improve the LSTD ? LSTD is a type of estimating function method, and evaluate the asymptotic estimation variance of LSTD. We derive the optimal estimating function with the minimum asymptotic estimation variance.

32. 32 The Suboptimal Estimating Function (LSTDc) GLSTD is required to estimate the functions depending on current state. To avoid estimating these functions, we simple replace them by constant value. Optimize it to minimize the asymptotic variance

33. 33 The Suboptimal Estimating Function (LSTDc) Theorem 2. The optimal shift is given by where

34. 34 Summary of This Talk We introduce a semiparametric statistical viewpoint for estimation of value function with linear model. Our aim –Evaluate the estimation variance of value functions –Develop more efficient estimation methods

35. 35 Estimating Functions Question Which function is appropriate when more than one estimating function exist ? Answer Choose the estimating function with minimum asymptotic variance

36. 36 Summary of Our Main Results 1.Formulate the estimation problem of linearly- represented value functions as a semiparametric inference problem 2.Evaluate the asymptotic variance of estimations of value function 3.Derive the optimal estimation method with the minimum asymptotic variance

37. 37 Estimating Functions Question Which function is appropriate when more than one estimating function exist ? Answer Choose the estimating function with minimum asymptotic variance

38. 38 Instrumental Variable (IV) Method Instrumental variable: –Correlated to the input variable, but uncorrected to the noise. Instrumental variable method

39. 39 Statistics approach

40. 40 What is the Semiparametric Approach ? Semiparametric model: –Parameter: –Nuisance parameter: Estimating function [Godambe, 1985] [Conditions] – converges to the true parameter We need to estimate the parameter regardless of the nuisance parameter. Show the detail in [Godambe, 1985]