1. Abstract
LSPI (Least-Squares Policy Iteration) works well in value function approximation
Gaussian kernel is a popular choice as a basis function but can not treat discontinuities well
Propose new type of basis function, Geodesic Gaussian Kernels
Apply our method to robot control tasks

2. Maze Problem Task: guide a robot to the goal from any placeUp
Reward:
+1 (reach the goal)
0 (otherwise)
Left
Right
Good Job!!
Position
Down
Task: guide a robot to the goal from any place
Condition: no supervision but reward at goal
Goal: select optimal action in each position

3. Markov Decision Process (MDP)
A model consisting of {S, A, T, R}
S: finite set of states, e.g. si = (xi, yi), i=1,2,3….
A: finite set of actions, e.g. up, down, left, right
T: transition function, specifies next state s’
R: immediate reward function
Assume MDP is given or can be estimated from data
Policy function: specifies action to take in each state
Goal: learn good policy function from MDP

4. Reinforcement Learning (RL)
Iterate 1 and 2 (Policy iteration) gives optimal π
1. Evaluate action-value function Q(s,a): discounted sum of future rewards when taking a in s and following π thereafter (Sutton 1998)
2. Update policy
Problem: Q(s,a) can not be evaluated directly
r(si,ai): immediate reward when taking action a in state s, γ: discount factor (0<γ<1)

5. Bellman Equation Q(s,a) can be evaluated through recursive form
Problem: number of parameters becomes very large in large state and action spaces
Slow learning
Overfitting

6. Least-Square Policy Iteration
Lagoudakis and Parr, 2005
Linear Architecture
φ(s,a): fixed basis function, w: learned weight, K: # of basis functions
Learn so as to optimally approximate Bellman equation in the Least-square sense
# of learning parameters can be reduced dramatically
Problem: How do we choose φi(s,a)?

7. Popular Choice: Gaussian Kernel
ED: Euclid Distance
Sc: centre state
Bell-shape centered on Sc
Smooth surface
Gaussian tail goes over the partition Sc
One kernel is placed near the partition Sc

8. Value Function with Discontinuities
Approximated value function by
20-randomly located Gaussian kernels
Less accurate around the partition
Undesired policies are obtained around the partition
Obtained Policy
Log scale
Optimal value function
Log scale

9. Aim of This Research
Gaussian kernels are not suited for approximating discontinuous value function
Value function is smooth along the maze but discontinuous across the partition
Goal: We propose new kernel, Geodesic Gaussian Kernels, based on the state space structure

10. Gaussian Kernels on Graph
Ordinary Gaussian
Geodesic Gaussian S Sc S Sc
Shortest Path
(Dijkstra Algorithm)
Euclidean Distance

11. Example of Kernels Ordinary Gaussian Geodesic Gaussian Sc Sc Sc
Tail does not go across the partition

12. Value Function by Geodesic Gaussian
Approximated value function by
20-randomly located Geodesic Gaussian kernels
Accurate around the partition
Desired policies are obtained around the walls
Obtained Policies
Log scale
Optimal value function
Log scale

13. Experimental Results Fraction of optimal states Average over 100 runs
Sutton’s maze
Three-room maze
Fraction of optimal states
Average over 100 runs

14. Discussions Ordinary Gaussian:
Large width suffers from the tail problem
Small width does not have the tail problem,
but is less smooth along the state space.
Geodesic Gaussian (with rather large width):
Smooth along the state space, while discontinuity across the partitions preserved.

15. Arm Robot Control Reward: 2-DOF robot arm Lead the hand to the apple.
+1 (reach the apple)
0 (otherwise)
2-DOF robot arm
Lead the hand to the apple.
State space

16. Learned Value Functions by Ordinary Gaussian
smooth over
the obstacle

17. Learned Value Functions by Geodesic Gaussian
smooth along
the state space

18. Summary of Results
Average over
30 runs

19. Khepera Robot Navigation
Khepera robot has 8 IR sensors measuring the distance to obstacles (0-1030)
Task: explore unknown maze without collision
6 actions
Reward:
+10 (a1)
+5 (a2/a3)
0 (a4/a5)
-4 (a6)
-20 (collision)

20. Difficulty of The Task
State space is high dimensional (8-D) and large (1030^8)
Entire state space can not be explored, thus need inter/extrapolation
State transition is highly stochastic

21. State Space and Graph
8D graph by Self-Organized Map is projected onto the 2D subspace for visualization
Partitions

22. Learned Value Functions by Ordinary Gaussian
Local maximum
When Khepera faces an obstacle,
it goes backward (and go forward).

23. Learned Value Functions by Geodesic Gaussian
When Khepera faces an obstacle,
it makes a turn (and go forward).

24. Summary of Results
Average over
30 runs

25. Conclusion Value function approximation: good basis function needed
Ordinary Gaussian kernel:
smooth over discontinuities
Geodesic Gaussian kernel:
smooth along the state space
Graph Gaussian is promising in high-dimensional continuous problems!