1. Reinforcement Learning Presented by: Kyle Feuz

3. Outline Motivation MDPs RL Model-Based Model-Free Q-Learning SARSA Challenges

4. Examples Pac-Man Spider

5. MDPs 4-tuple (State, Actions, Transitions, Rewards).

6. Important Terms Policy Reward Function Value Function Model

7. Model-Based RL Learn transition function Learn expected rewards Compute the optimal policy

8. Model-Free RL Learn expected rewards/values Skip learning transistion function Trade-offs?

9. Basic Equations

10. Examples Pac-Man Spider Mario

11. Q-Learning Q(s, a) = = (1 − α)Q(s, a) + α[R(s, s′ ) + Max Q(s′, a′ )]

12. Q-Learning Demo Video

13. SARSA Q-Learning Q(s, a) = = (1 − α)Q(s, a) + α[R(s, s′ ) + Q(s′, a′ )]

14. Challenges Explore vs. Exploit State Space representation Training Time Multiagent Learning Moving Target Competive or Cooperative

15. Transfer Learning for Reinforcement Learning on a Physical Robot Applied TL and RL on Nao robot TL using the q-value reuse approach RL uses SARSA variant State space is represented via CMAC Neural Network inspired by the cerebellum Acts as an associative memory Allows agents to generalize the state space

16. Agent Model

17. SARSA Update Rule Q(s, a) = = (1 − α)Q(s, a) + α[R(s, s′ ) + γe(s, a)Q(s′, a′ )]

18. Q-Value Reuse Q(s, a) = = Qsource (χX (s), χA (a)) + Qtarget (s, a)

19. Experimental Setup Seated Nao robot Hit the ball at 45 angle 5 Actions in Source – 9 Actions in Target

20. Robot Results

21. Simulator Results

22. Advanced Combinations

23. Examples Pac-Man Spider Mario Q-Learning Penalty Kick Others

24. References and Resources rl repository rl-community rl on PBWorks rl warehouse Reinforcement Learning: An Introduction Artificial Intelligence: A Modern Approach How to Make Software Agents do the Right Thing How to Make Software Agents do the Right Thing

25. Questions?