1. Reinforcement Learning for the Soccer Dribbling Task Arthur Carvalho Renato Oliveira

2. Introduction RoboCup soccer simulation –Scoring “A Data Mining Approach to Solve the Goal Scoring Problem” –Passing “A New Passing Strategy Based on Q-Learning Algorithm in RoboCup” –Dribbling ?

3. Soccer Dribbling Task

4. Outline The soccer dribbling task as a RL problem RL solution Experiments Conclusion

5. The Soccer Dribbling Task as a RL Problem Coach –Setting positions Dribbler is placed in the center-left region together with the ball Adversary is placed in a random position –Manage the play Adversary wins when he gains possession or when the ball goes out of the field Dribbler wins when he crosses the field with the ball

6. The Soccer Dribbling Task as a RL Problem When an episode ends, the coach starts a new one RoboCup soccer simulator operates in discrete time steps Episodic reinforcement-learning framework

7. The Soccer Dribbling Task as a RL Problem Actions –HoldBall() –Dribble(α, k) Dribble(30, 5), Dribble(330, 5), Dribble(0, 5), Dribble(0, 10) The dribbler can kick the ball forward (strongly and weakly), diagonally upward, and diagonally downward.

8. The Soccer Dribbling Task as a RL Problem State VariableMeaning posY (dribbler) Vertical position of the dribbler ang(dribbler)Global angle of the dribbler ang(dribbler; adversary) The relative angle between the dribbler and the adversary ang(ball; adversary) The relative angle between the ball and the adversary dist(ball; adversary) Distance between the ball and the adversary

9. Outline The soccer dribbling task as a RL problem RL solution Experiments Conclusion

10. RL Solution

11. CMAC –Partitioning the state space into several receptive fields (hyper-rectangles) Each one is associated with a weight –Multiple partitions of the state space (layers) are usually used –The CMAC’s response to a given input is equal to the sum of the weights of the excited receptive fields

12. RL Solution

13. Outline The soccer dribbling task as a RL problem RL solution Experiments Conclusion

14. Experiments

15. Adversary –Fixed policy –It computes a near-optimal interception point (UvA Trilearn 2003 team) Two phases –Training –Testing

16. Experiments Training Phase: 5 independent runs, each one lasting 50,000 episodes 53%

17. Experiments Qualitatively –Rule #1

18. Experiments Qualitatively –Rule #2

19. Experiments

20. Outline The soccer dribbling task as a RL problem RL solution Experiments Conclusion

21. Dribble –Soccer dribbling task –Reinforcement learning solution Benchmark Start point for dribbling tasks in other sports games –E.g., hockey, basketball, and football

22. Thank you! Source code available at: http://sites.google.com/site/soccerdribbling Arthur Carvalho Renato Oliveira a3carval@cs.uwaterloo.ca rmo@cin.ufpe.br