1. From Motor Babbling to Planning Cornelius Weber Frankfurt Institute for Advanced Studies Goethe University Frankfurt, Germany ICN Young Investigators’ Colloquium 26 th June 2008, Frankfurt am Main

2. Reinforcement Learning valueactor units fixed reactive system that always strives for the same goal Trained Weights

3. reinforcement learning does not use the exploration phase to learn a general model of the environment that would allow the agent to plan a route to any goal so let’s do this

4. Learning actor state space randomly move around the state space learn world models: ● associative model ● inverse model ● forward model variables: ► action ► current state ► next state

5. Learning: Associative Model weights to associate neighbouring states use these to find any possible routes between agent and goal

6. Learning: Inverse Model weights to “postdict” action given state pair use these to identify the action that leads to a desired state Sigma-Pi neuron model

7. Learning: Forward Model weights to predict state given state-action pair use these to predict the next state given the chosen action

8. Planning

22. goal actor units agent

23. Planning

26. Discussion - AI context... assumed links explained by learning - reinforcement learning... if no access to full state space - noise... wide “goal hills” will have flat slopes - shortest path... not taken; how to define? - biological plausibility... Sigma-Pi neurons; winner-take-all - to do: embedding... learn state space from sensor input - to do: embedding... let the goal be assigned naturally - to do: embedding... hand-designed planning phases

27. Acknowledgments Collaborators: Jochen Triesch FIAS J-W-Goethe University Frankfurt Stefan Wermter University of Sunderland Mark Elshaw University of Sheffield