1. MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra, Jürgen Schmidhuber IDSIA - Lugano - Switzerland

2. MindRACES, First Review Meeting, Lund, 11/01/2006 2 Overview Real world scenarios Fovea Software Framework Robertino Robot Simulated 3D scenarios Integration and Future Work

3. MindRACES, First Review Meeting, Lund, 11/01/2006 3 Instances of scenario 1: find an object Still Image Robot Lab Office Room Movie (LUCS) 2D Environments 3D Environments

4. MindRACES, First Review Meeting, Lund, 11/01/2006 4 Fovea Centralis Simulation Three regions with increasing resolutions

5. MindRACES, First Review Meeting, Lund, 11/01/2006 5 Fovea Centralis Simulation Three regions with increasing resolutions Subsample each region to 13x11 pixels

6. MindRACES, First Review Meeting, Lund, 11/01/2006 6 Fovea Centralis Simulation Three regions with increasing resolutions Subsample each region to 13x11 pixels Combine the subsampled image

7. MindRACES, First Review Meeting, Lund, 11/01/2006 7 ??? ???? ????? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley BinsObjects 1-D world Transformation Fovea Memory State 0

8. MindRACES, First Review Meeting, Lund, 11/01/2006 8 ??? ???? ????? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 1 BinsObjects 1-D world Transformation Fovea Memory

9. MindRACES, First Review Meeting, Lund, 11/01/2006 9 ??? ???? ????? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 2 BinsObjects 1-D world Transformation Fovea Memory

10. MindRACES, First Review Meeting, Lund, 11/01/2006 10 ??? ???? ????? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 3 BinsObjects 1-D world Transformation Fovea Memory

11. MindRACES, First Review Meeting, Lund, 11/01/2006 11 ???? ?? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 4 BinsObjects 1-D world Transformation Fovea Memory

12. MindRACES, First Review Meeting, Lund, 11/01/2006 12 ???? ?? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 5 BinsObjects 1-D world Transformation Fovea Memory

13. MindRACES, First Review Meeting, Lund, 11/01/2006 13 ??? ?? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 6 BinsObjects 1-D world Transformation Fovea Memory

14. MindRACES, First Review Meeting, Lund, 11/01/2006 14 ??? ?? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 7 BinsObjects 1-D world Fovea Memory Transformation

15. MindRACES, First Review Meeting, Lund, 11/01/2006 15 Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 8 BinsObjects 1-D world Fovea Memory Transformation

16. MindRACES, First Review Meeting, Lund, 11/01/2006 16 Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 9 BinsObjects 1-D world Fovea Memory Transformation

17. MindRACES, First Review Meeting, Lund, 11/01/2006 17 Simulated Vision: abstract 2-D Simulation Extended 1-D Simulation Only the centered sensor can differentiate between objects

18. MindRACES, First Review Meeting, Lund, 11/01/2006 18 Simulated Vision: 2-D Simulation Original image with 2 objects

19. MindRACES, First Review Meeting, Lund, 11/01/2006 19 Simulated Vision: 2-D Simulation Original image with 2 objects Simulated fovea images Center of the fovea Step 1

20. MindRACES, First Review Meeting, Lund, 11/01/2006 20 Simulated Vision: 2-D Simulation Original image with 2 objects Simulated fovea images Center of the fovea Detected!!! Step 1Step n

21. MindRACES, First Review Meeting, Lund, 11/01/2006 21 Software Framework Server Robertino Debian/GNU Linux operating system CAN bus interface FireWire (IEEE 1394) interface Video4Linux library Fovea simulation Client Robosim Linux or Windows Ogre framework Fovea simulation Simple physical and collision detection system Same interface as the real robot Robomon Linux or Windows Experiment management Interface for learning algorithms Full remote control of the robot/simulation TCP/IP Previously recorded data

22. MindRACES, First Review Meeting, Lund, 11/01/2006 22 Robertino - Software Framework Linux CANV2L Local API (network, sensors, actuators) Remote control servers Local Application Remote control client API Remote Application Fovea implementation

23. MindRACES, First Review Meeting, Lund, 11/01/2006 23 Robosim - Software Framework Ogre 3D simulation framework on various operation systems Robot simulation with specific environment Remote control servers Local Application Remote control client API Remote Application Fovea implementation

24. MindRACES, First Review Meeting, Lund, 11/01/2006 24 Robertino – Client Software Manual control of the robot Learning control interface Fovea image

25. MindRACES, First Review Meeting, Lund, 11/01/2006 25 Robertino - Overview Diameter: 40 cm Height: 43 cm Weight: 6.5 kg. Holonomic three wheeled drive PC-103 (industry standard) with a 500MHz Intel Mobile- Pentium II processor on-board WLAN (IEEE 802.11a) 2 cameras, used to simulate the fovea Actuators: the three wheels and the simulated fovea www.openrobertino.org

26. MindRACES, First Review Meeting, Lund, 11/01/2006 26 Omnidirectional Camera Web cam Omnidirectional mirror Image of an office environment, as seen by the robot

27. MindRACES, First Review Meeting, Lund, 11/01/2006 27 Image Transformation Used only for human convenience for navigation Simple transformation algorithm

28. MindRACES, First Review Meeting, Lund, 11/01/2006 28 High Quality Camera Imaging Source DFK21 AF04Image of an office environment, as seen by the robot

29. MindRACES, First Review Meeting, Lund, 11/01/2006 29 Robot Lab Environment Robot lab 550 cm x 280 cm Robot can navigate through the world and move the fovea Robot can be observed from a top view camera and local camera

30. MindRACES, First Review Meeting, Lund, 11/01/2006 30 Robot Lab Environment Camera image Fovea dataComposition No video for his presentation!

31. MindRACES, First Review Meeting, Lund, 11/01/2006 31 3D Simulation of the Robot Environment design in 3D Studio MAX or G-MAX. Simulation in Ogre3D Same objects as in the real world Shadows (optional)

32. MindRACES, First Review Meeting, Lund, 11/01/2006 32 Simulated vs. Real Vision (Fovea) Simulation Real Robot Fovea transformation brings both world more together Artificial noise simulates camera noise (optional)

33. MindRACES, First Review Meeting, Lund, 11/01/2006 33 Integration and Future Work Develop and evaluate anticipatory learning algorithms in the simulated environments (UW, ÖFAI); compare them also to non- anticipatory ones Share scenarios for attentive vision (LUCS, NBU) Real robot with simulation as anticipation for surprise studies (CNR) Simulation-based learning of anticipation (UW) Share fovea simulation (IST) Evaluate transfer of learned behavior to the real robot Systematically increase the complexity of simulated and real environments Possibly use a movable camera and zoom lens mounted on the robot

34. MindRACES, First Review Meeting, Lund, 11/01/2006 34 Conclusion Learn to use a fovea for searching objects Objects possibly occluded or partially invisible Several real-world and simulated scenarios developed for training First experiments with simulated 1D and 2D worlds successful Starting with 3D experiments

35. MindRACES, First Review Meeting, Lund, 11/01/2006 35 Questions?