1. ADAPT IST-2001-37173 Artificial Development Approach to Presence Technologies 2 nd Review Meeting Munich, June 7-9 th, 2004

2. Project funded by the Future and Emerging Technologies arm of the IST Programme Presence Research Initiative ConsortiumConsortium n Total cost: 1.335.141 € - Community funding: 469.000 € n Project start date: October 1 st, 2002 n Project duration: 36 months n Total cost: 1.335.141 € - Community funding: 469.000 € n Project start date: October 1 st, 2002 n Project duration: 36 months Participant RoleParticipant NameShort NameCountry Coordinator Giulio Sandini DIST – University of GenovaDISTI Partner Rolf Pfeifer University of Zurich, Dept. of Information Technology UNIZHCH Partner Jacquline Nadel UMR7593, CNRS, University Pierre & Marie Curie, Paris CNRSF

3. Project funded by the Future and Emerging Technologies arm of the IST Programme Presence Research Initiative GoalGoal We wish to… …understand the process of building a coherent representation of visual, auditory, haptic, and kinesthetic sensations process  development process  dynamic representation Perhaps, once we “know” how it works, we can “ask” a machine to use this knowledge to elicit the sense of presence

4. Project funded by the Future and Emerging Technologies arm of the IST Programme Presence Research Initiative So, we are asking… How do we represent our world and, in particular, how do we represent the objects we interact with? Our primary mode of interaction with objects is through manipulation, that is, by grasping objects! How do we represent our world and, in particular, how do we represent the objects we interact with? Our primary mode of interaction with objects is through manipulation, that is, by grasping objects!

5. Project funded by the Future and Emerging Technologies arm of the IST Programme Presence Research Initiative Two-pronged approach n Study how infants do it n Implement a “similar” process in an artificial system n Study how infants do it n Implement a “similar” process in an artificial system Learning by doing: modeling  abstract principles  build new devices

6. Project funded by the Future and Emerging Technologies arm of the IST Programme Presence Research Initiative Scientific prospect n From the theoretical point of view: –Studying the nature of “representation” n From development: developmental path –Interacting with objects: multi-sensory representation, object affordances –Interpreting others/object interaction: imitation n From embodiment and morphology –Why do we need a body? How morphology influences/supports computation? n Computational architecture –How can an artificial system learn representations to support similar behaviors? n From the theoretical point of view: –Studying the nature of “representation” n From development: developmental path –Interacting with objects: multi-sensory representation, object affordances –Interpreting others/object interaction: imitation n From embodiment and morphology –Why do we need a body? How morphology influences/supports computation? n Computational architecture –How can an artificial system learn representations to support similar behaviors?

7. Project funded by the Future and Emerging Technologies arm of the IST Programme Presence Research Initiative Vision Touch Streri & Gentaz (2003, 2004) Reversible cross-modal transfer between hand and eyes in newborn infants Transfer of shape is not reversible

8. Project funded by the Future and Emerging Technologies arm of the IST Programme Presence Research Initiative 6-month-olds detect a violation of intermodality between face and voice A teleprompter device allows to delay independently voice or image

9. Project funded by the Future and Emerging Technologies arm of the IST Programme Presence Research Initiative Grasping: morphological computation Robot hand with: - elastic tendons - soft finger tips (developed by Hiroshi Yokoi, AI Lab, Univ. of Zurich and Univ. of Tokyo) Result: - control of grasping - simple “close” - details: taken care of by morphology/materials

10. Project funded by the Future and Emerging Technologies arm of the IST Programme Presence Research Initiative VideoVideo

11. …how can the robot grasp an unknown object ? n Use a simple motor synergy to flex the fingers and close the hand n Exploit the intrinsic elasticity of the hand; the fingers bend and adapt to the shape of the object n Use a simple motor synergy to flex the fingers and close the hand n Exploit the intrinsic elasticity of the hand; the fingers bend and adapt to the shape of the object

12. Project funded by the Future and Emerging Technologies arm of the IST Programme Presence Research Initiative Result of clustering n 2D Self Organizing Map (100 neurons) n Input: proprioception (hand posture, touch sensors were not used) n 2D Self Organizing Map (100 neurons) n Input: proprioception (hand posture, touch sensors were not used) The SOM forms 7 classes (6 for the objects plus 1 for the no-object condition)

13. Project funded by the Future and Emerging Technologies arm of the IST Programme Presence Research Initiative Example: learning visual features n Only one modality (non-overlapping areas of visual field) guide feature extraction of each other n Learn invariant features from spatial context (it is well known that temporal context can be used for learning these features) n Only one modality (non-overlapping areas of visual field) guide feature extraction of each other n Learn invariant features from spatial context (it is well known that temporal context can be used for learning these features)

14. Project funded by the Future and Emerging Technologies arm of the IST Programme Presence Research Initiative

15. Future work n Continue and complete ongoing experiments n Experiment on affordant vs. non-affordant use of objects (CNRS, UGDIST) n Investigation on cross-modal transfer in newborn infants (CNRS) n Experiments on the robot (UGDIST, UNIZH) – Learning affordances – Learning visuo-motor features by unsupervised learning n Feature extraction on videos showing mother- infant interaction n Continue and complete ongoing experiments n Experiment on affordant vs. non-affordant use of objects (CNRS, UGDIST) n Investigation on cross-modal transfer in newborn infants (CNRS) n Experiments on the robot (UGDIST, UNIZH) – Learning affordances – Learning visuo-motor features by unsupervised learning n Feature extraction on videos showing mother- infant interaction

16. Project funded by the Future and Emerging Technologies arm of the IST Programme Presence Research Initiative Epirob04Epirob04 Genoa – August 25-27, 2004 http://www.epigenetic-robotics.org Invited speakers: Luciano Fadiga Dept. of Biomedical Sciences, University of Ferrara, Italy Claes von Hofsten Dept. of Psychology, University of Upssala, Sweden Jürgen Konczak Human Sensorimotor Control Lab, University of Minnesota, USA Jacqueline Nadel CNRS, University Pierre & Marie Curie, Paris, France