1. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Humanoids, Brain and Cognitive Sciences Giulio Sandini Italian Institute of Technology Robotics, Brain and Cognitive Sciences Dept. and LIRA-Lab, University of Genoa Information and Communication Technology Research Forum 2009 Khalifa University of Science, Technology and Research April 5, 2009 – Abu Dhabi

2. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Humanoids may have to do with “automation”, service robotics, rescue and intervention but what do they have to do with “communication technologies”?

3. Robotics, Brain and Cognitive Sciences Department Giulio Sandini I am going to argue that communication is at the very core of humanoid robotics and that research on humanoids can help advancing communication technologies

4. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Communication not in the sense of “transmission” but in the sense of “mutual understanding” and team work.

5. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Today’s humans are operating artificial systems by pushing, pulling, sliding, turning, dragging, and, to a limited extent, speaking and gesturing Artificial systems do not understand (anticipate) the intention of the human operator.

6. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Conversely communication in humans is a process of mutual understanding. At both ends of a communication channel there is always a human being

7. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Why there is still no communication channel involving at least one artificial system at one terminal?

8. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Because to understand each other two agents need to share not only the communication channel but also knowledge and experience (and to some extent also the body shape) The body is human’s communication channel

9. Robotics, Brain and Cognitive Sciences Department Giulio Sandini …humanoid robots could be the equivalent of today’s visual displays in the sense of “conveying information” but with “mutual understanding” …in the future… humanoids as communication terminals (with the added value that humans and humanoids will be able to act together)

10. Robotics, Brain and Cognitive Sciences Department Giulio Sandini..future of Humanoid Robots Copyright: Toyota Motor Corporation – Robot Technologies, 2007 Close interactions with humans supported by natural multimodal communication channels such as gesture, vision, body-language, haptics, speech and natural language …and its value chain

11. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Ubiquitous Robot Companion

12. Robotics, Brain and Cognitive Sciences Department Giulio Sandini

13. Robotics, Brain and Cognitive Sciences Department Giulio Sandini

14. Robotics, Brain and Cognitive Sciences Department Giulio Sandini For this we have to build systems that are able to communicate with humans in a natural way We have to build systems that “understand” what a human is doing and, to some extent, are able to predict what a human will be doing in the immediate future

15. Robotics, Brain and Cognitive Sciences Department Giulio Sandini

16. Robotics, Brain and Cognitive Sciences Department Giulio Sandini

17. Robotics, Brain and Cognitive Sciences Department Giulio Sandini This brings about the issue of “Brain Sciences”: to implement systems that understand what humans are doing requires a lot of “brain research” to go along with “humanoid research”

18. Robotics, Brain and Cognitive Sciences Department Giulio Sandini How does a human being understands what another human being is doing? How does a humanoid understands what a human being is doing? Brain Science Robotics Science Cognitive Science

19. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Humanoids Research Humanoid robotics research today can be seen as a human centered discipline advancing science and developing new technologies along three main streams

20. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Human Centered Technologies 1.Build state-of-the-art humanoids 2.Study humans (i.e. perceptual, motor, learning, communication abilities) 3.Exploit human-machine interaction Build Humanoids Study Humans Interaction Italian Institute of Technology

21. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Why a body?

22. Robotics, Brain and Cognitive Sciences Department Giulio Sandini The chess player… Up to a few years ago the prevalent opinion was that it was possible to study the “human system” by investigating its “sub-systems” separately (visual, auditory, motor etc.) and that it was possible to separate perception and reasoning from the physics and mechanics of the body  the chess player

23. Robotics, Brain and Cognitive Sciences Department Giulio Sandini The if-then-else approach The “mind” as a logical machine: If there is an obstacle in front then turn else go ahead. Is it possible to arrive always of a logical decision? Is it worth to look always for a logical decision? In the real world the possible alternatives may be so numerous that it is not possible to find a “logical” solution in time…

24. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Obstacle Cow-catcher What is an obstacle? Depends on the body What does it means “avoid”? Depends on the body

25. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Our understanding and our decisions depend in many different ways on the body and the actions that our body is able to perform

26. Robotics, Brain and Cognitive Sciences Department Giulio Sandini A point in space is represented by: Where it is In the visual space In the somatosensory (body) space In the auditory space How it can be reached –With hands –With eyes –… …

27. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Object properties can be coded in terms of motor primitives Position of objects can be coded in terms of the action required to reach it Trajectory of objects can be coded in terms of “collision trajectory” (which body part is going to hit) Size can be coded in terms of “grasp type” (small is whatever can be grasped with a pinch grasp) One can express size with gestures: how big it is?

28. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Recognize an object without the object How relevant is “action” in recognizing objects? Can the visual system exploit the view of haptic exploration to extract shape information of the explored object? Study done with Francesco Campanella

29. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Vision as an embodied process Understanding as a perceptual process Understanding with your eyes Understanding as a motor process (goal-driven) Understanding with your bottom A chair is whatever gives a goal to my sitting action

30. Robotics, Brain and Cognitive Sciences Department Giulio Sandini F5 Mirror Neurons Active when another individual is seen performing manipulative gestures From: Fadiga, L., L. Fogassi, V. Gallese, and G. Rizzolatti, Visuomotor Neurons: ambiguity of the discharge or "motor“ Perception? Internation Journal of Psychophysiology, 2000. 35: p. 165-177. SII 7b IT STs F1 F4 AIP VIP Reaching Grasping Object/hand recognition Visual cortex

31. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Understanding actions performed by others means to activate the areas of the brain that are activated when we execute the same actions

32. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Being able (learning) to act is essential to understand actions done by others. Understanding mirror neurons: a bio-robotic approach. G. Metta, G. Sandini, L. Natale, L. Craighero, L. Fadiga. Interaction Studies. Volume 7 Issue 2. 2006

33. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Perception provides the essential information about the context of the action Which are the actions compatible with the current context?

34. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Through the context it is possible to predict the future course of events… Object’s representation also contains “actions”

35. Robotics, Brain and Cognitive Sciences Department Giulio Sandini These multimodal motor representations need to be learned and adapted continuously…

36. Robotics, Brain and Cognitive Sciences Department Giulio Sandini …by active exploration and interaction

37. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Our Guiding Philosophy It is essential to study not only how the system works but also how it is constructed. Cognition is necessarily the product of a process of embodied development (cannot be hand-coded )

38. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Developmental approach

39. Robotics, Brain and Cognitive Sciences Department Giulio Sandini The “process of building” a human being is based on learning thought interaction with the environment and with other human beings Shaping the self Interacting with others

40. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Learning to control gaze 1998

41. Robotics, Brain and Cognitive Sciences Department Giulio Sandini RobotCub U. Genoa IIT Herthfordshire IST - Lisbon U. Salford Telerobot U. Ferrara EPFL - Lausanne U. Uppsala U. Zurich Scuola S. Anna Robotcub is a collaborative 5-years project in the area of “cognitive systems” We are now in the middle of the 4 th year

42. Robotics, Brain and Cognitive Sciences Department Giulio Sandini People Giorgio Metta, David Vernon Lorenzo Natale, Francesco Nori: Software, testing, calibration Marco Maggiali, Marco Randazzo: firmware, DSP libraries, tactile sensing Francesco Becchi, Paolo Pino, Giulio Maggiolo, Gabriele Careddu: design and integration Roberto Puddu, Gabriele Tabbita, Walter Fancellu: assembly Darwin Caldwell Nikos Tsagarakis, William Hinojosa: legs and spine, force/torque sensors Bruno Bonino, Fabrizio Larosa, Claudio Lorini: electronics Davide Dellepiane: wiring Mattia Salvi: CAD maintenance Alberto Zolezzi: managing quotes, orders and spare parts Ricardo Beira, Luis Vargas, Miguel Praca: design of the head and face Paul Fitzpatrick & Alessandro Scalzo: software middleware Alberto Parmiggiani: joint level sensing Alexander Schmitz: fingertips Ravinder Dahiya: FET-PVDF tactile senors Lorenzo Jamone: fingertips Auke Ijspeert and Ludovic Righetti: simulation and initial torque specification

43. Robotics, Brain and Cognitive Sciences Department Giulio Sandini The RobotCub Approach Scientific Stance: Cognition emerges through interactions (a body is essential). Research Methodology: Cognition is best studied through a programme of progressive development (gradual acquisition of a prospective capability) Research Strategy: Global scientific community is best served by creating an open platform and exploiting synergies

44. Robotics, Brain and Cognitive Sciences Department Giulio Sandini ….RobotCub two goals… Advance our understanding of several key issues in cognition Realize a physical platform for embodied cognitive research

45. Robotics, Brain and Cognitive Sciences Department Giulio Sandini The iCub platform The iCub is a full humanoid robot sized as a three and half year-old child. The total height is 104cm. It has 53 degrees of freedom, including articulated hands to be used for manipulation and gesturing. The robot will be able to crawl and sit and autonomously transition from crawling to sitting and vice-versa. The robot is GPL/FDL: software, hardware, drawings, documentation, etc.

46. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Degrees of freedom Head: vergence required + 3 dof neck Arms: 7 dof each –3 shoulder, elbow, 3 wrist Hands: 9 dof each ► 17 joints –5 fingers ► underactuated Legs: 6 dof each –3 hip, knee, 2 ankle Waist: 3 dof Σ = 53 dof

47. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Body cover: concept

48. Robotics, Brain and Cognitive Sciences Department Giulio Sandini It is small…

49. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Sensors… Cameras Skin/tactile Gyros/inertial Microphones 640x480 color - Remote head Very small but otherwise standard Small (0.67mm) Designing force/torque sensor Fingertip 3-axis sensor Joint torque measurement Conductive paint… QTC: rubber-like material Tension sensors? Organic FETs? Standard, condenser electret

50. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Artificial sensorized skin Principle of operation Multiple elements Skin structure By Marco Maggiali: IIT and Giorgio Cannata: UNIGE Patented

51. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Miniaturized electronics Hand sensor sampling PIC- based card Force/torque sensor fitted into the sensor Wired with 25micron coated wires Design and documentation Control cards 4 times smaller than the commercial equivalent Embedded electronics in a 6-axis sensor With: Alexander Schmitz, Lorenzo Natale Electronics: Bruno Bonino, Fabrizio Larosa

52. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Fingertip Outline 3D CAD Electrode fabrication Silicon cover Complete prototype Electronics

53. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Hands are key!

54. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Wiki CVS Part lists Drawings

55. Robotics, Brain and Cognitive Sciences Department Giulio Sandini iCubs in Europe 13 iCubs are on the make

56. Robotics, Brain and Cognitive Sciences Department Giulio Sandini.....and we are planning for more....

57. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Yoga…

58. Robotics, Brain and Cognitive Sciences Department Giulio Sandini …ICub Drummer Ludovic Righetti, Auke Jan Ijspeert: Programmable Central Pattern Generators: an Application to Biped Locomotion Control. ICRA 2006: 1585-1590

59. Robotics, Brain and Cognitive Sciences Department Giulio Sandini ICub helper Peter Ford Dominey Laboratoire d'Etude des Mécanismes Cognitifs (EMC) Equipe Neuroscience Cognitive et Representations Multimodales (NCRM) Université Lumière Lyon 2

60. Robotics, Brain and Cognitive Sciences Department Giulio Sandini ICub Observer Work by: Jonas Ruesch, Manuel Lopes, Alexandre Bernardino, Jonas Hornstein, José Santos Victor, Rolf Pfeifer – IST Lisbon and University of Zurich AI Lab.

61. Robotics, Brain and Cognitive Sciences Department Giulio Sandini …still a long way to go…

62. Robotics, Brain and Cognitive Sciences Department Giulio Sandini The challenges Mindware: –Machine learning with little human intervention (irrespective of the application but tailored to the AI application domain) –To tackle problems like perception and control (and possibly the two together) …vision…(still not solved..  ) –Robust sensorimotor coordination, manipulation, full-body control Bodyware: –Materials: stronger and lighter bones –Actuators: flexible, compliant, impact absorbing –Sensors: e.g. skin, but also more resolution, range, etc. –Computation: the brain of our robot is still bulky, wiring is also a complex problem –Energy: for autonomy –Safety

63. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Conclusions Humanoids and Human research may be mutually supportive Perception is an “action” process Critical mass and international collaboration (exploit body similarity) Communication is a key aspect of humanoid research Body materials will be a key aspect of humanoid research

64. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Thanks!

65. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Form and function are correlated in biological systems Physical growth and morphological change is an essential ingredient of nature’s smart solutions …are we missing something?

66. Robotics, Brain and Cognitive Sciences Department Giulio Sandini “Natural” minds are shaped by interactions

67. Robotics, Brain and Cognitive Sciences Department Giulio Sandini In nature also the bodies are shaped by interactions Lucia Galli, TRENDS in Neuroscience Vol. 25 No. 12 – December 2002 Micromechanical hypothesis of mosaic assembly

68. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Development and learning are associated to morphological changes at the musculoskeletal and at the neuronal level In biological systems body and mind co-develop Is this unique of biological systems? Can we think about “artificial growth”?

69. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Solutions in biological systems are always multi-technological Intelligent manipulation depends as much on how the controller works (mind) as on the physical properties of the muscles Rizzolatti discovery of Mirror Neurons: exploit motor representation for visual recognition Bizzi e Mussa-Ivaldi theory of force fields Exploit visco-elastic properties of muscles SII 7b IT STs F1 F4 AIP VIP Reaching Grasping Object/hand recognition Visual cortex

70. Robotics, Brain and Cognitive Sciences Department Giulio Sandini Exploit co-development of function and physical growth Living materials: living in the sense that their physical characteristics are changed by functions (e.g. they become harder or softer, they change shape, they “grow” differently).

71. Robotics, Brain and Cognitive Sciences Department Giulio Sandini …the missing dimension…? Life science “Robotics” Material Science …so far we have attempted to study the adaptation of the mind but the body is fixed…

72. Robotics, Brain and Cognitive Sciences Department Giulio Sandini