Humanoid Robots as Cooperative Partners for People paper by Breazeal, C., et al.. (2003) MIT Media Lab, Robotic Life Group presentation by Kósa Máté Ágoston cognitive Rijksuniversiteit Groningen 2010

building socially intelligent robots important implications for how we will be able to communicate with, work with, and teach robots in the future it is a critical competence for robots that will play a useful, rewarding, and long-term role in the daily lives of people

socially intelligent robot robots that show aspects of human-like social intelligence, based on deep models of human cognition and social competence contrasted to socially evocative / receptive / situated / embedded brings research closer to the „hard” problem of artificial intelligence (in small steps…) Fong, T., Nourbakhsh, I. & Dautenhahn, K. (2002)

socially intelligent robot Why? – We anthropomorphize by default – Personality lends coherence and consistence to behavior (to know someone is to predict his actions) – Natural learning – Scalability reflects in trust and sincerity (for when it gets out of hand see Blade Runner, Ridley Scott 1982)

theory of mind Assumption: – each participant has a set of mechanisms and representations for predicting and interpreting other’s actions, emotions, beliefs, desires, and other mental states Derived models: – joint attention, representation, empathy, intersubjectivity, reason (mental states to behavior), inference, social reference etc.

Collaborative approach vs. ML supervised learning techniques – the learning algorithm has no a priori knowledge about the structure of the state and action spaces, must discover any structure that exists on its own needs data, time, relatively stable enviroment problems with generalizing hard to guide for the laic bridges machine learning with HMC

Collaborative approach vs. Humans we are innate teachers we have a well-established social signaling we have infrastructure we have an affinity towards interdisciplinarity

Social Skills reciprocal cooperation is achieved with the goal to: – help the instructor maintain a good mental model of the learner – help the learner leverage from instruction and guidance to build the appropriate task models, representations, associations, etc. test of abilities: the button task

Social Skills Communication skill Deictic reference Joint attention Mutual beliefs

Communication Conversational policies – Cohen et. al. (1990) argue that much of task- oriented dialog can be understood in terms of Joint Intention Theory – Modeled after analysis of master-novice task Turn-taking skills – Modeled after human model, very robust – Envelope displays (para-linguistic cues)

Communication Conversational policies – Cohen et. al. (1990) argue that much of task- oriented dialog can be understood in terms of Joint Intention Theory – Modeled after analysis of master-novice task Turn-taking skills – Modeled after human model, very robust – Envelope displays (para-lingvistic cues) - same goal and the same plan of execution - different abilities, tools, partial knowledge and different beliefs referring to the state of the goal - communication is necessary to mobilize the potential - same goal and the same plan of execution - different abilities, tools, partial knowledge and different beliefs referring to the state of the goal - communication is necessary to mobilize the potential

Communication Conversational policies – Cohen et. al. (1990) argue that much of task- oriented dialog can be understood in terms of Joint Intention Theory – Modeled after analysis of master-novice task Turn-taking skills – Modeled after human model, very robust – Envelope displays (para-lingvistic cues) - same goal and the same plan of execution - different abilities, tools, partial knowledge and different beliefs referring to the state of the goal - communication is necessary to mobilize the potential - same goal and the same plan of execution - different abilities, tools, partial knowledge and different beliefs referring to the state of the goal - communication is necessary to mobilize the potential Organizational markers Elaborations Clarifications Confirmations Referential elaborations Confirmations of successful identification Organizational markers Elaborations Clarifications Confirmations Referential elaborations Confirmations of successful identification

Communication Conversational policies – Cohen et. al. (1990) argue that much of task- oriented dialog can be understood in terms of Joint Intention Theory – Modeled after analysis of master-novice task Turn-taking skills – Modeled after human model, very robust – Envelope displays (para-linguistic cues)

Deictic reference Estimating gaze via estimating head-pose – pan / tilt / rotation – objects in 3D spatial map projected on gaze vector – camera on the wall (panoramic view) Pointing – background and depth map extraction – candidates fit to ellipse, then presence of pointing finger is analyzed (kurtosis) – stereo camera ceiling-mounted (bird’s eye view)

Deictic reference Estimating gaze via estimating head-pose – pan / tilt / rotation – objects in 3D spatial map projected on gaze vector – camera on the wall (panoramic view) Pointing – background and depth map extraction – candidates fit to ellipse, then presence of pointing finger is analyzed (kurtosis) – stereo camera ceiling-mounted (bird’s eye view)

Deictic reference Estimating gaze via estimating head-pose – pan / tilt / rotation – objects in 3D spatial map projected on gaze vector – camera on the wall (panoramic view) Pointing – background and depth map extraction – candidates fit to ellipse, then presence of pointing finger is analyzed (kurtosis) – stereo camera ceiling-mounted (bird’s eye view)

Joint Attention – seeing vs. attending (in baby humans 7-9 months) – referential looking (in baby humans 6-18 months) – proto-declarative pointing (in b.h months) – exploiting all these at 14 months of age (in b.h.) two entities looking at the same thing is not necessarily joint attention (necessary-not-suff) updating mutual belief with a common referent is closer to the human-human model

Joint Attention To keep in mind: – Attention focus (what gets the attention) – Referent focus (the “subject” of communication) – Saliency determines a list, not a particular object – perceptual/internal/socially cued saliency – Decay of saliency – Leonardo’s model of own foci – Leonardo’s model of instructor’s foci

Beliefs – humans around the age of 3 note difference between perception and belief temporal integration of perceptual input (composite instances of real-world objects) percept tree > snapshot > belief. classification > data structure > create/update

Beliefs – humans around the age of 3 note difference between perception and belief temporal integration of perceptual input (composite instances of real-world objects) percept tree > snapshot > belief. classification > data structure > create/update when the robot shares a belief with a human, the belief gets labeled as “mutual belief” human’s attentional and referent focus are updated for the belief concerned when the robot shares a belief with a human, the belief gets labeled as “mutual belief” human’s attentional and referent focus are updated for the belief concerned

Learning From “internal” demonstration – telemetry suit  – robot interpolates exemplars using a dynamically weighted blend of the recorded button pressing trajectories Names of things – social cue feedback

Learning Task structure – task is either a (sub)task or an action, hierarchically organized – constraints exist as actions (currently used for sequential constraints but are expandable) – task goals are more than the sum of (sub)task goals – a goal can be either a state-change in world (attain a state) performance (just do it) Natural instruction

Performing in collaboration possible because of the goal-oriented approach (and the turn-taking implementation) communication of robot’s perceived SoW and intention leads to common ground which is the basis of joint intention/attention/planning knowledge of own abilities, negotiation of task with human importance of gestural cues during collaboration

Video time

Discussion Knowing what matters – restraining search-space by saliency – temporal cues + joint attention Knowing what to try – collaboration contrasted with imitation and experiment Knowing how to recognize success/faliure – goal types: change desired/performance, goal progress Knowing how to explore Knowing how to leverage the provided structure – experienced demonstration, mo’ generally social context