1. Human-Robot Interaction
Doctoral School – Robotics Program
Autonomous Robots Class
Human-Robot Interaction
Robots in education, therapy and rehabilitation
Aude G Billard
Learning Algorithms and Systems Laboratory - LASA
EPFL, Swiss Federal Institute of Technology
Lausanne, Switzerland
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

2. A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Overview of the Class
9h15-12h00
Interfaces and interaction modalities
non-verbal cues and expressiveness in interactions: gesture,
posture, social spaces and facial expressions
User-centred design of social robots: humanoids, androids, etc.
motivations and emotions in robots
social intelligence for robots
Social learning and skill acquisition via teaching and imitation
14h15-17h00:
Robots in education, therapy and rehabilitation
Evaluation methods and methodologies for HRI research
Ethical issues in human-robot interaction research
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

3. Robots in education, therapy and rehabilitation
Current real-life applications of HRI are in Education, Therapeutic and Rehabilitation
These applications rely often on toy-like robots that are small, and, as such convey a sense of security and cuteness
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

4. A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Robots in Therapy
Use for the disabled (autistic children, paraplegics, etc)
Use for the elderly
Rationale:
Aging population is increasing  loneliness of the elderly
Sick child or elderly person in a home  loneliness of the sick
a toy robot replaces a domestic animals in environments where those cannot be accepted (allergies)
petting the toy robot will improve the patient’s mood, fulfill its need to care and be cared for and enhance interaction and communication across patients (a mediator)
a robot allows to follow the patient’s development in longitudinal studies
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

5. A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Robots in Therapy
Paro, the Seal Robot (AIST, Japan)
8 actuators; 2 for upper and lower eyelids,
1 for rotation of eyes, 2 for the neck,
1 for each front fin, and 1 for two rear fins.
Paro weights about 2.8 kg.
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

6. A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Robots in Therapy
Paro has internal states that correspond to emotions.
Each state activity changes as a function of the interactions and otherwise decays with time. Interaction changes the internal states of Paro and, thus, the user creates the character of Paro.
Reinforcement learning: positive for stroking, negative for beating.
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

7. A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Robots in Therapy
Tests: 2-15 years old children – hospitalized; Paro robot was presented 3 time a day over the course of 11 days
Results:
The children’s moods improved on interaction with Paro, encouraging the
children to communicate with each other and caregivers.
In one striking instance, a young autistic patient recovered his appetite and his speech abilities during the weeks when Paro was at the hospital.
In another case, a hopitalized child patient who felt pain when she moved her body, arms, and legs and could not move from her bed. When Paro was given to her, she smiled and was willing to stroke Paro.
Paro had a rehabilitative function as well as a mental effect.
A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Wada, K.; Shibata, T.; Saito, T.; Tanie, K, Effects of robot-assisted activity for elderly people and nurses at a day service center, Proceedings of the IEEE, Volume 92,  Issue 11,  Nov Page(s):1780 – 1788.

8. A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Robots in Therapy
Tests: Elderly people with slight dementia; Paro robot was presented 3 day a week over the course of 5 weeks.
Results:
Paro improved the mood of elderly people and made them more active.
Paro encouraged elderly people to communicate, both with one another and with the nursing staff.
Urinary tests showed that the ability to recover from stress was improved in the elderly.
A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Wada, K.; Shibata, T.; Saito, T.; Tanie, K, Effects of robot-assisted activity for elderly people and nurses at a day service center, Proceedings of the IEEE, Volume 92,  Issue 11,  Nov Page(s):1780 – 1788.

9. Robota: A test of understanding self- agency
Robots for Therapy
Robota: A test of understanding self- agency
Collaboration with Prof. Jaqueline Nadel Hopital de la Salpetriere, Paris, France
Robota allows the child to understand that the robot’s movements originate
from his own movement (sense of agency) and are limited to a restricted
category of movements (intentional selection of his own action )
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

10. A tool to test the imitation capabilities of autistic children
Robots for Therapy
A tool to test the imitation capabilities of autistic children
Collaboration with Prof. Jaqueline Nadel Hopital de la Salpetriere, Paris, France
The educator demonstrates
The imitation game
The autistic child is let free to
play with the robot
Billard, A., Robins, B, Dautenhahn, K. and Nadel, J (2006) Building Robota, a Mini-Humanoid Robot for the Rehabilitation of Children with Autism. the RESNA Assistive Technology Journal. Vol. 19, Issue 1.
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

11. A tool to test and train the interactivity of autistic children
Robots for Therapy
A tool to test and train the interactivity of autistic children
Would the human-appearance affect the child’s interaction with the robot?
Children are much more responsive toward non-human looking doll
Robins, B., Dautenhahn, K., te Boekhorst, R. Billard, A. (2004) Effects of repeated exposure of a humanoid robot on children with autism In S.Keats, J.Clarckson, P. Langdon and P.Robinson (eds), Designing a More Inclusive World. London: Springer Verlag,
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

12. A tool to test and train the interactivity of autistic children
Robots for Therapy
A tool to test and train the interactivity of autistic children
Four autistic children, age 5-10, from the Enhanced Provision unit. Each child participated in about 13 trials on average.
Autistic children interact much more with the non-human looking robot
Robins, B., Dautenhahn, K., te Boekhorst, R. Billard, A. (2004) Effects of repeated exposure of a humanoid robot on children with autism In S.Keats, J.Clarckson, P. Langdon and P.Robinson (eds), Designing a More Inclusive World. London: Springer Verlag,
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

13. A tool to test and train the interactivity of autistic children
Robots for Therapy
A tool to test and train the interactivity of autistic children
Collaboration with Prof. Kerstin Dautenhahn,
University of Hertfordshire, London, UK
Robot as a mediator, an object of joint attention
A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Robins, B et al. (2004) London: Springer Verlag,

14. Eye-Gaze levels of all children
Robots for Therapy
A tool to test and train the interactivity of autistic children
Eye-Gaze levels of all children
A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Robins, B., Dautenhahn, K., te Boekhorst, R. Billard, A. Springer Verlag, 2004.

15. Robots for Education ROBOTA at the Museum
French National Science Museum , « La Cite de l’Espace », Paris,
November 2001 – January 2003
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

16. Robots for Education ROBOTA at the Museum
French National Science Museum, Toulouse, May 2004 – December 2006
Fondation Verdan, Lausanne, November 2004-March 2006
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

17. A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Robots for Education
ROBOTA at the Museum
Monitored the interactions of the users with Robota over 6 months
- Ran from 10am until 6pm every day
30 to 100 users per day (mean 43, std 13)
Interactions lasted between 2secunds and 5 minutes (mean 35 sec, std, 1.15min)
Wide range of nationalities (french, english, german, spanish, dutch, swedish)
Type of interactions:
~45% (body words), ~20%( naturalistic), ~35% (insulting or abusive language)
 Example of naturalistic trainings:
« Drink », « Have an Apple », «  Give me a kiss», «  sing »
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

18. Expressive Body Movements
Keepon (Kozima’s group, CRL, Japan):
Very simple but powerful design to convey joint attention and turn taking behavior
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

19. Expressive Body Movements
Attentive action Directing the head up/down and left/right so as to orient Keepon's face/body to a certain target in the environment. Keepon seems to be perceiving the target. This action includes eye-contact and joint attention.
Emotive action Keeping its attention in a certain direction, Keepon rocks its body from side to side and/or bobs its body up and down. Keepon seems to express emotions (like pleasure and excitement) about the target.
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

20. Progressive exploration by a 2-year old child
Robots for Therapy
Progressive exploration by a 2-year old child
Kozima, H.; Nakagawa, C.; Yasuda, Y.; Interactive robots for communication-care: a case-study in autism therapy Robot and Human Interactive Communication, ROMAN IEEE International Workshop on 3-15 Aug Page(s):
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

21. A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Robots for Therapy
For more than a year and half (over 500 child-sessions), observed a group of children with autism, PDD, Asperger's syndrome, Down's syndrome, and other developmental disorders;
Recorded live interactions between Keepon and the children from the perspective of Keepon
Child seen from Keepon's eyes
Kozima, H.; Nakagawa, C.; Yasuda, Y.; Interactive robots for communication-care: a case-study in autism therapy Robot and Human Interactive Communication, ROMAN IEEE International Workshop on 3-15 Aug Page(s):
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

22. A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Robots for Therapy
For more than a year and half (over 500 child-sessions), observed a group of children with autism, PDD, Asperger's syndrome, Down's syndrome, and other developmental disorders;
Recorded live interactions between Keepon and the children from the perspective of Keepon
In the longitudinal observations, the children showed various actions in relation to Keepon.
Sometimes they showed vivid facial expressions that even their parents had not seen before.
They also showed prosocial actions like trying to feed Keepon, putting on a cap on its head, and kissing it.
Kozima, H.; Nakagawa, C.; Yasuda, Y.; Interactive robots for communication-care: a case-study in autism therapy Robot and Human Interactive Communication, ROMAN IEEE International Workshop on 3-15 Aug Page(s):
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

23. A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Robots for Therapy
Case 1 . Emergence of Dyadic Interaction
The first case is a girl M with Kanner-type autism and mental retardation (MA 2:6 at CA 5:0; no apparent language). We describe the first 15 sessions, which lasted about 5 months (CA 3:9.4:1):
M showed strong interest from Session 1 (hereafter S1), but did not get close to Keepon. Through S1 to S7, M avoided being looked straight at by Keepon (i.e., aversion to eye-contact); however M often looked into Keepon's profile.
In S5, after observing another child R put a paper cylinder on Keepon's head, M dragged the nurse's arm, asking her to do the same thing to Keepon. When the nurse completed the task, M looked satisfied and left from Keepon. Through S5 to S10, the distance to Keepon gradually got shorter (to 40.50cm).
In S11,M touched the head of Keepon with a xylophone stick, then M directly touched with her hand as if M had examined the texture and smell of Keepon. After this first touch, M started social interactions including eye-contact and vocalization (S12.), putting a cap on Keepon, asking her mother to do the same (S13.), and hugging and kissing Keepon (S14).
We can see here the emergence of both spontaneous dyadic interactions, such as touching Keepon with a xylophone stick, and interpersonally triggered dyadic interaction, such as putting a paper cylinder on Keepon. Especially, the latter suggests that M was a good observer of others' behavior, while she seldom imitate when being instructed. Because R's action was mediated by Keepon and an object (e.g., the paper cylinder), which were being of N's interest, it would be relatively easy for N to emulate the same result.
Interactive robots for communication-care: a case-study in autism therapy Kozima, H.; Nakagawa, C.; Yasuda, Y.; Robot and Human Interactive Communication, ROMAN IEEE International Workshop on Aug Page(s):
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

24. A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Robots for Therapy
Case 2 . Emergence of Triadic Interaction
The second case is a 3-year-old girl N with Kannertype autism with mental retardation at a medium level (no language). We observed her interact with Keepon for totally 39 sessions, which lasted about 18 months: In S1, N gazed at Keepon for a long time. After observing another child W played with Keepon using a toy, N was encouraged to play with Keepon using the same toy, but N did not show interest in doing that. Through S2 to S14, N did not pay attention to Keepon, even when she was next to it. However, N often glanced at Keepon, when she noticed its sound, such as .Pong pong pong.
In S15, after observing another child R put a cap on Keepon's head, N touched Keepon with her finger.
In S16 (after 3-month blank from S15), N came close to Keepon and looked into its movement. In the snack time, N came to Keepon and poked its nose, to which Keepon responded by bobbing, and N showed surprise and smile . the mothers and nurses burst into laughter. During this play, N often made referential looks with smile to her mother and the nurse.
From S17, N often sat in front of Keepon with her mother; sometimes she touched Keepon to derive some response. From S20, N started exploring Keepon's ability by walking around it to see if it could follow her.
In the snack time of S33, N came to Keepon and started imitation play. its movements . when N performed one movement (bobbing, rocking, or bowing), soon Keepon mimicked; then N did another, and Keepon did the same. Through S33 to S39, N often played this imitation game. with Keepon, during which she often made referential looks to her mother and the nurse.
Interactive robots for communication-care: a case-study in autism therapy Kozima, H.; Nakagawa, C.; Yasuda, Y.; Robot and Human Interactive Communication, ROMAN IEEE International Workshop on Aug Page(s):
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

25. A.G. Billard, Autonomous Robots Class - EDPR/EDIC
Robots for Therapy
Keepon elicited curiosity in children showing difficulty in interpersonal communication (especially, those with autism and PDD); its small size seem to elicit a sense of security.
Some of the children extended their dyadic interaction with Keepon into triadic inter-personal interaction, where they tried to share with others the pleasure and surprise they found in Keepon.
Each child showed a different style of interaction that changed over time; this provides information on the child’s personality and developmental profile, that is not thoroughly explained by a diagnostic label such as autism..
Interactive robots for communication-care: a case-study in autism therapy Kozima, H.; Nakagawa, C.; Yasuda, Y.; Robot and Human Interactive Communication, ROMAN IEEE International Workshop on Aug Page(s):
A.G. Billard, Autonomous Robots Class - EDPR/EDIC

26. A.G. Billard, Autonomous Robots Class - EDPR/EDIC
SUMMARY
Promising results have been achieved in the use of toy-like robots for therapy with autistic children, sick children and with elderly people.
More studies need to be conducted to determine the level of acceptability of these toys by a large population;
Most of these toys are full-fledged prototypes ready to be commercialized.
Next step is to move to bigger robots, for use with normal adults in our everyday life.
A.G. Billard, Autonomous Robots Class - EDPR/EDIC