1. PETRA 2014
An Interactive Learning and Adaptation Framework for Socially Assistive Robotics: An Interactive Reinforcement Learning Approach
Konstantinos Tsiakas, Ph.D. Candidate
HERACLEIA Human-Centered Computing Laboratory
Computer Science and Engineering Department
University of Texas at Arlington, USA
Institute of Informatics and Telecommunications
National Center for Scientific Research, Demokritos, Greece
December 2016

2. Outline Introduction Socially Assistive Robotics
PETRA 2014
Outline
Introduction
Socially Assistive Robotics
Methodology and Approach
Preliminary Work
Issues and Challenges
Robot Behavior Modeling
Use Case
Preliminary Experiments
Motivation
Interactive Learning and Adaptation Framework
Description
Experimental Results
Conclusion
Ongoing Work
Research Plan
Discussion

3. Socially Assistive Robotics
PETRA 2014
Socially Assistive Robotics
Socially Assistive Robotics (SAR) is an HRI research area that studies how robots can assist users during a cognitive or physical task through social interaction (Feil & Mataric 2005)
Providing an adaptive and personalized interaction maximizes user engagement, motivation and thus performance (Fasola & Mataric 2010)
Adaptation and personalization can be achieved through task parameters adjustment (e.g., difficulty) and through robot behavior (feedback, encouragement, etc.)
A key challenge is to develop proper computational methods for personalization and adaptation in real-time, ensuring efficiency and safety

4. Socially Assistive Robotics
PETRA 2014
Socially Assistive Robotics

5. Methodology and Approach
PETRA 2014
Methodology and Approach
Reinforcement Learning
Reinforcement Learning (RL) provides an appropriate framework for behavior modeling and optimization
An RL agent learns behavior through trial-and-error interactions with its environment
Markov Decision Processes (MDP)
MDP is defined as a tuple <S, A, T, R>
Optimal policy π: S → A
π* = argmaxaQ(s,a)
environment
action
reward
state

6. Preliminary Work PETRA 2014
K. Tsiakas et. al,A Multimodal Adaptive Rehabilitation Session Manager for Physical Exercising", In PETRA 2015.

7. Robot Behavior Modeling
PETRA 2014
Robot Behavior Modeling

8. PETRA 2014
Learning Experiments

9. PETRA 2014
Transfer Experiments

10. Motivation Issues and Challenges
PETRA 2014
Motivation
Issues and Challenges
RL agents require many iterations and interaction data
Online learning methods (e.g., exploration) may be inappropriate for HRI
Human → Dynamic environment
Long-term adaptation
Short-term adaptation
Co-adaptation
Interactive Learning and Adaptation Framework
Integrates Interactive Reinforcement Learning techniques to facilitate the adaptation of a learning agent to each user, ensuring efficacy and safety
Allows the agent to dynamically adapt during the interaction
Allows human users to shape agent’s policy through feedback and guidance (implicitly and/or explicitly)
Supports the participation of an external supervisor that can guide the system during its early interaction steps

11. Interactive Learning and Adaptation
PETRA 2014
Interactive Learning and Adaptation
K. Tsiakas et. al “Adaptive Robot Assisted Therapy using Interactive Reinforcement Learning", In ICSR 2016.

12. Interactive Learning and Adaptation
PETRA 2014
Interactive Learning and Adaptation
Teaching on a budget (Torrey et al. 2013)
Q-augmentation (Chanel et al. 2008)

13. PETRA 2014
Experimental Results

14. PETRA 2014
Ongoing Work

15. PETRA 2014
Ongoing Work
DEMO:

16. Research Plan Thesis Proposal
PETRA 2014
Research Plan
Thesis Proposal
Publish HRI dataset and analysis (HRI conference)
EEG analysis
Correlation between performance, difficulty, feedback and engagement values → adaptive and personalized training
Reinforcement Learning
Train offline with learned user model
Repeat interactive learning and adaptation experiments
Develop supervisor GUI
Visualize state uncertainty and importance -- Active Learning
HRI
User study -- Adaptive vs. Non-Adaptive Training Session
Journal of Social Robotics

17. PETRA 2014
Related Publications
K. Tsiakas, M. Abujelala, A. Lioulemes, F. Makedon, "An Intelligent Interactive Learning and Adaptation Framework for Robot-Based Vocational Training", in IEEE Symposium Series in Computational Intelligence,C2RAT, SSCI 2016
K. Tsiakas, M. Dagioglou, V. Karkaletsis, F. Makedon., "Adaptive Robot Assisted Therapy using Interactive Reinforcement Learning", in International Conference on Social Robotics, ICSR 2016
K. Tsiakas, M. Papakostas, B. Cheeba, D. Ebert,. V. Karkaletsis, F. Makedon., "An Interactive Learning and Adaptation Framework for Adaptive Robot Assisted Therapy", In Proceedings of the 9th International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2016.
K.Tsiakas, C. Abellanoza, F. Makedon, "Interactive Learning and Adaptation for Robot Assisted Therapy for People with Dementia", In Proceedings of the 9th International Conference on PErvasive Technologies Related to Assistive Environments, PETRA BEST STUDENT PAPER AWARD
K. Tsiakas, "Facilitating Safe Adaptation of Learning Agents using Interactive Reinforcement Learning", Student Consortium, Intelligent User Interfaces, ACM IUI 2016
K. Tsiakas, M. Huber and F. Makedon. "A Multimodal Adaptive Rehabilitation Session Manager for Physical Exercising", In Proceedings of the 8th International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2015.

18. PETRA 2014
Questions
Contact