PETRA 2014 An Interactive Learning and Adaptation Framework for Socially Assistive Robotics: An Interactive Reinforcement Learning Approach Konstantinos Tsiakas, Ph.D. Candidate HERACLEIA Human-Centered Computing Laboratory Computer Science and Engineering Department University of Texas at Arlington, USA Institute of Informatics and Telecommunications National Center for Scientific Research, Demokritos, Greece December 2016

Outline Introduction Socially Assistive Robotics PETRA 2014 Outline Introduction Socially Assistive Robotics Methodology and Approach Preliminary Work Issues and Challenges Robot Behavior Modeling Use Case Preliminary Experiments Motivation Interactive Learning and Adaptation Framework Description Experimental Results Conclusion Ongoing Work Research Plan Discussion

Socially Assistive Robotics PETRA 2014 Socially Assistive Robotics Socially Assistive Robotics (SAR) is an HRI research area that studies how robots can assist users during a cognitive or physical task through social interaction (Feil & Mataric 2005) Providing an adaptive and personalized interaction maximizes user engagement, motivation and thus performance (Fasola & Mataric 2010) Adaptation and personalization can be achieved through task parameters adjustment (e.g., difficulty) and through robot behavior (feedback, encouragement, etc.) A key challenge is to develop proper computational methods for personalization and adaptation in real-time, ensuring efficiency and safety

Socially Assistive Robotics PETRA 2014 Socially Assistive Robotics

Methodology and Approach PETRA 2014 Methodology and Approach Reinforcement Learning Reinforcement Learning (RL) provides an appropriate framework for behavior modeling and optimization An RL agent learns behavior through trial-and-error interactions with its environment Markov Decision Processes (MDP) MDP is defined as a tuple <S, A, T, R> Optimal policy π: S → A π* = argmaxaQ(s,a) environment action reward state

Preliminary Work PETRA 2014 K. Tsiakas et. al,A Multimodal Adaptive Rehabilitation Session Manager for Physical Exercising", In PETRA 2015.

Robot Behavior Modeling PETRA 2014 Robot Behavior Modeling

PETRA 2014 Learning Experiments

PETRA 2014 Transfer Experiments

Motivation Issues and Challenges PETRA 2014 Motivation Issues and Challenges RL agents require many iterations and interaction data Online learning methods (e.g., exploration) may be inappropriate for HRI Human → Dynamic environment Long-term adaptation Short-term adaptation Co-adaptation Interactive Learning and Adaptation Framework Integrates Interactive Reinforcement Learning techniques to facilitate the adaptation of a learning agent to each user, ensuring efficacy and safety Allows the agent to dynamically adapt during the interaction Allows human users to shape agent’s policy through feedback and guidance (implicitly and/or explicitly) Supports the participation of an external supervisor that can guide the system during its early interaction steps

Interactive Learning and Adaptation PETRA 2014 Interactive Learning and Adaptation K. Tsiakas et. al “Adaptive Robot Assisted Therapy using Interactive Reinforcement Learning", In ICSR 2016.

Interactive Learning and Adaptation PETRA 2014 Interactive Learning and Adaptation Teaching on a budget (Torrey et al. 2013) Q-augmentation (Chanel et al. 2008)

PETRA 2014 Experimental Results

PETRA 2014 Ongoing Work

PETRA 2014 Ongoing Work DEMO: https://youtu.be/TfL3BITWzG0

Research Plan Thesis Proposal PETRA 2014 Research Plan Thesis Proposal Publish HRI dataset and analysis (HRI conference) EEG analysis Correlation between performance, difficulty, feedback and engagement values → adaptive and personalized training Reinforcement Learning Train offline with learned user model Repeat interactive learning and adaptation experiments Develop supervisor GUI Visualize state uncertainty and importance -- Active Learning HRI User study -- Adaptive vs. Non-Adaptive Training Session Journal of Social Robotics

PETRA 2014 Related Publications K. Tsiakas, M. Abujelala, A. Lioulemes, F. Makedon, "An Intelligent Interactive Learning and Adaptation Framework for Robot-Based Vocational Training", in IEEE Symposium Series in Computational Intelligence,C2RAT, SSCI 2016 K. Tsiakas, M. Dagioglou, V. Karkaletsis, F. Makedon., "Adaptive Robot Assisted Therapy using Interactive Reinforcement Learning", in International Conference on Social Robotics, ICSR 2016 K. Tsiakas, M. Papakostas, B. Cheeba, D. Ebert,. V. Karkaletsis, F. Makedon., "An Interactive Learning and Adaptation Framework for Adaptive Robot Assisted Therapy", In Proceedings of the 9th International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2016. K.Tsiakas, C. Abellanoza, F. Makedon, "Interactive Learning and Adaptation for Robot Assisted Therapy for People with Dementia", In Proceedings of the 9th International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2016. BEST STUDENT PAPER AWARD K. Tsiakas, "Facilitating Safe Adaptation of Learning Agents using Interactive Reinforcement Learning", Student Consortium, Intelligent User Interfaces, ACM IUI 2016 K. Tsiakas, M. Huber and F. Makedon. "A Multimodal Adaptive Rehabilitation Session Manager for Physical Exercising", In Proceedings of the 8th International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2015.

PETRA 2014 Questions Contact konstantinos.tsiakas@mavs.uta.edu