Examples of Nonlinear Systems: ROBOTS New Robotic Treatment Systems for Childhood Autism and Cerebral Palsy Joint Work with N. BugnariuA, D. HansonB, F. MakedonC ADepartment of Physical Therapy, University of North Texas Health Sciences Center (UNT HSC) BHanson Robotics Inc., Plano, TX, USA CDepartment of Computer Science & Engineering Department, University of Texas at Arlington, USA NGS Focus: Human-Robot Interaction Ph.D. Students: Isura Ranatunga, Nahum Torres This work was supported by: US National Science Foundation Grants #CPS 1035913, and #CNS 0923494. TxMed consortium grant: “Human-Robot Interaction System for Early Diagnosis and Treatment of Childhood Autism Spectrum Disorders (RoDiCA)”

NGS Robots with HRI and pHRI Two assistive robotic systems aimed at the treatment of children with certain motor and cognitive impairments. In the Neptune project [1] Mobile manipulator for children suffering from Cerebral-Palsy. Mobile robot base and a 6DOF robotic arm, interfaced via: Wii Remote, iPad, Neuroheadset, the Kinect, and Force sensing robotic skin Therapeutic outcomes Hand and head gesture recognition and reward. Hand motion excercises using IPAD Games (CPlay, CPMaze, ProlloquoToGo) held by the robot. The RoDiCA project [2] focuses on treating cognitive impairments in children suffering from ASD Zeno is a robotic platform developed by Hanson Robotics, based on a patented realistic skin. Real time subject tracking/joint attention Advanced head-eye and hand coordination Facial gesture recognition and synthesis Data logging and analysis. Neptune Mobile manipulator with iPad attached. Zeno (by Hanson RoboKind Inc.) generating facial expressions and maintaining eye contact. 9/18/2018 Multiscale Robots and Systems Lab University of Texas Arlington

Advanced Control for Human Robot Interaction Realistic & Intuitive Human-Robot Interaction Physical HRI Recognize & Synthesize poses and gestures Adaptive Interfaces Visual HRI Robot Touch HRI Neptune Control through Neural Headband Zeno Video 9/18/2018 Multiscale Robots and Systems Lab University of Texas Arlington

Neptune: Assistive Robotic System for CP 9/18/2018 Multiscale Robots and Systems Lab University of Texas Arlington

Multiscale Robots and Systems Lab University of Texas Arlington Adaptive Interfaces The supervisory control of multi-DOF robots is a demanding application. If a single operator is tasked with direct control, performing coordinated tasks becomes non-intuitive. We use Reinforcement Learning TD(lambda) scheme in order to adaptively change the mapping of DOF’s from the operator user interface to the robot. State Propagation Interface Mapping System Metrics Evaluation state Interface input action Update Reward function reward Critic Actor TD error Value Function Policy 9/18/2018 Multiscale Robots and Systems Lab University of Texas Arlington

Interface Mapping of Neural Headband to Robot Experiments No Mapping Update “Emotion Energy” 11th episode “mental workload” plot EE = 1867.19 With Mapping Update 11th episode “mental workload” plot 21th episode “mental workload” plot EE = 1644.07 EE = 1467.91

Advanced Control for Human Robot Interaction Realistic & Intuitive Human-Robot Interaction Physical HRI Recognize and control poses and gestures Adaptive Interfaces Visual HRI Robot Touch HRI Neptune Control through Neural Headband Zeno Video 9/18/2018 Multiscale Robots and Systems Lab University of Texas Arlington

Gesture Recognition and Synthesis with Zeno Synthesis of realistic motion distribution in redundant mechanisms, for instance: coordination of motion between neck and eyes during object tracking hand-body gesturing and facial gestures We formulated online optimization algorithms, including reinforcement learning, combined with visual servoing for these problems [5, 6, 7]. Block diagram of neck-eye motion control system for conversational interaction with Zeno Object Pose Tracking Error Good match with human tracking response 9/18/2018 Multiscale Robots and Systems Lab University of Texas Arlington

Control Diagram: Convergence of the Error Correction Algorithm Object Pose Tracking Error I’m going say a few words about the implementation very quickly. Proof of exponentially stable tracking on the paper. This error correction scheme yields exponentially stable tracking

Interaction through hand gestures Gestures performed by the user are recognized by Kinect or Wii Mote in real-time, then played back by robot arm or used as rewards during rehabilitation excercises Match percentage for Line gesture with 100 neurons Testing sets Neural Net Size Match (X) percentage Match (Y) Percentage MSE Set 1 100 78.26 93.23 Set 2 69.34 95.06 Set 3 79.98 95.14 Set 4 83.01 97.30 Set 5 68.88 89.97 9/18/2018 Multiscale Robots and Systems Lab University of Texas Arlington

Multiscale Robots and Systems Lab University of Texas Arlington Zeno mimicking user Zeno has both hand-gesture playback from user as well as scripted sequences to encourage hand coordination. 9/18/2018 Multiscale Robots and Systems Lab University of Texas Arlington

Advanced Control for Human Robot Interaction Realistic & Intuitive Human-Robot Interaction Physical HRI (pHRI) Recognize & Synthesize poses and gestures Adaptive Interfaces Visual HRI Robot Touch HRI Neptune Control through Neural Headband Zeno Video 9/18/2018 Multiscale Robots and Systems Lab University of Texas Arlington

Physical HRI using Robotic Skin A CRS A465 robot arm and an artificial skin patch used as a one dimensional force sensor Pressure sensors mounted on Neptune Ipad Actual force reading from PZT Artificial Skin

Physical HRI – Algorithm and Results Kalman filter, impedance controller, and computed-torque control Force measurement in 1D is sufficient for estimating interaction forces in all 3 directions Push force Model uncertainty leads to poor estimation J. Rajruangrabin, D.O. Popa, "Enhancement of Manipulator Interactivity Through Compliant Skin and Extended Kalman Filtering," in Proc. of IEEE Conference on Automation Science and Technology (CASE), Scottsdale, AZ, September 2007. With one dimension force measurement along x the estimation result in y and z is better even with the presence of 2% rms noise.