1. Examples of Nonlinear Systems: ROBOTS New Robotic Treatment Systems for Childhood Autism and Cerebral Palsy
Joint Work with
N. BugnariuA, D. HansonB, F. MakedonC
ADepartment of Physical Therapy, University of North Texas Health Sciences Center (UNT HSC)
BHanson Robotics Inc., Plano, TX, USA
CDepartment of Computer Science & Engineering Department,
University of Texas at Arlington, USA
NGS Focus: Human-Robot Interaction
Ph.D. Students: Isura Ranatunga, Nahum Torres
This work was supported by:
US National Science Foundation Grants #CPS , and #CNS
TxMed consortium grant: “Human-Robot Interaction System for Early Diagnosis and Treatment of Childhood Autism Spectrum Disorders (RoDiCA)”

2. NGS Robots with HRI and pHRI
Two assistive robotic systems aimed at the treatment of children with certain motor and cognitive impairments.
In the Neptune project [1]
Mobile manipulator for children suffering from Cerebral-Palsy.
Mobile robot base and a 6DOF robotic arm, interfaced via:
Wii Remote, iPad, Neuroheadset, the Kinect, and Force sensing robotic skin
Therapeutic outcomes
Hand and head gesture recognition and reward.
Hand motion excercises using IPAD Games (CPlay, CPMaze, ProlloquoToGo) held by the robot.
The RoDiCA project [2]
focuses on treating cognitive impairments in children suffering from ASD
Zeno is a robotic platform developed by Hanson Robotics, based on a patented realistic skin.
Real time subject tracking/joint attention
Advanced head-eye and hand coordination
Facial gesture recognition and synthesis
Data logging and analysis.
Neptune Mobile manipulator with iPad attached.
Zeno (by Hanson RoboKind Inc.) generating facial expressions and maintaining eye contact.
9/18/2018
Multiscale Robots and Systems Lab University of Texas Arlington

3. Advanced Control for Human Robot Interaction
Realistic & Intuitive Human-Robot Interaction
Physical HRI
Recognize & Synthesize poses and gestures
Adaptive Interfaces
Visual HRI
Robot Touch HRI
Neptune Control through
Neural Headband
Zeno Video
9/18/2018
Multiscale Robots and Systems Lab University of Texas Arlington

4. Neptune: Assistive Robotic System for CP
9/18/2018
Multiscale Robots and Systems Lab University of Texas Arlington

5. Multiscale Robots and Systems Lab University of Texas Arlington
Adaptive Interfaces
The supervisory control of multi-DOF robots is a demanding application.
If a single operator is tasked with direct control, performing coordinated tasks becomes non-intuitive.
We use Reinforcement Learning TD(lambda) scheme in order to adaptively change the mapping of DOF’s from the operator user interface to the robot.
State Propagation
Interface Mapping
System
Metrics
Evaluation
state
Interface input
action
Update
Reward
function
reward
Critic
Actor TD
error
Value
Function
Policy
9/18/2018
Multiscale Robots and Systems Lab University of Texas Arlington

6. Interface Mapping of Neural Headband to Robot Experiments
No Mapping Update
“Emotion Energy”
11th episode “mental workload” plot
EE =
With Mapping Update
11th episode “mental workload” plot
21th episode “mental workload” plot
EE =
EE =

7. Advanced Control for Human Robot Interaction
Realistic & Intuitive Human-Robot Interaction
Physical HRI
Recognize and control poses and gestures
Adaptive Interfaces
Visual HRI
Robot Touch HRI
Neptune Control through
Neural Headband
Zeno Video
9/18/2018
Multiscale Robots and Systems Lab University of Texas Arlington

8. Gesture Recognition and Synthesis with Zeno
Synthesis of realistic motion distribution in redundant mechanisms, for instance:
coordination of motion between neck and eyes during object tracking
hand-body gesturing
and facial gestures
We formulated online optimization algorithms, including reinforcement learning, combined with visual servoing for these problems [5, 6, 7].
Block diagram of neck-eye motion control system for conversational
interaction with Zeno
Object Pose
Tracking Error
Good match with human tracking response
9/18/2018
Multiscale Robots and Systems Lab University of Texas Arlington

9. Control Diagram: Convergence of the Error Correction Algorithm
Object Pose
Tracking Error
I’m going say a few words about the implementation very quickly. Proof of exponentially stable tracking on the paper.
This error correction scheme yields exponentially stable tracking

10. Interaction through hand gestures
Gestures performed by the user are recognized by Kinect or Wii Mote in real-time, then played back by robot arm or used as rewards during rehabilitation excercises
Match percentage for Line gesture with 100 neurons
Testing sets
Neural Net Size
Match (X)
percentage
Match (Y)
Percentage MSE
Set 1
100
78.26
93.23
Set 2
69.34
95.06
Set 3
79.98
95.14
Set 4
83.01
97.30
Set 5
68.88
89.97
9/18/2018
Multiscale Robots and Systems Lab University of Texas Arlington

11. Multiscale Robots and Systems Lab University of Texas Arlington
Zeno mimicking user
Zeno has both hand-gesture playback from user as well as scripted sequences to encourage hand coordination.
9/18/2018
Multiscale Robots and Systems Lab University of Texas Arlington

12. Advanced Control for Human Robot Interaction
Realistic & Intuitive Human-Robot Interaction
Physical HRI
(pHRI)
Recognize & Synthesize poses and gestures
Adaptive Interfaces
Visual HRI
Robot Touch HRI
Neptune Control through
Neural Headband
Zeno Video
9/18/2018
Multiscale Robots and Systems Lab University of Texas Arlington

13. Physical HRI using Robotic Skin
A CRS A465 robot arm and an artificial skin patch used as a one dimensional force sensor
Pressure sensors mounted on Neptune Ipad
Actual force reading from PZT Artificial Skin

14. Physical HRI – Algorithm and Results
Kalman filter, impedance controller, and computed-torque control
Force measurement in 1D is sufficient for estimating interaction forces in all 3 directions
Push force
Model uncertainty leads to poor estimation
J. Rajruangrabin, D.O. Popa, "Enhancement of Manipulator Interactivity Through Compliant Skin and Extended Kalman Filtering," in Proc. of IEEE Conference on Automation Science and Technology (CASE), Scottsdale, AZ, September 2007.
With one dimension force measurement along x the estimation
result in y and z is better even with the presence of 2% rms noise.