1 Lessons Learned using Responsive Virtual Humans for Assessing Interaction Skills R. Hubal & D. Fishbein, RTI International M.J. Paschall, Pacific Inst. for Research and Evaluation

2 Overview of Talk  Problem  Experimental Setting  Solution  Methods  Results  Implications This study was performed under awards # 5-R01-DA from the National Institute on Drug Abuse and # EIA from the National Science Foundation.

3 Problem  Many inner-city adolescents are at risk during social situations that require:  Emotional control.  Conflict resolution through negotiation, seeking information, expressing preferences.  Adverse consequences include violence, school suspension, criminal activity.  Training on these skills is well studied, but…  Assessment is unsatisfactory:  Want to assess adolescent’s learning before he’s “on the street”.  Existing methods use surveys or text-based cases, not “assessment-by-doing”.  Problem  Experimental Setting  Solution  Methods  Results  Implications

4 Experimental Setting  Part of a larger study:  Examining neurocognitive-emotive predictors of behavioral problems among minority adolescents in high-risk urban settings.  Identifying promising approaches to improve social-cognitive skills and reduce problem behaviors such as interpersonal violence and drug abuse.  Participants part of an ongoing longitudinal study.  Research team includes RTI, JHU, PIRE, others.  Problem  Experimental Setting  Solution  Methods  Results  Implications

5 Solution  Role playing is known to be effective in providing adolescents with social competency skills.  Use responsive virtual characters to simulate role-play scenarios for assessment:  Interactive 3D.  Natural language processing.  Cognitive, social, emotional models.  Problem  Experimental Setting  Solution  Methods  Results  Implications

6 Solution, continued  Responsive virtual human training applications:  Law officers learning to manage encounters with the mentally ill.  Medical practitioners practicing on virtual adult and pediatric standardized patients.  Survey field interviewers learning to avert non- response and to gain informed consent to participate.  Army ground tracked vehicle mechanics using a maintenance training system.  These were not intended for assessment…  Problem  Experimental Setting  Solution  Methods  Results  Implications

7 Solution, continued  Vignette storylines designed to target skills (engaging in negotiation, maintaining emotional control and being non-provocative, seeking information).  Language models designed to interpret participant input as reflecting one or more social skill.  Behavior models designed so that the virtual character attempts different strategies to require participants to demonstrate skills.  Methods devised to measure participant engagement.  Problem  Experimental Setting  Solution  Methods  Results  Implications

8 Methods  Participants:  Black male inner-city 10 th graders.  Approx. half diagnosed with Conduct Disorder.  Materials:  PACT video “Workin’ It Out”.  Virtual character vignettes: – 3 scenarios (stolen goods, invitation, confrontation). – 2 variations (pre/post).  Behavioral reports, teacher ratings, self-reports.  Problem  Experimental Setting  Solution  Methods  Results  Implications

9  Problem  Experimental Setting  Solution  Methods  Results  Implications Methods, continued  Positive Adolescents Choices Training (PACT) content:  Social skills training program developed specifically for black youth.  Uses videotapes containing vignettes of a conflict situation and demonstrating skills that could defuse a potentially violent situation.  Target skills include: – Giving negative feedback or expressing criticism calmly (“Givin’ It”). – Reacting appropriately to the criticism or anger of others (“Takin’ It”). – Identifying problems and solutions (“Workin’ It Out”).

10 Methods, continued  Procedure:  Participant given pre-test set of vignettes. – Wizard-of-Oz methodology. – Monitor how participants handle vignettes designed to elicit risky behavior.  Participant given standard text-based assessments.  Participant shown PACT video in presence of facilitator.  After up to 6 month delay, participant given post-test set of vignettes (again WoZ).  Behavioral reports, teacher ratings, self-reports collected.  Problem  Experimental Setting  Solution  Methods  Results  Implications

11 Methods, continued  Measures:  Engagement with vignettes.  Body language.  Verbalizations.  Number of conversational turns.  Response time.  Outcome: – Positive outcome is to decline or back away. – Negative outcome is to agree or escalate confrontation.  Problem  Experimental Setting  Solution  Methods  Results  Implications

12 Results  Scoring procedures used during pre/post test with vignettes identify:  Level of emotional control.  Interpersonal communication skills.  Analysis against established measures provides some support for construct and criterion validity.  Nearly all queried participants stated their virtual decision mirrored what would be their real-life decision.  Note: acceptability / usability not different among groups.  Problem  Experimental Setting  Solution  Methods  Results  Implications

13  Participants agree to help/go along with/escalate confrontation with the synthetic character one- third to half of time.  Simulation effective in differentiating:  Adolescents with clinical Conduct Disorder.  Adolescents who had participated in live training sessions on key skills. Results, continued  Problem  Experimental Setting  Solution  Methods  Results  Implications

14 Implications  Virtual characters can be useful for assessment as well as training in some contexts.  Military applications:  Greater understanding of signs and symptoms associated with PTSD.  Managing encounters with the mentally ill.  Interaction with child casualties and combatants.  Crowd control or checkpoint management.  Essentially, where need is to assess interaction skills in environment that is costly, dangerous, or difficult to replicate.  Problem  Experimental Setting  Solution  Methods  Results  Implications

15 Questions? This study was performed under awards # 5-R01-DA from the National Institute on Drug Abuse and # EIA from the National Science Foundation.