Presentation on theme: "Piloting a methodology for understanding the relationship between agents and genres David Novick Adriana Camacho Marisa Steans The University of Texas."— Presentation transcript:
Piloting a methodology for understanding the relationship between agents and genres David Novick Adriana Camacho Marisa Steans The University of Texas at El Paso What are the promise and the problems of assessing agent genres and capabilities?
Research focus: Human-ECA rapport, especially via paralinguistics 2013 study: Effect of increased familiarity – Experimental condition: greater gesture amplitude – System: “Escape from the Castle of the Vampire King” Current study: Effect of gesture perception – Experimental condition: Agent reacts meaningfully to human’s gestures – System: “Survival on Jungle Island” Advanced aGent ENgagement Team
Research directions “Vampire King” and “Jungle Island” are both games – Needed highly engaging application to induce subjects to return for second sessions – Needed strongly constrained (dynamic) contexts to attain acceptable recognition for speech and gesture Now looking beyond games for more meaningful applications – Need situation where we can study rapport – Need high levels of engagement – Need strong constraint of interaction contexts Reviewing genres for applications with suitable agent characteristics
Key features of agents Visual Features Human likeness Realism Facial expressions First vs. third person Motion Camacho, A., Rayon, A., Gris, I., and Novick, D. (2014). An exploratory analysis of ECA characteristics, Intelligent Virtual Agents 2014, Boston, MA, August, 2014, 95-98 Functional Features Non-scripted dialogue Verbal communication Level of interaction (modalities) Group social skills AI level Environment interaction Persona Nonverbal reaction
Research goals Find out if agents’ characteristics relate to their genre Explore the meaning and utility of genre for agents Validate the UTEP agent taxonomic analysis Develop and pilot a methodology that relates agents’ characteristics and genres
“Genre” Genres are regular groupings of stylistic, thematic, and compositional elements (Hanks, 1987) vs. Genres are orienting frameworks, interpretive procedures, and sets of expectations that are not part of discourse structure but of the ways that the actors relate to and use language (Bauman, 1986)
Coding example Paper AuthorsTitleAgentApplication Ramin Yaghoubzadeh, Marcel Kramer, Karola Pitsch, and Stefan Kopp Virtual Agents as Daily Assistants for Elderly or Cognitively Impaired People Billie Daily help (w/elderly) (Coder A) Daily assistant (Coder B) Visual Human-like (anthro) Facial Expressions 1 st vs 3 rd Person Body Language RealisticCoder 1110.61A 1010.81B Behavior Not Scripted Verbal Comm. Level of Interaction Group Social Skills AI Level Environment interaction Persona (Background) Non-verbal Reaction Coder 10.8 10.4111A 00.6 0 101B
Interrater reliability AnthroCoder AK undefined Coder BAnthroNot Anthro130 Not00 Facial ExprCoder AK = 0.55 Coder BExprNot Expr60 Not34 1st v. 3rdCoder AK = 0.00 Coder B1st3rd 1st121 3rd00 RealisticCoder AK = 0.45 Coder BRealisticNot Realistic80 Not32 UnscriptedCoder AK = 0.00 Coder BUnscriptedNot Unscripted00 Not211 NV React’nCoder AK = 0.11 Coder BNonverbalNot Nonverbal51 Not52
Genres ApplicationGenreImplementation Daily assistant AssistanceWoZ Health coach Training“Real” Self-service check-out CommercialStatic Public speaking training Training“Real” Word game EducationWoZ Vocabulary trainer EducationVideo Word game EducationWoZ Nurse training TrainingVideo GameEntertainment“Real” Negotiation?“Real” Med trainingTraining“Real”
Issues Many agents do not have an application – Coders sometimes put purpose of study where application not evident – E.g., “gesture study” Distinction between training and education may be meaningless Papers may not provide enough information about agent characteristics – When coders guess, we can at least rate reliability; but Kappas are proving to be too low – When information is incomplete, how summarize? “Unreal” agents may distort findings – WoZ – Video recordings
Pilot results Analyzed 20 IVA 2013 papers, of which 7 had usable codings AuthorApplicationGenre Visual (max=5) Functional (max=8) Total (max=13) Yaghoubzadehdaily assistantassistance3.8 7.6 Batrinca public speaking trainingtraining126.96.36.199 Leenurse trainingtraining188.8.131.52 Bergmannvocabulary trainereducation184.108.40.206 Kühneword gameeducation220.127.116.11 Yasavurhealth coachtraining18.104.22.168 Payneself-service check-outcommercial2.40
Next steps Analyze papers from IVA 2011, 2012, 2014 – Note IVA 2014 emphasis on biomedical applications – Avoid duplicate agents – Consider papers from ICMI, CHI, … Improve interrater reliability for agent characteristics Group applications into genres (how?) Explore correspondence between genres & characteristics Develop new applications for study of human-ECA rapport
David Novick firstname.lastname@example.org www.cs.utep.edu/novick Adriana Camacho Marisa Steans Advanced aGent ENgagement Team
Your consent to our cookies if you continue to use this website.