Presentation is loading. Please wait.

Presentation is loading. Please wait.

Piloting a methodology for understanding the relationship between agents and genres David Novick Adriana Camacho Marisa Steans The University of Texas.

Similar presentations


Presentation on theme: "Piloting a methodology for understanding the relationship between agents and genres David Novick Adriana Camacho Marisa Steans The University of Texas."— Presentation transcript:

1 Piloting a methodology for understanding the relationship between agents and genres David Novick Adriana Camacho Marisa Steans The University of Texas at El Paso What are the promise and the problems of assessing agent genres and capabilities?

2 Research focus: Human-ECA rapport, especially via paralinguistics 2013 study: Effect of increased familiarity – Experimental condition: greater gesture amplitude – System: “Escape from the Castle of the Vampire King” Current study: Effect of gesture perception – Experimental condition: Agent reacts meaningfully to human’s gestures – System: “Survival on Jungle Island” Advanced aGent ENgagement Team

3 Escape from the Castle of the Vampire King

4 Research directions “Vampire King” and “Jungle Island” are both games – Needed highly engaging application to induce subjects to return for second sessions – Needed strongly constrained (dynamic) contexts to attain acceptable recognition for speech and gesture Now looking beyond games for more meaningful applications – Need situation where we can study rapport – Need high levels of engagement – Need strong constraint of interaction contexts  Reviewing genres for applications with suitable agent characteristics

5 Key features of agents Visual Features Human likeness Realism Facial expressions First vs. third person Motion Camacho, A., Rayon, A., Gris, I., and Novick, D. (2014). An exploratory analysis of ECA characteristics, Intelligent Virtual Agents 2014, Boston, MA, August, 2014, Functional Features Non-scripted dialogue Verbal communication Level of interaction (modalities) Group social skills AI level Environment interaction Persona Nonverbal reaction

6

7 Research goals Find out if agents’ characteristics relate to their genre Explore the meaning and utility of genre for agents Validate the UTEP agent taxonomic analysis  Develop and pilot a methodology that relates agents’ characteristics and genres

8 “Genre” Genres are regular groupings of stylistic, thematic, and compositional elements (Hanks, 1987) vs. Genres are orienting frameworks, interpretive procedures, and sets of expectations that are not part of discourse structure but of the ways that the actors relate to and use language (Bauman, 1986)

9 (Meta-)Methodology Code IVA papers for agents’ genres & characteristics – Selected papers from IVA 2013 – Two coders Evaluate interrater reliability – Genre – Characteristics Assess meaningfulness – Genre – Characteristics

10 IVA 2013

11 Coding example Paper AuthorsTitleAgentApplication Ramin Yaghoubzadeh, Marcel Kramer, Karola Pitsch, and Stefan Kopp Virtual Agents as Daily Assistants for Elderly or Cognitively Impaired People Billie Daily help (w/elderly) (Coder A) Daily assistant (Coder B) Visual Human-like (anthro) Facial Expressions 1 st vs 3 rd Person Body Language RealisticCoder A B Behavior Not Scripted Verbal Comm. Level of Interaction Group Social Skills AI Level Environment interaction Persona (Background) Non-verbal Reaction Coder A B

12 Interrater reliability AnthroCoder AK undefined Coder BAnthroNot Anthro130 Not00 Facial ExprCoder AK = 0.55 Coder BExprNot Expr60 Not34 1st v. 3rdCoder AK = 0.00 Coder B1st3rd 1st121 3rd00 RealisticCoder AK = 0.45 Coder BRealisticNot Realistic80 Not32 UnscriptedCoder AK = 0.00 Coder BUnscriptedNot Unscripted00 Not211 NV React’nCoder AK = 0.11 Coder BNonverbalNot Nonverbal51 Not52

13 Genres ApplicationGenreImplementation Daily assistant AssistanceWoZ Health coach Training“Real” Self-service check-out CommercialStatic Public speaking training Training“Real” Word game EducationWoZ Vocabulary trainer EducationVideo Word game EducationWoZ Nurse training TrainingVideo GameEntertainment“Real” Negotiation?“Real” Med trainingTraining“Real”

14 Issues Many agents do not have an application – Coders sometimes put purpose of study where application not evident – E.g., “gesture study” Distinction between training and education may be meaningless Papers may not provide enough information about agent characteristics – When coders guess, we can at least rate reliability; but Kappas are proving to be too low – When information is incomplete, how summarize? “Unreal” agents may distort findings – WoZ – Video recordings

15 Pilot results Analyzed 20 IVA 2013 papers, of which 7 had usable codings AuthorApplicationGenre Visual (max=5) Functional (max=8) Total (max=13) Yaghoubzadehdaily assistantassistance Batrinca public speaking trainingtraining Leenurse trainingtraining Bergmannvocabulary trainereducation Kühneword gameeducation Yasavurhealth coachtraining Payneself-service check-outcommercial2.40

16 Next steps Analyze papers from IVA 2011, 2012, 2014 – Note IVA 2014 emphasis on biomedical applications – Avoid duplicate agents – Consider papers from ICMI, CHI, … Improve interrater reliability for agent characteristics Group applications into genres (how?) Explore correspondence between genres & characteristics Develop new applications for study of human-ECA rapport

17 David Novick Adriana Camacho Marisa Steans Advanced aGent ENgagement Team


Download ppt "Piloting a methodology for understanding the relationship between agents and genres David Novick Adriana Camacho Marisa Steans The University of Texas."

Similar presentations


Ads by Google