electronic visualization laboratory, university of illinois at chicago Towards Lifelike Interfaces That Learn Jason Leigh, Andrew Johnson, Luc Renambot, Steve Jones, Maxine Brown

electronic visualization laboratory, university of illinois at chicago The Electronic Visualization Laboratory Established in 1973 Jason Leigh, Director; Tom DeFanti, Co- Director; Dan Sandin, Director Emeritus 10 full time staff Interdisciplinary Computer Science, Art & Communication 30 students, 15 funded students, Research in: –Advanced display systems –Visualization and virtual reality –High speed networking –Collaboration & human computer interaction 34 years of collaboration with Science, Industry & Arts to apply new computer science techniques to these disciplines. Major support by NSF and ONR.

electronic visualization laboratory, university of illinois at chicago Goal in 3 Years Life-sized Avatar capable of reacting to speech input with naturalistic facial and gestural responses. A methodology of how to capture and translate human verbal and non- verbal communication into an interactive digital representation. Deeper understanding of how to create believable/credible avatars.

electronic visualization laboratory, university of illinois at chicago System Components AlexDSS Speech Recognition Natural Language Processing Responsive Avatar Facial Expression Recognition Responsive Avatar Engine Eye-tracking Speech Synthesis Lip Synch Gestural Articulation Facial Articulation Knowledge Processing Facial & Body Motion / Performance Capture Phonetic Speech Sampling Knowledge Capture Textual & Contextual Information

electronic visualization laboratory, university of illinois at chicago EVL Year 1 Digitize facial images and audio of Alex Shadow Alex to capture information about his mannerisms Create 3D Alex focusing largely on facial features Prototype initial RAE & merge initial avatar, speech recognition, AlexDSS, pre-recorded voices Validate provision of non-verbal avatar cues, evaluate efficacy of cues

electronic visualization laboratory, university of illinois at chicago EVL Year 2 Full-scale Motion & performance capture to create gestural responses to AlexDSS Speech synthesis using Alex’s voice patterns to create verbal responses to AlexDSS Use eye-tracking to begin to experiment with aspects of non- verbal communication Evaluate merging of verbal and non- verbal information in users’ understandings of –avatar believability and credibility (ethos) –information retrieved –avatar emotional appeals (pathos)

electronic visualization laboratory, university of illinois at chicago EVL Year 3 Utilize camera-based recognition of facial expressions as additional non-verbal input Conduct user studies: –relative to a believability and credibility (ethos) –to correlate attention to non- verbal communication relative to comprehension and retention –to assess value of avatar emotional appeals (pathos) –to address formation of longer-term relationship formation between avatar and user. Camera Microphone Life-sized projection

electronic visualization laboratory, university of illinois at chicago Thanks! This project was supported by grants from the National Science Foundation Award CNS and CNS