1. Presented by: Mónica Domínguez
A Knowledge-Based Conversation Agent -with social and cultural competence-
KRISTINA Consortium
Presented by: Mónica Domínguez

2. KRISTINA’s Motivation
07/11/2017
Yerun Event, Brussels

3. KRISTINA’s Settings Target users Target roles
Social companion in geriatric contexts (small talk, daily routines, cultural and physical activities, …)
Elderly (care recipients) with German and Turkish background
Care givers with Polish background
Basic care assistant (preferences of the care taker, care advices)
Migrants in need of health care information with North African background
Health care adviser (info on health care system, body care advices, …)
07/11/2017
Yerun Event, Brussels

4. A sample dialogue with KRISTINA
K: You look downhearted today. What is wrong?
U: I feel sad. Because of my eyes, I even can’t read the newspaper anymore.
K: Shall I read the newspaper aloud for you?
U: Yes, this would be great!
K: You certainly can still read the headings of the articles. Just tell me which one I shall read. ...
...
07/11/2017
Yerun Event, Brussels

5. KRISTINA’s Technologies Know how and when to interact
Knowledge
Base
Targeted IR
Emotion Analysis
Speech Recognition
Facial Expr. Recognition
Gesture
Analysis
Emotion Recognition
Language Analysis
Multimodal Fusion
Interaction Manager
Dialogue Manager
Idle Behavior Manager
Multimodal
Fission
&
Discourse Planning
Facial Expr. Generation
Generation
Language Synthesis
3 USER PARTNERS
Listen and watch
8 TECHNICAL PARTNERS
Think and learn
Speak and act
Know how and when to interact
07/11/2017
Yerun Event, Brussels

6. What is the Added Value of KRISTINA?
Target: individuals in need of advice and support in the context of basic care and healthcare.
Multilinguality: German, Spanish, Polish, Arabic and Turkish.
Versatility: socially and culturally adapted and designed and tested by experts in basic and health care.
Applicability: web-based integrated technologies.
07/11/2017
Yerun Event, Brussels

7. Thank you!
07/11/2017
Yerun Event, Brussels

8. Where do we stand? Third (and last) year
Working on the Final Demonstrator
The coverage of the number of conversation topics is (still) limited, but can be extended as needed
Language coverage: German, Polish, Spanish, Turkish
07/11/2017
Yerun Event, Brussels

9. Preliminary evaluation
07/11/2017
Yerun Event, Brussels

10. 3.1. Multimodal Communication Analysis (Verbal)
SPEECH
Feature Extraction
Decoder
Ortographic Transcription
ASR
Semantic Parser
Disambiguation
Dependency Parser LA
Ontology KB
07/11/2017
Yerun Event, Brussels

11. 3.2. Multimodal Communication Analysis (Non-verbal)FA
FACE
FUSION
VIDEO ER
Detection
Tracking
Classification
Ontology
GESTURES GA KB
07/11/2017
Yerun Event, Brussels

12. 3.1. KB Dialogue Management
Policy-based turn taking decision making
Is it appropriate to interrupt the user?
How to react upon user’s attempt to barge in?
Idle behavior
Mimicking the user’s affective state / gestures / facial expresion DM
07/11/2017
Yerun Event, Brussels

13. 3.3. Multimodal Communication GenerationKB
OWL
Dialogue Manager
Mode Selection
Text Generation (verbal)
Expressive characters (non-verbal)
KRISTINA Agent DB
CoNLL history
Prosody Prediction (verbal)
Speech synthesis (verbal)
07/11/2017
Yerun Event, Brussels