Semantic annotation framework Part 2: Dialogue acts ISO/TC37/SC4 N442 rev00 Harry Bunt Tilburg University ISO TC 37/SC 4 meeting Marrakech, May 25, 2008

Purpose and justification Dialogue acts are widely used in studies of dialogue phenomena, in dialogue annotation, and in the design of dialogue systems. Dialogue acts are particularly useful for: describing functional and intentional aspects of the dialogue utterance meaning; the design of dialogue management systems.

Dialogue acts Well-known examples of communicative functions (“core dialogue acts”): question WH-question YN-question check/verification statement/inform answer (WH-answer. YN-answer) confirmation, disconfirmation request instruct promise acknowledgement greeting

Purpose and justification (2) Alternative dialogue act schemas: TRAINS, Map Task, Verbmobil, DAMSL, SWBD-DAMSL, COCONUT,... with different: underlying approach to dialogue modelling definitions of basic concepts level of granularity and mutually inconsistent terminology Particularly unsatisfactory: Lack of solid foundations of definitions and multidimensionality Lack of interoperability

ISO approach Preparatory studies in TDG 3 in a joint effort with eContent project LIRICS. Focus: How to best support the annotation of dialogues with dialogue act information in an empirically and theoretically well-founded way. Outcome: 1.Design of a preliminary set of data categories for multidimensional dialogue act annotation, based on DAMSL, DIT++, and other schemas, tested for coverage and usability by annotators and endorsed by ISO TC37/SC4/TDG3. 2.Recommendation to set up an ISO project based on 1 as part the Semantic Annotation Framework project.

Summary Main points of project outlined in ISO/TC37/SC4 N442 (rev00): Aim to: 1.Provide more solid foundations for multidimensionality of DA tag sets 2.Design consistent truly semantic definitions of core dialogue acts 3.Develop agreed definitions in the form of ISO data categories and enter in ISO registry 4.Define annotation language with abstract syntax, concrete XML-based syntax, and semantics compliant with LAF

Theoretical foundations of DA annotation concepts Information-state change approach to dialogue semantics: the meaning of an utterance in dialogue is the way in which the information state of a listener is changed by understanding the utterance (Bunt & Romary, LREC 2002). A dialogue act has two components for describing utterance meanings: the information which the speakers makes available to the addressee - the “semantic content” the “communicative function”, capturing the way the speaker intends an addressee to update his information state with the semantic content.

Multifunctionality 1.U: Can you tell me what time is the first train to the airport on Sunday? 2.S: On Sunday morning the first train to the airport is at U: Thank you. - expression of thanks

Multifunctionality 1.U: Can you tell me what time is the first train to the airport on Sunday? 2.S: On Sunday morning the first train to the airport is at U: Thank you.

Multifunctionality 1.U: Can you tell me what time is the first train to the airport on Sunday? 2.S: On Sunday morning the first train to the airport is at U: Thank you. - expression of thanks

Multifunctionality 1.U: Can you tell me what time is the first train to the airport on Sunday? 2.S: On Sunday morning the first train to the airport is at U: Thank you. - expression of thanks - positive feedback (about understanding and acceptance)

Multifunctionality 1.U: Can you tell me what time is the first train to the airport on Sunday? 2.S: On Sunday morning the first train to the airport is at U: Thank you. - expression of thanks - positive feedback (about understanding and acceptance) - indication of dialogue closure

Multifunctionality 1.U: Can you tell me what time is the first train to the airport on Sunday? 1.S: The first train to the airport on Sunday is at... let me see positive auto-feedback about perception and interpretation - WH-answer (to indirect WH-question)

Multidimensionality Utterances have multiple functions ==> multiple annotation tags are required (or syntactically and semantically (!) complex tags -- cf. studies by Popescu-Belis), i.e. annotation must be multidimensional.

Multidimensional annotation Usual informal notion of dimension: Set of mutually exclusive tags Not satisfactory... See problems in multidimensional annotation according to DAMSL (Bunt, LREC 2006)

Dimensions in Dialogue Basic intuition: participants in a dialogue do multiple things simultaneously, such as: making progress in performing the activity (“task”) which motivates the dialogue; providing and eliciting communicative feedback; take and assign turns; monitor contact, attention, use of time,... greet, thank, apologize, say goodbye,...

Dimensions in Dialogue A dimension is an aspect of participating in a dialogue such that: 1.There is a class of dialogue acts for addressing this dimension (empirical foundation); 2.It can be addressed independently of other dimensions; 3.Within a dimension, an utterance has at most one communicative function.

Observed dimensions (TDG3/LIRICS) Performing a certain task or activity through or with support from the communication Monitoring the interaction - providing and eliciting feedback - editing one’s own or one’s partner’s speech - managing the turn-taking - managing the use of time - managing contact and attention - managing the opening and closing of (sub-) dialogues and thematic progression Dealing with social obligations: greeting, thanking, apologizing,…

Dimensions for dialogue acts Examples: dimension function example 1.Auto-feedback OverallPositive Okay. 2.Allo-feedback EvaluationElicitation Okay? 3.Turn management TurnGiving Yes 4.Time management Stalling Well, you know,.. 5.Contact man’t ContactChecking Hello? 6.Own comm. man’t Self-correction I mean... 7.Partner comm.man. Completion... completion 8.Topic management TopicShiftAnnounc. Something else. 9.Dialogue structuring DA-announcement Question: 10.Social oblig. man’t Valediction Bye 11.Task/domain OpenMeeting I open this meeting

Dimensions of dialogue acts Example: Inform The KL204 leaves at Task/domain

Dimensions of dialogue acts Example: Inform The KL204 leaves at Task/domain I see what you mean. Auto-feedback

Dimensions of dialogue acts Example: Inform The KL204 leaves at Task/domain I see what you mean. Auto-feedback We should first discuss the agenda. Topic management

Dimensions of dialogue acts Example: Inform The KL204 leaves at Task/domain I see what you mean. Auto-feedback We should first discuss the agenda. Topic management I’m very grateful for you help. Social obligation management

Dimensions of dialogue acts Example: Inform The KL204 leaves at Task/domain I see what you mean. Auto-feedback We should first discuss the agenda. Topic management I’m very grateful for you help. Social obligation management ==> Inform acts can be used in every dimension.

Dimensions for dialogue acts A number of the most commonly used types of dialogue act, such as questions, answers, statements, requests, instructions, or offers,.. do not belong to any dimension: they are ‘general purpose functions’: they can be used in any dimension. (DAMSL ‘dimensions’ like Info-request and Answer are clearly not proper dimensions.)

General-purpose functions Applicable in any dimension are:  Information-seeking functions WH-question, YN-question, Alternatives-question, Check,..  Information-providing functions Inform, WH-Answer, YN-Answer, Confirmation, Disconfirmation, Agreement, Correction,..  Commissive functions Offer, Promise, AcceptRequest,..  Directive functions Instruct, Request, Suggest,..

Core dimensions and dialogue acts Data categories from LIRICS: Set of 54 core dialogue act types 24 general-purpose functions 30 dimension-specific functions spread over 10 dimensions described in the form of ISO (12620) data categories. Compare: DAMSL: 12 dimensions, 30 functions SWBD-DAMSL: 60 functions DIT++: 11 dimensions, 95 functions

Validation of LIRICS data categories Usability for human annotators Inter-annotator agreement measurements for English and Dutch; 2 trained annotators working on raw text/audio Results: almost perfect agreement (Rietveld & van Hout, 1993: kappa ≥ 0.80)

Inter-annotator agreement scores Function classEnglishDutchaverage Information-seeking Information-providing Feedback Interaction management Social obligations management 0.94

Validation of LIRICS data categories Applicability also demonstrated for Italian (annotated test suite developed in Pisa). Application of to multi-party multimodal AMI dialogues (Petukhova & Bunt, IWCS-7); results comparing favourably with use of AMI or DAMSL annotation schemes. Machine learnability investigations are promising (Geertzen et al., SIGDIAL 2007).

Towards a dialogue act annotation language DA tag components: Examples: Note: for dimension-specific functions, the dimension name is in fact redundant.

Design of dialogue annotation language: DiaML Distinction in Linguistic Annotation Framework: annotations: information structures independent of representation format (“abstract syntax”) representations: annotations cast in a certain format (“concrete syntax”)

Design of dialogue annotation language: DiaML Distinction in Linguistic Annotation Framework: annotations: information structures independent of representation format (“abstract syntax”) representations: annnotations cast in a certain format (“concrete syntax”) In addition: semantics, defined for abstract syntax

DiaML abstract syntax Abstract Syntax: format-independent definition of information structures For dialogue acts: pairs of stretches (possibly discontinuous) of dialogue behaviour and sets of dialogue act types (at most one function in each dimension) Information to be expressed in DiaML: speaker and addressee(s) segments of dialogue behaviour dimensions communicative functions optionally: functional dependencies (e.g. an utterance is an Answer to which Question, or provides Feedback on which previous dialogue act)

DiaML abstract syntax Information to be expressed in DiaML: speaker and addressee(s) segment of dialogue behaviour dimension communicative function optionally: functional dependency Conceptual elements: finite set of dialogue participants finite, ordered set of segment begin/end indicators finite set of dimensions finite sets of domain-specific and general-purpose comm. functions

DiaML abstract syntax Information to be expressed in DiaML: speaker and addressee(s) segment of dialogue behaviour dimension communicative function Structure definitions: A DiaML segment (‘markable’) is a finite sequence of pairs of segment begin/end indicators, defining a stretch of source text A DiaML tag is an n-tuple of pairs A complete DiaML structure is a 4-tuple

DiaML concrete syntax

Current status Result of NWIP ballot? If NWIP approved: 1. Project (“editorial”) group: David Traum Claudia Soria Jae-Woong Choe Andrei Popescu-Belis Jan Alexandersson Alex Chengyu Fang Koiti Hasida (tbc) Time schedule and meetings: Moscow, August 2008? (TC 37 annual meeting) Pisa, October 2008, workshop Tilburg, January 2009 (IWCS-9), workshop

Dimensions in DAMSL FLF Dimensions: 1.Statement 2.Info-request 3.Influencing-addressee-future-action 4.Committing-speaker-future-action 5.Conventional Opening or Closing 6.Explicit-performative 7.Exclamation 8.Other

Dimensions in DAMSL (3) Example: 1.A: I hope you’ll have a good time! 2.B: Yeah, thanks. And you’ll be visiting friends in Italy. 3. A: That’s right.

Dimensions in DAMSL Definitions of some FLF Dimensions: 1.Statement: Speaker makes a claim about the world 2.Info-request: Speaker requests Addressee to provide information 3.Influencing-addressee-future-action 4.Committing-speaker-future-action 5.(.... 8)

Dimensions in DAMSL (2) BLF Dimensions: 1.Agreement 2.Understanding 3.Answer 4.Information-relation

Dimensions in DAMSL (4) Conclusion: ‘Question’ and ‘statement’ are mutually exclusive tags => having them in different dimensions is wrong

Dimensions in DAMSL (2) Definitions of some BLF Dimensions: 1.Agreement 2.Understanding: Utterances concerning the understanding between Speaker and Addressee 3.Answer: Speaker provides information requested by the Addressee 4.Information-relation

Dimensions in DAMSL (4) Can ‘question’ and ‘answer’ be alternatives in the same dimension? Consider: 1. S: Did you ask me something? 2. U: Can I change the contrast? - question

Dimensions in DAMSL (4) Can ‘question’ and ‘answer’ be alternatives in the same dimension? Consider: 1. S: Did you ask me something? 2. U: Can I change the contrast? - question - answer

Dimensions in DAMSL (4) Can ‘question’ and ‘answer’ be alternatives in the same dimension? Consider: 1. S: Did you ask me something? 2. U: Can I change the contrast? - question - answer So question and answer can co-occur => question and answer cannot be in the same dimension

Dimensions in DAMSL (3) Example: 1.A: I hope you’ll have a good time! 2.B: Yeah, thanks. And you’ll be visiting friends in Italy. 3. A: That’s right.

Dimensions in DAMSL (3) Example: 1.A: I hope you’ll have a good time! 2.B: Yeah, thanks. And you’ll be visiting friends in Italy. Statement? Question? 3. A: That’s right.

Dimensions in DAMSL (3) Example: 1.A: I hope you’ll have a good time! 2.B: Yeah, thanks. And you’ll be visiting friends in Italy. Statement? Question? DAMSL: - statement - info-request... but a speaker cannot at the same time state something and question its truth!

Dimensions in DAMSL (4) Conclusion: ‘Question’ and ‘statement’ are mutually exclusive tags => having them in different dimensions is wrong Can they be alternatives in the same dimension?

Dimensions in DAMSL (4) Can ‘question’ and ‘answer’ be alternatives in the same dimension? Consider: 1. S: Did you ask me something? 2. U: Can I change the contrast?

Dimensions in DAMSL (4) Can ‘question’ and ‘answer’ be alternatives in the same dimension? 1. S: Did you ask me something? 2. U: Can I change the contrast? - question about what the task doman - answer about what the speaker said => The task domain and what the speaker said are different ‘dimensions’; an utterance can be a question in one dimension and an answer in another.

Dimensions in DAMSL (5) Example: A: And what possibilities do you have on Tursday? B: Did you say Thursday? - function in (DAMSL’s) Understanding dimension, but which? - Signal-understanding? - Signal-non-understanding? - Check (function in DAMSL’s Info- request dimension)

Multidimensional annotation scheme for dialogue acts Two parts: 1.dimension-specific communicative functions for each dimension 2.general-purpose functions (hierarchically organized, reflecting degrees of specificity)