1. Towards an integrated scheme for semantic annotation of multimodal dialogue data Volha Petukhova and Harry Bunt

2. Motivation Several corpora with multimodal data transcriptions: AMI meeting corpus (http://www.amiproject.org); IFA Dialog Video corpus (http://www.fon.hum.uva.nl/ IFA-SpokenLanguageCorpora/IFADVcorpus); ISL meeting corpus (Burger et al., 2002)

3. Motivation coding schemes for the analysis of nonverbal actions in terms of behavioural low-level features: Facial Action Coding System (FACS) ; Ham-NoSys coding schemes of semantic and pragmatic information in visual expressions: SmartKom Coding scheme (Steininger, 2001) DIME-DAMSL (Pineta et al., 2005) MUMIN annotation scheme (Allwood et al., 2004)

4. Motivation the majority of these schemes are designed for a particular purpose and are used solely by their creators (Dybkjær and Bernsen, 2002) The AAMAS workshop ‘Towards a StandardMarkup Language for Embodied Dialogue Acts’ in 2008 and 2009 ISO project 24617-2 “Semantic annotation framework, Part 2: Dialogue acts”

5. Exploratory annotation study DIT++ dialogue act annotation scheme (http://dit.uvt.nl/) incorporates theoretical and empirical findings from other approaches (Petukhova & Bunt, 2009c and Bunt & Schiffrin, 2007) describe not only task-oriented communicative actions, but also actions related to other communicative dimensions: Task, Auto-Feedback, Allo-Feedback, Turn Management, Time Management, Contact Management, Discourse Structuring, Social Obligation Management, Own Communication Management, Partner Communication Management contains open classes, allowing suitable additions of those communicative functions which are specific for a certain modality offer flexible segmentation strategies

6. Exploratory annotation study Corpus material and annotations: two scenario- based dialogues with a total duration of 51 minutes from the AMI corpus Tool: ANVIL (http://www.dfki.de/˜kipp/anvil) Two annotations studies: (1) using only speech transcription and sound; (2) using speech transcription, sound and video provided with transcriptions of nonverbal signals (gaze, head, facial expression, posture orientation and hand movements).

7. Exploratory annotation study Transcriptions: Verbal elements: manually produced orthographic transcriptions for each speaker, including word-level timings Non-verbal elements: gaze direction; head movements; hand and arm gestures; eyebrow, eyes and lips movements; posture shifts; features: –form of movement (head: nod, shake, jerk; hands: pointing, shoulder- shrug, etc.; eyes: narrow, widen; lips: pout, compress, purse, flatten, (half)open, random moves); –direction (up, down, left, right, backward, forward); –trajectory (line, circle, arch); –size (large, small, medium, extra large); –speed (slow, medium, fast); –number of repetitions (up to 20 times); –FTO: difference between time that turn starts and moment that previous turn ends; –Duration Overall kappa =.76

8. Exploratory annotation study compared the annotations with respect to the number and nature of (1)functional segments identified; (2) communicative functions altered; (3) communicative functions specified; and (4) communicative functions assigned to single functional segments.

9. Results Nonverbal communicative behaviour may serve four purposes: 1.emphasizing or articulating the semantic content of dialogue acts; 2.emphasizing or supporting the communicative functions of synchronous verbal behaviour; 3.performing separate dialogue acts in parallel to what is contributed by the partner; 4. expressing a separate communicative function in parallel to what the same speaker is expressing verbally.

10. Results Full-fledged dialogue acts (20% new segments): Feedback acts (68.5%): positive (65.3%), negative (3.2%): Time Management (24.8%) Turn Management (4.7%) Discourse Structuring (2%)

11. Results are reflected in a significant majority of annotation schemes: We analyzed 18 well-known dialogue act annotation schemes: DAMSL, SWBD-DAMSL, LIRICS, DIT++, MRDA, Coconut, Verbmobil, HCRC MapTask, Linlin, TRAINS, AMI, SLSA, Alparon, C-Star, Primula, Matis, Chiba and SPAAC Feedback is not defined only in Linlin and Primula; Turn management acts are not defined in HCRC MapTask, Verbmobil, Linlin, Alparon and C-Star; Discourse Structuring is not defined in TRAINS and Alparon; and Time Management is not defined in MRDA, HCRC MapTask, Linlin, Maltus, Primula and Chiba.

12. Results Communicative function alteration and specification: adjustment of the level of feedback (understanding vs agreement) express degree of certainty about the validity of the proposition reveal speaker’s attitude towards the addressee(-s), towards the content of what he is saying, or towards the actions he is considering to perform signal speaker’s emotional or cognitive state (Pavelin (2002): modalizers)

13. Results Communicative function alteration and specification: - no existing dialogue act annotation scheme deals with this type of information Proposal: a set of qualifiers that can be attached to communicative function in order to describe the speaker’s behaviour more accurately

14. Results qualifier attribute qualifier valuescommunicative function category modalityuncertain,certaininformation-providing functions conditionalityconditional, unconditional action-discussion functions partialitypartial, completeresponsive general-purpose functions; feedback functions modeangry, happy, surprised,... information-providing functions; feedback functions

15. Results Multifunctionality in multimodal utterances: - A verbal functional segment has on average 1.3 communicative functions (also confirmed in Bunt, 2009) - multimodal segment has 1.4 functions on average - very often concerned with feedback and other interaction management dimensions such as Own Communication Management and Time Management; Task and Turn Management; Task and Discourse Structuring; Task and Allo-Feedback -Dialogue act taxonomies that take the multifunctionality of utterances into account such as DIT++, LIRICS, DAMSL, MRDA and Coconut, known as multidimensional dialogue act annotation schemes

16. Results Articulating semantic content (about 39%): They are relating to the propositional or referential meaning of an utterance For example deictic gestures: wording: Press this little presentation hand :........point................. pure semantic acts, as a rule do not have a communicative function on their own

17. Conclusions Multidimensional schemes could be used for annotation of multimodal data (such as DIT++, LIRICS, DAMSL, MRDA and Coconut) Extension is needed: with respect to uncertainty, speaker’s attitude and speaker’s emotions Proposal: Amused Suggestion’ and ’Uncertain Answer’ (undesirable) better to have qualifiers that can be attached to communicative function