1. Labelling databases Roddy Cowie et al QUB

2. Labelling databases: what is the issue? HUMAINE has to create labelling schemes for spontaneous emotionally coloured action & interaction Creating appropriate schemes is an intellectual challenge,  not routine drudgery. It is an empirical problem,  not one theory can solve for us (though theory has a role) It could be HUMAINE’s biggest achievement  If we get it right

3. Labelling databases: what is the issue? Labelling is about grouping events on at least 3 different levels  signs of emotion  emotion-related states  contexts of emotion in a way that facilitates the discovery of relationships between the levels

4. Labelling databases: what is the issue? It is not sufficient for labels to be true True labels may obscure relationships  by grouping events in at one level  when they are different at another (eg anger has subclasses which are vocally different)  by dividing events in ways that fail to show  they are similar if not identical (eg anger and irritation are similar, not utterly different)

5. Labelling databases: what is the issue? It is not sufficient for a scheme to be beautiful, it must also be practical. It must be potentially useful for applications and possible to apply within HUMAINE’s resources - which means being massively selective.

6. Labelling databases: the proposed framework Emotion labelling ‘trace’ type coarse labelling ‘word’ type intermediate labelling ‘appraisal’ type fine labelling Signs of emotion perceptual mechanical Context labels (categorical)

7. Labelling databases: emotion labelling ‘Trace’ type coarse labelling Continuous representations of the way some emotion-related judgment changes over time Eg E-trace shows general ebb & flow of emotion over 20 min period –  A guide to where closer analysis should concentrate  the kinds of unit to consider  rough unit boundaries  and for some purposes enough

8. Labelling databases: Emotion labelling ‘Trace’ type coarse labelling: other options include ACTtrace acted or genuine? VALIDtrace combines acted or stilted and how intense the emotion is These provide convenient, systematic ways to address issues Noam raised FEELtrace (activation-evaluation-useless) FEELtrace3 (activation-evaluation-power-useless) These provide a substantial part of the information a verbal labelling could in ~ real time (not 20x); and they are continuous

9. Labelling databases: Emotion labelling ‘Word’ type intermediate labelling Well chosen words are clearly useful. The problem is, how to choose?  anarchy gives incommensurable data  rigidity gives false impressions Proposal: an orderly way of selecting from a menu with known properties – ‘Basic English Euro Emotion Vocabulary’

10. Labelling databases: Emotion labelling ‘Word’ type intermediate labelling: a menu You have been reviewing a particular kind of material. How useful would you find each of these words to describe what you saw and/or heard? Not at all Some use A lot of use Essential admiration 0123 affection 0123 amusement 0123 annoyance 0123 anxiety 0123 tension 0123 trust 0123 wariness 0123 weariness 0123 worry 0123 to…

11. Labelling databases: Emotion labelling ‘Appraisal’ type fine labelling Scherer has proposed a scheme based on the idea that emotion is inseparable from the way the organism ‘appraises’ its situation. Some aspects of the theory are debatable, but that does not affect the descriptive power of the scheme. Appraisals are described under 5 broad headings. Intensity is effectively a 6 th category NOVELTY INTRINSIC PLEASANTNESS GOAL SIGNIFICANCE COPING POTENTIAL COMPATIBILITY with STANDARDS

12. Labelling databases: Emotion labelling ‘Appraisal’ type fine labelling The broad headings split to give 18 ‘dimensions’ NOVELTY  Suddenness High, medium, low  Familiarity High, medium, low  Predictability High, medium, low INTRINSIC PLEASANTNESS  global High, medium, low GOAL SIGNIFICANCE  Concern Relevance - person concerns (survival, bodily integrity, fulfillment of basic needs, self-esteem) - relationship concerns (establishment, continued existence and intactness of relationships, cohesion of social groups) - social order concerns (sense of orderliness, predictability in a social environment including fairness & appropriateness)  Outcome Probability High, medium, low  Relation to expectation Consonant, dissonant  Conduciveness to goals Conducive, obstrucive  Urgency High, medium, low

13. Labelling databases: Emotion labelling ‘Appraisal’ type fine labelling 18 ‘dimensions’ (contd) COPING POTENTIAL  Cause: Agent Self, other, nature  Cause: Motive Intent, negligence  Controllability of event High, medium, low  Power of agent High, medium, low  Adjustment possible to agent’s own goals? High, medium, low COMPATIBILITY with STANDARDS  External (norms or demands of a reference group ) High, medium, low  Internal (self ideal or internalized moral code ) High, medium, low

14. Labelling databases: Emotion labelling ‘Appraisal’ type fine labelling Data from Geneva suggest that there may be priveleged relationships between appraisal dimensions and signs (in both facial & vocal modalities) That is an exciting idea, and one HUMAINE is perfectly placed to explore.

15. Labelling databases: signs of emotion Automatic Properties that can be recovered by algorithms - basic (eg FAPs, HR, F0, pause boundaries) - selective (eg slope of F0 topline in a phrase) Perceptual Descriptions that humans find natural but machines don’t, eg ‘irregular rhythm’, ‘pointing’, ‘shrugging’ … Much to be worked out here - automatic type descriptions verified by humans, perceptual descriptions with parameters added by machine … - who needs what, and why? - for synthesis? for analysis?

16. Labelling databases: Context labels Slowest to fastest changing: Recording-context (recording style, acoustic quality, video quality) Agent characteristics (age, gender, race) Social setting (none, passive other, interactant, group) Intended audience (kin, colleagues, public) Physical constraint (unrestricted, posture constrained, hands constrained) Social constraint (pressure to expressiveness, neutral, pressure to formality) Overall communicative-goal (claim, sway, share feeling) Spatial focus (physical focus, imagined focus, none)

17. Labelling databases: finally… Labelling is about ‘discovering the relevant variables’ (Koch) Physics took 150 years to find the relevant variables to describe colour (Newton to Helmholtz). Psychology has been working on the relevant variables to describe personality for a century. If we make solid progress in 4 years, we will be doing very well. I guess we can.