HUMAINE summer school on databases: opening orientation Roddy Cowie, Ellen Douglas-Cowie, Edelle McMahon & Cate Cox

Practicalities  Welcome!  Fire alarms  Break and after  Groups  Logons  Toilets  Photo

INTRODUCTION: DATABASES in HUMAINE  The aim of this session is to put the Summer school in context.  To do that it reviews basic points about HUMAINE basic points about HUMAINE (context we are working in)(context we are working in) the kind of database work relevant to HUMAINE the kind of database work relevant to HUMAINE the databases that currently exist the databases that currently exist  Sets out natural targets for research on databases in the HUMAINE context based on report delivered in June (D5c) based on report delivered in June (D5c)

INTRODUCTION: DATABASES in HUMAINE  That background sets the scene for summer school  its function is getting hands-on feel for techniques we might use to meet the targets  and establishing relationships among people who will do it.

Basic points about HUMAINE (context we are working in)  HUMAINE is about developing a particular approach to research on emotion  It is spelled out in the Technical Annex (TA for short)  We are committed to follow through what we said in the TA,  and we need to be clear what it was.

Basic points about HUMAINE (context we are working in) Key points: HUMAINE is oriented to  Applications – but applications in the long term  Emotion – but emotion in a broad sense  Emotion as a factor in action and interaction, not emotions as isolated episodes These affect database objectives hugely

Basic points about HUMAINE: Applications in the long term  Humaine aims to bolster sound long-term development by clarifying the scientific, cultural and ethical underpinnings of emotion-oriented computing; clarifying the scientific, cultural and ethical underpinnings of emotion-oriented computing; developing accessible literature and standards …; developing accessible literature and standards …; and establishing enduring relationships among research groups in the key areas …. and establishing enduring relationships among research groups in the key areas ….  To achieve real European leadership in this area, coordinated efforts must go beyond the more engineering- and application-oriented developments undertaken in leading countries outside the EU…

Basic points about HUMAINE: emotion in a broad sense  We [will] consider emotion in an inclusive sense rather than in the narrow sense of episodes where a strong rush of feeling briefly dominates a person’s awareness.  … emotion in the broad sense pervades human communication and cognition. Human beings have positive or negative feelings about most things, people, events and symbols. These feelings strongly influence the way they attend, behave, plan, learn and select.

Basic points about HUMAINE: emotion in action & interaction  Human beings have positive or negative feelings about most things, people, events and symbols.  These feelings strongly influence the way they attend, behave, plan, learn and select.  The feelings are conveyed e.g. in faces, voices, gestures, and postures; and people judge others by the way they respond to such signals.  They seek out situations that are emotionally rich (not necessarily pleasant),  and are persuaded as much by emotional appeal as by factual content. The TA commits us to approach emotion in a way that embraces these aspects

Basic points about HUMAINE: terms to sum up Bounded emotions  specific kinds of mental state where a strong rush of feeling dominates a person’s awareness for a relatively short time Pervasive emotion  a quality that ‘colours’ almost any mental state to a greater or lesser extent  and therefore influences a wide range of actions & interactions HUMAINE is committed to address both – and therefore, so is its work on databases

Basic points about HUMAINE: relevant kinds of dbase work Databases can play two kinds of role in research  Supportive role of dbase is to let people fill in or check details of processes they understand in general terms, at least as well as they think they need to. role of dbase is to let people fill in or check details of processes they understand in general terms, at least as well as they think they need to.  Provocative role is to collect examples that help people to expand and restructure their thinking about the area. role is to collect examples that help people to expand and restructure their thinking about the area. To understand pervasive emotion in action & interaction, the database strand has to play a provocative role

Basic points about HUMAINE: summary By the nature of HUMAINE, we need dbases to  Lay the ground for long term applications  Deal with pervasive emotion as well as bounded episodes  Show how emotion figures in action and interaction  Plays a provocative role as well as a supportive one. How does available material measure up?

Background to HUMAINE database strand Review of existing material (based on D5c, available on the portal)

Review of existing material:database groups  3 main groupings of databases (see handout) multimodal multimodal face only face only speech only speech only  A lot of material, giving impression that we have the information  But on closer inspection …

Review of existing material  The older material Overwhelmingly goes for bounded emotions Overwhelmingly goes for bounded emotions Often relies on acted data Often relies on acted data Rarely shows how modalities combine Rarely shows how modalities combine Rarely shows emotion affecting action and interaction Rarely shows emotion affecting action and interaction Often uses a narrow or arbitrary range of emotions Often uses a narrow or arbitrary range of emotions  Suppose you have a practically important situation involving emotional colouring, signalled through multiple modalities, affecting action and interaction - how likely that you will find empirical evidence relevant to it?

Review of existing material  The progress Emerging work on multimodal databases – Belfast Naturalistic Database, SMARTKOM Emerging work on multimodal databases – Belfast Naturalistic Database, SMARTKOM Emerging interest in naturalistic sources Emerging interest in naturalistic sources Lost luggage, call centre Emerging interest in ‘colouring’ Emerging interest in ‘colouring’ AIBO robot, Campbell AIBO robot, Campbell  - trends we need to consolidate.

Agenda for HUMAINE work on databases Goal: generate an exemplar, ie  a piece of work which exemplifies how a key problem in the area can be solved in a principled way;  and which also demonstrates how work focused on that area can integrate with work focused on the other areas

Agenda for HUMAINE work on databases  It will be a library of data to be accessed by research on emotional states and their expression  But what will be the contents?  D5c outlines preliminary ideas  They define the kinds of technique we want to explore in the summer school - tooling up for the tasks we defined in D5c - tooling up for the tasks we defined in D5c

HUMAINE library of data: types of archive  1. Historical Key databases assembled or otherwise made accessible Key databases assembled or otherwise made accessible  2. Supportive (Reactive): Related to needs of other workpackages. eg annotated recordings for evaluation of signal processing routines eg annotated recordings for evaluation of signal processing routines gesture-rich samples support development of coding systems gesture-rich samples support development of coding systems (unless pre-existing material meets needs). (unless pre-existing material meets needs).  3. Provocative (Proactive): material that will challenge fundamental research in the area material that will challenge fundamental research in the area  4. Metaknowledge: procedures, systems, summary analyses, etc. procedures, systems, summary analyses, etc.

HUMAINE library of data: emotional scope  There should be a prestated framework defining the emotional and emotion-related states that should ideally be covered (even if there are gaps) (even if there are gaps)  Standard lists of ‘primaries’ are not an appropriate framework  D5c has a list based on Cowie et al 1999 and Scherer 2004 – What do we think of it? Watch this space … What do we think of it? Watch this space …

HUMAINE library of data: levels of representativeness Naturalistic material  the point of reference for other material  but we need to understand its limits as well as its value Acted material  we would prefer in principle to avoid  but how much of the table can we sample in any other ways? Induction techniques  we propose to invest effort in selecting and developing appropriate techniques

HUMAINE library of data: contexts (your team scored)  Solo (often in a laboratory) emotion (if it is expressed at all) is likely to be impoverished emotion (if it is expressed at all) is likely to be impoverished  Sedentary human interaction sitting talking to each other sitting talking to each other  Mobile human interaction possibility of whole-body movements & use of objects possibility of whole-body movements & use of objects  Social facilitation a group can amplify various kinds of action a group can amplify various kinds of action  Task-dominated scope for expression is constrained by the task (eg driving). Surface manifestations vary massively with the task. Are there deeper constants? scope for expression is constrained by the task (eg driving). Surface manifestations vary massively with the task. Are there deeper constants?  Machine interaction important for us – but should be understood in the broad range important for us – but should be understood in the broad range

HUMAINE library of data: modalities  audio and visual recorded as norm recorded as norm (but naturalism may outweigh …) (but naturalism may outweigh …)  performance measures (error rate, reaction time, etc) (error rate, reaction time, etc) where there is a task where there is a task  psychophysiological measures (heart rate, skin conductance, temperature, breathing) (heart rate, skin conductance, temperature, breathing) occasional – interesting but restrictive occasional – interesting but restrictive  brain scans, kinetic records … realistically beyond our scope. … realistically beyond our scope.

HUMAINE library of data: diversity and scale  We have to recognise as issues Gender Gender Culture Culture Age Age  Can we collect hundreds of samples per closely- defined cell? to be negotiated for supportive databases to be negotiated for supportive databases normally, establish the technique; normally, establish the technique; let others replicate. let others replicate.

HUMAINE library of data: measurement & labelling  Assessments of the person’s emotional state  Codings of the signs that reveal it  What is WP5’s responsibility?  Many techniques are not an option too labour-intensive too labour-intensive too restricted in application too restricted in application  It does make sense to explore techniques that are efficient efficient of general interest of general interest

HUMAINE library of data: structure and accessibility  Format people can access and understand  Systematic approach to choice of units  Mechanisms for distribution  … and more.

Background to the Summer School: summary By the nature of HUMAINE, we need dbases to  Deal with pervasive emotion as well as bounded episodes, show how emotion figures in action and interaction, play a provocative role as well as a supportive one  They need to cover a satisfying range of emotions, be acceptably natural, address context effects, cover key modalities, reflect basic forms of diversity, include basic labelling, be competently structured.

Background to the Summer School: summary HOW  Between us, we have a range of tools at our disposal  The summer school is about getting to grips with them -  bearing in mind what we want to achieve