1. On Defining Cephalic Gesture Categories
Natalya V. Sukhova,
Yulia V. Nikolaeva
Institute of Linguistics RAS,
Lomonosov Moscow State University
NUST MISiS

2. Outline of the Talk About the Corpus.
Annotation of Kinetic Phenomena. General Concept.
Cephalic Gestures. Approaches to Analysis.
Annotation of Cephalic Gestures. Our Approach.
ICMC 2017, Osnabrück

3. 1a Language As It Is: Multimodal (Multichannel) Resource
Goal is to create a resource approaching the actual richness of human discourse
Registered phenomena:
verbal structure
prosody
eye gaze
head movements
hand gestures
torso position
ICMC 2017, Osnabrück

4. Kinetic / visual modality
1b Spoken Multimodal Discourse
Discourse
Vocal modality
Verbal channel
Prosody
Kinetic / visual modality
Eye gaze
Posture
Gesture
Language?
Other modalities
Proxemics
Other
channels
Facial
expression
Intonation
Many other
components
Hands
Head
Most multimodal studies
ICMC 2017, Osnabrück 4

5. 1с RuPeCS Russian Pear Chats and Stories (Pear Film, Chafe ed. 1980):
24 conversations
96 native speakers
total duration is 10 hours
words
11 media files for each conversation
24 written retellings
Project web-site:
ICMC 2017, Osnabrück 5

6. 1d The Scene Tobii glasses Listener Listener Narrator Reteller
Нужно ли добавить подробности про процедуру на сам слайд?
Reteller
Commentator
Commentator 6 6 6 6

7. 2a Manual Gestures
Communicatively significant spontaneous hand/arm movements, accompanying speech
Discourse functions
Adding to speech contents, facilitating verbalization, appealing to an interlocutor, regulating turn-taking, supporting rhythmical structure
Standing out as a figure against kinetic background (other movements and stillness periods)
Kinetic structure
Distinct articulatory effort, velocity, trajectory, movement direction, hand shape and orientation, gesture range, location in gesture space
ICMC 2017, Osnabrück

8. 2b Distinguishing Movements and Positions
Neutral (resting) positions: set of preferred hand/arm positions a speaker regularly returns to. Hand/arm posture changes: hand/arm resting position changing movements. Self-adaptors (adaptors, manipulators): physiologically motivated movements or movements with practical purposes. Gestures Typical kinetic patterns of a speaker form a gesticulation portrait (cf. ‘prosodic portraits’ in Kibrik 2009).
Здесь я скажу, что у рук все очень хорошо различается, гораздо лучше, чем у головы
hand/arm resting position changing movements - вот это не очень понятно, что имеется в виду?
ICMC 2017, Osnabrück

9. 3a Cephalic Gesture Categories
MOVEMENT is all motor activity of a head.
GESTURE is all motor activity that potentially has a meaning.
HEAD POSTURE (point, position) brings about a change of a head position in at least one direction and may (or may not) last for some time.
ICMC 2017, Osnabrück

10. 3b Levels of Analysis Form – physical description
Meaning – e.g. nods, shakes
Function – a role in a discourse, e.g. backchannelling
See Hadar et al. 1985; Bull 1987; McClave 2000; Allwood, Cerrato 2003; Benoit, Caplier 2005; Kousidis et al
ICMC 2017, Osnabrück

11. 3c Our Approach to Cephalic Gesture Categories. Tiers in ELAN
What does it show?
Its assignment
Cephalic Gesture Chain
A set of cephalic gestures going together without a pause
To define a structural unit of gestures (form)
Cephalic Gesture
Tags
Comments
A movement which is physiologically prominent, visible and potentially meaningful for communication
To single out a salient head movement from a motor flux (form).
Tags give the physical description of a movement (form).
Comments provide the meaning of some movements designating nods, shakes and jerks if any.
Cephalic Posture
Posture change
A continuum which arises after a movement changes in any way (plane, axis): posture change brings it about
To see all discourse in terms of static and dynamic units (form and function)
ICMC 2017, Osnabrück

12. 4a Physical Form # Plane Axis Example 1 Frontal Anteroposterior
Backward / Forward 2
Horizontal
Vertical
Up / Down 3
Sagittal
Mediolateral
TurnLeft / TurnRight
ICMC 2017, Osnabrück

13. Vocabulary of movements
4b Formal Vocabulary
Vocabulary of movements
Description
SIMPLE
Down
Downward head move Up
Upward head move
TurnLeft
Move of the head turning on the left side
Straighten
Head is straightened
COMPOUND
Backward-forward
Head moves backward and forward in a single continuous movement
Slide
Displacement of the head from side to side
TurnRepeated
Head turns from side to side (shake)
DownUp (UpDown) Repeated
Head moves up and down in a single continuous movement (nod)
ICMC 2017, Osnabrück

14. Picture in ELAN
ICMC 2017, Osnabrück

15. CONCLUSIONS & PERSPECTIVES
Current annotation scheme:
provides detailed annotation of movements relevant in communication;
takes into account most frequent complications;
makes a part of a consistent annotation system for several kinetic channels (presuming that they are used simultaneously to pass along one message);
allows comparing head and manual gestures to vocal channel units and other kinetic channels’ units.
Further research:
extending annotated subcorpus;
measuring gesticulation portrait features.
ICMC 2017, Osnabrück

16. Thank you for attention!
ICMC 2017, Osnabrück