1. KTH-TMH Stockholm loredana@speech.kth.se
Loredana Cerrato
Looking for feedback expressions
in speech corpora
KTH-TMH Stockholm

2. General idea
Study of the distribution, typology, use and function of verbal and non-verbal feedback expressions in human communication and in human machine communication.
Why?
Try to model human behaviour and try to reproduce it in speech technology application

3. Definition of feedback in human communcation
Allwood J. et al. 1992, On the semantics and pragmatics of linguistic feedback
“Control strategies of the channel of communication and of the informative flux, used by participant to a conversation to support the interaction”.
”Feedback mechanism is one of the most important cohesion devices in human conversation”
”Feedback consists of a set of verbal and bodily means used, in a minimal intrusive way, both to elicit and give information about the basic communicative functions: contact, perception, comprehension, attitudinal reactions”
Jens Alwood considers feedback mechanism as one of the most important cohesion devices in human conversation Part of the task for a listener is to make sure that the other person taking part in the conversation knows that he-she is paying attention, and indicate that he-she is at the same state in the conversation. This is mainly done by means of feedback expressions: an exchange of information that support the interaction. Feedback can be expressed vocally, verballly, gesturally or by any combination of the three.

4. General ways of expressing verbal feedback
short verbal expressions: yes, no, ok together with some prosodical and phonological phenomena (like varying pitch contour and vocalic lengthening)
short non lexical items: ah, ah-ah, mm, mh-mh
repetition: either the last word uttered by the interlocutor or by repeating the core words of the last sentence
reformulation: of the meaning of the received message.

5. General ways of expressing non-verbal feedback
Facial expressions: smile, eyebrow rising, gaze.
Head movements: nods, shakes, tilts ..
Gestures: hand, shoulder movements
Postures: trunk position.
-Combination of verbal and non verbal expressions
Request feedback: look at the the other
raise eyebrows
Give feedback: look at the other
-nod the head

6. Looking for verbal feedback expressions in a corpus of Italian spoken dialogues (human-human communication)
Material:
12 spontaneous-elicited dialogues of ca. 12 minutes each
AVIP database: Archive of Spoken Italian Varieties
Napoli, Bari, Pisa
(Collected in , it containes 16 dialogues transcribed at different levels: ortographic, phonetic, phonemic, prosodic and DA)
Map task technique
Cerrato L. 1999

7. Map task technique Only audio recordings Task-oriented conversation
Giver: gives instruction
Follower: receives the instruction to accomplish the task
FOLLOWER
GIVER
Yes…
park? What park?
I don´t have any parks
Go straight
till the lake, then at
the park turn left.
Two subjects sat in a small recording studio, separated by a wooden wall.
Each of them received a schematic map, which consists of an outline and roughly a dozen of labelled features: lago rosa, prato di Lillá, bar da Mimí, cinema Toto´.
Most features are common to both maps, but not all of them, and the participant are informed about this. One has a route drawn in and the other has not.
So the task of one subject is to “give instructions” to the other, in order to make him draw the route.
The speaker that gives instructions is called: giver,
the one who receives the interactions is called follower.
the auditive channel of communication is maximised
no visual channel, therefore feedback expressions are produced in a great number.

8. Analysis Identification and labeling of feedback
Distributional analysis
3) Acoustic phonetic analysis:
Duration and Pitch contour analysis
With the aim of detecting acoustic cues related to the function carried out by feedback expressions:
show attention/continuation of contact
aknowledge reception
agreement, acceptance
refusal
require clarifications
require further explanation
require further indications
show reception and acquisition of knowledge

9. Distribution of feedback expressions
1/3 of the turns consist of feedback expressions If a dialogue counts c.a 260 turns,
90 are feedback expressions
Feedback is expressed in a non-intrusive way
Distinction between giving and requesting feedback
Regional variants
A contradiction: if feedback is expressed in a ”non-intrusive way” can it be counted as a turn of its own?
which means that the speaker who gives the feedback has no intention to ”take the floor”, and doesn´t want to interrupr the turn of the interlocutor. But for practical resaons we transcribed feedback expressions as being turns.

10. Characterisazion of feedback expressions and their functions
Type of expression
Minimal responses
mmh ah si no
Function
show attention,continuation of contact
acknowledge reception
agreement, acceptance
refusal
F0 contour
F0 flat or slightly falling
Extended responses
Repetitions
Reformulations
Questions
require clarifications
require further explanation
require further indications
show reception and acquisition of knowledge
Rising F0

11. “Ah”: acquisition of knowledge
Intentional F0 variation can modify the meaning of the feedback expression
“Ah”: acquisition of knowledge
Flat F0: the acquired knowledge satisfies a previous need of the listener
Rising-falling F0: the information received doesn’t satisfy a need, but disappoints the listener

12. Material: 4 real dialogues between
Looking for verbal feedback expressions in a corpus of Swedish spoken dialogues (human-human communication)
Material: 4 real dialogues between
1 travel agent and 4 customers
recorded in a travel agency in Gothenburg
Göteborg Spoken Language Corpus.
Cerrato, L. 2002a

13. Analysis Identification and labeling of feedback
Distributional analysis
Analysis of its function:
show continuation of contact and perception
acknowledge comprehension
express a point of view (attitudinal reactions)
4) Acoustic phonetic analysis:
Duration and Pitch contour analysis

14. Results: Customers-agent distribution for each dialogue
The number of feedback expressions of the customers reaches more than 50% in all the dialogues.
In particular in dialogue 2, which was the shortest of all (only 21 turns per interlocutor) 87% of the customer’s turns consisted of feedback expressions!
Most of the short feedback expressions are produced with the simple intention of showing continuation of contact.

15. Feedback functions Customers Agent Continuation of contact
Acknowledgement of information
Expression of a point of view

16. Results of the acoustic analysis
· short duration and flat pitch contour
active listening attitude
not intend to take the turn;
·  longer duration and rising pitch contour
active listening attitude intention to take the turn
290 ms
När återkopplingen längd är kort och F0 är plat betyder det att talare vill bara visa intressen (”aktive lyssning”) och inte vill ta ordet.
När återkopplingens längd är längre och F0 stiger betyder det att talare inte bara vill visa intressen men vill även gärna ta ordet.
Skillnaden mellan korta och utsträckta återkoppling (m {j}a nä ) är ungefär 200 msec
550 ms

17. Conclusions on feedback
It is possible to categorize verbal feedback expressions according to their typology and to the meaning they convey.
Intentional variation of F0 seems to play an important role for assigning a particular meaning to the expressions used.
Question and hypothesis:
Is it possible to categorize visual feedback in the same way?

18. The study of ”visual feedback” in collaboration with Jens Allwod & c.
What is the relation between co-occurring visual and verbal feedback expressions?
Do visual expressions emphasise the meaning of the verbal expression?
Or do they convey different shades of meaning?
When visual feedback occur independently of verbal feedback expressions what is its function and meaning?
Need for a multimodal database

19. Feedback expressions are produced along different modalities
Verbal: words and sentences
Prosodic: speech rhythm, pauses, intensity, stress, intonation
Gestural: hand and arm movements
Facial: head and eye movement, gaze, smile and other facial expressions
Bodily: posture, trunk and leg movements

20. Feedback coding in a ”multimodal corpus”
Tool: Multitool (Leif Grönqvist, Magnus Gunnarsson)
Corpus: GSLC- Travel agency activity
Annotation: Multitier annotation of feedback expressions
Text: transcription (MSO)
Function: reports the function of the feedback expression and the function of the utterance that precede the feedback expression
Gesture: reports the “non-verbal feedback expressions”, analysed in terms of movements.
Gaze: [should] report the direction of the speakers’ gaze
For our study of “multimodal feedback” we use a multi-tier annotation, which consist of the following tiers displayed on the partiture of Multitool:

21. Feedback in human machine interactions
System’s feedback
Process feedback
”I am active” ”I am retrieving information”
Information feedback
”I have understood”: explicit, implicit feedback that indicates the reception and comprehension of the message
Explicit feedback:
User: I want to go to London
System: did you say you wanted to go to London?
Implicit feedback:
System: At what time do you wish to arrive in London?
Users’ feedback
Explicit, Implicit
To show:
Perception,
Understanding
Acknowledgement
Acceptation,
Refusal
Ecc.

22. Analysis of users’ feedback in human-machine interactions
AUGUST
18% of users (of 1206) gave feedback at least once
6% of utterances (of 6876) contained feedback expressions
In 89% of these cases, feedback appeared in a turn of its own.
ADAPT
94% of users (of 33) gave feedback at least once
18% of utterances (of 1845) contained feedback expressions
Only in 6% of these cases, feedback appeared in a turn of its own (it occurred mostly at the beginning of longer sentences)
Results adapted from Bell L. & Gustafson J. 2000;

23. August
August was an experimental spoken dialogue system able to give information about tourist sights and restaurants in Stockholm, about KTH and the life and works of August Strindberg.
The August database counts more than utterances produced by 2685 users, all visitors at the Stockholm Cultural Centre, where the dialogue system was displayed as part of the Cultural Capital of Europe '98 program.
we are interested in:
Multimodality – interest in multimodal systems is increasing. Humans communciate in a multimodal way.
We are interested in comparing human-human dialogues to human-machine dialogues.
We are interested in task-oriented natural dialogues.
We needed to get the users real reactions.
Lack of multimodal databases
Only Waxholm August and Adapt
Waxholm does not really contain “natural” elicited speech
August is a huge corpus of spontaneous speech, but presents many problems
We have chosen those users that carried out at least 5 turns with the dialogue system
Adapt dialogues were recordewd by means of the Woz technique, but trying to elicit natuaral speech (the task was quite vague)

24. Adapt
Adapt is an experimental dialogue system in the domain of Stockholm’ s real-estate.
The Adapt database was collected in 1999 by means of the Wizard of Oz technique.
It includes a total of 50 dialogues produced by 33 users.

25. Comparison between feedback strategies in Human-human communication and in human-machine communication
Material:
4 dialogues from GCSL and 4 dialogues from ADAPT
Method:
Distributional, functional and acoustic analysis
Results:
Humans produce feedback also when interacting with machines, but in different ways.
The majority of feedback expressions in GSLC travel agency dialogues consist of short monosyllabic words produced often on their own in a non-intrusive way.
In the interaction with ADAPT this kind of short feedback were never produced, they rather appeared at the beginning of longer sentences.
Cerrato L. 2002b

26. Measuring head movements related to feedback in collaboration with Linköping university
Quantitative measurement of the entity of head movement related to feedback expressions with the Qualisys MacReflex motion tracking system.
A pilot study:
recording of 4 elicited dialogues between humans recorded in controlled situation,
transcription and coding of the dialogues
identification of “typical visual” feedback
quantification of the movements
implementation of the results in talking heads to make them look more natural.

27. Main References
Allwod J ., 1999, The Swedish Spoken Language Corpus at Göteborg University. In Proceedings of Fonetik 99, Gothenburg Papers in Theoretical Linguistics 81.
Allwod J., Nivre J., Ahlsén E. , 1992, On the Semantics and Pragmatics of Linguistic Feedback in Journal of Semantics 64.
Allwood J., Grönqvist L., Ahlsén E., Gunnarsson M. ,2002 Annotations and Tools for an Activity Based Spoken Language Corpus", Proc. of 2nd SIGdial Workshop on Discourse and Dialogue, Aalborg, Denmark
Bell L & Gustafson .,2000 Positive and negative user feedback in a spoken dialogue corpus, In Proc of ICSLP 2000, Beijing, China.
Cerrato L, 1999 Il feedback verbale in dialoghi elicitati con la tecnica del map task. In Atti del GFS 10 Napoli, Italia, 25-35
Cerrato L. 2002a, Some charactersitcs of feedback expressions in Swedish, in TMH.OPSR Vol.43 Fonetik 2002 p
Cerrato L. 2002b, A comparison between feedback strategies in Human-Human and Human-machine communciation in proceedings of ICSLP-2002, Denver, Colorado USA, (in press)