1. Using Dialog Corpora to train a Chatbot
Bayan Abu Shawar and Eric Atwell, University of LEEDS
The paper presents the following:
ALICE and Elizabeth chatbot systems.
Examples of the Dialogue Diversity Corpus and its problems.
A Java program to convert from dialogue transcript to AIML Format.
Using Wmatrix to compare human and chatbot dialogue.

2. A Chatbot
A chatbot is a conversational agent that interacts with users using
natural language.
ALICE and Elizabeth chatbots are presented in this paper.
Both were adopted from ELIZA (Weizenbaum 1966), which
emulated a psychotherapist.

3. ALICE System
ALICE: the Artificial Linguistic Internet Computer Entity; a software robot that you can chat with using natural language.
ALICE language knowledge is stored in AIML files.
AIML: The Artificial Intelligence Mark up Language.

4. AIML Files are made up of :
Topics : each Topic file contains a list of categories
Categories: contain
Pattern: to match with user input
Template: represents ALICE output
Patterns can match parts of input: “divide and conquer”

5. The AIML Format
< aiml version=”1.0” >
< topic name=” the topic” >
<category>
<pattern>PATTERN</pattern>
<template>Template</template>
</category> ..
</topic>
</aiml>

6. Example involving <srai> - recursion:
Halo, what is 2 and 2 ?
HALO WHAT IS 2 AND 2
<sr/> <srai>WHAT IS 2 AND 2 </srai>
HALO WHAT IS 2 AND *
HELLO WHAT IS 2 *
Well hello there!
Hi. I was waiting to talk
Hello there!
Two
Four
Six
Hello there! Four

7. Elizabeth system (Millican 2002)
Knowledge is stored as a script in a text file.
Each line starts with a script command notation.
These notations are:
W: Welcome message Q: quitting message
V: Void input I: Input transformation
K: Key word pattern R: key word response
N: No match O: Output transformation
M: Memorise phrase &: Action to be perform
/ : Comment

8. Pattern Matching is more complex in Elizabeth
The matching process involves five phases:
    1. Matching with Input Transformation Rules.
2. Matching with Keyword patterns.
3. Matching with Output transformation rules.
4. Matching with Void or No keyword messages.
5. Performing any Dynamic processes.
… ALICE categories are simpler, easier to Machine Learn, but we can also convert from AIML format to Elizabeth script.

9. Example:
Input: I think my mum loves my brother more than me
Match Algorithm:
I think my mother loves my brother more than me
WHY DO YOU THINK [my mother loves my brother more than me]?
WHY DO YOU THINK YOUR MOTHER LOVES YOUR BROTHER MORE THAN YOU?
Respond: WHY DO YOU THINK YOUR MOTHER LOVES YOUR BROTHER MORE THAN YOU?

10. Machine Learning from the Dialog Diversity Corpus
The DDC is a collection of links to different dialogue corpuses in different fields.
Examples of these dialogue corpuses are:
MICAS Corpus
CIRCLE Corpus
CSPA Corpus
The TRAINS Dialogue Corpus
ICE-Singapore Corpus
Mishler Book Medical Interview

11. <ROTATES CEILING> everything else moves and we all get
MICAS Corpus
Michigan of academic Spoken English, a collection of transcripts of academic speech events recorded at the University of Michigan.
Astronomy transcript:
S1: circumpolar stars. So if I keep my pointer there, [S2: oh ]
<ROTATES CEILING> everything else moves and we all get
sick. <SS LAUGH> and we go backwards in time. And that’s
even more fun.
S2: make it go really really fast.
Problems:
Long monologs
Overlapping
More than two speakers
Extra annotations recorded actions such as <SS Laugh>

12. A collection of transcripts holding different tutorial sessions on
Circle Corpus
Centre for interdisciplinary research on constructive learning environments
A collection of transcripts holding different tutorial sessions on
topics such as physics, algebra and geometry.
Algebra transcript
TUTOR [ Opening remarks and asks student to read out aloud
and begin]
STUD [Reads problem] Mike starts a job at McDonald’s that will pay him 5 dollars and hour, Mike gets dropped off by his parents at the start of is shift. Mike works a “h” hour shift. Write an expression for how much he makes in one night?
[Writes “h*5 = how much he makes”]

13. Physics transcripts
T: [student name], I’d like you to read the problem carefully,
and then tell me your strategy for solving this.
S: ok
[Pause 17 sec]
hmm.
[Pause 6 sec]
T: thinking out loud as much as possible is good
Problem:
Different format structure were used to distinguish speakers
and linguistic annotation

14. CSPA Corpus
Corpus of Spoken Professional American-English
Includes transcripts conversations of various types.
LANGER: Hello, I’m delighted to be here.
I have carefully read and heard about the University of Albany, the State University of New York. And I’m also the director of the National Research Center on English Learning and Achievement.
STRICKLAND: Her mother wrote the stances.
(Laughter)
Problems:
Long turn monologues.
The transcript were not “anonymised”.

15. The TRAINS Dialogue Corpus
A corpus of task-oriented spoken dialogue, that has been used in several studies of human-human dialogue.
utt10 : what you'll have to do is you'll have to uh pick out an <sli> uh an engine <sli> and schedule a train to do that
utt11 : u: okay <sli> um <sli> engine <sli> two
utt12 : s: + okay +
utt13 : u: + from + Elmira
utt14 : s: + mm-hm +
Problem
Dealing with extra linguistic annotation such as ‘+’ and
<sli>

16. ICE-Singapore
International Corpus of English, Singapore English
<$B>
<ICE-SIN: S1A-099#33:1:B>
How how are things otherwise
<ICE-SIN:S1A-099#34:1:B>
Are you okay
<$A>
<ICE-SIN:S1A-099#35:1:A>
Uhm okay lah
Problems
Unconstrained conversations
A lot of linguistic annotation
Great variation in turn length

17. Mishler Book Medical Interviews
A scanned text image, including dialogue between patient and physician.
Problems
Scanned image cannot be converted to text format
Extra linguistic annotation

18. Desired dialogue corpus characteristics for machine learning
We developed a Java program to read a transcript from the DDC and convert it to AIML format in order to retrain ALICE.
Problems arise when extracting ALICE categories from the DDC:
No standard formats to distinguish between speakers.
Extra-linguistic annotations were used.
No standard format in using linguistic annotations.
Long turns and monologues.
Irregular turn taking (overlapping).
More than one speaker.
Scanned text-image not converted to text format.

19. To extract AIML, corpus data must be “normalized” to make it
look like chatbot transcripts:
1. Two speakers.
2. Structured format.
3. Short, obvious turns without overlapping, and without any
unnecessary notes, extras-linguistic expressions etc.

20. The Java Program
Converts the dialogue transcript to AIML format.
The output AIML is used to retrain ALICE.
The first speaker is the pattern, the second is the template.

21. Example from the MICAS corpus:
S1: circumpolar stars. So if I keep my pointer there, [S2: oh ] <ROTATES CEILING> everything else moves and we all get sick. <SS LAUGH> and we go backwards in time. And that’s even more fun.
S2: make it go really really fast.
The AIML category generated by the program is:
<category>
<pattern> CIRCUMPOLAR STARS SO IF I KEEP MY POINTER THERE EVERYTHING ELSE MOVES AND WE ALL GET SICK AND WE GO BACKWARDS IN TIME AND THAT’S EVEN MORE FUN</pattern>
<template> make it go really really fast.</template>
</category>

22. Other differences we need to “learn” :
Using Wmatrix to compare human and chatbot dialogue
Wmatrix is a tool to provide a data driven method to compare
corpora, three levels: Word, PoS and semantic tag analysis.
The comparisons results are viewed as frequency lists ordered by
log-likelihood ratio (LL).
LL values indicate the most important differences between
corpora.
Wmatrix was used to compare human-to-human dialogues
extracted from the DDC corpora and human to computer
dialogues extracted from chatting with ALICE.

23. Sorted by log-likelihood value
ALICE and Astronomy Word Comparison
Sorted by log-likelihood value
Item O % O % LL do i we so
and
Emily
you
this

24. Sorted by log-likelihood value
ALICE and Astronomy POS Comparison
Sorted by log-likelihood value
Item O % O % LL
PPIS VD
PPIS CC
PPY CS ZZ DD

25. Sorted by log-likelihood value
ALICE and Astronomy Semantic Comparison
Sorted by log-likelihood value
Item O %1 O % LL
Z Personal names
E Liking
W The universe
M Location and direction
M Moving, coming
H Residence
F Food
Q Speech act

26. Sorted by log-likelihood value
French word Comparison between Chatbot and real dialogue
Sorted by log-likelihood value
Item O % O % LL
conversation
euh
danser
fais de
coucher
football

27. Conclusions
We train ALICE rather than Elizabeth because AIML format is closer to the markup language and the simple pattern matching technique used by ALICE.
2. Dialogue Diversity corpus (DDC) illustrates huge diversity in dialogues: genres, speaker background/register, mark-up and annotation.
3. It will be useful to agree standards for transcription and mark-up format.
4. Wmatrix has shown further differences between chatbot and real dialogue.

28. Future Work
Expanding AIML files using least frequent word and
investigating how to incorporate corpus-derived linguistic
annotation into an Elizabeth-style chatbot pattern file.