1. The MSR ESL Assistant: Detecting and correcting non-native errors in English
Michael Gamon, Chris Brockett, William B. Dolan, Jianfeng Gao, Dmitriy Belenko (Microsoft Research), Alexandre Klementiev (University of Illinois at Urbana Champaign), Claudia Leacock (Butler Hill Group)

2. Making NLP useful

3. Overview Motivation Part I: The system
Error statistics
Different solutions for different errors
Machine learned classifiers for preposition and determiner errors
Adding a language model and web-based examples
Part II: Evaluation on native and non-native data
Part III: Usage and interactions

4. Motivation: The Story of the Disappearing and Reappearing Slide
750M people use English as a second or foreign language (vs. 375M as first language)
74% of use of English is between non-native speakers
As many as 300M people study English in China

5. Error statistics Previous studies:
Articles and prepositions account for 20% - 50% of ESL errors
Prepositions are difficult for learners with various L1 backgrounds

6. Error statistics NICT Japanese Learners of English corpus:
26.6% of errors are determiner related
10% of errors are preposition related
CLEC Chinese Learners’ Corpus:
10% of errors determiner and number related
2% preposition related, 5% collocation errors (which often involve prepositional collocations)

7. Most frequent errors made by East Asian non-native speakers
Preposition presence and choice: Finally, the pollution on the world is serious.
Definite and indefinite determiner presence and choice: We should think whether we have ability to do it well.
Noun pluralization: So other works couldn't be done in adequate times.
Gerund/infinitive confusion: So, money is also important in improve people's spirit.
Auxiliary verb presence and choice: The fire will break out, it can do harmful to people.
Over-regularized verb inflection: It was builded in 1995.
Adjective/noun confusion: There was a wonderful women volleyball match between Chinese team and Cuba team.
Word order (adjective sequences and nominal compounds): A pop British band called "Spice Girl" has sung a song.

8. Different errors – different solutions
Prepositions and articles: much contextual information needed
Over-regularized verb morphology: local information is enough
Noun number: local information (mass noun, quantifier etc) is enough
Machine learned approaches for (1), simple heuristics for (2) and (3).
Total number of error modules: 4 machine-learned modules, 19 heuristic models

9. Modeling preposition and determiner errors
What data?
Domain
Sentences
Encarta encyclopedia
487,281
Reuters newswire
567,394
UN proceedings (Hansard)
500,000
Europarl
Web scraped, using an algorithm similar to STRAND (Resnik and Smith 2003)
Total
2,554,675

10. Modeling preposition and determiner errors
Preprocessing: tokenization, POStagging
Heuristic algorithm (based on POS tags): find left edges of NPs (potential sites for prepositions and articles)
For each potential site of a preposition or article:
Target feature 1: preposition/article present or absent
Target feature 2: choice of preposition/article (if present)
Contextual features (POS tags to the left/right, tokens to the left/right)
Maximum Entropy classifier

11. Modeling preposition and determiner errors
Training data: 2.5M sentences: Encarta, Reuters, UN, EU, web scraped
Classifier
Training cases
Article presence/absence
11.9M
Article choice
4.3M
Preposition presence/absence
16.1M
Preposition choice
6.5M

12. Adding a language model
LM accuracy alone is not sufficient: 58.36%

13. Adding web search
Observation: Non-native speakers often use the web to validate word choice

14. Show suggestions and originals in context

15. Evaluation (1): native text (correct usage of prepositions and determiners)
Splitting the original training data into 70% training, 30% test
Note: classification is split into two questions:
Should there be a determiner/preposition?
If yes, which one should it be? (Prepositions: limiting the set to 12 choices that are common in errors: about, as, at, by, for, from, in, like, of, on, since, to, with, "other“)

16. Articles: results on native text
Presence/absence
Choice model
Combined
Accuracy
89.94%
89.66%
86.76%
Baseline
64.04% (no article)
77.73% (definite)
58.91%
Presence/absence model
Precision
Recall
Presence
87.89%
83.54%
Absence
91.01%
93.54%
Choice model
Precision
Recall
the
91.48%
95.60%
a/an
81.77%
68.94%

17. Prepositions: results on native text
Presence/absence
Choice model
Combined
Accuracy
88.57%
66.23%
76.77%
Baseline
59.57% (no preposition)
27.07% (of)
42.00%
Presence/absence
Precision
Recall
Presence
86.76%
84.66%
Absence
89.75%
91.23%

18. Results on individual prepositions
Choice model
Precision
Recall as
77.28%
62.77% on
68.17%
56.69% of
71.91%
87.54%
about
60.17%
35.12% to
67.92%
64.48% by
63.37%
52.62% at
64.92%
52.85% in
61.81%
69.87%
since
62.62%
20.67%
with
63.45%
47.94%
from
59.58%
38.36%
other
56.97%
55.14%
for
58.46%
47.91%

19. Evaluation(2): Human evaluation
Spellchecked Chinese Learners’ Corpus (CLEC)
Test set scraped from the web
User data

20. Spellchecked Chinese Learners’ Corpus (CLEC)
1 million words of English compositions
collected from Chinese learners of English in China with differing levels of proficiency:
senior secondary school students
English-major university students
non-English-major university students

21. Web scraped data collected by a vendor for MSR
Scraped from 489 personal web pages and blogs of non-native speakers/students of English, of Korean, Chinese, or Japanese L1 background
6746 sentences, 1k selected randomly for our evaluation
Education level ranges from high school to graduate school, professionals are also included
Gender balanced

22. Intermission: Pie charts

23. Prepositions

24. Articles

25. Broader categories CLEC Web scraped adj related verb related
noun related
prep related
CLEC
Web
scraped

26. Usage of the prototype and evaluation of user data

27. Page views per day
Live Translator snafu
Beijing Olympics

28. User location country visits percentage China 51,285 26.80%
United States
28,916
15.10%
Taiwan
25,753
13.40%
Korea - South
12,934
6.80%
Hong Kong
8,826
4.60%
Brazil
4,648
2.40%
Canada
3,917
2.00%
Germany
3,077
1.60%
United Kingdom
2,928
1.50%
Japan
2,581
1.30%
Italy
2,579
Spain
2,557
Russian Federation
2,448
Saudi Arabia
2.021
1.10%

29. Users and Sessions

30. Repeat users (2)
Highlight the 4 or more category

31. Return visits

32. Collected data

33. User interactions

34. 84% of squiggles are examined by the user
This is for all frequent users that are being evaluated (eg, not include random sentences). Also not include Thesuarus

35. Are users accepting the right suggestions?
suggested
accepted

36. In summary Large market for ESL proofing tools
Detecting and correcting non-native errors is a non-trivial and interesting research problem
We may already be at a point where the technology starts to be useful

37. Some open questions
How does the accuracy of POStagging influence the accuracy of the overall system?
How can we best leverage the user behavior as a supervision signal?

38. Some ideas Using web result counts directly as an LM approximation
Using web result counts as (part of a) supervision signal for ML
Combining more sources of evidence: LMs trained on different data sets etc
Build one single model, including LM scores
Active learning to optimize thresholds