Presentation is loading. Please wait.

Presentation is loading. Please wait.

Using semantic associations for the detection of real-word spelling errors Jennifer Pedler School of Computer Science & Information Systems Birkbeck, University.

Similar presentations


Presentation on theme: "Using semantic associations for the detection of real-word spelling errors Jennifer Pedler School of Computer Science & Information Systems Birkbeck, University."— Presentation transcript:

1 Using semantic associations for the detection of real-word spelling errors Jennifer Pedler School of Computer Science & Information Systems Birkbeck, University of London.

2 Real-word spelling errors There is a bored (board) for messages. She wrote the appointment in her dairy (diary). One word mistakenly produced for another

3 Noun Pairs Word 1% TotalWord 2% Total college98%collage2% dinner96%diner4% road93%rod7% manner76%manor24% diary70%dairy30% reactor69%rector31% ear67%era33% lintel50%lentil50% Verb Pairs Word 1% TotalWord 2% Total unite99%untie1% ensure95%ensue5% inflict95%inflect5% expand94%expend6% relieve91%relive9% confirm87%conform13% mediate86%meditate14% carve84%crave16% depreciate66%deprecate34% inhibit64%inhabit36% Confusable Pairs

4 Semantic flavour Carve stone wood knife. oak walnut marble granite chisel Crave man food success. people chocolate attention

5 material stone wood marble granite oakwalnut WordNet Relationships Hyponymy/hypernymy ISA relation

6 stone, rock (countable, as in he threw a stone at me) stone, rock (countable, as in he threw a stone at me) stone, rock (uncountable, as in stone is abundant in New England) stone, rock (uncountable, as in stone is abundant in New England) stone (building material) stone (building material) gem, gemstone, stone gem, gemstone, stone stone, pit, endocarp (e.g. cherry stone) stone, pit, endocarp (e.g. cherry stone) stone (unit used to measure... weight) stone (unit used to measure... weight) stone (lack of feeling...) stone (lack of feeling...) WordNet senses for stone

7 material stone sandstonegranitelimestonemarble wood oakwalnutbeechash stone pericarp covering Stone, rock stone, pit, endocarp Branches for two senses for stone

8 material 0 (91) P 0.05 stone 19 (39) P 0.02 granite 4 P marble 12 P wood 10 (52) P 0.03 substance 0 (342) P 0.2 entity 0 (1028) P 0.6 Section of final carve hypernym tree

9 entity carve 0.96 crave 0.04 substance carve 0.87 crave 0.13 material carve 0.99 crave 0.01 food carve 0.41 crave 0.59 foodstuff carve 0.22 crave 0.78 stone carve 1.0 crave 0.0 Merged tree

10 Scoring at run-time Seventeenth century dolls carved from wood fetch very high prices... Final Scores: carve crave Score Co-occurrenceDistancecarvecrave doll century (0.338)0.568 (0.512) wood (0.891)0.010 (0.009) price (0.384)0.232 (0.116)

11 Spellchecking Test data: Flob Corpus I million words 1310 confusables Flob Original: Seventeenth century dolls carved from wood fetch very high prices... Flob Reversed: Seventeenth century dolls craved from wood fetch very high prices... Behaviour Accept as correct Flag as error Correct spellingError False alarm Ignored error Optimum performance: Minimise false alarms, maximise number of actual errors flagged

12 There is a famous early 15th century chest in the chapel bearing a carved scene of two jousters in action. Scores: carve , crave prefer carve This method in particular has interested dairy farmers because, being alkaline, it helps the animal's rumen to function efficiently. Scores: dairy , diary prefer dairy Selecting the confusable with the highest score - Examples Flob Original (confusable correctly accepted): Black hands caught him, united the rope... Scores: unite , untie prefer untie The ensuring fight is real Douglas Fairbanks Jnr stuff. Scores: ensure , ensue prefer ensue Flob Reversed (error correctly flagged):

13 CorpusFlob OriginalFlob Reversed Method Accept False alarm Flag Ignore Hyper Tree91%9%88%12% Choose more frequent 89%11%89%11% Selecting the confusable with the highest score - Results

14 ... reforms now in the final making should signify a new era in local government in which results count for more than ructions. Scores: ear , era prefer ear Setting a confidence threshold CorpusFlob OriginalFlob Reversed Threshold Accept False alarm Flag Ignore > 0.591%9%88%12% >0.795%5%80%18% >0.897%3%71%29% >0.999%1%51%49% > %0.1%5%95%

15 A fierce fight ensured.. Scores: ensure , ensue prefer ensue You've bought the manner House and you've got a Ferrari. Scores: manor , manner prefer manor Final Examples Less frequent word preferred with high level of confidence But the firm... it's gone bankrupt and we're all out on our ears. Scores: ear era prefer era The one remaining false alarm

16 Semantic flavour Diary entry (40) appointment (5) engagement (6).. Dairy product (101) farmer (62) cow (56)... chronicle (1) meeting (1) schedule (1) rota ?

17 Carve (substance) Crave (substance) Confusable (hypernym) P(H|C) N. Co- oc Tot Co-oc 393 P(C) Crave (foodstuff) Carve (foodstuff) entity carve 0.96 crave 0.04 substance carve 0.87 crave 0.13 material carve 0.99 crave 0.01 food carve 0.41 crave 0.59 foodstuff carve 0.22 crave 0.78 stone carve 1.0 crave 0.0 Merging the trees W(C|H) P(H)P(C|H)


Download ppt "Using semantic associations for the detection of real-word spelling errors Jennifer Pedler School of Computer Science & Information Systems Birkbeck, University."

Similar presentations


Ads by Google