Presentation on theme: "Using semantic associations for the detection of real-word spelling errors Jennifer Pedler School of Computer Science & Information Systems Birkbeck, University."— Presentation transcript:
Using semantic associations for the detection of real-word spelling errors Jennifer Pedler School of Computer Science & Information Systems Birkbeck, University of London.
Real-word spelling errors There is a bored (board) for messages. She wrote the appointment in her dairy (diary). One word mistakenly produced for another
Noun Pairs Word 1% TotalWord 2% Total college98%collage2% dinner96%diner4% road93%rod7% manner76%manor24% diary70%dairy30% reactor69%rector31% ear67%era33% lintel50%lentil50% Verb Pairs Word 1% TotalWord 2% Total unite99%untie1% ensure95%ensue5% inflict95%inflect5% expand94%expend6% relieve91%relive9% confirm87%conform13% mediate86%meditate14% carve84%crave16% depreciate66%deprecate34% inhibit64%inhabit36% Confusable Pairs
Semantic flavour Carve stone wood knife. oak walnut marble granite chisel Crave man food success. people chocolate attention
material stone wood marble granite oakwalnut WordNet Relationships Hyponymy/hypernymy ISA relation
stone, rock (countable, as in he threw a stone at me) stone, rock (countable, as in he threw a stone at me) stone, rock (uncountable, as in stone is abundant in New England) stone, rock (uncountable, as in stone is abundant in New England) stone (building material) stone (building material) gem, gemstone, stone gem, gemstone, stone stone, pit, endocarp (e.g. cherry stone) stone, pit, endocarp (e.g. cherry stone) stone (unit used to measure... weight) stone (unit used to measure... weight) stone (lack of feeling...) stone (lack of feeling...) WordNet senses for stone
material stone sandstonegranitelimestonemarble wood oakwalnutbeechash stone pericarp covering Stone, rock stone, pit, endocarp Branches for two senses for stone
material 0 (91) P 0.05 stone 19 (39) P 0.02 granite 4 P 0.002 marble 12 P 0.007 wood 10 (52) P 0.03 substance 0 (342) P 0.2 entity 0 (1028) P 0.6 Section of final carve hypernym tree
Scoring at run-time Seventeenth century dolls carved from wood fetch very high prices... Final Scores: carve 0.776 crave 0.223 Score Co-occurrenceDistancecarvecrave doll10.8990.101 century20.432 (0.338)0.568 (0.512) wood20.990 (0.891)0.010 (0.009) price60.768 (0.384)0.232 (0.116)
Spellchecking Test data: Flob Corpus I million words 1310 confusables Flob Original: Seventeenth century dolls carved from wood fetch very high prices... Flob Reversed: Seventeenth century dolls craved from wood fetch very high prices... Behaviour Accept as correct Flag as error Correct spellingError False alarm Ignored error Optimum performance: Minimise false alarms, maximise number of actual errors flagged
There is a famous early 15th century chest in the chapel bearing a carved scene of two jousters in action. Scores: carve 0.8326, crave 0.1674 prefer carve This method in particular has interested dairy farmers because, being alkaline, it helps the animal's rumen to function efficiently. Scores: dairy 0. 0.6797, diary 0. 0.3203 prefer dairy Selecting the confusable with the highest score - Examples Flob Original (confusable correctly accepted): Black hands caught him, united the rope... Scores: unite 0.2428, untie 0.7572 prefer untie The ensuring fight is real Douglas Fairbanks Jnr stuff. Scores: ensure 0.3233, ensue 0.6767 prefer ensue Flob Reversed (error correctly flagged):
CorpusFlob OriginalFlob Reversed Method Accept False alarm Flag Ignore Hyper Tree91%9%88%12% Choose more frequent 89%11%89%11% Selecting the confusable with the highest score - Results
... reforms now in the final making should signify a new era in local government in which results count for more than ructions. Scores: ear 0.5136, era 0.4864 prefer ear Setting a confidence threshold CorpusFlob OriginalFlob Reversed Threshold Accept False alarm Flag Ignore > 0.591%9%88%12% >0.795%5%80%18% >0.897%3%71%29% >0.999%1%51%49% >0.9999.9%0.1%5%95%
A fierce fight ensured.. Scores: ensure 0.0646, ensue 0.9354 prefer ensue You've bought the manner House and you've got a Ferrari. Scores: manor 0.9951, manner 0.0049 prefer manor Final Examples Less frequent word preferred with high level of confidence But the firm... it's gone bankrupt and we're all out on our ears. Scores: ear 0.0093 era 0.9907 prefer era The one remaining false alarm
Your consent to our cookies if you continue to use this website.