Domain-Specific Iterative Readability Computation Jin Zhao 13/05/2011

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 2WING, NUS Domain-Specific Resources

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Domain-Specific Resources 3WING, NUS Modular arithmetic page from Wikipedia Modular arithmetic page from Interactivate.com Domain-specific resources targets at varying audiences.

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Challenge for a Domain-Specific Search Engine 4WING, NUS How to measure readability for domain- specific resources?

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Literature Review Heuristic-based Readability Measures – Weighted sum of text feature values – Examples:  Flesch Kincaid Reading Ease (FKRE): [Flesch48]  Dale-Chall readability formula: [Dale&Chall48] 5WING, NUS Quick and indicative but often oversimplify

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Literature Review Natural Language Processing and Machine Learning Approaches – Extract deep text features and use supervised learning methods to generate models for readability measurement – Text Features  Unigram [Collins-Thompson04], Parse tree height [Schwarm05], Discourse relations [Pitler08] – Supervised learning techniques  Support Vector Machine (SVM) [Schwarm05], k-Nearest Neighbor (KNN) [Heilman07] 6WING, NUS More accurate but annotated corpus required and ignorant of the domain-specific concepts

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Literature Review Domain-Specific Readability Measures – Derive information of domain-specific concepts from expert knowledge sources – Examples:  Open Access and Collaborative Consumer Health Vocabulary [Kim07]  Medical Subject Headings ontology [Yan06] – Handles domain-specific concepts but expert knowledge sources are still expensive and not always available 7WING, NUS Key qualities of a good readability measure: effective, portable and domain-aware.

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Intuitions Use an iterative computation algorithm to estimate these two scores from each other Example: – Pythagorean theorem vs. ring theory 8WING, NUS A domain-specific resource is less readable if it contains more difficult concepts A domain-specific concept is more difficult if it appears in less readable resources

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Iterative Computation (IC) Algorithm Graph Construction – Construct a graph representing resources, concepts and occurrence information Score Computation – Initialize and iteratively compute the readability score of domain- specific resources and the difficulty score of domain-specific concepts – Two versions: heuristic and probabilistic Required Input – A collection of domain-specific resources – A list of domain-specific concepts 9WING, NUS

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Graph Construction 10WING, NUS …Pythagorean theorem can be written as a 2 + b 2 = c 2, where c represents the length of the hypotenuse… …The sine function (sin) can be defined as the ratio of the side opposite the angle to the hypotenuse… … right triangle Pythagorean theorem hypotenuse sine function cosine function … Resource 1 Resource 2 Concept List Pythagorean Theorem hypotenuse sine function Resource 1 Resource 2 right triangle cosine function

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Score Computation (Heuristic) 11WING, NUS wxyz abc Resource Nodes Concept Nodes Initialization – Resource Node (FKRE) – Concept Node (Average score of its adjacent nodes) wxyz abc Resource Nodes Concept Nodes Iterative Computation – Each node (Original score + average of the original scores of its adjacent nodes) Initialization Iteration 1

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Score Computation (Heuristic) 12WING, NUS wxyz abc Resource Nodes Concept Nodes wxyz abc Resource Nodes Concept Nodes Termination Condition – The rank order of the resource nodes stabilizes Iteration 2 Iteration 3

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Score Computation (Heuristic) Single-valued score for each node – Unable to handle concepts of varying difficulties Simple averaging in score computation – Difficult to incorporate sophisticated computational mechanisms 13WING, NUS

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Score Computation (Probabilistic) 14 wxyz abc Resource Nodes Concept Nodes Initialization – Resource Node (Sentence Sampling) – Concept Node (Resource Sampling) Initialization

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Score Computation (Probabilistic) 15 Iterative Computation – Modified Naïve Bayes Classification Original: Modified: Direct Adaptation: Resource Nodes Concept Nodes

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Evaluation Key qualities of a good readability measure – Effectiveness – Portability – Domain-awareness 16WING, NUS

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Effectiveness Corpus of Math Webpages Metrics: – Pairwise accuracy – Spearman’s rho Baseline: – Heuristic  FKRE – Supervised learning  NB, SVM, MaxEsnt  Binary concept features only 17WING, NUS PairwiseSpearmanIterations FKRE NB SVM Maxent HIC PIC

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Portability Different selection strategies – Resource selection at random – Concept selection at random – Resource selection by quality – Concept selection by TF.IDF Performance measurement at 5 levels – 20%, 40%, 60%, 80% and 100% of the original resource collection / concept list 18WING, NUS

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Portability 19WING, NUS Resource Selection Strategies Concept Selection Strategies

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Portability 20WING, NUS PairwiseSpearman FKRE NB SVM Maxent HIC PIC.75.55

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Domain-awareness Handling of domain-specific concepts – Simple yet effective – Concepts of multiple difficulty levels?  Converge to single value even in PIC  Splitting? (K-Means, GMM, etc.)  Other computational mechanisms? 21WING, NUS

Jin Zhao and Min-Yen Kan 13/05/2011 / 22 Conclusion Iterative Computation – Estimate the readability of domain-specific resources and difficulty of domain-specific concepts in a iterative manner – Effective, Portable and Domain-aware Future Work – Handling of concepts of multiple difficulty levels 22WING, NUS