July 9, 2003ACL An Improved Pattern Model for Automatic IE Pattern Acquisition Kiyoshi Sudo Satoshi Sekine Ralph Grishman New York University

July 9, 2003 ACL Automatic Pattern Acquisition The cost of manual construction of extraction patterns is very high. The cost of preparation of annotated data for supervised learning is still high. The recent trend of the researches on pattern acquisition is un- (semi-) supervised learning.

July 9, 2003 ACL Information Extraction Identifying entities from source text and mapping from source text to pre-defined table. “A smiling Palestinian suicide bomber triggered a massive explosion in the heavily policed heart of downtown Jerusalem today, …” Date: Location: Perpetrator: downtown Jerusalem A … suicide bomber today

July 9, 2003 ACL Local Context Local contexts provides a useful information to identify entities. “A smiling Palestinian suicide bomber triggered a massive explosion in the heavily policed heart of downtown Jerusalem today, …” Date: Location: Perpetrator: downtown Jerusalem A … suicide bomber today

July 9, 2003 ACL Extraction Pattern Generalize each instance of entity and its local context into an extraction pattern. “A smiling Palestinian suicide bomber triggered a massive explosion in the heavily policed heart of downtown Jerusalem today, …” triggered a massive explosion NE category Association Rule Perpetrator:

July 9, 2003 ACL Dependency Tree for Pattern Model Introducing syntax (dependency tree) clarify the relation of arguments with predicates. triggered a massive explosion A smiling Palestinian suicide bomber heart heavily policeddowntown Jerusalem today SBJ OBJ ADV IN

July 9, 2003 ACL Extraction Pattern models Predicate-Argument model (Yangarber et al. 2000) – Based on direct relation with a predicate Chain model (Sudo et al. 2001) – Based on a chain of modifiers of a predicate triggered explosion triggered triggered heart downtown Jerusalem

July 9, 2003 ACL Predicate-Argument model Predicate-Argument model is based on the direct relation of a predicate and its arguments. triggered a massive explosion heart heavily policeddowntown Jerusalem SBJ OBJ ADV IN

July 9, 2003 ACL Chain model Chain model can capture the chain of modifier with an arbitrary depth in the tree, regardless phrasal or clausal boundary. triggered a massive explosion heart heavily policed SBJ OBJ ADV IN (Sudo et al. 2001) reported 5% gain in recall with same level of precision over Predicate-Argument model.

July 9, 2003 ACL Problem Chain model contains only one node at each level of the tree. triggered a massive explosion heart heavily policeddowntown Jerusalem SBJ OBJ ADV IN

July 9, 2003 ACL Problem Lack of the context can make a pattern too general, causing a false match on irrelevant text. triggered a national financial crisis the Mexican peso last week SBJ OBJ ADV “ The Mexican peso was devalued and triggered a national financial crisis last week. ”

July 9, 2003 ACL Subtree model Generalization of Predicate-Argument and Chain model – Any connected subtree of a dependency tree will be considered as a candidate of extraction pattern. – Give reliable contexts as Predicate-Argument model does – Capable to capture long-distance relationship in dependency tree

July 9, 2003 ACL Subtree model Subtree model can provide more relevant contexts, as well as have a flexibility in traversing arbitrary depth in the tree. triggered a massive explosion heart heavily policeddowntown Jerusalem SBJ OBJ ADV IN

July 9, 2003 ACL Experiment Entity Extraction task – Identify if an NE instance is involved in scenario or not Management Succession – Person, Organization, Post (Position_Title) Murder Arrest – Arresting Agency (Organization), Suspect (Person), Charge – Source: Japanese newspaper 117,109 articles (Mainichi 1995) – Test: accumulated from Mainichi 1994 Succession 148 documents Arrest 205 documents

July 9, 2003 ACL Acquisition Method The target scenario is specified by TREC-like narrative description – “ Management Succession at the level of executives of a company. The topic of interest should not be limited to the promotion inside the company mentioned, but also includes hiring executives from outside the company of their resignation. ” [Translated from Japanese] Preprocessing – Dependency Analysis, NE-tagging Document Retrieval R

July 9, 2003 ACL Acquisition Method Count all possible subtrees in R – subtree-mining algorithm (Zaki et al. 2002) – make a Pattern List of those that conform the pattern model Rank each subtree R For each subtree i, number of times subtree i occurred in the documents in R

July 9, 2003 ACL Acquisition Method Count all possible subtrees in R – subtree-mining algorithm (Zaki et al. 2002) – make a Pattern List of those that conform the pattern model Rank each subtree R For each subtree i, number of documents in the source which contain subtree i

July 9, 2003 ACL Overlapping patterns Pattern List contains many overlapping patterns – (19) ( report) (( -wa) Happyo_suru) – (480) ( report that … be appointed) (( -wa) (Shunin_suru-to) Happyo-suru)  works as a weight on patterns with more relevant context [Translated from Japanese]

July 9, 2003 ACL  comparison

July 9, 2003 ACL Unsupervised Parameter Tuning Unsupervised text classification task by pattern matching – retrieved … 300 documents retrieved – random … 300 randomly selected – For each precision-recall curve for , calculate the area that the curve covers. Pearson correlation coefficient – r p = 0.80 with 2% confidence

July 9, 2003 ACL Extraction Performance

July 9, 2003 ACL Lessons learned Subtree vs. Chain – Too-general patterns got more penalized for Subtree model Penalize by Inversed Document Frequency (Subtree, Chain) More scenario-specific patterns got promoted (Subtree)

July 9, 2003 ACL Lessons learned Subtree vs. Predicate-Argument – Patterns with nominalized predicates Extraction patterns for headlines e.g. (promotion of ) (( -no) Shokaku) – Noun phrase patterns with chain of modifiers e.g. ( with ministerial authority) (((Daihyoken-no (Aru-  ( ))) [Translated from Japanese]

July 9, 2003 ACL Lessons to be learned Enhanced scoring function by modern IR technique. – Some techniques directly helps pattern acquisition e.g. relevance feedback – However, note the crucial difference between Pattern acquisition and IR Same pattern does not appear twice in a document. Generic variable instead of sticking to Named Entity categories as place holder. – How robust can a pattern be without semantic restriction?

July 9, 2003 ACL Conclusion We proposed Subtree model as a generalization of – Predicate-Argument model – Chain model Subtree model patterns overly performed better than other models in Entity Extraction tasks. Scoring function needs a special consideration for overlapping patterns. Unsupervised parameter tuning by text classification task.