Reyyan Yeniterzi Weakly-Supervised Discovery of Named Entities Using Web Search Queries Marius Pasca Google CIKM 2007

Motivation  Name entities  essential during the construction of knowledge bases from Web  helpful in various NLP tasks; like parsing, coreference resolution …  constitute a significant part of the Web search queries  helpful in building verticals in Web search

Previous works  Mining query logs  to improve various IR tasks re-ranking of retrieved documents query expansion spelling correction  Large-scale IE  mainly on document collections  ignoring the collective knowledge embedded in noisy search queries  This is the first work that applies name entity finding to Web search query logs

Extraction from Query Logs  Given  A set of target classes  A set of seed instances  The goal  To extract relevant class instances from query logs  Without using Any domain knowledge Any handcrafted extraction pattern

Overview of the System

Step 1: identification of query templates that match the seed instances

Step 2: identification of candidate instances

Step 3: internal representation of candidate instances  query: prefix candidate_instance postfix  entry: prefix postfix  weight of an entry = frequency of the query

Step 4: internal representation of seed instances  introducing weak supervision in the extraction process  the vectors associated with the seed instance are merged into a reference search-signature vector  a loose search fingerprint of the desired output type with respect to the class

Step 5: instance ranking  ranking based on the similarity score (computed with Jensen-Shannon) between each candidate vector and class vector

Reference search-signature vectors  A series of queries that can be asked about instances of a class  Given a set of candidate phrases, the system guess which candidate phrases are more likely to belong to the target class by looking at the queries

Experimental setting - 1  Target Classes  10 classes with 5 seed instances for each class City Country Drug Food Location Movie Newspaper Person University VideoGame

Experimental setting - 2  Data  A random sample of 50 million unique fully-anonymized queries submitted to Google  Evaluation Procedure  Top 250 candidates of each class are manually assigned a correctness label 1 : correct 0 : incorrect  Precision at rank N has been calculated for several N values

Quality of Extracted Instances

Does the popularity of seed instances in query logs correlated with precision ? more queries with seed instances more accurate scoring better internal representation ?

Comparing the usefulness of query logs vs. Web documents in NE finding  M. Pasca. Acquisition of categorized named entities for Web search. CIKM 2004  Target classes are incrementally acquired from Web documents along with their respective instances by using hand crafted extraction patterns (D-patt) Class [such as|including] Instance  Manual one-to-one mapping of chosen target classes with acquired classes

Comparing the usefulness of query logs vs. Web documents in NE finding  Instances extracted from Web documents are also manually evaluated as correct and incorrect  Except City, Newspaper and Country classes, seed based extraction from queries outperformed D-patt in every other class

Conclusion  Search queries, which are thought as noisy, keyword based approximations of underspecified user information needs, proved to be useful in name entity discoveries even with a small set of seed instances  with absolute precision (or precision improvement relative to web based hand crafted system) 0.96 (29%) for 0.90 (26%) for 0.80 (15%) for

Questions ?