Presentation is loading. Please wait.

Presentation is loading. Please wait.

Sudeepa Roy University of Washington with Laura Chiticariu Vitaly Feldman Frederick R. Reiss Huaiyu Zhu IBM Research-Almaden.

Similar presentations


Presentation on theme: "Sudeepa Roy University of Washington with Laura Chiticariu Vitaly Feldman Frederick R. Reiss Huaiyu Zhu IBM Research-Almaden."— Presentation transcript:

1 Sudeepa Roy University of Washington with Laura Chiticariu Vitaly Feldman Frederick R. Reiss Huaiyu Zhu IBM Research-Almaden

2 Information Extraction (IE) 6/24/2013SIGMOD 20132 Person Extractor Extracting structured information (entity, relation) from unstructured text Michael Kelly Chelsea Cynthia Rowley September … … : true positive : false positive Mention (span) (doc3, offset = [10, 21])

3 We want to improve the quality of the extractors… i.e. output as many true positives as possible and, as few false positives as possible 6/24/2013SIGMOD 20133 this work... by Dictionary Refinement (can be used by itself or in addition to other approaches)

4 What are Dictionaries? 6/24/2013SIGMOD 20134 Person Extractor Michael Kelly Chelsea Cynthia Rowley September … … : true positive : false positive Set of dictionaries Set of extraction rules Tiziana Daniela Guido Luciano Cesare … Damp Glandorf Cobbenrode Bad Laer Aach … Time Warner Comcast Corp. Walt Disney Co. News Corp. DirecTV Group … Locations (German cities) Organizations (media) Names (Italian first names) Dictionary 1Dictionary 2Dictionary 3 Dictionaries in IE List of entries - person names, locations, organizations, etc. - also called Gazetteers/Lists Examples: Rule-Based IE systems like SystemT from IBM Almaden Our approach can be useful also for machine-learning-based IE systems

5 A Toy Example 6/24/2013SIGMOD 2013 R1: create view FirstName as Dictionary(first.dict, Document, text); R2: create view LastName as Dictionary(last.dict, Document, text); R3: create view FullName as Dictionary(full.dict, Document, text); R4: create view FirstLast as select Merge(F.match, L.match) as match from FirstName F, LastName L where FollowsTok(F.match, L.match, 0, 0); R5: create table Person(match span); insert into Person ( select * from FullName A ) union ( select * from FirstLast A ) ) union ( select * from FirstName A where Not(MatchesRegex([ ]*[A-Z].*, RightContextTok(A.match, 1 full.dict first.dict chelsea john april david …… last.dict smith lee ……… april smith john brown …… Set of extraction rules Set of dictionaries Output union of 1. FirstName, 2. FullName and 3. FirstName followed by LastName (FirstLast) as Person Declarative Rule language (AQL/Annotation Query Language in SystemT) Details not needed in the talk Set of dictionaries Set of extraction rules

6 A Toy Example - Execution 6/24/2013SIGMOD 2013 full.dict first.dict chelsea john april david …… last.dict ………This April, mark your calendars for the first derby of the season: Arsenal at Chelsea. …………………….,..April Smith and John Lee reporting live from …….. David said that…… smith lee ……… april smith john brown …… Set of dictionaries Input Document April Chelsea April Smith John Lee David Smith Lee 6 April Smith April Chelsea April John David FirstName LastName April Chelsea April Smith John Lee David Person April Smith John Lee FirstLast Views/Results FullName Extract first names (from first.dict) Extract last names (from last.dict) Extract full names (from full.dict) Find FirstLast = firstnames followed by lastnames Output union of FirstName, FullName and FirstLast as Person Two different mentions (spans) of April in Document

7 1. Better Dictionaries Better Result Quality Entries are collected from diverse sources, or generated automatically by a previous IE step therefore noisy 6/24/2013SIGMOD 20137 So, refined dictionaries are desirable Even more reasons… 2. To create specialized dictionaries based on Domains (sports articles) or Languages (German newspaper) e.g. from All First or Last Names to German Last Names 3. To add sophisticated extraction rules First find ambiguous entries, then treat them differently e.g. Chelsea, April

8 What do we mean by dictionary refinement? Output the entries that are responsible for many false positives and not for too many true positives 6/24/2013SIGMOD 20138 In practice, a human supervisor decides whether to actually delete these entries/ add a new rule to handle them if deleted get deleted do not get deleted

9 Next, Our Problem Definition Why is Provenance useful in IE? Measuring Output Quality 6/24/2013SIGMOD 20139

10 Provenance in IE 6/24/2013SIGMOD 2013 full.dict first.dict chelsea john april david …… last.dict smith lee ……… april smith john brown …… Set of dictionaries April Chelsea April Smith John Lee David Smith Lee 10 April Smith April Chelsea April John David FirstName LastName April Chelsea April Smith John Lee David Person April Smith John Lee FirstLast Views/Results FullName Connection with Relational DB and Queries Assign unique id-s to entries Interpret extraction rules as an RA query Propagate provenance annotation to the output (ref. Provenance Semiring by Green et. al. 2007) Relations w5w5 w7w7 w4w4 w6w6 w1w1 w2w2 w3w3 w 5 + w 3 w 4 w3w3 w3w3 w4w4 w4w4 w5w5 w5w5 w 3 w 4 w8w8 Many-to-many dependency Which entries should be deleted to delete April and retain April Smith?

11 Measuring Output Quality Standard Measures in IE Precision (P) = /( + ) in output = 3/5 Recall (R) = in output / actual = 3/3 F-score (F) = 2/(1/P + 1/R) = 3/4 6/24/2013SIGMOD 201311 April Chelsea April Smith John Lee David w 1 : chelsea w 2 : john w 3 : april ……… w 4 : smith w 6 : lee ……… w 5 : april smith w 7 : john brown …… 3/4 6/7 3/3 w 5 + w 3 w 4 w3w3 w3w3 (to balance deletion of false and true positives) 2/3 false w5w5 When entries are deleted, results whose provenance becomes False get deleted We use F-score: Higher F-score = higher extractor quality Max value 1 Delete w 3 Delete {w 3, w 5 } Provenance and F-score

12 6/24/2013SIGMOD 201312 Dictionary Refinement Problem Objective: Maximize F-Score Select a set S of entries to remove … that maximizes the new F-score … subject to |S| k or new recall r Select a set S of entries to remove … that maximizes the new F-score … subject to |S| k or new recall r Size Constraint (limit #deleted entries) Size Constraint (limit #deleted entries) Recall Constraint (limit #true positives deleted) Recall Constraint (limit #true positives deleted) April Chelsea April Smith John Lee David w 5 + w 3 w 4 w3w3 w3w3 Input: Provenance of each result / Label of each result Input: Provenance of each result / Label of each result w7w7 w7w7 w1w1 w1w1 w 2 w 6 We also studied the incomplete labeling case S = Possible output New F-score = 1 w1: chelsea w3: april

13 We have an optimization problem: Maximize F-score What are the main challenges? 6/24/2013SIGMOD 201313

14 6/24/2013SIGMOD 2013 full.dict first.dict chelsea john april david …… last.dict smith lee ……… april smith john brown …… Set of dictionaries April Chelsea April Smith John Lee David Smith Lee 14 April Smith April Chelsea April John David FirstName LastName April Chelsea April Smith John Lee David Person April Smith John Lee FirstLast Outputs/Results FullName Complex Input-Output Dependency w5w5 w4w4 w3w3 w 5 + w 3 w 4 w3w3 w3w3 w4w4 w4w4 w5w5 w5w5 w 3 w 4 Multiple Inputs One output One Input Multiple outputs (true/false positives) Many-to-many Which entries should we delete to retain April Smith and delete April?

15 15 --------------------------------------- create view ValidLastNameAll as select N.lastname as lastname from LastNameAll N -- do not allow partially all capitalized words where Not(MatchesRegex(/(\p{Lu}\p{M}*)+-.*([\p{Ll}\p{Lo}]\p{M}*).*/, N.lastname)) and Not(MatchesRegex(/.*([\p{Ll}\p{Lo}]\p{M}*).*- (\p{Lu}\p{M}*)+/, N.lastname)); create view LastName as select C.lastname as lastname --from Consolidate(ValidLastNameAll.lastname) C; from ValidLastNameAll C consolidate on C.lastname; -- Find dictionary matches for all first names -- Mostly US first names create view StrictFirstName1 as select D.match as firstname from Dictionary('strictFirst.dict', Doc.text) D -- where MatchesRegex(/\p{Upper}\p{Lower}[\p{Alpha}]{0,20}/, D.match); -- changed to enable unicode match where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); -- German first names create view StrictFirstName2 as select D.match as firstname from Dictionary('strictFirst_german.dict', Doc.text) D -- where MatchesRegex(/\p{Upper}\p{Lower}[\p{Alpha}]{0,20}/, D.match); --where MatchesRegex(/\p{Upper}.{1,20}/, D.match); -- changed to enable unicode match where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); -- nick names for US first names create view StrictFirstName3 as select D.match as firstname from Dictionary('strictNickName.dict', Doc.text) D -- where MatchesRegex(/\p{Upper}\p{Lower}[\p{Alpha}]{0,20}/, D.match); --where MatchesRegex(/\p{Upper}.{1,20}/, D.match); -- changed to enable unicode match where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); -- german first name from blue page create view StrictFirstName4 as select D.match as firstname from Dictionary('strictFirst_german_bluePages.dict', Doc.text) D -- where MatchesRegex(/\p{Upper}\p{Lower}[\p{Alpha}]{0,20}/, D.match); --where MatchesRegex(/\p{Upper}.{1,20}/, D.match); -- changed to enable unicode match where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); -- Italy first name from blue pages create view StrictFirstName5 as select D.match as firstname from Dictionary('names/strictFirst_italy.dict', Doc.text) D where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); -- France first name from blue pages create view StrictFirstName6 as select D.match as firstname from Dictionary('names/strictFirst_france.dict', Doc.text) D where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); -- Spain first name from blue pages create view StrictFirstName7 as select D.match as firstname from Dictionary('names/strictFirst_spain.dict', Doc.text) D where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); -- Indian first name from blue pages -- TODO: still need to clean up the remaining entries create view StrictFirstName8 as select D.match as firstname from Dictionary('names/strictFirst_india.partial.dict', Doc.text) D where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); -- Israel first name from blue pages create view StrictFirstName9 as select D.match as firstname from Dictionary('names/strictFirst_israel.dict', Doc.text) D where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); -- union all the dictionary matches for first names create view StrictFirstName as (select S.firstname as firstname from StrictFirstName1 S) union all (select S.firstname as firstname from StrictFirstName2 S) union all (select S.firstname as firstname from StrictFirstName3 S) union all (select S.firstname as firstname from StrictFirstName4 S) union all (select S.firstname as firstname from StrictFirstName5 S) union all (select S.firstname as firstname from StrictFirstName6 S) union all (select S.firstname as firstname from StrictFirstName7 S) union all (select S.firstname as firstname from StrictFirstName8 S) union all (select S.firstname as firstname from StrictFirstName9 S); -- Relaxed versions of first name create view RelaxedFirstName1 as select CombineSpans(S.firstname, CP.name) as firstname from StrictFirstName S, StrictCapsPerson CP where FollowsTok(S.firstname, CP.name, 1, 1) and MatchesRegex(/\-/, SpanBetween(S.firstname, CP.name)); create view RelaxedFirstName2 as select CombineSpans(CP.name, S.firstname) as firstname from StrictFirstName S, StrictCapsPerson CP where FollowsTok(CP.name, S.firstname, 1, 1) and MatchesRegex(/\-/, SpanBetween(CP.name, S.firstname)); -- all the first names create view FirstNameAll as (select N.firstname as firstname from StrictFirstName N) union all (select N.firstname as firstname from RelaxedFirstName1 N) union all (select N.firstname as firstname from RelaxedFirstName2 N); create view ValidFirstNameAll as select N.firstname as firstname from FirstNameAll N where Not(MatchesRegex(/(\p{Lu}\p{M}*)+-.*([\p{Ll}\p{Lo}]\p{M}*).*/, N.firstname)) and Not(MatchesRegex(/.*([\p{Ll}\p{Lo}]\p{M}*).*- (\p{Lu}\p{M}*)+/, N.firstname)); create view FirstName as select C.firstname as firstname --from Consolidate(ValidFirstNameAll.firstname) C; from ValidFirstNameAll C consolidate on C.firstname; -- Combine all dictionary matches for both last names and first names create view NameDict as select D.match as name from Dictionary('name.dict', Doc.text) D -- where MatchesRegex(/\p{Upper}\p{Lower}[\p{Alpha}]{0,20}/, D.match); --where MatchesRegex(/\p{Upper}.{1,20}/, D.match); -- changed to enable unicode match where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); create view NameDict1 as select D.match as name from Dictionary('names/name_italy.dict', Doc.text) D where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); create view NameDict2 as select D.match as name from Dictionary('names/name_france.dict', Doc.text) D where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); create view NameDict3 as select D.match as name from Dictionary('names/name_spain.dict', Doc.text) D where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); create view NameDict4 as select D.match as name from FirstName FN, InitialWord IW, CapsPerson CP where FollowsTok(FN.firstname, IW.word, 0, 0) and FollowsTok(IW.word, CP.name, 0, 0); /** * Translation for Rule 3r2 * * This relaxed version of rule '3' will find person names like Thomas B.M. David * But it only insists that the second word is in the person dictionary */ /* CAPSPERSON INITIALWORD CAPSPERSON */ create view Person3r2 as select CombineSpans(CP.name, LN.lastname) as person from LastName LN, InitialWord IW, CapsPerson CP where FollowsTok(CP.name, IW.word, 0, 0) and FollowsTok(IW.word, LN.lastname, 0, 0); /** * Translation for Rule 4 * * This rule will find person names like David Thomas */ /* CAPSPERSON CAPSPERSON */ create view Person4WithNewLine as select CombineSpans(FN.firstname, LN.lastname) as person from FirstName FN, LastName LN where FollowsTok(FN.firstname, LN.lastname, 0, 0); -- Yunyao: 05/20/2008 revised to Person4WrongCandidates due to performance reason -- NOTE: current optimizer execute Equals first thus make Person4Wrong very expensive --create view Person4Wrong as --select CombineSpans(FN.firstname, LN.lastname) as person --from FirstName FN, -- LastName LN --where FollowsTok(FN.firstname, LN.lastname, 0, 0) -- and ContainsRegex(/[\n\r]/, SpanBetween(FN.firstname, LN.lastname)) -- and Equals(GetText(FN.firstname), GetText(LN.lastname)); create view Person4WrongCandidates as select FN.firstname as firstname, LN.lastname as lastname from FirstName FN, LastName LN where FollowsTok(FN.firstname, LN.lastname, 0, 0) and ContainsRegex(/[\n\r]/, SpanBetween(FN.firstname, LN.lastname)); create view Person4 as (select P.person as person from Person4WithNewLine P) minus (select CombineSpans(P.firstname, P.lastname) as person from Person4WrongCandidates P where Equals(GetText(P.firstname), GetText(P.lastname))); /** * Translation for Rule4a * This rule will find person names like Thomas, David */ /* CAPSPERSON \, CAPSPERSON */ create view Person4a as select CombineSpans(LN.lastname, FN.firstname) as person from FirstName FN, LastName LN where FollowsTok(LN.lastname, FN.firstname, 1, 1) and ContainsRegex(/,/,SpanBetween(LN.lastname, FN.firstname)); -- relaxed version of Rule4a -- Yunyao: split the following rules into two to improve performance -- TODO: Test case for optimizer -- create view Person4ar1 as -- select CombineSpans(CP.name, FN.firstname) as person --from FirstName FN, -- CapsPerson CP --where FollowsTok(CP.name, FN.firstname, 1, 1) --and ContainsRegex(/,/,SpanBetween(CP.name, FN.firstname)) --and Not(MatchesRegex(/(.|\n|\r)*(\.|\?|!|'|\sat|\sin)( )*/, LeftContext(CP.name, 10))) --and Not(MatchesRegex(/(?i)(.+fully)/, CP.name)) --and GreaterThan(GetBegin(CP.name), 10); create view Person4ar1temp as select FN.firstname as firstname, CP.name as name from FirstName FN, CapsPerson CP where FollowsTok(CP.name, FN.firstname, 1, 1) and ContainsRegex(/,/,SpanBetween(CP.name, FN.firstname)); create view Person4ar1 as select CombineSpans(P.name, P.firstname) as person from Person4ar1temp P where Not(MatchesRegex(/(.|\n|\r)*(\.|\?|!|'|\sat|\sin)( )*/, LeftContext(P.name, 10))) --' and Not(MatchesRegex(/(?i)(.+fully)/, P.name)) and GreaterThan(GetBegin(P.name), 10); create view Person4ar2 as select CombineSpans(LN.lastname, CP.name) as person from CapsPerson CP, LastName LN where FollowsTok(LN.lastname, CP.name, 0, 1) and ContainsRegex(/,/,SpanBetween(LN.lastname, CP.name)); /** * Translation for Rule2 * * This rule will handles names of persons like B.M. Thomas David, where Thomas occurs in some person dictionary */ /* INITIALWORD CAPSPERSON CAPSPERSON */ create view Person2 as select CombineSpans(IW.word, CP.name) as person from InitialWord IW, PersonDict P, CapsPerson CP where FollowsTok(IW.word, P.name, 0, 0) and FollowsTok(P.name, CP.name, 0, 0); /** * Translation for Rule 2a * * The rule handles names of persons like B.M. Thomas David, where David occurs in some person dictionary */ /* INITIALWORD CAPSPERSON NEWLINE ? CAPSPERSON */ create view Person2a as select CombineSpans(IW.word, P.name) as person from InitialWord IW, CapsPerson CP, PersonDict P where FollowsTok(IW.word, CP.name, 0, 0) and FollowsTok(CP.name, P.name, 0, 0); /* CAPSPERSON NEWLINE ? CAPSPERSON */ create view Person4r1 as select CombineSpans(FN.firstname, CP.name) as person from FirstName FN, CapsPerson CP where FollowsTok(FN.firstname, CP.name, 0, 0); /** * Translation for Rule 4r2 * * This relaxed version of rule '4' will find person names Thomas, David * But it only insists that the SECOND word is in some person dictionary */ /* ANYWORD CAPSPERSON NEWLINE ? CAPSPERSON */ create view Person4r2 as select CombineSpans(CP.name, LN.lastname) as person from CapsPerson CP, LastName LN where FollowsTok(CP.name, LN.lastname, 0, 0); /** * Translation for Rule 5 * * This rule will find other single token person first names */ /* INITIALWORD ? CAPSPERSON */ create view Person5 as select CombineSpans(IW.word, FN.firstname) as person from InitialWord IW, FirstName FN where FollowsTok(IW.word, FN.firstname, 0, 0); /** * Translation for Rule 6 * * This rule will find other single token person last names */ /* INITIALWORD ? CAPSPERSON */ create view Person6 as select CombineSpans(IW.word, LN.lastname) as person from InitialWord IW, LastName LN where FollowsTok(IW.word, LN.lastname, 0, 0); -- ================================================== ======== -- End of rules -- -- Create final list of names based on all the matches extracted -- -- ================================================== ======== /** * Union all matches found by strong rules, except the ones directly come * from dictionary matches */ create view PersonStrongWithNewLine as (select P.person as person from Person1 P) --union all -- (select P.person as person from Person1a_more P) union all (select P.person as person from Person3 P) union all (select P.person as person from Person4 P) union all (select P.person as person from Person3P1 P); create view PersonStrongSingleTokenOnly as (select P.person as person from Person5 P) union all (select P.person as person from Person6 P) union all (select P.firstname as person from FirstName P) union all (select P.lastname as person from LastName P) union all (select P.person as person from Person1a P); -- Yunyao: added 05/09/2008 to expand person names with suffix create view PersonStrongSingleTokenOnlyExpanded1 as select CombineSpans(P.person,S.suffix) as person from PersonStrongSingleTokenOnly P, PersonSuffix S where FollowsTok(P.person, S.suffix, 0, 0); -- Yunyao: added 04/14/2009 to expand single token person name with a single initial -- extend single token person with a single initial create view PersonStrongSingleTokenOnlyExpanded2 as select CombineSpans(R.person, RightContext(R.person,2)) as person from PersonStrongSingleTokenOnly R where MatchesRegex(/ +[\p{Upper}]\b\s*/, RightContext(R.person,3)); create view PersonStrongSingleToken as (select P.person as person from PersonStrongSingleTokenOnly P) union all (select P.person as person from PersonStrongSingleTokenOnlyExpanded1 P) union all (select P.person as person from PersonStrongSingleTokenOnlyExpanded2 P); /** * Union all matches found by weak rules */ create view PersonWeak1WithNewLine as (select P.person as person from Person3r1 P) union all (select P.person as person from Person3r2 P) union all (select P.person as person from Person4r1 P) union all (select P.person as person from Person4r2 P) union all (select P.person as person from Person2 P) union all (select P.person as person from Person2a P) union all (select P.person as person from Person3P2 P) union all (select P.person as person from Person3P3 P); -- weak rules that identify (LastName, FirstName) create view PersonWeak2WithNewLine as (select P.person as person from Person4a P) union all (select P.person as person from Person4ar1 P) union all (select P.person as person from Person4ar2 P); --include 'core/GenericNE/Person-FilterNewLineSingle.aql'; --include 'core/GenericNE/Person-Filter.aql'; create view PersonBase as (select P.person as person from PersonStrongWithNewLine P) union all (select P.person as person from PersonWeak1WithNewLine P) union all (select P.person as person from PersonWeak2WithNewLine P); output view PersonBase; from Dictionary('names/name_israel.dict', Doc.text) D where MatchesRegex(/\p{Lu}\p{M}*.{1,20}/, D.match); create view NamesAll as (select P.name as name from NameDict P) union all (select P.name as name from NameDict1 P) union all (select P.name as name from NameDict2 P) union all (select P.name as name from NameDict3 P) union all (select P.name as name from NameDict4 P) union all (select P.firstname as name from FirstName P) union all create view PersonDict as select C.name as name --from Consolidate(NamesAll.name) C; from NamesAll C consolidate on C.name; --========================================================== -- Actual Rules --========================================================== -- For 3-part Person names create view Person3P1 as select CombineSpans(F.firstname, L.lastname) as person from StrictFirstName F, StrictCapsPersonR S, StrictLastName L where FollowsTok(F.firstname, S.name, 0, 0) --and FollowsTok(S.name, L.lastname, 0, 0) and FollowsTok(F.firstname, L.lastname, 1, 1) and Not(Equals(GetText(F.firstname), GetText(L.lastname))) and Not(Equals(GetText(F.firstname), GetText(S.name))) and Not(Equals(GetText(S.name), GetText(L.lastname))) and Not(ContainsRegex(/[\n\r\t]/, SpanBetween(F.firstname, L.lastname))); create view Person3P2 as select CombineSpans(P.name, L.lastname) as person from PersonDict P, StrictCapsPersonR S, StrictLastName L where FollowsTok(P.name, S.name, 0, 0) --and FollowsTok(S.name, L.lastname, 0, 0) and FollowsTok(P.name, L.lastname, 1, 1) and Not(Equals(GetText(P.name), GetText(L.lastname))) and Not(Equals(GetText(P.name), GetText(S.name))) and Not(Equals(GetText(S.name), GetText(L.lastname))) and Not(ContainsRegex(/[\n\r\t]/, SpanBetween(P.name, L.lastname))); create view Person3P3 as select CombineSpans(F.firstname, P.name) as person from PersonDict P, StrictCapsPersonR S, StrictFirstName F where FollowsTok(F.firstname, S.name, 0, 0) --and FollowsTok(S.name, P.name, 0, 0) and FollowsTok(F.firstname, P.name, 1, 1) and Not(Equals(GetText(P.name), GetText(F.firstname))) and Not(Equals(GetText(P.name), GetText(S.name))) and Not(Equals(GetText(S.name), GetText(F.firstname))) and Not(ContainsRegex(/[\n\r\t]/, SpanBetween(F.firstname, P.name))); /** * Translation for Rule 1 * Handles names of persons like Mr. Vladimir E. Putin */ /* CANYWORD CAPSPERSON INITIALWORD CAPSPERSON */ create view Person1 as select CombineSpans(CP1.name, CP2.name) as person from Initial I, CapsPerson CP1, InitialWord IW, CapsPerson CP2 where FollowsTok(I.initial, CP1.name, 0, 0) and FollowsTok(CP1.name, IW.word, 0, 0) and FollowsTok(IW.word, CP2.name, 0, 0); --and Not(ContainsRegex(/[\n\r]/, SpanBetween(I.initial, CP2.name))); /** * Translation for Rule 1a * Handles names of persons like Mr. Vladimir Putin */ /* CANYWORD CAPSPERSON {1,3} */ -- Split into two rules so that single token annotations are serperated from others -- Single token annotations create view Person1a1 as select CP1.name as person from Initial I, CapsPerson CP1 where FollowsTok(I.initial, CP1.name, 0, 0) --- start changing this block --- disallow allow newline and Not(ContainsRegex(/[\n\t]/,SpanBetween(I.initial,CP1.name))) --- end changing this block ; -- Yunyao: added 05/09/2008 to match patterns such as "Mr. B. B. Buy" /* create view Person1a2 as select CombineSpans(name.block, CP1.name) as person from Initial I, BlockTok(0, 1, 2, InitialWord.word) name, CapsPerson CP1 where FollowsTok(I.initial, name.block, 0, 0) and FollowsTok(name.block, CP1.name, 0, 0) and Not(ContainsRegex(/[\n\t]/,CombineSpans(I.initial, CP1.name))); */ create view Person1a as -- ( select P.person as person from Person1a1 P -- ) -- union all -- (select P.person as person from Person1a2 P) ; /* create view Person1a_more as select name.block as person from Initial I, BlockTok(0, 2, 3, CapsPerson.name) name where FollowsTok(I.initial, name.block, 0, 0) and Not(ContainsRegex(/[\n\t]/,name.block)) --- start changing this block -- disallow newline and Not(ContainsRegex(/[\n\t]/,SpanBetween(I.initial,name.block))) --- end changing this block ; */ /** * Translation for Rule 3 * Find person names like Thomas B.M. David */ /* CAPSPERSON INITIALWORD CAPSPERSON */ create view Person3 as select CombineSpans(P1.name, P2.name) as person from PersonDict P1, --InitialWord IW, WeakInitialWord IW, PersonDict P2 where FollowsTok(P1.name, IW.word, 0, 0) and FollowsTok(IW.word, P2.name, 0, 0) and Not(Equals(GetText(P1.name), GetText(P2.name))); /** * Translation for Rule 3r1 * * This relaxed version of rule '3' will find person names like Thomas B.M. David * But it only insists that the first word is in the person dictionary */ /* CAPSPERSON INITIALWORD CAPSPERSON */ create view Person3r1 as create view Initial as --'Junior' (Yunyao: comments out to avoid mismatches such as Junior National [team player], -- If we can have large negative dictionary to eliminate such mismatches, -- then this may be recovered --'Name:' ((Yunyao: comments out to avoid mismatches such as 'Name: Last Name') -- for German names -- TODO: need further test,'herr', 'Fraeulein', 'Doktor', 'Herr Doktor', 'Frau Doktor', 'Herr Professor', 'Frau professor', 'Baron', 'graf' ); -- Find dictionary matches for all title initials select D.match as initial --'Name:' ((Yunyao: comments out to avoid mismatches such as 'Name: Last Name') -- for German names -- TODO: need further test,'herr', 'Fraeulein', 'Doktor', 'Herr Doktor', 'Frau Doktor', 'Herr Professor', 'Frau professor', 'Baron', 'graf' ); -- Find dictionary matches for all title initials from Dictionary('InitialDict', Doc.text) D; -- Yunyao: added 05/09/2008 to capture person name suffix create dictionary PersonSuffixDict as ( ',jr.', ',jr', 'III', 'IV', 'V', 'VI' ); create view PersonSuffix as select D.match as suffix from Dictionary('PersonSuffixDict', Doc.text) D; -- Find capitalized words that look like person names and not in the non-name dictionary create view CapsPersonCandidate as select R.match as name --from Regex(/\b\p{Upper}\p{Lower}[\p{Alpha}]{1,20}\b/, Doc.text) R --from Regex(/\b\p{Upper}\p{Lower}[\p{Alpha}]{0,10}(['-][\p{Upper}])?[\p{Alpha}]{1,10}\b/, Doc.text) R -- change to enable unicode match --from Regex(/\b\p{Lu}\p{M}*[\p{Ll}\p{Lo}]\p{M}*[\p{L}\p{M}*]{0,10}(['-][\p{Lu}\p{M}*])?[\p{L}\p{M}*]{1,10}\b/, Doc.text) R --from Regex(/\b\p{Lu}\p{M}*[\p{Ll}\p{Lo}]\p{M}*[\p{L}\p{M}*]{0,10}(['-][\p{Lu}\p{M}*])?(\p{L}\p{M}*){1,10}\b/, Doc.text) R -- Allow fully capitalized words --from Regex(/\b\p{Lu}\p{M}*(\p{L}\p{M}*){0,10}(['-][\p{Lu}\p{M}*])?(\p{L}\p{M}*){1,10}\b/, Doc.text) R from RegexTok(/\p{Lu}\p{M}*(\p{L}\p{M}*){0,10}(['-][\p{Lu}\p{M}*])?(\p{L}\p{M}*){1,10}/, 4, Doc.text) R --' where Not(ContainsDicts( 'FilterPersonDict', 'filterPerson_position.dict', 'filterPerson_german.dict', 'InitialDict', 'StrongPhoneVariantDictionary', 'stateList.dict', 'organization_suffix.dict', 'industryType_suffix.dict', 'streetSuffix_forPerson.dict', 'wkday.dict', 'nationality.dict', 'stateListAbbrev.dict', 'stateAbbrv.ChicagoAPStyle.dict', R.match)); create view CapsPerson as select C.name as name from CapsPersonCandidate C where Not(MatchesRegex(/(\p{Lu}\p{M}*)+-.*([\p{Ll}\p{Lo}]\p{M}*).*/, C.name)) and Not(MatchesRegex(/.*([\p{Ll}\p{Lo}]\p{M}*).*-(\p{Lu}\p{M}*)+/, C.name)); -- Find strict capitalized words with two letter or more (relaxed version of StrictCapsPerson) --============================================================ --TODO: need to think through how to deal with hypened name -- one way to do so is to run Regex(pattern, CP.name) and enforce CP.name does not contain ' -- need more testing before confirming the change create view CapsPersonNoP as select CP.name as name from CapsPerson CP where Not(ContainsRegex(/'/, CP.name)); --' create view StrictCapsPersonR as select R.match as name --from Regex(/\b\p{Lu}\p{M}*(\p{L}\p{M}*){1,20}\b/, CapsPersonNoP.name) R; from RegexTok(/\p{Lu}\p{M}*(\p{L}\p{M}*){1,20}/, 1, CapsPersonNoP.name) R; --============================================================ -- Find strict capitalized words --create view StrictCapsPerson as create view StrictCapsPerson as select R.name as name from StrictCapsPersonR R where MatchesRegex(/\b\p{Lu}\p{M}*[\p{Ll}\p{Lo}]\p{M}*(\p{L}\p{M}*){1,20}\b/, R.name); -- Find dictionary matches for all last names create view StrictLastName1 as select D.match as lastname from Dictionary('strictLast.dict', Doc.text) D --where MatchesRegex(/\p{Upper}\p{Lower}[\p{Alpha}]{0,20}/, D.match); -- changed to enable unicode match where MatchesRegex(/((\p{L}\p{M}*)+\s+)?\p{Lu}\p{M}*.{1,20}/, D.match); create view StrictLastName2 as select D.match as lastname from Dictionary('strictLast_german.dict', Doc.text) D --where MatchesRegex(/\p{Upper}\p{Lower}[\p{Alpha}]{0,20}/, D.match); --where MatchesRegex(/\p{Upper}.{1,20}/, D.match); -- changed to enable unicode match where MatchesRegex(/((\p{L}\p{M}*)+\s+)?\p{Lu}\p{M}*.{1,20}/, D.match); create view StrictLastName3 as select D.match as lastname from Dictionary('strictLast_german_bluePages.dict', Doc.text) D --where MatchesRegex(/\p{Upper}\p{Lower}[\p{Alpha}]{0,20}/, D.match); --where MatchesRegex(/\p{Upper}.{1,20}/, D.match); -- changed to enable unicode match where MatchesRegex(/((\p{L}\p{M}*)+\s+)?\p{Lu}\p{M}*.{1,20}/, D.match); create view StrictLastName4 as select D.match as lastname from Dictionary('uniqMostCommonSurname.dict', Doc.text) D --where MatchesRegex(/\p{Upper}\p{Lower}[\p{Alpha}]{0,20}/, D.match); --where MatchesRegex(/\p{Upper}.{1,20}/, D.match); -- changed to enable unicode match where MatchesRegex(/((\p{L}\p{M}*)+\s+)?\p{Lu}\p{M}*.{1,20}/, D.match); create view StrictLastName5 as select D.match as lastname from Dictionary('names/strictLast_italy.dict', Doc.text) D where MatchesRegex(/((\p{L}\p{M}*)+\s+)?\p{Lu}\p{M}*.{1,20}/, D.match); create view StrictLastName6 as select D.match as lastname from Dictionary('names/strictLast_france.dict', Doc.text) D where MatchesRegex(/((\p{L}\p{M}*)+\s+)?\p{Lu}\p{M}*.{1,20}/, D.match); create view StrictLastName7 as select D.match as lastname from Dictionary('names/strictLast_spain.dict', Doc.text) D where MatchesRegex(/((\p{L}\p{M}*)+\s+)?\p{Lu}\p{M}*.{1,20}/, D.match); create view StrictLastName8 as select D.match as lastname from Dictionary('names/strictLast_india.partial.dict', Doc.text) D where MatchesRegex(/((\p{L}\p{M}*)+\s+)?\p{Lu}\p{M}*.{1,20}/, D.match); create view StrictLastName9 as select D.match as lastname from Dictionary('names/strictLast_israel.dict', Doc.text) D where MatchesRegex(/((\p{L}\p{M}*)+\s+)?\p{Lu}\p{M}*.{1,20}/, D.match); create view StrictLastName as (select S.lastname as lastname from StrictLastName1 S) union all (select S.lastname as lastname from StrictLastName2 S) union all (select S.lastname as lastname from StrictLastName3 S) union all (select S.lastname as lastname from StrictLastName4 S) union all (select S.lastname as lastname from StrictLastName5 S) union all (select S.lastname as lastname from StrictLastName6 S) union all (select S.lastname as lastname from StrictLastName7 S) union all (select S.lastname as lastname from StrictLastName8 S) union all (select S.lastname as lastname from StrictLastName9 S); -- Relaxed version of last name create view RelaxedLastName1 as select CombineSpans(SL.lastname, CP.name) as lastname from StrictLastName SL, StrictCapsPerson CP where FollowsTok(SL.lastname, CP.name, 1, 1) and MatchesRegex(/\-/, SpanBetween(SL.lastname, CP.name)); create view RelaxedLastName2 as select CombineSpans(CP.name, SL.lastname) as lastname from StrictLastName SL, StrictCapsPerson CP where FollowsTok(CP.name, SL.lastname, 1, 1) and MatchesRegex(/\-/, SpanBetween(CP.name, SL.lastname)); -- all the last names create view LastNameAll as (select N.lastname as lastname from StrictLastName N) union all (select N.lastname as lastname from RelaxedLastName1 N) union all (select N.lastname as lastname from RelaxedLastName2 N); create view ValidLastNameAll as select N.lastname as lastname --------------------------------------- -- Document Preprocessing --------------------------------------- create view Doc as select D.text as text from DocScan D; ---------------------------------------- -- Basic Named Entity Annotators ---------------------------------------- -- Find initial words create view InitialWord1 as select R.match as word --from Regex(/\b([\p{Upper}]\.\s*){1,5}\b/, Doc.text) R from RegexTok(/([\p{Upper}]\.\s*){1,5}/, 10, Doc.text) R -- added on 04/18/2008 where Not(MatchesRegex(/M\.D\./, R.match)); -- Yunyao: added on 11/21/2008 to capture names with prefix (we use it as initial -- to avoid adding too many commplex rules) create view InitialWord2 as select D.match as word from Dictionary('specialNamePrefix.dict', Doc.text) D; create view InitialWord as (select I.word as word from InitialWord1 I) union all (select I.word as word from InitialWord2 I); -- Find weak initial words create view WeakInitialWord as select R.match as word --from Regex(/\b([\p{Upper}]\.?\s*){1,5}\b/, Doc.text) R; from RegexTok(/([\p{Upper}]\.?\s*){1,5}/, 10, Doc.text) R -- added on 05/12/2008 -- Do not allow weak initial word to be a word longer than three characters where Not(ContainsRegex(/[\p{Upper}]{3}/, R.match)) -- added on 04/14/2009 -- Do not allow weak initial words to match the timezon and Not(ContainsDict('timeZone.dict', R.match)); ----------------------------------------------- -- Strong Phone Numbers ----------------------------------------------- create dictionary StrongPhoneVariantDictionary as ( 'phone', 'cell', 'contact', 'direct', 'office', -- Yunyao: Added new strong clues for phone numbers 'tel', 'dial', 'Telefon', 'mobile', 'Ph', 'Phone Number', 'Direct Line', 'Telephone No', 'TTY', 'Toll Free', 'Toll-free', -- German 'Fon', 'Telefon Geschaeftsstelle', 'Telefon Geschäftsstelle', 'Telefon Zweigstelle', 'Telefon Hauptsitz', 'Telefon (Geschaeftsstelle)', 'Telefon (Geschäftsstelle)', 'Telefon (Zweigstelle)', 'Telefon (Hauptsitz)', 'Telefonnummer', 'Telefon Geschaeftssitz', 'Telefon Geschäftssitz', 'Telefon (Geschaeftssitz)', 'Telefon (Geschäftssitz)', 'Telefon Persönlich', 'Telefon persoenlich', 'Telefon (Persönlich)', 'Telefon (persoenlich)', 'Handy', 'Handy-Nummer', 'Telefon arbeit', 'Telefon (arbeit)' ); --include 'core/GenericNE/Person.aql'; create dictionary FilterPersonDict as ( 'Travel', 'Fellow', 'Sir', 'IBMer', 'Researcher', 'All','Tell', 'Friends', 'Friend', 'Colleague', 'Colleagues', 'Managers','If', 'Customer', 'Users', 'User', 'Valued', 'Executive', 'Chairs', 'New', 'Owner', 'Conference', 'Please', 'Outlook', 'Lotus', 'Notes', 'This', 'That', 'There', 'Here', 'Subscribers', 'What', 'When', 'Where', 'Which', 'With', 'While', 'Thanks', 'Thanksgiving','Senator', 'Platinum', 'Perspective', 'Manager', 'Ambassador', 'Professor', 'Dear', 'Contact', 'Cheers', 'Athelet', 'And', 'Act', 'But', 'Hello', 'Call', 'From', 'Center', 'The', 'Take', 'Junior', 'Both', 'Communities', 'Greetings', 'Hope', 'Restaurants', 'Properties', 'Let', 'Corp', 'Memorial', 'You', 'Your', 'Our', 'My', 'His','Her', 'Their','Popcorn', 'Name', 'July', 'June','Join', 'Business', 'Administrative', 'South', 'Members', 'Address', 'Please', 'List', 'Public', 'Inc', 'Parkway', 'Brother', 'Buy', 'Then', 'Services', 'Statements', 'President', 'Governor', 'Commissioner', 'Commitment', 'Commits', 'Hey', 'Director', 'End', 'Exit', 'Experiences', 'Finance', 'Elementary', 'Wednesday', 'Nov', 'Infrastructure', 'Inside', 'Convention', 'Judge', 'Lady', 'Friday', 'Project', 'Projected', 'Recalls', 'Regards', 'Recently', 'Administration', 'Independence', 'Denied', 'Unfortunately', 'Under', 'Uncle', 'Utility', 'Unlike', 'Was', 'Were', 'Secretary', 'Speaker', 'Chairman', 'Consider', 'Consultant', 'County', 'Court', 'Defensive', 'Northwestern', 'Place', 'Hi', 'Futures', 'Athlete', 'Invitational', 'System', 'International', 'Main', 'Online', 'Ideally' -- more entries,'If','Our', 'About', 'Analyst', 'On', 'Of', 'By', 'HR', 'Mkt', 'Pre', 'Post', 'Condominium', 'Ice', 'Surname', 'Lastname', 'firstname', 'Name', 'familyname', -- Italian greeting 'Ciao', -- Spanish greeting 'Hola', -- French greeting 'Bonjour', -- new entries 'Pro','Bono','Enterprises','Group','Said','Says','Assistant',' Vice','Warden','Contribution', 'Research', 'Development', 'Product', 'Sales', 'Support', 'Manager', 'Telephone', 'Phone', 'Contact', 'Information', 'Electronics','Managed','West','East','North','South', 'Teaches','Ministry', 'Church', 'Association', 'Laboratories', 'Living', 'Community', 'Visiting', 'Officer', 'After', 'Pls', 'FYI', 'Only', 'Additionally', 'Adding', 'Acquire', 'Addition', 'America', -- short phrases that are likely to be at the start of a sentence 'Yes', 'No', 'Ja', 'Nein','Kein', 'Keine', 'Gegenstimme', -- TODO: to be double checked 'Another', 'Anyway','Associate', 'At', 'Athletes', 'It', 'Enron', 'EnronXGate', 'Have', 'However', 'Company', 'Companies', 'IBM','Annual', -- common verbs appear with person names in financial reports -- ideally we want to have a general comprehensive verb list to use as a filter dictionary 'Joins', 'Downgrades', 'Upgrades', 'Reports', 'Sees', 'Warns', 'Announces', 'Reviews' -- Laura 06/02/2009: new filter dict for title for SEC domain in filterPerson_title.dict ); create dictionary GreetingsDict as ( 'Hey', 'Hi', 'Hello', 'Dear', -- German greetings 'Liebe', 'Lieber', 'Herr', 'Frau', 'Hallo', -- Italian 'Ciao', -- Spanish 'Hola', -- French 'Bonjour' ); create dictionary InitialDict as ( 'rev.', 'col.', 'reverend', 'prof.', 'professor.', 'lady', 'miss.', 'mrs.', 'mrs', 'mr.', 'pt.', 'ms.', 'messrs.', 'dr.', 'master.', 'marquis', 'monsieur', 'ds', 'di' --'Dear' (Yunyao: comments out to avoid mismatches such as Dear Member), --'Junior' (Yunyao: comments out to avoid mismatches such as Junior National [team player], -- If we can have large negative dictionary to eliminate such mismatches, -- then this may be recovered Extracts Person, Organization, Locations, … > 400 extraction rules > 200 dictionaries (in addition to features like regular expressions, parts of speech, capitalization, punctuation…) Extracts Person, Organization, Locations, … > 400 extraction rules > 200 dictionaries (in addition to features like regular expressions, parts of speech, capitalization, punctuation…) Actual Extractors in IBM-Almaden SystemT

16 6/24/2013SIGMOD 201316 Complex Objective Function 2 * G -s G o + G -s + B -s New F-score after deleting S = G o = original #true positives G -s = remaining #true positives after deleting S B -s = remaining #false positives after deleting S Both numerator and denominator depend on S (even if we try to rewrite the expression)

17 Rest of the Talk… Optimization Problem A Simple Class of Extractors (Simple-Rules) General Case (Complex Rules) Handling Incomplete Labeling Experimental Evaluation Related Work and Conclusions 6/24/2013SIGMOD 201317

18 Simple Rules Find matches of dictionary entries from text 6/24/2013SIGMOD 201318 name.dict …This April, mark your calendars for the first derby of the season: Arsenal at Chelsea….April Smith and John Lee reporting live from ….David said that…… April Chelsea April John David Person w3w3 w3w3 w8w8 w8w8 w1w1 w1w1 w 1 : chelsea w 2 : john w 3 : april w 8 : david ……… w3w3 w3w3 w8w8 w8w8 1.Provenance has a simple form 2.One-to-many (not many-to-many) Has independent applications Optimization is not that simple!

19 Results: Simple Rules 6/24/2013SIGMOD 201319 Greedy is not optimal NP-hard (reduction from the subset-sum problem) Near optimal Algorithm (simple, provably close to optimal) Optimal Algorithm Some details next

20 Sketch of Optimal Algorithm for Simple Rules, Size Constraint |S| k 6/24/2013SIGMOD 201320 1. Guess the optimal F-score θ 2. Verify if there exists a subset S, |S| k, giving this F-score θ 3. Repeat by binary-search in [0, 1] until the optimal θ is found 2 * G -s G o + G -s + B -s F -s = G -s (2 - θ) - θB -s - θG o 0 θ G -s = G o - w S G w B -s = B o - w S B w w S f(G w,B w ) Const, where |S| k Does not work for general case (many-to-many) Binary search on real numbers in [0, 1] (still poly-time) Top-k problem, poly-time! We want to handle the F-score ratio

21 What about General Case (Complex Rules)? 6/24/2013SIGMOD 201321 Arbitrary extraction rules Arbitrary provenance Many-to-many dependency April Smith John Lee David w 5 + w 3 w 4 w3w3 w3w3 w 2 w 6

22 Results: Complex Rules Simple Rules Provenance: w Optimal Algorithm NP-hard Near optimal Algorithm 6/24/2013SIGMOD 201322 Efficient Heuristics Sketch: Find an initial solution Improve solution by hill-climbing Efficient Heuristics Sketch: Find an initial solution Improve solution by hill-climbing NP-hard even for two dictionaries (reduction from the k-densest subgraph problem) Requires evaluation of Boolean provenance

23 What if not all the results are labeled? 6/24/2013SIGMOD 201323 April Chelsea April Smith John Lee David w 5 + w 3 w 4 w3w3 w3w3 w7w7 w7w7 w1w1 w1w1 w 2 w 6 Labeling is expensive Ignoring unlabeled results may lead to over-fitting We estimate missing labels So far we assumed all results are labeled as true positive / false positive

24 6/24/2013SIGMOD 201324 Simple RulesComplex Rules Estimating Missing Labels April Smith John Lee David w 5 + w 3 w 4 w3w3 w3w3 w 2 w 6 April Chelsea David April Chelsea April David w3w3 w3w3 w7w7 w7w7 w1w1 w1w1 w3w3 w3w3 w7w7 w7w7 w3w3 w3w3 w1w1 w1w1 Possible approach: Label of an entry = Empirical fraction of true positives w 3 april: 0.33 w 1 chelsea: 0.50 w 7 david: 1.00 Empirical estimation does not work! Arbitrary monotone Boolean expressions Very few or no labels available! We assume a statistical model and estimate labels using Expectation-Maximization (EM) algorithm 0 1 w7w7 w7w7 w3w3 w3w3 w1w1 w1w1 Reduces to Chelsea April David 0.33

25 Experimental Evaluation 6/24/2013SIGMOD 201325

26 Evaluation Experimental Setting IE system: SystemT (IBM Almaden) Rule language: AQL (Annotation Query Language) Dataset: CoNLL, ACE Person/Organization extractors (1 rule to more than 400 rules) Experimental Setting IE system: SystemT (IBM Almaden) Rule language: AQL (Annotation Query Language) Dataset: CoNLL, ACE Person/Organization extractors (1 rule to more than 400 rules) 6/24/2013SIGMOD 201326 Purpose of the experiments Optimization: Compare the final F-scores and Running time of algorithms For all cases of Simple/Complex rules and Size/Recall constraints Label Estimation: Compare our label estimation approach with optimization only on the labeled data Qualitative Evaluation: whether the returned entries are meaningful Purpose of the experiments Optimization: Compare the final F-scores and Running time of algorithms For all cases of Simple/Complex rules and Size/Recall constraints Label Estimation: Compare our label estimation approach with optimization only on the labeled data Qualitative Evaluation: whether the returned entries are meaningful

27 Sample Experiment – Optimization 6/24/2013SIGMOD 201327 Simple rules, size constraint Optimal algorithm performs better than other approaches (and is also efficient-in the paper)

28 Sample Experiment – Label Estimation 6/24/2013SIGMOD 201328 Simple rules, size constraint -F-score improves when the number of labeled results increase -Label estimation gives better result Fraction of Labeled Results 100% 50%25% 12.5%6.25%3.125%

29 Sample Experiment – Qualitative Evaluation 6/24/2013SIGMOD 201329 More experiments can be found in the paper 10% Labeled (test) dataset Actual dataset china(0, 12)(0, 100) kong(0, 11)(0, 70) june(0, 9)(0, 97) hong(1, 10)(2, 71) september(0, 8)(0, 101) king(0, 5)(6, 20) louis(1, 6)(4, 33) Top-7 Output by our optimal algorithm (Simple rules, size constraint) On person-name dictionary with 13k entries Incomplete 10% Labeled Data Bad/Ambiguous entries as person names!

30 Related Work Entity Extraction Agichtein et. al.00, Elmeleegy et. al. 09, Riloff et. al. 93, Yates et. al. 07, etc. Named Entity Disambiguation Hoffart et. al. 11, etc. Rule Refinement in IE Liu et. al. 10, Chai et. al. 09, Shen et. al. 08 etc. Causality Meliou et. al. 11 etc. Deletion Propagation Buneman et. al. 02, Kimelfeld et. al. 11 etc 6/24/2013SIGMOD 201330 Our work.. Orthogonal to these approaches and can be used in addition to them Alternative Objective Special Case

31 Conclusions Summary of our contributions: Dictionary refinement as an optimization problem Theoretical analysis and Experimental Evaluation Handling incomplete labels Future Work Better model for label estimation that considers correlation Adaptively labeling results 6/24/2013SIGMOD 201331

32 Thank you 6/24/2013SIGMOD 201332 Questions?


Download ppt "Sudeepa Roy University of Washington with Laura Chiticariu Vitaly Feldman Frederick R. Reiss Huaiyu Zhu IBM Research-Almaden."

Similar presentations


Ads by Google