1. Introduction to “Event Extraction” Jan 18, 2007

2. What is “Information Extraction” Filling slots in a database from sub-segments of text. As a task: October 14, 2002, 4:00 a.m. PT For years, Microsoft Corporation CEO Bill Gates railed against the economic philosophy of open-source software with Orwellian fervor, denouncing its communal licensing as a "cancer" that stifled technological innovation. Today, Microsoft claims to "love" the open- source concept, by which software code is made public to encourage improvement and development by outside programmers. Gates himself says Microsoft will gladly disclose its crown jewels--the coveted code behind the Windows operating system--to select customers. "We can be open source. We love the concept of shared source," said Bill Veghte, a Microsoft VP. "That's a super-important shift for us in terms of code access.“ Richard Stallman, founder of the Free Software Foundation, countered saying… NAME TITLE ORGANIZATION Bill Gates CEO Microsoft Bill Veghte VP Microsoft Richard Stallman founder Free Soft.. IE

3. What is “Information Extraction” Information Extraction = segmentation + classification + clustering + association As a family of techniques: October 14, 2002, 4:00 a.m. PT For years, Microsoft Corporation CEO Bill Gates railed against the economic philosophy of open-source software with Orwellian fervor, denouncing its communal licensing as a "cancer" that stifled technological innovation. Today, Microsoft claims to "love" the open- source concept, by which software code is made public to encourage improvement and development by outside programmers. Gates himself says Microsoft will gladly disclose its crown jewels--the coveted code behind the Windows operating system--to select customers. "We can be open source. We love the concept of shared source," said Bill Veghte, a Microsoft VP. "That's a super-important shift for us in terms of code access.“ Richard Stallman, founder of the Free Software Foundation, countered saying… Microsoft Corporation CEO Bill Gates Microsoft Gates Microsoft Bill Veghte Microsoft VP Richard Stallman founder Free Software Foundation Also known as “named entity extraction”

4. What is “Information Extraction” Information Extraction = segmentation + classification + association + clustering As a family of techniques: October 14, 2002, 4:00 a.m. PT For years, Microsoft Corporation CEO Bill Gates railed against the economic philosophy of open-source software with Orwellian fervor, denouncing its communal licensing as a "cancer" that stifled technological innovation. Today, Microsoft claims to "love" the open- source concept, by which software code is made public to encourage improvement and development by outside programmers. Gates himself says Microsoft will gladly disclose its crown jewels--the coveted code behind the Windows operating system--to select customers. "We can be open source. We love the concept of shared source," said Bill Veghte, a Microsoft VP. "That's a super-important shift for us in terms of code access.“ Richard Stallman, founder of the Free Software Foundation, countered saying… Microsoft Corporation CEO Bill Gates Microsoft Gates Microsoft Bill Veghte Microsoft VP Richard Stallman founder Free Software Foundation also aka “named entity extraction”

5. What is “Information Extraction” Information Extraction = segmentation + classification + association + clustering As a family of techniques: October 14, 2002, 4:00 a.m. PT For years, Microsoft Corporation CEO Bill Gates railed against the economic philosophy of open-source software with Orwellian fervor, denouncing its communal licensing as a "cancer" that stifled technological innovation. Today, Microsoft claims to "love" the open- source concept, by which software code is made public to encourage improvement and development by outside programmers. Gates himself says Microsoft will gladly disclose its crown jewels--the coveted code behind the Windows operating system--to select customers. "We can be open source. We love the concept of shared source," said Bill Veghte, a Microsoft VP. "That's a super-important shift for us in terms of code access.“ Richard Stallman, founder of the Free Software Foundation, countered saying… Microsoft Corporation CEO Bill Gates Microsoft Gates Microsoft Bill Veghte Microsoft VP Richard Stallman founder Free Software Foundation

6. What is “Information Extraction” Information Extraction = segmentation + classification + association + clustering As a family of techniques: October 14, 2002, 4:00 a.m. PT For years, Microsoft Corporation CEO Bill Gates railed against the economic philosophy of open-source software with Orwellian fervor, denouncing its communal licensing as a "cancer" that stifled technological innovation. Today, Microsoft claims to "love" the open- source concept, by which software code is made public to encourage improvement and development by outside programmers. Gates himself says Microsoft will gladly disclose its crown jewels--the coveted code behind the Windows operating system--to select customers. "We can be open source. We love the concept of shared source," said Bill Veghte, a Microsoft VP. "That's a super-important shift for us in terms of code access.“ Richard Stallman, founder of the Free Software Foundation, countered saying… Microsoft Corporation CEO Bill Gates Microsoft Gates Microsoft Bill Veghte Microsoft VP Richard Stallman founder Free Software Foundation NAME TITLE ORGANIZATION Bill Gates CEOMicrosoft Bill Veghte VP Microsoft RichardStallman founder Free Soft.. * * * *

7. Some things to think about We’ve seen sliding windows, non-sequential token tagging, and sequential token tagging. –Which of these are likely to work best, and when? –Are there other ways to formulate NER as a learning task? –Is there a benefit from using more complex graphical models? What potentially useful information does a linear- chain CRF not capture? –Can you combine sliding windows with a sequential model? Next lecture will survey IE of sets of related entities (e.g., person and his/her affiliation). –How can you formalize that as a learning task? –Some case studies…

8. ACE: Automatic Content Extraction A case study, or: yet another NIST bake-off

9. About ACE http://www.nist.gov/speech/tests/ace/ and http://projects.ldc.upenn.edu/ace/ The five year mission: “develop technology to extract and characterize meaning in human language”…in newswire text, speech, and images –EDT: Develop NER for: people, organizations, geo-political entities (GPE), location, facility, vehicle, weapon, time, value … plus subtypes (e.g., educational organizations) –RDC: identify relation between entities: located, near, part-whole, membership, citizenship, … –EDC: identify events like interaction, movement, transfer, creation, destruction and their arguments

12. … and their arguments (entities)

18. Events, entities and mentions In ACE there is a distinction between an entity—a thing that exists in the Real World—and an entity mention—which is something that exists in the text (a substring). Likewise, and event is something that (will, might, or did) happen in the Real World, and an event mention is some text that refers to that event. –An event mention lives inside a sentence (the “extent”) with a “trigger” (or anchor) –An event mention is defined by its type and subtype (e.g, Life:Marry, Transaction:TransferMoney) and its arguments –Every argument is an entity mention that has been assigned a role.

19. Events, entities and mentions –An event mention lives inside a sentence (the “extent”) with a “trigger” (or anchor) –An event mention is defined by its type and subtype (e.g, Life:Marry, Transation:TransferMoney) and its arguments –Every argument is an entity mention that has been assigned a role. ITHACA, N.Y. -- The John D. and Catherine T. MacArthur Foundation today (Sept. 20) named Jon Kleinberg, Cornell professor of computer science, among the 25 new MacArthur Fellows -- the so-called "Genius Awards" -- for 2005. Dr. Kleinberg will receive $500,000 in no-strings-attached support over the next five years. Jon Kleinberg will receive $500,000 from the MacArthur Foundation over the next five years.

20. How to find events? The simple approach in Ahn, “Stages of event extraction”: Find all the entities and sentences –Ahn uses ground-truth labels for entities (an ACE thing) –Entities=candidate event arguments; sentences=candidate event extents; these will be classified and paired up Find the event mentions –Classify words as anchors (for event type T:S) or not: 35 classes, mostly None –Classify (event-mention,entity-mention) pairs as arguments (with role R) or not: 36 classes, mostly None Q: Why not just classify entity-mentions by Role? –Classify event mentions by modality, polarity, … –Classify (event-mention i,event-mention j ) pairs as co-referent or not. Treat all of these tasks as separate classification problems POS tag and parse everything, convert parse tree to dependency relations, and use all of these as features

21. Event Anchors - Features

22. Event Anchors – Results (MaxEnt and TIMBL)

23. Argument identification Function of both anchor and entity mention

24. Event co-reference …using greedy left-to-right clustering where you repeatedly decide “should I link new event mention M new with previous mention M 1, M 2, …. based on Pr(M new co-referent with M j )

25. Ahn: The punchline

26. The Webmaster Project: Yet Another Case Study with Einat Minkov (LTI), Anthony Tomasic (ISRI) See IJCAI-2005 paper

27. Overview and Motivations What’s new : –Adaptive NLP components –Learn to adapt to changes in domain of discourse –Deep analysis in limited but evolving domain Compared to past NLP systems : –Deep analysis in narrow domain (Chat-80, SHRDLU,...) –Shallow analysis in broad domain (POS taggers, NE recognizers, NP-chunkers,...) –Learning used as tool to develop non-adaptive NLP components Details : –Assume DB-backed website, where schema changes over time No other changes allowed (yet) –Interaction: User requests (via NL email) changes in factual content of website (assume update of one tuple) System analyzes request System presents preview page and editable form version of request Key points : –partial correctness is useful –user can verify correctness (vs case for DB queries, q/a,...) => source of training data...something in between...

28. POS tags NP chunks words,... features entity1, entity2,.... email msg Shallow NLPFeature Building C C C C requestType targetRelation targetAttrib Classification newEntity1,... oldEntity1,... keyEntity1,... otherEntity1,... NER database web page templates Update Request Construction preview page user-editable form version of request confirm? LEARNER offline training data User

29. Outline Training data/corpus –look at feasibility of learning the components that need to be adaptive, using a static corpus Analysis steps: –request type –entity recognition –role-based entity classification –target relation finding –target attribute finding –[request building] Conclusions/summary

30. Training data User1 User2 User3.... Mike Roborts should be Micheal Roberts in the staff listing, pls fix it. Thanks - W On the staff page, change Mike to Michael in the listing for “Mike Roberts”.

31. Training data User1 User2 User3.... Add this as Greg Johnson’s phone number: 412 281 2000 Please add “412-281-2000” to greg johnson’s listing on the staff page.

32. Training data – entity names are made distinct User1 User2 User3.... Add this as Greg Johnson’s phone number: 412 281 2000 Please add “543-341-8999” to fred flintstone’s listing on the staff page. Modification: to make entity-extraction reasonable, remove duplicate entities by replacing them with alternatives (preserving case, typos, etc)

33. Training data User1 User2 User3.... Request1 Request2 Request3.... message(user 1,req 1) message(user 2,req 1).... message(user 1,req 2) message(user 2,req 2).... message(user 1,req 3) message(user 2,req 3)....

34. Training data – always test on a novel user? User1 User2 User3.... Request1 Request2 Request3.... message(user 1,req 1) message(user 2,req 1).... message(user 1,req 2) message(user 2,req 2).... message(user 1,req 3) message(user 2,req 3).... test train Simulate a distribution of many users (harder to learn)

35. Training data – always test on a novel request? User1 User2 User3 Request1 Request2 Request3 message(user 1,req 1) message(user 2,req 1).... message(user 1,req 3) message(user 2,req 3).... test train message(user 1,req 2) message(user 2,req 2).... Simulate a distribution of many requests (much harder to learn) 617 emails total + 96 similar ones

36. Training data – limitations One DB schema, one off-line dataset –May differ from data collected on-line –So, no claims made for tasks where data will be substantially different (i.e., entity recognition) –No claims made about incremental learning/transfer All learning problems considered separate One step of request-building is trivial for the schema considered: –Given entity E and relation R, to which attribute of R does E correspond? –So, we assume this mapping is trivial (general case requires another entity classifier)

37. POS tags NP chunks words,... features entity1, entity2,.... email msg Shallow NLPFeature Building C C C C requestType targetRelation targetAttrib newEntity1,... oldEntity1,... keyEntity1,... otherEntity1,... Information Extraction

38. Entity Extraction Results We assume a fixed set of entity types –no adaptivity needed (unclear if data can be collected) Evaluated: –hand-coded rules ( approx cascaded FST in “Mixup” language) –learned classifiers with standard feature set and also a “tuned” feature set, which Einat tweaked –results are in F1 (harmonic avg of recall and precision) –two learning methods, both based on “token tagging” Conditional Random Fields (CRF) Voted-perception discriminative training for an HMM (VP-HMM)

39. Entity Extraction Results – v2 (CV on users)

40. POS tags NP chunks words,... features entity1, entity2,.... email msg Shallow NLPFeature Building C C C C requestType targetRelation targetAttrib newEntity1,... oldEntity1,... keyEntity1,... otherEntity1,... Information Extraction

41. Entity Classification Results Entity “roles”: –keyEntity: value used to retrieve a tuple that will be updated (“delete greg’s phone number”) –newEntity: value to be added to database (“William’s new office # is 5307 WH”). –oldEntity: value to be overwritten or deleted (“change mike to Michael in the listing for...”) –irrelevantEntity: not needed to build the request (“please add.... – thanks, William”) Features: closest preceding preposition closest preceding “action verb” (add, change, delete, remove,...) closest preceding word which is a preposition, action verb, or determiner (in “determined” NP) is entity followed by ‘s

42. POS tags NP chunks words,... features entity1, entity2,.... email msg Shallow NLPFeature Building C C C C requestType targetRelation targetAttrib newEntity1,... oldEntity1,... keyEntity1,... otherEntity1,... Information Extraction

43. POS tags NP chunks words,... features entity1, entity2,.... email msg Shallow NLPFeature Building C C C C requestType targetRelation targetAttrib newEntity1,... oldEntity1,... keyEntity1,... otherEntity1,... Information Extraction Reasonable results with “bag of words” features.

44. POS tags NP chunks words,... features entity1, entity2,.... email msg Shallow NLPFeature Building C C C C requestType targetRelation targetAttrib newEntity1,... oldEntity1,... keyEntity1,... otherEntity1,... Information Extraction

45. Request type classification: addTuple, alterValue, deleteTuple, or deleteValue? Can be determined from entity roles, except for deleteTuple and deleteValue. –“Delete the phone # for Scott” vs “Delete the row for Scott” Features : –counts of each entity role –action verbs –nouns in NPs which are (probably) objects of action verb –(optionally) same nouns, tagged with a dictionary Target attributes are similar Comments: Very little data is available Twelve words of schema-specific knowledge: dictionary of terms like phone, extension, room, office,...

46. POS tags NP chunks words,... features entity1, entity2,.... email msg Shallow NLPFeature Building C C C C requestType targetRelation targetAttrib newEntity1,... oldEntity1,... keyEntity1,... otherEntity1,... Information Extraction

47. POS tags NP chunks words,... features entity1, entity2,.... email msg Shallow NLPFeature Building C C C C requestType targetRelation targetAttrib newEntity1,... oldEntity1,... keyEntity1,... otherEntity1,... Information Extraction

48. Training data User1 User2 User3.... Request1 Request2 Request3.... message(user 1,req 1) message(user 2,req 1).... message(user 1,req 2) message(user 2,req 2).... message(user 1,req 3) message(user 2,req 3)....

49. Training data – always test on a novel user? User1 User2 User3.... Request1 Request2 Request3.... message(user 1,req 1) message(user 2,req 1).... message(user 1,req 2) message(user 2,req 2).... message(user 1,req 3) message(user 2,req 3).... test train Simulate a distribution of many users (harder to learn)

50. Training data – always test on a novel request? User1 User2 User3 Request1 Request2 Request3 message(user 1,req 1) message(user 2,req 1).... message(user 1,req 3) message(user 2,req 3).... test train message(user 1,req 2) message(user 2,req 2).... Simulate a distribution of many requests (much harder to learn) 617 emails total + 96 similar ones

51. Other issues: a large pool of users and/or requests usr

52. Webmaster: the punchline

53. Conclusions? System architecture allows all schema-dependent knowledge to be learned –Potential to adapt to changes in schema –Data needed for learning can be collected from user Learning appears to be possible on reasonable time-scales –10s or 100s of relevant examples, not thousands –Schema-independent linguistic knowledge is useful F1 is eighties is possible on almost all subtasks. –Counter-examples are rarely changed relations (budget) and distinctions for which little data is available There is substantial redundancy in different subtasks –Opportunity for learning suites of probabilistic classifiers, etc Even an imperfect IE system can be useful…. –With the right interface…

54. POS tags NP chunks words,... features entity1, entity2,.... email msg Shallow NLPFeature Building C C C C requestType targetRelation targetAttrib newEntity1,... oldEntity1,... keyEntity1,... otherEntity1,... Information Extraction micro- form query DB

57. Webmaster: the Epilog (VIO) Faster for request-submitter Zero time for webmaster Zero latency More reliable (!) Tomasic et al, IUI 2006 Entity F1 ~= 84, Micro-form selection accuracy =~ 80 Used UI for experiments on real people (human-human, human-VIO)

58. Conclusions and comments Two case studies of non-trivial IE pipelines illustrate: –In any pipeline, errors propogate –What’s the right way of training components in a pipeline? Independently? How can (and when should) we make decisions using some flavor of joint inference? Some practical questions for pipeline components: –What’s downstream? What do errors cost? –Often we can’t see the end of the pipeline… How robust is the method ? –new users, new newswire sources, new upsteam components… –Do different learning methods/feature sets differ in robustness? Some concrete questions for learning relations between entities: –(When) is classifying pairs of things the right approach? How do you represent pairs of objects? How to you represent structure, like dependency parses? Kernels? Special features?