1. NELL Knowledge Base of Verbs
Derry Wijaya

2. Knowledge base of verbs
Contain: (1) Mapping of typed verb phrases (verbs + prepositions) to NELL KB relations
Approach:
Semi supervised learning trained on NELL facts (relation instances) as labeled data and ~600M Subject-Verb-Object (SVO) triples from ClueWeb as unlabeled data. Learn correspondences (mapping) between typed verb phrases and NELL relations. Typed verb phrases are verb phrases whose arguments are typed with NELL categories.
Use of background knowledge:
NELL KB enables distant training from unlabeled SVO triples whose subjects and objects are tagged with NELL categories. NELL KB also provides initial labeled training instances, initial verb phrases to relations mapping, and constraints (subsumption, exclusion) between relations.

3. Mapping Verbs and Relations
Task: which verbs express which relations?
Idea: map verbs to KB relations using Web as a kind of “interlingua” [Wijaya,2016]
wasBornIn
(person, city)
teamPlaysInCity
(sportsTeam, city)
(Steelers, Pittsburgh)
(Cavaliers, Cleveland)
(Madonna, Bay_City)
(Barack_Obama, Honolulu)
Knowledge Base
Obama, born in Honolulu, …
Madonna was born in Bay City, …
Steelers is based in Pittsburgh, PA
The Cavs played in Cleveland, Ohio
Web
The task is to find which verbs express which relations in the knowledge base. One solution is to find a mapping between the relations in the knowledge base and the verbs using Web text as a kind of interlingua.
So given relations in a knowledge base at one side and the verbs on the other, find mentions of the relation instances in the web text and the verbs that co-occur between them.
Using statistics of these find the mapping between relations and verbs.
The web text that we are using is about 600 million subject-verb-object triples extracted from ClueWeb.
~ 600 million
Subject-Verb-Object
(SVO) triples
From ClueWeb09
be born in
be based in
play in
Verbs

4. Mapping Verbs To Relations
Observation:
Prior knowledge: NELL’s CPL extraction patterns
Argument type of a verb can help disambiguate
<musician> “play” <musicInstrument>
musicianPlaysInstrument
teamPlaysSport
<sportsTeam> “play” <sport>
actorStarredInMovie
<actor> “play” <movie>
“play”
musicianPlaysInstrument
teamPlaysSport …
actorStarredInMovie
To add to this, we observe that prior knowledge – relation extraction patterns that CPL has learned can be useful to start the mapping with.
Furthermore, the argument type of the verbs can also help disambiguate which relations they express.
For example, the verb “play” can express many relations: musicianPlaysInstrument, athletePlaysSport, or actorPlaysMovie among others.
Using the types of the arguments of the verb “play” can help us disambiguate which relation the particular “play” expresses.
Hence, instead of mapping verbs to relations, we will be mapping typed-verbs (verbs whose arguments are typed with categories from the knowledge base) to relations.
<type_1> verb <type_2>
<type_1> KB relation(s) <type_2>

5. Mapping Verbs To Relations
Initialization
wasBornIn
(person, city)
teamPlaysInCity
(sportsTeam, city)
(Steelers, Pittsburgh)
(Cavaliers, Cleveland)
(Madonna, Bay_City)
(Barack_Obama, Honolulu)
Knowledge Base
Obama, born in Honolulu, …
Madonna was born in Bay City, …
Steelers is based in Pittsburgh, PA
The Cavs played in Cleveland, Ohio
SVO
Our method starts by building an initial classifier using only labeled instances and the verbs that co-occur with them as features.
Then, while log likelihood of the parameters of θ improve, do the E- and the M-step.
Build a classifier θ using only labeled instances
While the log probability of the data improves, l(θ|D), do the E- and M-step

6. Mapping Verbs To Relations
E-step
Bill Clinton, born in Hope, …
Bill Clinton was born in Hope, …
Pirates is based in Pittsburgh, PA
The New England Patriots played in Foxborough
SVO
teamPlaysInCity(Pirates, Pittsburgh)
teamPlaysInCity(The_New_England_Patriots, Foxborough)
wasBornIn(Bill_Clinton, Hope)
In the E-step we use the current classifier θ to label unlabeled instances using verbs that co-occur with them in SVO as features.
We incorporate type checking in that an unlabeled instance is only labeled with relations that match its entities’ types.
Use current classifier θ to label unlabeled instances, P(r | du(e1, e2) )
An unlabeled entity pair is only labeled with relations whose argument types match those of the entity pair

7. Mapping Verbs To Relations
M-step
teamPlaysInCity(Steelers, Pittsburgh)
teamPlaysInCity(Cavaliers, Cleveland)
wasBornIn(Barack_Obama, Honolulu)
wasBornIn(Madonna, Bay_City)
teamPlaysInCity(Pirates, Pittsburgh)
teamPlaysInCity(The_New_England_Patriots, Foxborough)
wasBornIn(Bill_Clinton, Hope)
Use labeled instances and the estimated labels to re-estimate the parameters of θ, P(r) and P(v|r)
A verb is only mapped to relations whose argument types match the types of at least one of the entity pairs that co-occur with the verb
Use NELL’s CPL patterns as prior, Pe(v | r)
In the M-step, we use the estimated labels and our labeled instances to re-estimate the parameters of our classifier.
We incorporate type checking in that a verb is only mapped to relations whose argument types match the types of at least one entity pair that co-occur with the verb in the SVO.
We also incorporate CPL patterns that contain verbs as prior for computing the parameters of our classifier.
At the end of the iteration we’ll have relation labels for unlabeled entity pairs and the probabilities of verbs given relations from the features of the classifier.

8. Mapping Verbs To Relations
Made publicly available at:
We make this mapping from verbs to relations available in our website.

9. English Verbs To Relations
personHasJobPosition
These are some examples of mapping from English verbs to the relation personHasJobPosition.

10. English Relation Instances (newly discovered)
writerWasBornInCity
These are some examples of instances that our method is able to discover for writerWasBornInCity relation.

11. Portuguese Verbs To Relations
bacteriaeoAgenteCausadorDeCondicaoFisiologica
These are some examples of mapping from Portuguese verbs to the relation bacteria caused physiological condition.
Note that CPL has not extracted patterns for this relation; our method can in this case complement CPL in finding patterns that express the relation.
No verb CPL patterns

12. Portuguese Relation Instances (newly discovered)
estadioLocalizadoNaCidade
These are some examples of new instances that our method is able to extract for the stadium located in city relation.
Note that Portuguese NELL has only one example for this relation, our method is able to extract more instances for the relation.
Only one seed Portuguese instance

13. Knowledge base of verbs
Contain: (2) Mapping of verb phrases to addition/deletion of relation instances
Approach:
We use Wikipedia edit history: the correspondence between addition/deletion of verb phrases on a Wikipedia page and updates to the infobox (KB) slots to distantly supervise a method for automatically learning verb phrases and KB changes. In addition, the method uses constraints among infobox slots to effectively map verbs to infobox changes.
Use of background knowledge:
Concurrent changes of texts and infobox slots in Wikipedia pages enables distant training from unlabeled verb phrases to state changes in the underlying KB.

14. Mapping Verbs and Changes in Relations
Task: which verbs (express events and therefore) initiate/terminate which relations? [Wijaya,2015]
Observation:
verbs expressing events discoverable in text
events + state-change correspondence effective for knowledge base updates
Idea: learn correspondence between verbs and state-changes in KB
from a corpus that has both text change and the accompanying KB change
Given the observation that verbs that express events are discoverable in text and that finding correspondence between events and state-changes is effective for updating knowledge base, we want to learn this correspondence between verbs and state-changes in KB from a corpus that has both text and accompanying KB and which changes over time

15. Wikipedia Revision infobox change text change
One such corpora is Wikipedia revision. When an event happens to an entity, its Wikipedia page is revised with some texts being added or removed and some infobox relations being initiated or terminated.
For example, in the marriage event that happens to this entity, the text is changed with verbs “be married in” added. The infobox is also changed with the initiation of a new spouse relation, with the addition of a new spouse value and start time.
Wikipedia revision of on 25 May 2014

16. Mapping Verbs and Changes in Relations
Made publicly available at:

17. Mapping Verbs and Changes in Relations
Begin - spouse

18. Mapping Verbs and Changes in Relations
End - spouse

19. Mapping Verbs and Changes in Relations
Begin – death date

20. Knowledge base of verbs
Contain: (3) Extension of NELL Relation Ontology to Higher Coverage
Approach:
Collect ~600M SVO triples by parsing 500M web pages (ClueWeb), a total of 2.1B mentions
Annotate S, O in the triple with NELL category (~94M NELL-typed SVO triples, 479K un-typed verb phrases)
Get typed verb phrases by computing selectional preference of each verb phrase, e.g. the verb phrase “eat with” prefers a subject of type Person and an object of type TableItem (~270K typed verb phrases, 61K untyped verb phrases, a total of 2B mentions)
Typed verb phrases are clustered to form relations by similarity in S-O pairs
Clusters of typed verb phrases that are not yet mapped to NELL relations are candidates for new relations
Use of background knowledge:
NELL KB enables assignment of semantic categories to each S, O of unlabeled SVO triples that allows typing of verb phrases
Thesis:
Nearly every binary relations can be described* by a verb or verb+prep, plus NELL argument types
AthletePlaysSport, AthletePlaysForTeam, PersonResemblesPerson, PersonFeelsEmotion, PersonDiesInLocation, …
The sense of a verb can be disambiguated to some extent by the types of its arguments
PersonPlaysSport, PersonPlaysMusicalInstrument, PersonPlaysPerson, ...
* described, not extracted

21. Knowledge Base of Verbs
(1), (2) and (3) will be made publicly available at:

22. Knowledge Base of Verbs
Main page

23. Knowledge Base of Verbs
Typed verb patterns and their mapping to Relations

24. Knowledge Base of Verbs
Actual mentions of the typed verb pattern in clueweb09 corpus

25. Knowledge base of verbs
Usage: NELL micro reader as the nell-verb component
Approach:
During reading: NELL KB provides background knowledge on fine-grained semantic types of noun phrases that are arguments of verb phrases. Typed verb phrases are then mapped to relations.
NELL-provided Noun Phrase types:

26. NELL-verb: (typed verb phrases to relations mapping)

27. References
Wijaya, Derry, Ndapandula Nakashole, and Tom M. Mitchell. “A Spousal Relation Begins with a Deletion of engage and Ends with an Addition of divorce”: Learning State Changing Verbs from Wikipedia Revision History. EMNLP 2015.
Wijaya, Derry and Mitchell, Tom. Mapping Verbs in Different Languages to Knowledge Base Relations using Web Text as Interlingua. NAACL 2016
These are some papers that I’ve referenced in this talk.
Thank you and hope you’ve enjoyed the talk.