1. Reasoning With Neural Tensor Networks for Knowledge Base Completion
Author: Richard Socher
Presented by Batselem
숭실대학교

2. Overview Introduction to Knowledge Base Knowledge Base Reasoning
Knowledge Base Completion
Neural Tensor Network
Knowledge Base Completion Experiment

3. Knowledge Base
A knowledge base is a special kind of database for knowledge management.
Designed for storing complex information
Captures human knowledge and places it into a computer system
Used to solve complex problems that require a high level of human expertise.
Knowledge
from an expert
Non-expert
user
User Interface
query
Inference
Engine
(추론 시스템)
Knowledge
base
advice
“Captures human knowledge and places it into a computer system where it is used to solve complex problems requiring a high level of human expertise.“
“Knowledge is Relevant information that one is able to recall from memory. A database is a structured collection of records or data”
A collection of information organized in such a way that a computer program can quickly select desired pieces of data.
It is possible to make conclusions from the information held in the knowledge base.

4. Knowledge Base Reasoning
Knowledge base consists of a large amount of facts (triples)
Represented as triples of a relation between two entities T = (e1, R, e2)
Reasoning over a large knowledge base (KB)
Entity1
Entity2
Relation
Shakespeare
Hamlet
authorOf
Knowledge Base Facts:
ex:book1 rdf:type ex:Publication .
ex:book2 rdf:type ex:Article .
ex:Article rdfs:subClassOf ex:Publication .
ex:publishes rdfs:range ex:Publication .
ex:MITPress ex:publishes ex:book3 . …
rdf:type
ex:book1
ex:Publication
rdfs:subClassOf
ex:book2
rdfs:range
rdf:type
ex:Article
ex:publishes
ex:book3
ex:MITPress

5. Knowledge Base Reasoning
Knowledge base consists of a large amount of facts (triples)
Represented as triples of a relation between two entities T = (e1, R, e2)
Reasoning over a large knowledge base (KB)
Entity1
Entity2
Relation
Shakespeare
Hamlet
authorOf
Logic (IF-THEN) Rules:
IF X is subclass of Y and Z is type of X
THEN Z is type of Y
rdf:type
ex:book1
ex:Publication
Infer new
relations
rdf:type
rdfs:subClassOf
rdf:type
ex:book2
rdfs:range
rdf:type
ex:Article
ex:publishes
ex:book3
ex:MITPress

6. Knowledge Base Completion
Knowledge bases are an important resource for question answering and other natural language processing (NLP) tasks
Question Answering (Apple Siri, IBM Watson),
Information Retrieval
Providing structured knowledge to users.
Problems with KB:
Suffer from incompleteness
Lack of ability to reason over their discrete entities and relationships
This paper adopts the goal of predicting the likely truth of additional facts based on existing facts in the knowledge base
predict
KB: Existing
facts
KB: Additional
facts

7. Knowledge Base Completion
Introduce a model that can accurately predict additional true facts using only an existing database.
(e1, R, e2) = (Ben Affleck, occupation, FilmMaker) is true or false
Score
Argo
Violet Anne
Relation
director
Neural
Tensor
Network
playedIn
childOf
Actor
Ben Affleck
gender
occupation
male
livedIn
occupation
won
FilmMaker
(Tom cruise, playedIn, Mission Impossible)
(Tom cruise, occupation, Actor)
(Matt Damon, playedIn, Martian)
(Matt Damon, occupation, Actor) ….
Los Angeles
Oscar Award

8. Neural Tensor Network Word2vec
Each relation is described by a neural network and pairs of entities are given as input to the model.
Each entity has a vector representation, which can be constructed by its word vectors.
The model returns a high score if they are in that relationship and a low one otherwise.
This allows any fact, whether implicitly or explicitly mentioned in the database to be answered with a certainty score.
Large text data
Word2vec

9. Neural Tensor Network
It represents each entity as the average of its word vectors, .
an entity vector as the composition of its word vectors. For example
The goal of their approach is to be able to state whether two entities (e1, e2) are in a certain relationship R.
For instance, whether the relationship (e1, R, e2) = (Bengal tiger, has part, tail) is true and with what certainty.
To this end, they define a set of parameters indexed by R for each relation’s scoring function
VectorBank_of_China
VectorBank_of_China= 0.5*(VectorBank+ VectorChina)
VectorBank
VectorChina

10. Neural Tensor Network
The Neural Tensor Network (NTN) replaces a standard linear neural network layer with a bilinear tensor layer
that directly relates the two entity vectors across multiple dimensions.
The model computes a score of how likely it is that two entities are in a certain relationship by the following NTN-based function g(e1, R, e2):
f = tanh is a standard nonlinearity applied element-wise
WR[1:k] ∈ ℛ d×d×k is a tensor and the bilinear tensor product
Standard layer weight VR ∈ ℛ k×2d

11. Neural Tensor Network R
Reference:An MLP-based representation of neural tensor networks for the RDF data models

12. Neural Tensor Network
Training objective: T(i)c = (e(i)1, R(i),ec) is a triplet with a random entity corrupted from a correct triplet T (i) = (e(i)1, R(i), e(i)2)
Score the correct relation triplet higher than its corrupted one up to a margin of 1.
For example, given a correct triplet (Pablo Picaso, nationality, Spain), a potential negative example is (Pablo Picaso, nationality, United States).
For each correct triplet we sample C random corrupted triplets.

13. Neural Tensor Network
The model is trained by taking derivatives with respect to the five groups of parameters.
Cost Function
Update W using Backpropagation: 𝐽
𝜕𝐽/𝜕𝑊
𝑙𝑜𝑐𝑎𝑙 𝑚𝑖𝑛𝑖𝑚𝑢𝑚 𝑝𝑜𝑖𝑛𝑡
𝑊=𝑊 − 𝜕𝐽/𝜕𝑊

14. Experiment Datasets Relation Triplets Classification
They randomly switch entities from correct testing triplets resulting in an equal number of positive and negative examples.
We predict the relation (e1, R, e2) holds if g(e1, R, e2) ≥ TR

15. Experiment

16. Experiment
Given place of birth is Florence and profession is historian, relation model can accurately predict that Francesco Guicciardini’s gender is male and his nationality is Italy.
These might be inferred from two pieces of common knowledge: (i) Florence is a city of Italy; (ii) Francesco is a common name among males in Italy.

17. Q&A
Thank you