Natural Language Questions for the Web of Data Mohamed Yahya, Klaus Berberich, Gerhard Weikum Max Planck Institute for Informatics, Germany Shady Elbassuoni Qatar Computing Research Institute Maya Ramanath Dept. of CSE, IIT-Delhi, India Volker Tresp Siemens AG, Corporate Technology, Munich, Germany EMNLP 2012

Q NL Translation to Q NL : Natural Language Questions “Which female actor played in Casablanca and is married to a writer who was born in Rome?”. Q FL : SPARQL 1.0 ?x hasGender female ?x marriedTo ?w ?x isa actor?w isa writer ?x actedIn Casablanca_(film) ?w bornIn Rome Translation Problem : This complex query is difficult for the user Soluction : automatically Translate q NL to q FL Natural Language Questions for the Web of Data

YAGO2 is a huge semantic knowledge base, derived from Wikipedia, WordNet and GeoNames. Natural Language Questions for the Web of Data Knowledge base Relation ClassEntities

Architecture of System DEANNA (DEep Answers for maNy Naturally Asked questions) Natural Language Questions for the Web of Data

Phrase detection A detected phrase p is a pair Toks : phrase l : label (l ∈ {concept, relation}) Natural Language Questions for the Web of Data Phrase detection Q NL Phrase P r : { } P c : { }

Phrase detection e.q. “Which female actor played in Casablanca and is married to a writer who was born in Rome?” use a detector that works against a phrase-concept dictionary concept phrase detection : phrase-concept dictionary : instances of the means relation in Yago2 Natural Language Questions for the Web of Data

Phrase detection relation phrase detection : rely on a relation detector based on ReVerb (Fader et al., 2011) with additional POS tag patterns e.q. “Which female actor played in Casablanca and is married to a writer who was born in Rome?” Natural Language Questions for the Web of Data

Phrase Mapping Two kinds of phrase Mapping: – The mapping of concept phrases – The mapping of relation phrases Phrase Mapping Phrase Mappings Natural Language Questions for the Web of Data

Phrase Mapping the mapping of concept phrases: e.q. “Which female actor played in Casablanca and is married to a writer who was born in Rome?” phrase-concept dictionary : instances of the means relation in Yago2 also use a detector that works against a phrase-concept dictionary Natural Language Questions for the Web of Data

Phrase Mapping the mapping relation phrases: rely on a corpus of textual patterns to relation mappings e.q. “Which female actor played in Casablanca and is married to a writer who was born in Rome?” textual patterns relation Natural Language Questions for the Web of Data

Q-Unit Generation Mapping Candidate graph Dependency parsing q-unit is a triple of sets of phrases Two parts of q-uint generation step: Natural Language Questions for the Web of Data

Q-Unit Generation Dependency parsing : identifies triples of tokens:, where t rel, t arg1, t arg2 ∈ q NL who was born in Rome? nsubjpass(born-3, who-1) auxpass(born-3, was-2) root(ROOT-0, born-3) prep_in(born-3, Rome-5) e.q. born whoRome t rel t arg1 t arg2 root nsubjpass in, Natural Language Questions for the Web of Data

Q-Unit Generation q-unit is a triple of sets of phrases,t rel ∈ p rel, t arg1 ∈ p arg1, and t arg2 ∈ p arg2. born was born,, a writerRome PrPr PcPc PcPc Natural Language Questions for the Web of Data

Joint Disambiguation Rule 2: each phrase is assigned to at most one semantic item Rule 1: resolves the phrase boundary ambiguity (only nonoverlapping phrases are mapped) Natural Language Questions for the Web of Data e

Joint Disambiguation Disambiguation Graph Joint disambiguation takes place over a disambiguation graph DG = (V, E), – V = V s ∪ V p ∪ V q – E = E sim ∪ E coh ∪ E q Natural Language Questions for the Web of Data

Joint Disambiguation V s : the set of s-node V p : the set of p-node V rp : the set of relation phrases V rc : the set of concept phrases V q : a set of placeholder nodes for q–units Disambiguation Graph: Vertices Natural Language Questions for the Web of Data

Disambiguation Graph Disambiguation Graph: Edges Esim: E sim ⊆ V p × V s a set of weighted similarity edges Ecoh: E coh ⊆ V s × V s a set of weighted coherence edges Eq: E q ⊆ V q × V p × d d ∈ {rel, arg1, arg2} Q-edges sim-edges Ecoh: Natural Language Questions for the Web of Data

Disambiguation Graph Edge Weights Coh sem (Semantic Coherence) – between two semantic items s1 and s2 as the Jaccard coefficient of their sets of inlinks. Three kinds of inlink – InLinks(e) – InLinks(c) – InLinks(r) Natural Language Questions for the Web of Data

Disambiguation Graph: Edge Weights Coh sem : inlinks of entity InLinks(e): – the set of Yago2 entities whose corresponding Wikipedia pages link to the entity. E.q. – InLinks(Casablanca) = {Marwan_al-Shehhi, Ingrid_Bergman, …, Morocco,…} InLinks(Casablanca) sb.mpg.de/webyagospo/Browser Natural Language Questions for the Web of Data

Disambiguation Graph: Edge Weights Coh sem : inlinks of class InLinks(c) = ∪ e ∈ c Inlinks(e) E.q. – InLinks(wikicategory_Metropolitan_areas_of_Morocco) = InLinks(Casablanca) ∪ InLinks(Marrakech) ∪ … ∪ InLinks(Rabat) Natural Language Questions for the Web of Data entities class

Disambiguation Graph: Edge Weights Coh sem : inlinks of ralation InLinks(r) = ∪ (e1, e2) ∈ r (InLinks(e 1 ) ∩ InLinks(e 2 )) Natural Language Questions for the Web of Data

Similarity Weights Similarity Weights of entities – how often a phrase refers to a certain entity in Wikipedia. Similarity Weights of classes – reflects the number of members in a class Similarity Weights of relations – reflects the maximum n-gram similarity between the phrase and any of the relation’s surface forms Natural Language Questions for the Web of Data

Joint Disambiguation Disambiguation Graph Processing The result of disambiguation is a subgraph of the disambiguation graph, yielding the most coherent mappings. We employ an ILP(integer linear program) to this end. Natural Language Questions for the Web of Data ILP e

Joint Disambiguation : ILP Natural Language Questions for the Web of Data Definitions :

Joint Disambiguation : ILP objective function : Natural Language Questions for the Web of Data

Joint Disambiguation : ILP Constraints: Natural Language Questions for the Web of Data

Joint Disambiguation : ILP resulting subgraph Natural Language Questions for the Web of Data e

Query Generation not assign subject/object roles in triploids and q-units Replacing each semantic class with distinct type-constrained variable Example: – “Which singer is married to a singer?” ?x type singer, ?x marriedTo ?y, and ?y type singer Natural Language Questions for the Web of Data

Query Generation E.q. Natural Language Questions for the Web of Data e ?x Replacing each semantic class ?x ?y Q-uint: arg1 rel arg2 Generation ?x type writer ?y type person bornInRome ?yactedInCasablanca ?ymarried ?x

Evaluation Three part of Evaluation: Datasets Evaluation Metrics Results & Discussion Natural Language Questions for the Web of Data

Datasets Experiments are based on two datasets: – QALD-1 1st Workshop on Question Answering over Linked Data (QALD-1) the context of the NAGA project – NAGA collection The NAGA collection is based on linking data from the Yago2 knowledge base Training set: – 23 QALD-1 questions – 43 NAGA questions Test set: – 27 QALD-1 questions – 44 NAGA questions hyperparameters (α, β, γ) in the ILP objective function. 19 QALD-1 questions in Test set Natural Language Questions for the Web of Data

Evaluation Metrics evaluated the output of DEANNA at three stages – after the disambiguation of phrases – after the generation of the SPARQL query – after obtaining answers from the underlying linked-data sources Judgement – two human assessors – If they were in disagreement then a third person resolved the judgment. Natural Language Questions for the Web of Data

Evaluation Metrics disambiguation stage looked at each q-node/s-node pair. whether the mapping was correct or not. whether any expected mappings were missing. Natural Language Questions for the Web of Data e

Evaluation Metrics query-generation stage Looked at each triple pattern. whether the pattern was meaningful for the question or not. whether any expected triple pattern was missing. e.q. (triple pattern) ?x bornIn Rome ?y actedIn Casablanca ?y married ?x Natural Language Questions for the Web of Data

query-answering stage the judges were asked to identify if the result sets for the generated queries are satisfactory. Natural Language Questions for the Web of Data

Results question q item set s correct(q, s) : – the number of correct items in s ideal(q) : the size of the ideal item set retrieved(q, s) : the number of retrieved items define: coverage and precision as follows: – cov(q, s) = correct(q, s) / ideal(q) – prec(q, s) = correct(q, s) / retrieved(q, s). Natural Language Questions for the Web of Data

Micro-averaging aggregates over all assessed items regardless of the questions to which they belong. Macro-averaging first aggregates the items for the same question, and then averages the quality measure over all questions. For a question q and item set s in one of the stages of evaluation correct(q, s) : the number of correct items in s ideal(q) : the size of the ideal item set retrieved(q, s) : the number of retrieved items define coverage and precision as follows: cov(q, s) = correct(q, s) / ideal(q) prec(q, s) = correct(q, s) / retrieved(q, s).

Results Example questions, the generated SPARQL queries and their answers Natural Language Questions for the Web of Data the relation bornIn relates people to cities and not countries in Yago2.

Results Natural Language Questions for the Web of Data Relaxation use (Elbassuoni et al., 2009)

Natural Language Questions for the Web of Data

Conclusions Author presented a method for translating natural language questions into structured queries. Although author’s model, in principle, leads to high combinatorial complexity, they observed that the Gurobi solver could handle they judiciously designed ILP very efficiently. Author’s experimental studies showed very high precision and good coverage of the query translation, and good results in the actual question answers. Natural Language Questions for the Web of Data