1. Reading Report on Question Answering
论文阅读
瞿裕忠 南京大学计算机系

2. Articles Open Domain Question Answering via Semantic Enrichment
WWW2015, Microsoft, UIUC
HAWK – Hybrid Question Answering using Linked Data
ESWC2015, University of Leipzig

3. Agenda Introduction Methodology Feature development (Ranking)
Answer type prediction (Probabilistic models)
Experiments
Conclusion

4. Introduction Web Search  open domain QA
KB-based QA (structured knowledge bases)
Corpus-based QA (unstructured corpus)

5. Introduction Difficulty for KB-based QA systems
Incompleteness of KB: Information required to answer a question may not always exist in KBs.
Semantic parsing: Although semantic parsing [2, 3] has been a hot research topic recently, the problem of mapping natural language utterances to logical-form queries is still considered largely unsolved.
Weakness of Web-based QA systems
Insufficient knowledge about the generated answer candidates.
Different mentions of the same entity such as “President Obama” and “Barack Obama” are viewed as different answer candidates.
Answer type checking relies on a generic named entity recognition component that provides a small set of crude type labels.

6. Introduction A new QA system framework, named QuASE
Question Answering via Semantic Enrichment

7. Who was the first American in space?
Example
Who was the first American in space?
Submit it to a search engine
Get a set of relevant sentences
On May 5, 1961, Shepard piloted the Freedom 7 mission... ;
Alan Shepard became the first American in space when the Freedom 7...;
Linking entity to Freebase
Freedom 7, Alan Shepard, Sally Ride, …
Such linked entities are treated as answer candidates to the given question.
Semantic features are integrated into a ranking algorithm.
Alan Shepard

8. Example
Who was the first American in space?

9. Methodology Web Sentence Selection via Search Engine
Answer Candidate Generation via Entity Linking
Feature Generation and Ranking.

10. Methodology (1) Web Sentence Selection via Search Engine
Submit the question as a query to a search engine
Collect the top-50 returned snippets, as well as the top-50 documents.
Compute the word count vector based on the returned snippets to represent the information for the query, denoted as wq.
For each sentence from the top-50 returned documents, we compute its word count vector ws, and select those sentences with a high cos(ws, wq) into the high-quality sentence set.

11. Methodology (2) Answer Candidate Generation via Entity Linking
Primarily focus on those questions targeted at certain entities in KBs. ( e.g. excluding “when” question)
Use one of the entity linking systems [13] to identify answer candidates linked to Freebase.
This system achieves the best scores at TAC-KBP 2013, by several novel designs such as postponing surface form boundary detections and discriminating concepts and entities in Wikipedia pages.

12. Methodology (3) Feature Generation and Ranking
For each answer candidate, Freebase contains a wealth of information, such as their description texts and entity types.
A set of semantic features shall be developed based on such rich information, and subsequently utilized in a ranking algorithm to evaluate the appropriateness of each candidate as the true answer.

13. Features
Count: high frequency of answer candidate serves as a significant indicator of being the correct answer
Textual Relevance
Answer Type Related Features
Propose probabilistic models to directly measure the matching degree between a question and an answer candidate’s Freebase types.

14. Word to Answer Type (WAT) Model

15. JQA model
Joint <question, answer type> association (JQA) model

16. TREC dataset: factoid questions from TREC 8-12
Experimental Setup
TREC dataset: factoid questions from TREC 8-12
Among the remaining 1902 questions (entity-oriented)
Bing query: crowdsourcing, entity answers in Freebase.
Approximately 6000 question-answer pairs

17. Training Dataset for JQA and WAT
Experimental Setup
Training Dataset for JQA and WAT
Around 1.3 million <question, answer types> pairs based on Bing query logs.
<query, clicked url> pairs from the query click logs
each entity in Freebase is also linked to some urls that are related to this entity (mostly Wikipedia pages or official web sites of this entity).
Freebase types of the entity corresponding to the clicked url as the answer types.

18. Answer Candidate Ranking
Experimental Setup
Answer Candidate Ranking
Use an in-house fast implementation of the MART gradient boosting decision tree algorithm [9, 21] to learn the ranker using the training set of our data.

19. Experimental Setup Evaluation Measures
F1 score, which is the harmonic mean of precision and recall.
Mean Reciprocal Rank (MRR).

20. Alternative QA systems
Experimental Setup
Alternative QA systems
KB-based QA system: Sempre[2, 3]
Web-based QA system: AskMSR+ [39]

21. Experimental Results

22. Experimental Results

23. Analysis of QuASE and AskMSR+ on their respective failed questions
Experimental Results
Analysis of QuASE and AskMSR+ on their respective failed questions
Compared with AskMSR+, features in QuASE for ranking are effective as long as true answers are in the candidate list.

24. Experimental Results
For QuASE, how to improve entity linking performance, in order to include true answers in the candidate list, can be important to further improve QA performance.

25. Conclusion Contributions Future work A new QA Framework, QuASE
Answer Type Checking Models
Extensive Experimental Evaluation
Future work
Relationships among entities can also be explored as semantic features to be incorporated in our system
To improve entity linking performance

26. HAWK--Hybrid QA using Linked Data
Introduction
Methodology
Evaluation
Conclusion

27. Introduction Hybrid question answering
Find and combine information stored in both structured and textual data sources
Document Web
Labels and abstracts in Linked Data sources
Question Answering over Linked Data (QALD-4)
Task 3: Hybrid question answering

28. Example
Which recipients of the Victoria Cross died in the Battle of Arnhem?
Cannot be answered by solely DBpedia or Wikipedia abstracts
?uri dbo:award dbr:Victoria_Cross
The abstract for John Hollington Grayburn
‘he went into action in the Battle of Arnhem [...] but was killed after standing up in full view of a German tank’.

29. Architectural overview of HAWK
Introduction
Architectural overview of HAWK

30. Methodology POS-Tagging Entity Annotation Dependency Parsing
Linguistic Pruning
Semantic Annotation
Generating SPARQL Queries
Semantic Pruning of SPARQL Queries
Ranking

31. Predicate-argument tree
By Example
Which recipients of the Victoria Cross died in the Battle of Arnhem?
Predicate-argument tree
Tree after pruning

32. Methodology Semantic Annotation
nouns correspond to object type properties and classes
verbs correspond to object type properties
question words (e.g., who or where) correspond to classes (e.g., Person or Place)

33. Methodology Generating SPARQL Queries
Traverses the tree in a pre-order walk
Related information are situated close to each other in tree
Information are more restrictive from left to right
Triple pattern for generating queries while traversal
E.g. a variable bound to the class Place will not have an outgoing predicate birthPlace.
Semantic Pruning of SPARQL Queries

34. Methodology
Ranking
HAWK ranks queries using supervised training based on the gold standard answer set from the QALD-4 benchmark.
Feature selection?

35. Evaluation To evaluate HAWK
focus on this hybrid training dataset comprising 25 questions,
17 out of which are entity searches
Using only DBpedia type information
No aggregation process.

36. Experimental Results Red: inability to generate correct query,
Green: missing recall.
Blue: missing precision

37. Experimental Results

38. Failing entity annotation
Error Analysis
Failing entity annotation
Queries 1, 11 and 15.
Jane Austin, G8, Los Alamos.
Without matching entity annotations a full-text search retrieves too many matches for reaching high precision values on limited result set.
Query structure
Queries 11 or 15
complex query structures lead to a multitude of interpretations
Missing type information
Some of the resources of the gold standard do not have appropriate type information leading to a high amount of queries that need to be ranked correctly.

39. Error Analysis Query example
1. Give me the currencies of all G8 countries.
11.Who composed the music for the film that depicts the early life of Jane Austin?
15. Of the people that died of radiation in Los Alamos, whose death was an accident?

40. Conclusion Contributions Future work
HAWK, the first hybrid QA system for the Web of Data
A generic approach to generate SPARQL queries out of predicate argument structures
Achieve up to 0.68 F-measure on the QALD-4 benchmark.
Future work
Finding the correct ranking approach to map a predicate-argument tree to a possible interpretation
Computational complexity
Domain-specific applications (higher F-measures)

41. 致谢
欢迎提问！