1. Web Logs and Question Answering Richard Sutcliffe 1, Udo Kruschwitz 2, Thomas Mandl 3 1 - University of Limerick, Ireland 2 - University of Essex, UK 3 - University of Hildesheim, Germany

2. Outline Question Answering (QA) Query Log Analysis (QLA) Characteristics of QA and QLA QA & QLA: 8 Key Questions Workshop Papers Key Questions addressed by Papers Conclusions

3. Question Answering (QA) A Question Answering (QA) system takes as input a short natural language question and a document collection and produces an exact answer to the question, taken from the collection Origins go back to TREC-8 (Voorhees and Harman, 1999)

4. Query Log Analysis (QLA) A Query Log is a record of a person’s internet search Log comprises query plus related information Query Log Analysis looks at Logs mainly in order to improve search engines Early study was Spink and Saracevic (2000)

5. Strengths & Weaknesses of QA Following TREC, CLEF and NTCIR, we know how to build efficient monolingual factoid QA systems However, range of questions asked is extremely narrow Also, work is based on fixed document collections Most evaluation is offline, using artificial queries Real users and real information needs have been ignored Thus, QA is not a solved problem

6. Strengths & Weaknesses of QLA Potentially there is a huge amount of data, increasing all the time Queries entered are ‘naturally occurring’ because users do not know they are monitored! On the other hand, huge data sets pose problems; manual analysis cannot be used but machine learning etc must be used We must infer from behaviour what users were thinking, what they wanted and whether a search succeeded Also, logs are mostly owned by search engine companies

7. QA & QLA – 8 Key Questions 1. Can the meaning of queries in logs be deduced? 2. Can NLP techniques such as Named Entity Recognition be applied in QLA? 3. Can QLA tell us new types of questions for QA research? 4. Can queries within a session be interpreted as a dialogue with the user giving the questions and the system providing the answers?

8. QA & QLA – 8 Key Questions Cont. 5. What can logs from real QA systems like lexxe.com or questions from sites like answers.com tell us? 6. Are QA logs different from IR logs? 7. Can click-through data enable us to deduce new QA question types? 8. What analysis could be done on logs made from telephone QA systems (e.g. cinema booking)

9. Papers -1 Bernardi and Kirschner: From artificial questions to real user interaction logs Real logs vs. not real questions at TREC etc Three sets (TREC, Bertomeu & BoB) analysed as dialogues TREC differs significantly from BoB (query length, no. of anaphora) Conclusion: future TREC-style evaluation should take these differences into account to make task more realistic

10. Papers - 2 Leveling: QA evaluation queries vs. real world queries Compares queries to a search engine, to answers.com, and used at TREC and CLEF (six sets) Infers the QA question type of a bare IR query (keywords) and converts it back into a syntactic QA query Conclusion: This process could be used to answer IR queries properly with a QA system

11. Papers - 3 Zhu et al.: Question Answering based on Community QA Considers whether Q-A pairs from Yahoo Answers can be used a log-like resource to improve QA Given input query, similar queries are identified in logs. Sentences from answers to these are selected by summarisation algorithm to use as response to query

12. Papers - 4 Momtazi and Klakow: Yahoo! Answers for sentence retrieval Two statistical frameworks developed for capturing relationships between words in Q-A pairs in Yahoo! Answers These were then used in sentence selection task based on TREC 2006 queries Conclusion: Best results exceeded the baseline

13. Papers - 5 Small and Strzalkowski: Collaborative QA using web trails Logs were made of users in an interactive QA study Information stored includes the documents users saved Docs are placed in standard order to allow comparison between users; docs saved by different users overlap When previously observed seq of docs is saved by user, rest of that seq could be presented to user

14. Papers - 6 Sutcliffe, White and Kruschwitz: NE recognition in intranet query log A log of queries to a university web site was first analysed by hand This resulted in a list of topic types and a list of Named Entity types Training data for NEs was extracted from web pages and used to train maximum entropy recogniser NE recogniser was evaluated; uses of NEs in answering queries were discussed

15. Papers - 7 Mandl and Schulz: Log-based evaluation resources for QA Concerned with link between query logs and well-formed questions answered by QA systems Proposes a system switching between IR-mode and QA- mode Discusses log resources available and related tracks at CLEF Presents preliminary analysis of questionl-like queries in MSN log

16. Papers vs. Workshop Goals - 1 Bernardi and Kirschner investigate Question 6 Leveling investigates Question 1 and Question 3 Momtazi and Klakow look at Question 5 Zhu et al. also look at Question 5 Small and Strzalkowski investigate Question 4

17. Papers vs. Workshop Goals - 2 Sutcliffe et al. look at Question 2 Mandl and Schulz also look at Question 3 Only Questions 7 and 8 are not addressed at the workshop!

18. Conclusions It looks like an interesting field We look forward to your papers There will be time at the end for discussion