1. Next Question Prediction
Tom Chao Zhou
Co-work with
Chin-Yew Lin, Irwin King, Young-In Song and Yunbo Cao
This work was done at Microsoft Research Asia.

2. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Outline
Introduction
Our Approach
Experiments
Conclusions
Future Work
Q & A
Tom Chao Zhou

3. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Introduction
Community-based Question Answering (CQA) service:
Ask questions, and answer other people’s questions.
E.g. Yahoo! Answers, Baidu Zhidao.
Why good? Returning explicit, self-contained answers.
Question Search:
Why? Time lag.
How? Finding semantically equivalent questions.
Limitation: it cannot guide the users to explore resources. that may interest him/her in a CQA service.
Tom Chao Zhou

4. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Introduction
Our thought:
A user posts a question in CQA because he/she is interested in a particular topic, but his/her current question may only capture one aspect of the topic.
It would be quite helpful to predict questions in this Table to guide the user to explore resources in CQA.
Tom Chao Zhou

5. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Introduction
Our thought:
CQA service is not an ideal source to learn the semantic relations among questions. No explicit relations between questions in CQA services.
Online forums
Highly interactive discussions of questions.
Questions posted by the thread starter are semantically related.
Forums contain much larger number of questions than CQA.
Tom Chao Zhou

6. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Introduction
discussion topic:
‘travel in orange beach’
Tom Chao Zhou

7. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Introduction
Next Question Prediction:
Definition:
Predict a ‘next’ question that is semantically or pragmatically related to the queried question.
Benefits:
Assisting the user in exploring different aspects of a topic tailored to his/her information needs.
Allowing the system to populate the question database before hand so to make it more complete.
Providing a relevance feedback mechanism for guided search.
1: Understand information needs.
Make a well-thought-out plan.
Reduce total time lag for a user to get a thorough knowledge.
Tom Chao Zhou

8. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Introduction
The system framework of next question prediction and a user case scenario.
Tom Chao Zhou

9. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Our Approach
Question Detection:
Problem:
Simple rules, such as ? and 5W1H words are not adequate for detecting question.
Questions can be expressed by imperative sentences.
Question marks are omitted.
Classification-based approach.
Mining labeled sequential patterns.
Discovered LSP forms a binary feature as the input for a classifier.
When where who what why how
Tom Chao Zhou

10. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Our Approach
Question Detection:
Mining labeled sequential patterns (LSP):
LSP: S->l, S is a sequence of patterns, and l is a class label.
s2=<b1,…,bn> contains a sequence s1=<a1,…,am> if there exists integers i1,…,im such that 1<=i1<…<im<=n and aj=bij.
A LSP p1 is contained by p2 if the sequence of p1 is contained by the sequence and they have the same class label.
sup(p) : support of p, percentage of tuples in a sequence database that contain the p.
conf(p): sup(p)/sup(p.S)
Classifier:
Each discovered LSP forms a binary feature as the input for a classification model.
Ripper: a rule based classification algorithm.
mine the LSPs by setting minimum support and minimum confidence
Tom Chao Zhou

11. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Our Approach
Translation Model-based Approach:
Translation model with Jelinek-Mercer smoothing:
q: queried question, D is candidate question, t is a query term in q, w is a term that appears in the collection, θC is the background collection, λ is a smoothing parameter, and pmle() is the maximum likelihood estimator.
Why we employ translation model?
It can bridge the lexical chasm between the queried question and candidate question.
The behavior of translation model is determined by how the word translation probabilities are obtained.
Tom Chao Zhou

12. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Our Approach
Learning word translation probabilities:
Questions are the focus of forum discussions.
Questions posted by the thread starter are very likely to explore different aspects of a discussion topic.
Words in these questions are semantically related.
Utilizing the co-occurrence properties of words that appear in semantically related questions may help find semantically related words.
Proposing an Association Rule Mining (ARM) based algorithm to explore word translation probabilities.
Tom Chao Zhou

13. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Our Approach
Learning word translation probabilities:
ARM-based algorithm.
Employ the ARM to mine the length 2 association rule: w->t
Use the confidence value of rule w->t to represent the word translation probability p(t|w) in the translation model.
Tom Chao Zhou

14. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Our Approach
TransLDA: Translation Model with LDA Ensemble
Translation model:
A fine-grained framework.
Limitation:
Only employing unigram level knowledge may suffer the weakness that the trained model is not robust against the noise.
Topic modeling:
Enhance document representation, has been applied in Ad-hoc retrieval before.
Target:
Next questions should explore different aspects of a discussion topic to the queried question.
Topic modeling view: the candidate questions should share similar topic distributions with the queried question.
Tom Chao Zhou

15. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Our Approach
TransLDA: Translation Model with LDA Ensemble
Latent Dirichlet Allocation (LDA):
Possessing fully generative semantics.
Motivation:
Both unigram level knowledge and latent factors should be considered in measuring semantic relations between questions.
TransLDA:
Incorporates LDA into the translation model.
Tom Chao Zhou

16. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Experiments
We ask ourselves three research questions:
RQ 1: How are the proposed methods compared with other approaches on labeled questions in next question prediction problem?
RQ 2: How are the proposed methods compared with other approaches on topic matching task under the next question prediction scenario?
RQ 3: How are the proposed methods compared with other approaches on joint topic distributions’ similarity with ground truth?
Tom Chao Zhou

17. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Experiments
Data Set:
Crawled from TripAdvisor, a popular travel forum.
Development Set: DEV SET
Unlabeled Testing Set: TST SET
Labeled Testing Set: TST LABEL SET
Training Set: TRAIN SET
Tom Chao Zhou

18. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Experiments
Data Set:
Statistics:
TRAIN SET:
1,976,522 questions extracted from 971,859 threads.
Tom Chao Zhou

19. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Experiments
Data Analysis:
Analyzing thread starter’s activities in TRAIN SET.
Over 40% of thread starters will reply on average 1.9 posts to the thread he or she initialized, indicating our expectation that forum discussions are quite interactive.
Over 68.8% thread starters do ask questions in their posts, and on average they ask 2 questions in their posts. Prove the motivation that question is a focus of forum discussions and forum data is an ideal source to train the proposed model.
Tom Chao Zhou

20. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Experiments
Data Analysis:
Analyzing thread starter’s activities in TRAIN SET.
Tom Chao Zhou

21. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Experiments
Data Preprocessing:
Stemming:
Porter Stemmer.
Stop word list:
Used by SMART system.
Document representation:
Unigram. (each word is a unit)
Metrics:
Precision at rank n, Precision at rank R, Mean Average Precision, Mean Reciprocal Rank, Bpref, Kullback-Leibler divergence, Top-k Recall.
Tom Chao Zhou

22. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Experiments
Comparison methods:
Cosine Similarity (COS):
Query Likelihood Language Model (QL):
Our methods:
Trans-ns1, Trans-ns2, Trans-ns3
TransLDA-ns1, TransLDA-ns2, TransLDA-ns3
ns is the neighborhood size parameter.
Tom Chao Zhou

23. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Experiments
Parameter Tuning:
Data Set:
DEV SET.
Metric:
MRR.
Method:
Exhaustive search.
Tuned parameters:
Dirichlet smoothing parameter in QL.
Translation model parameter.
TransLDA parameter.
Number of topics in LDA.
Tom Chao Zhou

24. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Experiments
Experiment on Labeled Question:
TST LABEL SET, 200 queried questions, 31,018 questions in the repository.
Tom Chao Zhou

25. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Experiments
Experiment on Topic Matching:
TST SET, 10,000 queried questions, 31,018 questions in question repository.
Given a query q, denoting the first next question of q in the actual thread as q’, and consider q’ to be relevant to q.
Function topic(X) returns the most probable topic that X belongs to.
For each method, return a ranked list of results {D1,D2,…Dn}, we denote Di to be relevant to q:
Tom Chao Zhou

26. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Experiments
Experiment on Topics’ Joint Probability Distribution:
TST SET, 10,000 queried questions, 31,018 questions in question repository.
For each q, we consider its first next question q’ in the thread as its relevant result. We aggregate the counts of topic transitions in the actual thread as ground truth, and apply maximum likelihood estimation to calculate topics’ joint probability (100*100 matrix):
For each method, we consider the first result as its predicted next question, and calculate topics’ joint probability distribution as the same way.
Tom Chao Zhou

27. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Conclusions
Defining the next question prediction problem.
Proposing an ARM-based algorithm to learn the word translation probabilities, and adapting it in to the translation model.
Proposing the TransLDA, which incorporates LDA into translation model.
Experimental results show that ARM-based algorithm to learn word translation probabilities is essential to translation model, and TransLDA outperforms alternative approaches.
Tom Chao Zhou

28. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Future Work
Conducting experiments on CQA services.
Investigating how to measure and how to diversify the predicted questions.
Query suggestion for long queries would also benefit from next question prediction.
Tom Chao Zhou

29. Your Question, Our passion!
Tom Chao Zhou

30. Tom Chao Zhou czhou@cse.cuhk.edu.hk
Thanks!
Tom Chao Zhou