1. Recommending Forum Posts to Designated Experts
Jason Hyun Duk Cho1,3, Yanen Li2, Roxana Girju1, Chengxiang Zhai1
1Department of Computer Science,
University of Illinois at Urbana-Champaign
2LinkedIn

2. Designated Experts in Online Domains
Rises in expert participations in forums
Examples include education (Coursera, Piazza), health (MedHelp), or legal (ask-a-lawyer).

3. Designated Experts in Online Domains
5.2 Million Students over 10,000 courses
First example is Coursera. Here, instructors answer questions that students may have, or fellow students answer questions. Notice not all of them are answered

4. Designated Experts in Online Domains
12 Million visitors per month
MedHelp has ‘Ask A Doctor’ forums where doctors respond to patients’ questions. Here, a patient asks a doctor what a ‘rapid cycler’ is.

5. Designated Experts in Online Domains
Rises in expert participations in forums
Examples include education (Coursera, Piazza), health (MedHelp), or legal (ask-a-lawyer).
We call people who have credentials ‘Designated Experts.’

6. Problem Number of users/questions overwhelm experts!
62.1% of online forums benefit from medical experts, but only 4.7% had responses from experts [1].
5.2M students over 532 courses on Coursera [2]
Each course has an average of 10,000 students

7. Solutions Hire more designated experts
Not very realistic
Model designated experts’ behaviors and route questions that they are most likely to answer.
Hiring more designated experts not realistic, and lower returns.

8. Approach Utilize existing framework
We used matrix factorization framework, by combining collaborative filtering, and encoding user/document information
Explore experts’ behaviors to improve recommendation performance
More on second point in the next slide…

9. Designated Expert Behavior
Most forum posts had either zero or one designated expert responses!
This was taken from MedHelp

10. Outline Experimental Setup Expert-document modeling
Document-word modeling
Expert-word modeling
Expert Similarities
Expert behavior constraints
Analysis

11. Experimental Setup
We used matrix factorization framework to run the experiments
Combination of collective matrix factorization and regularization
We used MedHelp’s Ask A Doctor forum for evaluation
56,194 threads across 18 forum categories
168 designated experts
Used stochastic gradient descent – parallelizable, so it can be used on big data

12. Framework We used matrix factorization to model the problem:
Matrices X and Y are low rank matrices. The goal is to infer k latent features.
These are often solved using SGD or Least squares

13. Expert-Document Model document-expert matrix
Where matrix C corresponds to weight of a given row and column
Sim(U,P) is cosine similarity
R corresponds to feedback matrix. It is set to 1 if an expert responded to a thread, 0 otherwise.

14. Document-Words Encode words in the objective function
Matrix D is modeled using TF-IDF weighting.

15. Evaluation Results Did not perform well
Does not capture experts’ preferences
Can we improve performance by adding words that experts prefer?

16. Expert-Words Encode words in the objective function
Matrix E is modeled using TF-IDF weighting.

17. Evaluation Results Performs significantly better than previously
Encoding both documents and expert profiles help tremendously
Can we do better?

18. Expert Similarity
There are not that much collaborative filtering going on
Vast majority of the posts have only one expert response.
Encode expert-expert similarity to mitigate this issue
Cosine similarity used for matrix S.

19. Evaluation Results Performs somewhat better.d
Performs somewhat better.
Can we explicitly encode experts’ behavior?

20. Constraint 1 – Propensity to answer
Some experts respond more than others
We should capture these characteristics
CS 440 – Introduction to AI Spring 2015 d

21. Constraint 2 – One expert per thread
Once an expert has answered a forum post, another expert is highly unlikely to respond to the post.
We still try to give a response to each forum

22. Evaluation Results
Adding the constraints improved the performance quite significantly

23. Objective Function There are lots of parameters to tune.
How sensitive is the algorithm to different parameters?

24. Sensitivity Analysis

25. Sensitivity Analysis

26. Sensitivity Analysis
Other than modeling expert-word matrices, algorithm was not very sensitive to parameters

27. Impact of Data Size
Study was conducted across all 18 forum categories. Circles indicate cases where the combined method performed better. Seems to consistently perform better. Notice we set parameters constant throughout the experiment

28. Impact of Data Size

29. Impact of Data Size
In all cases, combining all the method yielded the best performance in terms of MAP.
MAP chosen because it is standard

30. Conclusion
We introduced new type of experts called Designated Experts.
By utilizing the experts’ behavior, we can improve recommendation performance.
Our proposed algorithm was not sensitive to parameters, nor data size.
Designated experts – websites give such power.

31. Future Works
We would like to apply our algorithm on other domains, such as Coursera.
Modeling the interaction between experts and average users may be of interest.
We used Bag-of-words model. We would also like to model semantics of how experts talk, and see if they differ from average users.

32. Acknowledgements
We would like to thank the anonymous reviewers for their helpful comments. We would like to thank @WalmartLabs for partially funding this research.

33. Q&A
Thank you!

34. Appendix 0
Mean Average Precision (MAP)
Mean Reciprocal Rank (MRR)

35. Appendix 1
Stochastic Gradient Descent for inference

36. Appendix 2 In-depth explanation of the constraints
One expert per thread
Strict rule
NP complete problem – change from ILP to LP.