Recommending Forum Posts to Designated Experts Jason Hyun Duk Cho1,3, Yanen Li2, Roxana Girju1, Chengxiang Zhai1 1Department of Computer Science, University of Illinois at Urbana-Champaign 2LinkedIn 3@WalmartLabs

Designated Experts in Online Domains Rises in expert participations in forums Examples include education (Coursera, Piazza), health (MedHelp), or legal (ask-a-lawyer).

Designated Experts in Online Domains 5.2 Million Students over 10,000 courses First example is Coursera. Here, instructors answer questions that students may have, or fellow students answer questions. Notice not all of them are answered

Designated Experts in Online Domains 12 Million visitors per month MedHelp has ‘Ask A Doctor’ forums where doctors respond to patients’ questions. Here, a patient asks a doctor what a ‘rapid cycler’ is.

Designated Experts in Online Domains Rises in expert participations in forums Examples include education (Coursera, Piazza), health (MedHelp), or legal (ask-a-lawyer). We call people who have credentials ‘Designated Experts.’

Problem Number of users/questions overwhelm experts! 62.1% of online forums benefit from medical experts, but only 4.7% had responses from experts [1]. 5.2M students over 532 courses on Coursera [2] Each course has an average of 10,000 students

Solutions Hire more designated experts Not very realistic Model designated experts’ behaviors and route questions that they are most likely to answer. Hiring more designated experts not realistic, and lower returns.

Approach Utilize existing framework We used matrix factorization framework, by combining collaborative filtering, and encoding user/document information Explore experts’ behaviors to improve recommendation performance More on second point in the next slide…

Designated Expert Behavior Most forum posts had either zero or one designated expert responses! This was taken from MedHelp

Outline Experimental Setup Expert-document modeling Document-word modeling Expert-word modeling Expert Similarities Expert behavior constraints Analysis

Experimental Setup We used matrix factorization framework to run the experiments Combination of collective matrix factorization and regularization We used MedHelp’s Ask A Doctor forum for evaluation 56,194 threads across 18 forum categories 168 designated experts Used stochastic gradient descent – parallelizable, so it can be used on big data

Framework We used matrix factorization to model the problem: Matrices X and Y are low rank matrices. The goal is to infer k latent features. These are often solved using SGD or Least squares

Expert-Document Model document-expert matrix Where matrix C corresponds to weight of a given row and column Sim(U,P) is cosine similarity R corresponds to feedback matrix. It is set to 1 if an expert responded to a thread, 0 otherwise.

Document-Words Encode words in the objective function Matrix D is modeled using TF-IDF weighting.

Evaluation Results Did not perform well Does not capture experts’ preferences Can we improve performance by adding words that experts prefer?

Expert-Words Encode words in the objective function Matrix E is modeled using TF-IDF weighting.

Evaluation Results Performs significantly better than previously Encoding both documents and expert profiles help tremendously Can we do better?

Expert Similarity There are not that much collaborative filtering going on Vast majority of the posts have only one expert response. Encode expert-expert similarity to mitigate this issue Cosine similarity used for matrix S.

Evaluation Results Performs somewhat better. d Performs somewhat better. Can we explicitly encode experts’ behavior?

Constraint 1 – Propensity to answer Some experts respond more than others We should capture these characteristics CS 440 – Introduction to AI Spring 2015 d

Constraint 2 – One expert per thread Once an expert has answered a forum post, another expert is highly unlikely to respond to the post. We still try to give a response to each forum

Evaluation Results Adding the constraints improved the performance quite significantly

Objective Function There are lots of parameters to tune. How sensitive is the algorithm to different parameters?

Sensitivity Analysis

Sensitivity Analysis

Sensitivity Analysis Other than modeling expert-word matrices, algorithm was not very sensitive to parameters

Impact of Data Size Study was conducted across all 18 forum categories. Circles indicate cases where the combined method performed better. Seems to consistently perform better. Notice we set parameters constant throughout the experiment

Impact of Data Size

Impact of Data Size In all cases, combining all the method yielded the best performance in terms of MAP. MAP chosen because it is standard

Conclusion We introduced new type of experts called Designated Experts. By utilizing the experts’ behavior, we can improve recommendation performance. Our proposed algorithm was not sensitive to parameters, nor data size. Designated experts – websites give such power.

Future Works We would like to apply our algorithm on other domains, such as Coursera. Modeling the interaction between experts and average users may be of interest. We used Bag-of-words model. We would also like to model semantics of how experts talk, and see if they differ from average users.

Acknowledgements We would like to thank the anonymous reviewers for their helpful comments. We would like to thank @WalmartLabs for partially funding this research.

Q&A Thank you!

Appendix 0 P@K Mean Average Precision (MAP) Mean Reciprocal Rank (MRR)

Appendix 1 Stochastic Gradient Descent for inference

Appendix 2 In-depth explanation of the constraints One expert per thread Strict rule NP complete problem – change from ILP to LP.