1. KDD 2011 Research Poster Content - Driven Trust Propagation Framwork V. G. Vinod Vydiswaran, ChengXiang Zhai, and Dan Roth University of Illinois at Urbana-Champaign Incorporating text in trust models Model parameters Can you trust news stories?  Even reputed sources make mistakes.  Not all claims made by a source is equally trustworthy.  Some claims are purposefully misleading.  How to verify free-text claims? Acknowledgments This research was supported by the Multimodal Information Access and Synthesis (MIAS) Center at the University of Illinois at Urbana-Champaign, part of CCICADA, a DHS Science and Technology Center of Excellence, and grants from the Army Research Laboratory under agreement W911NF-09-2-0053. Contact details vgvinodv@illinois.edu, czhai@cs.uiuc.edu, danr@illinois.edu Claim 1 Claim n Claim 2...... EvidenceClaims Sources Web sources Evidence passages Claim sentences  Incorporates semantics in trust computation using evidence.  Claims need not be structured tuples – they can be free-text sentences.  Framework does not assume that accurate Information Extraction is available.  A source can have different trust profile for different claims – not all claims from a source get equal weight. Advantages over traditional models Traditional two-layer fact-finder models Claim 1 Claim n Claim 2 … [Yin, et al., 2007; Pasternack & Roth, 2010]  Computed scores : Claim veracity : Evidence trust : Source trust  Influence factors : Evidence similarity : Relevance : Source - Evidence influence Iterative formulation #TopicRetrievalTwo-stage models Our model 1Healthcare0.8860.8950.932 2Obama administration0.8520.8760.927 3Bush administration0.9310.9210.971 4Democratic policy0.8940.7690.922 5Republican policy0.7740.8480.936 6Immigration0.8200.9520.983 7Gay rights0.8320.8640.807 8Corruption0.8740.8410.941 9Election reform0.8640.8890.908 10WikiLeaks0.8860.8600.825 Average0.8610.8690.915 +6.3% Relative  There is a need to determine the truth value of a claim.  This value depends on its source as well as on evidence. Evidence documents influence each other and have different relevance to claims.  We developed a trust propagation framework that associates relevant evidence to claims and sources.  Global analysis of this data, taking into account relations between the stories, their relevance and their sources allows us to make progress in determining trustworthiness values over sources and claims.  Experiments with news trustworthiness show promising results on incorporating evidence in trustworthiness computation and improving “credibility” of retrieved results. Conclusions Data characteristics Experimental results D. Using trust model to boost evidence retrieval C. Does it depend on news genres? A. Computing trust scores and trusted sources for specific claim topics B. Finding trustworthy news sources and news reporters  Model brings credible documents to the top of the result list  Improvement in NDCG scores statistically significant.  Model helps bring out the disparity in credibility of reporting on specific topics  Model scores show influence of both popularity and average rating of articles.  Specific news sources appear to be trusted more for specific news genres.  23,164 news articles from 23 genres collected from Politics category of NewsTrust.org  All news articles were rated by human volunteers based on journalistic principles  Scored in the range [1,5], mean 3.70  Investigative reports most trusted (4.10), Advertisements least (2.43) Veracity of news reporting Trustworthiness of news stories Credibility of news sources