1. 1 Can People Collaborate to Improve the relevance of Search Results? Florian Eiteljörge eiteljoerge@stud.uni-hannover.de June 11, 2013Florian Eiteljörge

2. Outline Web search & social search techniques Phase one: Study setup & results Phase two: Study setup & results Discussion June 11, 2013Florian Eiteljörge

3. Web Search Search engines heavily used on internet studies indicate: 50% of web search sessions fail Idea: use social search techniques to improve web search June 11, 2013Florian Eiteljörge

4. Social search techniques Idea people search for something and give (implicit) feedback by clicking on result items Most clicked items seem to be mostly relevant – so they will be ranked higher next time. Problem users tend to click on the top result items popular sites get even more popular, even if there are new high-quality pages that would be more relevant ("rich-get-richer" phenomenon) June 11, 2013Florian Eiteljörge

5. What is the paper about? Authors had three hypothesis related to social search techniques: H1: Users will prefer to rate results at the top of the result lists, whether the results are randomized, or in the order that Google presents them. H2: Users explicit relevance rankings are not biased by the rank of the result list [while implicit feedback is biased] H3: For some types of queries people's collaborative effort can produce better ordering of search results. The authors developed a search engine environment to capture user respond by presenting Google's top ten results in randomized order to test the above hypothesis June 11, 2013Florian Eiteljörge

6. Study setup – phase one (rating) 145 participants were invited by mail to rate search results for their relevance participants had the possibility to rate any number of results of preselected queries in the most frequent categories (shopping, health, technology, business, computers, arts) participants were free to choose categories and queries they wanted to rate the result items were presented in random order Google-like result item layout relevance was measured on a 4-point scale: highly relevant, relevant, don’t know, not relevant after rating queries, each participant was asked to answer a short survey to determine how experience in searching affects the relevance perception June 11, 2013Florian Eiteljörge

7. Results June 11, 2013Florian Eiteljörge first bar: percent of selection of the item for rating second bar: percent of times when item was rated as highly relevant

8. Results – phase one June 11, 2013Florian Eiteljörge participants preferred to rate the first two items (H1 confirmed) participants explicit feedback not biased in general (H2 mostly confirmed) feedback for the first item is biased: rated highly relevant in 70% of the times (even if participants were told the order is randomized)

9. Study setup – phase two (evaluation) 20 participants were invited to choose if they prefer the results based on the explicit user-feedback or the Google-results the invited participants self-identified themselves as novice searchers both result-lists were displayed side-by-side the new ranking was created with the following formula: score = 3 x highly-relevant-count + 2 x relevant-count + don’t-know-count + (-1) x not-relevant-count June 11, 2013Florian Eiteljörge

10. Results – phase two in some categories the users rate result items very different from Google e.g.: shopping (digital cameras, walking shoes) – a mean difference in ranking of 4.2 in some categories users agree with the Google ranking e.g.: Business (Microsoft Bid for Yahoo, Online Advertisement) – a mean difference of 0.8 70% of the participants rated the user-based ordering higher than the Google-ordering; these participants chose to rate queries in the categories shopping, computers and arts the other 30% preferred the Google-ranking while choosing to rate queries of the categories business and technology June 11, 2013Florian Eiteljörge

11. Conclusion people prefer the top result items explicit feedback is not biased in general in some categories the Google-ranking is very inconsistent to the users ranking June 11, 2013Florian Eiteljörge

12. June 11, 2013Florian Eiteljörge Discussion

13. Presentation based on Morris MR, Horvitz E. SearchTogether: an interface for collaborative web search. Symposium on User Interface Software and Technology. 2007:3-12 http://www.grouplens.org/system/files/p283-agrahri.pdf http://www.grouplens.org/system/files/p283-agrahri.pdf June 11, 2013Florian Eiteljörge