Information Retrieval Effectiveness of Folksonomies on the World Wide Web P. Jason Morrison

Information retrieval (IR) on the Web Traditionally, there are 2 options: 1. Search Engines – documents added to collection automatically, full text searching using some algorithm; 2. Subject Directories – documents collected and organized into a hierarchy or taxonomy by experts. Many sites now use a new system: 3. Folksonomies – documents collected and tagged with keywords by all users, brought together into a loose organizational system.

Folksonomies Very little empirical study has been done on Folksonomies. Used by social bookmarking sites like Del.icio.us, photography sites like Flickr, and video sites like YouTube. Even large, established retailers like Amazon are starting to experiment with tagging.

Research Questions: 1. Do web sites that employ folksonomies return relevant results to users performing information retrieval tasks, specifically searching? 2. Do folksonomies perform as well as subject directories and search engines?

Hypotheses: 1. Despite different index sizes and categorization strategies, the top results from search engines, directories, and folksonomies will show some overlap. Items that show up in the results of more than one will be more likely to be judged. 2. There will be significant difference between the IR effectiveness of search engines, expert-maintained directories, and folksonomies. 3. Folksonomies will perform as well or better than search engines and directories for information needs that fall into entertainment or current event categories. They will perform less well for factual or specific-document searches.

Gordon and Pathak’s (1999) Seven Features: 1. Searches should use real information needs 2. Studies should try to capture the information need, not just the query used, if possible 3. A large enough number of searches must be done to do a meaningful evaluation. 4. Most major search engines should be included 5. The special features of each engine should be utilized. 6. Relevance should be judged by the person with the information need.

Gordon and Pathak’s Seven Features, cont: 7) Experiments need to be conducted so they provide meaningful measures: Good experimental design, such as returning results in a random order; Use of accepted IR measurements like Recall and Precision; Use of appropriate statistical tests.

Hawking, et al.’s (2001) additional feature: 8) Search topics should include different types of information needs Four different types based on the desired results: 1. A short factual statement that directly answers a question; 2. A specific document or web site that the user knows or suspects exists; 3. A selection of documents that pertain to an area of interest; or 4. An exhaustive list of every document that meets their need. (

Leighton and Srivastava (1997) Gordon and Pathak (1999) Hawking et al (2001)Can et al (2003) The Present Study Information Needs Provided by Library reference desk, other studies Faculty membersQueries from web logsComputer Science Students and Professors Graduate students Queries Created by The researchers Skilled searchersQueries from web logsSame Relevance Judged by The researchers (by consensus) Same faculty members Research AssistantsSame Participants233 Faculty members Total queries

Leighton and Srivastava (1997) Gordon and Pathak (1999) Hawking et al (2001)Can et al (2003) The Present Study Engines tested Results evaluated per engine 20 Total results evaluated / evaluator: or 320About 160 Relevancy Scale 4 categories4-point scaleBinary Precision Measures: P(20), weighted groups by rank P(1-5), P(1-10), P(5-10), P(15-20) P(1), P(1-5), P(5) P(20) P(10), P(20) P(20), P(1-5) Recall Measures: noneRelative recall; R(15-20), R(15-25), R(40-60), R( ), R( ) noneRelative recall: R(10), R(20) Relative recall: R(20), R(1-5)

IR systems studied Two directories: Open Directory and Yahoo. Three search engines: Alta Vista, Live (Microsoft), and Google. Three social bookmarking systems representing the folksonomies: Del.icio.us, Furl, and Reddit.

General results 34 users, 103 queries and 9266 total results returned. The queries generated by participants were generally similar to previous studies in terms of word count and use of operators. Previous studies of search engine logs have shown that users rarely try multiple searches and rarely look past the first set or results. This fits the current study. For many queries, some IR systems did not return the full 20 results. In fact there were many queries where some IR systems returning 0 results.

Hypothesis 1: Overlap in results Number of engines returning the URL Number of unique results Relevancy rateSD Total

IR system type combination Engine types returning same URLNMean DirectoryFolksonomySearch Engine no yes noyesno yesno noyes yesnoyes yes no yes Total

Overlap of results findings Almost 90% of results were returned by just one engine – fits well with previous studies. Results found by both search engines and folksonomies were significantly more likely to be relevant The directory/search engine group had a higher relevancy rate than the folksonomy/search engine group, but the difference was not significant. Allowing tagging or meta-searching a folksonomy could improve search engine performance. Hypothesis 1 is supported.

Hypothesis 2: Performance differences Performance measures: Precision Relative Recall Retrieval Rate also calculated

Performance (dcv 20) IR SystemPrecisionRecallRetrieval Rate Open DirectoryMean N Yahoo DirectoryMean N Del.icio.usMean N FurlMean N RedditMean N GoogleMean N LiveMean N Alta VistaMean N TotalMean N

Precision at positions 1-20

Recall at positions 1-20

Average performance at dcv 1-5 IR System TypeAvg PrecisionAvg Recall Avg Retrieval Rate DirectoryMean N FolksonomyMean N Search EngineMean N

Performance differences findings There are statistically significant differences among individual IR systems and IR system types. Search engines had the best performance by all measures. In general directories had better precision than folksonomies, but difference not usually statistically significant. Del.icio.us performed as well or better than the directories. Hypothesis 2 is supported.

Hypothesis 3: Performance for different needs Do Folksonomies perform better than the other IR systems for some information needs, and worse for others?

Comparing information need categories Info Need Category IR System Type Avg Precision Avg RecallAvg Retrieval Short Factual Answer DirectoryMean N1228 FolksonomyMean N2842 Search Engine Mean N4042 Specific Item DirectoryMean N FolksonomyMean N Search Engine Mean N Selection of Relevant Items DirectoryMean N FolksonomyMean N Search Engine Mean N

News and entertainment searches Information Need IR System Type Avg Precision Avg RecallRetrieval Rate NewsDirectory Mean N44042 Folksonomy Mean N Search Engine Mean N EntertainmentDirectory Mean N61618 Folksonomy Mean N Search Engine Mean N252427

Factual and exact site searches Information Need IR System Type Avg Precision Avg RecallRetrieval Rate FactualDirectory Mean N1228 Folksonomy Mean N2842 Search Engine Mean N4042 Exact SiteDirectory Mean N Folksonomy Mean N Search Engine Mean N615763

Performance for different info needs findings Significant differences were found among folksonomies, search engines, and directories for the three info need categories. When comparing within info need categories, the search engines had significantly better precision. Recalls scores were similar but not significant. Folksonomies did not perform significantly better for news and entertainment searches; but They did perform significantly worse than search engines for factual and exact site searches. Hypothesis 3 only partly supported.

What other factors impacted performance? For the study as a whole, the use of query operators correlated negatively with recall and retrieval rate. Non-boolean operators correlated negatively with precision scores. When looking at just folksonomy searches, query operator use lead to even lower recall and retrieval scores. Some specific cases were not handled by the folksonomies. A search for movie show times at a certain zip code (“showtimes borat”) had zero results on all folksonomies. Queries that were limited by geography and queries with obscure topics can perform poorly in folksonomies because users might not have added/tagged items yet.

User factors For the most part, user experience did not correlate significantly with performance measures. Expert users were more likely to have lower precision scores. Same correlation found when correcting for query factors Experienced users probably less likely to deem something relevant.

Recommendations Further research is needed Additional folksonomies should be studied as well. It might be useful to collect additional types of data, such as whether or not participants clicked through to look at sites before judging. Additional analysis on ranking would be interesting. Any similar study must also deal with difficult technical issues like server and browser timeouts.

Conclusions The overlap between folksonomy results and search engine results could be used to improve Web IR performance. The search engines, with their much larger collections, performed better than directories and folksonomies in almost every case. Folksonomies may be better than directories for some needs, but more data is required. Folksonomies are particularly bad at finding a factual answer or one specific site.

Conclusions (cont.) Although search engines had better performance across the board, folksonomies are promising because: 1.They are relatively new and may improve with time and additional users; 2.Search results could be improved with relatively small changes to the way query operators and search terms are used. 3.There are many variations in organization to be tried.

Future research Look at the difference between systems that primarily use tagging (Del.icio.us, Furl) and those that use ranking (Reddit, Digg) Which variations are more successful? Tags, titles, categories, descriptions, comments, and even full text are collected by various folksonomies. Where should weight be placed? Should a document that matches the query closely rank higher than one with many votes, or vice versa?

Future research (cont.) Artificial situations could be set up to study absolute recall and searches for an exhaustive list of items. Similar studies on IR systems covering smaller domains, like video, should be done. Blog search systems in particular would be interesting. What about other IR behaviors such as browsing? There are many other fascinating topics such as the social networks in some folksonomies and what motivates users to tag items among others.