1. Basics of Information Retrieval - Focus: the Web Lillian N. Cassel February 2008 For CSC 2500 : Survey of Information Science A number of these slides are taken or adapted from Source: http://www.stanford.edu/class/cs276/cs276-2006-syllabus.html

2. Basic ideas  Information overload  The challenging byproduct of the information age  Huge amounts of information available -- how to find what you need when you need it  What kinds of information do you manage? What do you expect to find when you want/need it? How frequently do you access each kind?  Think about addresses, e-mail messages, files of interesting articles, product brochures, business cards, photos, charts and tables, etc.  Information retrieval is the formal study of efficient and effective ways to extract the right bit of information from a collection.  The web is a special case, as we will discuss.

3. Some distinctions  Data, information, knowledge  How do you distinguish among them?  Data: symbols  Information: data that are processed to be useful; provides answers to "who", "what", "where", and "when" questions  Knowledge: application of data and information; answers "how" questions  Understanding: appreciation of "why”  Wisdom: evaluated understanding. Russell Ackoff as reported in http://www.systems- thinking.org/dikw/dikw.htmhttp://www.systems- thinking.org/dikw/dikw.htm  What are we seeking when we search the web?

4. Organization  Information sources - organization:  Very well organized, indexed, controlled  Give some examples  Totally unorganized, uncharacterized, uncontrolled  Give some examples  Something in between  Give some examples

5. Databases  Databases hold specific data items  Organization is explicit  Keys relate items to each other  Queries are constrained, but effective in retrieving the data that is there  Databases generally respond to specific queries with specific results  Browsing is difficult  Searching for items not anticipated by the designers can be difficult  Give an example of a database with which you interact regularly.  What is a query that works easily?  Have you tried unsuccessfully to get information you know is there?

6. The Web  The Web contains many kinds of elements  Organization?  There are no keys to relate items to each other  Queries are unconstrained; effectiveness depends on the tools used.  Web queries generally respond to general queries with specific results  Browsing is possible, though somewhat complicated  There are no designers of the overall Web structure.  Describe how you frequently use the web  What works easily?  What has been difficult?

7. How high is Mt Everest The answer, with source -- not just a link to the place where an answer could be found

8. How fast is the Web growing? Source: http://news.netcraft.com/archives/web_server_survey.htmlhttp://news.netcraft.com/archives/web_server_survey.html An increase of 50 million sites during 2007

9. The servers What server software is providing content on the Web Source: http://news.netcraft.com/archives/web_server_survey.htmlhttp://news.netcraft.com/archives/web_server_survey.html

10. Digital Library  Something in between the very structured database and the unstructured Web.  Content is controlled. Someone makes the entries. (Maybe a lot of people make the entries, but there are rules for admission.)  Searching and browsing are somewhat open, not controlled by fixed keys and anticipated queries.  Nature of the collection regulates indexing somewhat.

11. Digital Library examples  American Memory: http://memory.loc.gov/ammem/index.html http://memory.loc.gov/ammem/index.html  The National Science Digital Library www.nsdl.org www.nsdl.org  CITIDEL: www.citidel.orgwww.citidel.org

12. In all cases  Trying to connect an information user to the specific information wanted.  Concerned with efficiency and effectiveness  Effectiveness - how well did we do?  Efficiency - how well did we use available resources?  We will focus on the Web, but some of the concepts apply in other situations also.

13. Effectiveness  Two measures:  Precision  Of the results returned, what percentage are meaningful to the goal of the query?  Recall  Of the materials available that match the query, what percentage were returned?  Ex. Search returns 590,000 responses and 195 are relevant. How well did we do?  Not enough information.  Did the 590,000 include all relevant responses? If so, recall is perfect. We have no way of knowing whether or not this is all the relevant resources What is important is whether the right response is there.  195/590,000 is not good precision!

14. The Web Web spider Indexer Search User Source: http://www.stanford.edu/class/cs276/cs276-2006-syllabus.html Basic structure of Web Search

15. What is involved in text retrieval

16. The process Query entered Query Interpreted Items retrieved Index searched Results Ranked

17. The Collection  Where does the collection come from?  How is the index created?  Those are important distinguishing characteristics  Inverted Index -- Ordered list of terms related to the collected materials. Each term has an associated pointer to the related material(s).  www.cs.cityu.edu.hk/~deng/5286/T51.doc www.cs.cityu.edu.hk/~deng/5286/T51.doc

18. An example: Unstructured data - circa 1650  Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia?  One could grep all of Shakespeare’s plays for Brutus and Caesar, then strip out lines containing Calpurnia?  Slow (for large corpora)  NOT Calpurnia is non-trivial  Other operations (e.g., find the word Romans near countrymen) not feasible  Ranked retrieval (best documents to return) Source: http://www.stanford.edu/class/cs276/cs276-2006-syllabus.html

19. A table showing incidence of the search terms in the documents 1 means the word is in the document What are the entries in the rows for the query: Brutus AND Caesar NOT Calpurnia Source: http://www.stanford.edu/class/cs276/cs276-2006-syllabus.html

20. The vectors  Brutus: 1 1 0 1 0 0  Caesar:1 1 0 1 1 1  Calpurnia:0 1 0 0 0 0  NOT Calpurnia:1 0 1 1 1 1  Brutus AND Caesar AND NOT Calpurnia:  AND them:1 0 0 1 0 0  Plays # 1 and 4 satisfy the query

21. The query terms in the documents Antony and Cleopatra, Act III, Scene ii Agrippa [Aside to DOMITIUS ENOBARBUS]: Why, Enobarbus, When Antony found Julius Caesar dead, He cried almost to roaring; and he wept When at Philippi he found Brutus slain. Hamlet, Act III, Scene ii Lord Polonius: I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. Source: http://www.stanford.edu/class/cs276/cs276-2006-syllabus.html

22. Inverted index  For each term T, we must store a list of all documents that contain T.  Do we use an array or a list for this? Brutus Calpurnia Caesar 12358132134 248163264128 1316 What happens if the word Caesar is added to document 14?

23. Inverted index  Linked lists generally preferred to arrays  Dynamic space allocation  Insertion of terms into documents easy  Space overhead of pointers Brutus Calpurnia Caesar 248163264128 2358132134 1316 1 Dictionary Postings lists Sorted by docID (more later on why). Posting

24. Inverted index construction Tokenizer Token stream. Friends RomansCountrymen Linguistic modules Modified tokens. friend romancountryman Indexer Inverted index. friend roman countryman 24 2 13 16 1 More on these later. Documents to be indexed. Friends, Romans, countrymen.

25.  Sequence of (Modified token, Document ID) pairs. I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. Doc 1 So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious Doc 2 Indexer steps

26.  Sort by terms. Core indexing step.

27.  Multiple term entries in a single document are merged.  Frequency information is added. Why frequency? Will discuss later.

28.  The result is split into a Dictionary file and a Postings file.

29.  Where do we pay in storage? Pointers Terms

30. Query processing: AND  Consider processing the query: Brutus AND Caesar  Locate Brutus in the Dictionary;  Retrieve its postings.  Locate Caesar in the Dictionary;  Retrieve its postings.  “Merge” the two postings: 128 34 248163264123581321 Brutus Caesar

31. 34 12824816 3264 12 3 581321 The merge  Walk through the two postings simultaneously, in time linear in the total number of postings entries 128 34 248163264123581321 Brutus Caesar 2 8 If the list lengths are x and y, the merge takes O(x+y) operations. Crucial: postings sorted by docID.

32. Boolean queries: Exact match  The Boolean Retrieval model is being able to ask a query that is a Boolean expression:  Boolean Queries are queries using AND, OR and NOT to join query terms  Views each document as a set of words  Is precise: document matches condition or not.  Primary commercial retrieval tool for 3 decades.  Professional searchers (e.g., lawyers) still like Boolean queries:  You know exactly what you’re getting.

33. Example: WestLaw http://www.westlaw.com/  Largest commercial (paying subscribers) legal search service (started 1975; ranking added 1992)  Tens of terabytes of data; 700,000 users  Majority of users still use boolean queries  Example query:  What is the statute of limitations in cases involving the federal tort claims act?  LIMIT! /3 STATUTE ACTION /S FEDERAL /2 TORT /3 CLAIM  /3 = within 3 words, /S = in same sentence

34. Example: WestLaw http://www.westlaw.com/  Another example query:  Requirements for disabled people to be able to access a workplace  disabl! /p access! /s work-site work-place (employment /3 place  Note that SPACE is disjunction, not conjunction!  Long, precise queries; proximity operators; incrementally developed; not like web search  Professional searchers often like Boolean search:  Precision, transparency and control  But that doesn’t mean they actually work better….

35. Boolean queries: More general merges  Exercise: Adapt the merge for the queries: Brutus AND NOT Caesar Brutus OR NOT Caesar

36. Query optimization  What is the best order for query processing?  Consider a query that is an AND of t terms.  For each of the t terms, get its postings, then AND them together. Brutus Calpurnia Caesar 12358162134 248163264128 1316 Query: Brutus AND Calpurnia AND Caesar

37. Query optimization example  Process in order of increasing freq:  start with smallest set, then keep cutting further. Brutus Calpurnia Caesar 12358132134 248163264128 1316 This is why we kept freq in dictionary Execute the query as (Caesar AND Brutus) AND Calpurnia.

38. What’s ahead in IR? Beyond term search  What about phrases?  Stanford University  Proximity: Find Gates NEAR Microsoft.  Need index to capture position information in docs. More later.  Zones in documents: Find documents with (author = Ullman) AND (text contains automata).

39. Evidence accumulation  1 vs. 0 occurrence of a search term  2 vs. 1 occurrence  3 vs. 2 occurrences, etc.  Usually more seems better  Need term frequency information in docs

40. Ranking search results  Boolean queries give inclusion or exclusion of docs.  Often we want to rank/group results  Need to measure proximity from query to each doc.  Need to decide whether docs presented to user are singletons, or a group of docs covering various aspects of the query.

41. IR vs. databases: Structured vs unstructured data  Structured data tends to refer to information in “tables” EmployeeManagerSalary SmithJones50000 ChangSmith60000 50000IvySmith Typically allows numerical range and exact match (for text) queries, e.g., Salary < 60000 AND Manager = Smith.

42. Unstructured data  Typically refers to free text  Allows  Keyword queries including operators  More sophisticated “concept” queries e.g.,  find all web pages dealing with drug abuse  Classic model for searching text documents

43. Semi-structured data  In fact almost no data is “unstructured”  E.g., this slide has distinctly identified zones such as the Title and Bullets  Facilitates “semi-structured” search such as  Title contains data AND Bullets contain search … to say nothing of linguistic structure

44. More sophisticated semi- structured search  Title is about Object Oriented Programming AND Author something like stro*rup  where * is the wild-card operator  Issues:  how do you process “about”?  how do you rank results?  The focus of XML search.

45. Clustering and classification  Given a set of docs, group them into clusters based on their contents.  Given a set of topics, plus a new doc D, decide which topic(s) D belongs to.

46. The web and its challenges  Unusual and diverse documents  Unusual and diverse users, queries, information needs  Beyond terms, exploit ideas from social networks  link analysis, clickstreams...  How do search engines work? And how can we make them better?

47. More sophisticated information retrieval  Cross-language information retrieval  Question answering  Summarization  Text mining  …

48. Crawling the web  Misnomer as the spider or robot does not actually move about the web  Program sends a normal request for the page, just as a browser would.  Retrieve the page and parse it.  Look for anchors -- pointers to other pages. Put them on a list of URLs to visit  Extract key words (possibly all words) to use as index terms related to that page  Take the next URL and do it again  Actually, the crawling and processing are parallel activities

49. Responding to search queries  Use the query string provided  Form a boolean query  Join all words with AND? With OR?  Find the related index terms  Return the information available about the pages that correspond to the query terms.  Many variations on how to do this. Usually proprietary to the company.

50. Making the connections  Stemming  Making sure that simple variations in word form are recognized as equivalent for the purpose of the search: exercise, exercises, exercised, for example.  Indexing  A keyword or group of selected words  Any word (more general)  How to choose the most relevant terms to use as index elements for a set of documents.  Build an inverted file for the chosen index terms.

51. The Vector model  Let  N be the total number of documents in the collection  n i be the number of documents which contain k i  freq(i,j) raw frequency of k i within d j  A normalized tf (term frequency) factor is given by  tf(i,j) = freq(i,j) / max(freq(i,j))  where the maximum is computed over all terms which occur within the document d j  The idf (index term frequency) factor is computed as  idf(i) = log (N/n i )  the log is used to make the values of tf and idf comparable. It can also be interpreted as the amount of information associated with the term k i.

52. Anatomy of a web page  Metatags: Information about the page  Primary source of indexing information for a search engine.  Ex. Title. Never mind what has an H1 tag (though that may be considered), what is in the brackets?  Other tags provide information about the page. This is easier for the search engine to use than determining the meaning of the text of the page.  Dealing with the cheaters  False information provided in the web page to make the search engine return this page  False metatags, invisible words (repeated many times), etc

53. Standard Metatags  The Dublin Core (http://dublincore.org/)http://dublincore.org/ 15 common items to use in labeling any web document TitleContributorSource CreatorDateLanguage Subject Resources typeRelation DescriptionFormatCoverage PublisherIdentifierRights

54. Hubs and authorities  Hub points to a lot of other places.  CITIDEL is a hub for computing information  NSDL is a hub for science, technology, engineering and mathematics education.  Authorities are pointed to by a lot of other places.  W3C.org is an authority for information about the web.  When Hub or Authority status is captured, the search can be more accurate.  If several pages match a query, and one is an authority page, it will be ranked higher.  When a hub matches a query, the pages it points to are likely to be relevant.

55. An exercise  We have a document collection (sort of)  We build an index so that we can process a query  Do this:  Take the list of documents  Each group go through the items assigned and list every word that appears and the number of times the word appears. Use stemming. Only count nouns.  We will then combine these lists and pick the 20 most frequent words to use as the index. Make the inverted list (file index) corresponding to each of the 20 words.  Now, suppose we have the query I will give you. Which documents in our collection satisfy the query? Given our 20 words list, how would we match that to our query?

56. Conclusions  The plan was to introduce the basic concepts of information retrieval in a form accessible to most students.  There is a lot more, but this gives some flavor of the way the systems work and may help you use them more effectively.  A word about the pattern for these slides …

57. Extra references  Data, Information, Knowledge, and Wisdom by Gene Bellinger, Durval Castro, Anthony Mills: http://www.systems-thinking.org/dikw/dikw.htm http://www.systems-thinking.org/dikw/dikw.htm  Some slides from the course CS276 Information Retrieval and Web Mining - Standford University: http://www.stanford.edu/class/cs276/cs276-2006-syllabus.html http://www.stanford.edu/class/cs276/cs276-2006-syllabus.html

58. http://infolab.stanford.edu/~backrub/google.html http://www.seochat.com/c/a/Search-Engine-Optimization-Help/Search-Engines-and-Algorithms- Semantic-Search/ http://www.columbia.edu/~js1353/pubs/wolf-www02.pdf http://mics.cs.uwec.edu/CD%20image/papers/paper89.pdf http://www.engl.niu.edu/ehoffman/search_engines.html http://moon.ouhsc.edu/kboyce/sdms/otherlinks/spidap4.htm http://www.cse.lehigh.edu/~brian/course/sem/notes/searchengines.pdf http://www9.org/w9cdrom/159/159.html http://www9.org/w9cdrom/293/293.html http://perso.fundp.ac.be/~lgoffine/Hypertext/semantic_links.html http://dbpubs.stanford.edu:8090/pub/2002-6 http://searchenginewatch.com/showPage.html?page=2167891 http://searchenginewatch.com/showPage.html?page=2168031 http://www.webreference.com/content/search/ http://www.onlamp.com/pub/a/onlamp/2003/08/21/better_search_engine.html (this site has other related articles links) http://www.webopedia.com/DidYouKnow/Internet/2003/HowWebSearchEnginesWork.asp http://art-support.com/seo_algorithms.htm http://computer.howstuffworks.com/search-engine.htm Some additional articles on Web search. Note the first one.