1. Information Retrieval
September 19, 2005
Information Retrieval
Dragomir R. Radev
University of Michigan
(C) 2005, The University of Michigan

2. About the instructor Dragomir R. Radev
Associate Professor, University of Michigan
School of Information
Department of Electrical Engineering and Computer Science
Department of Linguistics
Head of CLAIR (Computational Linguistics And Information Retrieval) at U. Michigan
Treasurer, North American Chapter of the ACL
Ph.D., 1998, Computer Science, Columbia University
Home page:
(C) 2005, The University of Michigan

3. Introduction
(C) 2005, The University of Michigan

4. IR systems Google Vivísimo AskJeeves NSIR Lemur MG Nutch
(C) 2005, The University of Michigan

5. Examples of IR systems
Conventional (library catalog). Search by keyword, title, author, etc.
Text-based (Lexis-Nexis, Google, FAST). Search by keywords. Limited search using queries in natural language.
Multimedia (QBIC, WebSeek, SaFe) Search by visual appearance (shapes, colors,… ).
Question answering systems (AskJeeves, NSIR, Answerbus) Search in (restricted) natural language
(C) 2005, The University of Michigan

6. (C) 2005, The University of Michigan

7. (C) 2005, The University of Michigan

8. Need for IR
Advent of WWW - more than 8 Billion documents indexed on Google
How much information? 200TB according to Lyman and Varian 2003.
Search, routing, filtering
User’s information need
(C) 2005, The University of Michigan

9. Some definitions of Information Retrieval (IR)
Salton (1989): “Information-retrieval systems process files of records and requests for information, and identify and retrieve from the files certain records in response to the information requests. The retrieval of particular records depends on the similarity between the records and the queries, which in turn is measured by comparing the values of certain attributes to records and information requests.” Kowalski (1997): “An Information Retrieval System is a system that is capable of storage, retrieval, and maintenance of information. Information in this context can be composed of text (including numeric and date data), images, audio, video, and other multi-media objects).”
(C) 2005, The University of Michigan

10. Sample queries (from Excite)
In what year did baseball become an offical sport?
play station codes . com
birth control and depression
government
"WorkAbility I"+conference
kitchen appliances
where can I find a chines rosewood
tiger electronics
58 Plymouth Fury
How does the character Seyavash in Ferdowsi's Shahnameh exhibit characteristics of a hero?
emeril Lagasse
Hubble
M.S Subalaksmi
running
(C) 2005, The University of Michigan

11. Mappings and abstractions
Reality
Data
Information need
Query
From Korfhage’s book
(C) 2005, The University of Michigan

12. Typical IR system (Crawling) Indexing Retrieval User interface
(C) 2005, The University of Michigan

13. Key Terms Used in IR
QUERY: a representation of what the user is looking for - can be a list of words or a phrase.
DOCUMENT: an information entity that the user wants to retrieve
COLLECTION: a set of documents
INDEX: a representation of information that makes querying easier
TERM: word or concept that appears in a document or a query
(C) 2005, The University of Michigan

14. Documents
(C) 2005, The University of Michigan

15. Documents Not just printed paper collections vs. documents
data structures: representations
Bag of words method
document surrogates: keywords, summaries
encoding: ASCII, Unicode, etc.
(C) 2005, The University of Michigan

16. Document preprocessing
Formatting
Tokenization (Paul’s, Willow Dr., Dr. Willow, , New York, ad hoc)
Casing (cat vs. CAT)
Stemming (computer, computation)
Soundex
(C) 2005, The University of Michigan

17. Document representations
Term-document matrix (m x n)
term-term matrix (m x m x n)
document-document matrix (n x n)
Example: 3,000,000 documents (n) with 50,000 terms (m)
sparse matrices
Boolean vs. integer matrices
(C) 2005, The University of Michigan

18. Document representations
Term-document matrix
Evaluating queries (e.g., (AB)C)
Storage issues
Inverted files
Evaluating queries
Advantages and disadvantages
(C) 2005, The University of Michigan

19. IR models
(C) 2005, The University of Michigan

20. Major IR models Boolean Vector Probabilistic Language modeling
Fuzzy retrieval
Latent semantic indexing
(C) 2005, The University of Michigan

21. Major IR tasks Ad-hoc Filtering and routing Question answering
Spoken document retrieval
Multimedia retrieval
(C) 2005, The University of Michigan

22. Venn diagrams z x w y D1 D2
(C) 2005, The University of Michigan

23. Boolean model A B
(C) 2005, The University of Michigan

24. Boolean queries What types of documents are returned? Stemming
restaurants AND (Mideastern OR vegetarian) AND inexpensive
What types of documents are returned?
Stemming
thesaurus expansion
inclusive vs. exclusive OR
confusing uses of AND and OR
dinner AND sports AND symphony
4 OF (Pentium, printer, cache, PC, monitor, computer, personal)
(C) 2005, The University of Michigan

25. Boolean queries Weighting (Beethoven AND sonatas) precedence
coffee AND croissant OR muffin
raincoat AND umbrella OR sunglasses
Use of negation: potential problems
Conjunctive and Disjunctive normal forms
Full CNF and DNF
(C) 2005, The University of Michigan

26. Transformations De Morgan’s Laws: CNF or DNF?
NOT (A AND B) = (NOT A) OR (NOT B)
NOT (A OR B) = (NOT A) AND (NOT B)
CNF or DNF?
Reference librarians prefer CNF - why?
(C) 2005, The University of Michigan

27. Boolean model Partition Partial relevance?
Operators: AND, NOT, OR, parentheses
(C) 2005, The University of Michigan

28. Exercise D1 = “computer information retrieval”
D2 = “computer retrieval”
D3 = “information”
D4 = “computer information”
Q1 = “information  retrieval”
Q2 = “information  ¬computer”
(C) 2005, The University of Michigan

29. Exercise 1
Swift 2
Shakespeare 3 4
Milton 5 6 7 8
Chaucer 9 10 11 12 13 14 15
((chaucer OR milton) AND (NOT swift)) OR ((NOT chaucer) AND (swift OR shakespeare))
(C) 2005, The University of Michigan

30. Stop lists
most common words in English account for 50% or more of a given text.
Example: “the” and “of” represent 10% of tokens. “and”, “to”, “a”, and “in” - another 10%. Next 12 words - another 10%.
Moby Dick Ch.1: 859 unique words (types), 2256 word occurrences (tokens). Top 65 types cover 1132 tokens (> 50%).
Token/type ratio: 2256/859 = 2.63
(C) 2005, The University of Michigan

31. Vector models Term 1 Doc 1 Doc 2 Term 3 Doc 3 Term 2
(C) 2005, The University of Michigan

32. Vector queries Each document is represented as a vector
non-efficient representations (bit vectors)
dimensional compatibility
(C) 2005, The University of Michigan

33. The matching process Document space
Matching is done between a document and a query (or between two documents)
distance vs. similarity
Euclidean distance, Manhattan distance, Word overlap, Jaccard coefficient, etc.
(C) 2005, The University of Michigan

34. Miscellaneous similarity measures
The Cosine measure
 (di x qi)
|X  Y|
 (D,Q) = =
|X| * |Y|
 (di)2 *
 (qi)2
The Jaccard coefficient
|X  Y|
 (D,Q) =
|X  Y|
(C) 2005, The University of Michigan

35. Exercise
Compute the cosine measures  (D1,D2) and  (D1,D3) for the documents: D1 = <1,3>, D2 = <100,300> and D3 = <3,1>
Compute the corresponding Euclidean distances, Manhattan distances, and Jaccard coefficients.
(C) 2005, The University of Michigan

36. Evaluation
(C) 2005, The University of Michigan

37. Relevance Difficult to change: fuzzy, inconsistent
Methods: exhaustive, sampling, pooling, search-based
(C) 2005, The University of Michigan

38. Contingency table retrieved not retrieved relevant w x n1 = w + x
not relevant y z N
n2 = w + y
(C) 2005, The University of Michigan

39. Precision and Recall w w+x w w+y Recall: Precision:
(C) 2005, The University of Michigan

40. Exercise
Go to Google ( and search for documents on Tolkien’s “Lord of the Rings”. Try different ways of phrasing the query: e.g., Tolkien, “JRR Melville”, +”JRR Tolkien” +Lord of the Rings”, etc. For each query, compute the precision (P) based on the first 10 documents returned by AltaVista.
Note! Before starting the exercise, have a clear idea of what a relevant document for your query should look like. Try different information needs.
Later, try different queries.
(C) 2005, The University of Michigan

41. [From Salton’s book]
(C) 2005, The University of Michigan

42. Interpolated average precision (e.g., 11pt)
Interpolation – what is precision at recall=0.5?
(C) 2005, The University of Michigan

43. Issues Why not use accuracy A=(w+z)/N? Average precision
Average P at given “document cutoff values”
Report when P=R
F measure: F=(b2+1)PR/(b2P+R)
F1 measure: F1 = 2/(1/R+1/P) : harmonic mean of P and R
(C) 2005, The University of Michigan

44. Kappa N: number of items (index i) n: number of categories (index j)
k: number of annotators
(C) 2005, The University of Michigan

45. Kappa example (from Manning, Schuetze, Raghavan)
J1+
J1-
J2+
300 10
J2- 20 70
(C) 2005, The University of Michigan

46. Kappa (cont’d) P(A) = 370/400 P (-) = (10+20+20+70)/800 = 0.2125
P (E) = * * = 0.665
K = ( )/( ) = 0.776
Kappa higher than 0.67 is tentatively acceptable; higher than 0.8 is good
(C) 2005, The University of Michigan

47. Relevance collections
TREC ad hoc collections, 2-6 GB
TREC Web collections, 2-100GB
(C) 2005, The University of Michigan

48. Sample TREC query <top> <num> Number: 305
<title> Most Dangerous Vehicles
<desc> Description:
Which are the most crashworthy, and least crashworthy,
passenger vehicles?
<narr> Narrative:
A relevant document will contain information on the crashworthiness of a given vehicle or vehicles that can be used to draw a comparison with other vehicles. The document will have to describe/compare vehicles, not drivers. For instance, it should be expected that vehicles preferred by year-olds would be involved in more crashes, because that age group is involved in more crashes. I would view number of fatalities per 100 crashes to be more revealing of a vehicle's crashworthiness than the number of crashes per 100,000 miles, for example.
</top> LA FT LA LA LA LA FT LA FT LA LA FT LA
LA LA LA LA LA LA LA LA LA FT LA LA
(C) 2005, The University of Michigan

49. (C) 2005, The University of Michigan
<DOCNO> LA </DOCNO>
<DOCID> </DOCID>
<DATE><P>March 16, 1989, Thursday, Home Edition </P></DATE>
<SECTION><P>Business; Part 4; Page 1; Column 5; Financial Desk </P></SECTION>
<LENGTH><P>586 words </P></LENGTH>
<HEADLINE><P>AGENCY TO LAUNCH STUDY OF FORD BRONCO II AFTER HIGH RATE OF ROLL-OVER ACCIDENTS </P></HEADLINE>
<BYLINE><P>By LINDA WILLIAMS, Times Staff Writer </P></BYLINE>
<TEXT>
<P>The federal government's highway safety watchdog said Wednesday that the Ford Bronco II appears to be involved in more fatal roll-over
accidents than other vehicles in its class and that it will seek to determine if the vehicle itself contributes to the accidents. </P>
<P>The decision to do an engineering analysis of the Ford Motor Co. utility-sport vehicle grew out of a federal accident study of the
Suzuki Samurai, said Tim Hurd, a spokesman for the National Highway Traffic Safety Administration. NHTSA looked at Samurai accidents after
Consumer Reports magazine charged that the vehicle had basic design flaws. </P>
<P>Several Fatalities </P>
<P>However, the accident study showed that the "Ford Bronco II appears to have a higher number of single-vehicle, first event roll-overs,
particularly those involving fatalities," Hurd said. The engineering analysis of the Bronco, the second of three levels of investigation
conducted by NHTSA, will cover the Bronco II models, the agency said. </P>
<P>According to a Fatal Accident Reporting System study included in the September report on the Samurai, 43 Bronco II single-vehicle
roll-overs caused fatalities, or 19 of every 100,000 vehicles. There were eight Samurai fatal roll-overs, or 6 per 100,000; 13 involving
the Chevrolet S10 Blazers or GMC Jimmy, or 6 per 100,000, and six fatal Jeep Cherokee roll-overs, for 2.5 per 100,000. After the
accident report, NHTSA declined to investigate the Samurai. </P>
...
</TEXT>
<GRAPHIC><P> Photo, The Ford Bronco II "appears to have a higher
number of single-vehicle, first event roll-overs," a federal official
said. </P></GRAPHIC>
<SUBJECT>
<P>TRAFFIC ACCIDENTS; FORD MOTOR CORP; NATIONAL HIGHWAY TRAFFIC SAFETY ADMINISTRATION; VEHICLE INSPECTIONS;
RECREATIONAL VEHICLES; SUZUKI MOTOR CO; AUTOMOBILE SAFETY </P>
</SUBJECT>
</DOC>
(C) 2005, The University of Michigan

50. TREC (cont’d) http://trec.nist.gov/tracks.html
(C) 2005, The University of Michigan

51. Word distribution models
(C) 2005, The University of Michigan

52. Shakespeare Romeo and Juliet:
And, 667; The, 661; I, 570; To, 515; A, 447; Of, 382; My, 356; Is, 343; That, 343; In, 314; You, 289; Thou, 277; Me, 262; Not, 257; With, 234; It, 224; For, 223; This, 215; Be, 207; But, 181; Thy, 167; What, 163; O, 160; As, 156; Her, 150; Will, 147; So, 145; Thee, 139; Love, 135; His, 128; Have, 127; He, 120; Romeo, 115; By, 114; She, 114; Shall, 107; Your, 103; No, 102; Come, 96; Him, 96; All, 92; Do, 89; From, 86; Then, 83; Good, 82; Now, 82; Here, 80; If, 80; An, 78; Go, 76; On, 76; I'll, 71; Death, 69; Night, 68; Are, 67; More, 67; We, 66; At, 65; Man, 65; Or, 65; There, 64; Hath, 63; Which, 60; …
A-bed, 1; A-bleeding, 1; A-weary, 1; Abate, 1; Abbey, 1; Abhorred, 1; Abhors, 1; Aboard, 1; Abound'st, 1; Abroach, 1; Absolved, 1; Abuse, 1; Abused, 1; Abuses, 1; Accents, 1; Access, 1; Accident, 1; Accidents, 1; According, 1; Accursed, 1; Accustom'd, 1; Ache, 1; Aches, 1; Aching, 1; Acknowledge, 1; Acquaint, 1; Acquaintance, 1; Acted, 1; Acting, 1; Action, 1; Acts, 1; Adam, 1; Add, 1; Added, 1; Adding, 1; Addle, 1; Adjacent, 1; Admired, 1; Ado, 1; Advance, 1; Adversary, 1; Adversity's, 1; Advise, 1; Afeard, 1; Affecting, 1; Afflicted, 1; Affliction, 1; Affords, 1; Affray, 1; Affright, 1; Afire, 1; Agate-stone, 1; Agile, 1; Agree, 1; Agrees, 1; Aim'd, 1; Alderman, 1; All-cheering, 1; All-seeing, 1; Alla, 1; Alliance, 1; Alligator, 1; Allow, 1; Ally, 1; Although, 1;
(C) 2005, The University of Michigan

53. The BNC (Adam Kilgarriff)
the det
be v
of prep
and conj
a det
in prep
to infinitive-marker
have v
it pron
to prep
for prep
i pron
that conj
you pron
he pron
on prep
with prep
do v
at prep
by prep
Kilgarriff, A. Putting Frequencies in the Dictionary. International Journal of Lexicography 10 (2) Pp
(C) 2005, The University of Michigan

54. Stop lists
most common words in English account for 50% or more of a given text.
Example: “the” and “of” represent 10% of tokens. “and”, “to”, “a”, and “in” - another 10%. Next 12 words - another 10%.
Moby Dick Ch.1: 859 unique words (types), 2256 word occurrences (tokens). Top 65 types cover 1132 tokens (> 50%).
Token/type ratio: 2256/859 = 2.63
(C) 2005, The University of Michigan

55. Zipf’s law Rank x Frequency  Constant
(C) 2005, The University of Michigan

56. Zipf's law is fairly general!
Frequency of accesses to web pages
in particular the access counts on the Wikipedia page,
with s approximately equal to 0.3
page access counts on Polish Wikipedia (data for late July 2003)
approximately obey Zipf's law with s about 0.5
Words in the English language
for instance, in Shakespeare’s play Hamlet with s approximately 0.5
Sizes of settlements
Income distributions amongst individuals
Size of earthquakes
Notes in musical performances
(C) 2005, The University of Michigan

57. Zipf’s law (cont’d) Limitations: Power law with coefficient c = -1
Low and high frequencies
Lack of convergence
Power law with coefficient c = -1
Y=kxc
Li (1992) – typing words one letter at a time, including spaces
(C) 2005, The University of Michigan

58. Heap’s law Size of vocabulary: V(n) = Knb
In English, K is between 10 and 100, β is between 0.4 and 0.6.
V(n) n
(C) 2005, The University of Michigan

59. Heap’s law (cont’d) Related to Zipf’s law: generative models
Zipf’s and Heap’s law coefficients change with language
Alexander Gelbukh, Grigori Sidorov. Zipf and Heaps Laws’ Coefficients Depend on Language. Proc. CICLing-2001, Conference on Intelligent Text Processing and Computational Linguistics, February 18–24, 2001, Mexico City. Lecture Notes in Computer Science N 2004, ISSN , ISBN , Springer-Verlag, pp. 332–335.
(C) 2005, The University of Michigan

60. Indexing
(C) 2005, The University of Michigan

61. Methods Manual: e.g., Library of Congress subject headings, MeSH
Automatic
(C) 2005, The University of Michigan

62. LOC subject headings http://www.loc.gov/catdir/cpso/lcco/lcco.html
A -- GENERAL WORKS B -- PHILOSOPHY. PSYCHOLOGY. RELIGION C -- AUXILIARY SCIENCES OF HISTORY D -- HISTORY (GENERAL) AND HISTORY OF EUROPE E -- HISTORY: AMERICA F -- HISTORY: AMERICA G -- GEOGRAPHY. ANTHROPOLOGY. RECREATION H -- SOCIAL SCIENCES J -- POLITICAL SCIENCE K -- LAW L -- EDUCATION M -- MUSIC AND BOOKS ON MUSIC N -- FINE ARTS P -- LANGUAGE AND LITERATURE Q -- SCIENCE R -- MEDICINE S -- AGRICULTURE T -- TECHNOLOGY U -- MILITARY SCIENCE V -- NAVAL SCIENCE Z -- BIBLIOGRAPHY. LIBRARY SCIENCE. INFORMATION RESOURCES (GENERAL)
(C) 2005, The University of Michigan

63. Medicine CLASS R - MEDICINE Subclass R R5-920 Medicine (General)
R General works
R History of medicine. Medical expeditions
R Medicine as a profession. Physicians
R Medicine and the humanities. Medicine and disease in relation to history, literature, etc.
R Directories
R Missionary medicine. Medical missionaries
R Medical philosophy. Medical ethics
R Medicine and disease in relation to psychology. Terminal care. Dying
R Medical personnel and the public. Physician and the public
R Practice of medicine. Medical practice economics
R Medical education. Medical schools. Research
R Medical technology
R Biomedical engineering. Electronics. Instrumentation
R Computer applications to medicine. Medical informatics
R864 Medical records
R Medical physics. Medical radiology. Nuclear medicine
(C) 2005, The University of Michigan

64. Finding the most frequent terms in a document
Typically stop words: the, and, in
Not content-bearing
Terms vs. words
Luhn’s method
(C) 2005, The University of Michigan

65. Luhn’s method E
FREQUENCY
WORDS
(C) 2005, The University of Michigan

66. Computing term salience
Term frequency (IDF)
Document frequency (DF)
Inverse document frequency (IDF)
(C) 2005, The University of Michigan

67. Applications of TFIDF Cosine similarity Indexing Clustering
(C) 2005, The University of Michigan

68. Vector-based matching
The cosine measure S
(dk . ck . idf(k))
sim (D,C) = k S S
(dk)2 .
(ck)2 k k
(C) 2005, The University of Michigan

69. IDF: Inverse document frequency
TF * IDF is used for automated indexing and for topic discrimination:
N: number of documents dk: number of documents containing term k fik: absolute frequency of term k in document i wik: weight of term k in document i
idfk = log2(N/dk) + 1 = log2N - log2dk + 1
(C) 2005, The University of Michigan

70. Asian and European news
deng
china
beijing
chinese
xiaoping
jiang
communist
body
party
died
leader
state
people
nato
albright
belgrade
enlargement
alliance
french
opposition
russia
government
told
would
their
which
(C) 2005, The University of Michigan

71. Other topics 120.385 shuttle 99.487 space 90.128 telescope
hubble
rocket
astronauts
discovery
canaveral
cape
mission
florida
center
compuserve
massey
salizzoni
bob
online
executive
interim
chief
service
second
world
president
(C) 2005, The University of Michigan

72. Compression
(C) 2005, The University of Michigan

73. Compression Methods Fixed length codes Huffman coding Ziv-Lempel codes
(C) 2005, The University of Michigan

74. Fixed length codes Binary representations ASCII
Representational power (2k symbols where k is the number of bits)
(C) 2005, The University of Michigan

75. Variable length codes Alphabet: A .- N -. 0 ----- B -... O --- 1 .----
C -.-.  P .--. 
D -..  Q --.-  —
E .  R
F S
G --. T - 
H U ..- 
I ..  V ...- 
J .---  W .-- 
K -.-  X -..-
L .-..  Y -.—
M --  Z --..
Demo:
(C) 2005, The University of Michigan

76. Most frequent letters in English
E T A O I N S H R D L U
Demo:
Also: bigrams:
TH HE IN ER AN RE ND AT ON NT
(C) 2005, The University of Michigan

77. Useful links about cryptography
(C) 2005, The University of Michigan

78. Huffman coding Developed by David Huffman (1952)
Average of 5 bits per character (37.5% compression)
Based on frequency distributions of symbols
Algorithm: iteratively build a tree of symbols starting with the two least frequent symbols
(C) 2005, The University of Michigan

79. (C) 2005, The University of Michigan

80. 1 1 1 g 1 1 1 i j f c 1 1 b d a 1 e h
(C) 2005, The University of Michigan

81. (C) 2005, The University of Michigan

82. Exercise
Consider the bit string:
Use the Huffman code from the example to decode it.
Try inserting, deleting, and switching some bits at random locations and try decoding.
(C) 2005, The University of Michigan

83. Ziv-Lempel coding Two types - one is known as LZ77 (used in GZIP)
Code: set of triples <a,b,c>
a: how far back in the decoded text to look for the upcoming text segment
b: how many characters to copy
c: new character to add to complete segment
(C) 2005, The University of Michigan

84. <8,2,r> peter_piper <6,3,c> peter_piper_pic
<0,0,p> p
<0,0,e> pe
<0,0,t> pet
<2,1,r> peter
<0,0,_> peter_
<6,1,i> peter_pi
<8,2,r> peter_piper
<6,3,c> peter_piper_pic
<0,0,k> peter_piper_pick
<7,1,d> peter_piper_picked
<7,1,a> peter_piper_picked_a
<9,2,e> peter_piper_picked_a_pe
<9,2,_> peter_piper_picked_a_peck_
<0,0,o> peter_piper_picked_a_peck_o
<0,0,f> peter_piper_picked_a_peck_of
<17,5,l> peter_piper_picked_a_peck_of_pickl
<12,1,d> peter_piper_picked_a_peck_of_pickled
<16,3,p> peter_piper_picked_a_peck_of_pickled_pep
<3,2,r> peter_piper_picked_a_peck_of_pickled_pepper
<0,0,s> peter_piper_picked_a_peck_of_pickled_peppers
(C) 2005, The University of Michigan

85. Links on text compression
Data compression:
Calgary corpus:
Huffman coding: LZ
(C) 2005, The University of Michigan

86. Relevance feedback and query expansion
(C) 2005, The University of Michigan

87. Relevance feedback
Problem: initial query may not be the most appropriate to satisfy a given information need.
Idea: modify the original query so that it gets closer to the right documents in the vector space
(C) 2005, The University of Michigan

88. Relevance feedback Automatic Manual Method: identifying feedback terms
Q’ = a1Q + a2R - a3N
Often a1 = 1, a2 = 1/|R| and a3 = 1/|N|
(C) 2005, The University of Michigan

89. Example Q = “safety minivans”
D1 = “car safety minivans tests injury statistics” - relevant
D2 = “liability tests safety” - relevant
D3 = “car passengers injury reviews” - non-relevant
R = ?
S = ?
Q’ = ?
(C) 2005, The University of Michigan

90. Pseudo relevance feedback
Automatic query expansion
Thesaurus-based expansion (e.g., using latent semantic indexing – later…)
Distributional similarity
Query log mining
(C) 2005, The University of Michigan

91. Examples Lexical semantics (Hypernymy): Distributional clustering:
Book: publication, product, fact, dramatic composition, record
Computer: machine, expert, calculator, reckoner, figurer
Fruit: reproductive structure, consequence, product, bear
Politician: leader, schemer
Newspaper: press, publisher, product, paper, newsprint
Distributional clustering:
Book: autobiography, essay, biography, memoirs, novels
Computer: adobe, computing, computers, developed, hardware
Fruit: leafy, canned, fruits, flowers, grapes
Politician: activist, campaigner, politicians, intellectuals, journalist
Newspaper: daily, globe, newspapers, newsday, paper
(C) 2005, The University of Michigan

92. Examples (query logs) Book: booksellers, bookmark, blue
Computer: sales, notebook, stores, shop
Fruit: recipes cake salad basket company
Games: online play gameboy free video
Politician: careers federal office history
Newspaper: online website college information
Schools: elementary high ranked yearbook
California: berkeley san francisco southern
French: embassy dictionary learn
(C) 2005, The University of Michigan

93. Problems with automatic query expansion
Adding frequent words may dilute the results (example?)
(C) 2005, The University of Michigan

94. Stemming
(C) 2005, The University of Michigan

95. Goals Motivation: Representing related words as one token
Computer, computers, computerize, computational, computerization
User, users, using, used
Representing related words as one token
Simplify matching
Reduce storage and computation
Also known as: term conflation
(C) 2005, The University of Michigan

96. Methods Manual (tables) Affix removal (Harman 1991, Frakes 1992)
Achievement  achiev
Achiever  achiev
Etc.
Affix removal (Harman 1991, Frakes 1992)
if a word ends in “ies” but not “eies” or “aies” then “ies”  “y”
If a word ends in “es” but not “aes”, “ees”, or “oes”, then “es”  “e”
If a word ends in “s” but not “us” or “ss” then “s”  NULL
(apply only the first applicable rule)
(C) 2005, The University of Michigan

97. Porter’s algorithm (Porter 1980)
Home page:
Reading assignment:
Consonant-vowel sequences:
CVCV ... C
CVCV ... V
VCVC ... C
VCVC ... V
Shorthand: [C]VCVC ... [V]
(C) 2005, The University of Michigan

98. Porter’s algorithm (cont’d)
[C](VC){m}[V]
{m} indicates repetition
Examples:
m=0 TR, EE, TREE, Y, BY
m=1 TROUBLE, OATS, TREES, IVY
m=2 TROUBLES, PRIVATE, OATEN
Conditions:
*S - the stem ends with S (and similarly for the other letters).
*v* - the stem contains a vowel.
*d - the stem ends with a double consonant (e.g. -TT, -SS).
*o - the stem ends cvc, where the second c is not W, X or Y (e.g. -WIL, -HOP).
(C) 2005, The University of Michigan

99. SSES -> SS caresses -> caress
Step 1a
SSES -> SS caresses -> caress
IES -> I ponies -> poni ties -> ti
SS -> SS caress -> caress
S -> cats -> cat
Step 1b
(m>0) EED -> EE feed -> feed agreed -> agree
(*v*) ED -> plastered -> plaster bled -> bled
(*v*) ING -> motoring -> motor sing -> sing
Step 1b1
If the second or third of the rules in Step 1b is successful,
the following is done:
AT -> ATE conflat(ed) -> conflate
BL -> BLE troubl(ed) -> trouble
IZ -> IZE siz(ed) -> size (*d and not (*L or *S or *Z)) -> single letter
hopp(ing) -> hop
tann(ed) -> tan
fall(ing) -> fall
hiss(ing) -> hiss
fizz(ed) -> fizz
(m=1 and *o) -> E fail(ing) -> fail fil(ing) -> file
(C) 2005, The University of Michigan

100. (*v*) Y -> I happy -> happi sky -> sky Step 2
Step 1c
(*v*) Y -> I happy -> happi sky -> sky
Step 2
(m>0) ATIONAL -> ATE relational -> relate
(m>0) TIONAL -> TION conditional -> condition rational -> rational
(m>0) ENCI -> ENCE valenci -> valence
(m>0) ANCI -> ANCE hesitanci -> hesitance
(m>0) IZER -> IZE digitizer -> digitize
(m>0) ABLI -> ABLE conformabli -> conformable
(m>0) ALLI -> AL radicalli -> radical
(m>0) ENTLI -> ENT differentli -> different
(m>0) ELI -> E vileli - > vile
(m>0) OUSLI -> OUS analogousli -> analogous
(m>0) IZATION -> IZE vietnamization -> vietnamize
(m>0) ATION -> ATE predication -> predicate
(m>0) ATOR -> ATE operator -> operate
(m>0) ALISM -> AL feudalism -> feudal
(m>0) IVENESS -> IVE decisiveness -> decisive
(m>0) FULNESS -> FUL hopefulness -> hopeful
(m>0) OUSNESS -> OUS callousness -> callous
(m>0) ALITI -> AL formaliti -> formal
(m>0) IVITI -> IVE sensitiviti -> sensitive
(m>0) BILITI -> BLE sensibiliti -> sensible
(C) 2005, The University of Michigan

101. Step 3
(m>0) ICATE -> IC triplicate -> triplic
(m>0) ATIVE -> formative -> form
(m>0) ALIZE -> AL formalize -> formal
(m>0) ICITI -> IC electriciti -> electric
(m>0) ICAL -> IC electrical -> electric
(m>0) FUL -> hopeful -> hope
(m>0) NESS -> goodness -> good
Step 4
(m>1) AL -> revival -> reviv
(m>1) ANCE -> allowance -> allow
(m>1) ENCE -> inference -> infer
(m>1) ER -> airliner -> airlin
(m>1) IC -> gyroscopic -> gyroscop
(m>1) ABLE -> adjustable -> adjust
(m>1) IBLE -> defensible -> defens
(m>1) ANT -> irritant -> irrit
(m>1) EMENT -> replacement -> replac
(m>1) MENT -> adjustment -> adjust
(m>1) ENT -> dependent -> depend
(m>1 and (*S or *T)) ION -> adoption -> adopt
(m>1) OU -> homologou -> homolog
(m>1) ISM -> communism -> commun
(m>1) ATE -> activate -> activ
(m>1) ITI -> angulariti -> angular
(m>1) OUS -> homologous -> homolog
(m>1) IVE -> effective -> effect
(m>1) IZE -> bowdlerize -> bowdler
(C) 2005, The University of Michigan

102. (m>1) E -> probate -> probat rate -> rate
Step 5a
(m>1) E -> probate -> probat rate -> rate
(m=1 and not *o) E -> cease -> ceas
Step 5b
(m > 1 and *d and *L) -> single letter controll -> control roll -> roll
(C) 2005, The University of Michigan

103. Porter’s algorithm (cont’d)
Example: the word “duplicatable”
duplicat rule 4 duplicate rule 1b1 duplic rule 3
The application of another rule in step 4, removing “ic,” cannot be applied since one rule from each step is allowed to be applied.
% cd /clair4/class/ir-w03/tf-idf
% ./stem.pl computers
computers comput
(C) 2005, The University of Michigan

104. Porter’s algorithm
(C) 2005, The University of Michigan

105. Stemming Not always appropriate (e.g., proper names, titles)
The same applies to casing (e.g., CAT vs. cat)
(C) 2005, The University of Michigan

106. String matching
(C) 2005, The University of Michigan

107. String matching methods
Index-based
Full or approximate
E.g., theater = theatre
(C) 2005, The University of Michigan

108. Index-based matching Inverted files Position-based inverted files
Block-based inverted files
This is a text. A text has many words. Words are made from letters.
Text: 11, 19
Words: 33, 40
From: 55
(C) 2005, The University of Michigan

109. Inverted index (trie) Letters: 60 l d Made: 50 a m n Many: 28 t
Text: 11, 19 w
Words: 33, 40
(C) 2005, The University of Michigan

110. Sequential searching No indexing structure given
Given: database d and search pattern p.
Example: find “words” in the earlier example
Brute force method
try all possible starting positions
O(n) positions in the database and O(m) characters in the pattern so the total worst-case runtime is O(mn)
Typical runtime is actually O(n) given that mismatches are easy to notice
(C) 2005, The University of Michigan

111. Knuth-Morris-Pratt Average runtime similar to BF
Worst case runtime is linear: O(n)
Idea: reuse knowledge
Need preprocessing of the pattern
(C) 2005, The University of Michigan

112. Knuth-Morris-Pratt (cont’d)
Example (
database: ABC ABC ABC ABDAB ABCDABCDABDE
pattern: ABCDABD
index
char A B C D A B D –
pos
ABCDABD
(C) 2005, The University of Michigan

113. Knuth-Morris-Pratt (cont’d)
ABC ABC ABC ABDAB ABCDABCDABDE
ABCDABD ^
(C) 2005, The University of Michigan

114. Boyer-Moore Used in text editors Demos
(C) 2005, The University of Michigan

115. Word similarity Hamming distance - when words are of the same length
Levenshtein distance - number of edits (insertions, deletions, replacements)
color --> colour (1)
survey --> surgery (2)
com puter --> computer ?
Longest common subsequence (LCS)
lcs (survey, surgery) = surey
(C) 2005, The University of Michigan

116. Levenshtein edit distance
Examples:
Theatre-> theater
Ghaddafi->Qadafi
Computer->counter
Edit distance (inserts, deletes, substitutions)
Edit transcript
Done through dynamic programming
(C) 2005, The University of Michigan

117. Recurrence relation Three dependencies Simple edit distance: D(i,0)=i
D(0,j)=j
D(i,j)=min[D(i-1,j)+1,D(1,j-1)+1,D(i-1,j-1)+t(i,j)]
Simple edit distance:
t(i,j) = 0 iff S1(i)=S2(j)
(C) 2005, The University of Michigan

118. Example Gusfield 1997 W R I T E S 1 2 3 4 5 6 7 V N1 2 3 4 5 6 7 V N
(C) 2005, The University of Michigan
Gusfield 1997

119. Example (cont’d) Gusfield 1997 W R I T E S 1 2 3 4 5 6 7 V N *1 2 3 4 5 6 7 V N *
(C) 2005, The University of Michigan
Gusfield 1997

120. Tracebacks Gusfield 1997 W R I T E S 1 2 3 4 5 6 7 V N *1 2 3 4 5 6 7 V N *
(C) 2005, The University of Michigan
Gusfield 1997

121. Weighted edit distance
Used to emphasize the relative cost of different edit operations
Useful in bioinformatics
Homology information
BLAST
Blosum
(C) 2005, The University of Michigan

122. Web sites: http://www.merriampark.com/ld.htm
(C) 2005, The University of Michigan

123. Clustering
(C) 2005, The University of Michigan

124. Clustering Exclusive/overlapping clusters Hierarchical/flat clusters
The cluster hypothesis
Documents in the same cluster are relevant to the same query
(C) 2005, The University of Michigan

125. Representations for document clustering
Typically: vector-based
Words: “cat”, “dog”, etc.
Features: document length, author name, etc.
Each document is represented as a vector in an n-dimensional space
Similar documents appear nearby in the vector space (distance measures are needed)
(C) 2005, The University of Michigan

126. Hierarchical clustering Dendrograms
E.g., language similarity:
(C) 2005, The University of Michigan

127. Another example Kingdom = animal Phylum = Chordata
Subphylum = Vertebrata
Class = Osteichthyes
Subclass = Actinoptergyii
Order = Salmoniformes
Family = Salmonidae
Genus = Oncorhynchus
Species = Oncorhynchus kisutch (Coho salmon)
(C) 2005, The University of Michigan

128. Clustering using dendrograms
Example: cluster the following sentences:
A B C B A
A D C C A D E
C D E F C D A
E F G F D A
A C D A B A
REPEAT
Compute pairwise similarities
Identify closest pair
Merge pair into single node
UNTIL only one node left
Q: what is the equivalent Venn diagram representation?
(C) 2005, The University of Michigan

129. Methods Single-linkage Complete-linkage Average-linkage
One common pair is sufficient
disadvantages: long chains
Complete-linkage
All pairs have to match
Disadvantages: too conservative
Average-linkage
Centroid-based (online)
Look at distances to centroids
Demo:
/clair4/class/ir-w05/clustering
(C) 2005, The University of Michigan

130. k-means Needed: small number k of desired clusters
hard vs. soft decisions
Example: Weka
(C) 2005, The University of Michigan

131. k-means 1 initialize cluster centroids to arbitrary vectors
2 while further improvement is possible do
3 for each document d do
find the cluster c whose centroid is closest to d
assign d to cluster c
6 end for
7 for each cluster c do
recompute the centroid of cluster c based on its documents
9 end for
10 end while
(C) 2005, The University of Michigan

132. Example Cluster the following vectors into two groups: A = <1,6>
B = <2,2>
C = <4,0>
D = <3,3>
E = <2,5>
F = <2,1>
(C) 2005, The University of Michigan

133. Complexity
Complexity = O(kn) because at each step, n documents have to be compared to k centroids.
(C) 2005, The University of Michigan

134. Weka
A general environment for machine learning (e.g. for classification and clustering)
Book by Witten and Frank
(C) 2005, The University of Michigan

135. Demos http://vivisimo.com/
(C) 2005, The University of Michigan

136. Human clustering
Significant disagreement in the number of clusters, overlap of clusters, and the composition of clusters (Maczkassy et al. 1998).
(C) 2005, The University of Michigan

137. Lexical networks
(C) 2005, The University of Michigan

138. Lexical Networks Used to represent relationships between words
Example: WordNet - created by George Miller’s team at Princeton
Based on synsets (synonyms, interchangeable words) and lexical matrices
(C) 2005, The University of Michigan

139. Lexical matrix
(C) 2005, The University of Michigan

140. Synsets Disambiguation Synonyms {board, plank} {board, committee}
substitution
weak substitution
synonyms must be of the same part of speech
(C) 2005, The University of Michigan

141. $ ./wn board -hypen
Synonyms/Hypernyms (Ordered by Frequency) of noun board
9 senses of board
Sense 1
board
=> committee, commission
=> administrative unit
=> unit, social unit
=> organization, organisation
=> social group
=> group, grouping
Sense 2
=> sheet, flat solid
=> artifact, artefact
=> object, physical object
=> entity, something
Sense 3
board, plank
=> lumber, timber
=> building material
(C) 2005, The University of Michigan

142. Sense 4
display panel, display board, board
=> display
=> electronic device
=> device
=> instrumentality, instrumentation
=> artifact, artefact
=> object, physical object
=> entity, something
Sense 5
board, gameboard
=> surface
Sense 6
board, table
=> fare
=> food, nutrient
=> substance, matter
(C) 2005, The University of Michigan

143. Sense 7
control panel, instrument panel, control board, board, panel
=> electrical device
=> device
=> instrumentality, instrumentation
=> artifact, artefact
=> object, physical object
=> entity, something
Sense 8
circuit board, circuit card, board, card
=> printed circuit
=> computer circuit
=> circuit, electrical circuit, electric circuit
Sense 9
dining table, board
=> table
=> furniture, piece of furniture, article of furniture
=> furnishings
(C) 2005, The University of Michigan

144. Antonymy “x” vs. “not-x” “rich” vs. “poor”?
{rise, ascend} vs. {fall, descend}
(C) 2005, The University of Michigan

145. Other relations
Meronymy: X is a meronym of Y when native speakers of English accept sentences similar to “X is a part of Y”, “X is a member of Y”.
Hyponymy: {tree} is a hyponym of {plant}.
Hierarchical structure based on hyponymy (and hypernymy).
(C) 2005, The University of Michigan

146. Other features of WordNet
Index of familiarity
Polysemy
(C) 2005, The University of Michigan

147. Familiarity and polysemy
board used as a noun is familiar (polysemy count = 9)
bird used as a noun is common (polysemy count = 5)
cat used as a noun is common (polysemy count = 7)
house used as a noun is familiar (polysemy count = 11)
information used as a noun is common (polysemy count = 5)
retrieval used as a noun is uncommon (polysemy count = 3)
serendipity used as a noun is very rare (polysemy count = 1)
(C) 2005, The University of Michigan

148. Compound nouns advisory board appeals board backboard backgammon board
baseboard
basketball backboard
big board
billboard
binder's board
binder board
blackboard
board game
board measure
board meeting
board member
board of appeals
board of directors
board of education
board of regents
board of trustees
(C) 2005, The University of Michigan

149. Overview of senses
1. board -- (a committee having supervisory powers; "the board has seven members")
2. board -- (a flat piece of material designed for a special purpose; "he nailed boards across the windows")
3. board, plank -- (a stout length of sawn timber; made in a wide variety of sizes and used for many purposes)
4. display panel, display board, board -- (a board on which information can be displayed to public view)
5. board, gameboard -- (a flat portable surface (usually rectangular) designed for board games; "he got out the board and set up the pieces")
6. board, table -- (food or meals in general; "she sets a fine table"; "room and board")
7. control panel, instrument panel, control board, board, panel -- (an insulated panel containing switches and dials and meters for controlling electrical devices; "he checked the instrument panel"; "suddenly the board lit up like a Christmas tree")
8. circuit board, circuit card, board, card -- (a printed circuit that can be inserted into expansion slots in a computer to increase the computer's capabilities)
9. dining table, board -- (a table at which meals are served; "he helped her clear the dining table"; "a feast was spread upon the board")
(C) 2005, The University of Michigan

150. Top-level concepts {act, action, activity} {animal, fauna} {artifact}
{attribute, property}
{body, corpus}
{cognition, knowledge}
{communication}
{event, happening}
{feeling, emotion}
{food}
{group, collection}
{location, place}
{motive}
{natural object}
{natural phenomenon}
{person, human being}
{plant, flora}
{possession}
{process}
{quantity, amount}
{relation}
{shape}
{state, condition}
{substance}
{time}
(C) 2005, The University of Michigan

151. WordNet and DistSim wn reason -hypen - hypernyms
wn reason -synsn - synsets
wn reason -simsn - synonyms
wn reason -over overview of senses
wn reason -famln - familiarity/polysemy
wn reason -grepn - compound nouns
/data2/tools/relatedwords/relate reason
(C) 2005, The University of Michigan

152. System comparison
(C) 2005, The University of Michigan

153. Comparing two systems Comparing A and B One query?
Average performance?
Need: A to consistently outperform B
[this slide: courtesy James Allan]
(C) 2005, The University of Michigan

154. The sign test Example 1: Example 2: A > B (12 times)
p < (significant at the 5% level)
Example 2:
A > B (18 times)
A < B (9 times)
p < (not significant at the 5% level)
[this slide: courtesy James Allan]
(C) 2005, The University of Michigan

155. Other tests The t test: The sign test:
Takes into account the actual performances, not just which system is better
The sign test:
(C) 2005, The University of Michigan

156. Techniques for dimensionality reduction: SVD and LSI
(C) 2005, The University of Michigan

157. Techniques for dimensionality reduction
Based on matrix decomposition (goal: preserve clusters, explain away variance)
A quick review of matrices
Vectors
Matrices
Matrix multiplication
(C) 2005, The University of Michigan

158. SVD: Singular Value Decomposition
A=USVT
This decomposition exists for all matrices, dense or sparse
If A has 5 columns and 3 rows, then U will be 5x5 and V will be 3x3
In Matlab, use [U,S,V] = svd (A)
(C) 2005, The University of Michigan

159. Term matrix normalization
D1 D2 D3 D4 D5
D D D D D5
(C) 2005, The University of Michigan

160. Example (Berry and Browne)
T1: baby
T2: child
T3: guide
T4: health
T5: home
T6: infant
T7: proofing
T8: safety
T9: toddler
D1: infant & toddler first aid
D2: babies & children’s room (for your home)
D3: child safety at home
D4: your baby’s health and safety: from infant to toddler
D5: baby proofing basics
D6: your guide to easy rust proofing
D7: beanie babies collector’s guide
(C) 2005, The University of Michigan

161. Document term matrix
(C) 2005, The University of Michigan

162. Decomposition (C) 2005, The University of Michigan u =
v =
(C) 2005, The University of Michigan

163. Decomposition Spread on the v1 axis s = 1.5849 0 0 0 0 0 0
(C) 2005, The University of Michigan

164. Rank-4 approximation s4 = 1.5849 0 0 0 0 0 0 0 1.2721 0 0 0 0 0
(C) 2005, The University of Michigan

165. Rank-4 approximation u*s4*v'
(C) 2005, The University of Michigan

166. Rank-4 approximation u*s4 -1.1056 -0.1203 0.0207 -0.5558 0 0 0
(C) 2005, The University of Michigan

167. Rank-4 approximation s4*v'
(C) 2005, The University of Michigan

168. Rank-2 approximation s2 = 1.5849 0 0 0 0 0 0 0 1.2721 0 0 0 0 0
(C) 2005, The University of Michigan

169. Rank-2 approximation u*s2*v'
(C) 2005, The University of Michigan

170. Rank-2 approximation u*s2 -1.1056 -0.1203 0 0 0 0 0
(C) 2005, The University of Michigan

171. Rank-2 approximation s2*v'
(C) 2005, The University of Michigan

172. Documents to concepts and terms to concepts
A(:,1)'*u*s
>> A(:,1)'*u*s4
>> A(:,1)'*u*s2
>> A(:,2)'*u*s2
>> A(:,3)'*u*s2
(C) 2005, The University of Michigan

173. Documents to concepts and terms to concepts
>> A(:,4)'*u*s2
>> A(:,5)'*u*s2
>> A(:,6)'*u*s2
>> A(:,7)'*u*s2
(C) 2005, The University of Michigan

174. Cont’d (C) 2005, The University of Michigan >> (s2*v'*A(1,:)')'
>> (s2*v'*A(2,:)')'
>> (s2*v'*A(3,:)')'
>> (s2*v'*A(4,:)')'
>> (s2*v'*A(5,:)')'
(C) 2005, The University of Michigan

175. Cont’d (C) 2005, The University of Michigan >> (s2*v'*A(6,:)')'
>> (s2*v'*A(7,:)')'
>> (s2*v'*A(8,:)')'
>> (s2*v'*A(9,:)')‘
(C) 2005, The University of Michigan

176. Properties A is a document to term matrix. What is A*A’, what is A’*A?
A'*A
(C) 2005, The University of Michigan

177. Latent semantic indexing (LSI)
Dimensionality reduction = identification of hidden (latent) concepts
Query matching in latent space
(C) 2005, The University of Michigan

178. Useful pointers http://lsa.colorado.edu
(C) 2005, The University of Michigan

179. Models of the Web
(C) 2005, The University of Michigan

180. Size
The Web is the largest repository of data and it grows exponentially.
320 Million Web pages [Lawrence & Giles 1998]
800 Million Web pages, 15 TB [Lawrence & Giles 1999]
8 Billion Web pages indexed [Google 2005]
Amount of data
roughly 200 TB [Lyman et al. 2003]
1.5 million terabytes = 1.5 petabytes = 1.5 E +18.
At 2,000 bytes/page = 1 E 15 pages = 1 billion years at 1 page/minute
1 in every 28 page views on the Web is a search results page (3.5% of all page views) (June 1, 1999, Alexa Insider)
(C) 2005, The University of Michigan

181. Bow-tie model of the Web
TEND 44M
SCC 56 M
IN 44 M
OUT 44 M
24% of pages reachable from a given page
DISC 17 M
(C) 2005, The University of Michigan
Bröder & al. WWW 2000, Dill & al. VLDB 2001

182. Power laws Web site size (Huberman and Adamic 1999)
Power-law connectivity (Barabasi and Albert 1999): exponents 2.45 for out-degree and 2.1 for the in-degree
Others: call graphs among telephone carriers, citation networks (Redner 1998), e.g., Erdos, collaboration graph of actors, metabolic pathways (Jeong et al. 2000), protein networks (Maslov and Sneppen 2002). All values of gamma are around 2-3.
(C) 2005, The University of Michigan

183. Small-world networks
Diameter = average length of the shortest path between all pairs of nodes. Example…
Milgram experiment (1967)
Kansas/Omaha --> Boston (42/160 letters)
diameter = 6
Albert et al – average distance between two verstices is d = log10n. For n = 109, d=18.89.
Six degrees of separation
(C) 2005, The University of Michigan

184. Clustering coefficient
Cliquishness (c): between the kv (kv – 1)/2 pairs of neighbors.
Examples: n k d
drand C
crand
Actors
225226 61
3.65
2.99
0.79
Power grid
4941
2.67
18.7
12.4
0.08
0.005
C. Elegans
282 14
2.65
2.25
0.28
0.05
(C) 2005, The University of Michigan

185. Models of the Web
Evolving networks: fundamental object of statistical physics, social networks, mathematical biology, and epidemiology
Erdös/Rényi 59, 60
Barabási/Albert 99
Watts/Strogatz 98 A B a b
Kleinberg 98
Indegree/outdegree plots
Evolving networks: fundamental object of statistical physics
Social networks
Bow tie model (not?)
My Erdos number is 4
Growth of the Web (p 130)
Why is the Web different (MN) – how do users really create the Web
Fat-tailed distributions, small worlds (W&S, # of triangles), hard to destroy
Communities (Kleinberg)
Evolutionary networks, equilibrium/non-equilibrium
Evaluation metrics: degree distribution, clustering coefficient, diameter (give comparison)
W/S model is equlibrium (term from statistical mechanics)
EN have history, memory
Small word networks (+result)
D&M
Zipf
What are typical clustering coefficients and diameters
Random graph – clustering coefficient is much smaller
Mesoscopic objects
Phase transitions
Bipartite networks
Pagerank
Google changed the way the Web works
Menczer 02
Radev 03
(C) 2005, The University of Michigan

186. Self-triggerability across hyperlinkspj pi
Document closures for information retrieval
Self-triggerability [Mosteller&Wallace 84]  Poisson distribution
Two-Poisson [Bookstein&Swanson 74]
Negative Binomial, K-mixture [Church&Gale 95]
Triggerability across hyperlinks? p p’
by with from p p’
photo dream path
Perltree/Lexpagerank/Web Models/Lex/Link
Predict communities based on a few words
Realistic attachment model (based on words)
Dilemma: add links/remove links
Property-driven attachment
(C) 2005, The University of Michigan

187. Evolving Word-based Web
Observations:
Links are made based on topics
Topics are expressed with words
Words are distributed very unevenly (Zipf, Benford, self-triggerability laws)
Model
Pick n
Generate n lengths according to a power-law distribution
Generate n documents using a trigram model
Model (cont’d)
Pick words in decreasing order of r.
Generate hyperlinks with random directionality
Outcome
Generates power-law degree distributions
Generates topical communities
Natural variation of PageRank: LexRank
(C) 2005, The University of Michigan

188. Social network analysis for IR
(C) 2005, The University of Michigan

189. Social networks Induced by a relation Symmetric or not Examples:
Friendship networks
Board membership
Citations
Power grid of the US
WWW
(C) 2005, The University of Michigan

190. Krebs 2004
(C) 2005, The University of Michigan

191. Prestige and centrality
Degree centrality: how many neighbors each node has.
Closeness centrality: how close a node is to all of the other nodes
Betweenness centrality: based on the role that a node plays by virtue of being on the path between two other nodes
Eigenvector centrality: the paths in the random walk are weighted by the centrality of the nodes that the path connects.
Prestige = same as centrality but for directed graphs.
Pagerank/HITS
(C) 2005, The University of Michigan

192. Graph-based representations
Square connectivity (incidence) matrix
Graph G (V,E) 1 2 3 4 5 6 7 8 1 2 3 4 5 7 6 8
Directed, undirected
(C) 2005, The University of Michigan

193. Markov chains
A homogeneous Markov chain is defined by an initial distribution x and a Markov kernel E.
Path = sequence (x0, x1, …, xn). Xi = xi-1*E
The probability of a path can be computed as a product of probabilities for each step i.
Random walk = find Xj given x0, E, and j.
(C) 2005, The University of Michigan

194. Stationary solutions
The fundamental Ergodic Theorem for Markov chains [Grimmett and Stirzaker 1989] says that the Markov chain with kernel E has a stationary distribution p under three conditions:
E is stochastic
E is irreducible
E is aperiodic
To make these conditions true:
All rows of E add up to 1 (and no value is negative)
Make sure that E is strongly connected
Make sure that E is not bipartite
Example: PageRank [Brin and Page 1998]: use “teleportation”
Stochastic, aperiodic, irreducible (strongly connected)
Unique solutions
(C) 2005, The University of Michigan

195. Example 1 2 3 4 5 7 6 8
t=0
t=1
This graph E has a second graph E’ (not drawn) superimposed on it: E’ is the uniform transition graph.
(C) 2005, The University of Michigan

196. Eigenvectors An eigenvector is an implicit “direction” for a matrix.
Mv = λv, where v is non-zero, though λ can be any complex number in principle.
The largest eigenvalue of a stochastic matrix E is real: λ1 = 1.
For λ1, the left (principal) eigenvector is p, the right eigenvector = 1
In other words, ETp = p.
(C) 2005, The University of Michigan

197. Computing the stationary distribution
Solution for the stationary distribution
function PowerStatDist (E):
begin
p(0) = u; (or p(0) = [1,0,…0])
i=1;
repeat
p(i) = ETp(i-1)
L = ||p(i)-p(i-1)||1;
i = i + 1;
until L < 
return p(i)
end
Power methods (KHMG)
(C) 2005, The University of Michigan

198. Example
t=0 1 2 3 4 5 7 6 8
t=1
t=10
(C) 2005, The University of Michigan

199. How Google works Crawling Anchor text Fast query processing Pagerank
(C) 2005, The University of Michigan

200. More about PageRank
Named after Larry Page, founder of Google (and UM alum)
Reading “The anatomy of a large-scale hypertextual web search engine” by Brin and Page.
Independent of query (although more recent work by Haveliwala (WWW 2002) has also identified topic-based PageRank.
(C) 2005, The University of Michigan

201. HITS Query-dependent model (Kleinberg 97)
Hubs and authorities (e.g., cars, Honda)
Algorithm
obtain root set using input query
expanded the root set by radius one
run iterations on the hub and authority scores together
report top-ranking authorities and hubs
(C) 2005, The University of Michigan

202. The link-content hypothesis
Topical locality: page is similar () to the page that points to it ().
Davison (TF*IDF, 100K pages)
0.31 same domain
0.23 linked pages
0.19 sibling
0.02 random
Menczer (373K pages, non-linear least squares fit)
Chakrabarti (focused crawling) - prob. of losing the topic
1=1.8, 2=0.6,
(C) 2005, The University of Michigan
Van Rijsbergen 1979, Chakrabarti & al. WWW 1999, Davison SIGIR 2000, Menczer 2001

203. Measuring the Web
(C) 2005, The University of Michigan

204. Bharat and Broder 1998
Based on crawls of HotBot, Altavista, Excite, and InfoSeek
10,000 queries in mid and late 1997
Estimate is 200M pages
Only 1.4% are indexed by all of them
(C) 2005, The University of Michigan

205. Example (from Bharat&Broder)
A similar approach by Lawrence and Giles yields 320M pages
(Lawrence and Giles 1998).
(C) 2005, The University of Michigan

206. Crawling the web
(C) 2005, The University of Michigan

207. Basic principles The HTTP/HTML protocols Following hyperlinks
Some problems:
Link extraction
Link normalization
Robot exclusion
Loops
Spider traps
Server overload
(C) 2005, The University of Michigan

208. Example U-M’s root robots.txt file: http://www.umich.edu/robots.txt
User-agent: *
Disallow: /~websvcs/projects/
Disallow: /%7Ewebsvcs/projects/
Disallow: /~homepage/
Disallow: /%7Ehomepage/
Disallow: /~smartgl/
Disallow: /%7Esmartgl/
Disallow: /~gateway/
Disallow: /%7Egateway/
(C) 2005, The University of Michigan

209. Example crawler E.g., poacher
/data0/projects/perltree-index
(C) 2005, The University of Michigan

210. &ParseCommandLine();
&Initialise();
$robot->run($siteRoot)
#=======================================================================
# Initialise() - initialise global variables, contents, tables, etc
# This function sets up various global variables such as the version number
# for WebAssay, the program name identifier, usage statement, etc.
sub Initialise {
$robot = new WWW::Robot(
'NAME' => $BOTNAME,
'VERSION' => $VERSION,
' ' => $ ,
'TRAVERSAL' => $TRAVERSAL,
'VERBOSE' => $VERBOSE, );
$robot->addHook('follow-url-test', \&follow_url_test);
$robot->addHook('invoke-on-contents', \&process_contents);
$robot->addHook('invoke-on-get-error', \&process_get_error); }
# follow_url_test() - tell the robot module whether is should follow link
sub follow_url_test {}
# process_get_error() - hook function invoked whenever a GET fails
sub process_get_error {}
# process_contents() - process the contents of a URL we've retrieved
sub process_contents
run_command($COMMAND, $filename) if defined $COMMAND;
(C) 2005, The University of Michigan

211. Focused crawling Topical locality The radius-1 hypothesis
Pages that are linked are similar in content (and vice-versa: Davison 00, Menczer 02, 04, Radev et al. 04)
The radius-1 hypothesis
given that page i is relevant to a query and that page i points to page j, then page j is also likely to be relevant (at least, more so than a random web page)
Focused crawling
Keeping a priority queue of the most relevant pages
(C) 2005, The University of Michigan

212. Question answering
(C) 2005, The University of Michigan

213. People ask questions
Excite corpus of 2,477,283 queries (one day’s worth)
8.4% of them are questions
43.9% factual (what is the country code for Belgium)
56.1% procedural (how do I set up TCP/IP) or other
In other words, 100 K questions per day
(C) 2005, The University of Michigan

214. People ask questions
In what year did baseball become an offical sport?
Who is the largest man in the world?
Where can i get information on Raphael?
where can i find information on puritan religion?
Where can I find how much my house is worth?
how do i get out of debt? Where can I found out how to pass a drug test? When is the Super Bowl? who is California's District State Senator? where can I buy extra nibs for a foutain pen? how do i set up tcp/ip ? what time is it in west samoa? Where can I buy a little kitty cat? what are the symptoms of attention deficit disorder? Where can I get some information on Michael Jordan? How does the character Seyavash in Ferdowsi's Shahnameh exhibit characteristics of a hero? When did the Neanderthal man live? Which Frenchman declined the Nobel Prize for Literature for ideological reasons? What is the largest city in Northern Afghanistan?
Each question can have multiple answers
How does the character Seyavash in Ferdowsi's Shahnameh exhibit characteristics of a hero?
When did the Neanderthal man live?
(C) 2005, The University of Michigan

215. (C) 2005, The University of Michigan

216. Question answering What is the largest city in Northern Afghanistan?
(C) 2005, The University of Michigan

217. Possible approaches Map?
Knowledge base Find x: city (x)  located (x,”Northern Afghanistan”)   ¬exists (y): city (y)  located (y,”Northern Afghanistan”)   greaterthan (population (y), population (x))
Database?
World factbook?
Search engine?
(C) 2005, The University of Michigan

218. The TREC Q&A evaluation
Run by NIST [Voorhees and Tice 2000]
2GB of input
200 questions
Essentially fact extraction
Who was Lincoln’s secretary of state?
What does the Peugeot company manufacture?
Questions are based on text
Answers are assumed to be present
No inference needed
(C) 2005, The University of Michigan

219. Question answering
Q: When did Nelson Mandela become president of South Africa?
A: 10 May 1994
Q: How tall is the Matterhorn?
A: The institute revised the Matterhorn 's height to 14,776 feet 9 inches
Q: How tall is the replica of the Matterhorn at Disneyland?
A: In fact he has climbed the 147-foot Matterhorn at Disneyland every week end for the last 3 1/2 years
Q: If Iraq attacks a neighboring country, what should the US do?
A: ??
(C) 2005, The University of Michigan

220. (C) 2005, The University of Michigan

221. NSIR Current project at U-M Reading:
Reading:
[Radev et al., 2005a]
Dragomir R. Radev, Weiguo Fan, Hong Qi, Harris Wu, and Amardeep Grewal. Probabilistic question answering on the web. Journal of the American Society for Information Science and Technology 56(3), March 2005
(C) 2005, The University of Michigan

222. (C) 2005, The University of Michigan

223. ... Afghanistan, Kabul, 2, Administrative capital and largest city (1997 est ... Undetermined. Panama, Panama City, 450, of the Gauteng, Northern Province, Mpumalanga
... died in Kano, northern Nigeria's largest city, during two days of anti-American riots led by Muslims protesting the US-led bombing of Afghanistan, according to
... air strikes on the city. ... the Taliban militia in northern Afghanistan in a significant blow ... defection would be the largest since the United States k
... Kabul is the capital and largest city of Afghanistan met. area pop. 2,029,889), is the largest city in Uttar Pradesh, a state in northern India school.discovery.com/homeworkhelp/worldbook/atozgeography/ k/k1menu.html
... Gudermes, Chechnya's second largest town. The attack ... location in Afghanistan's outlying regions ... in the city of Mazar-i-Sharif, a Northern Alliance-affiliated english.pravda.ru/hotspots/2001/09/17/
... Get Worse By RICK BRAGG Pakistan's largest city is getting a jump on the ... Region: Education Offers Women in Northern Afghanistan a Ray of Hope
... within three miles of the airport at Mazar-e-Sharif, the largest city in northern Afghanistan, held since 1998 by the Taliban. There was no immediate comment uk.fc.yahoo.com/photos/a/afghanistan.html
(C) 2005, The University of Michigan
Google

224. Query modulation Document retrieval Sentence retrieval
What is the largest city in Northern Afghanistan?
Query modulation
(largest OR biggest) city “Northern Afghanistan”
Document retrieval
Sentence retrieval
Gudermes, Chechnya's second largest town … location in Afghanistan's outlying regions
within three miles of the airport at Mazar-e-Sharif, the largest city in northern Afghanistan
Answer extraction
Gudermes
Mazer-e-Sharif
Answer ranking
(C) 2005, The University of Michigan
Mazer-e-Sharif
Gudermes

225. (C) 2005, The University of Michigan

226. (C) 2005, The University of Michigan

227. Research problems Source identification: Query modulation:
semi-structured vs. text sources
Query modulation:
best paraphrase of a NL question given the syntax of a search engine?
Compare two approaches: noisy channel model and rule-based
Sentence ranking
n-gram matching, Okapi, co-reference?
Answer extraction
question type identification
phrase chunking
no general-purpose named entity tagger available
Answer ranking
what are the best predictors of a phrase being the answer to a given question: question type, proximity to query words, frequency
Evaluation (MRDR)
accuracy, reliability, timeliness
(C) 2005, The University of Michigan

228. Document retrieval
Use existing search engines: Google, AlltheWeb, NorthernLight
No modifications to question
CF: work on QASM (ACM CIKM 2001)
(C) 2005, The University of Michigan

229. Sentence ranking Weighted N-gram matching:
Weights are determined empirically, e.g., 0.6, 0.3, and 0.1
(C) 2005, The University of Michigan

230. Probabilistic phrase reranking
Answer extraction: probabilistic phrase reranking. What is: p(ph is answer to q | q, ph)
Evaluation: TRDR
Example: (2,8,10) gives .725
Document, sentence, or phrase level
Criterion: presence of answer(s)
High correlation with manual assessment
(C) 2005, The University of Michigan

231. Phrase types
PERSON PLACE DATE NUMBER DEFINITION ORGANIZATION DESCRIPTION ABBREVIATION KNOWNFOR RATE LENGTH MONEY REASON DURATION PURPOSE NOMINAL OTHER
(C) 2005, The University of Michigan

232. Question Type Identification
Wh-type not sufficient:
Who: PERSON 77, DESCRIPTION 19, ORG 6
What: NOMINAL 78, PLACE 27, DEF26, PERSON 18, ORG 16, NUMBER 14, etc.
How: NUMBER 33, LENGTH 6, RATE 2, etc.
Ripper:
13 features: Question-Words, Wh-Word, Word-Beside-Wh-Word, Is-Noun-Length, Is-Noun-Person, etc.
Top 2 question types
Heuristic algorithm:
About 100 regular expressions based on words and parts of speech
(C) 2005, The University of Michigan

233. Ripper performance - 20.69% TREC8,9,10 30% 17.03% TREC10 TREC8,9 24%
22.4%
TREC8
TREC9
Test Error Rate
Train Error Rate
Test
Training
(C) 2005, The University of Michigan

234. Regex performance 7.6% 5.5% 4.6% TREC8,9,10 18.2% 6% 7.4% TREC8,9 18%
15%
7.8%
TREC9
Test on TREC10
Test on TREC8
Test on TREC9
Training
(C) 2005, The University of Michigan

235. Phrase ranking
Phrases are identified by a shallow parser (ltchunk from Edinburgh)
Four features:
Proximity
POS (part-of-speech) signature (qtype)
Query overlap
Frequency
(C) 2005, The University of Michigan

236. Proximity Phrasal answers tend to appear near words from the query
Average distance = 7 words, range = 1 to 50 words
Use linear rescaling of scores
(C) 2005, The University of Michigan

237. Part of speech signature
Penn Treebank tagset (DT = determiner, JJ = adjective)
NO (100%) NO (86.7%) PERSON (3.8%) NUMBER (3.8%) ORG (2.5%) PERSON (37.4%) PLACE (29.6%) DATE (21.7%) NO (7.6%) NO (75.6%) NUMBER (11.1%) PLACE (4.4%) ORG (4.4%) PLACE (37.3%) PERSON (35.6%) NO (16.9%) ORG (10.2%) ORG (55.6%) NO (33.3%) PLACE (5.6%) DATE (5.6%)
VBD DT NN NNP DT JJ NNP NNP NNP DT NNP
Phrase Types
Signature
Example: “Hugo/NNP Young/NNP” P (PERSON | “NNP NNP”) = .458
Example: “the/DT Space/NNP Flight/NNP Operations/NNP contractor/NN”
P (PERSON | “DT NNP NNP NNP NN”) = 0
(C) 2005, The University of Michigan

238. Query overlap and frequency
What is the capital of Zimbabwe?
Possible choices: Mugabe, Zimbabwe, Luanda, Harare
Frequency:
Not necessarily accurate but rather useful
(C) 2005, The University of Michigan

239. Reranking Proximity = .5164 Rank Probability and phrase
the_DT Space_NNP Flight_NNP Operations_NNP contractor_NN ._ International_NNP Space_NNP Station_NNP Alpha_NNP International_NNP Space_NNP Station_NNP to_TO become_VB a_DT joint_JJ venture_NN United_NNP Space_NNP Alliance_NNP NASA_NNP Johnson_NNP Space_NNP Center_NNP will_MD form_VB The_DT purpose_NN prime_JJ contracts_NNS First_NNP American_NNP this_DT bulletin_NN board_NN Space_NNP :_: 'Spirit_NN '_'' of_IN Alan_NNP Shepard_NNP
Proximity = .5164
(C) 2005, The University of Michigan

240. Reranking Qtype = .7288 Proximity * qtype = .3763
Rank Probability and phrase
Space_NNP Administration_NNP ._ SPACE_NNP CALENDAR_NNP _ First_NNP American_NNP International_NNP Space_NNP Station_NNP Alpha_NNP her_PRP$ third_JJ space_NN mission_NN NASA_NNP Johnson_NNP Space_NNP Center_NNP the_DT American_NNP Commercial_NNP Launch_NNP Industry_NNP the_DT Red_NNP Planet_NNP ._ First_NNP American_NNP Alan_NNP Shepard_NNP February_NNP Space_NNP International_NNP Space_NNP Station_NNP
Qtype = Proximity * qtype = .3763
(C) 2005, The University of Michigan

241. Reranking All four features Rank Probability and phrase
Neptune_NNP Beach_NNP ._ February_NNP Go_NNP Space_NNP Go_NNP Alan_NNP Shepard_NNP First_NNP American_NNP Space_NNP May_NNP First_NNP American_NNP woman_NN Life_NNP Sciences_NNP Space_NNP Shuttle_NNP Discovery_NNP STS-60_NN the_DT Moon_NNP International_NNP Space_NNP Station_NNP Space_NNP Research_NNP A_NNP Session_NNP
All four features
(C) 2005, The University of Michigan

242. (C) 2005, The University of Michigan

243. (C) 2005, The University of Michigan

244. (C) 2005, The University of Michigan

245. Document level performance
TREC 8 corpus (200 questions)
164
163
149
#>0
1.3361
1.0495
0.8355
Avg
Google
NLight
AlltheWeb
Engine
(C) 2005, The University of Michigan

246. Sentence level performance
135
137
159
119
121 99
148
#>0
0.49
0.54
2.55
0.44
0.48
2.53
0.26
0.31
2.13
Avg
GO O
GO L
GO U
NL O
NL L
NL U
AW O
AW L
AW U
Engine
(C) 2005, The University of Michigan

247. Phrase level performance
0.199
0.157
0.117
0.105
Combined
0.0646
0.058
0.054
0.038
Global proximity
0.068
0.048
0.026
Appearance order
1.941
2.698
2.652
2.176
Upperbound
Google S+P
Google D+P
NorthernLight
AlltheWeb
Experiments performed Oct-Nov. 2001
(C) 2005, The University of Michigan

248. (C) 2005, The University of Michigan

249. (C) 2005, The University of Michigan

250. Text classification
(C) 2005, The University of Michigan

251. Introduction
Text classification: assigning documents to predefined categories
Hierarchical vs. flat
Many techniques: generative (maxent, knn, Naïve Bayes) vs. discriminative (SVM, regression)
Generative: model joint prob. p(x,y) and use Bayesian prediction to compute p(y|x)
Discriminative: model p(y|x) directly.
(C) 2005, The University of Michigan

252. Generative models: knn
K-nearest neighbors
Very easy to program
Issues: choosing k, b?
(C) 2005, The University of Michigan

253. Feature selection: The 2 testIt 1 C
k00
k01
k10
k11
For a term t:
Testing for independence: P(C=0,It=0) should be equal to P(C=0) P(It=0)
P(C=0) = (k00+k01)/n
P(C=1) = 1-P(C=0) = (k10+k11)/n
P(It=0) = (k00+K10)/n
P(It=1) = 1-P(It=0) = (k01+k11)/n
(C) 2005, The University of Michigan

254. Feature selection: The 2 test
High values of 2 indicate lower belief in independence.
In practice, compute 2 for all words and pick the top k among them.
(C) 2005, The University of Michigan

255. Feature selection: mutual information
No document length scaling is needed
Documents are assumed to be generated according to the multinomial model
(C) 2005, The University of Michigan

256. Naïve Bayesian classifiers
Assuming statistical independence
(C) 2005, The University of Michigan

257. Spam recognition (C) 2005, The University of Michigan
Return-Path:
X-Sieve: CMU Sieve 2.2
From: "Ibrahim Galadima"
Reply-To:
To:
Date: Tue, 14 Jan :06:
Subject: Gooday
DEAR SIR
FUNDS FOR INVESTMENTS
THIS LETTER MAY COME TO YOU AS A SURPRISE SINCE I HAD
NO PREVIOUS CORRESPONDENCE WITH YOU
I AM THE CHAIRMAN TENDER BOARD OF INDEPENDENT
NATIONAL ELECTORAL COMMISSION INEC I GOT YOUR
CONTACT IN THE COURSE OF MY SEARCH FOR A RELIABLE
PERSON WITH WHOM TO HANDLE A VERY CONFIDENTIAL
TRANSACTION INVOLVING THE ! TRANSFER OF FUND VALUED AT
TWENTY ONE MILLION SIX HUNDRED THOUSAND UNITED STATES
DOLLARS US$20M TO A SAFE FOREIGN ACCOUNT
THE ABOVE FUND IN QUESTION IS NOT CONNECTED WITH
ARMS, DRUGS OR MONEY LAUNDERING IT IS A PRODUCT OF
OVER INVOICED CONTRACT AWARDED IN 1999 BY INEC TO A
(C) 2005, The University of Michigan

258. Well-known datasets 20 newsgroups Reuters-21578 WebKB
Reuters-21578
Cats: grain, acquisitions, corn, crude, wheat, trade…
WebKB
course, student, faculty, staff, project, dept, other
NB performance (2000)
P=26,43,18,6,13,2,94
R=83,75,77,9,73,100,35
(C) 2005, The University of Michigan

259. Support vector machines
Introduced by Vapnik in the early 90s.
(C) 2005, The University of Michigan

260. Semi-supervised learning
Co-training
Graph-based
(C) 2005, The University of Michigan

261. Additional topics Soft margins VC dimension Kernel methods
(C) 2005, The University of Michigan

262. NB also good in many circumstances
SVMs are widely considered to be the best method for text classification (look at papers by Sebastiani, Christianini, Joachims), e.g. 86% accuracy on Reuters.
NB also good in many circumstances
(C) 2005, The University of Michigan

263. Readings
Books:
1. Ricardo Baeza-Yates and Berthier Ribeiro-Neto; Modern Information Retrieval, Addison-Wesley/ACM Press, 1999.
2. Pierre Baldi, Paolo Frasconi, Padhraic Smyth; Modeling the Internet and the Web: Probabilistic Methods and Algorithms; Wiley, 2003, ISBN:
Papers:
Barabasi and Albert "Emergence of scaling in random networks" Science (286) , 1999
Bharat and Broder "A technique for measuring the relative size and overlap of public Web search engines" WWW 1998
Brin and Page "The Anatomy of a Large-Scale Hypertextual Web Search Engine" WWW 1998
Bush "As we may thing" The Atlantic Monthly 1945
Chakrabarti, van den Berg, and Dom "Focused Crawling" WWW 1999
Cho, Garcia-Molina, and Page "Efficient Crawling Through URL Ordering" WWW 1998
Davison "Topical locality on the Web" SIGIR 2000
Dean and Henzinger "Finding related pages in the World Wide Web" WWW 1999
Deerwester, Dumais, Landauer, Furnas, Harshman "Indexing by latent semantic analysis" JASIS 41(6) 1990
(C) 2005, The University of Michigan

264. Readings
Erkan and Radev "LexRank: Graph-based Lexical Centrality as Salience in Text Summarization" JAIR 22, 2004
Jeong and Barabasi "Diameter of the world wide web" Nature (401) , 1999
Hawking, Voorhees, Craswell, and Bailey "Overview of the TREC-8 Web Track" TREC 2000
Haveliwala "Topic-sensitive pagerank" WWW 2002
Kumar, Raghavan, Rajagopalan, Sivakumar, Tomkins, Upfal "The Web as a graph" PODS 2000
Lawrence and Giles "Accessibility of information on the Web" Nature (400) , 1999
Lawrence and Giles "Searching the World-Wide Web" Science (280) , 1998
Menczer "Links tell us about lexical and semantic Web content" arXiv 2001
Page, Brin, Motwani, and Winograd "The PageRank citation ranking: Bringing order to the Web" Stanford TR, 1998
Radev, Fan, Qi, Wu and Grewal "Probabilistic Question Answering on the Web" JASIST 2005
Singhal "Modern Information Retrieval: an Overview" IEEE 2001
(C) 2005, The University of Michigan

265. More readings
Gerard Salton, Automatic Text Processing, Addison-Wesley (1989)
Gerald Kowalski, Information Retrieval Systems: Theory and Implementation, Kluwer (1997)
Gerard Salton and M. McGill, Introduction to Modern Information Retrieval, McGraw-Hill (1983)
C. J. an Rijsbergen, Information Retrieval, Buttersworths (1979)
Ian H. Witten, Alistair Moffat, and Timothy C. Bell, Managing Gigabytes, Van Nostrand Reinhold (1994)
ACM SIGIR Proceedings, SIGIR Forum
ACM conferences in Digital Libraries
(C) 2005, The University of Michigan

266. Thank you!
Благодаря!
(C) 2005, The University of Michigan