1. The Search Engine Architecture
CSCI 572: Information Retrieval and Search Engines
Summer 2011

2. Outline Introduction Google Summary The PageRank algorithm
The Google Architecture
Architectural components
Architectural interconnections
Architectural data structures
Evaluation of Google
Summary

3. Problems with search engines circa the last decade
Human maintenance
Subjective
Example: Ranking hits based on $$$
Automated search engines
Quality of result
Neglect to take user’s context into account
Searching process
High quality results aren’t always at the top of the list

4. The Typical Search Engine Process
In what stages is the most time spent?

5. How to scale to modern times?
Currently
Efficient index
Petabyte scale storage space
Efficient Crawling
Cost effectiveness of hardware
Future
Qualitative context
Maintaining localization data
Perhaps send indexing to clients
Client computers help gather Google’s index in a distributed, decentralized fashion?

6. Google The whole idea is to keep up with the growth of the web
Design Goals: -Remove Junk Results -Scalable document indices
Use of link structure to improve quality filtering
Use as an academic digital library
Provide search engine datasets
Search engine infrastructure and evolution

7. Google Archival of information Leverage of usage data
Use of compression
Efficient data structures
Proprietary file system
Leverage of usage data
PageRank algorithm
Sort of a “lineage” of a source of information
Citation graph

8. PageRank Algorithm With damping factor d
Numerical method to calculate page’s importance
this approach might well be followed by people doing research
Page Rank of a page A
With damping factor d
Where PR(x) = Page Rank of page X
Where C(x) = the amount of outgoing links from page x
Where T1…Tn is the set of pages with incoming links to page A
PR(A)=(1-d)+d(PR(T1)/C(T1)+…+PR(Tn)/C(Tn))
It’s actually a bit more complicated than it first looks
For instance, what’s PR(T1) and PR(T2) and so on?

9. PageRank Algorithm An excellent explanation
Since the PR(A) equation is a probability distribution over all web pages linking to web page A…
And because of the (1-d) term and the d*(PR….) term
The PageRanks of all the web pages on the web will sum to 1

10. PageRank: Example So, where do you start?
It turns out that you can effectively “guess” what the PageRanks for the web pages are initially
In our example, guess 0 for all of the pages
Then you run the PR function to calculate PR for all the web pages iteratively
You do this until…
The page ranks for each web page stop changing in each iteration
They “settle down”

11. Below is the iterative calculation that we would run
PageRank: Example
Below is the iterative calculation that we would run
PR(a) = 1 - $damp + $damp * PR(c);
PR(b) = 1 - $damp + $damp * (PR(a)/2)
PR(c) = 1 - $damp + $damp * (PR(a)/2 + PR(b) + PR(d));
PR(d) = 1 - $damp;

12. PageRank Algorithm: First 18 iterations
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
Still changing too much

13. PageRank: next 13 iterations
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d:
Starting to stabilize

14. PageRank: Last 9 iterations
a: b: c: d:
a: b: c: d:
a: b: c: d:
a: b: c: d: Average pagerank =
Stabilized

15. Google Architecture Key components Interconnections Data structures
A reference architecture for search engines?

16. Google Data Components
BigFiles
Repository
Use zlib to compress
Lexicon
Word base
Hit Lists
Word->document ID map
Document Indexing
Forward Index
Inverted Index

17. Google File System (GFS)
BigFiles
A.k.a. Google’s Proprietary Filesystem
64-bit addressable
Compression
Conventional operating systems don’t suffice
No explanation of why?
GFS:

18. Google Key Data Components
Repository
Stores full text of web pages
Use zlib to compress
Zlib less efficient than bzip
Tradeoff of time complexity versus space efficiency
Bzip more space efficient, but slower
Why is it important to compress the pages?

19. Google Lexicon Lexicon Why is it important to have a lexicon?
Contains 14 million words
Implemented as a hash table of pointers to words
Full explanation beyond the scope of this discussion
Why is it important to have a lexicon?
Tokenization
Analysis
Language Identification
SPAM

20. Mapping queries to hits
HitLists
wordID->(docID,position,font,capitalization) mapping
Takes up most of the space in the forward and inverted indices
Types: Fancy,Plain,Anchor

21. Document Indexing Document Indexing Forward Index Inverted Index
docIDs->wordIDs
Partially sorted
Duplicated doc IDs
Makes it easier for final indexing and coding
Inverted Index
wordIDs->docIDs
2 sets of inverted barrels

22. Crawling and Indexing Crawling Indexing Distributed, Parallel
Social issues
Bringing down web servers: politeness
Copyright issues
Text versus code
Indexing
Developed their own web page parser
Barrels
Distribution of compressed documents
Sorting

23. Google’s Query Evaluation
1: Parse the query
2: Convert words into WordIDs
Using Lexicon
3: Select the barrels that contain documents which match the WordIDs
4: Search through documents in the selected barrels until one is discovered that matches all the search terms
5: Compute that document’s rank (using PageRank as one of the components)
6: Repeat step 4 until no documents are found and we’ve went through all the barrels
7: Sort the set of returned documents by document rank and return the top k documents

24. Google Evaluation Performed by generating numerical results
Query satisfaction
Bill Clinton Example
Storage requirements
55GB Total
System Performance
9 days to download 26 million pages
63 hours to get the final 11 million (at the time)
Search Performance
Between 1 and 10 seconds for most queries (at the time)

25. Wrapup Loads of future work Even at that time, there were issues of:
Information extraction from semi-structured sources (such as web pages)
Still an active area of research
Search engines as a digital library
What services, APIs and toolkits should a search engine provide?
What storage methods are the most efficient?
From 2005 to 2010 to ???
Enhancing metadata
Automatic markup and generation
What are the appropriate fields?
Automatic Concept Extraction
Present the Searcher with a context
Searching languages: beyond context-free queries
Other types of search: Facet, GIS, etc.

26. The Future? User poses keyword query search
“Google-like” result page comes back
Along with each link returned, there will be
A “Concept Map” outlining – using extraction methods – what the “real” content of the document is
This basically allows you to “visually” see what the page rank is
Discover information visually
Existing evidence that this works well
Carrot2/3 clustering

27. Software Architecture
Concept Map
Chris’s Homepage
Data
Publications
Software
Data Grid
Science Data Systems
Software Architecture