1. 1 Searching the Web Prepared By: Hasan Ba-Abdullah. 425121603 Supervised By: Dr. Mourad Ykhlef King Saud University College of Computer & Information Sciences Information System Department IS 531:Document Storage and Retrieval Systems

2. 2 Agenda 1. Introduction 2. Challenges of Searching the Web 3. Measuring the Web 4. Searching Engines (Google) 5. Web Directories 6. Metasearchers 7. Google Searching Guidelines

3. 3 1. Introduction The Web can be seen as a very large, unstructured but ubiquitous database. So we need for efficient tools to manage, retrieve and filter the information. There are 3 different forms of searching the Web: 1. Search Engines, which index a portion of Web pages as a full-text database. 2. Web Directories, which classify selected Web documents by subject. 3. Searching by hyperlinks structure.

4. 4 2. Challenges of Searching the Web Problem with the data itself: Distributed data High percentage of volatile data it is estimated that 40% of the Web changes every month. Unstructured and redundant data No conceptual model, no organization, no constraints. By some estimates, about 30% of the Web is redundant. Quality of data Data can be false, invalid, outdated, poorly written or with many errors. Heterogeneous data Multiple media types, multiple formats, languages and alphabets. Problems regarding the user and his interaction with the retrieval system: How to specify a query How to interpret the answer provided by the system.

5. 5 3. Measuring the Web Detailed Domain Counts and Internet Statistic. Source: http://www.whois.sc/internet-statistics/

6. 6 3. Measuring the Web (Cont.) on April 1st 2006. Source: http://www.whois.sc/internet-statistics/country-ip-counts.html

7. 7 4. Search Engine A search engine is a program designed to help find information stored on a computer system such as the World Wide Web, or a personal computer. The search engine allows one to ask for content meeting specific criteria and retrieves a list of references that match those criteria. Two main architectures: 1. Centralized : Using crawlers, information is gathered into a single site, where it is indexed; the site then processes all user queries. 2. Distributed : Searching is a coordinated effort of many information gatherers and brokers.

8. 8 4.1 Centralized Architecture Most search engines uses a centralized crawler-indexer architecture. Components: Crawlers, Index, Query Engine, and Interface.

9. 9 4.2 Distributed Architecture  Harvest is an example of distributed architecture.  Main drawback: requires the coordination of several Web servers.  Components: 1.Gatherers: Extracts information from the documents stored on one or more Web servers. Can handle documents in many formats: HTML, PDF, Postscript, etc. 2.Broker: provides the indexing mechanism and query interface. 3. Replicator: to replicate servers. 4. Object Cache: reduces network and server load.

10. 10 4.2 Distributed Architecture (Cont.)

11. 11 4.3 About Google? The name "Google" is a play on the word "googol", which refers to the number represented by 1 followed by one hundred zeros. Google receives over 200 million queries each day through its various services. As of January 2006, Google has indexed 9.7 billion web pages, 1.3 billion images, and over one billion Usenet messages — in total, approximately 12 billion items. It also caches much of the content that it indexes.

12. 12 User Interfaces

13. 13 Google Services and Tools Source: http://en.wikipedia.org/wiki/List_of_Google_services_and_tools

14. 14 How Google works

15. 15 Google finds important pages The idea is that the documents on the web have different degrees of "importance". Google will show the most important pages first. The ideas is that more important pages are likely to be more relevant to any query than non-important pages.

16. 16 Google Relevance Factors Google's considers over 100 factors, including: 1. PageRank algorithm. 2. Popularity of page. 3. Position and size of the search terms within page. 4. Unique Content. 5. Terms order. 6. Page size and load time. 7. Error free websites. 8. Important incoming links. 9. Website Optimization.

17. 17 Google PageRank Numeric value to measure how important a page is. PageRank (PR) is the actual ranking of a page, as determined by Google. A probability is expressed as a numeric value between 0 and 1. Toolbar PageRankReal PageRank 0/100.15-0.9 1/100.9-5.4 2/105.4-32.4 3/1032.4-194.4 4/10194.4-1,166.4 5/101,166.4-6,998.4 6/106,998.4-41,990.4 7/1041,990.4-251,942.4 8/10251,942.4-1,511,654.4 9/101,511,654.4-9,069,926.4 10/109,069,926.4-0.85 × N + 0.15

18. 18 Google System Features A T1 T2 Tn C1 C2 Cm PageRank – Bring order to the web –PR(A) = (1-d) + d (PR(T1)/C(T1) + ….. + PR (Tn)/C(Tn)) PR(A) is the PageRank of page A. PR(T1) is the PageRank of the page that links to our (A) page. C(T1) is the number of links going out of page T1. d is a damping factor, usually set to 0.85.

19. 19 Example PageRank calculation: PR(A) = 0.5 + 0.5 PR(C) PR(B) = 0.5 + 0.5 (PR(A) / 2) PR(C) = 0.5 + 0.5 (PR(A) / 2 + PR(B)) These equations can easily be solved. We get the following PageRank values for the single pages: PR(A) = 14/13 = 1.07692308 PR(B) = 10/13 = 0.76923077 PR(C) = 15/13 = 1.15384615

20. 20 For example, the word "civil" might occur in documents 3, 8, 22, 56, 68, and 92, while the word "war" might occur in documents 2, 8, 15, 22, 68, and 77. Suppose someone comes to Google and types in civil war. In order to present and score the results, we need to do two things: 1. Find the set of pages that contain the user’s query somewhere 2. Rank the matching pages in order of relevance Indexing

21. 21 Web crawler A web crawler (also known as a web spider) is a program which browses the World Wide Web in a methodical, automated manner. Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine, that will index the downloaded pages to provide fast searches. It starts with a list of URLs to visit. As it visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, recursively browsing the Web according to a set of policies.

22. 22 Google Search Engine Architecture URL Server- Provides URLs to be fetched Crawler is distributed Store Server - compresses and stores pages Repository - holds pages for indexing Indexer - parses documents, records words, positions, font size, capitalization Lexicon - list of unique words found Barrels hold Anchors - keep information about link found in web pages URL Resolver - converts relative URLs to absolute Sorter - generates Doc Index Doc Index - inverted index of all words in all documents (except stop words) Links - stores info about links to each page (used for Pagerank) Pagerank - computes a rank for each page retrieved Searcher - answers queries

23. 23 5. Web Directories Web directory : A classification of Web pages by subject. Principles: Classification is by a hierarchical taxonomy. Directory may be specific to a subject, a region, a language. Pages are submitted and reviewed before they are included. Automatic classification is not successful enough. Advantage: if found, the answer will be useful in most cases; Disadvantage: classification is not specialized enough; not all Web pages are classified;

24. 24 6. Metasearchers Metasearcher : web server that sends a given query to several search engines and Web directories, collects the answers and unifies them. –Examples: Metacrawler, Savvysearch, MetaSearch, Mamma. Advantages: –Combine the results of many sources. –Save users from the need to pose queries to multiple searchers. –Ability to sort the results by different attributes. –Pages retrieved by multiple searchers are more relevant. –Improve coverage: individual searchers cover a small fraction of the Web. Issues: –How to translate the given query to the specific language of each search Engine? –How to rank the unified results?

25. 25 7. Google Searching Guidelines Query modifiers –Use these commands in the search window. intitle:test allintitle:test results inurl:testresults allinurl:testresults personality allintext:test results personality allinanchor:test results personality site:loc.gov filetype:doc

26. 26 intitle:test results This search returns sites with the word test in the title and results anywhere in the document.

27. 27 inurl:test results inurl:test results – only test must be found in the web address (URL)

28. 28 allintext Sometimes you get pages that do not have your search term/phrase in them. Use allintext to get only those pages that have your search terms in them. –Compare the searches in the next two slides…

29. 29 Example: crash test results

30. 30 allintext:crash test results Different pages float to the top of your “hit list”. And you get fewer pages than before.

31. 31 site: Limit your search to a specific web site. Enter search terms then qualifier. EXAMPLES: –“students” site:ksu.edu Finds student(s) on the King Saud University site

32. 32 filetype: You can specify a type of document to search. EXAMPLES: –pdf – Adobe readable files –doc – Microsoft Word documents –mdb – Microsoft Access databases –jpg, gif, tif – graphics, photos –ppt – Microsoft PowerPoint presentations

33. 33 define: will provide definitions of the words, gathered from various online sources.

34. 34 Funny Google News Google Bombing “Miserable Failure” قررت شركة غوغل استغلال سطح القمر للترويج عن لنفسها ومنتجاتها وذلك من خلال وضع شعارها على سطح القمر

35. 35 Summary Search engines are among the most important applications or services on the web. The success of the Google search engine was mainly due to its simple, easy-to-use, no-ad interface, and its powerful PageRank algorithm.

36. 36 Thanks Any Questions