1. Exploring the Internet Topic: Searching the Web 91.113-021 Instructor: Michael Krolak 91.113-031 Instructor: Patrick Krolak See Lecturer notes http://www.cs.uml.edu/~pkrolak/lab18/lab18.html http://www.cs.uml.edu/~pkrolak/lab18/lab18.html Authors: P. D. & M. S. Krolak Copyright 2005

2. Search for Information on the Web Finding information on the web requires some concepts of how the various types search engines work. Archives that capture the changes in the documents on the web are highly useful for those in the social sciences, technology, and business dynamics. “Intelligence is not the ability to store information, but to know where to find it.“ - Albert Einstein

3. How do we find information? Memory Media –Books –Movies –Music –Art Observe Ask other people

4. The Problem with the Internet The “Surface Web” contains 2.5 Billion pages. Each day 7.5 million web pages are added to the World Wide Web Information is submitted to the web without any context or test of validity

5. The Archives of the Web 1.Archival of the Web’s websites 2.Google’s archive of the Internet newsgroups. 3.Google’s archive of the world’s newspaper archives

6. The Way Back Machine Frustrated by dead links – there is an answer. The WayBack Machine at http://www.archive.org/http://www.archive.org/ Just fill in the URL of the dead link and the links history will give the history of the link (how the page changed over time) and allow you to view the dead link.

7. Google’s Newsgroup archive http://groups.google.com/ http://groups.google.com/ Archives over 100,000 groups Goes back for some groups over 30 years. There are for fee sites that provide competitive services. Depending on the group it can provide a treasure trove of insight into the cyber information society and it early history. Not all messages in the database are true, have merit or redeeming value, or are appropriate for children.

8. Google Newsgroup Google has a major effort to archive newsgroups

9. Google’s News Project Google plans to digitize the 250 years of published news and other data online.

10. Google’s news archive http://news.google.com/archivesearch http://news.google.com/archivesearch Google is creating an archive that will provide a free search of both free and fee publishers archives The goal is to provide search for articles going back to 1700. The project involves a partnership of many major information publishers

11. Google’s News archive The archive is located at: http://news.google.com/ar chivesearch The archive works like any other Google search and the results can be requested in time line order.

12. Searching for Information on the web 1.Search engines 2.Directory searches 3.Meta-Search engines

13. What is a Search Engine? search engine n. 1. A software program that searches a database and gathers and reports information that contains or is related to specified terms. 2. A website whose primary function is providing a search engine for gathering and reporting information available on the Internet or a portion of the Internet. Source: The American Heritage® Dictionary Copyright © 2002, 2001, 1995 by Houghton Mifflin Company. Published by Houghton Mifflin Company.

14. Search Engines Search engines have two parts: 1.The search sends out onto the Internet a software called a spider or bot (robot). Traces all the links and returns all the pages found. The pages are characterized by algorithms and stored in databases 2.The retrieval system that takes a query and maps against the databases. The retrieval rank orders the responses by relevance Each search engine uses a unique technique for retrieval and ranking.

15. What is a spider? n. 1. An automated program which crawls over the World Wide Web, gathering web pages for search engines. Spiders will ignore sites that explicitly state not be indexed by the search engines. Also referred to as a webcrawler, crawler, or bot

16. What are Meta Tags meta tags n. 1. Attributes that describe information about the content of the document. Some spiders use these tags to determine the relevance of a site to future queries. Example

17. How do search engines work?

18. Meta Search Engines Meta search engines are search engines that use their own resources for answering the question but they mostly form the query from the user input and package it and send it off to many other search engines simultaneously (the process is called spawning) and then wait until the replies come back. After a fixed time the meta takes the responses received and pulls them together into a report. There are many ways to create a meta search based on the idea. Some allow you to search only the web, others newsgroups, newspapers, and scientific journal.

19. Why is an understanding of how a search engine works important? From the view of a user: –The user wants to find the information with as few downloads as possible. –The easier to use and the more accurate the ranking the better. From the view of a web site developer: –The developer wants the site to found by in the first 5-10 ranked responses to a query. –The merit of a web design is often based on the search rankings. This requires a knowledge of a given search engine ranks a page.

20. When in doubt ask a librarian: The librarian is a trained professional and are well versed in using the various WWW resources for finding answers to a vast array of subjects. The librarian should be used for difficult searches; but the student will wisely observe, learn, and contemplate the librarian's techniques, resources, and methods.

21. What is a Subject Directory? subject directory n. 1. An Internet research tool on the World Wide Web that organizes Internet resources by subject headings and subheadings. Subject directories are usually compiled by human beings who apply some selection criteria to resources included in the database.

22. Examples of Subject Directories www.yahoo.com Yahoo!www.yahoo.com http://bubl.ac.uk/ BUBLhttp://bubl.ac.uk/ http://www.ipl.org/ Internet Public Libraryhttp://www.ipl.org/ www.about.com About.comwww.about.com www.jumpcity.com Jump Citywww.jumpcity.com http://www.joeant.com/ Joe Anthttp://www.joeant.com/

23. What is a Meta Search Engine? meta search engine n. 1. Meta search engines are search engines that use their own database as well as sending the query to many other search engines simultaneously (called spawning) and report the unique responses from other search engines. 2. Meta search engines that are limited to only the web, newsgroups, newspapers, and scientific journals.

24. Examples of Meta Search Engines Ask Jeeves -- frequently get the answer in the first pass. Jeeves allows queries in natural language.Ask Jeeves Dogpile -- for its variety of sources (web, newsgroups, newspapers)Dogpile Ixquick Metacrawler ProFusion

25. Current Information on search engines Search Engine Watch Is a source for comparing search engines and keeping up with innovations as they occur in the field.Search Engine Watch Recently Google was asked to turn over records about their customers search topics and the number of times pornographic information was accessed. The Federal government was looking to prove its case for protecting children. MSN and Yahoo had complied with the request. Google also recently complied with the Chinese government request to censure political inquiries on the Chinese version of Google.

26. The Deep Web

27. What is the invisible or deep web Invisible Web (n.) Also referred to as the deep Web, the term refers to either Web pages that cannot be indexed by a typical search engine or Web pages that a search engine purposely does not index, rendering the data “invisible” to the general user. One of the most common reasons that a Web site’s content is not indexed is because of the site’s use of dynamic databases, which opens the door for a potential spider trap. Web pages can also fall into the invisible Web if there are no links leading to them, since search engine spiders typically crawl through links that lead them from one destination to another. Data on the invisible Web is not inaccessible; the information is out there—it is stored on a Web server somewhere and can be accessed using a browser—but the data must be found using means other than the general-purpose search engines, such as Google and Yahoo!.WebWeb pagesindexed search engineWeb pagessearch enginedynamicdatabasesspider trapWeb serverbrowser search engines Source: http://www.webopedia.com/TERM/I/invisible_Web.html

28. The deep web The deep web is not mysterious, it simply means that normal search engines that use spiders that go from one link to another will not work with pages that are generated on the fly from data requested from a database, or not linked to other data, etc. Example of a deep website are the yellow or white pages, catalogues, and patents. Google can index search pdf, text, and word documents

29. What is the Deep Web? Estimated to be 500 times (1.25 trillion web sites) the size of the surface web.

30. Using the Search tools to find information of the web

31. Successful searching Plan your search: 1.What are the words that will only be on the right web page. Should they all be there or are there alternatives. The most specific concept is the best. 2.If you do not know your ideal topic well, use a meta search engine to get the smart. Then refine your search with a search engine like google or altavista. 3.Use a virtual library site to find information reviewed by experts if it is technical.

32. What is Boolean Logic? We use Boolean Logic to evaluate the truth of one or more propositions. There are three important operators: AND, OR, NOT AND – only true if A and B are both true. OR - only true if either A or B is true. NOT - only true when A is false. When searching for information, we use Boolean logic to find results that are relevant to our search terms. If a web page is relevant to a search term, the search engine evaluates the page as true.

33. Examples of Searching with Boolean Logic Yankees and Choke –All web pages that contain the terms Yankees and Choke. Yankees or Choke –All web pages that contain the word Yankees. –All web pages that contain the word Choke –All web pages that contain the terms Yankees and Choke Choke and not Yankees –All web pages that contain the word Choke, but don’t contain the word Yankees

34. More Advanced Uses of Boolean Logic If you are looking for a proper name, a phrase, or an other collection of words that normally are found together, then enclose them in double quotes, i.e. "President Gerald Ford". If the web page should have one or more words that must be on the page, then use the logical And, i.e. President And Ford And "United States". If the web page may have different forms of the name, or titles, etc. then use the logical Or, i.e. President Or "Vice President" Or Representative And "Gerald Ford". If document should exclude a word or phrase, then use the logical Not, i.e. "Gerald Ford" Not "Ford automotive" and Not "Ford car" and Not "Ford truck".

35. Other Helpful Hints While not Boolean logic, some search engines allow concepts like -- NEAR and FOLLOWED BY are also allowed, to indicate the relationship of the words or phrases other words and phrases. Normally these relations can be which comes first or whether the word is within a certain number of words to the first word. This concept is called proximity logic. Not all search engines use the AND, OR, NOT notation some like Alta Vista use " +" for AND and "-" for NOT.

36. Tips for Using Search Engines When searching for a large scale database, it is important to be extremely precise. Avoid using vague or common words that will only produce millions of pages. Read the instructions for each new search engine you use. There are many different methods of searching between the search engines and subject directories.

37. Finding Audio and Video http://www.alltheweb.com/ video, audio, newshttp://www.alltheweb.com/ http://images.google.com – Good source of imageshttp://images.google.com www.dogpile.com – One of the few search engines that provides searches for video.www.dogpile.com www.fazzle.com – Provides limited video and image searching capabilitieswww.fazzle.com http://video.google.com/ -- A new beta product may have bugs.http://video.google.com/ http://www.tssphoto.com/ Stock photo imageshttp://www.tssphoto.com/

38. Finding Movies and Films http://www.archive.org/details/movies

39. Sources of Audio Information An Audio Archive (software & Music, etc.) http://archive.museophile.org/audio/ http://archive.museophile.org/audio/ Speeches – http://www.historychannel.com/speeches/ http://www.historychannel.com/speeches/

40. Dogpile for finding non- text based files The number of sites that allow so called “anonyms or guest” ftp directories is now greatly diminished. Due to security considerations most sites do not have non-text directories that are open to search and file download. Hence Dogpile no longer maintains a search engine that can find files in ftp sites, but it still allows searches for images, audio, and videos.Dogpile Similarly tools like Archie and Gopher are now obsolete