1. Search Engines & Subject Directories
Internet Research
Search Engines & Subject Directories

2. Search engines
Search engines are the means by which most people search the Web.
Common examples are Google, Altavista, Direct Hit.

3. Yet they don’t search the Internet
Yet a search engine does not actually search the Web during your search.
A search engine searches itself.
It’s a three-step process.

4. 1) Bots index words
Search engines continually send out hundreds of “robots” or “bots” (or “spiders” or “crawlers” )
Bots visit web sites, read word by word, and then index those words.

5. 2) A database is created
A huge database of Web sites thus is gathered and indexed by word.
These databases can be huge, with millions of links.

6. 3) The Interface gives you access
Using the keywords you give it, a search engine then searches its own current index.

7. Interfaces are based on rankings
Search engines return results based on a ranking system.
Ranking is the order that files are listed when they are retrieved.

8. The ranking system is secret
These systems are proprietary and often “secret.” In general:
Altavista ranks web pages higher if your search terms are found in the first few words of the page
Google ranks by document “popularity” with other similar searches
Direct Hit ranks by the length of time other users spent at the site

9. Not even half the Web
With all of this software and sophistication, even the best search engines cover only 40-50% of the Web.
And they miss much else on the Internet.

10. Bots hit and miss Bots miss: XML pages, pdf files
Dynamically created HTML pages
Frames-based pages
New pages or recent updated text
Some say the Invisible Web is 500 times larger than Web

11. Subject Directories
A subject directory is also a database of web sites and references.
But a subject directory is organized not by keywords but by category or subject.

12. Yahoo! Yahoo! Is the most popular subject directory.
www. about.com takes the idea a step further with subject guides for selected topics.

13. Subjects are organized by people.
Information is selected, organized and cataloged by a person, not software.
You can usually be more assured that the search results will make sense.

14. You get an index of sites.
Subject directories will not often provide you with ranked web sites.
Instead, you will get a broad index related to your topic, divided further by subheadings.

15. Use for early searching.
Use a subject directory early in your search process to learn about your subject.
You will get fewer links of higher quality.
When you get more specific questions, you should use a search engine.