Uncovering the Invisible Web

Back in the day… Students used to research using resources hand-picked by librarians and teachers. These materials were selected because: They were written by expert authors They presented information without bias They were written at your reading levels Now, with access to the Web, you have to learn to find and evaluate resources too!

Today you will learn: How much information can search engines find on the web? What kind of information can you not typically find through a search engine? How does information in a library database compare to information found through a Google search?

It finds all we need to find, right guys?

Think again. Search engines access a relatively small part of the Web, known as “The Free Web” The large part of the Web that search engines can’t access is known as “The Invisible Web” or “The Deep Web”

Subscription databases Archives E-Books Estimated to be times size of the visible Web!

What else is not on the Free Web? Print books still under the protection of copyright. This means around 90% of the books on our library’s shelves! Full-text, searchable archives of journal, magazine, and newspaper articles

The “Invisible web” is information you cannot retrieve from search results. Information from this part of the web is not “crawled” by a search engine and is thus “invisible” to a searcher who does not know it exists. What’s the Invisible Web?

Search engines “crawl” the web like a spider, indexing some of the information found on a web page. Indexing is the process of recording that information along with its location so that people can search for it and find it in a database. A database is really just a collection of information that is indexed or organized for people to search. Spiders crawl the web?

How do search engines find web pages? Search engines find some sites through links on existing web pages, and other sites are submitted directly to the search engine by the person who created the website. It can take 2-6 months for a search engine to crawl a new web site. Each search engine crawls the web and sorts the information found in a different way.

Why search engines can’t access every page on the Web: The material is in a database that you have to pay to access (library databases) The page is available only after registration A security firewall prevents access to the page The information is private and owned by a company or organization

Some more reasons… The page is available by some search engines but not others. No two search engines are the same. The search engine crawler does not search a particular file format or non-text interface (Flash files) The site may be new, and hasn’t been crawled yet. Some pages just aren’t linked. You must know the URL.

Smarter search engines The larger search engines like Google already index information contained within media such as PDF files, word processor documents, spreadsheets, etc. But some information will never be available for free on the Internet.

Why can’t we get this stuff for free?

Publishing is a business! Professional authors are in the business of earning money for their hard work. Publishers also expect to earn money from publishing and selling an author’s work. Copyright laws protect the work of authors, artists, and anyone who creates information or other media.

Your teachers expect quality! Because material from library books and subscription databases is written by experts, your teachers may prefer you to use these resources instead of searching the Free Web. You may even have to defend some of the sources you find on the free Web!