Week 9 Search Engines and the Invisible Web

Resource Pages Collections of Links Compiled by “experts” Sometimes annotated Targeted Information for a Specific User Group Examples: Voice of the Shuttle: Computer Science Research Guide:

Anatomy of a Search Engine Basically, there are three parts to a search engine: “Spider” or “Crawler” - Finds the pages - Brings them home “Index” or “Database” - Storehouse of pages - Size matters, frequency of updates matters “Search Tool” - What we use to find the pages in the engine’s index - This is the user interface; the only part we see

How Search Engines Rank Pages Relevance retrieval Location of search terms Frequency of search terms Meta-tags (in the HTML source code of a Web page)

Positions of Words Term Co-Occurrence Proximity Pay for Placement “Featured Web Sites!”“Featured Web Sites!” Link Analysis Search Engine Showdown Chart: Other Ranking Methods

What Many Search Engines Cannot Find Some file types: some engines can, some cannot Dynamically-generated pages Pages locked behind firewalls or in fee-based online databases (such as Dialog) Lots of the “Deep Web” stuff:

Differences Between the “Deep Web” and Search Engine Results The Deep Web is another phrase for the Invisible Web Deep Web resources are usually: Subject specific / more focused Less content but tends to be of higher quality Updated more frequently Have specialized search interfaces Have a target audience in mind

Overview of the Deep Web What is Still Invisible: Disconnected, loose pages Password-protected pages and sites “robots.txt” files Dynamically-created pages: no static URLs Information bound in database structures that are uncrawlable by many search engines

When to Consider the Deep Web When you are familiar with a topic When you want authoritative information When you want specific information When you want timely information

Popular Deep Web Information Clinical Trials Environmental Information Grant Information Historical Documents and Images Art Collections Patents Demographic and Economic Data Government Information

Look at Some Deep Web Resources Salary.com Database U.S. Patent & Trademark Office Los Angeles Municipal Code

How to Find the Deep Web Use a search engine: search “database” as a term Use a print directory: try OCLC WorldCat to find those specific to your subject need Ask your colleagues Take note in the professional literature

How to Find the Deep Web (cont.) Use Alerting Services: The Scout Report (Internet Scout Project) INFOMINE

Evaluation of Web-Based Information Continuously evaluate as you look at “information” on the free Web. The key principles to look for are: Currency / Timeliness Authenticity Objectivity Completeness and Accuracy Verifiability Example: Thinking Critically about Web 2.0 and Beyond

Staying Current Subscribe to alerting services for Deep Web resources Look at reviewing tools Research Buzz: Search Engine Watch and Search Engine Report

Search Engines Don’t Find Information—People Do! Use the right combination of tools for the job, including offline (paper) resources Use the right tools the best way possible Sometimes a search engine, Deep Web resource or other Web finding tool is not appropriate to the information need A “good” search engine is one that finds what you want.