The Invisible Web Also called: deep content hidden internet dark matter
The Invisible Web The vast number of pages that search engines cannot or will not index Restricted: login, password (such as intranets, databases; private, proprietary) Sites not linked from anywhere (undiscovered) Sites that use a robots.txt file to keep files off limits from spiders Unsearchable or un-indexable file formats Non-static - searchable databases that only produce results dynamically in response to a specific search request (such as CGI, ASP, CFM) Real-time data – changes rapidly – too “fresh” Sites that are too “deep”
The Invisible Web Search engines often avoid indexing web pages that are delivered dynamically, such as via database programs: Often, the search engine may not like the URL used in order to retrieve the document. Many dynamic delivery mechanisms make use of the ? symbol. For example, a page may be found this way: http://www.website.com/cgi-bin/getpage.cgi?name=sitemap Most search engines will not read past the ? in that URL.
The Invisible Web Invisible Web sources tend to be: More current More comprehensive Searchable (however, not by SE’s) More specific/targeted Deeper breadth Often better quality
The Invisible Web Top types of “invisible” information News RSS Blogs Public company filings, stock prices Customized maps and directions Clinical trials Telephone numbers and addresses, postal codes Definitions Job postings Grant information Statistics Weather Museum, gallery, and library holdings
Traditional Search Engines Traditional Search Engines incorporation of “Invisible” Databases Weather Maps Phone directories Catalogs Stock prices
Traditional Search Engines Unless specially, programmed, though, spiders can’t find all the valuable resources available
Specialized Search Engines Search deeper into sites: Go beyond top page, or homepage Choose sources to spider—topical sites only “Smart” ranking and indexing based on knowledge of the specific subject
Specialized Search Engines There are hundreds of specialized search engines for almost every topic- Search Engine Guide Specialty Search Engines
Vortals Vortals: vertical-portal. Instead of being a horizontal, all-inclusive entry point into the Web, they are vertical, specialized entry points. Comprehensive sites focusing on gathering and providing links to the best resources in a specific topic. Usually are combined subject-specific search engines and subject-specific directories Also called “focused crawlers”; metasites; guru; authority; industry guide; subject directory site
Vortals Advantages – best of directories and subject specific search engines More up-to-date - crawl subject specific pages more often Deeper crawl - gets more of the content on each server More precision, less recall
Searching the Invisible Web How do you find these sites? Use directories known directories to find invisible web searching and browsing tools: Librarians’ Index to the Internet Open Directory Google Directory Teoma works well, too.
Searching the Invisible Web Rethink your search: Think key terms specific details – macro vs. micro Example you want to find the melting point of hydrogen peroxide. On the general web, you’d put in the key words melting, point, and “hydrogen peroxide” On the invisible web, you look for chemical databases, which included melting points as one feature of the database, once in the database, then you’d search for hydrogen peroxide
Searching the Invisible Web Remember some concepts are assumed Do not use the subject a search term Example: If you are looking for information on gender inequity in math education, exclude terms like education from your search in AskERIC, an education specific search tool
Mining the Invisible Web Tips: Certain kinds of sites can prove to be clearinghouses of information: Government - statistics of all kinds Professional organizations - archives of relevant research and statistics Media sites (TV and Radio) – transcripts and speeches College and university professor sites – lectures and personal publications
Mining the Invisible Web Look for library guides and commercial portals for more guidance in finding the hidden, valuable content available for free on the Web (more on this in the next lesson): My Ready Reference on the Web Resource