Crawling the Hidden Web Sriram Raghavan Hector Garcia-Molina Computer Science Department Stanford University Reviewed by Pankaj Kumar

Introduction What are web crawlers? Programs, that traverses Web graph in a structured manner, retrieving web pages. Are they really crawling the whole web graph? Their target: Publicly Index-able Web (PIW) They are missing something… 4/30/2015 Crawling Hidden Web2

What about results, which can only be obtained by: Search Forms Web pages, that need authorization. Let’s face the truth: Size of hidden web with respect to PIW High Quality information are present out there. Example – Patents & Trademark Office, News Media 4/30/2015 Crawling Hidden Web3

Now…The Goal: To create a web crawler, which can crawl and extract information from hidden database. Indexing, analysis and mining of hidden web content. But, the path is not easy: Automatic parsing and processing of form-based interfaces. Input to the form of search queries. 4/30/2015 Crawling Hidden Web4

Our approach: a. Task-specificity – Resource Discovery (will NOT focus in this paper) Content Extraction b. Human Assistance – It is critical, as it enables the crawler to use relevant values. gathers additional potential values. 4/30/2015 Crawling Hidden Web5

Hidden Web Crawlers A new operational model – developed at Stanford University. First of all… How a user interacts with a web form: 4/30/2015 Crawling Hidden Web6

Now, how a crawler should interact with a web form: Wait…what is this all about ??? - Let’s understand the terminologies first. That will help us. 4/30/2015 Crawling Hidden Web7

Terminologies: Form Page: Actual web page containing the form. Response Page: Page received in response to a form submission. Internal Form Representation: Created by the crawler, for a certain web form, F. F = ({E 1, E 2,…, E n }, S, M) Task-specific Database: Information, that the crawler needs. Matching Function: It implements the “Match” algorithm to produce value assignments for the form elements. Match(({E 1, E 2,…, E n }, S, M), D) = [E 1  v 1, E 2  v 2,…, E n  v n ] Response Analysis: Receives and stores the form submission in the crawler’s repository. 4/30/2015 Crawling Hidden Web8

Submission Efficiency (Performance): Let, N total = Total # of forms submitted by the crawler, N success = # of submissions which result in a response page containing one or more search results, and N valid = # of semantically correct form submissions. Then, a.Strict Submission Efficiency (SE strict ) = (N success ) / (N total ) b.Lenient Submission Efficiency (SE lenient ) = (N valid ) / (N total ) 4/30/2015 Crawling Hidden Web9

HiWE: Hidden Web Exposer HiWE Architecture: 4/30/2015 Crawling Hidden Web10

But, how does this fit in our operational model ???? Form Representation Task Specific Database (LVS Table) Matching Function Computing Weights 4/30/2015 Crawling Hidden Web11

LITE: Layout-based Information Extraction Technique What is it ?? A technique where page layout aids in label extraction. Prune the form page. Approximately layout the pruned page using Custom Layout Engine. Identify and rank the Candidate. The highest ranked candidate is the label associated with the form element. 4/30/2015 Crawling Hidden Web12

Experiments Task Description: Collect Web pages containing “News articles, reports, press releases, and white papers relating to the semiconductor industry, dated sometime in the last ten years”. Parameter values: ParametersValues Number of sites visited50 Number of forms encountered218 Number of forms chosen for submission94 Label matching threshold ( σ ) 0.75 Minimum form size ( α ) 3 Value assignment ranking function ρ fuz Minimum acceptable value assignment rank ( ρ min) 0.6 4/30/2015 Crawling Hidden Web13

Effect of Value Assignment Ranking function ( ρ fuzz, ρ avg and ρ prob ): Label Extraction: a.LITE: 93% b.Heuristic purely based on Textual Analysis : 72% c.Heuristic based on Extensive manual observation: 83% Ranking FunctionN total N success SE strict ρ fuz ρ avg ρ prob /30/2015 Crawling Hidden Web 14

Effect of α : Effect of crawler input to LVS table: 4/30/2015 Crawling Hidden Web15

Pros and Cons… Pros More amount of information is crawled Quality of information is very high More focused results Crawler inputs increases the number of successful submissions Cons Crawling becomes slower Task-specific Database can limit the accuracy of results Unable to process simple form element dependencies Lack of support for partially filled out forms 4/30/2015 Crawling Hidden Web16

Where does our course fit in here…?? In Content Extraction Given the set of resources, i.e. sites and databases, automate the information retrieval In Label Matching (Matching Function) Label Normalization Edit Distance Calculation In LITE-based heuristic for extracting labels Identify and Rank Candidates In maintaining Crawler’s repository 4/30/2015 Crawling Hidden Web17

Related Works… J. Madhavan et al, VLDS, 2008, Google's Deep Web Crawl J. Madhavan et al, CIDR, Jan. 2009, Harnessing the Deep Web: Present and Future Manuel Álvarez, Juan Raposo, Fidel Cacheda and Alberto Pan, Aug. 2006, A Task-specific Approach for Crawling the Deep Web Lu Jiang, Zhaohui Wu, Qian Feng, Jun Liu, Qinghua Zheng, Efficient Deep Web Crawling Using Reinforcement Learning Manuel Álvarez et al, Crawling the Content Hidden Behind Web Forms Yongquan Dong, Qingzhong Li, 2012, A Deep Web Crawling Approach Based on Query Harvest Model Alexandros Ntoulas, Petros Zerfos, Junghoo Cho, Downloading Hidden Web Content Rosy Madaan, Ashutosh Dixit, A.K. Sharma, Komal Kumar Bhatia, 2010, A Framework for Incremental Hidden Web Crawler Ping Wu, Ji-Rong Wen, Huan Liu, Wei-Ying Ma, Query Selection Techniques for Efficient Crawling of Structured Web Sources 4/30/2015 Crawling Hidden Web18

So…what’s the “Conclusion” ? Traditional Crawler’s limitations Issues related to extending the Crawlers for accessing the “Hidden Web” Need for narrow application focus Promising results of HiWE Limitations (of HiWE): Inability to handle simple dependencies between form elements Lack of support for partial filled out forms 4/30/2015Crawling Hidden Web19