1. Crawling the Hidden Web Sriram Raghavan Hector Garcia-Molina Computer Science Department Stanford University Reviewed by Pankaj Kumar

2. Introduction What are web crawlers? Programs, that traverses Web graph in a structured manner, retrieving web pages. Are they really crawling the whole web graph? Their target: Publicly Index-able Web (PIW) They are missing something… 4/30/2015 Crawling Hidden Web2

3. What about results, which can only be obtained by: Search Forms Web pages, that need authorization. Let’s face the truth: Size of hidden web with respect to PIW High Quality information are present out there. Example – Patents & Trademark Office, News Media 4/30/2015 Crawling Hidden Web3

4. Now…The Goal: To create a web crawler, which can crawl and extract information from hidden database. Indexing, analysis and mining of hidden web content. But, the path is not easy: Automatic parsing and processing of form-based interfaces. Input to the form of search queries. 4/30/2015 Crawling Hidden Web4

5. Our approach: a. Task-specificity – Resource Discovery (will NOT focus in this paper) Content Extraction b. Human Assistance – It is critical, as it enables the crawler to use relevant values. gathers additional potential values. 4/30/2015 Crawling Hidden Web5

6. Hidden Web Crawlers A new operational model – developed at Stanford University. First of all… How a user interacts with a web form: 4/30/2015 Crawling Hidden Web6

7. Now, how a crawler should interact with a web form: Wait…what is this all about ??? - Let’s understand the terminologies first. That will help us. 4/30/2015 Crawling Hidden Web7

8. Terminologies: Form Page: Actual web page containing the form. Response Page: Page received in response to a form submission. Internal Form Representation: Created by the crawler, for a certain web form, F. F = ({E 1, E 2,…, E n }, S, M) Task-specific Database: Information, that the crawler needs. Matching Function: It implements the “Match” algorithm to produce value assignments for the form elements. Match(({E 1, E 2,…, E n }, S, M), D) = [E 1  v 1, E 2  v 2,…, E n  v n ] Response Analysis: Receives and stores the form submission in the crawler’s repository. 4/30/2015 Crawling Hidden Web8

9. Submission Efficiency (Performance): Let, N total = Total # of forms submitted by the crawler, N success = # of submissions which result in a response page containing one or more search results, and N valid = # of semantically correct form submissions. Then, a.Strict Submission Efficiency (SE strict ) = (N success ) / (N total ) b.Lenient Submission Efficiency (SE lenient ) = (N valid ) / (N total ) 4/30/2015 Crawling Hidden Web9

10. HiWE: Hidden Web Exposer HiWE Architecture: 4/30/2015 Crawling Hidden Web10

11. But, how does this fit in our operational model ???? Form Representation Task Specific Database (LVS Table) Matching Function Computing Weights 4/30/2015 Crawling Hidden Web11

12. LITE: Layout-based Information Extraction Technique What is it ?? A technique where page layout aids in label extraction. Prune the form page. Approximately layout the pruned page using Custom Layout Engine. Identify and rank the Candidate. The highest ranked candidate is the label associated with the form element. 4/30/2015 Crawling Hidden Web12

13. Experiments Task Description: Collect Web pages containing “News articles, reports, press releases, and white papers relating to the semiconductor industry, dated sometime in the last ten years”. Parameter values: ParametersValues Number of sites visited50 Number of forms encountered218 Number of forms chosen for submission94 Label matching threshold ( σ ) 0.75 Minimum form size ( α ) 3 Value assignment ranking function ρ fuz Minimum acceptable value assignment rank ( ρ min) 0.6 4/30/2015 Crawling Hidden Web13

14. Effect of Value Assignment Ranking function ( ρ fuzz, ρ avg and ρ prob ): Label Extraction: a.LITE: 93% b.Heuristic purely based on Textual Analysis : 72% c.Heuristic based on Extensive manual observation: 83% Ranking FunctionN total N success SE strict ρ fuz 3214285388.8 ρ avg 3760312683.1 ρ prob 4316281065.1 4/30/2015 Crawling Hidden Web 14

15. Effect of α : Effect of crawler input to LVS table: 4/30/2015 Crawling Hidden Web15

16. Pros and Cons… Pros More amount of information is crawled Quality of information is very high More focused results Crawler inputs increases the number of successful submissions Cons Crawling becomes slower Task-specific Database can limit the accuracy of results Unable to process simple form element dependencies Lack of support for partially filled out forms 4/30/2015 Crawling Hidden Web16

17. Where does our course fit in here…?? In Content Extraction Given the set of resources, i.e. sites and databases, automate the information retrieval In Label Matching (Matching Function) Label Normalization Edit Distance Calculation In LITE-based heuristic for extracting labels Identify and Rank Candidates In maintaining Crawler’s repository 4/30/2015 Crawling Hidden Web17

18. Related Works… J. Madhavan et al, VLDS, 2008, Google's Deep Web Crawl J. Madhavan et al, CIDR, Jan. 2009, Harnessing the Deep Web: Present and Future Manuel Álvarez, Juan Raposo, Fidel Cacheda and Alberto Pan, Aug. 2006, A Task-specific Approach for Crawling the Deep Web Lu Jiang, Zhaohui Wu, Qian Feng, Jun Liu, Qinghua Zheng, Efficient Deep Web Crawling Using Reinforcement Learning Manuel Álvarez et al, Crawling the Content Hidden Behind Web Forms Yongquan Dong, Qingzhong Li, 2012, A Deep Web Crawling Approach Based on Query Harvest Model Alexandros Ntoulas, Petros Zerfos, Junghoo Cho, Downloading Hidden Web Content Rosy Madaan, Ashutosh Dixit, A.K. Sharma, Komal Kumar Bhatia, 2010, A Framework for Incremental Hidden Web Crawler Ping Wu, Ji-Rong Wen, Huan Liu, Wei-Ying Ma, Query Selection Techniques for Efficient Crawling of Structured Web Sources http://deepweb.us/ 4/30/2015 Crawling Hidden Web18

19. So…what’s the “Conclusion” ? Traditional Crawler’s limitations Issues related to extending the Crawlers for accessing the “Hidden Web” Need for narrow application focus Promising results of HiWE Limitations (of HiWE): Inability to handle simple dependencies between form elements Lack of support for partial filled out forms 4/30/2015Crawling Hidden Web19