1. Automating the Extraction of Data Behind Web Forms Automating the Extraction of Data Behind Web Forms Brigham Young University Sai Ho Yau

2. NextPrevious Hurdles Against Automating Data Extraction There are enormous amounts of information available from the Web, but it is difficult to extract the data automatically due to several reasons: Web information is stored in databases Form interfaces Relevant information can be obtained only after a Web form is filled out and submitted

3. NextPrevious Problems Dealing with Forms No general Web form design Required text fields One form may lead to another Resulting information embedded within forms Returned error messages versus valid data Elimination of possible duplicate data

4. NextPreviousMotivations Eliminate duplicate data and merge resulting information. We want to automatically: Fill in Web forms. Extract information behind forms. Screen out errors.

5. NextPrevious The Framework

6. NextPrevious Method: Construct the Query String

7. NextPrevious Method: Construct the Query String

8. NextPrevious Method: Construct the Query String

9. NextPrevious Returned Web Page

10. NextPreviousSolutions Two phases to deal with many possible responses to a query*: Sampling phase Exhaustive phase * Assuming no HTTP error

11. NextPrevious Sampling Phase Submit the default form. Randomly select N form-field settings and submit the form N times. If no new information, STOP and send the result downstream (N is set so that the probability of subsequent submissions yielding new data is less than 5%). Otherwise, ENTER the Exhaustive Phase.

12. NextPrevious Exhaustive Phase Estimate the total time and quantity of data. If below threshold, exhaustively obtain the rest of the information. Otherwise, return the results of the sampling and report to the user the estimate of time and quantity of data.

13. NextPrevious Data Retrieving Strategy Locate possible duplicate information from subsequent retrieved Web pages during Sampling and Exhaustive Phases.

14. NextPrevious Retrieved Web Pages

15. NextPrevious Data Retrieving Strategy Locate possible duplicate information from subsequent retrieved Web pages during Sampling and Exhaustive Phases. Discard duplicates and merge new information.

16. NextPrevious Duplicates Discarded and New Information Merged

17. NextPrevious Data Retrieving Strategy Locate possible duplicate information from subsequent retrieved Web pages during Sampling and Exhaustive Phases. Discard duplicates and merge new information. Send fully merged data downstream for data extraction.

18. NextPreviousConclusions Filter duplicate data and merge resulting information. We can automate data extraction process by automatically: Fill in Web forms. Retrieve information behind forms. Handle errors.