1. Data Connections -- Options, Functionality, and Performance

2. Objective Use VBA code to extract and store data from web pages when the underlying database is not accessible. Step one – Accessing the web pages Step two – Extracting the data

3. Accessing the Web Pages Internet Explorer Object WinHttpRequest Object XMLHttpRequest object

4. Document Object Model (DOM) Regular Expressions Retrieving the Data

5. IE Object Dim objIE As Object Dim varTables As Variant Set objIE = CreateObject("InternetExplorer.Application") URL = "http://google.com" objIE.Visible = False objIE.navigate URL Do Until Not objIE.Busy DoEvents Loop While objIE.Document.ReadyState <> "complete" Wend Set varTables = objIE.Document.all.tags("TABLE")

6. Advantages View web page in action Credentials Background tasks DOM Disadvantages Hangs up Can interfere with other browsers More resources IE Object

7. WinHttpRequest Object Dim winHttpReq As Object Set winHttpReq = CreateObject("WinHttp.WinHttpRequest.5.1") URL = "http://greensboro.usps.gov/Operations/SETIarea/SETI_ReProcessScans.cfm? requesttimeout=5000&SDate=" & ProcDate winHttpReq.SetTimeouts 6000000, 6000000, 6000000, 6000000 winHttpReq.Open "GET", URL, False winHttpReq.SetCredentials “username", “password", HTTPREQUEST_SETCREDENTIALS_FOR_SERVER winHttpReq.Send result = winHttpReq.responseText Set winHttpReq = Nothing

8. Advantages More timeout control Waits for webpage to complete Disadvantages Cannot see what is returned May need Credentials No DOM WinHttpRequest Object

9. Time Trial IE ObjectWinHttpRequest Object 470497 452691 414259 Using two different versions of the Scan Error Tracking program, we ran three separate scans of 542 zip codes. Execution time is in seconds.

10. DOM vs RegEx The DOM is preferred when parsing web pages where the data is in uniform locations, using tables and rows. Regular Expressions work best when attempting to find data on a page where the location of the information, or the structure of the page is not known in advance.