X-PERT: Accurate Identification of Cross-Browser Issues in Web Applications Shauvik Roy Choudhary, Mukul Prasad, Alessandro Orso Labs of America Partially supported by

Web Applications 2

3

4 Mozilla Firefox Internet Explorer

5 Mozilla Firefox Internet Explorer

6 Google Chrome

7 Apple Safari

8 Mozilla Firefox

9

10 Mozilla Firefox Internet Explorer

11 Mozilla Firefox Internet Explorer

Web Developer

Browser versions last year Source: StatCounter.com

Browser versions last year with at least 1% market share 150mn visits/month ( StatCounter.com)

Challenges Manual inspection is expensive DOM differs between browsers Mimic end user’s perception Produce readable, effective & actionable reports Engineering Issues… Modern apps have many dynamic screens

Detection of Cross-Browser Issues (XBI) Web Application Model Generation Model Comparison Error Report

Model Definition and Comparison WebDiff [ICSM’10] Roy Choudhary and Orso CrossCheck [ICST’12] Roy Choudhary, Prasad and Orso CrossT [ICSE’11] Mesbah and Prasad Screen Transition Graph Screen Model Screen image + geometry DOM Tree

Study of Real World Cross-Browser Issues (XBIs) Examined 100 websites Found 23 with XBIs XBI Types o Behavioral XBIs o Content XBIs o Structural XBIs 9% 22% 30% 57% Text Visual Random URL Generator

Limitations of existing techniques Miss Structural XBIs Focus on symptoms rather than causes o Duplicate reports o Low Precision

Comprehensive XBI Detection Behavioral XBIs Content XBIs Structural XBIs Text Visual

Example

Example

Example Behavior XBI 9% Behavior XBI 9%

Example Content XBI (Visual) – 30% Content XBI (Visual) – 30% Content XBI (Text) – 22% Content XBI (Text) – 22% Structural XBI 57% Structural XBI 57%

Comprehensive XBI Detection Behavioral XBIs Content XBIs Structural XBIs Text Visual

Comprehensive XBI Detection Behavioral XBIs Content XBIs Structural XBIs Text Visual Behavior check using Graph Isomorphism CrossCheck [ICST’12] Roy Choudhary, Prasad and Orso CrossT [ICSE’11] Mesbah and Prasad

Comprehensive XBI Detection Behavioral XBIs Content XBIs Structural XBIs Text Visual Text Content check using String equality WebDiff [ICSM’10] Roy Choudhary and Orso CrossCheck [ICST’12] Roy Choudhary, Prasad and Orso CrossT [ICSE’11] Mesbah and Prasad

Comprehensive XBI Detection Behavioral XBIs Content XBIs Structural XBIs Text Visual Visual Content check using Image Matching Computer Vision WebDiff [ICSM’10] Roy Choudhary and Orso CrossCheck [ICST’12] Roy Choudhary, Prasad and Orso

Detecting Structural XBIs Intuition: Disruption in the relative arrangement of web page elements leads to Structural XBIs Idea: Abstraction to represent relative arrangement and compare it to expose errors

Alignment Graph (AG) A graph capturing structural relationships between elements (rectangles) on the webpage An AG has: o Containment relationships o Alignment relationships o Attributes above right-align left-align

Step 1: AG Construction

Step 1: AG Construction a. Containment Relationship e contains e’ iff ( x 1 ≤ x 1 ’ ∧ y 1 ≤ y 1 ’ ∧ x 2 ≥ x 2 ’ ∧ y 2 ≥ y 2 ’ ), and if same bounds and XPath(e) ≤ XPath(e’) e e’

Step 1: AG Construction a. Containment Relationship

Step 1: AG Construction b. Alignment Relationship

Step 1: AG Construction c. Attributes Button 1 Button 2 What alignment attributes can we infer? Button 1 is on the left of Button 2 Button 1 and Button 2 have their tops aligned Button 1 and Button 2 have their bottoms aligned leftOf top-align bottom-align y 1 = y 1 ’y 2 = y 2 ’ x 2 < x 1 ’

Step 1: AG Construction c. Attributes Button 1 Button 2 leftOf top-align bottom-align

Step 2: Matching AGs

Text Visual Comprehensive XBI Detection Behavioral XBIs Content XBIs Structural XBIs Text Visual Report Generation Model Generation Behavior Checker Element Matcher Structure Checker Content Checker Model Comparison ✔ ✔ ✔

Empirical Evaluation Tool: X-PERT (Cross-Platform Error ReporTer) Research Questions: RQ1: Can X-PERT find XBIs in real web applications? RQ2: How does X-PERT compare to the state-of-art?

Experimental Protocol v14.0.1v9.0.9 v/s Ran X-PERT on Subject Applications Subjects + = XBIs (Ground Truth) Manual Check Error Reports Manual Check = False Positives & Negatives +

Subjects NameTypeStatesTransitions DOM Nodes (per screen) maxminaverage OrganizerProductivity GrantaBooksPublisher DesignTrustBusiness DivineLifeSpiritual SaiBabaReligious BreakawaySport ConferenceInformation FishermanRestaurant ValleyforgeLodge UniMelbUniversity KonquerorSoftware UBCClub BMVBSMinistry StarWarsMovie K elements Prior Art (6) Example and Survey (4) Random (4)

Effectiveness NAMEXBI BEHAVIORSTRUCTURE CONTENT TOTAL TEXTIMAGE TPFPTPFPTPFPTPFPTPFP Organizer GrantaBooks DesignTrust DivineLife SaiBaba Breakaway Conference Fisherman Valleyforge UniMelb Konqueror UBC BMVBS StarWars TOTAL RQ1: Can X-PERT find XBIs in real web applications? Answer: Yes RQ1: Can X-PERT find XBIs in real web applications? Answer: Yes

Improvement NAMEXBI X-PERTCROSSCHECK PrecFPReclDpPrecFPReclDp Organizer10100%0 080%2 13 GrantaBooks27100%0 096%1100%0 DesignTrust770%3 100 %05%12286%3 DivineLife1165%6 100 %029%2491%3 SaiBaba531%980%07%5380%10 Breakaway1383%277%113%4954%12 Conference7100% Fisherman571%2 100 %044%580%8 Valleyforge360%2 100 % 050%133%0 UniMelb267%1 100 % 07%27100%0 Konqueror0NA6 00%11NA0 UBC0NA0 00%1NA0 BMVBS1NA0 00%2 0 StarWars12100%0 010%9183%3 TOTAL10377% %132%38974%52 X-PERT Precision = 77% (45%  ) Recall = 95% (14%  ) RQ2: How does X-PERT compare to the state-of-art? Answer: X-PERT has better precision and recall RQ2: How does X-PERT compare to the state-of-art? Answer: X-PERT has better precision and recall

Experimental Data & Tool Release o Experiments (Crawl Data + Reports) o Layout testing algorithm implementation

Related Work o Industrial tools Test Suites for Web Browsers – Acid and test262 MS Expression Web, BrowserShots, BrowserStack Browsera, MogoTest, BrowserBite o Research techniques Eaton & Memon [ IJWET’07 ] Dallmeier, Burger, Orth & Zeller [ JSTools’12 ] 51

Contributions Empirical evaluation shows effectiveness New Structural XBI Detection Algorithm Comprehensive Technique to detect XBIs Study of Real World XBIs

Thank You Shauvik Roy Choudhary, Mukul Prasad, Alex Orso