1. Scenario-Based Requirements for Web Macro Tools Christopher Scaffidi, Allen Cypher, Sebastian Elbaum, Andhy Koesnandar, Brad Myers

2. 2 PBE and browser automation Programming-by-example (PBE) is a success story –Research systems: Eager, Marquise –Systems in widespread use: Microsoft Office macros Researchers have applied PBE to browser automation –e.g.: Internet Scrapbook, Koala, Miro, Robofox, Turquoise –These tools… 1.Watch the user perform a task 2.Infer the user’s intent 3.Record a web macro representing the intent 4.Allow the user to replay the macro on new data introduction  scenarios  requirements  Robofox  conclusion

3. Step 1:Open http://www.oanda.com/converter/classichttp://www.oanda.com/converter/classic Step 2:Submit the amount, date, foreign currency type, and target currency. Step 3:Retrieve the result in the target currency.

4. 4 PBE web macro tools are not widely adopted The web context introduces new challenges to PBE: –Web site changes can break macros without warning  Need features for detecting and responding to site change –Office tasks involve not just browsers but also spreadsheets and other applications  Need integration with other applications What other unmet requirements are there? introduction  scenarios  requirements  Robofox  conclusion

5. 5 Missing: a methodical characterization of requirements for web macro tools Many papers use scenarios to explain and motivate a PBE tool’s features. Such scenarios… –Are retrospective: The scenarios show the features that actually work, rather than opportunities for future work. –Lack pedigree: Are the scenarios hypothetical or based on a user study? Will the tool features be useful in practice? introduction  scenarios  requirements  Robofox  conclusion

6. 6Outline We have… –Analyzed 10 scenarios where web macros would be desirable –Characterized 25 requirements that tools must support to be useful for many real-world tasks –Used these requirements as a benchmark to identify opportunities for enhancing a particular tool (Robofox) introduction  scenarios  requirements  Robofox  conclusion

7. 7 Catalog of scenarios Using a web site to look up information – 6 scenarios –Currency converter, Package tracker, Per diem lookup, Staff lookup, Stock analysis, Watcher for eBay –Typical users: office workers Using a web form to order products – 1 scenario –Path to procurement –Typical users: online shoppers “Scraping” a large part of a site – 3 scenarios –Peoplesoft scraper, Person locator scraper, Scraper for CMS –Typical users: System administrators introduction  scenarios  requirements  Robofox  conclusion

8. 8 Scenarios came from a variety of sources Contextual inquiry – 3 scenarios –Currency converter, Per diem lookup, Staff lookup Co-workers – 2 scenarios –Path to procurement, Peoplesoft scraper Our own experiences – 2 scenarios –Package tracker, Scraper for CMS Online sources – 3 scenarios –Person locator scraper, Stock analysis, Watcher for eBay introduction  scenarios  requirements  Robofox  conclusion

9. 9 What is in a scenario? * Title Pithy name to make it easy to refer to the scenario * Typical User Tells what type of person is likely to perform the task * Overview Context and forces that would prompt a user to do the task * Starting Conditions Pre-conditions that hold true before the scenario * Result Post-conditions that should hold true after the scenario * Actions Steps that the user performs (maybe with a tool) Action Details Screenshots or other clarifying information Variations Discusses likely tweaks of the main scenario Macro Maintenance Examines how scenario evolution prompting macro changes * Scenario Source Summarizes empirical data that generated this scenario * = required section introduction  scenarios  requirements  Robofox  conclusion

10. 10 Requirements: Triggering macros On-demand execution –Required to support users’ work tasks Scheduled execution –Especially for scrapers Event-based triggers –Watcher for eBay executes when user visits Google homepage Subroutines –Especially for lookup macros introduction  scenarios  requirements  Robofox  conclusion

11. 11 Requirements: Using objects on pages Text snippets –Almost all scenarios require reading text Tabular information - –Almost all scenarios read tabular data Web form widgets –Read/write textbox, dropdown, checkbox widgets Other HTML structures –Watcher for eBay introduction  scenarios  requirements  Robofox  conclusion

12. 12 Requirements: Adapting to site changes Adaptation to changing page layout –Changes to text location, text color, page structure Adaptation to changing form fields - –Changes to field names and field values Adaptation to changing URLs –“This page has moved to X. Please update your links.” introduction  scenarios  requirements  Robofox  conclusion

13. 13 Requirements: Reading and writing data outside pages Browser APIs –Read/write highlighted text, show popup window Spreadsheets and other files - –Often one record per spreadsheet row or XML node Parameters containing user input –Prompting for an input (e.g. password) during playback introduction  scenarios  requirements  Robofox  conclusion

14. 14 Requirements: Transforming data Reformat to equivalent value - –For comparing values and for making outputs consistent Extracting values’ parts –For comparing values Combining values –Such as arithmetic and string concatenation introduction  scenarios  requirements  Robofox  conclusion

15. 15 Requirements: Executing control structures Looping operations - –Especially in scrapers Conditional operations –Macro behavior depends on data provided by user as well as data returned by web sites introduction  scenarios  requirements  Robofox  conclusion

16. 16 Requirements: Recovering from failure Partial restarts –Especially for scrapers Exception handlers –Integration with assertions introduction  scenarios  requirements  Robofox  conclusion

17. 17 Requirements: Supporting macro maintenance User-understandable representation –For maintenance and for sharing macros with other users Editable macros –Analogous to features supported by textual editors Features for debugging –Traces, breakpoints, step-by-step execution, etc Maintenance at runtime –On error, enter maintenance mode introduction  scenarios  requirements  Robofox  conclusion

18. 18 Example benchmark use: Robofox Triggering macros On-demand execution Yes Scheduled execution Yes Event-based triggers Yes Subroutines No Using objects on web pages Text snippets Yes Tabular information No Web form widgets Yes Other HTML structures No Adapting to site changes Adaptation to changing page layout Limited Adaptation to changing form fields Yes Adaptation to changing URLs No Inserts “sanity checks” to detect changing page structure. Includes heuristics for “refinding” page elements based on color, label, DOM location. Some changes (e.g.: form field renaming) are not handled automatically. introduction  scenarios  requirements  Robofox  conclusion

19. 19 Example benchmark use: Robofox Can read/write spreadsheets, but only if they have a specific layout Reading and writing data outside pages Browser APIs Limited Spreadsheets and other files Limited Parameters containing user input Limited Transforming data Reformat to equivalent value No Extracting values’ parts Yes Combining values No Executing control structures Looping operations Limited Conditional operations Yes Can do a “foreach” loop but not a “while(condition)” loop introduction  scenarios  requirements  Robofox  conclusion

20. 20 Example benchmark use: Robofox User-defined exception handlers can fire when assertions fail Recovering from failure Partial restarts No Exception handlers Yes Supporting macro maintenance User-understandable representation Yes Editable macros Yes Features for debugging Yes Maintenance at runtime Yes introduction  scenarios  requirements  Robofox  conclusion

21. 21 Summary of Robofox limitations Of 25 requirements… –Supported: 13 –Limited support: 5 –Unsupported: 7 Of 10 scenarios… –Supported: 3 –Limited support: 2 (unsupported scenario variations) –Unsupported: 5 introduction  scenarios  requirements  Robofox  conclusion

22. 22 Opportunities to enhance Robofox Adding support for 2 requirements… –Tabular information –Reformat to equivalent value … would leave only… –2 unsupported scenarios –2 unsupported scenario variations introduction  scenarios  requirements  Robofox  conclusion

23. 23 Extending the benchmark These 25 requirements are “Version 1” of a benchmark for evaluating web macro tools. As researchers meet existing requirements, new scenarios and new requirements will appear. We created a wiki where YOU can upload scenarios. –http://softwaresurvey.cs.cmu.edu/wmcorpus.htmlhttp://softwaresurvey.cs.cmu.edu/wmcorpus.html introduction  scenarios  requirements  Robofox  conclusion

24. 24 Thank You… To VL/HCC for the opportunity to present. To EUSES Consortium members for suggestions. To NSF and EUSES for funding (CCF-0325273, CCF-0438929, CCF-0324861). introduction  scenarios  requirements  Robofox  conclusion