1. Building an automation framework around open source technologies

2. Agenda Challenges Choosing an automation framework Generic Architecture How does it work? Features of an ideal test automation framework Additions to Selenium framework Other components of test automation framework Beyond test automation 12/11/2010Datamatics Confidential for Limited Circulation Only2

3. Challenges today ROITechnologyAutomationTools 12/11/2010Datamatics Confidential for Limited Circulation Only3

4. ROI Facts Frequency of usage Coverage of Tests Reusability in the system Creation efforts Maintenance efforts ROI TechnologyAutomationTools 12/11/2010Datamatics Confidential for Limited Circulation Only4

5. Technology Facts ROI Technology AutomationTools Over 73% of applications developed today are web enabled applications Prevalence of Firefox, IE, Safari and Chrome as the major browsers, with multiple versions Incompatibilities between these browsers primarily due to javascript implementation Testing needs for Web Applications is multiplied due to permutations of these challenges 12/11/2010Datamatics Confidential for Limited Circulation Only5

6. ROITechnology Automation Tools Test Automation Facts New requirements are introduced on a daily basis Testing window is getting smaller day by day Test automation can never replace human testers Not all test can / should be automated Test automation is a supplement to help / boost overall testing Use test automation for area that are least prone to changes and yet require frequent tests 12/11/2010Datamatics Confidential for Limited Circulation Only6

7. ROITechnologyAutomation Tools Test Automation Tool Facts Only people that claim 100% automation Learning curve is very higher with every variant Product support for emerging technologies Extendibility 12/11/2010Datamatics Confidential for Limited Circulation Only7

8. Myth#1: Test automation is a goal never reached Automation is a project on top of our goal list Many testers intimidated with test automation Developers don’t want to learn test automation GUI automation is hard so lets not do automation Automate everything and fire the testers For some - panacea and others - waste of time Automation testing is an investment – will take time and effort to bring to fruition 12/11/2010Datamatics Confidential for Limited Circulation Only8

9. Myth#2: Record / Playback Main feature recommended by any test automation tool vendors –Capture user actions and later on play them back for regression testing. Simple steps to follow –Design test case in test automation tool –Use capture feature to record user actions –Enhance script with verification points –Add delay and wait state points for synchronization –Playback and observe results in log file 12/11/2010Datamatics Confidential for Limited Circulation Only9

10. Record / Playback Myth Learning’s –Scripts resulting in hard-coded values –Costs associated with maintenance is astronomical –Scripts are often not reliable (pop-up, messages) –If tester makes a mistake – re-record the script –If application changes – re-record the script –Scripts cannot be recorded until software works Nothing is ever tested by this approach 12/11/2010Datamatics Confidential for Limited Circulation Only10

11. Test Automation using Open Source Framework Automation Approach Technology Layer Automation Tool 12/11/2010Datamatics Confidential for Limited Circulation Only11

12. Choosing a test automation approach Test script modularity Test library architecture Keyword or table driven testing Data driven testing Hybrid-Keyword & Data driven both. 12/11/2010Datamatics Confidential for Limited Circulation Only12

13. Choosing a technology layer Platform Independence Execute unattended runs Distributed test execution Extensible and Customizable Ease of debugging and logging Status reporting Repeat tests as needed (CI) Version control 12/11/2010Datamatics Confidential for Limited Circulation Only13

14. Generic Architecture Application Under Tests (AUT) Automation Tool Data ManagerTest Manager Test Artifacts Version Control Integration (CI) 12/11/2010Datamatics Confidential for Limited Circulation Only14

15. Choosing a Test Automation Tools Abbot Java GUI Test Framework Anteater Apodora Arbiter Autonet AutoTestFlash Avignon Blerby Test Runner Canoo WebTest Capedit Concordion Crosscheck csvdiff CubicTest Cucumber DBFeeder DbFit Eclipse TPTP EMOS Framework Enterprise Web Test Expect expect-lite Feed4TestNG Frankenstein GITAK Harness httest IdMUnit Imprimatur ItIN - Infopath testing in.Net ITP ItsNat, Natural AJAX WebDriver WebInject WebTst ivalidator Jacobie Jameleon jDiffChaser Jemmy JFunc: JUnit Functional Testing Extension JSystem Latka Linux Test Project LogiTest LReport MActor Marathon Maveryx MaxQ miTester for SIP SIP Testing Tool Mockito MozUnit OLVER Om org.tigris.mbt PAMIE Pounder pywinauto QAliber QAT QMTest Rasta Robot Framework Sahi Samie Sauce Labs Selenium SharpRobo soapui STAF Solex SWAT SWTBot Systin T-Plan Robot tclwebtest TestPlan TextTest Tomato Toster Watij WatiN Watir WET 12/11/2010Datamatics Confidential for Limited Circulation Only15

16. Test automation tool 12/11/2010Datamatics Confidential for Limited Circulation Only16

17. How does it work? 12/11/2010Datamatics Confidential for Limited Circulation Only17

18. Why Selenium? Support for Windows, Linux and Mac OS Supports browsers – Explorer, Firefox, Safire Supports AJAX calls Non-developers can develop test cases Graphical Selenium IDE Export test cases as JUnit test cases When test fails, Selenium can send an email w/ screenshot Selenium Grid enabled test execution on multiple machines in parallel reducing overall execution time 12/11/2010Datamatics Confidential for Limited Circulation Only18

19. Features of an ideal test automation framework Scripting Script Recording Parameterization Control Statements Database Integration Modal Dialog Support Keyword Support Execution Across Multiple OS and Browsers Across Multiple machines in parallel Ability to recover testing from failures Analysis Detailed Reporting for failures Snapshots on failures Multilayer Reporting 12/11/2010Datamatics Confidential for Limited Circulation Only19

20. Features Missing within the selenium suite Scripting Script Recording Parameterization Control Statements Database Integration Modal Dialog Support Keyword Support Execution Across Multiple OS and Browsers Across Multiple machines in parallel Ability to recover testing from failures Analysis Detailed Reporting for failures Snapshots on failures Multilayer Reporting 12/11/2010Datamatics Confidential for Limited Circulation Only20

21. Additions to Selenium framework KeywordsBusiness ComponentsDatasheets Utility / Generic Functions Recovery Functions Reusable Framework Reporting Functions SELENIUM Application Under Test (AUT) 12/11/2010Datamatics Confidential for Limited Circulation Only21

22. Features of an ideal test automation framework FeaturesBest found in Scripting Phase Script recordingSelenium IDE ParameterizationDatasheets Control StatementsImplementation specific Database IntegrationUtility / Generic Functions Modal Dialog SupportUtility / Generic Functions Keyword SupportKeywords 12/11/2010Datamatics Confidential for Limited Circulation Only22

23. Features of an ideal test automation framework FeaturesBest found in Execution Phase Across Multiple OS and BrowsersSelenium RC Across Multiple machines in ParallelSelenium Grid Ability to recover testing from failuresRecovery Function Analysis Phase Detailed Reporting for FailuresSelenium Snapshots on failuresSelenium Multilayer ReportingReporting Function 12/11/2010Datamatics Confidential for Limited Circulation Only23

24. Generic Architecture Application Under Tests (AUT) Automation Tool Data ManagerTest Manager Test Artifacts Version Control Integration (CI) 12/11/2010Datamatics Confidential for Limited Circulation Only24

25. Continuous Integration Apache Continuum Apache Gump Beebox Bitten Buildbot Cabie Cerberus Control Tier Cruise Control cDash Draco Hudson Jhbuild LuntBuild Sin Tinderbox 12/11/2010Datamatics Confidential for Limited Circulation Only25

26. Version Control Aegis ArX GNU arch Bazaar Bonsai Cervisia Codeville Cogito CVSNT OpenCVS OpenGrok Revision Control System Apache Subversion SVN Tortoise Vesta DCVS Darcs Fossil Git Subversion Libre Source Mercurial Monotone Repo CVS 12/11/2010Datamatics Confidential for Limited Circulation Only26

27. Looking beyond automation testing Test reports are automatically prepared Defects are automatically captured in defect tracking system Test metrics are updated automatically Traceability matrix is updated automatically Select test cases automatically based defect status updates Calculating code and test coverage automatically Suggesting improvements in test coverage 12/11/2010Datamatics Confidential for Limited Circulation Only27

28. Thank You!!!