1. Acceptance Testing Jerzy.Nawrocki@put.poznan.pl www.cs.put.poznan.pl/jnawrocki/require/ Copyright, 2004  Jerzy R. Nawrocki Requirements Engineering & Project Management Lecture 11

2. J.Nawrocki, Acceptance Testing XPrince Roles Project Manager Analyst Architect

3. J.Nawrocki, Acceptance Testing Introduction Test-driven development Acceptance tests: customer Automated accept. testing A. testing: customer+tester Progress measurement based on acceptance testing

4. J.Nawrocki, Acceptance Testing Presentation Contents Testing Life Cycle The Scribble Application The Manual Test Process Test Automation Use-Case-Based Acceptance Testing

5. J.Nawrocki, Acceptance Testing What is test automation? That’s simple! 1. Buy a test execution tool. 2. Record the manual tests. 3. Play the tests back whenever you want to. „Just as there is more to software design than knowing a programming language, there is more to automating testing than knowing a testing tool.” -- M. Fewster & D. Graham

6. J.Nawrocki, Acceptance Testing Test activities Identify test conditions (‘what’ to test) and prioritize  val i > Value subject to  size i  Capacity Knapsack problem 10 911 12 88 10 15 size i val i Capacity Value

7. J.Nawrocki, Acceptance Testing Test activities Identify test conditions (‘what’ to test) and prioritize Design test cases (‘how’ to test) Build test cases (scripts, data etc.) Execute test cases Compare test outcomes to expected outcomes

8. J.Nawrocki, Acceptance Testing Quality attributes of a test case How effective in detecting defects? How exemplary ? (the more exemplary, the less test cases needed) How economic ? How evolvable ? (maintenance effort)

9. J.Nawrocki, Acceptance Testing Test automation effects Effective Exemplary Evolvable Economic

10. J.Nawrocki, Acceptance Testing An example test case Prerequisites: logged into the purchase order system as a data entry clerk; database system must contain the standard Data Set; no other new purchase order activity on the system. StepInputExp.outCond. 1 Create a new order for any one standard order, setting quantity to 100 Order confirmation message displayed VB10 VB23 2 Confirm the orderPurchase order printed VB10 3 Print a new orders reportNew orders report printed showing just this one new order VB10 VB23 4 Cancel the orderPurchase order cancellation notice printed V8

11. J.Nawrocki, Acceptance Testing Presentation Contents Testing Life Cycle The Scribble Application The Manual Test Process Test Automation Use-Case-Based Acceptance Testing

12. J.Nawrocki, Acceptance Testing The Scribble application 1 Belgium 2 Denmark 3 Netherlands New item added in order 1 Netherlands Initial state 1 Netherlands 2 Belgium New item added – end 1 Belgium 2 Netherlands Sorted 1 Denmark 2 Netherlands Old item deleted Enter: Belgium Enter: Denmark Enter: 1 Sort

13. J.Nawrocki, Acceptance Testing Manual test for Scribble Belgium Netherlands UK Add 2 new items to the sorted list. Move an item (the list gets unsorted). Add an item to the unsorted list. Delete an item. Try to delete an invalid item (position) countries.dcm countries2.dcm

14. J.Nawrocki, Acceptance Testing The screen shots – Starting the Scribble Scribble File List

15. J.Nawrocki, Acceptance Testing The screen shots – Openning a file Scribble File List New Close Print Exit Open Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands UK

16. J.Nawrocki, Acceptance Testing The screen shots – Adding an item (Sweden) Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands UK Add Item OK Cancel Sweden| Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands UK New List Sort List Del Item Mov Item Add Item Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands Sweden UK

17. J.Nawrocki, Acceptance Testing The screen shots – Adding an item (USA) Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands Sweden UK Add Item OK Cancel USA| Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands Sweden UK New List Sort List Del Item Mov Item Add Item Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands Sweden UK USA

18. J.Nawrocki, Acceptance Testing The screen shots – Moving an item Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands Sweden UK USA New List Sort List Add Item Del Item Mov Item Scribble File List File Name countries List Sorted Countries on our mailing list are: 1 Belgium 2 Netherlands 3 Sweden 4 UK 5 USA Move Item OK Cancel 4 From 1 To Scribble File List File Name countries List Unsorted Countries on our mailing list are: UK Belgium Netherlands Sweden USA

19. J.Nawrocki, Acceptance Testing The screen shots – Adding an item (Norway) Scribble File List File Name countries List Unsorted Countries on our mailing list are: UK Belgium Netherlands Sweden USA Add Item OK Cancel Norway| Scribble File List File Name countries List Unsorted Countries on our mailing list are: UK Belgium Netherlands Sweden USA New List Sort List Del Item Mov Item Add Item Scribble File List File Name countries List Unsorted Countries on our mailing list are: UK Belgium Netherlands Sweden USA Norway

20. J.Nawrocki, Acceptance Testing The screen shots – Deleting a valid item Scribble File List File Name countries List Unsorted Countries on our mailing list are: 1 UK 2 Belgium 3 Netherlands 4 Sweden 5 USA 6 Norway Delete Item From Position OK Cancel 2 Scribble File List File Name countries List Unsorted Countries on our mailing list are: UK Belgium Netherlands Sweden USA Norway New List Sort List Add Item Mov Item Del Item Scribble File List File Name countries List Unsorted Countries on our mailing list are: UK Netherlands Sweden USA Norway

21. J.Nawrocki, Acceptance Testing The screen shots – Deleting an invalid item Scribble File List File Name countries List Unsorted Countries on our mailing list are: 1 UK 2 Netherlands 3 Sweden 4 USA 5 Norway Delete Item From Position OK Cancel 7 Scribble File List File Name countries List Unsorted Countries on our mailing list are: UK Netherlands Sweden USA Norway New List Sort List Add Item Mov Item Del Item Scribble File List File Name countries List Unsorted Countries on our mailing list are: 1 UK 2 Netherlands 3 Sweden 4 USA 5 Norway Delete Error Position Not Valid OK

22. J.Nawrocki, Acceptance Testing The screen shots – Closing Scribble File List File Name countries List Unsorted Countries on our mailing list are: UK Netherlands Sweden USA Norway Open New Print Exit Close Scribble File List File Name countries List Unsorted Countries on our mailing list are: 1 UK 2 Netherlands 3 Sweden 4 USA 5 Norway Close Save the document? OK i i No

23. J.Nawrocki, Acceptance Testing The screen shots – Exiting Scribble File List Scribble File List Open New Close Print Exit

24. J.Nawrocki, Acceptance Testing What was the test input? 1 Move mouse to Scribble icon 2 Double click

25. J.Nawrocki, Acceptance Testing What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu Scribble File List

26. J.Nawrocki, Acceptance Testing What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click Scribble File List

27. J.Nawrocki, Acceptance Testing What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option Scribble File List New Close Print Exit Open

28. J.Nawrocki, Acceptance Testing What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click Scribble File List New Close Print Exit Open

29. J.Nawrocki, Acceptance Testing What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries.dcm

30. J.Nawrocki, Acceptance Testing What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries.dcm 8 Double click

31. J.Nawrocki, Acceptance Testing What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries.dcm 8 Double click 9 Move mouse to List menu Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands UK

32. J.Nawrocki, Acceptance Testing What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries.dcm 8 Double click 9 Move mouse to List menu 10 Click Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands UK

33. J.Nawrocki, Acceptance Testing What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries.dcm 8 Double click 9 Move mouse to List menu 10 Click 11 Move mouse to Add Item Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands UK New List Sort List Del Item Mov Item Add Item

34. J.Nawrocki, Acceptance Testing What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries.dcm 8 Double click 9 Move mouse to List menu 10 Click 11 Move mouse to Add Item 12 Click Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands UK New List Sort List Del Item Mov Item Add Item

35. J.Nawrocki, Acceptance Testing What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries.dcm 8 Double click 9 Move mouse to List menu 10 Click 11 Move mouse to Add Item 12 Click 13 Type Sweden Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands UK Add Item OK Cancel Sweden|

36. J.Nawrocki, Acceptance Testing What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries.dcm 8 Double click 9 Move mouse to List menu 10 Click 11 Move mouse to Add Item 12 Click 13 Type Sweden 14 Move mouse to OK button Scribble File List File Name countries List Sorted Countries on our mailing list are: Belgium Netherlands UK Add Item OK Cancel Sweden|

37. J.Nawrocki, Acceptance Testing Presentation Contents Testing Life Cycle The Scribble Application The Manual Test Process Test Automation Use-Case-Based Acceptance Testing

38. J.Nawrocki, Acceptance Testing The manual test process Unscripted manual testing: 1.Identify conditions to test 2.Think up specific inputs 3.Enter the inputs 4.Checked it worked OK

39. J.Nawrocki, Acceptance Testing The manual test process Unscripted manual testing Vague manual scripts: 1.Read what to do 2.Think up specific inputs 3.Enter the inputs 4.Checked it worked OK

40. J.Nawrocki, Acceptance Testing Vague manual script 1 Run up Scribble File menu displayed 2 Open file with sorted ListFile contents displayed 3 Add some items to ListItems added in order 4 Move an itemItem moved, List now unsorted 5 Add an itemItem added at end of List 6 Delete item form listItem deleted 7 Use invalid position numberError message displayed to delete an item 8 Save changes in new fileEnd of test

41. J.Nawrocki, Acceptance Testing The manual test process Unscripted manual testing Vague manual scripts Detailed manual scripts: 1.Read what to do 2.Enter the inputs 3.Checked it worked OK

42. J.Nawrocki, Acceptance Testing Detailed manual script 1 Click on Scribble iconScribble opened, File menu enabled 2 Move mouse to File menu, clickOptions available: Open, New, Exit 3 Move mouse to Open option,Filenames displayed including clickcountries.dcm 4 Move mouse to countries.dcmText plus List of 3 countries: Belgium, Netherlands, UK, File and List menus displayed...

43. J.Nawrocki, Acceptance Testing Presentation Contents Testing Life Cycle The Scribble Application The Manual Test Process Test Automation Use-Case-Based Acceptance Testing

44. J.Nawrocki, Acceptance Testing Manual or Automatic Testing? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries.dcm 8 Double click 9 Move mouse to List menu 10 Click 11 Move mouse to Add Item 12 Click 13 Type Sweden 14 Move mouse to OK button The amount of effort to automate a test case: 2 – 10 times the effort required to run the test manually (!)

45. J.Nawrocki, Acceptance Testing Introduction – What is Rational Robot? Baseline Robot

46. J.Nawrocki, Acceptance Testing Introduction – Rational Robot and test-first coding Remarks: When you start testing an application is ready. Rational Robot is for regression testing, not for test-first coding. The only chance for test-first coding is SQABasic.

47. J.Nawrocki, Acceptance Testing Testing HTML Applications StartBrowser [ URL$, ] [ WindowTag= Name$ ] Starts the browser, enables HTML testing, and loads a page if specified. Web page address Identifies this instance of the browser StartBrowser "http://www.cs.put.poznan.pl/jnawrocki/models/", "WindowTag= Parent"

48. J.Nawrocki, Acceptance Testing Testing HTML Applications Browser action$, recMethod$, parameters$ Performs an action on a Web browser Back, Forward, NewPage, SetFrame, CloseWin,.. Wait= % default time is 30s Browser NewPage, "" HTMLTitle= $, Name= $, "".. Wait for the topmost frame

49. J.Nawrocki, Acceptance Testing Testing HTML Applications HTML action%, recMethod$, parameters$ Performs a mouse action on an HTML tag. Click,.. Coords= x, y HTML Click, "Name= Autor", "Coords= 10,12" HTMLTitle= $, Name= $, ""..

50. J.Nawrocki, Acceptance Testing TestDatastore\DefaultTestScriptDatastore\TMS_Script\vp\test7. Weryfikacja_daty. base.grd SQABasic HTMLDocument Click, "Type=HTMLDocument;Index=1", " Coords=25,14 " Result = HTMLTableVP (CompareData, "Type=HTMLTable;Index=1", "VP= Weryfikacja_daty ")

51. J.Nawrocki, Acceptance Testing An alternative to Rational Robot httpUnit Based on jUnit For programmers!

52. J.Nawrocki, Acceptance Testing Data-driven scripts Data-driven scripting = Test inputs stored in a separate (data) file. countries, Sweden, USA, countries2 countries, France, Germany, test2 countries, Austria, Italy, test3

53. J.Nawrocki, Acceptance Testing Data-driven scripts Data-driven scripting = Test inputs stored in a separate (data) file. OpenFile ‘ScribbleData’ For each record in ScribbleData Read INPUTFILE Read NAME1 Read NAME2 Read OUTPUTFILE Call ScribbleOpen (INPUTFILE) FocusOn ‘Scribble’ SelectOption ‘List/Add Item’ FocusOn ‘Add Item’ Type NAME1 LeftMouseClick ‘OK’... EndFor countries, Sweden, USA, countries2 countries, France, Germany, test2 countries, Austria, Italy, test3 ScribbleData Control script

54. J.Nawrocki, Acceptance Testing A more sophisticated data-driven script OpenFile ‘ScribbleData’ Read INPUTFILE Call ScribbleOpen (INPUTFILE) Go to next record (ie row) For each record in ScribbleData Read ADDNAME If ADDNAME <> Blank Then { FocusOn ‘Scribble’ SelectOption ‘List/Add Item’ FocusOn ‘Add Item’ Type ADDNAME LeftMouseClick ‘OK} Read MOVEFROM Read MOVETO... EndFor ScribbleData countries Sweden USA 41 Norway 2 7 countries2 Control script

55. J.Nawrocki, Acceptance Testing Data-driven scripts               Software under test Key automated regression test cases Additional automated regression test cases

56. J.Nawrocki, Acceptance Testing Keyword-driven scripts Keyword-driven script = A data-driven scripts augmented with keywords representing user actions. SQABasic??? ScribbleOpen countries AddToList Sweden USA SaveAs countries2 For each TEST_ID OpenFile TEST_ID For each record in test file Read KEYWORD Call KEYWORD EndFor CloseFile TEST_ID EndFor Test file Control script ScribbleOpen... AddToList... SaveAs... Supporting scripts

57. J.Nawrocki, Acceptance Testing Shared scripts Shared script = A script used (shared) by more than one test case. SQABasic Call ScribbleOpen (‘countries’) FocusOn ‘Scribble’ SelectOption ‘List/Add Item’... Call ScribbleSaveAs (‘countries2’)

58. J.Nawrocki, Acceptance Testing Automating test execution Automated scripts are not like manual scripts Don’t automate testing by simply recording Automated execution with manual verification? How much shoul be compared (whole, minimum, something between)? Dynamic comparison and post-execution comparison Automated comparison messages have to be manually checked While tests fail the second time?

59. J.Nawrocki, Acceptance Testing Presentation Contents Testing Life Cycle The Scribble Application The Manual Test Process Test Automation Use-Case-Based Acceptance Testing

60. J.Nawrocki, Acceptance Testing Test activities Identify test conditions (‘what’ to test) and prioritize Design test cases (‘how’ to test) Build test cases (scripts, data etc.) Execute test cases Compare test outcomes to expected outcomes

61. J.Nawrocki, Acceptance Testing Test activities Identify important scenarios (criticality/frequency) and prioritize Instantiate the selected scenarios (1 -> *) Implement (scripts, data etc.) Execute test cases Compare test outcomes to expected outcomes

62. J.Nawrocki, Acceptance Testing Use case goal levels Book tripBook hotelBook flight User Goal Level Book trip Summary Level Book trip Book hotelBook flight Find flight Reserve seat Find hotel Reserve room Subfunction Level

63. J.Nawrocki, Acceptance Testing Test activities Identify important scenarios (criticality/frequency) and prioritize Instantiate the selected scenarios (1 -> *) Implement (scripts, data etc.) Execute test cases Compare test outcomes to expected outcomes

64. J.Nawrocki, Acceptance Testing Test coverage GetSoftwareG1:0 SelectAProjectG1:0 GetInfoAboutProjectsG1:0 RegisterACustomerG1:1Z1:1 SubmitAProposalG1:0R1:0U1:0 Scenario Total calls

65. J.Nawrocki, Acceptance Testing Use-Case Example Success Guarantees Success Guarantees: Requestor has goods, correct budet ready do be debited. Trigger Trigger: Requestor decides to buy something. Main Success Scenario 1.Requestor 1.Requestor: Initiate a request. 2.Approver 2.Approver: Check money in the budget, check price of goods, complete request for submission. 3.Buyer 3.Buyer: Check contents of storage, find best vendor for goods. 4.Authorizer 4.Authorizer: Validate Approver’s signature....Extensions 1a. Requestor does not know vendor or price: leave those parts blank and continue.

66. J.Nawrocki, Acceptance Testing Test scenario Aktor: Reprezentant klienta Cel: Zarejestrowanie organizacji klienta w bazie danych SDS Pochodzi z: przypadku użycia RegisterACustomer [RegCust] Autor: J.Nawrocki; Zmodyfikowano: 28.11.2003 Scenariusz: Główny scenariusz – G1 Scenariusze pomocniczych przypadków: - Warunki wstępne: Firma X nie jest zarejestrowana jako reprezentant klienta. Punkt wejścia: Przeglądarka internetowa (Internet Explorer 5.0) Krok Akcja aktoraSpodziewany rezultat Pro1Otwórz stronę SDS-u.Strona SDS jest otwarta. 1Otwórz formularz rejestracji.Formularz otwarty [ReqCust- Form]. 2Wprowadź dane Firmy X [CustInfo] i wyślij je. Zostaje wyświetlony komunikat “Registration confirmed” [ReqCust-Confirm].

67. J.Nawrocki, Acceptance Testing Test scenario – Screen shots RegCust-FormRegCust-Confirm

68. J.Nawrocki, Acceptance Testing Summary SDS test development: 1. Use cases 2. Manual test scenarios 3. Automated test scripts 4. Test cases sharing the scripts httpUnit or Rational Robot?

69. J.Nawrocki, Acceptance Testing Questions?