Presentation is loading. Please wait.

Presentation is loading. Please wait.

Introduction to Software Engineering

Similar presentations


Presentation on theme: "Introduction to Software Engineering"— Presentation transcript:

1 Introduction to Software Engineering
Chapter 6 Software Testing Software Engineering Consortium Taiwan (SECT) March 2005 © by SEC

2 Some General Concepts about Software Testing
The objective is: Ensure Quality Detect bug {error, failure, inconsistent with specs} Hint the cause of bugs Test Plan  Test Cases  Testing Report  Bug tracking Automation Tools

3 A Typical Test Case 案例 (操作/輸入) 應產生行為(輸出) 實際產生結果 通過測試與否
2.3於使用者登入密碼欄內輸入正確密碼後按Enter。 程式必須告知使用者登入成功。 程式告知使用者輸入登入成功。  成功  2.4於使用者登入密碼欄內輸入錯誤密碼後按Enter。 程式必須告知使用者輸入密碼錯誤。 程式告知使用者輸入密碼錯誤。 2.5 於使用者登入密碼欄內不輸入密碼後按Enter。 程式必須告知使用者未輸入密碼。 失敗 未顯示錯誤訊息,直接跳回輸入畫面

4 Software Testing 6.1 Software Testing Fundamentals
Chapter 6 Software Testing 6.1 Software Testing Fundamentals 6.2 Testing Principles 6.3 Testing Techniques 6.3.1 White-Box Testing Techniques 6.3.2 Black-Box Testing Techniques 6.4 Software Testing Strategies 6.4.1 Verification and Validation 6.4.2 The V Model 6.4.3 Unit Testing 6.4.4 Integration Testing 6.4.5 Regression Testing 6.4.6 Validation Testing 6.4.7 Alpha Testing 6.4.8 Beta Testing 6.4.9 System Testing 315 © by SEC

5 6.1 Software Testing Fundamentals
What is software testing? – Myers: The process of executing a program with the intent of finding errors. – Beizer: The act of executing tests. Tests are designed and then executed to demonstrate the correspondence between an element and its specification. – IEEE: The process of exercising or evaluating a system or system component by manual or automated means to verify that it satisfies specified requirements or to identify differences between expected and actual results. 316 © by SEC

6 6.2 Testing Principles Davis in his book, 201 Principles of Software
Development, suggests a set of testing principles: – All tests should be traceable to customer requirements. – Tests should be planned long before testing begins. – The Pareto principle applies to software testing. 80% of all errors uncovered during testing will likely be traceable to 20% of all program modules. 318 © by SEC

7 6.2 Testing Principles (cont’d)
– Testing should begin “in the small” and progress toward testing “in the large”. – Exhaustive testing is not possible. – To be most effective, testing should be conducted by an independent third party. 319 © by SEC

8 Attributes of A Good Test
Kaner, Falk, and Nguyen in their book, Testing Computer Software, suggest the following attributes of a “good” test – A good test has a high probability of finding an error – A good test is not redundant – A good test should be “best of breed” – A good test should be neither too simple nor too complex 320 © by SEC

9 Wason作業 要檢驗「假如一張卡片一面是母音字母,那它的另一面就是偶數」,需要翻哪些卡片才能確定?(Wason & Johnson, 1972) 要檢驗「假如一個人在喝酒,那他一定滿18歲」,需要翻哪幾張卡片才能確定? E F 4 7 喝酒 喝可樂 22歲 16歲 9

10 Who Tests the Software? Development Engineers
– Understand the system, but test “gently” – Driven by “delivery” – Only perform unit tests and integration tests Software Test Engineers and Testers – Need to learn the system, but attempt to break it – Driven by “quality” – Define test cases, write test specifications, run tests 321 © by SEC

11 Another example A function takes two numbers and return the average of the two numbers How many tests would be needed to make sure that the function has no bug? (10, 20)15, (12,21)16.5, …

12 Exhaustive Testing Is it possible to develop test cases to exercise all logical paths of a program? – Even small programs, the number of possible logical paths can be very large – Example: Consider a program contains 2 nested loops that each executes 1 to 20 times depending on the input data. The interior loop has 4 if-then-else constructs There are 1014 possible paths! If we exercise one test per millisecond, it would take 3170 years to complete test Exhaustive testing is impractical for large software system – Selective testing is required 322 © by SEC

13 When Testing Can Stop? When we run out of budget or time…
When the test adequate criteria is met Testing is never done, the burden simply shifts from you to the customer Only when the product has been irrevocably retired 323 © by SEC

14 Question A program has two versions. The engineering team detected and fixed 100 bugs in n version A, while 25 bugs were detected and fixed in version B. After the fix, which program would you take as the final version?

15 6.3 Testing Techniques Two general software testing approaches:
– White-box testing: Design tests to exercise internal structures of the software to make sure they operates according to specifications and designs – Black-box testing: Design tests to exercise each function of the software and check its errors. – White-box and black-box testing approaches can uncover different class of errors and are complement each other 324 © by SEC

16 Exercise A program reads three integer values as the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene(不等邊), isosceles(等腰), or equilateral(等邊). Write a set of test cases that you feel would adequately test this program. Will it make any difference whether you know the algorithm of the program or not?

17 6.3.1 White-Box Testing White-box testing
– Also known as glass-box testing or structural testing – Has the knowledge of the program’s structures – A test case design method that uses the control structure of the procedural design to derive test cases – Focus on the control structures, logical paths, logical conditions, data flows, internal data structures, and loops. – W. Hetzel describes white-box testing as “testing in the small” 325 © by SEC

18 Basis Path Testing Basic path testing (a white-box testing technique):
– First proposed by Tom McCabe. – Can be used to derive a logical complexity measure for a procedure design. – Used as a guide for defining a basis set of execution path. – Guarantee to execute every statement in the program at least one time. 327 © by SEC

19 How to test this program?
Read Weight (w) Read Height (h) IF w > 400 THEN Print “invalid weight” ENDIF If h > 3 THEN Print “invalid height” BMI=w/(h*h) Print ( “BMI = “ BMI) IF BMI < 20 THEN Print “underweight” ELSE IF BMI >= 20 AND BMI <=25 THEN Print “ideal weight” ELSE IF BMI > 25 THEN Print “overweight” ENDIF

20 Test Coverage

21 The basic structured-constructs in a flow graph :
Basis Path Testing (cont’d) The basic structured-constructs in a flow graph : Sequence If Else While . . . Until Case 328 © by SEC

22 Basis Path Testing (cont’d)
Flow graph notation (control flow graph) – Node represents one or more procedural statements. A sequence of process boxes and a decision diamond can map into a single node A predicate node is a node with two or more edges emanating from it – Edge (or link) represents flow of control – Region: areas bounded by edges and nodes When counting regions, include the area outside the graph as a region 329 © by SEC

23 Occurs when one or more Boolean operators (OR, AND, NAND,
Basis Path Testing (cont’d) – Compound condition Occurs when one or more Boolean operators (OR, AND, NAND, NOR) is present in a conditional statement A separate node is created for each of the conditions C1 and C2 in the statement IF C1 AND C2 predicate nodes if (c1 AND c2) then print T; else print F; end if; c1 F c2 F regions T 330 © by SEC

24 binarySearch() Example
public int binarySearch(int sortedArray[ ], int searchValue) { int bottom = 0; int top = sortedArray.length() - 1; int middle, locationOfsearchValue; boolean found = FALSE; locationOfsearchValue = -1; /* the location of searchValue in the sortedArray */ /* location = -1 means that searchValue is not found */ while ( bottom <= top && !found) middle = (top + bottom)/2; if (searchValue == sortedArray[ middle ]) found = TRUE; locationOfsearchValue = middle; } 6 else if (searchValue < sortedArray[ middle ]) top = middle - 1; else bottom = middle + 1; } // end while 9 10 return locationOfsearchValue; 1 { 3 5 331 © by SEC

25 The Control Flow Graph (CFG) of Function binarySearch() 1
2 3 4 Please count the number of: Nodes Arcs Predictable Nodes Regions 6 5 8 7 9 10 332 © by SEC

26 Cyclomatic Complexity
Cyclomatic complexity is a software metric – provides a quantitative measure of the global complexity of a program. – When this metric is used in the context of the basis path testing the value of cyclomatic complexity defines the number of independent paths in the basis set of a program the value of cyclomatic complexity defines an upper bound of number of tests (i.e., paths) that must be designed and exercised to guarantee coverage of all program statements 333 © by SEC

27 – An independent path is any path of the program that
Cyclomatic Complexity (cont’d) Independent path – An independent path is any path of the program that introduce at least one new set of procedural statements or a new condition – In a flow graph, an independent path must move along at least one edge that has not been traversed before the path is defined Examples: consider the CFG of binarySearch() – Path 1: – Path 2: independent paths – Path 3: – Path 4: (not an independent path) © by SEC 334

28 Cyclomatic Complexity (cont’d)
Three ways to compute cyclomatic complexity: – The number of regions (R) of the flow graph correspond to the cyclomatic complexity. V(G)= R – Cyclomatic complexity, V(G), for a flow graph G is defined as V(G) = E - N + 2 where E is the number of flow graph edges and N is the number of flow graph nodes. – Cyclomatic complexity, V(G) = P + 1 where P is the number of predicate nodes contained in the flow graph G. 335 © by SEC

29 Cyclomatic Complexity of Function binarySearch() predicate nodes
2 3 1 R3 4 R2 R1 regions 6 R4 7 5 R5 8 9 10 336 © by SEC

30 Deriving Basis Test Cases
The following steps can be applied to derive the basis set: 1. Using the design or code as a foundation, draw the corresponding flow graph. 2. Determine the cyclomatic complexity of the flow graph. V(G) = 5 regions V(G) = 13 edges – 10 nodes + 2 = 5 V(G) = 4 predicate nodes + 1 = 5 337 © by SEC

31 Deriving Basis Test Cases (cont’d)
3. Determine a basis set of linearly independent paths. Path 1: Path 2: Path 3: … Path 4: … Path 5: … 4. Prepare test cases that force the execution of each path in the basis set Path 1 test case: – Inputs: sortedArray = { }, searchValue = 2 – Expected results: locationOfSearchValue = -1 338 © by SEC

32 Deriving Basis Test Cases (cont’d)
Path 2 test case: cannot be tested stand-alone! – Inputs: sortedArray = {2, 4, 6}, searchValue = 8 – Expected results: locationOfSearchValue = -1 Path 3 test case: – Inputs: sortedArray = {2, 4, 6, 8, 10}, searchValue = 6 – Expected results: locationOfSearchValue = 2 Path 4 test case: – Inputs: sortedArray = {2, 4, 6, 8, 10}, searchValue = 4 – Expected results: locationOfSearchValue = 1 Path 5 test case: – Inputs: sortedArray = {2, 4, 6, 8, 10}, searchValue = 10 – Expected results: locationOfSearchValue = 4 339 © by SEC

33 Deriving Basis Test Cases (cont’d)
Each test cases is executed and compared to its expected results. Once all test cases have been exercised, we can be sure that all statements are executed at least once Note: some independent paths cannot be individually tested because the input data are determined after traversing the paths. – In binarySearch(), the initial value of variable found is FALSE, hence path 2 can only be tested as part of path 3, 4, and 5 tests 340 © by SEC

34 練習 請畫出本程式的flow graph 計算V(G) 列出所有的basis path 針對每個basis path設計測試資料
public double calculate(int amount) { double rushCharge = 0; if (nextday.equals("yes") ) { rushCharge = 14.50; } double tax = amount * .0725; if (amount >= 1000) { shipCharge = amount * rushCharge; } else if (amount >= 200) { shipCharge = amount * rushCharge; } else if (amount >= 100) { shipCharge = rushCharge; } else if (amount >= 50) { shipCharge = rushCharge; } else if (amount >= 25) { shipCharge = rushCharge; } else { shipCharge = rushCharge; total = amount + tax + shipcharge; return total; } //end calculate 請畫出本程式的flow graph 計算V(G) 列出所有的basis path 針對每個basis path設計測試資料

35 path nextday amount expected result
V(G) = E - N + 2         =         =  7 Path 1:  Path 2:  Path 3:  Path 4:  Path 5:  Path 6:  Path 7:  path     nextday   amount     expected result 1            yes           10                       no           1500        ????.?? 3            no            300                      no            150                      no            75                        no            30                        no            10           

36 Graph Matrices A graph matrix
– A tabular representation of a flow graph – A square matrix with a size equal to the number of nodes on the flow graph – Matrix entries correspond to the edges between nodes – Adding link weight to each edge to represent The connection between nodes The probability of the edge to be executed The resource (e.g., processing time or memory) required for traversing the edge 341 © by SEC

37 Graph Matrices (cont’d)
A connection matrix – A graph matrix with the link weight is 1 (representing a connection exists) or 0 (representing a connection does not exist) – Each row of the matrix with two or more entries represents a predicate node – Provide another method for computing the cyclomatic complexity of a flow graph 342 © by SEC

38 Graph Matrices (cont’d)
Connected to node 1 a Node 4 5 1 2 b d a b c 3 4 5 c f g Graph Matrix 1 4 3 2 e 5 2 3 f g e d 5 Connections 4 1 1 1-1=0 2 1 1 2-1=1 3 4 5 1 1 Cyclomatic complexity 2-1=1 Flow Graph 1 Connection Matrix 3+1=4 343 © by SEC

39 Summery of Basis Path Testing
Derive the Control Flow Graph & calculate V(G): number of basis paths Identify the basis paths Design test cases for each basis path Conduct the tests according to the test cases

40 Condition Testing Condition Testing Logical conditions
– Test case design method that exercise the logical conditions in a program module Logical conditions – Simple condition: (a rel-op b) where rel-op={<, ≤, =, ≠, ≥, >} (may be negated with NOT), e.g., a≤b; NOT(a≤b) – Compound condition: two or more simple conditions connected with AND, OR, e.g., (a>b) AND (c<d) 344 © by SEC

41 Condition Testing (cont’d)
– Relational expression: (E1 rel-op E2) where E1 and E2 are arithmetic expressions, e.g., ((a*b+c)>(a+b+c)) – Boolean expression A condition without relational expressions If a condition is incorrect, then at least one component of the condition is incorrect. 345 © by SEC

42 Condition Testing (cont’d)
The type of errors in a condition include : – Boolean operator (incorrect/missing/extra Boolean operators) – Boolean variable error – Boolean parenthesis error – Relational operator error – Arithmetic expression error The purpose of condition testing is to detect not only errors in the conditions of a program but also other errors in the program. – Detect faults in the condition, statements executed before the condition, and the statements executed after the condition 346 © by SEC

43 – The simplest condition testing strategy
Condition Testing (cont’d) Branch testing – The simplest condition testing strategy – For a compound condition C, test (1) true and false branches of C and (2) every simple condition of C at least once – Example: for the statement: C = (a>b) AND (c<d) we test for TRUE and FALSE C a>b c<d TRUE and FALSE 347 © by SEC

44 Condition Testing (cont’d)
Domain testing – For a relational expression, require 3 or 4 tests to be derived – For an expression E1 rel-op E2, test for E1 greater than, equal to, or less than E2 – Guarantees detection of rel-op error if E1 and E2 are correct – To detect errors in E1 and E2, the difference between E1 and E2 for the tests E1 greater or less than E2 should be as small as possible 348 © by SEC

45 examples if (A or B) then print C If (A > B) then print C
Test case: input A:true, B: true, output:? Test case: input A:true, B: false, output:? Test case: input A:false, B: true, output:? Test case: input A:false, B: false, output:? If (A > B) then print C Test case: input A:10, B: 9, output:? Test case: input A:11, B: 12, output:? Test case: input A: 13, B: 13, output:?

46 Condition Tests Function getTaxRate (amount, prdType) {
If ( amount < 1000) { Return 5% } if (amount > 1000 AND (prdType == FOOD OR prdType == DRINKS) ) { Return 10% } else { Return 15% }

47 Data Flow Testing Data flow testing
– A testing technique that selects test paths of a program according to the locations of definitions and uses of variables in the program. – DEF(S) = {X| statement S contains a definition of X} – USE(S) = {X| statement S contains a use of X} – The definition of variable X at statement S is said to be live at statement S’ if there exists a path from statement S to statement S’ that contains no other definition of X 352 © by SEC

48 Data Flow Testing (cont’d)
– A definition-use (DU) chain of variable X is of the form [X, S, S’], where S and S’ are statement numbers, X is DEF(S) and USE(S’), and the definition of X in statement S is live at statement S’. DU testing strategy – A strategy for selecting the derived DU chains as test cases All use coverage: every DU chain to be covered at least once 353 © by SEC

49 Example i = get_random(); j = get_random(); if (a<30) {
(1) Which lines are definition occurrences of variable i? (2) Which lines are use occurrences of variable i? i = get_random(); j = get_random(); if (a<30) { i= get_random(); } else { j=5; } if (a>b) { i=8; j=6; } else { printf("%d\n",j); } if (b<30) { printf("%d\n",i); j=6; (1) Lines 1, 4, and 8. (2) Lines 14 and 17. (3) There are six (3*2) def-use pairs, 5 of which are feasible. 1-17 (a=35, b=32) 1-14(a=35, b=29) 4-14 (a=25, b=24) 4-17 (not feasible) 8-14 (a=29, b=25) 8-187 (a=35, b=34)

50 Identify the def-use pairs for variable j
Design test cases for these def-use pairs

51 Loop Testing Simple Loops (n iterations) – skip the loop entirely
– only one pass through the loop – two passes through the loop – m passes through the loop where m < n – n-1, n, n+1 passes through the loop 354 © by SEC

52 Loop Testing (cont’d) Nested Loops – conduct simple loop tests for the
innermost loop while holding the outer loops at their minimum iteration – work outward, conducting tests for the next innermost loop – continue until all the loops have been tested – tests grow geometrically as the level of nesting increases 355 © by SEC

53 Loop Testing (cont’d) Concatenated Loops
– independent loops using simple loop testing – dependent loops using nested loop testing 356 © by SEC

54 Loop Testing (cont’d) Unstructured Loops
– Whenever possible, redesign the code to reflect the use of structured programming constructs. – Then apply the appropriate loop testing strategy. 357 © by SEC

55 Loop Testing Example Inputs:
ordered integer array A[1], ..., A[n] such that A[ i ] < A[ i+1] for i between 1 and n-1 Key: an integer to search, n: number of the array Outputs: integer: the location of ‘key’ in the array (0 for not found)

56 Binary Search found := false low := 1 high := n
while ((low <= high) && not found) do mid := floor( (low + high) /2 ) if A[mid] = key then index := mid found := true Else if A[mid]<key then low := mid + 1 else high := mid - 1 end if end while if not found then index := 0

57 Summery: White Box Testing
Basis Path Testing Identify every basis path ( V(G) ) Test every basis path Condition Testing Branch Testing: test every branch and condition component (True/False Combinations) Domain Testing: test close values around a relational condition (>, ==, <) Data Flow Testing Test every Definition-Use path Loop Testing Test every loop from inside out

58 In Class Exercise procedure bubbleSort( A : list of sortable items ) defined as: do swapped := false for each i in 0 to length(A) - 2 do: if A[i] > A[i+1] then swap( A[i], A[i+1] ) swapped := true end if end for while swapped end procedure

59 Test Adequacy Criteria
– Stopping rule determinates whether sufficient testing has been done How do you know when you have tested enough? How do you pick test cases? Control flow coverage criteria – Statement coverage – Edge coverage – Condition coverage – Path coverage 358 © by SEC

60 6.3.2 Black-Box Testing Black-box testing
– Also known as functional testing, behavioral testing, or specification-based testing – Does not have the knowledge of the program’s structures – Discover program errors based on program requirements and product specifications – Derive sets of inputs to fully exercise all functional requirements for a program – Complementary to white-box testing 359 © by SEC

61 6.3.2 Black-Box Testing (cont’d)
– Focuses on the functional requirements of the software including functions, operations, external interfaces, external data and information – Attempts to find errors in the following: Incorrect of missing functions, Interface errors, Errors in data structures or external database access, Performance errors, and Initialization and termination errors. 360 © by SEC

62 Equivalence Partitioning
– Divide the input domain of a program into classes of data – Derive test cases based on these partitions – Test case design for equivalence partitioning is based on an evaluation of equivalence classes for an input domain 361 © by SEC

63 Equivalence Partitioning (cont’d)
An equivalence class represents a set of valid or invalid states for input condition. An input condition is: – a specific numeric value, a range of values – a set of related values, or a Boolean condition 362 © by SEC

64 Equivalence Partitioning (cont’d)
Equivalence classes can be defined using the following guidelines: – If an input condition specifies a range, one valid and two invalid equivalence class are defined. – If an input condition requires a specific value, one valid and two invalid equivalence classes are defined. – If an input condition specifies a member of a set, one valid and one invalid equivalence classes are defined. – If an input condition is Boolean, one valid and one invalid classes are defined. 363 © by SEC

65 Equivalence Partitioning: Automated
Banking Example Consider an automated banking application. The user can dial the bank from a personal computer, provide a six-digit password, and follow with a series of keyword commands that activate the banking function. The software for the application accepts data in the following form: Area Code Prefix Suffix Password commands Blank or three-digit number Three-digit number, not beginning with 0 or 1 Four-digit number Six-character alphanumeric "Check", "deposit", "pay", etc. 364 © by SEC

66 Equivalence Partitioning: Automated Banking Example (cont’d)
Data Item Area Code Prefix Suffix Password Commands Input Condition Boolean range value Remarks The area code may or may not be present Values between 200 and 999 with specific exception Specified value > 200 Four-digit length Password may or not be present Six-character string "Check", "deposit", "pay", etc. set Test cases for each input domain data item can be developed and executed. Test cases are selected so that the largest number of attributes of an equivalence class are exercised at once 365 © by SEC

67 In Class Exercise Grocery Store Example
Consider a software module that is intended to accept: the name of a grocery item a list of the different sizes the item comes in, specified in ounces. The specifications state that the item name is to be alphabetic characters 2 to 15 characters in length. Each size may be a value in the range of 1 to 48, whole numbers only. The sizes are to be entered in ascending order (smaller sizes first). A maximum of five sizes may be entered for each item. The item name is to be entered first, followed by a comma, then followed by a list of sizes. A comma will be used to separate each size. Spaces (blanks) are to be ignored anywhere in the input. Please derive the equivalence partitions of the inputs. Source:

68 Derived Equivalence Classes
Item name is alphabetic (valid) Item name is not alphabetic (invalid) Item name is less than 2 characters in length (invalid) Item name is 2 to 15 characters in length (valid) Item name is greater than 15 characters in length (invalid) Size value is less than 1 (invalid) Size value is in the range 1 to 48 (valid) Size value is greater than 48 (invalid) Size value is a whole number (valid) Size value is a decimal (invalid) Size value is numeric (valid) Size value includes nonnumeric characters (invalid) Size values entered in ascending order (valid) Size values entered in non-ascending order (invalid) No size values entered (invalid) One to five size values entered (valid) More than five sizes entered (invalid) Item name is first (valid) Item name is not first (invalid) A single comma separates each entry in list (valid) A comma does not separate two or more entries in the list (invalid) The entry contains no blanks (???) The entry contains blanks (????)

69 Derived Black Box Test Cases
# Test Data Expected Outcome Classes Covered 1 xy,1 T 1,4,7,9,11,13,16,18,20,22 2 AbcDefghijklmno,1,2,3  ,4,48 1,4,7,9,11,13,16,18,20,23 3 a2x,1 F 4 A,1 5 abcdefghijklmnop 6 Xy,0 7 XY,49 8 Xy,2.5 10 9 xy,2,1,3,4,5 14 Xy 15 11 XY,1,2,3,4,5,6 17 12 1,Xy,2,3,4,5 19 13 XY2,3,4,5,6 21 AB,2#7

70 Selecting Test Values: Boundary Values
More errors occur at the boundaries of an input/output domain than in the “center”. As well as selecting test data “inside” an equivalence class, we also select data at the “edges” of the equivalence class. user output FK queries mouse formats input picks input domain data prompts output domain 366 © by SEC

71 Boundary Value Analysis
Boundary value analysis (BVA) – A test case design technique complements to equivalence partition – Rather selecting any element from an equivalence class, BVA leads to the selection of test cases that exercise bounding values (“edge” of the class) – Unlike equivalence partition that derives test cases only from input conditions, BVA derives test cases from both input conditions and output domain 367 © by SEC

72 Boundary Value Analysis (cont’d)
Guidelines: 1. If an input condition specifies a range [a, b], test cases should be designed with value a and b, just above and below a and b Example: Integer D with input condition [-3, 10], BVA test values are -3, 10, 11, -2, 0 2. If an input condition specifies a number values, test cases should be developed to exercise the minimum, number, maximum number, and values just above and below minimum and maximum Example: Enumerate data E with input condition: {3, 5, 100, 102}, BVA test values are 3, 102, 2, 4, 101, 103 368 © by SEC

73 Boundary Value Analysis (cont’d)
3. Guidelines 1 and 2 are applied to output condition 4. If internal program data structures have prescribed boundaries, be certain to design a test case to exercise the data structure at its boundary Array input condition: – empty, single element, full element, out-of-boundary Search output condition: – element is inside array or the element is not inside array 369 © by SEC

74 Intuition and Experiences
Also known as error guessing Identify potential errors and design test cases based on intuition and experiences Test cases can be derived by making a list of possible errors or error-prone situations – Empty or null lists/strings – Zero instances/occurrences – Blanks or null characters in strings – Negative numbers – Keep a defect history 373 © by SEC

75 Summery: Black Box Testing
Equivalence Partitions Boundary Value Analysis Intuition and Experience

76 請為下列情況設計test case 教務系統從文字檔匯入某一課程成績資料。輸入課號後,上傳檔案, 格式為學號,成績\n學號,成績\n….
若已有舊資料則覆蓋掉 成績須為 0-100之整數 學號若不存在,或成績不合法,則捨棄該行資料,並列印錯誤訊息後,繼續處理下一行 處理完成後,列印尚未輸入成績學生的學號清單

77 請為下列情況設計test case 計算業務員薪水 底薪3萬 達到業績目標 獎金 5000 超過目標的業績,可抽成3%,抽成最高可到5萬
業績比上月成長50%(含)以上者獎金 1000

78 6.4 Software Testing Strategies
Testing begins at the module level and works outward toward the integration of the entire computer-based system. Different testing techniques are appropriate at different points in time. Testing is conducted by the developer of the software and (for large projects) an independent test group. Testing and debugging are different activities, but debugging must be accommodated in any testing strategy 374 © by SEC

79 Phases of Software Testing
Unit Testing Module Integration Testing Subsystem and System Software System System Testing Software System Software + Hardware System Emulation / Parallel Testing Physical Environment, Installation and data download 9 © 2006 by SEC

80 Classification of Software Testing Software Testing
Static Testing Black Box Testing Dynamic Testing White Box Testing Data Flow Testing Control Flow Testing 10 © 2006 by SEC

81 Classification of Software Testing
Black Box Partition Testing Boundary Testing Transaction Flow Testing Syntax Testing Logic-based Testing Random Testing White Box Control Flow Testing Statement Coverage Branches Coverage Conditions Coverage Loops Coverage Data Flow Testing All-Def-Path All-Use-Path All-DU-Path 11 © 2006 by SEC

82 6.4.1 Verification and Validation
Software testing is one element of a broader topic that is often referred to as verification and validation (V&V). Verification – are we building the product correctly? – Refers to the set of activities that ensure that software correctly implements a specific function. Validation – are we building the correct product? – Refers to the set of activities that ensure that the software that has been built is traceable to customer requirements 375 © by SEC

83 6.4.1 Verification and Validation (cont’d)
The definition of V& V encompasses many of the activities that we have referred to as software quality assurance (SQA) including – Formal technical review, quality and configuration audits, performance monitoring, simulation, feasibility study, documentation review, database review, algorithm analysis, development testing, qualification testing, and installation testing 376 © by SEC

84 6.4.2 The V Model The V model – Emerged in reaction to some waterfall models that showed testing as a single phase following the traditional development phases of requirements analysis, high-level design, detailed design and coding. – The V model portrays several distinct testing levels and illustrates how each level addresses a different stage of the software lifecycle. – The V shows the typical sequence of development activities on the left-hand (downhill) side and the corresponding sequence of test execution activities on the right-hand (uphill) side. 377 © by SEC

85 6.4.2 The V Model (cont’d) Requirements Analysis Architecture Design
Detailed Code Acceptance Testing System Integration Unit Testing 378 © by SEC

86 6.4.2 The V Model (cont’d) The V model is valuable because it highlights the existence of several levels of testing and delineates how each relates to a different development phase: – Unit testing: concentrates on each unit (i.e., component) of the software (white box) – Integration testing: focuses on design and the construction of the software architecture (black box, limited amount of white box) 379 © by SEC

87 6.4.2 The V Model (cont’d) – System testing: verifies that all elements mesh properly and that overall system function/performance is achieved. – Acceptance testing: are ordinarily performed by the business/users to confirm that the product meets the business requirements. 380 © by SEC

88 6.4.3 Unit Testing Unit testing
– Is normally considered as an adjunct to the coding step – Focuses verification effort on the smallest unit of software design – the software component or module – Using the component-level design description as a guide – Provide a release criterion for a programming task – Encourage early defect discovery – Discourage big-bang integration with its late defect discovery – Find "side effect" defects, especially in highly coupling designs. – Is white-box oriented and the step can be conducted in parallel for multiple components 381 © by SEC

89 6.4.3 Unit Testing (cont’d) interface local data structures
boundary conditions independent paths error-handling paths Test Cases Module 382 © by SEC

90 Considerations of Unit Test
Interface – Ensures that information properly flows in and out of the component Local data structures – Ensures that data stored temporarily maintains its integrity during execution Boundary conditions – Ensures that the component operates properly at boundaries established to limit or restrict processing Independent paths (basis paths) – Ensures that all paths in a component have been executed at least once Error-handling paths – Ensures that errors are correctly handled 383 © by SEC

91 Unit Test Cases Unit test cases should be designed to uncover errors
due to erroneous computation, incorrect comparisons, or improper control flow Common errors in computation – Misunderstood or incorrect arithmetic precedence – Mixed mode operations – Incorrect initialization – Precision inaccuracy – Incorrect symbolic representation of an expression 384 © by SEC

92 Unit Test Cases (cont’d)
Common errors in comparison and control flow – Comparison of different data types – Incorrect logical operators or precedence – Expectation of equality when precision error makes equality unlikely – Incorrect comparison of variables – Improper or nonexistent loop termination – Failure to exit when divergent iteration is encountered – Improperly modified loop variables 385 © by SEC

93 Unit Test Cases (cont’d)
What should be tested when error handling is evaluated? – Error description is unintelligible – Error noted does not correspond to error encountered – Error condition causes system intervention prior to error handling – Exception-condition processing is incorrect – Error description does not provide enough information to assist in the location of the cause of the error Boundary testing – An important task of the unit test step – Test cases that exercise data structure, control flow, and data values just below, at, and just above maxima and minima are vary likely to uncover errors 386 © by SEC

94 Unit Test Procedure Review of design information (provided as guidance) to establish test cases that are likely to uncover errors previous noted Each test case should be couple with a set of expected results Develop driver and/or stub software for each unit test – Driver: nothing more than a “main program” that accept test case data, passes such data to the component to be tested, and prints relevant results – Stub: serves to replace modules that are subordinate called by the component to be tested. – Drivers and stubs represent overhead Execute and evaluate the unit test 387 © by SEC

95 Unit Test Environment interface local data structures
Driver interface local data structures boundary conditions independent paths error-handling paths Test Module to be tested Stub Stub results Cases 388 © by SEC

96 Component Testing When testing objects, complete test coverage should include The testing in isolation of all operations associated with the object. The setting and interrogation of all attributes associated with the object. The exercise of the object in all possible states. All events that cause a state change in the object should be simulated.

97 Component Testing (cont’d)
Test the attribute Check if it has been set up when the weather station is installed. Test the operation Testing reportWeather, calibrate, test, startup, shutdown in isolation or in a specific sequence.

98 Component Testing (cont’d)
Testing states A state model is needed. Identify sequences of state transitions that have to be tested and define event sequences to force these transitions. Examples of state sequences that should be tested. Shutdown ® Waiting ® Shutdown Waiting ® Calibrating ® Testing ® Transmitting ® Waiting Waiting ® Collecting ® Waiting ® Summarising ® Transmitting ® Waiting

99 Component Testing (cont’d)
When inheritance is used, it is more difficult to test All subclasses should be tested with all inherited operations. Reason: the inherited operation may make assumptions about other operations and attributes and these may have been changed when inherited.

100 State Model of the class “WeatherStation”
Operation calibrate() Calibrating calibration OK Shutdown Startup() test() Waiting Testing transmission done test complete shutdown() Transmitting collection done clock reportWeather() weather summary complete Collecting Summarising

101 6.4.4 Integration Testing Big bang (non-incremental integration)
– All components are combined in advance. The entire program is tested as a whole – When a set of errors is encountered, correction is difficult because isolation of causes is complicated by the vast expanse of entire program – Once these errors are corrected, new ones appear and the process continues in a seemingly endless loop 389 © by SEC

102 6.4.4 Integration Testing (cont’d)
Incremental integration – The program is constructed and tested in small increments, where errors are easier to isolate and correct – Interfaces are more likely to be tested completely – A systematic test approach may be applied Top-down integration Bottom-up integration Sandwich testing (combination of above two approaches) 390 © by SEC

103 Top Down Integration Modules are integrated by moving download through
the control hierarchy – Beginning with the main control module (main program) – Modules subordinate (and ultimately subordinate) to main control module are incorporated into the structure in either a depth-first or breadth-first manner 391 © by SEC

104 Top Down Integration (cont’d)
The integration process: – The main control module is used as a test driver and stubs are substituted for all components directly subordinate to the main control module – Depending the depth-first or breadth-first, subordinate stubs are replaced one at a time with actual components – Test are conducted as each component is integrated – On completion of each set of tests, another stub is replaced with the real component – Regression testing may be conducted to ensure that new errors have not been introduced – Repeat from step 2 until the entire program structure is built 392 © by SEC

105 Top Down Integration (cont’d)
top module is tested with stubs M1 M2 M3 M4 stubs are replaced one at a time M5 M6 M7 as new modules are integrated, M8 some subset of tests is re-run 393 © by SEC

106 Top Down Integration (cont’d)
Advantages – Is better at discovering errors in the system architecture (verifies major control or decision points early) – Can demonstrate a complete function of the system early (if depth-first integration is selected) Disadvantages – Logistical problems can raise   – Need to write and test stubs (costly) To solve the logistical problem: – Delay many tests until stubs are replaced with actual modules – Develop stubs that perform limited functions that simulate the actual module (requires significant overhead) – Integrate the software from the bottom of the hierarchy upward 394 © by SEC

107 Bottom Up Integration Begins construction and testing with atomic modules (i.e., components at the lowest levels in the program structure). The integration process: – Low-level components are combined into clusters (sometimes called builds) that perform a specific software subfunction – A test driver is written to coordinate test case input and output – The cluster is tested – The test drivers are removed and clusters are combined moving upward in the program structure. As the integration moves upward, the need for separate test drivers lessens. 395 © by SEC

108 Bottom Up Integration (cont’d) Mc
Ma Mb D1 D2 D3 Cluster 3 Cluster 1 Cluster 2 396 © by SEC

109 Bottom Up Integration (cont’d)
Advantages – Easier test case design and no need for stubs – Can starts at an early stage in the development process (not require to wait until the architectural design of the system to be completed) – Potentially reusable modules are adequately tested Disadvantages – User interface components are tested last – The program as an entity does not exist until the last module is added – Major design faults show up late 397 © by SEC

110 Sandwich Integration Testing
The advantages of one integration strategy tend to result in disadvantages for the other strategy. In practice, most integration involves a combination of both the top-down and bottom-up integration strategies, i.e., sandwich integration and testing The sandwich integration and testing – The decision modules (logic modules) are integrated and tested top- down so that major design faults can be revealed early – The worker modules (operational modules) are integrated and tested bottom-up so that the potentially reusable modules are adequately tested while reducing the constructions of test stubs – The sandwich integration and testing has the strengths of both top- down and bottom-up integration strategies while avoiding their weaknesses 398 © by SEC

111 6.4.5 Regression Testing Each time a new module is added as part of
integration testing, or a bug is fixed (as the results of uncovering errors through testing) in the software, the software changes. These changes may cause problems with functions that previously worked. Regression testing is the re-execution of some subset of tests that have already been conducted to ensure that changes have not generated unintended side effects. 399 © by SEC

112 6.4.5 Regression Testing (cont’d)
Regression testing may be conducted – Manually by re-executing a subset of all test cases – Using automated capture/playback tools – enable software engineer to capture test cases and results for subsequent playback and comparison. The regression test suite contains three different classes of test cases: – A representative sample of tests that will exercise all software functions – Additional tests that focus on software functions that are likely to be affected by the change – Tests that focus on the software components that have been changed 400 © by SEC

113 6.4.5 Regression Testing (cont’d)
As integration testing proceeds, the number of regression tests can grown quite large – The regression test suite should be designed to include only those tests that address one or more classes of errors in each of the major program functions. – It is impractical and inefficient to re-execute every test for every program function once a change has occurred 401 © by SEC

114 6.4.6 Validation Testing Validation testing: Validation test criteria
– validation succeeds when software functions in a manner that can be reasonably expected by the customer – reasonable expectations are defined in the Software Requirements Specification: a section called Validation Criteria Validation test criteria – All functional requirements are satisfied – All behavioral characteristics are achieved – All performance requirements are attained – Documentation is correct – Human engineering and other requirements are met (transportability, compatibility, error recovery, maintainability) 402 © by SEC

115 6.4.6 Validation Testing (cont’d)
Configuration review (sometimes called an audit) – An important element of the validation process – ensures that all elements of the software delivered to the customer (the software configuration) have been properly developed and documented Alpha and beta testing – When software is developed as a product to be used by many customers, a process called alpha and beta testing is used to uncover errors that only end-user seems able to find. 403 © by SEC

116 6.4.7 Alpha Testing Alpha Tests – Conducted at developer’s site
– Conducted by customer – Developer looks over the shoulder – Developer records the errors and problems – Conducted in a controlled environment 404 © by SEC

117 6.4.8 Beta Testing Beta Tests
– Conducted at one or more customer sites – Conducted by end-user – Developer is not present – Uncontrolled environment – Errors may be real or imagined – Customer reports the errors – Accordingly modifications are made – Final product is released For commercial products α - β testing is done 405 © by SEC

118 6.4.9 System Testing System test: The focus areas are:
– The system software is tested as a whole. It verifies all elements mesh properly to make sure that all system functions and performance are achieved in the target environment. The focus areas are: System functions and performance System reliability and recoverability (recovery test) System installation (installation test) System behavior in the special conditions (stress and load test) System user operations (acceptance test/alpha test) Hardware and software integration and collaboration Integration of external software and the system System testers: test engineers in ITG or SQA people. 406 © by SEC

119 6.4.9 System Testing (cont’d)
Recovery tests: – verify that the system can recover when forced to fail in various ways database recovery is particularly important – e.g., measure time to recover (MTTR) Stress tests: – verify that the system can continue functioning when confronted with many simultaneous requests (abnormal situations) – execute the system by demanding resource in abnormal quantity, frequency, or volume (subject to extreme data & event traffic ) – excessive interrupt, high input data rate, maximum memory, … – how high can we go? Do we fail-soft or collapse? – sensitivity testing (a variation of stress testing) – attempts to uncover data combinations within valid input classes that may cause instability or improper processing (performance degradation) 407 © by SEC

120 6.4.9 System Testing (cont’d)
Security tests: – Verify that access protection mechanisms work make penetration cost more than value of entry – Subject to compromise attempts – E.G., Measure average time to break in Performance: – Is designed to test the run-time performance of software (real-time and embedded systems) within the context of an integrated system – Measure speed, resource utilization under various circumstances. – Is often coupled with stress testing and usually requires both hardware and software instrumentation – Occurs throughout all steps in the testing process 408 © by SEC

121 Testing on Web-Based Softwares More Difficult than Traditional Testing
The Heterogeneous Nature of their components The Heterogeneous Nature of the running environments The extremely wide number of users distributed all around the world and accessing the application services in a concurrent and global way The possibility to generate software components at run time 23 © 2006 by SEC

122 Special Requirements of Web-based Software Testing
Browser Independent No Buffering, Caching Fronts and Preferences Object Mode Tables and Forms Frames 26 © 2006 by SEC

123 Web Testing Tools Web testing tools Web log Analyzers
Weblint; W3C Validator; Net Mechanic; Doctor HTML; TCAT for Java; SilkTest; Visual Test; Web log Analyzers FastStats; Analog; 84 © 2006 by SEC

124 A Poor Test Case Example
Test ID Description Expected Result Actual Result 1 Player 1 rolls dice and moves Player 1 moves on board 2 Player 2 rolls dice and moves Player 2 moves on board

125 A Better Test Case Example
Test ID Description Expected Result Actual Result 1 Game begins and initiated with two players Player 1 has $1200 and in location 0 Player 2 has $1200 and in location 0 2 Precondition: Test 1 success Player 1 dice roll: 3 Player 1 moves to location 3 3 Test 1 & 2 has completed Player 2 dice roll: 4 Player 1 is on location 3 Player 2 moves to location 4

126 A Monopoly Example Requirement:
When a player lands on the “Go to Jail” cell, the player goes directly to jail, does not pass go, does not collect $200 On the next turn, the player must pay $50 to get out of jail and does not roll the dice or advance. If the player does not have enough money, he or she is out of the game

127 Monopoly: derived tests
Does the player get sent to jail after landing on “Go to Jail”? Does the player receive $200 if “Go” is between the current space and jail? Is $50 correctly deducted if the player has more than $50? Is the player out of the game if he or she has less than $50?

128 Derived Test Case 1 Precondition: Only 1 player Player has $1200
Test ID Description Expected Result Actual Result 1 Precondition: Only 1 player Player has $1200 Player is 3 steps to “Go to Jail” Cell “Go” is between the Jail and “Go to Jail” Cell Player dice roll: 3 Player clicks “End Turn” button Player is sent to jail Player’s money is $1200 On next turn, the “Get Out of Jail” button is enabled and “Roll Dice” Button is disabled 1.1 Player clicks “Get Out of Jail” button. Player’s money is $1150 The “Get Out of Jail” button is disabled

129 Derived Test Case 2 Precondition: Only 1 player Player has $20
Test ID Description Expected Result Actual Result 2 Precondition: Only 1 player Player has $20 Player is 3 steps to “Go to Jail” Cell “Go” is between the Jail and “Go to Jail” Cell Player dice roll: 3 Player clicks “End Turn” button Player is sent to jail Player’s money is $20 On next turn, the “Get Out of Jail” button is enabled and “Roll Dice” Button is disabled 2.1 Player clicks “Get Out of Jail” button. Screen displays ”You are out!” The game ends

130 Requirements: If a Player (A) lands on property owned by another player (B), A must pay rent to B. If A does not have enough money to pay B, A is out of the game.

131 Derived Test Case 3 Precondition: Player A has $1200
Test ID Description Expected Result Actual Result 3 Precondition: Player A has $1200 A is 3 steps to B’s Cell: “Golden Garden” Rent for “Golden Garden” is $100 A dice roll: 3 A clicks “End Turn” button A moves to Cell: “Golden Garden” Screen displays “Pay Rent to B: $100” A’s money is $1100

132 Derived Test Case 3 Precondition: Player A has $50
Test ID Description Expected Result Actual Result 3 Precondition: Player A has $50 A is 3 steps to B’s Cell: “Golden Garden” Rent for “Golden Garden” is $100 A dice roll: 3 A clicks “End Turn” button A moves to Cell: “Golden Garden” Screen displays “The Rent is $100, Player A is broke!” A is kicked out of the game


Download ppt "Introduction to Software Engineering"

Similar presentations


Ads by Google