1. 1 CS 501 Spring 2005 CS 501: Software Engineering Lecture 21 Reliability 3

2. 2 CS 501 Spring 2005 Administration

3. 3 CS 501 Spring 2005 Validation and Verification Validation: Are we building the right product? Verification: Are we building the product right? In practice, it is sometimes difficult to distinguish between the two. That's not a bug. That's a feature!

4. 4 CS 501 Spring 2005 The Testing Process Unit, System and Acceptance Testing are major parts of a software project It requires time on the schedule It may require substantial investment in test data, equipment, and test software. Good testing requires good people! Documentation, including management and client reports, are important parts of testing. What is the definition of "done"?

5. 5 CS 501 Spring 2005 Test Design Testing can never prove that a system is correct. It can only show that (a) a system is correct in a special case, or (b) that it has a fault. The objective of testing is to find faults. Testing is never comprehensive. Testing is expensive.

6. 6 CS 501 Spring 2005 Testing Strategies Bottom-up testing. Each unit is tested with its own test environment. Top-down testing. Large components are tested with dummy stubs. user interfaces work-flow client and management demonstrations Stress testing. Tests the system at and beyond its limits. real-time systems transaction processing

7. 7 CS 501 Spring 2005 Methods of Testing Closed box testing Testing is carried out by people who do not know the internals of what they are testing. Open box testing Testing is carried out by people who know the internals of what they are testing. (a) What is the advantage of each approach? (b)In each case, how do you set about selecting test cases?

8. 8 CS 501 Spring 2005 Stages of Testing Testing is most effective if divided into stages Unit testing unit test System testing integration test function test performance test installation test Acceptance testing

9. 9 CS 501 Spring 2005 Testing: Unit Testing Tests on small sections of a system, e.g., a single class Emphasis is on accuracy of actual code Test data is chosen by developer(s) based on their understanding of specification and knowledge of the unit Can be at various levels of granularity Open box: by the developer(s) of the unit If unit testing is not thorough, system testing becomes almost impossible. If your are working on a project that is behind schedule, do not rush the unit testing.

10. 10 CS 501 Spring 2005 Testing: System and Sub-System Testing Tests on components or complete system, combining units that have already been thoroughly tested Emphasis is on integration and interfaces Uses trial data that is typical of the actual data, and/or stresses the boundaries of the system, e.g., failures, restart Is carried out systematically, adding components until the entire system is assembled Open or closed box: by development team or by special testers System testing is finished fastest if each component is completely debugged before assembling the next

11. 11 CS 501 Spring 2005 Testing:Acceptance Testing Closed box: by the client The entire system is tested as a whole The emphasis is on whether the system meets the requirements Uses real data in realistic situations, with actual users, administrators, and operators The acceptance test must be successfully completed before the new system can go live or replace a legacy system Completion of the acceptance test may be a contractual requirement before the system is paid for

12. 12 CS 501 Spring 2005 Variants of Acceptance Testing Alpha Testing: Clients operate the system in a realistic but non-production environment Beta Testing: Clients operate the system in a carefully monitored production environment Parallel Testing: Clients operate new system alongside old production system with same data and compare results

13. 13 CS 501 Spring 2005 Test Cases Test cases are specific tests that are chosen because they are likely to find faults. Test cases are chosen to balance expense against chance of finding serious faults. Cases chosen by the development team are effective in testing known vulnerable areas. Cases chosen by experienced outsiders and clients will be effective in finding gaps left by the developers. Cases chosen by inexperienced users will find other faults.

14. 14 CS 501 Spring 2005 Test Case Selection: Coverage of Inputs Objective is to test all classes of input Classes of data -- major categories of transaction and data inputs. Cornell example: (undergraduate, graduate, transfer,...) by (college, school, program,...) by (standing) by (...) Ranges of data -- typical values, extremes Invalid data, reversals, and special cases.

15. 15 CS 501 Spring 2005 Test Case Selection: Program Objective is to test all functions of each computer program Paths through the computer programs Program flow graph Check that every path is executed at least once Dynamic program analyzers Count number of times each path is executed Highlight or color source code Can not be used with time critical software

16. 16 CS 501 Spring 2005 Test Strategies: Program (a) Statement analysis (b) Branch testing If every statement and every branch is tested is the program correct?

17. 17 CS 501 Spring 2005 Statistical Testing Determine the operational profile of the software Select or generate a profile of test data Apply test data to system, record failure patterns Compute statistical values of metrics under test conditions

18. 18 CS 501 Spring 2005 Statistical Testing Advantages: Can test with very large numbers of transactions Can test with extreme cases (high loads, restarts, disruptions) Can repeat after system modifications Disadvantages: Uncertainty in operational profile (unlikely inputs) Expensive Can never prove high reliability

19. 19 CS 501 Spring 2005 Regression Testing REGRESSION TESTING IS ONE OF THE KEY TECHNIQUES OF SOFTWARE ENGINEERING Applied to modified software to provide confidence that modifications behave as intended and do not adversely affect the behavior of unmodified code. Basic technique is to repeat entire testing process after every change, however small.

20. 20 CS 501 Spring 2005 Regression Testing: Program Testing 1.Collect a suite of test cases, each with its expected behavior. 2.Create scripts to run all test cases and compare with expected behavior. (Scripts may be automatic or have human interaction.) 3.When a change is made, however small (e.g., a bug is fixed), add a new test case that illustrates the change (e.g., a test case that revealed the bug). 4.Before releasing the changed code, rerun the entire test suite.

21. 21 CS 501 Spring 2005 Documentation of Testing Testing should be documented for thoroughness, visibility and for maintenance (a) Test plan (b) Test specification and evaluation (c) Test description (d) Test analysis report

22. 22 CS 501 Spring 2005 A Note on User Interface Testing User interfaces need two categories of testing. During the design phase, user interface testing is carried out with trial users to ensure that the design is usable. This design testing is also used to develop graphical elements and to validate the requirements. During the implementation phase, the user interface goes through the standard steps of unit and system testing to check the reliability of the implementation. Acceptance testing is then carried out on the complete system.

23. 23 CS 501 Spring 2005 Fixing Bugs Isolate the bug Intermittent --> repeatable Complex example --> simple example Understand the bug Root cause Dependencies Structural interactions Fix the bug Design changes Documentation changes Code changes

24. 24 CS 501 Spring 2005 Moving the Bugs Around Fixing bugs is an error-prone process! When you fix a bug, fix its environment Bug fixes need static and dynamic testing Repeat all tests that have the slightest relevance (regression testing) Bugs have a habit of returning! When a bug is fixed, add the failure case to the test suite for the future.

25. 25 CS 501 Spring 2005 Security in the Software Development Process The security goal The security goal is to make sure that the agents (people or external systems) who interact with a computer system, its data, and its resources, are those that the owner of the system would wish to have such interactions. Security considerations need to be part of the entire software development process. They may have a major impact on the architecture chosen. Example. Integration of Internet Explorer into Windows

26. 26 CS 501 Spring 2005 Agents and Components A large system will have many agents and components: each is potentially unreliable and insecure components acquired from third parties may have unknown security problems The software development challenge: develop secure and reliable components protect whole system from security problems in parts of it

27. 27 CS 501 Spring 2005 Techniques: Barriers Place barriers that separate parts of a complex system: Isolate components, e.g., do not connect a computer to a network Firewalls Require authentication to access certain systems or parts of systems Every barrier imposes restrictions on permitted uses of the system Barriers are most effective when the system can be divided into subsystems with simple boundaries

28. 28 CS 501 Spring 2005 Techniques: Authentication & Authorization Authentication establishes the identity of an agent: What the agent knows (e.g., password) What the agent possess (e.g., smart card) Where does the agent have access to (e.g., controller) What are the physical properties of the agent (e.g., fingerprint) Authorization establishes what an authenticated agent may do: Access control lists Group membership

29. 29 CS 501 Spring 2005 Example: An Access Model for Digital Content Digital material Attributes User Roles Actions Operations Access Policies

30. 30 CS 501 Spring 2005 Techniques: Encryption Allows data to be stored and transmitted securely, even when the bits are viewed by unauthorized agents Private key and public key Digital signatures Encryption Decryption X Y Y X

31. 31 CS 501 Spring 2005 Security and People People are intrinsically insecure: Careless (e.g, leave computers logged on, use simple passwords, leave passwords where others can read them) Dishonest (e.g., stealing from financial systems) Malicious (e.g., denial of service attack) Many security problems come from inside the organization: In a large organization, there will be some disgruntled and dishonest employees Security relies on trusted individuals. What if they are dishonest?

32. 32 CS 501 Spring 2005 Design for Security: People Make it easy for responsible people to use the system Make it hard for dishonest or careless people (e.g., password management) Train people in responsible behavior Test the security of the system Do not hide violations

33. 33 CS 501 Spring 2005 Suggested Reading Trust in Cyberspace, Committee on Information Systems Trustworthiness, National Research Council (1999) http://www.nap.edu/readingroom/books/trust/ Fred Schneider, Cornell Computer Science, was the chair of this study.