1. CS 5150 Software Engineering Lecture 21 Reliability 2

2. CS 5150 2 Administrivia Presentation Feedback Generally a big improvement Transitions Demos Code Jargon Conclusion Quiz 3 Design patterns Security/privacy Reliability

3. CS 5150 3 Final Milestone Presentation Demo Report Cumulative

4. CS 5150 4 SE in the News

5. CS 5150 5 Reliability Analysis - Consequences

6. CS 5150 6 Iron Law of Software Reliability For most non-trivial applications, 100% reliability is effectively impossible

7. CS 5150 7 Verification and Validation Does the software do what it is intended to do? Validation Does the design match the intended behavior? Verification Does the implementation match the design? Informal and formal

8. CS 5150 8 Verification Does the implementation exhibit any particular bad behaviors? Can we guarantee that the implementation does what it is designed to do? “How did software get so good without proof?” Dijkstra

9. CS 5150 9 Static and Dynamic Static Examination of source code without actually running it Can be manual or tool-assisted Dynamic Testing Debugging Profiling

10. CS 5150 10 Informal Static Verification Code reviews Can be light or heavy Colleague reads over patch before committing to repository Meetings with multiple participants Benefits: “given enough eyeballs, all bugs are shallow” Opportunity to share knowledge Opportunity to do housekeeping Exposes inconsistent assumptions between developers

11. CS 5150 11 Getting the Most from Code Reviews Senior team members must show leadership heavy-weight reviews require preparation The tone must remain constructive Code reviews are not mini personnel reviews

12. CS 5150 12 Review Checklists Data faults: Initialization, constants, array bounds, character strings Control faults: Conditions, loop termination, compound statements, case statements Input/output faults: All inputs used; all outputs assigned a value Interface faults: Parameter numbers, types, and order; structures and shared memory Storage management faults: Modification of links, allocation and de-allocation of memory Exceptions: Possible errors, error handlers

13. CS 5150 13 Working in Pairs Idea: achieve benefits of continuous review by doing all work in pairs Hard for management to swallow because it looks like doubling the personnel cost of development Some evidence that it works well Generally highest value with a senior/junior pair Junior person should ‘drive’

14. CS 5150 14 Static Analysis Tools Scan source code for possible problems Compiler warnings on steroids Superficial syntactic warnings Deeper analysis based on symbolic execution void foo( int *p ) { printf( “%i”, *p ); }

15. CS 5150 15 Scaling of Static Analysis In general, static analyses cannot be both complete and sound Complete - no false positives Sound - no false negatives Static analysis tools have become more common in the last 10 years Careful engineering trade-offs between false positives and false negatives

16. CS 5150 16 Heavy-Weight Formal Verification Hoare logic Pre-conditions/post-conditions Invariants Model checking Assertions

17. CS 5150 17 Testing Testing Testing Testing comes in many flavors User testing Unit testing Integrations testing Performance testing Stress testing Regression testing Acceptance testing Manual/automated testing Close/open-box (black/white-box) testing

18. CS 5150 18 Fundamental Facts About Testing Testing is indispensable for software verification, but can never prove that non-trivial software is defect-free Each test proves that the system works in one particular circumstance Most non-trivial software has infinitely many possible execution contexts (i.e., input, environment, etc) Every project should have a verification plan that includes testing strategies

19. CS 5150 19 Unit Testing Each unit test should verify that a single ‘unit’ of software works in isolation Can be challenging to write unit tests for code buried deep in a project Overlap with type systems Basic sanity checks Should run automatically e.g. overnight

20. CS 5150 20 Regression Testing Every time you find a bug in your project, make a test case that reproduces the bug The test case should be committed to your repository no later than the bug fix itself A good set of regression tests provide a ‘ratcheting’ mechanism Helps prevent the recurrence of bugs

21. CS 5150 21 Determinism => Reproducibility It is very hard to fix bugs that cannot be reproduced reliably Do not introduce unnecessary non- determinism into your programs Random numbers Address-based data structures Multithreading