Presentation on theme: "Non-Functional Testing Last update: 21-05-2014 Philippe CHARMAN"— Presentation transcript:
Non-Functional Testing Last update: Philippe CHARMAN
Non exhaustive list of non-functional testing Performance Testing –Load testing –Stress Testing –Endurance Testing Usability Testing Accessibility Testing Compatibility Testing Compliance Testing Internationalization and localization Testing Doc & Training Testing Recovery Testing Etc.
Load Testing Load testing is the process of putting demand on a system or device and measuring its response Some examples: –simulating multiple users accessing a web server –giving a huge document to a word processor –subjecting a mail server to a large amount of traffic –writing and reading data to and from a hard disk continuously It helps to identify the maximum operating capacity of an application as well as any bottlenecks and determine which element is causing degradation Synonyms: volume testing, scalability testing, etc.
Stress Testing Testing used to determine the stability of a system Testing beyond normal operational capacity, often to a breaking point, in order to observe the results The goals may be to ensure the software does not crash in conditions of insufficient computational resources (such as memory or disk space),
Endurance Testing Kind of load testing over a longer period of time During endurance tests, memory utilization is monitored to detect potential leaks Synonyms: reliability testing, soak testing
Performance Tests Main typesWhat we are mainly concerned at Load Tests Scalability Tests System response time, big memory peak, bottlenecks Stress TestsWill it crash ? Endurance Tests Reliability Tests Soak Tests System response time, memory/resource consumption over a long period of time
Usability Evaluation Methods Usability : how well users can learn and use a product to achieve their goals and how satisfied they are with that process How to evaluate usability: –Ease of learning –Efficiency of use –Memorability –Error frequency and severity –Subjective satisfaction Some evaluation methods: –Inspection –Testing –Inquiry –Rapid prototyping
Usability Inspection Usability inspection is aimed at finding usability problems in the design, An evaluator inspects a user interface Some inspection methods: –Cognitive walkthrough –Heuristic evaluation
Usability Testing Technique used to evaluate a product by testing it on real users How many users to test ? Eye Tracking
Eye Tracking Demo
Accessibility Testing Software accessibility is a general term used to describe the degree to which an application is available to as many people as possible Access issues: –Visual impairments –Motor skills –Hearing impairments –Photosensitive epilepsy
Accessibility Testing Typical test cases: –Make sure that all functions are available via keyboard only (do not use mouse) –Make sure that information is visible when display setting is changed to High Contrast modes –Make sure that screen reading tools can read all the text available and every picture/Image have corresponding alternate text associated with it –Make sure that product defined keyboard actions do not affect accessibility keyboard shortcuts –Make sure that Web pages do not contain anything that flashes more than two times in any one second period –etc...
Accessibility Testing Do a simple test: –Start menu > Control Panel > Accessibility Options > Display > –Check the checkbox "Use High Contrast" –Are your main applications (web browser, reader, etc.) still usable ? Another test: –Install the tool Color Oracle from –Is it still easy to distinguish colors differences with your main applications?
Color impaired vision
Compatibility Testing Testing conducted on the application to evaluate the application's compatibility with the computing environment that can be composed of: –Hardware platform : x86, x64, UltraSPARC-I, POWER7 etc. –OS: Windows XP 32-bit, Debian 4.0, Solaris 11, etc. –Database: Oracle, Sybase, DB2, etc. –Browser: IE, Firefox, Chrome, –Peripheral: printer, scanner, –Etc. Compatibility Matrix Pairwise Testing methods
C/C++ Standards Some examples Avoid long and complex functions. Rationale: long functions are difficult to understand and to test Do not use #define to use a constant Use "const« instead Use #include to include.h only and not.cpp files
Compliance Tests More: MISRA web site (Motor Industry Software Reliability Association) JSF web site (Joint Strike Fighter) Web site of the tool Understand
Internationalization & Localization Testing Globalisation = i18n + l10n I18N & l10n –OS –Keyboard –Display text –Date format –Number format –Etc.
Internationalization & Localization Testing To be performed once functional testing has been completed and all the functionality related defects are identified and removed 2 phases: First, to make sure that application’s GUI or functionality will not be broken with the translated text Second to make sure that translation of all the strings have happened properly
Recovery Testing Testing how well an application is able to recover from crashes, hardware failures and other similar problems. Recovery testing != reliability testing
References Srinivasan Desikan & Gopalaswamy Ramesh, Software Testing: Principles and Practice, Pearson Education J. Nielsen, "Heuristic Evaluation". In Jakob Nielsen and Robert L. Mack, editors, "Usability Inspection Methods". John Wiley and Sons, Inc Mauro, C.L.: Professional usability testing and return on investment as it applies to user interface design for web-based products and services