Presentation is loading. Please wait.

Presentation is loading. Please wait.

What, why and how? Johnny Bigert, 2012-10-25

Similar presentations

Presentation on theme: "What, why and how? Johnny Bigert, 2012-10-25"— Presentation transcript:

1 What, why and how? Johnny Bigert, 2012-10-25

2 My View on Unit Testing Part of something bigger – testing on many levels 1. Fast and early feedback! Tight feedback loop, results within seconds Early implies lower cost 2. Confidence! Less fear of refactoring 3. Better code! Testability requires decoupling and usable APIs 4. Reliability! Possible to remove all non-determinism

3 Unit test, functional test, system test – what does it mean?

4 Definitions and Terminology There are no definitive answers Different dimensions of automated tests: Level – Unit, component, integration, system API – Developer test, external test Characteristic – Functional, non-functional Acceptance testing, regression testing

5 What Level Are We Testing On? Unit – The smallest unit of program code, normally a single function, class with methods or data structure with functions Component – A set of units integrated together, normally a module or subsystem comprising an isolated set of classes or files. Integration – Several layers of a stack integrated, normally a stand-alone application (e.g. client or server) System – Several stand-alone applications integrated (e.g. clients and servers), normally the full system

6 What API Are We Using For Testing? Developer tests – Function calls to an internal or external interface, normally test and production code running in the same process External tests – Invokation of a non-programming interface, normally running in separate processes or machines. For example, UI (simulated user input), network (HTTP etc), IPC.

7 What Aspect Are We Testing? Functional – The behavior of the system, normally expressed as a step-by-step instruction with some alternative paths Non-functional – The characteristics of the system, normally expressed as measurements and numbers. For example, latency, round-trip time, memory/CPU/bandwidth consumption, scalability, availability.

8 Functional DeveloperExternal Unitunit testN/A Componentcomponent test Integrationfunctional test SystemN/Asystem test Component test – personal favorite. Provides stability just like a functional test (test code less brittle than unit tests). Provides flexibility, reliability and fast feedback just like a unit test (quick to write, easy to mock out problematic parts, results within milliseconds).

9 Non-Functional DeveloperExternal UnitN/A Componente.g. test threading model of a component Integratione.g. measure CPU consumption SystemN/Ae.g. load test

10 Choose How to Test When choosing, consider Cost (of writing, maintaining, execution resources etc.) Feedback loop (time) Reliability Trade-off example: Mocking or using system x? Mocking: cost of writing and maintaining, risk of modeling wrong behavior. Using: cost/risk of non-determinism, cost of test resources and longer execution times.

11 Regression And Acceptance Testing Definition: regression or regression bug Something that was once considered done but does not fulfill requirements anymore Acceptance testing: the use of a test to determine if a feature fulfills the requirements Acceptance tests becomes part of the regression test suite once a feature is done

12 Necessities of effective unit and component testing

13 Dependencies Assumption: we want to achieve fast feedback and reliable tests Note, there are other kinds of tests that also have merit Remove non-determinism: we need to be in control Randomness, time, file system, network, databases, multi-threading Remove dependencies to non-deterministic components

14 Dependencies What kind of problematic dependencies to you have in your code base?

15 Getting Rid of Dependencies, C++ Example: Randomness class A { public: void foo() { bool coinflip = rand() % 2 == 0; if(coinflip) … } }; Problem: dependency to non-deterministic component (randomness)

16 Getting Rid of Dependencies, C++ C++ solution: introduce interface and use dependency injection class IRandomness { virtual bool coinflip() = 0; };

17 Getting Rid of Dependencies, C++ class A { public: A(IRandomness &r) : randomness(r) {} // ctor inject void foo() { if(randomness.coinflip()) … } private: IRandomness &randomness; };

18 What Is a Mock Object? Conforms to interface Programmable behavior For example, if someone calls your send function with argument hello, return 7 Can verify function invocations Mock frameworks available E.g. gmock (Google) Much more elegant in reflective languages

19 Getting Rid of Dependencies, C++ Test code: void testFoo() { RandomnessMock random; // inherits IRandomness A a(random);; … }

20 Getting Rid of Dependencies, C++ Production code: void main() { Randomness random; // inherits IRandomness A a(random); … }

21 Bi-Directional Dependencies sendData (inherits) onDataReceived (inherits) Application Network INetworkReceiverINetworkSender

22 Getting Rid of Dependencies, C Function pointers = hard-to-read code Link-time tricks Link with other translation unit defining e.g. rand()

23 Getting Rid of Dependencies, C RandMock m; // mixed C/C++ test code RandMock &getRandMock() { return m; // used by test case to access mock } int rand() { m.rand(); // forward to mock object } Drawbacks: hard to understand test case code, one test executable per component to test = harder to maintain Or clean C solution

24 Code Coverage Dont focus on high levels of code coverage per se Necessary, but not sufficient Instead, focus on covering the most important user scenarios After that, use code coverage to determine which code has not been covered Analyze to understand what user scenarios are missing

25 Unit Test Frameworks Test runners/frameworks GTest UnitTest++ CppUnit C Unit Testing Framework Mock frameworks GMock Hippo Mockpp Amop

26 Agree on what to aim for

27 Fast Feedback Example From a previous job: Proprietary hardware/software/OS platform Very expensive hardware = bottleneck in testing Emulator to test software locally on Linux machine Ran unit tests in emulator, turn-around time = 15 mins Unacceptable

28 Fast Feedback Example Main problem: emulator start-up time Challenge: proprietary types used (e.g. string classes) Solution: fake proprietary types (string with std::string, many others just empty implementations) Ran unit tests on local machine, turn-around time < 15 seconds (from 15 mins)

29 Fast Feedback Example Invest to speed up feedback Invest to remove testability obstacles Unit testing should be easy, fast and fun

30 Development Excellence Defect prevention Continuous improvement - feedback from bugs in order to improve software development process Relentless testing Regression detection Continuous integration running all regression tests often Focus on both functional and non-functional requirements Always releasable Keep CI builds green Keep the product working

31 Continuous Integration Staged Stage 1 – build, unit tests, code coverage and functional smoke tests – 10 mins Stage 2 – extensive functional tests, smoke non- functional tests – 1 hour Stage 3 – full functional and non-functional regression tests, static and dynamic analysis – 12 hours Run stages per Commit (fine-grained tracability, expensive), or Regularly (1. commit, 2. hourly, 3. nightly)

32 Code Analysis Tools Static analysis (run on source code) Lint – free, local analysis only, lots of false positives Klocwork – commercial, global analysis Coverity – expensive, global analysis, good precision Dynamic analysis (need representative data) IBM Rational Purify – commercial, memory leaks, double frees, buffer overflow, use of uninitialized memory. Instrumentation. Valgrind – free, memory problems, heap, usage, cache usage, call graphs, threading problems. No instrumentation.

Download ppt "What, why and how? Johnny Bigert, 2012-10-25"

Similar presentations

Ads by Google