1. TestNG … and improving use of JUnit Dec 11, 2008 C. Gordon Huffman

2. 2 Agenda (not chronological) TestNG testing framework Improving use of JUnit Testing integrated classes vs. 1 class

3. 3 What is TestNG? Automated testing framework NG = Next Generation Similar to JUnit (especially JUnit 4) Not a JUnit extension (but inspired by JUnit) Designed to be better than JUnit, especially when testing integrated classes Created by Dr. Cédric Beust (of Google) Open source (http://testng.org)

4. 4 History sUnit for Smalltalk (~1998, Kent Beck) JUnit (~2000, Kent Beck & Erich Gamma) –Latest 3.x: 3.8.2 (March 2006) TestNG (~2004) –Latest: 5.8 (March 2008) JUnit 4.x (2006) –Latest: 4.5 (August 2008)

5. 5 TestNG philosophy Use more Java and OO features Feature-rich (JUnit: simplicity is valued) Support testing integrated classes (e.g., by default, don’t create a new test class instance for every test method). Separate compile-time test code from run- time configuration/data info.

6. 6 Basic Testing Terms Test class Class(es) under test (“Production” class) “CUT” (class under test) “SUT” (system under test)

7. 7 “Rhythm” of an Automated Test Setup (most of it in a separate method) Exercise (call) production class(es) Assertion(s); true => silence; false => fail Teardown (if needed, in a separate method)

8. 8 Test class/method (JUnit 3 vs. 4) // JUnit 3 import junit.framework.TestCase; public class MyTestClass extends TestCase { public void testSomething1() throws... {... } public void testSomething2() throws... {... } } // JUnit 4 import org.junit.Test; public class MyTestClass { @Test public void aTest1() throws... {... } @Test public void aTest2() throws... {... } }

9. 9 Test class/method (JUnit 4 vs. TestNG) import org.junit.Test; // JUnit 4... or import org.testng.annotations.Test; // TestNG public class MyTestClass { @Test public void aTestMethod() throws... {... } }

10. 10 TestNG Assertions import static org.testng.Assert.*; import org.testng.annotations.Test; public class MyTest { @Test public void myTestMethod() { //... Possibly some setup //... Call production method(s) assertTrue(boolExpression); //... more assertions }

11. 11 Assertions: TestNG vs. JUnit 3/4 JUnit 3 assertions inherited by base TestCase JUnit 4: similar to TestNG TestNG assertEquals(…) assertion signatures are different than JUnit’s: –JUnit: [msg,] expected, actual –TestNG: actual, expected [, msg] Can use JUnit 3/4 assertions (library issues) Can use TestNG’s copy of JUnit 3 assertions (org.testng.AssertJUnit): migration aid. JUnit4/TestNG assertions richer than JUnit 3; each contains features that the other doesn’t.

12. 12 Expected Exceptions JUnit 4 (can simplify test code) @Test(expected = AThrowable.class) TestNG (can have more than one exception) @Test(expectedExceptions = AThrowable.class) Or... @Test(expectedExceptions = { T1.class,... })

13. 13 TestNG “Groups” Each test method is tagged with any # of groups. –@Test // no groups –@Test (groups = “group1”) –@Test (groups = { “g1”, “g2”,... }) A group therefore contains any # of test methods. Can also annotate test class this way; test method groups are union of any explicit test method groups, and any of its test class. Groups can span classes. Groups can also be externally defined (TestNG xml configuration file). A group is identified by a unique string (don’t use white space). –There are no pre-defined group names. –E.g., “slow”, “db”, “ui”, “unit”, “integration”, “broken.unknownReason”, “check-in”, “week-end”, “functional.billing”

14. 14 TestNG Groups (continued) TestNG community suggests hierarchical names from more general to less. E.g.: –db.table.CUSTOMER –state.broken.fix-in-progress Design group names so that you can select them with prefix patterns. Groups complement other features (as we’ll see).

15. 15 Setup/Teardown (JUnit 3/4) JUnit 3 protected void setUp() throws Exception {... } protected void tearDown() throws Exception {... } JUnit 4 (also, @After*; annotation package: org.junit) @BeforeClass public static void before1()... @Before public void before2()... Multiple per class (order not defined) These 4 annotations are “inherited” (order is [kind of] defined).

16. 16 Setup/Teardown (TestNG) @BeforeMethod @BeforeClass (no need to be static) @BeforeGroups ({“group1”, …}) @BeforeTest @BeforeSuite And @After*

17. 17 TestNG run-time concepts A org.testng.TestNG instance runs tests. –Invoked by IDE, ant, or on command line. –Run params specified as setters, or in xml. Hierarchy of suite/test/class/method. A TestNG run executes 1 or more suites. –Can be transparent, e.g., run 1 test method via Eclipse.

18. 18 testng.xml Controls test runs Optional File can have any name (.xml suffix advised) –TestNG creates “testng-failed.xml”, for easy re-run. Root: (then / / ) Suite can also point to a list of additional xml suite files. Some of the available specifications are described next.

19. 19 testng.xml (continued) Which potential test methods to run: suite/test: list of java package name patterns to include/exclude; test: list of fully qualified class names to include; within a class, optional (default = include all) list of method names to include/exclude; test: list of group name patterns to include/exclude (and can define new groups for this file with a list of existing group names). Parameter name/value pairs: Suite/test: (primitive values) for @Parameters (test values override suite values). JUnit semantics: test-level flag (default: false); migration aid.

20. 20 Disable Tests TestNG –@Test(enabled = false) –Add to a group which is excluded –Exclude in other ways in testng.xml JUnit 4: @Ignore, @Ignore(“reason”) –Can also annotate test class. –Runners report # ignored

21. 21 Test Timeouts TestNG @Test(timeout = 1000) testng.xml time-out attribute JUnit 4 @Test(timeout = 1000)

22. 22 The Costs of Testing Time, money, schedule, production scope. Doubled (very roughly) code base. –Maintenance (production and test). Sloppily written tests?? Is test code quality as good as production? Slow tests? True buy in from business sponsors/owners, project managers, user reps?

23. 23 “High View” of Test Benefits “High View”: using tests to best advantage “Do the thing right” “Do the right thing” (acceptance tests) Safety net for modifications/refactoring Defect localization (1 fail is more info than 100) Documentation/training Minimize costs of testing This view leads to better tests

24. 24 Integration vs. Unit Tests Unit test: tests 1 invocation of 1 method of 1 isolated class with 1 thread Integration test: tests more than 1 method invocation of an isolated class, often much more, or more than 1 thread Distinction: unit test vs. a test written in JUnit Examples –Test hits db –Life-cycle of an object (e.g., a customer order) –Testing units by testing integrated classes

25. 25 Integration vs. Unit (continued) Should both be tested? Unit –How big do you want the holes in your refactoring safety net? –How many permutations can integration tests handle? Integration –If the units work independently, will they work together?

26. 26 If Unit Tests are Really Integration Tests Slow tests. False sense of security for refactoring safety net. Permutations of business rules likely not covered. Poor defect isolation: 100 tests failing is less information than 1 test failing. High maintenance for test/production modification. Counter to the goals of TDD and Agility.

27. 27 Test Method Dependencies JUnit 3 : Antithetical concept; each test method gets its own test class instance; high isolation; unit oriented by intentional design TestNG: –@Test(dependsOnMethods = {“m1”, “m2”,...}) –@Test(dependsOnGroups = {“g1”, “g2”,...}) –E.g.: configuration dependencies; production lifecycle JUnit 4: tests ignored if “assumptions” (similar to assertions) are violated: assumeTrue(bool), assumeNoException(Throwable), assumeNotNull(refs …), assumeThat(…) high-readability method

28. 28 Parallel (multiple threads) @Test(threadPoolSize = #, invocationCount = #, timeout = #) testng.xml: –<suite|test parallel =“tests|methods|none” thread-count=“#” (5); 1 if not parallel time-out =“#” ms (default: 0=none)...> Testing production code with multiple threads (unless not supported); stress testing; speeding up tests!

29. 29 Test Method Parameters I @Test @Parameters({“name1”, “name2”}) public void aTest(String name1, int name2) {... } Values come from testng.xml ( is checked first, then ). Only basic types (String, [Bb]ool, #’s, …).

30. 30 Test Method Parameters II @Test(dataProvider = “myDataProvider”) public void aTestMethod(/* params! */) {... } @DataProvider(name = “myDataProvider”) public Object[][] getSomeTestData() {... } Runs (possibly multiple: first []) parameterized tests. Multiple parameters: second [] Return type can also be Iterator Data provider itself can take 0, 1, or 2 args: Method, and test context

31. 31 Test Class Factories @Factory public static Object[] myTestFactory() { // return instances of test classes } Those instances will also be interrogated for @Factory methods, and so on. (Each such method invoked once.) Example use: you want some of your test classes to have non-standard constructors; the factory constructs them.

32. 32 Migrating JUnit 3 => TestNG TestNG tool that modifies source code by adding imports, @Test, @BeforeTest, @AfterTest, and an optional list of desired group names; tool accessible via command line, ant, or Eclipse/IDEA. Can use testng.xml for tests that need further migration analysis. That’s not necessarily enough; for other warnings and tips, see Appendix D of [TestNG].

33. 33 More TestNG and JUnit 4 TestNG and JUnit 4 each have other features not mentioned here. For more TestNG features, see [TestNG]. For more JUnit 4 features, see JUnit 4 docs and the source code (e.g., Hamcrest).

34. 34 Reference [TestNG] Next Generation Java TM Testing ; TestNG and Advanced Concepts; Cédric Beust and Hani Suleiman; Addison-Wesley; ©2008