JUnit test distribution Bettina Scharpf ACM 2 University of Applied Science Furtwangen

04/07/2005Bettina Scharpf, ACM 22 Specification “Develop a Grid Service that partitions JUnit tests, does a balanced distribution and saves the results.” Technologies: JUnit ANT Web and Grid Services Globus Toolkit 4 and GRAM Proof of concept: partitioning efficient distribution

04/07/2005Bettina Scharpf, ACM 23 Agenda Introduction to JUnit Test partitioning –atoms Efficient distribution –execution hosts –algorithms Concept

04/07/2005Bettina Scharpf, ACM 24 Introduction to JUnit tests visible behavior –black and white box tests “units” of code -> Java classes by Erich Gamma and Kent Beck “test first”

04/07/2005Bettina Scharpf, ACM 25 Distinction between –errors (e.g. exceptions) and –failures (e.g. value not as expected) test passes or not - nothing inbetween Test runners –with graphical user interface –command line –... Introduction to JUnit

04/07/2005Bettina Scharpf, ACM 26 Organisation of JUnit tests TestSuite Test.run(testResult) TestCase setUp() tearDown() TestCase setUp() tearDown() TestCase setUp() tearDown() TestCase setUp() tearDown() TestCase setUp() tearDown() TestCase setUp() tearDown() TestCase setUp() tearDown() TestSetup setUp() tearDown() setUp() tearDown() TestCase setUp() tearDown()

04/07/2005Bettina Scharpf, ACM 27 Agenda Introduction to JUnit Test partitioning –atoms Efficient distribution –execution hosts –algorithms Concept

04/07/2005Bettina Scharpf, ACM 28 Single test cases Single test case wrapped by TestSetup’s setUp() and tearDown() but: TestSetup partitioning at any price? Test partitioning Atoms in JUnit TestSetup setUp() tearDown() setUp() tearDown() TestCase setUp() tearDown() TestCase setUp() tearDown() setUp() tearDown() TestCase setUp() tearDown() setUp() tearDown() TestCase setUp() tearDown() TestCase setUp() tearDown() =>+

04/07/2005Bettina Scharpf, ACM 29 Agenda Introduction to JUnit Test partitioning –atoms Efficient distribution –execution hosts –algorithms Concept

04/07/2005Bettina Scharpf, ACM 210 Efficient distribution Execution Hosts Problem: Hosts perform differently Solution: Equalize hosts Run reference test on each host Find slowest host Relate each host to the slowest one (using factor) Calculate execution time per slowest host Calculate quota for each particular host (using factor) Sum up known execution times

04/07/2005Bettina Scharpf, ACM 211 Optimal balance vs. preparation time: Check all possibilities (exponential) –Knapsack: must be run (n-1) times for n hosts + exact - very high complexity or Iterate: Assign largest test to host with most capacity left + not very complex + for most cases exact enough Efficient distribution Algorithms

04/07/2005Bettina Scharpf, ACM 212 Agenda Introduction to JUnit Test partitioning –atoms Efficient distribution –execution hosts –algorithms Concept

04/07/2005Bettina Scharpf, ACM 213 Problems: 1.) Partitioning: Optimal partitioning vs. longer cumulative execution time 2.) Efficient distribution: Optimal balance vs. test preparation time Solution: Must be something inbetween... Concept

04/07/2005Bettina Scharpf, ACM ) Partitioning of TestSetup objects – if necessary single test case contained in TestSetup object is too big complete TestSetup object is too big or –if fixture is not too big, amount of TestCases not too high (e. g. depending on the total execution time) Concept Solutions

04/07/2005Bettina Scharpf, ACM ) Balancing the tests –use knapsack for a small number of test cases (configurable) -> optimal “packaging” of tests for one host (each test has a weight and a value which are both the same here) -> iterate: remove assigned tests and “pack a knapsack” for the next host –“expensive first” assign tests iteratively, starting with the most expensive test case and the most performant host Concept Solutions else

04/07/2005Bettina Scharpf, ACM 216 The problem of atoms which are “too big” –has longer execution time than the most performant host (or the most performance left) –Solution put atom on the host with the most performance left calculate: how much longer will the test now take? re-calculate the execution time per slowest host apply “slowest host time” to all execution hosts (won’t extend the entire test duration) Distributing tests with unknown execution time: Round Robin, beginning with the most performant host Concept Solutions

04/07/2005Bettina Scharpf, ACM 217 Concept An Example host A host B host C host D test cases 1.) obtain factor 2.) calculate tests and execution time per host 2.0 (39 sec.) 3.) distribute s s s s14s 7 s 10.5s 14s s left 6.5 s left 8 s left 0 s left 4 s left 2.5 s left 3 s left2 s left1 s left 1.5 s left 2 s left 1 s left 0.5 s left

04/07/2005Bettina Scharpf, ACM 218 Fragen?