A Safety-Critical Java Technology Compatibility Kit Hans Søndergaard Stephan Korsholm VIA University College, Horsens, Denmark & Anders P. Ravn Aalborg University JTRES 2014 October 2014

Test Suite – why? Java Community Process (JCP) The formal process for developing or revising Java technology Specifications. Final Release The final stage in a JSR when the Specification, RI, and TCK have been completed. Technology Compatibility Kit (TCK) The suite of tests, tools, and documentation that allows an organization to determine if its implementation is compliant with the Specification. 2

Ingredients in a TCK Tests Test cases and test datasets Tools for test execution Execute the tests and generate results Test report documents Classification of results 3

Test Case Development Checks conformance with a specification  Must be implementation independent: - data representations must be hidden  Black-box testing using public entities only Must check specification of classes  Invariants and Pre- and Post-conditions ?Intuition or formalization in development ? !JML (Java Modeling Language) 4

Background Anders P. Ravn and Hans Søndergaard. A Test Suite for Safety- Critical Java using JML Yoonsik Cheon and Gary T. Leavens. A Simple and Practical Approach to Unit Testing: The JML and JUnit Way A. Sarcar and Y. Cheon. A new Eclipse-based JML compiler built using AST merging JML4c L. Zhao, D. Tang, and J. Vitek. A Technology Compatibility Kit for Safety Critical Java

Specification of Invariants Class: public abstract class HighResolutionTime Specification: ” A time object in normalized form represents negative time if both components are nonzero and negative, or one is nonzero and negative and the other is zero.” Invariant: public invariant (this.getMilliSeconds() >= 0 && (0 <= this.getNanoSeconds() && this.getNanoSeconds() < )) || (this.getMilliSeconds() <= 0 && ( < this.getNanoSeconds() && this.getNanoSeconds() <= 0)); 6

Pre/Post-conditions Normal behaviour Class: public abstract class HighResolutionTime Specification: public void set(HighResolutionTime time) “Change the value represented by this to that of the given time. … Throws ClassCastException if the time parameter is not of the same class as this. Throws IllegalArgumentException if the time parameter is null....” public normal_behaviour requires time != null && this.getClass() == time.getClass(); ensures this.getMilliseconds() == time.getMilliseconds(); ensures this.getNanoseconds() == time.getNanoseconds(); ensures this.getClock() == time.getClock(); 7

Exceptions Class: public abstract class HighResolutionTime Specification: public void set(HighResolutionTime time) “Change the value represented by this to that of the given time. … Throws ClassCastException if the time parameter is not of the same class as this. Throws IllegalArgumentException if the time parameter is null....” also public exceptional_behaviour requires time == null; signals (IllegalArgumentException) true; public exceptional_behaviour requires time != null && this.getClass() != time.getClass(); signals (ClassCastException) true; 8

JML annotations compiled and executed Compilation of JML annotations – Uses a JML compiler – Compiled to Java bytecode – Translated to runtime assertion checks Execution of the extended Java bytecode – Constructor: Runtime check of Precondition Constructor execution Runtime check of Postcondition and Invariant – Method: Runtime check of Invariant and Precondition Method execution Runtime check of Postcondition and Invariant Violation of runtime checks – Throws a JMLAssertionError

Test Case Using class TestCase from JUnit public class TestAbsoluteTime extends TestCase {... public void test(int i) { AbsoluteTime abs; switch (i) {... // AbsoluteTime(long millis, int nanos) case 32: new AbsoluteTime(0,0); break; case 33: new AbsoluteTime(0, ); break;... // wrap around value case 37: abs = new AbsoluteTime(Long.MAX_VALUE, ); assert abs.getNanoseconds() == ; break;...

Test Cases - exceptions // AbsoluteTime(AbsoluteTime time) case 39: abs = new AbsoluteTime(); new AbsoluteTime(abs); break; case 40: abs = null; try{ new AbsoluteTime(abs); assert false; } catch (IllegalArgumentException e){}; break;... case 91:... default: break; } public static final int testCount = 91;

Beyond Unit Testing 12 Class: public abstract class Clock Specification: public abstract RelativeTime getResolution(); “Gets the resolution of the clock defined as the nominal interval between ticks. “ public behaviour requires true; ensures \result != null; ensures (\result.getMilliSeconds() > 0 || \result.getMilliSeconds() == 0 && \result.getNanoSeconds() > 0); Is unit testing sufficient for a Clock?

Specification of clock test method 13 Specification: behaviour requires true; ensures !failure; ensures (\forall int i; 0 < i && i < SIZE; sample[i-1].compareTo(sample[i]) < 0); // time is moving forward ensures (\forall int i; 0 < i && i < SIZE; (sample[i].subtract(sample[i-1])).compareTo(c.getResolution()) >= 0 ); // distance between two samples >= resolution Use JML to specify test methods! case 14: clockTest(clk); break;

Implementation of the clock test method 14 boolean failure; AbsoluteTime[] sample; RelativeTime resolution; behaviour... // JML specification here void clockTest(Clock c) { resolution = c.getResolution(); sample = new AbsoluteTime[SIZE]; failure = false; sample[0] = c.getTime(sample[0]); for (int i = 1; i < SIZE; i++){ int j = 0; do { sample[i] = c.getTime(sample[i]); j++; } while ( sample[i].subtract(sample[i-1]). compareTo (resolution) < 0 && j < MAXLOOPS ); if (j == MAXLOOPS) { failure = true; break;} }

Tools for test execution Requirements – Executable on different VMs, e.g. VMs for resource constrained platforms HVM has been extended with reflection JML compilers – jmlc works with Java 1.4 source files – jml4c works with Java 1.5, including generics built on the Eclipse Java compiler JUnit – a subset of junit is used package framework – class TestResult has been extended add JML error to a list of errors 15

TestSuite assembly and execution public class AllTests { public static final TestResult result = new TestResult(); public static void main (String[] args) { TestSuite suite = new TestSuite(); suite.addTest(test_AbsoluteTime); suite.addTest(test_RelativeTime);... suite.run(result);... // TestCase objects defined here (next slide) } 16

TestCase object public class AllTests { public static final TestResult result = new TestResult();... public static TestCase test_AbsoluteTime = new TestAbsoluteTime(”AbsoluteTime”) { public void runTest () { try { for (int i = 1; i <= TestAbsoluteTime.testCount; i++) test(i); } catch (JMLAssertionError e) {result.addJMLError(this, e);} catch (Throwable e) { result.addError(this, e); } } }; } 17

Test report No errors Test cases: 2 Test errors: 0 JML errors: 0 If e.g. an JML error Test number is 67, By method AbsoluteTime.add Regarding specifications at File "./src/javax/realtime/AbsoluteTime.java", line 216, character 15 With values nanos: millis: 0

Where are we? javax.realtime 47 classes and interfaces 24 finished 15 are Empty

Where are we? javax.safetycritical 35 classes and interfaces 12 (nearly) finished 3 are Empty

Conclusion Tests JML is an efficient vehicle for generating test conditions and may even improve the informal specifications. Concrete Test cases Guided by specifications and also additional cases based on “intuition”. Tools for test execution The tools involved are able to run on a resource constrained VM (HVM). Test report documents A simple test report. 21

Conformance tests of other SCJ implementations Extract the JML specifications from our SCJ implementation (simple) Merge this extract with another SCJ implementation (more complicated) Use the test cases (no change) Compile classes with JML annotations into Java bytecode, using jml4c (? depends of the VM) Run the test suites, using (part of) JUnit (simple). 22