1. 1 Testing the OPN Language: Rule Coverage and Fuzz Testing Wujie Zheng

2. 2 Outline The OPN Language Task 1: Rule Coverage Collection  Verifying the OPN definitions  A measurement of the test suite for the OPN compiler Task 2: Fuzz Testing  Test the type checker Conclusions and Future Work

3. 3 The OPN Language OPN (Open Protocol Notation) is a notation for describing architecture, messages, and behavior of network protocols.

4. The OPN Language Components  OPN definitions  The OPN compiler Parser (and lexer) Type checker Code generator  Runtime 4

5. The OPN Language Components  OPN definitions  The OPN compiler Parser (and lexer) Type checker Code generator  Runtime 5

6. The OPN Language Examples For Statements EmbeddedStatement ::= for ( ForInitializer ; Expression ; StatementList, ? ) EmbeddedStatement ForInitializer ::= VariableDeclaration (, VariableDeclaration )* | StatementList, ? StatementList ::= StatementExpression (, StatementExpression )* Endpoint Declarations EndpointDeclaration ::= VisibilityModifier* endpoint Identifier EndpointAddress? EndpointTransports Aspects? ContractElement+ ( { EndpointMember* } AspectsAfterBlock? | Aspects? ; ) 6

7. 7 Task 1: Rule Coverage Collection Rule Coverage  A criterion for testing grammars  A test case is said to cover a grammar rule if that rule is used at least once in deriving the test case For example, The following test module test; int x; covers the grammar rule Document ::= module QualifiedIdentifier Aspects? ; UsingDeclaration* ModuleMember*, and not cover the rule Document ::= protocol QualifiedIdentifier Aspects? ; UsingDeclaration* ProtocolMember*

8. Task 1: Rule Coverage Collection Rule Coverage Collection  Define a set of grammar rules (sub-classes of each rule) based on the OPN definitions  Implement an OPNRulePrinter to traverse the POM objects to get the rules covered by the test cases  Report the rule coverage result 8

9. 9

10. Task 1: Rule Coverage Collection Purpose 1: verifying the OPN definitions  To collect the rule coverage, we need to identify the sub-rules of each grammar rule and visit each component of a rule, which can identify Missing definitions Wrong definitions Some bugs can be identified by experience, some may due to conflicts between the definitions and the implementations 10

11. Task 1: Rule Coverage Collection Purpose 1: verifying the OPN definitions  Results: find 17 bugs in OPN definitions Missing definitions Bug 6083: Definition of “ReferenceExpression” missed in the specification Bug 15386: [Doc] Range expressions are implemented but not defined … Wrong definitions Bug 14971: [Doc] the definition of Switch Statements misses DefaultCase … 11

12. Task 1: Rule Coverage Collection Purpose 2: provide a measurement of the test suite for the OPN compiler  To understand how well the OPN grammar has been tested  Compiler errors may occur in some cases of a grammar rule but not in other cases Define sub-classes of each rule, so as to check corner cases For example, for the definition Document ::= protocol QualifiedIdentifier Aspects? ; UsingDeclaration* ProtocolMember* 6 rules are defined: Document_Protocol_Aspects, Document_Protocol_NoAspects, Document_Protocol_NoProtocolMember, Document_Protocol_ProtocolMember, Document_Protocol_NoUsingDeclaration, and Document_Protocol_UsingDeclaration. 12

13. Task 1: Rule Coverage Collection Purpose 2: provide a measurement of the test suite for the OPN compiler  Results: Define 518 rules Measure the rule coverage of the Tests.OpnExportRoundtrip tests: 459/518=88.6% A detailed rule coverage report 13

14. Task 2: Fuzz Testing Fuzz Testing  A software testing technique that provides invalid, unexpected, or random data to the inputs of a program.  If the program fails (for example, by crashing, hanging, or failing built-in code assertions), the defects can be noted. 14

15. Task 2: Fuzz Testing Randomly Mutate The Inputs 15 Original Input Mutated Input

16. Task 2: Fuzz Testing Fuzz Testing for the type checker  Given a test input, randomly mutates some parts of it and generates more test inputs.  Two kinds of mutation Unstructured mutation  Mutate lines or words of an OPN input.  Supports three operators: change, add, and remove. Structured mutation  Mutate identifiers of a POM input (generated from an OPN input).  Supports only the change operator now. 16

17. Task 2: Fuzz Testing Fuzz Testing for the type checker  How to check the test result of mutated inputs? The program under test (mainly the type checker) should not crash or hang. If a mutated input passes the type checker, it is suspicious The type checker should not throw exceptions without any error messages. A mutated input that increases the code coverage may deserve an investigation. Randomly inspect some mutated inputs and the execution results. 17

18. Task 2: Fuzz Testing Fuzz Testing for the type checker  How to check the test result of mutated inputs? Log the mutated inputs and the execution results for investigation  The mutated inputs and the changes  Whether the mutated inputs pass/fail the type checker  Detailed error messages 18

19. Task 2: Fuzz Testing Fuzz Testing for the type checker  Results Find 9 bugs in the type checker  Hangs Bug 14982: [TypeChecker] "endpoint U over U" crashes the type checker  Exceptions with error messages Bug 15015: [TypeChecker] type checker throws exception without any error message for code related to Aspects  Type checker passes Bug 15616: [TypeChecker] type checker should report errors for the comparison between 'int' and 'method‘ … 19

20. Task 2: Fuzz Testing Fuzz Testing for the type checker  Experience Structured mutation is much more effective than unstructured mutation  Generate error-inducing inputs more quickly The mutated inputs are syntactically correct, so we can focus on testing the type checker  More easy to identify suspicious results for manual inspection Change of identifiers can result in conflict of type information more easily, so it is more suspicious for the type checker to pass the mutated inputs 20

21. 21 Rule coverage collection is good for verifying OPN definitions  Need to check the definitions carefully, otherwise don’t know how to implement the rule coverage collecting program  Also check the consistence between the definitions and the implementations The test suites have achieved high rule coverage Future work  To define rules for the combinations of components of each rule Conclusions and Future Work

22. 22 Fuzz testing is good for verifying the type checker  Especially helpful for ensuring the robustness  Also find semantic bugs Future work  To systematically mutate the POM objects using symbolic execution techniques Conclusions and Future Work

23. 23 Thank you! Q&A