Improving Test Oracles in a Java-based Combinatorial Testing Framework

Combinatorial testing (CT) is an effective and efficient approach to functional testing. Based on a test model with parameters and values, test inputs are generated using an algorithm (also called combination strategy) such that the resulting test suite satisfies a predefined coverage criteria. In related work, we designed a coverage criterion and implemented a combination strategy for combinatorial robustness testing (CRT). The implementation is integrated into coffee4j, which is a Java-based framework for combinatorial test generation, execution and fault characterization (debugging). A JUnit 5 extension seamlessly integrates the popular test automation framework.

One major yet unsolved problem in test automation relates to the test oracle, i.e. the entity that decides whether a test case succeeded or failed. A lot of programs are considered to be non-testable because no test oracle exists and the development of an automated test oracle or the use of a human test oracle is impractical because it would require an extraordinary amount of time and money. The idea of metamorphic testing is to create an approximate test oracle by defining relations over input and output of different test cases. For instance, when testing a search engine, a relation could specify that a more general search should yield more results than a more specific search. Then, a failure can be detected when one test case with no further restrictions on the search yields 10.00.000 results while another test cases that uses the same keyword but limits the results to  the last two years results in 10.500.000 findings.

The goal of this topic is to integrate metamorphic testing into the coffee4j framework, such that the test oracle for combinatorial tests can be defined via relationships.


  • Type: Bachelor Thesis, Master Thesis
  • Status: Open