Mutation Testing to Evaluate Combinatorial Tests

Combinatorial testing (CT) is an effective and efficient approach to functional testing. Based on a test model with parameters and values, test inputs are generated using an algorithm (also called combination strategy) such that the resulting test suite satisfies a predefined coverage criteria.

In related work, we designed a coverage criterion and implemented a combination strategy for combinatorial robustness testing (CRT). The implementation is integrated into coffee4j, which is a Java-based framework for combinatorial test generation, execution and fault characterization (debugging). A JUnit 5 extension seamlessly integrates the popular test automation framework. First artificial experiments show that the new coverage criterion improves fault detection effectiveness and requires fewer test inputs compared to traditional CT coverage criteria.

Mutation testing is an approach to evaluate the quality of test cases. First, the system under test (SUT) is mutated: Faults are injected by small changes to the source code, e.g. in an expression + is replaced by -. Then, the same test cases are executed on the mutated SUT to check if the test cases are actually capable of detecting failures.

In this thesis, mutation testing should be used as an approach to evaluate combinatorial coverage criteria. Possible tasks in this thesis are as follows.

  • Define robustness-specific mutation operators
  • Define metrics to compare combinatorial coverage criteria
  • Extend an existing or develop your own mutation testing framework
  • Apply the mutation testing to Java applications


  • Type: Bachelor Thesis, Master Thesis
  • Status: Open