Making Combinatorial Testing Aware of Expected Results

Combinatorial testing evolved as an effective and efficient approach to generate test cases to reveal interaction faults [1, 2]. While there are a lot of different algorithms that allow to generate test suites of minimal size, the integration of expected results is still an open issue in combinatorial testing [3] and testing in general [4].

In practice, it is very unlikely to have automated oracles to compute expected results. Expected results are often hand-crafted for each test case manually which is a labour-intensive and expensive task. We believe the hand-crafted results are fine as long as they can be reused for a sufficiently long time. Hand-crafting expected results is labour-intensive and therefore costly. By reusing the results, the costs can be considered as initial upfront investments that can be distributed over time.

However, the input model is often developed iteratively which limits the reusability of expected results a lot: Every time a value is added or removed from the input model, the generated test suites differs a lot making reuse of expected results quite impossible.


The goal of this thesis is to adjust the test case generation to prefer test cases for which expected results already exist while still preserving the combinatorial coverage criteria. Since the reuse of handcrafted results is motivated by cost reduction, the approach should be based on a cost function and an algorithm that tries to minimise that function. There is already some work on cost-aware test case generation [2, 5]. However, it is not clear if the approach is applicable to the concept of expected results and how the cost function should look like.

This thesis topic includes the creation of a concept that includes likely changes that are iteratively applied to the input model and a cost model for hand-crafting expected results. Furthermore, it include the development of an integrated approach that allows to apply the ideas in practice.

Further Information

  1. Kuhn, D. R.; Kacker, R. N. & Lei, Y. Introduction to Combinatorial Testing. Chapman & Hall/CRC, 2013
  2. Yilmaz, C.; Fouche, S.; Cohen, M. B.; Porter, A.; Demiroz, G. & Koc, U. Moving forward with combinatorial interaction testing IEEE Computer Society, IEEE, 2014
  3. Kruse, P. M. Test Oracles and Test Script Generation in Combinatorial Testing 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2016
  4. Barr, E. T.; Harman, M.; McMinn, P.; Shahbaz, M. & Yoo, S.The Oracle Problem in Software Testing: A Survey IEEE Transactions on Software Engineering, Institute of Electrical and Electronics Engineers (IEEE), 2015
  5. Gulsen Demiroz and Cemal Yilmaz. Cost-Aware Combinatorial Interaction Testing Proceedings of International Conference on Advances in System Testing and Validation Lifecycle 2012


  • Type: Master Thesis
  • Status: Current
  • ID: 2019-015
  • Student: Kateryna Prokhorova
  • Language: German, English