ROBUSTA – Combinatorial Robustness Testing

Combinatorial testing (CT) is a well-known black-box test design approach. Based on the system’s specification, test inputs are generated to systematically detect errors in a system under test (SUT).

ROBUSTA: The degree to which systems can deal with invalid inputs and stressful environmental conditions, i.e. robustness, is an important characteristic of software systems. Robust is often implemented by if-statements that detect invalid inputs and exceptional control-flows that activate error-handlers to recover the system. Because the exceptional control-flow. When errors are detected, the normal control-flow ist left. These systems require different test inputs to prevent the so-called input masking effect. Therefore, this research project aims to improve the activities of modelling and generating test inputs to test the robustness of software systems.

But, the generation of effective test suites is not the only challenge in CT. After test execution, the test inputs of failing tests require further analysis to reveal failure-inducing combinations. Manual analysis is a tedious and cumbersome task, especially for SUTs with many input parameters and values.

COFFEe: Therefore, this research project aims to to fully automate the combinatorial (robustness) testing by deeply integrating testing lifecycle, the activities of test input modelling, test input generation, test method execution and fault characterization. COFFEe is a framework which integrates the aforementioned steps as a JUnit5 extension, which makes it easy to integrate it into software development and build processes.

coffee4j is the reference implementation of ROBUSTA and COFFEe and can be found accessed via



  • [URL] K. Fögen, H. Lichter (2019): Repairing Over-Constrained Models for Combinatorial Robustness Testing. In IEEE International Workshop of Combinatorial Testing and its Applications (to be published).
  • [URL] K. Fögen, H. Lichter (2019): Combinatorial Robustness Testing with Negative Test Cases. In 19th IEEE International Conference on Software Quality, Reliability, and Security (to be published).
  • [URL] [DOI] J. Bonn, K. Fögen, H. Lichter (2019): A Framework for Automated Combinatorial Test Generation, Execution, and Fault Characterization. In Proceedings of the 8th International Workshop on Combinatorial Testing (IWCT 2019), IEEE, 224-233.
  • [URL] K. Fögen, H. Lichter (2018): A Case Study on Robustness Fault Characteristics for Combinatorial Testing – Results and Challenges. In Proceedings of the 6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018) co-located with APSEC 2018, Nara, Japan, December 4, 2018,, Vol. 2273, 22-29.
  • [URL] [DOI] K. Fögen, H. Lichter (2018): Combinatorial Testing with Constraints for Negative Test Cases. In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Västerås, Sweden, 328-331.
  • [URL] V. von Hof, K. Fögen, H. Kuchen (2017): Detecting Spring Configurations Errors. In Proceedings of the 32nd ACM Symposium on Applied Computing, SAC 2017, Marrakech, Marocco, April 3-7, 2017, 1505–1512.
  • [URL] H. Lichter, K. Fögen, T. Sunetnanta, T. Anwar (2016): Report on the 4th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2016). In Joint Proceedings of the 4th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2016) and 1st International Workshop on Technical Debt Analytics (TDA 2016) co-located with APSEC 2016, Hamilton, New Zealand, December 6, 2016,, Vol. 1559, 1-3.
  • [URL] V. von Hof, K. Fögen, H. Kuchen (2016): Compilezeit-Prüfung von Spring-Konfigurationen. In Gemeinsamer Tagungsband der Workshops der Tagung Software Engineering 2016 (SE 2016), Wien,, Vol. 1559, 96-108.
  • [URL] K. Fögen, V. H. H. von Kuchen (2015): Checking Spring Annotations. In 18. Kolloquium Programmiersprachen und Grundlagen der Programmierung (KPS) 2015, Pörtschach am Wörthersee; 2015-10-04 – 2015-10-07.



Generali Deutschland Informatik Services