Mathematics in Computer Science

, Volume 12, Issue 1, pp 51–67 | Cite as

An Efficient Design and Implementation of the In-Parameter-Order Algorithm

  • Kristoffer Kleine
  • Dimitris E. SimosEmail author


The In-Parameter-Order-General algorithm is one of the most widely used algorithms for covering array generation. It has found wide adoption in the field of combinatorial testing as it produces acceptably small covering arrays while being faster than alternative algorithms. In this paper we describe in detail how to efficiently design an efficient implementation, referred to as Fast In-Parameter-Order-General algorithm, which improves test generation times significantly while maintaining a low memory footprint compared to existing implementations. Besides the general algorithmic engineering and optimization considerations, we analyse and evaluate their benefits in an experimental study.


Covering arrays Generation Optimization Algorithm Performance 

Mathematics Subject Classification

Primary 05B15 Secondary 68W01 68R99 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



The research presented in this paper has been funded in part by the Austrian Research Promotion Agency (FFG) under grant 851205 (Security ProtocoL Interaction Testing in Practice - SPLIT) and the Austrian COMET Program (FFG).


  1. 1.
    Bozic, J., Garn, B., Simos, D.E., Wotawa, F.: Evaluation of the IPO-family algorithms for test case generation in web security testing. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–10. IEEE (2015)Google Scholar
  2. 2.
    Bryce, R.C., Colbourn, C.J.: The density algorithm for pairwise interaction testing. Softw. Test. Verif. Reliab. 17(3), 159–182 (2007)CrossRefGoogle Scholar
  3. 3.
    Bryce, R.C., Colbourn, C.J.: A density-based greedy algorithm for higher strength covering arrays. Softw. Test. Verif. Reliab. 19(1), 37–53 (2009)CrossRefGoogle Scholar
  4. 4.
    Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: an approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 23(7), 437–444 (1997)CrossRefGoogle Scholar
  5. 5.
    Colbourn, C.J., Dinitz, J.H.: Handbook of Combinatorial Designs. CRC Press, Boca Raton (2006)CrossRefzbMATHGoogle Scholar
  6. 6.
    Duan, F., Lei, Y., Yu, L., Kacker, R.N., Kuhn, D.R.: Improving IPOG’s vertical growth based on a graph coloring scheme. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–8. IEEE (2015)Google Scholar
  7. 7.
    Forbes, M., Lawrence, J., Lei, Y., Kacker, R.N., Kuhn, D.R.: Refining the in-parameter-order strategy for constructing covering arrays. J. Res. Natl. Inst. Stand. Technol. 113(5), 287 (2008)CrossRefGoogle Scholar
  8. 8.
    Gao, S.-W., Lv, J.-H., Bing-Lei, D., Colbourn, C.J., Ma, S.-L.: Balancing frequencies and fault detection in the in-parameter-order algorithm. J. Comput. Sci. Technol. 30(5), 957–968 (2015)CrossRefGoogle Scholar
  9. 9.
    Gao, S., Lv, J., Du, B., Jiang, Y., Ma, S.: General optimization strategies for refining the in-parameter-order algorithm. In: 2014 14th International Conference on Quality Software (QSIC), pp. 21–26. IEEE (2014)Google Scholar
  10. 10.
    Hartman, A., Raskin, L.: Problems and algorithms for covering arrays. Discret. Math. 284(1), 149–156 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Kitsos, P., Simos, D.E., Torres-Jimenez, J., Voyiatzis, A.G.: Exciting FPGA cryptographic trojans using combinatorial testing. In: 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), pp. 69–76. IEEE (2015)Google Scholar
  12. 12.
    Kleine, K., Simos, D.E.: Coveringcerts: combinatorial methods for x. 509 certificate testing. In: 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 69–79. IEEE (2017)Google Scholar
  13. 13.
    Kreher, D.L., Stinson, D.R.: Combinatorial Algorithms: Generation, Enumeration, and Search, vol. 7. CRC Press, Boca Raton (1998)zbMATHGoogle Scholar
  14. 14.
    Kuhn, D.R., Bryce, R., Duan, F., Ghandehari, L.S., Lei, Y., Kacker, R.N.: Combinatorial testing: theory and practice. Adv. Comput. 99, 1–66 (2015)CrossRefGoogle Scholar
  15. 15.
    Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. 30(6), 418–421 (2004)CrossRefGoogle Scholar
  16. 16.
    Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: IPOG: a general strategy for t-way software testing. In: 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems, 2007. ECBS’07, pp. 549–556. IEEE (2007)Google Scholar
  17. 17.
    Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: IPOG/IPOG-D: efficient test generation for multi-way combinatorial testing. Softw. Test. Verif. Reliab. 18(3), 125–148 (2008)CrossRefGoogle Scholar
  18. 18.
    Lei, Y., Tai, K.-C.: In-parameter-order: a test generation strategy for pairwise testing. In: High-Assurance Systems Engineering Symposium, 1998. Proceedings. Third IEEE International, pp. 254–261. IEEE (1998)Google Scholar
  19. 19.
    Seroussi, G., Bshouty, N.H.: Vector sets for exhaustive testing of logic circuits. IEEE Trans. Inf. Theory 34(3), 513–522 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Torres-Jimenez, J., Izquierdo-Marquez, I.: Survey of covering arrays. In: 2013 15th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), pp. 20–27. IEEE (2013)Google Scholar
  21. 21.
    Younis, M.I., Zamli, K.Z.: MIPOG-an efficient t-way minimization strategy for combinatorial testing. Int. J. Comput. Theory Eng. 3(3), 388 (2011)CrossRefGoogle Scholar
  22. 22.
    Yu, L., Lei, Y., Kacker, R.N., Kuhn, D.R.: Acts: a combinatorial test generation tool. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (ICST), pp. 370–375. IEEE (2013)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2017

Authors and Affiliations

  1. 1.SBA ResearchViennaAustria

Personalised recommendations