Abstract
Software systems generally have a large number of configurable options interacting with each other. Such systems are more likely to be prone to errors, crashes, and faulty executions that are usually caused by option interactions. To avoid such errors, testing all possible configurations during the development phase is usually not feasible, since the number of all possible configurations is exponential in the order of number of options. A t-way covering array (CA) is a 2-dimensional combinatorial object that helps to efficiently cover all t-length option interactions of the system under test. Generating a CA with a small number of configurations is important to shorten the testing phase. However, the testing cost (e.g. the testing time) may differ from one configuration to another. Currently, most sequential tools can generate optimum CAs in terms of number of configurations, but they are not cost-aware, i.e., they cannot handle the varying costs of configurations. In this work, we implement a parallel, cost-aware CA-generation tool based on a sequential tool, Jenny, to generate lower-cost CAs faster. Experimental results show that our cost-aware CA construction approach can generate \(32\%\) and \(21\%\) lower cost CAs on average for t = 2 and t = 3, respectively, compared to state-of-the-art CA-generation tools. Moreover, the cost-awareness comes for free, i.e., we speed up our algorithm by leveraging parallel computation. The cost models and cost reduction techniques we propose could also be adapted for other existing CA generation tools.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Kuhn, D.R., Wallace, D.R., Gallo Jr., A.M.: Software fault interactions and implications for software testing. IEEE Trans. Software Eng. 30(6), 418–421 (2004)
Yilmaz, C., Fouche, S., Cohen, M.B., Porter, A., Demiroz, G., Koc, U.: Moving forward with combinatorial interaction testing. Computer 2, 37–45 (2014)
Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. (CSUR) 43(2), 11 (2011)
Seroussi, G., Bshouty, N.H.: Vector sets for exhaustive testing of logic circuits. IEEE Trans. Inf. Theory 34(3), 513–522 (1988)
Lei, Y., Tai, K.-C.: In-parameter-order: a test generation strategy for pairwise testing. In: High-Assurance Systems Engineering Symposium, Proceedings Third IEEE International, pp. 254–261. IEEE (1998)
Cohen, M.B., Colbourn, C.J., Ling, A.C.: Augmenting simulated annealing to build interaction test suites. In: 14th International Symposium on Software Reliability Engineering, ISSRE 2003, pp. 394–405. IEEE (2003)
Czerwonka, J.: Pairwise testing in the real world: practical extensions to test-case scenarios. In: Proceedings of 24th Pacific Northwest Software Quality Conference, pp. 419–430. Citeseer (2006)
Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: Ipog/ipog-d: efficient test generation for multi-way combinatorial testing. Softw. Test. Verification Reliab. 18(3), 125–148 (2008)
Garvin, B.J., Cohen, M.B., Dwyer, M.B.: Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Softw. Eng. 16(1), 61–102 (2011)
Khalsa, S.K., Labiche, Y.: An orchestrated survey of available algorithms and tools for combinatorial testing. In: 2014 IEEE 25th International Symposium on Software Reliability Engineering (ISSRE), pp. 323–334. IEEE (2014)
Demiroz, G., Yilmaz, C.: Cost-aware combinatorial interaction testing. In: In the Proceedings of VALID 2012, The Fourth International Conference on Advances in System Testing and Validation Lifecycle, pp. 9–16, November 2012
Demiroz, G., Yilmaz, C.: Using simulated annealing for computing cost-aware covering arrays. Appl. Soft Comput. 49, 1129–1144 (2016)
Demiroz, G.: Cost-aware combinatorial interaction testing (doctoral symposium). In: Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA 2015), pp. 440–443. ACM, July 2015
Bryce, R.C., Colbourn, C.J.: Prioritized interaction testing for pair-wise coverage with seeding and constraints. Inf. Softw. Technol. 48(10), 960–970 (2006). Advances in Model-based Testing
Jenkins, B.: jenny: A pairwise testing tool (2005). http://www.burtleburtle.net/bob/index.html
Pairwise testing available tools. http://www.pairwise.org/tools.asp
Johansen, M.F., Haugen, O., Fleurey, F.: An algorithm for generating t-wise covering arrays from large feature models. In: Proceedings of the 16th International Software Product Line Conference, SPLC 2012, vol. 1, pp. 46–55. ACM (2012)
Demiroz, G., Yilmaz, C.: Towards automatic cost model discovery for combinatorial interaction testing. In: Proceedings of the 2016 International Workshop on Combinatorial Testing (IWCT 2016). IEEE, April 2016
Yu, L., Lei, Y., Kacker, R.N., Kuhn, D.R.: Acts: a combinatorial test generation tool. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (ICST), pp. 370–375. IEEE (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Taş, M.K., Mercan, H., Demiröz, G., Kaya, K., Yilmaz, C. (2018). Generating Cost-Aware Covering Arrays for Free. In: Itsykson, V., Scedrov, A., Zakharov, V. (eds) Tools and Methods of Program Analysis. TMPA 2017. Communications in Computer and Information Science, vol 779. Springer, Cham. https://doi.org/10.1007/978-3-319-71734-0_15
Download citation
DOI: https://doi.org/10.1007/978-3-319-71734-0_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-71733-3
Online ISBN: 978-3-319-71734-0
eBook Packages: Computer ScienceComputer Science (R0)