Model-based testing is a promising approach for increasing the efficiency of the testing process and for improving software quality. It has been employed in the industry for more than a decade. Nevertheless, there are still challenges regarding its application in different domains. Some of these challenges are general, while some others are domain-specific. In this paper, we explain our experiences in enhancing model-based testing for its adoption in the consumer electronics domain, in particular for Digital TV systems. We applied the so-called industry-as-laboratory approach to define/refine research problems and evaluate our research results. We summarize these results and provide an evaluation of relevant research problems for our context. We observed that the industry-as-laboratory approach is highly effective for industry-academia collaboration and technology transfer in the scope of model-based software testing.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Vestel currently employs MaTeLo (http://www.all4tec.net) for developing test models and generating test cases from these models.
In each of the case studies, we used the same version of the system to compare the results before and after model refinement.
Aksit, M., Tekinerdogan, B., Sozer, H., Safi, H., Ayas, M. (2015). The DESARC method: An effective approach for university-industry cooperation. In Proceedings of the international conference on advances in computing, control and networking (pp. 51–53).
Apfelbaum, L., Doyle, J. (1997). Model-based testing. In Software quality week conference (pp. 296–300).
Arias, T. (2011). Execution architecture views for evolving software-intensive systems. Ph.D. Thesis, University of Groningen.
Barr, E., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering, 41(5), 507–525.
Belli, F. (2001). Finite state testing and analysis of graphical user interfaces. In Proceedings of 12th international symposium on software reliability engineering, ISSRE2001 (pp. 34–43).
Belli, F., Beyazit, M., Guler, N. (2011). Event-based GUI testing and reliability assessment techniques—An experimental insight and preliminary results. In Proceedings of the 4th IEEE international conference on software testing, verification and validation workshops (pp. 212–221).
Belli, F., Endo, A., Linschulte, M., & Simao, A. (2014). A holistic approach to model-based testing of web service compositions. Software: Practice and Experience, 44(2), 201–234.
Boberg, J. (2008). Early fault detection with model-based testing. In Proceedings of the 7th ACM SIGPLAN workshop on ERLANG (pp. 9–20).
Boogerd, C., Moonen, L. (2006). Prioritizing software inspection results using static profiling. In Proceedings of the 6th international workshop on source code analysis and manipulation (pp. 149–160).
Brinksma E, Hooman J (2008) Dependability for high-tech systems: an industry-as-laboratory approach. In Proceedings of the conference on design, automation and test in Europe (pp. 1226–1231).
Dalal, S., Jain, A., Karunanithi, N., Leaton, J., Lott, C., Patton, G., Horowitz, B. (1999). Model-based testing in practice. In Proceedings of the international conference on software engineering (pp. 285–294).
Damm, L., Lundberg, L., & Wohlin, C. (2008). A model for software rework reduction through a combination of anomaly metrics. Journal of Systems and Software, 81(11), 1968–1982.
de Visser, I. (2008). Analyzing user perceived failure severity in consumer electronics products : Incorporating the user perspective into the development process. Ph.D. thesis, Eindhoven University of Technology.
Dustin, E., Rashka, J., & Paul, J. (1999). Automated software testing: Introduction, management, and performance. Boston, MA: Addison-Wesley Longman Publishing Co. Inc.
Engstrm, E., & Runeson, P. (2011). Software product line testing a systematic mapping study. Information and Software Technology, 53(1), 2–13.
Eriksson, M. (2007). Engineering families of softwareintensive systems using features, goals and scenarios. Ph.D. Thesis, Umea University.
Gebizli, C., Metin, D., Sozer, H. (2015). Combining model-based and risk-based testing for effective test case generation. In Proceedings of the 8th IEEE conference on software testing, verification and validation workshops (pp. 1–4).
Gebizli, C., Sozer, H. (2014). Improving models for model-based testing based on exploratory testing. In Proceedings of the 6th IEEE workshop on software test automation (pp. 656–661).
Gebizli, C, Sozer, H. (2015). Automated refinement of models for model-based testing using exploratory testing. Software Quality Journal (under review).
Grottke, M., Matias, R., Trivedi, K.S. (2008). The fundamentals of software aging. In Proceedings of the 19th international symposium on software reliability engineering workshops (pp. 1–6).
Guen, H.L., Marie, R., Thelin, T. (2004). Reliability estimation for statistical usage testing using markov chains. In Proceedings of the 15th international symposium on software reliability engineering (pp. 54–65).
Joye, C. (2014). Matelo test case generation algorithms: Explanation on available algorithms for test case generation. http://www.all4tec.net/MaTeLo-How-To/understanding-of-test-cases-generation-algorithms.html.
Keranen, J., & Raty, T. (2011). Model-based testing of embedded systems in hardware in the loop environment. IET Software, 6(4), 364–376.
Klockwork, (2016). C and C++ checker reference, [available online]. https://developer.klocwork.com/documentation/en/insight/10-1/c-and-c-checker-reference.
Lackner, H., Thomas, M., Wartenberg, F., Weißleder, S. (2014). Model-based test design of product lines: Raising test design to the product line level. In Proceedings of the 7th IEEE international conference on software testing, verification and validation (pp. 51–60).
Lindvall, M., Ganesan, D., Ardal, R., Wiegand, R. (2015). Metamorphic model-based testing applied on NASA DAT—An experience report. In Proceedings of the 37th international conference on software engineering (pp. 129–138).
Malik, Q.A., Jskelinen, A., Virtanen, H., Katara, M., Abbors, F., Truscan, D., Lilius, J. (2010). Model-based testing using system vs. test models—What is the difference? In Proceedings of the 17th IEEE international conference and workshops on engineering of computer based systems (pp. 291–299).
Muller, G. (2012). Industry-as-laboratory applied in practice: The boderc project. http://www.gaudisite.nl/IndustryAsLaboratoryAppliedPaper.pdf. Accessed 2015.
Neto, A., Subramanyan, R., Vieira, M., Travassos, G. (2007). A survey on model-based testing approaches: A systematic review. In Proceedings of the 1st ACM international workshop on Empirical assessment of software engineering languages and technologies (pp. 31–36).
Nguyen, B., & Memon, A. (2014). An observe-model-exercise* paradigm to test event-driven systems with undetermined input spaces. IEEE Transactions on Software Engineering, 40(3), 216–234.
Nguyen, B., Robbins, B., Banerjee, I., & Memon, A. (2014). GUITAR: an innovative tool for automated testing of gui-driven software. Automated Software Engineering, 21(1), 65–105.
Potts, C. (1993). Software-engineering research revisited. IEEE Software, 10(5), 19–28.
Samih, H., Bogusch, R. (2014). MPLM—Matelo product line manager: [Relating variability modelling and model-based testing]. In Proceedings of the 18th international software product line conference: Companion volume for workshops, demonstrations and tools, (Vol. 2, pp. 138–142).
Schieferdecker, I. (2012). Model-based testing. IEEE Software, 29(1), 14–18.
Sozer, H., Hofmann, C., Tekinerdogan, B., Aksit, M. (2011). Runtime verification of component-based embedded software. In Proceedings of the 26th international symposium on computer and information sciences (pp. 471–477).
Sozer, H., Tekinerdogan, B., & Aksit, M. (2009). FLORA: A framework for decomposing software architecture to introduce local recovery. Software: Practice and Experience, 39(10), 869–889.
Sozer, H., Tekinerdogan, B., & Aksit, M. (2013). Optimizing decomposition of software architecture for local recovery. Software Quality Journal, 21(2), 203–240.
Tekinerdogan, B., Sozer, H., & Aksit, M. (2008). Software architecture reliability analysis using failure scenarios. Journal of Systems and Software, 81(4), 558–575.
Utting, M., & Legeard, B. (2007). Practical model-based testing: A tools approach. San Francisco, CA: Morgan Kaufmann Publishers Inc.
Utting, M., Pretschner, A., & Legeard, B. (2012). A taxonomy of model-based testing approaches. Software Testing Verification and Reliability, 22(5), 297–312.
van de Laar, P. (2010). Observations from the industry-as-laboratory research project darwin. In Proceedings of the 8th conference on systems engineering research (pp. 658–667).
Weißleder, S. (2010). Test models and coverage criteria for automatic model-based test generation with uml state machines. Ph.D. Thesis, Humboldt-Universitt zu Berlin.
Weißleder, S., Lackner, H. (2010). System models vs. test models -distinguishing the undistinguishable?. In Informatik 2010: Service Science - Neue Perspektiven für die Informatik, Beiträge der 40. Jahrestagung der Gesellschaft für Informatik e.V. (GI), Band 2, 27.09. - 1.10.2010, Leipzig (pp. 321–326).
Whittaker, J. (2009). Exploratory software testing: Tips, tricks, tours, and techniques to guide test design (1st ed.). Boston: Addison-Wesley Professional.
Zoeteweij, P., Abreu, R., Golsteijn, R., van Gemund, A. (2007). Diagnosis of embedded software using program spectra. In Proceedings of the 14th annual IEEE international conference and workshop on engineering of computer based systems (pp. 213–220).
This work is supported by the joint Grant of Vestel Electronics and the Turkish Ministry of Science, Industry and Technology (909.STZ.2015). The contents of this article reflect the ideas and positions of the authors and do not necessarily reflect the ideas or positions of Vestel Electronics and the Turkish Ministry of Science, Industry and Technology. We would like to thank software developers and software test engineers at Vestel Electronics for sharing their code base with us and supporting our case studies.
About this article
Cite this article
Sözer, H., Gebizli, C.Ş. Model-based testing of digital TVs: an industry-as-laboratory approach. Software Qual J 25, 1185–1202 (2017). https://doi.org/10.1007/s11219-016-9321-y
- Software testing
- Model-based testing
- Case studies
- Industrial projects
- Industry-as-laboratory approach