Software & Systems Modeling

, Volume 16, Issue 4, pp 1049–1082 | Cite as

FLAME: a formal framework for the automated analysis of software product lines validated by automated specification testing

  • Amador Durán
  • David Benavides
  • Sergio Segura
  • Pablo Trinidad
  • Antonio Ruiz-Cortés
Regular Paper


In a literature review on the last 20 years of automated analysis of feature models, the formalization of analysis operations was identified as the most relevant challenge in the field. This formalization could provide very valuable assets for tool developers such as a precise definition of the analysis operations and, what is more, a reference implementation, i.e., a trustworthy, not necessarily efficient implementation to compare different tools outputs. In this article, we present the FLAME framework as the result of facing this challenge. FLAME is a formal framework that can be used to formally specify not only feature models, but other variability modeling languages (VML s) as well. This reusability is achieved by its two-layered architecture. The abstract foundation layer is the bottom layer in which all VML-independent analysis operations and concepts are specified. On top of the foundation layer, a family of characteristic model layers—one for each VML to be formally specified—can be developed by redefining some abstract types and relations. The verification and validation of FLAME has followed a process in which formal verification has been performed traditionally by manual theorem proving, but validation has been performed by integrating our experience on metamorphic testing of variability analysis tools, something that has shown to be much more effective than manually designed test cases. To follow this automated, test-based validation approach, the specification of FLAME, written in Z, was translated into Prolog and 20,000 random tests were automatically generated and executed. Tests results helped to discover some inconsistencies not only in the formal specification, but also in the previous informal definitions of the analysis operations and in current analysis tools. After this process, the Prolog implementation of FLAME is being used as a reference implementation for some tool developers, some analysis operations have been formally specified for the first time with more generic semantics, and more VML s are being formally specified using FLAME.


Formal specification Specification testing Software product lines Feature models 



The authors would like to thank José A. Galindo for his help implementing the BeTTy module for generating the tests in Prolog. We would also like to thank Miguel Toro, Pere Botella, Isidro Ramos, Frank van der Linden, Ernesto Pimentel, Vicente Pelechano, Daniel Le Berre, Sven Apel, Patrick Heymans, Paolo Borba, Maurice ter Beek, Rob Hierons, Michael Hinchey and the anonymous reviewers, for their helpful comments on earlier versions of this article. Finally, we thank Marwa Benabdelali for using a very early version of the reference implementation of FLAME at the Institut Supérieur de Gestion de Tunis and provide early feedback.


  1. 1.
    Acher, M., Collet, P., Lahire, P., France, R.: Familiar: a domain-specific language for large scale management of feature models. Sci. Comput. Program. 78(6), 657–681 (2013)CrossRefGoogle Scholar
  2. 2.
    Achour, I., Labed, L., Ben Ghezala, H.: Towards an extended tool for analysis of extended feature models. In: International Symposium on Networks, Computers and Communications, pp. 1–5 (June 2014)Google Scholar
  3. 3.
    AHEAD Tool Suite. Accessed March 2015
  4. 4.
    Bachmeyer, R., Delugach, H.: A conceptual graph approach to feature modeling. In: International Conference on Conceptual Structures, pp. 179–191 (2007)Google Scholar
  5. 5.
    Batory, D.: Feature models, grammars, and propositional formulas. In: Software Product Lines Conference, pp. 7–20 (2005)Google Scholar
  6. 6.
    Beizer, B.: Software Testing Techniques, 2nd edn. Van Nostrand Reinhold Co., Hoboken (1990)zbMATHGoogle Scholar
  7. 7.
    Benavides, D.: On the Automated Analysis of Software Product Lines using Feature Models. PhD thesis, Univ. of Seville (2007)Google Scholar
  8. 8.
    Benavides, D., Ruiz-Cortés, A., Trinidad, P.: Automated reasoning on feature models. In: International Conference on Advanced Information, Systems Engineering, pp. 491–503 (2005)Google Scholar
  9. 9.
    Benavides, D., Segura, S., Ruiz-Cortés, A.: Automated analysis of feature models 20 years later: a literature review. Inf. Syst. 35(6), 615–636 (2010)CrossRefGoogle Scholar
  10. 10.
    Benavides, D., Segura, S., Trinidad, P., Ruiz-Cortés, A.: FAMA: tooling a framework for the automated analysis of feature models. In: International Workshop on Variability Modeling of Software-Intensive Systems, pp. 129–134 (2007)Google Scholar
  11. 11.
    Berger, T., She, S., Lotufo, R., Wasowski, A., Czarnecki, K.: Variability modeling in the real: a perspective from the operating systems domain. In: International Conference on Automated Software Engineering, pp. 73–82 (2010)Google Scholar
  12. 12.
    BigLever. Biglever software gears. Accessed March 2015
  13. 13.
    Bowen, J., Hinchey, M.: Ten commandments of formal methods\(\ldots \) ten years on. In: Hinchey, M., Coyle, L. (eds.) Conquering Complexity, pp. 237–251. Springer, London (2012)CrossRefGoogle Scholar
  14. 14.
    Chan, W., Cheung, S., Leung, K.: A metamorphic testing approach for online testing of service-oriented software applications. Int. J. Web Serv. Res. 4(2), 61–81 (2007)CrossRefGoogle Scholar
  15. 15.
    Chen, T.Y., Cheung, S.C., Yiu, S.M.: Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01, Univ. of Science and Tech., Hong Kong (1998)Google Scholar
  16. 16.
    Chen, T.Y., Feng, J., Tse, T.H.: Metamorphic testing of programs on partial differential equations: a case study. In: International Computer Software and Applications Conference, pp. 327–333 (2002)Google Scholar
  17. 17.
    Chen, T.Y., Huang, D.H., Tse, T.H., Zhou, Z.Q.: Case studies on the selection of useful relations in metamorphic testing, pp. 569–583. In: Ibero-American Symposium on Software Engineering and Knowledge, Engineering(2004)Google Scholar
  18. 18.
    Classen, A., Boucher, Q., Heymans, P.: A text-based approach to feature modelling: syntax and semantics of TVL. Sci. Comput. Program. 76(12), 1130–1143 (2011)CrossRefGoogle Scholar
  19. 19.
    Clements, P., Northrop, L.: Software Product Lines: Practices and Patterns. Addison-Wesley, Reading (2001)Google Scholar
  20. 20.
    Clocksin, W.F., Mellish, C.S.: Programming in Prolog: Using the ISO Standard, 5th edn. Springer, Berlin (2003)CrossRefzbMATHGoogle Scholar
  21. 21.
    Czarnecki, K., Grünbacher, P., Rabiser, R., Schmid, K., Wasowski, A.: Cool features and tough decisions: a comparison of variability modeling approaches. In: International Workshop on Variability Modeling of Software-Intensive Systems, pp. 173–182 (2012)Google Scholar
  22. 22.
    Czarnecki, K., Helsen, S., Eisenecker, U.: Formalizing cardinality-based feature models and their specialization. Softw. Process Improv. Pract. 10(1), 7–29 (2005)CrossRefGoogle Scholar
  23. 23.
    El-Sharkawy, S., Dederichs, S., Schmid, K.: From feature models to decision models and back again. In: International Software Product Line Conference, pp. 126–135 (2012)Google Scholar
  24. 24.
    Fagereng Johansen, M., Haugen, O., Fleurey, F.: An algorithm for generating t-wise covering arrays from large feature models. In: International Software Product Line Conference (2012)Google Scholar
  25. 25.
    Fan, S., Zhang, N.: Feature model based on description logics. In: International Conference on Knowledge-Based Intelligent Information and Engineering Systems, pp. 1144–1151 (2006)Google Scholar
  26. 26.
    Fernandez-Amorós, D., Heradio, R., Cerrada, J.: Inferring information from feature diagrams to product line economic models. In: Software Product Line Conference, pp. 41–50 (2009)Google Scholar
  27. 27.
    Fernández-Amorós, D., Heradio, R., Cerrada, J.A., Cerrada, C.: A scalable approach to exact model and commonality counting for extended feature models. IEEE Trans. Softw. Eng. 40(9), 895–910 (2014)CrossRefGoogle Scholar
  28. 28.
    Feature Modeling Plug-in. Accessed March 2015
  29. 29.
    Galindo, J., Alférez, M., Acher, M., Baudry, B., Benavides, D.: A variability-based testing approach for synthesizing video sequences. In: International Symposium on Software Testing and Analysis, pp. 293–303 (2014)Google Scholar
  30. 30.
    Galindo, J.A., Turner, H., Benavides, D., White, J.: Testing variability intensive systems using automated analysis: an application in android. Softw. Qual. J. (2014). doi: 10.1007/s11219-014-9258-y
  31. 31.
    García-Galán, J., Trinidad, P., Rana, O. F., Ruiz-Cortés, A.: Automated configuration support for infrastructure migration to the cloud. Future Gener. Comput. Syst. (2015). doi: 10.1016/j.future.2015.03.006
  32. 32.
    Gheyi, R., Massoni, T., Borba, P.: A theory for feature models in alloy. In: First Alloy, Workshop, pp. 71–80 (2006)Google Scholar
  33. 33.
    Gheyi, R., Massoni, T., Borba, P.: Algebraic laws for feature models. J. Univ. Comput. Sci 14(21), 3573–3591 (2008)zbMATHGoogle Scholar
  34. 34.
    Henard, C., Papadakis, M., Perrouin, G., Klein, J., Le Traon, Y.: Multi-objective test generation for software product lines. In: International Software Product Line Conference, pp. 62–71 (2013)Google Scholar
  35. 35.
    Hewitt, M., O’Halloran, C., Sennett, C.: Experiences with PiZA, an animator for Z. In: Z user meeting, pp. 35–51 (1997)Google Scholar
  36. 36.
    ISA Research Group. FaMa Tool Suite. Accessed March 2015
  37. 37.
    ISO/IEC: Information technology—Z formal specification notation—Syntax, type system and semantics. International Standard ISO/IEC 13568:2002 (2002)Google Scholar
  38. 38.
    Jackson, D.: Software Abstractions: Logic, Language, and Analysis, revised edition. MIT Press, Cambridge (2012)Google Scholar
  39. 39.
    Kang, K., Cohen, S., Hess, J., Novak, W., Peterson, S.: Feature-Oriented Domain Analysis (FODA) Feasibility Study. Technical Report CMU/SEI-90-TR-21, Soft. Engineering Institute (1990)Google Scholar
  40. 40.
    King, P.: Printing Z and Object-Z LaTeX documents. University of Queensland (1990)Google Scholar
  41. 41.
    Le Berre, D., Rapicault, P.: Dependency management for the eclipse ecosystem: eclipse p2, metadata and resolution, pp. 21–30. In: International Workshop on Open Component, Ecosystems (2009)Google Scholar
  42. 42.
    Lopez-Herrejon, R., Linsbauer, L., Galindo, J., Parejo, J.A., Benavides, D., Segura, S., Egyed, A.: An assessment of search-based techniques for reverse engineering feature models. J. Syst. Softw. 103, 353–369 (2015)CrossRefGoogle Scholar
  43. 43.
    Mendonca, M., Branco, M., Cowan, D.: SPLOT: software product lines online tools. In: Companion to the International Conference on Object-Oriented Programming, Systems, Languages, and Applications, pp. 761–762 (2009)Google Scholar
  44. 44.
    Mendonca, M., Wasowski, A., Czarnecki, K., Cowan, D.: Efficient compilation techniques for large scale feature models. In: Generative Programming and Component Engineering Conference, pp. 13–22 (2008)Google Scholar
  45. 45.
    Müller, C., Resinas, M., Ruiz-Cortés, A.: Automated analysis of conflicts in WS-agreement. IEEE Trans. Serv. Comput. 7(4), 530–544 (2014)CrossRefGoogle Scholar
  46. 46.
    Myers, G.J., Sandler, C.: The Art of Software Testing. Wiley, New York (2004)Google Scholar
  47. 47.
    Olaechea, R., Stewart, S., Czarnecki, K., Rayside, D.: Modeling and multi-objective optimization of quality attributes in variability-rich software. In: International Workshop on Non-functional System Properties in Domain Specific Modeling Languages (2012)Google Scholar
  48. 48.
    Pérez Lamancha, B. Polo Usaola, M.: Testing product generation in software product lines using pairwise for features coverage. In: International Conference on Testing Software and Systems, pp. 111–125 (2010)Google Scholar
  49. 49.
    Perrouin, G., Oster, S., Sen, S., Klein, J., Baudry, B., Le Traon, Y.: Pairwise testing for software product lines: comparison of two approaches. Softw. Qual. J. 20(3–4), 605–643 (2011)Google Scholar
  50. 50.
    Perrouin, G., Sen, S., Klein, J., Baudry, B., Le Traon, Y.: Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines. In: International Conference on Software Testing, Verification and Validation, pp. 459–468 (2010)Google Scholar
  51. 51.
    Pohl, K., Böckle, G., van der Linden, F.: Software Product Line Engineering: Foundations, Principles, and Techniques. Springer, Berlin (2005)CrossRefzbMATHGoogle Scholar
  52. 52.
    pure::variants. Accessed March 2015
  53. 53.
    Roos-Frantz, F.: Automated Analysis of Software Product Lines with Orthogonal Variability Models. PhD thesis, Univ. of Seville (2012)Google Scholar
  54. 54.
    Roos-Frantz, F., Benavides, D., Ruiz-Cortés, A., Heuer, A., Lauenroth, K.: Quality-aware analysis in product line engineering with the orthogonal variability model. Softw. Qual. J. 20(3–4), 519–565 (2012)CrossRefGoogle Scholar
  55. 55.
    Saaltink, M.: The Z/EVES system. In: Z User Meeting, pp. 72–85 (1997)Google Scholar
  56. 56.
    Sayyad, A.S., Menzies, T., Ammar, H.: On the value of user preferences in search-based software engineering: a case study in software product lines. In: International Conference on Software Engineering, pp. 492–501 (2013)Google Scholar
  57. 57.
    Schmid, K., John, I.: A customizable approach to full lifecycle variability management. Sci. Comput. Program. 53(3), 259–284 (2004)MathSciNetCrossRefGoogle Scholar
  58. 58.
    Schmid, K., Rabiser, R., Grünbacher, P.: A comparison of decision modeling approaches in product lines. In: Working on Variability Modeling of Software-Intensive Systems, pp. 119–126 (2011)Google Scholar
  59. 59.
    Schobbens, P., Trigaux, J.C., Heymans, P., Bontemps, Y.: Generic semantics of feature diagrams. Comput. Netw. 51(2), 456–479 (2007)CrossRefzbMATHGoogle Scholar
  60. 60.
    Segura, S.: Automated analysis of feature models using atomic sets. In: Workshop on Analyses of Software Product Lines, pp. 201–207 (2008)Google Scholar
  61. 61.
    Segura, S., Benavides, D., Ruiz-Cortés, A.: Functional testing of feature model analysis tools: a test suite. IET Softw. 5(1), 70–82 (2011)CrossRefGoogle Scholar
  62. 62.
    Segura, S., Durán, A., Sánchez, A.B., Le Berre, D., Lonca, E., Ruiz-Cortés, A.: Automated metamorphic testing of variability analysis tools. Softw. Test. Verif. Reliab. 25(2), 138–163 (2015)CrossRefGoogle Scholar
  63. 63.
    Segura, S., Galindo, J.A., Benavides, D., Parejo, J.A., Ruiz-Cortés, A.: Betty: benchmarking and testing on the automated analysis of feature models. In Workshop on Variability Modelling of Software-intensive Systems, pp. 63–71 (2012)Google Scholar
  64. 64.
    Segura, S., Hierons, R.M., Benavides, D., Ruiz-Cortés, A.: Automated metamorphic testing on the analyses of feature models. Inf. Softw. Technol. 53(3), 245–258 (2011)CrossRefGoogle Scholar
  65. 65.
    Spivey, J.M.: The Z Notation: A Reference Manual. Prentice-Hall, Englewood Cliffs (1992)zbMATHGoogle Scholar
  66. 66.
    Sun, J., Zhang, H., Li, Y.F., Wang, H.: Formal semantics and verification for feature modeling. In: International Conference on Engineering of Complex Computer Systems, pp. 303–312 (2005)Google Scholar
  67. 67.
    Thüm, T., Batory, D., Kästner, C.: Reasoning about edits to feature models. In International Conference on Software Engineering, pp. 254–264 (2009)Google Scholar
  68. 68.
    Thüm, T., Kastner, C., Erdweg, S., Siegmund, N.: Abstract features in feature modeling. In Software Product Lines Conference, pp. 191–200 (2011)Google Scholar
  69. 69.
    Treinen, R., Zacchirol, S.: Common upgradeability description format (cudf) 2.0. Technical Report 003, The Mancoosi project (FP7) (2009)Google Scholar
  70. 70.
    Trinidad, P., Benavides, D., Durán, A., Ruiz-Cortés, A., Toro, M.: Automated error analysis for the agilization of feature modeling. J. Syst. Softw. 81(6), 883–896 (2008)CrossRefGoogle Scholar
  71. 71.
    Trinidad, P., Benavides, D., Ruiz-Cortés, A., Segura, S., Jimenez, A.: Fama framework. In: International Software Product Line Conference Tool Demonstrations, p. 359 (2008)Google Scholar
  72. 72.
    von der Massen, T., Litcher, H.: Determining the variation degree of feature models. In: Software Product Lines Conference, pp. 82–88 (2005)Google Scholar
  73. 73.
    West, M.M., Eaglestone, B.M.: Software development: two approaches to animation of Z specifications using prolog. Softw. Eng. J. 7(4), 264–276 (1992)CrossRefGoogle Scholar
  74. 74.
    Weyuker, E.J.: On testing non-testable programs. Comput. J. 25(4), 465–470 (1982)CrossRefGoogle Scholar
  75. 75.
    White, J., Benavides, D., Schmidt, D.C., Trinidad, P., Dougherty, B., Ruiz-Cortés, A.: Automated diagnosis of feature model configurations. J. Syst. Softw. 83(7), 1094–1107 (2010)CrossRefGoogle Scholar
  76. 76.
    White, J., Galindo, J., Saxena, T., Doughtery, B., Benavides, D., Schmidt, D.: Evolving feature model configurations in software product lines. J. Syst. Softw. 87, 119–136 (2014)CrossRefGoogle Scholar
  77. 77.
    Wielemaker, J.: Prolog unit tests. Accessed March 2015
  78. 78.
    Zhang, W., Yan, H., Zhao, H., Jin, Z.: A BDD-based approach to verifying clone-enabled feature models’ constraints and customization. In: International Conference on Software Reuse, pp. 186–199 (2008)Google Scholar
  79. 79.
    Zhang, W., Zhao, H., Mei, H.: A propositional logic-based method for verification of feature models. In: International Conference on Formal Methods and Software Engineering, pp. 115–130 (2004)Google Scholar
  80. 80.
    Zhou, Z.Q., Huang, DH., Tse, TH., Yang, Z., Huang, H., Chen, TY.: Metamorphic testing and its applications. In: International Symposium on Future Software Technology, pp. 346–351 (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Amador Durán
    • 1
  • David Benavides
    • 1
  • Sergio Segura
    • 1
  • Pablo Trinidad
    • 1
  • Antonio Ruiz-Cortés
    • 1
  1. 1.Department of Computer Languages and SystemsUniversidad de Sevilla, E.T.S.I. InformáticaSevilleSpain

Personalised recommendations