Skip to main content

Challenges and practices in aligning requirements with verification and validation: a case study of six companies

Abstract

Weak alignment of requirements engineering (RE) with verification and validation (VV) may lead to problems in delivering the required products in time with the right quality. For example, weak communication of requirements changes to testers may result in lack of verification of new requirements and incorrect verification of old invalid requirements, leading to software quality problems, wasted effort and delays. However, despite the serious implications of weak alignment research and practice both tend to focus on one or the other of RE or VV rather than on the alignment of the two. We have performed a multi-unit case study to gain insight into issues around aligning RE and VV by interviewing 30 practitioners from 6 software developing companies, involving 10 researchers in a flexible research process for case studies. The results describe current industry challenges and practices in aligning RE with VV, ranging from quality of the individual RE and VV activities, through tracing and tools, to change control and sharing a common understanding at strategy, goal and design level. The study identified that human aspects are central, i.e. cooperation and communication, and that requirements engineering practices are a critical basis for alignment. Further, the size of an organisation and its motivation for applying alignment practices, e.g. external enforcement of traceability, are variation factors that play a key role in achieving alignment. Our results provide a strategic roadmap for practitioners improvement work to address alignment challenges. Furthermore, the study provides a foundation for continued research to improve the alignment of RE with VV.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. 1.

    Reference to source is given by interviewee code, see Table 2.

References

  1. Barmi ZA, Ebrahimi AH, Feldt R (2011) Alignment of requirements specification and testing: A systematic mapping study. In: Proc 4th Int. Conf. On Software Testing, Verification and Validation Workshops (ICSTW) pp. 476–485

  2. Cao L, Ramesh B (2008) Agile requirements engineering practices: An empirical study. In: IEEE Software Jan/Feb 2008

  3. Cheng BH, Atlee JM (2007) Research directions in requirements engineering. In: Proc. Future of Software Engineering (FOSE) pp. 285–303

  4. Cleland-Huang J, Chang CK, Christensen M (2003) Event-based traceability for managing evolutionary change. In: IEEE Transactions on Software 29(9)

  5. Damian D, Chisan J, Vaidyanathasamy L, Pal Y (2005) Requirements engineering and downstream software development: findings from a case study. Empir Softw Eng 10:255–283

    Article  Google Scholar 

  6. Damian D, Chisan J (2006) An empirical study of the complex relationship between requirements engineering processes and other processes that lead to payoffs in productivity, quality, and risk management. IEEE Trans Softw Eng 32(7):33–453

    Article  Google Scholar 

  7. De Lucia A, Fasano F, Oliveto R, Tortora G (2007) Recovering traceability links in software artifact management systems using information retrieval methods. In: ACM Transactions on Software Engineering and Methodology, 16(4):Article 13

  8. Dias Neto AC, Arilo C, Subramanyan R, Vieira M, Travassos GH (2007) A survey on model-based testing approaches: A systematic review. In: Proc of 1st ACM Int workshop on Empirical Assessment of Software Engineering Languages and Technologies, pp. 31–36

  9. Ferguson RW, Lami G (2006) An empirical study on the relationship between defective requirements and test failures. In: Proc of 30th Annual IEEE/NASA Software Engineering Workshop SEW-30 (SEW’06)

  10. Fogelström N, Gorschek T (2007) Test-case driven versus checklist-based inspections of software requirements—an experimental evaluation, In: Proc. of 10th Workshop on Requirements Engineering (WER’07)

  11. Gotel O, Finkelstein A (1994) An analysis of the requirements traceability problem. In: Proc. First Int Conf. Requirements Eng., pp. 94–101

  12. Gorschek T, Davis AM (2007) Requirements engineering: in search of the dependent variables. Inf Softw Technol 50(1–2):67–75

    Google Scholar 

  13. Gorschek T, Wohlin C (2004) Packaging software process improvement issues—a method and a case study. Softw Pract & Experience 34:1311–1344

    Article  Google Scholar 

  14. Gorschek T, Wohlin C (2006) Requirements abstraction model. Requir Eng J 11:79–101

    Article  Google Scholar 

  15. Graham D (2002) Requirements and testing: seven missing-link myths. IEEE Softw 19:15–17

    Article  Google Scholar 

  16. Grieskamp W, Kicillof N, Stobie K, Braberman V (2011) Model-based quality assurance of protocol documentation: tools and methodology. Softw Test Verification Reliab 21(1):55–71

    Article  Google Scholar 

  17. Hasling B, Goetz H, Beetz K (2008) Model based testing of system requirements using UML use case models. In: Proc of 2008 international Conference on Software Testing, Verification, and Validation

  18. Hayes JH, Dekhtyar A, Sundaram SK, Holbrook EA, Vadlamudi S, April A (2007) REquirements TRacing On target (RETRO): improving software maintenance through traceability recovery. Innov Syst Softw Eng 3(3):193–202

    Article  Google Scholar 

  19. Höst M, Feldt R, Lüders F (2010) Support for different stakeholders in software engineering master thesis projects. IEEE Trans Educ 52(2):288–296. doi:10.1109/TE.2009.2016106

    Article  Google Scholar 

  20. IEEE (1990) IEEE standard glossary of software engineering terminology. Technical Report 610.12–1990. IEEE, New York

    Google Scholar 

  21. ISO/IEC. (2001) 9126–1:2001(E), International standard software engineering product quality part 1: Quality model. Technical report, ISO/IEC

  22. Jarke, M (1998) Requirements Traceability. Commun ACM 41(12):32–36

    Google Scholar 

  23. Jones JA, Grechanik M, van der Hoek A (2009) Enabling and enhancing collaborations between software development organizations and independent test agencies. In: Cooperative and Human Aspects of Software Engineering (CHASE’09), May 17, 2009, Vancouver, Canada

  24. Kraut RE, Streeter L (1995) Coordination in software development. Commun ACM 38(3):69–81

    Article  Google Scholar 

  25. Kukkanen J, Vakevainen K, Kauppinen M, Uusitalo E (2009) Applying a systematic approach to link requirements and testing: A case study, In: Proc of Asia-Pacific Software Engineering Conference (APSEC ’09):482–488

  26. Lormans M, van Deursen A, Gross H (2008) An industrial case study in reconstructing requirements views, In: Empirical Software Engineering, online first, September 03, 2008

  27. Lubars M, Potts C, Richter C (1993) A review of the state of the practice in requirements modelling. In: Proc of 1st IEEE Int. Symposium on Requirements Engineering, pp. 2–14

  28. Martin R, Melnik G (2008) Tests and requirements, requirements and tests a Möbius strip. IEEE Softw 25(1):54–59

    Article  Google Scholar 

  29. Melnik G, Maurer F, Chiasson M (2006) Executable acceptance tests for communicating business requirements: Customer perspective. In: Proc. of Agile Conference, Minneapolis, USA, pp. 12–46

  30. Mohagheghi P, Dehlen V (2008) Where is the proof? - A review of experiences from applying MDE in industry. In: LNCS, Model driven architecture—foundations and applications, vol 5095:432–443

  31. Nebut C, Fleurey F, Traon YL, Jézéquel J (2006) Automatic test generation: a use case driven approach. IEEE Trans Softw Eng 32(3):140–155

    Article  Google Scholar 

  32. Paci F, Massacci F, Bouquet F, Debricon S (2012) Managing evolution by orchestrating requirements and testing engineering processes. In: Proc. of IEEE 5th Int. Conf. On, pp.834–841, 17–21 April 2012

  33. Pettersson F, Ivarsson M, Gorschek (2008) A practitioner’s guide to light weight software process assessment and improvement planning. J Syst Softw 81(6):972–995

    Article  Google Scholar 

  34. Post H, Sinz C, Merz F, Gorges T, Kropf T (2009) Linking functional requirements and software verification. In: Proceedings of 17th IEEE International Requirements Engineering Conference, pp. 295–302

  35. Ramesh B, Stubbs C, Powers T, Edwards M (1997) Requirements traceability: theory and practice. Ann Softw Eng 3(1):397–415

    Article  Google Scholar 

  36. Ramesh B (1998) Factors influencing requirements traceability practice. In: Communications of the ACM CACM Homepage archive, 41(12):37–44

  37. Randell B (1969) Towards a methodology of computing system design. In: Naur P, Randell B (eds) NATO working conference on software engineering 1968. Report on a Conference Sponsored by NATO Scientific Committee, Garmisch, pp 204–208

    Google Scholar 

  38. Regnell B, Runeson P (1998) Combining scenario-based requirements with static verification and dynamic testing. In: Proc. 4th Int. Working Conf. Requirements Engineering: Foundation for Software Quality, pp.195–206

  39. Regnell B, Runeson P, Wohlin C (2000) Towards integration of use case modelling and usage-based testing. J Syst Softw 50(2):117–130

    Article  Google Scholar 

  40. Ricca F, Torchiano M, Di Penta M, Ceccato M, Tonella P (2009) Using acceptance tests as a support for clarifying requirements: a series of experiments. Inf Softw Technol 51:270–283

    Article  Google Scholar 

  41. Robson C (2002) Real world research: A resource for social scientists and practitioner researchers, 2nd edition. Blackwell Publishing

  42. Runeson P, Andersson C, Höst M (2003) Test processes in software product evolution—a qualitative survey on the state of practice. J Softw Maint Evol Res Pract 15(1):41–59

    MATH  Article  Google Scholar 

  43. Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131–164

    Article  Google Scholar 

  44. Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software engineering—guidelines and examples. Wiley

  45. Sabaliauskaite G, Loconsole A, Engström E, Unterkalmsteiner M, Regnell B, Runeson P, Gorschek T, Feldt R (2010) Challenges in aligning requirements engineering and verification in a large-scale industrial context. In: Proceedings of REFSQ 2010

  46. Sikora E, Tenbergen B, Pohl K (2012) Industry needs and research directions in requirements engineering for embedded systems. Requir Eng 17(1):57–78

    Article  Google Scholar 

  47. Uusitalo EJ, Komassi M, Kauppinen M, Davis AM (2008) Linking requirements and testing in practice. In: Proceedings of 16th IEEE International Requirements Engineering Conference, pp. 295–302

  48. Watkins R, Neal M (1994) Why and how of requirements tracing. IEEE Softw 11(4):104–106

    Article  Google Scholar 

  49. Yue T, Briand L, Labiche Y (2011) A systematic review of transformation approaches between user requirements and analysis models. Requir Eng 16(2):75–99

    Article  Google Scholar 

Download references

Acknowledgment

We want to thank Börje Haugset for acting as interviewer in three of the interviews. We would also like to thank all the participating companies and anonymous interviewees for their contribution to this project. The research was funded by EASE Industrial Excellence Center for Embedded Applications Software Engineering (http://ease.cs.lth.se).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Elizabeth Bjarnason.

Additional information

Communicated by: Nachiappan Nagappan

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Bjarnason, E., Runeson, P., Borg, M. et al. Challenges and practices in aligning requirements with verification and validation: a case study of six companies. Empir Software Eng 19, 1809–1855 (2014). https://doi.org/10.1007/s10664-013-9263-y

Download citation

Keywords

  • Requirements engineering
  • Verification
  • Validation
  • Testing
  • Alignment
  • Case study