Empirical Software Engineering

, Volume 19, Issue 6, pp 1809–1855

Challenges and practices in aligning requirements with verification and validation: a case study of six companies

  • Elizabeth Bjarnason
  • Per Runeson
  • Markus Borg
  • Michael Unterkalmsteiner
  • Emelie Engström
  • Björn Regnell
  • Giedre Sabaliauskaite
  • Annabella Loconsole
  • Tony Gorschek
  • Robert Feldt
Article

Abstract

Weak alignment of requirements engineering (RE) with verification and validation (VV) may lead to problems in delivering the required products in time with the right quality. For example, weak communication of requirements changes to testers may result in lack of verification of new requirements and incorrect verification of old invalid requirements, leading to software quality problems, wasted effort and delays. However, despite the serious implications of weak alignment research and practice both tend to focus on one or the other of RE or VV rather than on the alignment of the two. We have performed a multi-unit case study to gain insight into issues around aligning RE and VV by interviewing 30 practitioners from 6 software developing companies, involving 10 researchers in a flexible research process for case studies. The results describe current industry challenges and practices in aligning RE with VV, ranging from quality of the individual RE and VV activities, through tracing and tools, to change control and sharing a common understanding at strategy, goal and design level. The study identified that human aspects are central, i.e. cooperation and communication, and that requirements engineering practices are a critical basis for alignment. Further, the size of an organisation and its motivation for applying alignment practices, e.g. external enforcement of traceability, are variation factors that play a key role in achieving alignment. Our results provide a strategic roadmap for practitioners improvement work to address alignment challenges. Furthermore, the study provides a foundation for continued research to improve the alignment of RE with VV.

Keywords

Requirements engineering Verification Validation Testing Alignment Case study 

References

  1. Barmi ZA, Ebrahimi AH, Feldt R (2011) Alignment of requirements specification and testing: A systematic mapping study. In: Proc 4th Int. Conf. On Software Testing, Verification and Validation Workshops (ICSTW) pp. 476–485Google Scholar
  2. Cao L, Ramesh B (2008) Agile requirements engineering practices: An empirical study. In: IEEE Software Jan/Feb 2008Google Scholar
  3. Cheng BH, Atlee JM (2007) Research directions in requirements engineering. In: Proc. Future of Software Engineering (FOSE) pp. 285–303Google Scholar
  4. Cleland-Huang J, Chang CK, Christensen M (2003) Event-based traceability for managing evolutionary change. In: IEEE Transactions on Software 29(9)Google Scholar
  5. Damian D, Chisan J, Vaidyanathasamy L, Pal Y (2005) Requirements engineering and downstream software development: findings from a case study. Empir Softw Eng 10:255–283CrossRefGoogle Scholar
  6. Damian D, Chisan J (2006) An empirical study of the complex relationship between requirements engineering processes and other processes that lead to payoffs in productivity, quality, and risk management. IEEE Trans Softw Eng 32(7):33–453CrossRefGoogle Scholar
  7. De Lucia A, Fasano F, Oliveto R, Tortora G (2007) Recovering traceability links in software artifact management systems using information retrieval methods. In: ACM Transactions on Software Engineering and Methodology, 16(4):Article 13Google Scholar
  8. Dias Neto AC, Arilo C, Subramanyan R, Vieira M, Travassos GH (2007) A survey on model-based testing approaches: A systematic review. In: Proc of 1st ACM Int workshop on Empirical Assessment of Software Engineering Languages and Technologies, pp. 31–36Google Scholar
  9. Ferguson RW, Lami G (2006) An empirical study on the relationship between defective requirements and test failures. In: Proc of 30th Annual IEEE/NASA Software Engineering Workshop SEW-30 (SEW’06)Google Scholar
  10. Fogelström N, Gorschek T (2007) Test-case driven versus checklist-based inspections of software requirements—an experimental evaluation, In: Proc. of 10th Workshop on Requirements Engineering (WER’07)Google Scholar
  11. Gotel O, Finkelstein A (1994) An analysis of the requirements traceability problem. In: Proc. First Int Conf. Requirements Eng., pp. 94–101Google Scholar
  12. Gorschek T, Davis AM (2007) Requirements engineering: in search of the dependent variables. Inf Softw Technol 50(1–2):67–75Google Scholar
  13. Gorschek T, Wohlin C (2004) Packaging software process improvement issues—a method and a case study. Softw Pract & Experience 34:1311–1344CrossRefGoogle Scholar
  14. Gorschek T, Wohlin C (2006) Requirements abstraction model. Requir Eng J 11:79–101CrossRefGoogle Scholar
  15. Graham D (2002) Requirements and testing: seven missing-link myths. IEEE Softw 19:15–17CrossRefGoogle Scholar
  16. Grieskamp W, Kicillof N, Stobie K, Braberman V (2011) Model-based quality assurance of protocol documentation: tools and methodology. Softw Test Verification Reliab 21(1):55–71CrossRefGoogle Scholar
  17. Hasling B, Goetz H, Beetz K (2008) Model based testing of system requirements using UML use case models. In: Proc of 2008 international Conference on Software Testing, Verification, and ValidationGoogle Scholar
  18. Hayes JH, Dekhtyar A, Sundaram SK, Holbrook EA, Vadlamudi S, April A (2007) REquirements TRacing On target (RETRO): improving software maintenance through traceability recovery. Innov Syst Softw Eng 3(3):193–202CrossRefGoogle Scholar
  19. Höst M, Feldt R, Lüders F (2010) Support for different stakeholders in software engineering master thesis projects. IEEE Trans Educ 52(2):288–296. doi:10.1109/TE.2009.2016106 CrossRefGoogle Scholar
  20. IEEE (1990) IEEE standard glossary of software engineering terminology. Technical Report 610.12–1990. IEEE, New YorkGoogle Scholar
  21. ISO/IEC. (2001) 9126–1:2001(E), International standard software engineering product quality part 1: Quality model. Technical report, ISO/IECGoogle Scholar
  22. Jarke, M (1998) Requirements Traceability. Commun ACM 41(12):32–36Google Scholar
  23. Jones JA, Grechanik M, van der Hoek A (2009) Enabling and enhancing collaborations between software development organizations and independent test agencies. In: Cooperative and Human Aspects of Software Engineering (CHASE’09), May 17, 2009, Vancouver, CanadaGoogle Scholar
  24. Kraut RE, Streeter L (1995) Coordination in software development. Commun ACM 38(3):69–81CrossRefGoogle Scholar
  25. Kukkanen J, Vakevainen K, Kauppinen M, Uusitalo E (2009) Applying a systematic approach to link requirements and testing: A case study, In: Proc of Asia-Pacific Software Engineering Conference (APSEC ’09):482–488Google Scholar
  26. Lormans M, van Deursen A, Gross H (2008) An industrial case study in reconstructing requirements views, In: Empirical Software Engineering, online first, September 03, 2008Google Scholar
  27. Lubars M, Potts C, Richter C (1993) A review of the state of the practice in requirements modelling. In: Proc of 1st IEEE Int. Symposium on Requirements Engineering, pp. 2–14Google Scholar
  28. Martin R, Melnik G (2008) Tests and requirements, requirements and tests a Möbius strip. IEEE Softw 25(1):54–59CrossRefGoogle Scholar
  29. Melnik G, Maurer F, Chiasson M (2006) Executable acceptance tests for communicating business requirements: Customer perspective. In: Proc. of Agile Conference, Minneapolis, USA, pp. 12–46Google Scholar
  30. Mohagheghi P, Dehlen V (2008) Where is the proof? - A review of experiences from applying MDE in industry. In: LNCS, Model driven architecture—foundations and applications, vol 5095:432–443Google Scholar
  31. Nebut C, Fleurey F, Traon YL, Jézéquel J (2006) Automatic test generation: a use case driven approach. IEEE Trans Softw Eng 32(3):140–155CrossRefGoogle Scholar
  32. Paci F, Massacci F, Bouquet F, Debricon S (2012) Managing evolution by orchestrating requirements and testing engineering processes. In: Proc. of IEEE 5th Int. Conf. On, pp.834–841, 17–21 April 2012Google Scholar
  33. Pettersson F, Ivarsson M, Gorschek (2008) A practitioner’s guide to light weight software process assessment and improvement planning. J Syst Softw 81(6):972–995CrossRefGoogle Scholar
  34. Post H, Sinz C, Merz F, Gorges T, Kropf T (2009) Linking functional requirements and software verification. In: Proceedings of 17th IEEE International Requirements Engineering Conference, pp. 295–302Google Scholar
  35. Ramesh B, Stubbs C, Powers T, Edwards M (1997) Requirements traceability: theory and practice. Ann Softw Eng 3(1):397–415CrossRefGoogle Scholar
  36. Ramesh B (1998) Factors influencing requirements traceability practice. In: Communications of the ACM CACM Homepage archive, 41(12):37–44Google Scholar
  37. Randell B (1969) Towards a methodology of computing system design. In: Naur P, Randell B (eds) NATO working conference on software engineering 1968. Report on a Conference Sponsored by NATO Scientific Committee, Garmisch, pp 204–208Google Scholar
  38. Regnell B, Runeson P (1998) Combining scenario-based requirements with static verification and dynamic testing. In: Proc. 4th Int. Working Conf. Requirements Engineering: Foundation for Software Quality, pp.195–206Google Scholar
  39. Regnell B, Runeson P, Wohlin C (2000) Towards integration of use case modelling and usage-based testing. J Syst Softw 50(2):117–130CrossRefGoogle Scholar
  40. Ricca F, Torchiano M, Di Penta M, Ceccato M, Tonella P (2009) Using acceptance tests as a support for clarifying requirements: a series of experiments. Inf Softw Technol 51:270–283CrossRefGoogle Scholar
  41. Robson C (2002) Real world research: A resource for social scientists and practitioner researchers, 2nd edition. Blackwell PublishingGoogle Scholar
  42. Runeson P, Andersson C, Höst M (2003) Test processes in software product evolution—a qualitative survey on the state of practice. J Softw Maint Evol Res Pract 15(1):41–59MATHCrossRefGoogle Scholar
  43. Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131–164CrossRefGoogle Scholar
  44. Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software engineering—guidelines and examples. WileyGoogle Scholar
  45. Sabaliauskaite G, Loconsole A, Engström E, Unterkalmsteiner M, Regnell B, Runeson P, Gorschek T, Feldt R (2010) Challenges in aligning requirements engineering and verification in a large-scale industrial context. In: Proceedings of REFSQ 2010Google Scholar
  46. Sikora E, Tenbergen B, Pohl K (2012) Industry needs and research directions in requirements engineering for embedded systems. Requir Eng 17(1):57–78CrossRefGoogle Scholar
  47. Uusitalo EJ, Komassi M, Kauppinen M, Davis AM (2008) Linking requirements and testing in practice. In: Proceedings of 16th IEEE International Requirements Engineering Conference, pp. 295–302Google Scholar
  48. Watkins R, Neal M (1994) Why and how of requirements tracing. IEEE Softw 11(4):104–106CrossRefGoogle Scholar
  49. Yue T, Briand L, Labiche Y (2011) A systematic review of transformation approaches between user requirements and analysis models. Requir Eng 16(2):75–99CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Elizabeth Bjarnason
    • 1
  • Per Runeson
    • 1
  • Markus Borg
    • 1
  • Michael Unterkalmsteiner
    • 2
  • Emelie Engström
    • 1
  • Björn Regnell
    • 1
  • Giedre Sabaliauskaite
    • 5
  • Annabella Loconsole
    • 3
  • Tony Gorschek
    • 2
  • Robert Feldt
    • 2
    • 4
  1. 1.Department of Computer ScienceLund UniversityLundSweden
  2. 2.Blekinge Institute of TechnologySchool of ComputingKarlskronaSweden
  3. 3.Malmö UniversityMalmöSweden
  4. 4.Department of Computer Science and EngineeringChalmers University of TechnologyGöteborgSweden
  5. 5.Singapore University of Technology and DesignSingaporeSingapore

Personalised recommendations