Skip to main content
SpringerLink
Log in

Challenges and practices in aligning requirements with verification and validation: a case study of six companies

Empirical Software Engineering volume 19, pages 1809–1855 (2014)Cite this article

Abstract

Weak alignment of requirements engineering (RE) with verification and validation (VV) may lead to problems in delivering the required products in time with the right quality. For example, weak communication of requirements changes to testers may result in lack of verification of new requirements and incorrect verification of old invalid requirements, leading to software quality problems, wasted effort and delays. However, despite the serious implications of weak alignment research and practice both tend to focus on one or the other of RE or VV rather than on the alignment of the two. We have performed a multi-unit case study to gain insight into issues around aligning RE and VV by interviewing 30 practitioners from 6 software developing companies, involving 10 researchers in a flexible research process for case studies. The results describe current industry challenges and practices in aligning RE with VV, ranging from quality of the individual RE and VV activities, through tracing and tools, to change control and sharing a common understanding at strategy, goal and design level. The study identified that human aspects are central, i.e. cooperation and communication, and that requirements engineering practices are a critical basis for alignment. Further, the size of an organisation and its motivation for applying alignment practices, e.g. external enforcement of traceability, are variation factors that play a key role in achieving alignment. Our results provide a strategic roadmap for practitioners improvement work to address alignment challenges. Furthermore, the study provides a foundation for continued research to improve the alignment of RE with VV.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. Reference to source is given by interviewee code, see Table 2.

References

  • Barmi ZA, Ebrahimi AH, Feldt R (2011) Alignment of requirements specification and testing: A systematic mapping study. In: Proc 4th Int. Conf. On Software Testing, Verification and Validation Workshops (ICSTW) pp. 476–485

  • Cao L, Ramesh B (2008) Agile requirements engineering practices: An empirical study. In: IEEE Software Jan/Feb 2008

  • Cheng BH, Atlee JM (2007) Research directions in requirements engineering. In: Proc. Future of Software Engineering (FOSE) pp. 285–303

  • Cleland-Huang J, Chang CK, Christensen M (2003) Event-based traceability for managing evolutionary change. In: IEEE Transactions on Software 29(9)

  • Damian D, Chisan J, Vaidyanathasamy L, Pal Y (2005) Requirements engineering and downstream software development: findings from a case study. Empir Softw Eng 10:255–283

    Article  Google Scholar 

  • Damian D, Chisan J (2006) An empirical study of the complex relationship between requirements engineering processes and other processes that lead to payoffs in productivity, quality, and risk management. IEEE Trans Softw Eng 32(7):33–453

    Article  Google Scholar 

  • De Lucia A, Fasano F, Oliveto R, Tortora G (2007) Recovering traceability links in software artifact management systems using information retrieval methods. In: ACM Transactions on Software Engineering and Methodology, 16(4):Article 13

  • Dias Neto AC, Arilo C, Subramanyan R, Vieira M, Travassos GH (2007) A survey on model-based testing approaches: A systematic review. In: Proc of 1st ACM Int workshop on Empirical Assessment of Software Engineering Languages and Technologies, pp. 31–36

  • Ferguson RW, Lami G (2006) An empirical study on the relationship between defective requirements and test failures. In: Proc of 30th Annual IEEE/NASA Software Engineering Workshop SEW-30 (SEW’06)

  • Fogelström N, Gorschek T (2007) Test-case driven versus checklist-based inspections of software requirements—an experimental evaluation, In: Proc. of 10th Workshop on Requirements Engineering (WER’07)

  • Gotel O, Finkelstein A (1994) An analysis of the requirements traceability problem. In: Proc. First Int Conf. Requirements Eng., pp. 94–101

  • Gorschek T, Davis AM (2007) Requirements engineering: in search of the dependent variables. Inf Softw Technol 50(1–2):67–75

    Google Scholar 

  • Gorschek T, Wohlin C (2004) Packaging software process improvement issues—a method and a case study. Softw Pract & Experience 34:1311–1344

    Article  Google Scholar 

  • Gorschek T, Wohlin C (2006) Requirements abstraction model. Requir Eng J 11:79–101

    Article  Google Scholar 

  • Graham D (2002) Requirements and testing: seven missing-link myths. IEEE Softw 19:15–17

    Article  Google Scholar 

  • Grieskamp W, Kicillof N, Stobie K, Braberman V (2011) Model-based quality assurance of protocol documentation: tools and methodology. Softw Test Verification Reliab 21(1):55–71

    Article  Google Scholar 

  • Hasling B, Goetz H, Beetz K (2008) Model based testing of system requirements using UML use case models. In: Proc of 2008 international Conference on Software Testing, Verification, and Validation

  • Hayes JH, Dekhtyar A, Sundaram SK, Holbrook EA, Vadlamudi S, April A (2007) REquirements TRacing On target (RETRO): improving software maintenance through traceability recovery. Innov Syst Softw Eng 3(3):193–202

    Article  Google Scholar 

  • Höst M, Feldt R, Lüders F (2010) Support for different stakeholders in software engineering master thesis projects. IEEE Trans Educ 52(2):288–296. doi:10.1109/TE.2009.2016106

    Article  Google Scholar 

  • IEEE (1990) IEEE standard glossary of software engineering terminology. Technical Report 610.12–1990. IEEE, New York

    Google Scholar 

  • ISO/IEC. (2001) 9126–1:2001(E), International standard software engineering product quality part 1: Quality model. Technical report, ISO/IEC

  • Jarke, M (1998) Requirements Traceability. Commun ACM 41(12):32–36

    Google Scholar 

  • Jones JA, Grechanik M, van der Hoek A (2009) Enabling and enhancing collaborations between software development organizations and independent test agencies. In: Cooperative and Human Aspects of Software Engineering (CHASE’09), May 17, 2009, Vancouver, Canada

  • Kraut RE, Streeter L (1995) Coordination in software development. Commun ACM 38(3):69–81

    Article  Google Scholar 

  • Kukkanen J, Vakevainen K, Kauppinen M, Uusitalo E (2009) Applying a systematic approach to link requirements and testing: A case study, In: Proc of Asia-Pacific Software Engineering Conference (APSEC ’09):482–488

  • Lormans M, van Deursen A, Gross H (2008) An industrial case study in reconstructing requirements views, In: Empirical Software Engineering, online first, September 03, 2008

  • Lubars M, Potts C, Richter C (1993) A review of the state of the practice in requirements modelling. In: Proc of 1st IEEE Int. Symposium on Requirements Engineering, pp. 2–14

  • Martin R, Melnik G (2008) Tests and requirements, requirements and tests a Möbius strip. IEEE Softw 25(1):54–59

    Article  Google Scholar 

  • Melnik G, Maurer F, Chiasson M (2006) Executable acceptance tests for communicating business requirements: Customer perspective. In: Proc. of Agile Conference, Minneapolis, USA, pp. 12–46

  • Mohagheghi P, Dehlen V (2008) Where is the proof? - A review of experiences from applying MDE in industry. In: LNCS, Model driven architecture—foundations and applications, vol 5095:432–443

  • Nebut C, Fleurey F, Traon YL, Jézéquel J (2006) Automatic test generation: a use case driven approach. IEEE Trans Softw Eng 32(3):140–155

    Article  Google Scholar 

  • Paci F, Massacci F, Bouquet F, Debricon S (2012) Managing evolution by orchestrating requirements and testing engineering processes. In: Proc. of IEEE 5th Int. Conf. On, pp.834–841, 17–21 April 2012

  • Pettersson F, Ivarsson M, Gorschek (2008) A practitioner’s guide to light weight software process assessment and improvement planning. J Syst Softw 81(6):972–995

    Article  Google Scholar 

  • Post H, Sinz C, Merz F, Gorges T, Kropf T (2009) Linking functional requirements and software verification. In: Proceedings of 17th IEEE International Requirements Engineering Conference, pp. 295–302

  • Ramesh B, Stubbs C, Powers T, Edwards M (1997) Requirements traceability: theory and practice. Ann Softw Eng 3(1):397–415

    Article  Google Scholar 

  • Ramesh B (1998) Factors influencing requirements traceability practice. In: Communications of the ACM CACM Homepage archive, 41(12):37–44

  • Randell B (1969) Towards a methodology of computing system design. In: Naur P, Randell B (eds) NATO working conference on software engineering 1968. Report on a Conference Sponsored by NATO Scientific Committee, Garmisch, pp 204–208

    Google Scholar 

  • Regnell B, Runeson P (1998) Combining scenario-based requirements with static verification and dynamic testing. In: Proc. 4th Int. Working Conf. Requirements Engineering: Foundation for Software Quality, pp.195–206

  • Regnell B, Runeson P, Wohlin C (2000) Towards integration of use case modelling and usage-based testing. J Syst Softw 50(2):117–130

    Article  Google Scholar 

  • Ricca F, Torchiano M, Di Penta M, Ceccato M, Tonella P (2009) Using acceptance tests as a support for clarifying requirements: a series of experiments. Inf Softw Technol 51:270–283

    Article  Google Scholar 

  • Robson C (2002) Real world research: A resource for social scientists and practitioner researchers, 2nd edition. Blackwell Publishing

  • Runeson P, Andersson C, Höst M (2003) Test processes in software product evolution—a qualitative survey on the state of practice. J Softw Maint Evol Res Pract 15(1):41–59

    Article  MATH  Google Scholar 

  • Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131–164

    Article  Google Scholar 

  • Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software engineering—guidelines and examples. Wiley

  • Sabaliauskaite G, Loconsole A, Engström E, Unterkalmsteiner M, Regnell B, Runeson P, Gorschek T, Feldt R (2010) Challenges in aligning requirements engineering and verification in a large-scale industrial context. In: Proceedings of REFSQ 2010

  • Sikora E, Tenbergen B, Pohl K (2012) Industry needs and research directions in requirements engineering for embedded systems. Requir Eng 17(1):57–78

    Article  Google Scholar 

  • Uusitalo EJ, Komassi M, Kauppinen M, Davis AM (2008) Linking requirements and testing in practice. In: Proceedings of 16th IEEE International Requirements Engineering Conference, pp. 295–302

  • Watkins R, Neal M (1994) Why and how of requirements tracing. IEEE Softw 11(4):104–106

    Article  Google Scholar 

  • Yue T, Briand L, Labiche Y (2011) A systematic review of transformation approaches between user requirements and analysis models. Requir Eng 16(2):75–99

    Article  Google Scholar 

Download references

Acknowledgment

We want to thank Börje Haugset for acting as interviewer in three of the interviews. We would also like to thank all the participating companies and anonymous interviewees for their contribution to this project. The research was funded by EASE Industrial Excellence Center for Embedded Applications Software Engineering (http://ease.cs.lth.se).

Author information

Authors and Affiliations

  1. Department of Computer Science, Lund University, Box 118, SE-221 00, Lund, Sweden

    Elizabeth Bjarnason, Per Runeson, Markus Borg, Emelie Engström & Björn Regnell

  2. Blekinge Institute of Technology, School of Computing, SE-371 79, Karlskrona, Sweden

    Michael Unterkalmsteiner, Tony Gorschek & Robert Feldt

  3. Malmö University, Teknik och samhälle, SE-205 06, Malmö, Sweden

    Annabella Loconsole

  4. Department of Computer Science and Engineering, Chalmers University of Technology, SE-412 96, Göteborg, Sweden

    Robert Feldt

  5. Singapore University of Technology and Design, 20 Dover Drive, Singapore, 138682, Singapore

    Giedre Sabaliauskaite

Authors
  1. Elizabeth Bjarnason
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Per Runeson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Markus Borg
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Unterkalmsteiner
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Emelie Engström
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Björn Regnell
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Giedre Sabaliauskaite
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Annabella Loconsole
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Tony Gorschek
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Robert Feldt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elizabeth Bjarnason.

Additional information

Communicated by: Nachiappan Nagappan

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Bjarnason, E., Runeson, P., Borg, M. et al. Challenges and practices in aligning requirements with verification and validation: a case study of six companies. Empir Software Eng 19, 1809–1855 (2014). https://doi.org/10.1007/s10664-013-9263-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-013-9263-y

Keywords

search

Navigation