Empirical Software Engineering

, Volume 18, Issue 1, pp 139–176 | Cite as

A study of methods for textual satisfaction assessment

  • Elizabeth Ashlee Holbrook
  • Jane Huffman Hayes
  • Alex Dekhtyar
  • Wenbin Li


Software projects requiring satisfaction assessment are often large scale systems containing hundreds of requirements and design elements. These projects may exist within a high assurance domain where human lives and millions of dollars are at stake. Satisfaction assessment can help identify unsatisfied requirements early in the software development lifecycle, when issues can be corrected with less impact and lower cost. Manual satisfaction assessment is expensive both in terms of human effort and project cost. Automated satisfaction assessment assists requirements analysts during the satisfaction assessment process to more quickly determine satisfied requirements and to reduce the satisfaction assessment search space. This paper introduces two new automated satisfaction assessment techniques and empirically demonstrates their effectiveness, as well as validates two previously existing automated satisfaction assessment techniques. Validation shows that automatically generated satisfaction assessments have high accuracy, thus reducing the workload of the analyst in the satisfaction assessment process.


Methods for SQA and V&V Requirements/specifications Validation Tracing 



This work is funded in part by the National Science Foundation under NSF grant CCF-0811140. This work was partially sponsored by NASA under grant NNG05GQ58G. We thank David Pruett and the other evaluators for their help. We thank Hakim Sultanov and Bill Kidwell. Thanks to Stephanie Ferguson, Marcus Fisher, Ken McGill, Tim Menzies, Lisa Montgomery, and everyone at the NASA IV&V facility. Thanks also to fellow graduate students Jody Larsen, Senthil Sundaram, Liming Zhao, and Sravanthi Vadlamudi. We thank statistics professor Dr. Arnold Stromberg.


  1. Altman DG (1991) Practical statistics for medical research. Chapman & HallGoogle Scholar
  2. Antoniol G (2002) Recovering traceability links between code and documentation. IEEE Trans on Software Engineering 28(10):970–983CrossRefGoogle Scholar
  3. Asplaugh TA, Antón AI (2008) Scenario support for effective requirements. Inf Softw Technol 50(3):198–220CrossRefGoogle Scholar
  4. Baeza-Yates R, Ribeiro-Neto B (2003) Modern information retrieval. Addison-WesleyGoogle Scholar
  5. Cleland-Huang J (2002) Automating speculative queries through event-based requirements traceability. Joint Conference on Requirements EngineeringGoogle Scholar
  6. Cleland-Huang J, Settimi R, BenKhadra O, Berezhanskaya E, Christina S (2005) Goal-centric traceability for managing non-functional requirements. In: Proceedings of the 27th international conference on Software engineering (ICSE '05). ACM, New York, NY, pp 362–371. doi: 10.1145/1062455.1062525.
  7. Cuddeback D, Dekhtyar A, Hayes JH (2010) Automated requirements traceability: the study of human analysts. In Proc. 18th International Conference on Requirements Engineering, Sydney, AustraliaGoogle Scholar
  8. Di Lucca GA, Di Penta M, Antoniol G, Casazza G (2001) An approach for reverse engineering of web-based applications. BP - 231, Dipartimento di Informatica e SistemisticaGoogle Scholar
  9. Diallo MH, Naslavsky L, Ziv H, Alspaugh TA, Richardson DA (2007) Evaluating software architectures against requirements-level scenarios. Workshop on the Role of Software Architecture for Testing and AnalysisGoogle Scholar
  10. Durán A, Ruiz A, Toro M (2001) An automated approach for verification of software requirements. Jornadas de Ingeniería de Requisitos Aplicada, Seville, SpainGoogle Scholar
  11. Gotel OCZ, Finkelstein ACW (1996) Extended requirements traceability: a framework for changing requirements. Workshop on Requirements Engineering in a Changing WorldGoogle Scholar
  12. Greenspan S, Mylopoulos J, Borgida J (1994) On formal requirements modeling languages: RML revisited. Proc. 16th International Conference on Software Engineering, p 135–147, Sorrento, ItalyGoogle Scholar
  13. Hayes JH, Dekhtyar A (2005) Humans in the traceability loop: can't live with 'em, can't live without 'em. Proc. of the 3rd International Workshop on Traceability in Emerging Forms of Software Engineering, Long Beach, California. TEFSE '05. ACM, New York, NY, 20–23Google Scholar
  14. Hayes JH, Dekhtyar A, Osbourne J (2003) Improving requirements tracing via information retrieval. International Conference on Requirements EngineeringGoogle Scholar
  15. Hayes JH, Dekhtyar A, Sundaram S (2006a) Advancing requirements tracing: the study of methods. IEEE Trans Software Engineering 32(1):4–19CrossRefGoogle Scholar
  16. Hayes JH, Dekhtyar A, Sundaram S (2006) Advances in dynamic generation of traceability links. Tech Report, (TR 451–06)Google Scholar
  17. Hayes JH, Dekhtyar A, Sundaram S, Holbrook A, Vadlamudi S, April A (2007) REquirements TRacing On target (RETRO): improving software maintenance through traceability recovery. Innovations in Systems and Software Engineering: A NASA Journal 3(3):193–202Google Scholar
  18. Holbrook EA (2009) Satisfaction assessment of textual software engineering artifacts, PhD dissertation, Dept. of Computer Science., University of Kentucky, Lexington, KYGoogle Scholar
  19. Holbrook EA, Hayes JH, Dekhtyar A (2009) Toward automating requirements satisfaction assessment. Requirements Engineering Conference, 2009. RE '09. 17th IEEE International, vol., no., pp 149–158Google Scholar
  20. ISO9000:2000, Quality Standard, International Organization for StandardizationGoogle Scholar
  21. Lecceuche R (2000) Finding comparatively important concepts between texts. Automated Software Engineering (ASE’00). Washington, DC, 55Google Scholar
  22. Letier E, van Lamsweerde A (2004) Reasoning about partial goal satisfaction for requirements and design engineering. SIGSOFT Softw Eng Notes 29(6):53–62CrossRefGoogle Scholar
  23. Marcus A, Maletic JI (2003) Recovering documentation-to-source code traceability links using latent semantic indexing. In: Proceedings of the 25th International Conference on Software Engineering (ICSE '03). IEEE Computer Society, Washington, DC, pp 125–135Google Scholar
  24. Marcus MP, Santorini B, Marcinkiewicz MA (1993) Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics 19:313–330Google Scholar
  25. Marcus A, Maletic JI, Sergeyev A (2005) Recovery of traceability links between software documentation and source code. Int J Softw Eng Knowl Eng, World Scientific 15(5):811–836Google Scholar
  26. Porter A, Votta L (1998) Comparing detection methods for software requirements inspections. Empir Softw Eng 3(4):355–379CrossRefGoogle Scholar
  27. Robinson WN (2005) Implementing rule-based monitors within a framework for continuous requirements monitoring. In: Proceedings of the Proceedings of the 38th Annual Hawaii International Conference on System Sciences - Volume 07 (HICSS '05), Vol. 7. IEEE Computer Society, Washington, DC, 188.1-. doi: 10.1109/HICSS.2005.306.
  28. Robinson WN (2009) Seeking quality through user-goal monitoring. IEEE Software, pp 58–65Google Scholar
  29. Robinson WN, Pawlowski S (1999) Managing requirements inconsistency with development goal monitors. IEEE Trans. on Software EngGoogle Scholar
  30. Salton G (1983) Introduction to modern information retrieval. McGraw-HillGoogle Scholar
  31. Shull G, Rus I, Basili VR (2000) How perspective-based reading can improve requirements inspections. IEEE Computer 33(7):73–79CrossRefGoogle Scholar
  32. Spanoudakis G, Zisman A (2005) Software traceability: a roadmap. In: Chang SK (ed) Handbook of Software Engineering and Knowledge Engineering, vol. 3: Recent Advancements. World Scientific PublishingGoogle Scholar
  33. Spanoudakis G, d’Avila A, Garcez A, Zisman A (2003) Revising rules to capture requirements traceability relations: a machine learning approach. In: Proceedings of the 15th International Conference in Software Engineering and Knowledge Engineering (SEKE 2003), pp 570–577, San FranciscoGoogle Scholar
  34. Spanoudakis G, Zisman A, Pérez-Miñana E, Krause P (2004) Rule-based generation of requirements traceability relations. J Syst Softw 2(72):105–127CrossRefGoogle Scholar
  35. Spivey J (1988) Understanding Z. CambridgeGoogle Scholar
  36. Sutcliffe A (1998) Scenario-based requirement analysis. Requir Eng 3(1):48–65. doi: 10.1007/BF02802920. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  • Elizabeth Ashlee Holbrook
    • 1
  • Jane Huffman Hayes
    • 2
  • Alex Dekhtyar
    • 3
  • Wenbin Li
    • 2
  1. 1.LexmarkLexingtonUSA
  2. 2.Department of Computer ScienceUniversity of KentuckyLexingtonUSA
  3. 3.Department of Computer ScienceCalifornia Polytechnic State UniversitySan Luis ObispoUSA

Personalised recommendations