Requirements Engineering

, Volume 11, Issue 1, pp 102–107 | Cite as

Requirements engineering paper classification and evaluation criteria: a proposal and a discussion

  • Roel WieringaEmail author
  • Neil Maiden
  • Nancy Mead
  • Colette Rolland
Original Article


In recent years, members of the steering committee of the IEEE Requirements Engineering (RE) Conference have discussed paper classification and evaluation criteria for RE papers. The immediate trigger for this discussion was our concern about differences in opinion that sometimes arise in program committees about the criteria to be used in evaluating papers. If program committee members do not all use the same criteria, or if they use criteria different from those used by authors, then papers might be rejected or accepted for the wrong reasons. Surely not all papers should be evaluated according to the same criteria. Some papers describe new techniques but do not report on empirical research; others describe new conceptual frameworks for investigating certain RE problems; others report on industrial experience with existing RE techniques. Other kinds of papers can also be easily recognized. All of these types of papers should be evaluated according to different criteria....


Requirements engineering research Research methods Paper classification Paper evaluation criteria 



This paper benefited from discussions with Al Davis, Sol Greenspan, Hans Heerkens, Ann Hickey, and Pamela Zave and from comments by the anonymous reviewers.


  1. 1.
    Archer LB (1984) Systematic method for designers. In: Cross N (eds) Developments in design methodology. Wiley, Chichester, pp 57–82Google Scholar
  2. 2.
    Asimov M (1962) Introduction to design. Prentice-Hall, Englewood CliffsGoogle Scholar
  3. 3.
    Auyang S (2004) Engineering: an endless frontier. Harvard University Press, CambridgeGoogle Scholar
  4. 4.
    Brooks F (1996) The computer scientist as toolsmith II. Commun ACM 39(3):61–68CrossRefMathSciNetGoogle Scholar
  5. 5.
    Bush D (2005) Modelling support for early identification of safety requirements: a preliminary investigation. In: Proceedings RHAS’2005 workshop, 13th international IEEE conference on requirements engineering, Paris, August 2005Google Scholar
  6. 6.
    Cross N (1994) Engineering design methods: strategies for product design, 2nd edn. Wiley, ChichesterGoogle Scholar
  7. 7.
    Cross N (2001) Design cognition: results from protocol and other empirical studies of design activity. In: Eastman C, McCracken M, Newstetter W (eds) Design knowing and learning: cognition in design education. Elsevier, Amsterdam, pp 79–103Google Scholar
  8. 8.
    Davis AM, Hickey AM (2004) A new paradigm for planning and evaluating requirements engineering research. In: Second international workshop on comparative evaluation in requirements engineering, Kyoto, Japan, pp 7–16Google Scholar
  9. 9.
    Easterbrook S, Yu E, Aranda J, Fan Y, Horkoff J, Leica M, Qadir RA (2005) Do viewpoints lead to better conceptual models? An exploratory case study. In: Proceedings of 13th IEEE international conference on requirements engineering, IEEE Computer Society Press, pp 199–208Google Scholar
  10. 10.
    El Emam K, Madhavji NH (1995) A field study of requirements engineering practices in information systems development. In: Proceedings of 2nd IEEE international symposium on requirements engineering, IEEE Computer Society Press, pp 68–80Google Scholar
  11. 11.
    Feynman RP (1994) Six easy pieces. Perseus BooksGoogle Scholar
  12. 12.
    Glass R, Ramesh V, Vessey I (2004) An analysis of research in the computing disciplines. Commun ACM 47(6):89–94CrossRefGoogle Scholar
  13. 13.
    Hall AD (1962) A methodology for systems engineering. Van Nostrand, PrincetonGoogle Scholar
  14. 14.
    Hicks MJ (1991) Problem solving in business and management. Hard, soft and creative approaches. International Thomson Business PressGoogle Scholar
  15. 15.
    IEEE (2004) Genre submission guide. Cited 22 May 2005Google Scholar
  16. 16.
    Jones JC (1984) A method for systematic design. In: Cross N (ed), Developments in design methodology. Wiley, pp 9–31Google Scholar
  17. 17.
    Maiden NAM, Jones SV, Manning S, Greenwood J, Renou L (2004) Model-driven requirements engineering: synchronising models in an air traffic management case study. In: Proceedings CaiSE’2004, Springer-Verlag LNCS 3084, pp 368–383Google Scholar
  18. 18.
    March JG (1994) A primer on decision making. How decisions happen. The Free PressGoogle Scholar
  19. 19.
    Merriam-Webster. Scholar
  20. 20.
    Roozenburg NFM, Eekels J (1995) Product design: fundamentals and methods. WileyGoogle Scholar
  21. 21.
    Tichy W, Lukowicz P, Prechelt L, Heinz E (1997) Experimental evaluation in computer science: a quantitative study. J Syst Softw 28:9–18CrossRefGoogle Scholar
  22. 22.
    Vincenti WG (1990) What engineers know and how they know it. Johns Hopkins University Press, BaltimoreGoogle Scholar
  23. 23.
    Wieringa RJ (1996) Requirements engineering: frameworks for understanding. Wiley, New YorkGoogle Scholar
  24. 24.
    Wieringa RJ, Heerkens H (2004) Evaluating the structure of research papers: a case study. In: Gervasi V, Zowghi D, Sim SE (eds) Second international workshop in comparative evaluation of requirements engineering, pp 29–38Google Scholar
  25. 25.
    Witte E (1972) Field research on complex decision-making processes—the phase theorem. Int Stud Manage Organ 2:156–182Google Scholar
  26. 26.
    WordNet. Scholar
  27. 27.
    Yu E, Mylopoulos JM (1994) Understanding “why” in software process modelling, analysis and design. In: Proceedings of 16th international conference on software engineering, IEEE Computer Society Press, pp 159–168Google Scholar
  28. 28.
    Zave P (1997) Classification of research efforts in requirements engineering. ACM Comput Surv 29(4):315–321CrossRefGoogle Scholar
  29. 29.
    Zelkowitz M, Wallace D (1997) Experimental validation in software engineering. Inf Softw Technol 39:735–743CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2005

Authors and Affiliations

  • Roel Wieringa
    • 1
    Email author
  • Neil Maiden
    • 2
  • Nancy Mead
    • 3
  • Colette Rolland
    • 4
  1. 1.University of TwenteTwenteThe Netherlands
  2. 2.City UniversityLondonUK
  3. 3.Software Engineering InstituteCarnegie Mellon universityPittsburgUSA
  4. 4.Université Paris1, Panthéon SorbonneParisFrance

Personalised recommendations