Requirements Engineering

, Volume 9, Issue 4, pp 248–260 | Cite as

A framework for empirical evaluation of conceptual modeling techniques

  • Andrew GeminoEmail author
  • Yair Wand
Original Article


The paper presents a framework for the empirical evaluation of conceptual modeling techniques used in requirements engineering. The framework is based on the notion that modeling techniques should be compared via their underlying grammars. The framework identifies two types of dimensions in empirical comparisons—affecting and affected dimensions. The affecting dimensions provide guidance for task definition, independent variables and controls, while the affected dimensions define the possible mediating variables and dependent variables. In particular, the framework addresses the dependence between the modeling task—model creation and model interpretation—and the performance measures of the modeling grammar. The utility of the framework is demonstrated by using it to categorize existing work on evaluating modeling techniques. The paper also discusses theoretical foundations that can guide hypothesis generation and measurement of variables. Finally, the paper addresses possible levels for categorical variables and ways to measure interval variables, especially the grammar performance measures.


System analysis and design Information systems development Conceptual modeling Empirical comparison Requirements engineering 



This work was supported in part by grants from the Social Sciences and Humanities and Natural Sciences and Engineering Research Councils of Canada.


  1. 1.
    Avison DE, Fitzgerald G (1995) Information systems development: methodologies, techniques, and tools, 2nd edn. McGraw-Hill, LondonGoogle Scholar
  2. 2.
    Agarwal R, Sinha A, Tanniru M (1996) Cognitive fit in requirements engineering: a study of object and process models. J Manag Inf Syst 13(2):137–162Google Scholar
  3. 3.
    Agarwal R, De P, Sinha AP (1999) Comprehending object and process models: an empirical study. IEEE Trans Softw Eng 25(4):541–556CrossRefGoogle Scholar
  4. 4.
    Batra D, J Hoffer, R Bostrom (1990) Comparing representations with relational and EER models. Commun ACM 33(2):126–139CrossRefGoogle Scholar
  5. 5.
    Bodart F, Sim M, Patel A, Weber R (2001) Should optional properties be used in conceptual modelling? A theory and three empirical tests. Inf Syst Res 12(4):384–405CrossRefGoogle Scholar
  6. 6.
    Brooks FP (1998) The mythical man-month: essays of software engineering, Anniversary edition. Addison-WesleyGoogle Scholar
  7. 7.
    Brosey M, Schniederman B (1978) Two experimental comparisons of relational and hierarchical database models. Int J Man Machine Stud 10:625–637CrossRefGoogle Scholar
  8. 8.
    Bubenko JA Jr (1986) Information system methodologies: a research review. In: Olle TW, Sol HG, Verrjin-Stuart AA (eds) Information system design methodologies: improving the practice. Elsevier, North Holland, pp 289–317Google Scholar
  9. 9.
    Burton-Jones A, Meso P (2002) How good are these UML diagrams? An empirical test of the Wand and Weber good decomposition model. In: Applegate L, Galliers R, DeGross JI (eds) Proceedings of the international conference on information systems 2002, December, 2002Google Scholar
  10. 10.
    Chaos (1995) Standish group report on information system development. Accessed 12 Jul 1996]
  11. 11.
    Collins AM, Quillan MR (1969) Retrieval time from semantic memory. J Verbal Learn Behav 8:240–247Google Scholar
  12. 12.
    Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technologies. MIS Q 13(3):319–340Google Scholar
  13. 13.
    Floyd C (1986) A comparative evaluation of systems development methods. In: Olle TW, Sol HG, Verrjin-Stuart AA (eds) Proceedings of the IFIP WG 8.1 working conference on comparative review of information systems design methodologies: improving the practice. North-Holland, Amsterdam, pp 19–54Google Scholar
  14. 14.
    Gemino A (1999) Empirical comparison of system analysis techniques. PhD Thesis, University of British ColumbiaGoogle Scholar
  15. 15.
    Gemino A (2004) Empirical comparisons of animation and narration in requirements validation. Req Eng J 9(3):153–168. DOI 10.1007/s00766-003-0182-0 Google Scholar
  16. 16.
    Gemino A, Wand Y (2003) Evaluation of modeling techniques based on models of learning. Commun ACM 46(10):79–84CrossRefGoogle Scholar
  17. 17.
    Jarvenpaa S, Machesky J (1989) Data analysis and learning: an experimental study of data modeling tools. Int J Man Mach Stud 31:367–391CrossRefGoogle Scholar
  18. 18.
    Johnson J, Boucher KD, Connors K, Robinson J (2001) The criteria for success. Softw Mag 21(1):s3–s8Google Scholar
  19. 19.
    Kim YG, March S (1995) Comparing data modeling formalisms. Commun ACM 38(4):103–115CrossRefGoogle Scholar
  20. 20.
    Kim J, Hahn J, Hahn H (2000) How do we understand a system with (so) many diagrams? Cognitive integration processes in diagrammatic reasoning. Info Syst Res 11(3):284–303CrossRefGoogle Scholar
  21. 21.
    Kung CH, Solvberg A (1986) Activity modelling and behaviour modelling. In: Olle TW, Sol HG, Verrjin-Stuart AA (eds) Proceedings of the IFIP WG 8.1 working conference on comparative review of information systems design methodologies: improving the practice. North-Holland, Amsterdam, pp 145–171Google Scholar
  22. 22.
    Larkin J, Simon HA (1987) When a diagram is (sometimes) worth ten thousand words. Cogn Sci 11(1):65–99Google Scholar
  23. 23.
    Mayer RE (1989) Models for understanding. Rev Educ Res 59(1):43–64Google Scholar
  24. 24.
    Mayer RE (2001) Multimedia learning. Cambridge University Press, New YorkGoogle Scholar
  25. 25.
    Moore GC, I (1991) Development of an instrument to measure perceptions of adopting an information technology innovation. Inf Syst Res 2(3):192–222zbMATHGoogle Scholar
  26. 26.
    Morris MG, Spier C, Hoffer JA (1999) An examination of procedural and object-oriented system analysis methods: does prior experience help or hinder performance. Decis Sci 30(1):107–136Google Scholar
  27. 27.
    Mylopoulos J (1992) Conceptual modeling and telos. In: Loucopoulos P, Zicari R (eds) Conceptual modeling, databases, and case: an integrated view of information systems development, chap 2. Wiley, New York, pp 49–68Google Scholar
  28. 28.
    Norman D (1986) Cognitive engineering. In: Norman D, Draper S (eds) User centered design: new perspectives on human computer interaction. Lawrence Erlbaum Associates, Hillsdale, pp 31–61Google Scholar
  29. 29.
    Nosek J, Ahrens J (1986) An experiment to test user validation of requirements: data flow diagrams vs. task oriented menus. Int J Man Mach Stud 25:675–684Google Scholar
  30. 30.
    Oei JLH, van Hemmen LJ, Falkenberg ED, Brinkkemper S (1992) The meta model hierarchy: a framework for information systems concepts and techniques. Technical Report No. 92-17, Department of Informatics, Faculty of Mathematics and Informatics, Katholieke Universiteir, Nijmegen, pp 1–30Google Scholar
  31. 31.
    Ramsey R, Atwood M, Van Doren J (1993) Flowcharts versus program design languages: an experimental comparison. Commun ACM 26(6):445–449CrossRefGoogle Scholar
  32. 32.
    Siau KL (1996) Empirical studies in information modeling. PhD Thesis, University of British ColumbiaGoogle Scholar
  33. 33.
    Vessey I, Conger S (1994) Requirements specification: learning object, process, and data methodologies. Commun ACM 37(5):102–113CrossRefGoogle Scholar
  34. 34.
    Wand Y, Weber R (1993) On the ontological expressiveness of information systems analysis and design grammars. J Inf Syst 3:217–237Google Scholar
  35. 35.
    Wand Y, Weber R (1995) On the deep structure of information systems. Inf Syst J 5:203–223Google Scholar
  36. 36.
    Wand Y, Weber R (2002) Information systems and conceptual modeling a research agenda. Inf Syst Res 13(4):203–223Google Scholar
  37. 37.
    Wang S (1996) Two MIS analysis methods: an experimental comparison. J Educ Bus Jan/Feb:136–141Google Scholar
  38. 38.
    Yadav S, Bravoco R, Chatfield A, Rajkumar T (1988) Comparison of analysis techniques for information requirements determination. Commun ACM 31(9):1090–1097CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2004

Authors and Affiliations

  1. 1.Faculty of Business AdministrationSimon Fraser UniversityBurnabyCanada
  2. 2.Sauder School of BusinessUniversity of British ColumbiaVancouverCanada

Personalised recommendations