Abstract
Goal models have long been considered to be useful tools for representing and analyzing complex decision problems in various stages of the software development lifecycle. Their usefulness for such tasks lies in their ability to compactly represent large numbers of alternative solutions to requirements problems and to capture the impact of each solution to high-level qualities of interest. In this way, goal models allow identification of optimal choices with respect to such quality priorities. To represent the impact of solutions to qualities, goal models utilize contribution links, a special diagrammatic modeling construct. Researchers of goal modeling languages have introduced various ways to visualize the particular construct and to define formal semantics for it. However, there is little evidence that, during actual use, the proposed visualizations evoke a way of performing diagrammatic inferences that is consistent with the corresponding formal semantics. We conduct an experimental study aimed at comparing two visualization choices for contribution links, symbolic versus numeric, with respect to their ability to evoke inferences that are consistent with their formal semantics. The experiment also explores if individual psychological differences including trait cognitive style, mathematics anxiety, and mental math ability, affect this evocation. Participants are asked to make a series of diagrammatic inferences over two sets of goal models each adopting one of the two competing visualization formats, symbolic vs. numeric. We measure accuracy, that is, the level to which participant decisions are consistent with the formal semantics proposed for each visualization, and investigate the effect to accuracy of various relevant factors – visualization choice, individual differences, and reasoning method adopted. Findings include that most participants adopt specific inference rules instead of working intuitively, that such rules are more consistent with the formal semantics in numeric models, that the utilization of negative contributions and notions of goal denial may hinter accuracy, and that the individual differences considered do not play an important role in either accuracy or choice of inference method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability Statement
The datasets generated during and/or analyzed during the current study are available in the York University’s Dataverse repository: https://doi.org/10.5683/SP3/T38E48.
References
Abrahão S, Insfran E, de Guevara FGL, Fernández-Diego M, Cano-Genoves C, Pereira de Oliveira R (2019) Assessing the effectiveness of goal-oriented modeling languages: A family of experiments. Inf Softw Technol 116:106171. https://doi.org/10.1016/j.infsof.2019.08.003. https://www.sciencedirect.com/science/article/pii/S0950584919301673
Allinson CW, Hayes J (2012) The Cognitive Style Index: Technical Manual and User Guide. Tech rep. https://www.cognitivestyleindex.com/_files/ugd/832dc3_da009728f13415dbc4e08da a0943f8f.pdf
Allinson CW, Hayes J (1996) The Cognitive Style Index: A Measure of Intuition-Analysis For Organizational Research. J Manag Stud 33(1):119–135. https://doi.org/10.1111/j.1467-6486.1996.tb00801.x
Amazon Mechanical Turk (2022). https://www.mturk.com/
Amyot D, Mussbacher G (2011) User Requirements Notation: The First Ten Years, The Next Ten Years (Invited Paper). J Softw (JSW) 6(5):747–768
Amyot D, Ghanavati S, Horkoff J, Mussbacher G, Peyton L, Yu ESK (2010) Evaluating goal models within the goal-oriented requirement language. Int J Intell Syst 25(8):841–877
Armstrong SJ, Qi M (2020) The Influence of Leader-Follower Cognitive Style Similarity on Followers’ Organizational Citizenship Behaviors. Front Psychol 11. https://doi.org/10.3389/fpsyg.2020.01265
Armstrong SJ (2000) The Influence of Individual Cognitive Style on Performance in Management Education. Educ Psychol 20(3):323–339. https://doi.org/10.1080/014434100750018020
Asahi T, Turo D, Shneiderman B (1995) Using Treemaps to Visualize the Analytic Hierarchy Process. Inf Syst Res 6(4):357–375. https://doi.org/10.1287/isre.6.4.357
Ashcraft MH (2002) Math anxiety: Personal, educational, and cognitive consequences. Curr Dir Psychol Sci 130. https://doi.org/10.1111/1467-8721.00196
Ashcraft MH, Kirk EP (2001) The relationships among working memory, math anxiety, and performance. J Exp Psychol Gen. https://doi.org/10.1037//0096-3445.130.2.224
Blazhenkova O, Kozhevnikov M (2009) The new object-spatial-verbal cognitive style model: Theory and measurement. Appl Cogn Psychol 23:638–663
Bork D, Roelens B (2021) A technique for evaluating and improving the semantic transparency of modeling language notations. Softw Syst Model 20(4):939–963. https://doi.org/10.1007/s10270-021-00895-w
Brooke J (1995) SUS: A quick and dirty usability scale. Usability Evaluation In Industry 189
Caire P, Genon N, Heymans P, Moody DL (2013) Visual notation design 2.0: Towards user comprehensible requirements engineering notations. In: Proceedings of the 21st IEEE International Requirements Engineering Conference (RE’13), pp 115–124. https://doi.org/10.1109/RE.2013.6636711
Chandler D (2007) Semiotics: The Basics, 2nd edn. Routledge
Corbin J, Strauss A (2012) Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory. Sage Publications. https://doi.org/10.4135/9781452230153
Crump MJC, McDonnell JV, Gureckis TM (2013) Evaluating Amazon’s Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS ONE 8(3):1–18. https://doi.org/10.1371/journal.pone.0057410
Dalpiaz F, Franch X, Horkoff J (2016) iStar 2.0 Language Guide. The Computing Research Repository (CoRR) abs/1605.0 (2016). arXiv:1605.07767
Dardenne A, van Lamsweerde A, Fickas S (1993) Goal-Directed Requirements Acquisition. Sci Comput. Program 20:3–50
Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 13(3):319–340. http://www.jstor.org/stable/249008
Dimara E, Bezerianos A, Dragicevic P (2018) Conceptual and Methodological Issues in Evaluating Multidimensional Visualizations for Decision Support. IEEE Trans Vis Comput Graph 24(1):749–759. https://doi.org/10.1109/TVCG.2017.2745138
Engin A, Vetschera R (2017) Information representation in decision making: The impact of cognitive style and depletion effects. Decis Support Syst 103:94–103. https://doi.org/10.1016/j.dss.2017.09.007. https://www.sciencedirect.com/science/article/pii/S0167923617301744
Epstein S, Pacini R, Denes-Raj V, Heier H (1996) Individual Differences in Intuitive-Experiential and Analytical-Rational Thinking Styles. J Pers Soc Psychol 71:390–405. https://doi.org/10.1037/0022-3514.71.2.390
Evans C, Harkins MJ, Young JD (2008) Exploring teaching styles and cognitive styles: evidence from school teachers in Canada. N Am J Psychol 10:567
Figl K, Recker J (2016) Exploring cognitive style and task-specific preferences for process representations. Requir Eng 21:63–85
Genero M, Poels G, Piattini M (2008) Defining and validating metrics for assessing the understandability of entity-relationship diagrams. Data and Knowledge Engineering 64(3):534–557. https://doi.org/10.1016/j.datak.2007.09.011
Gettingera J, Kieslingb E, Stummerc C, Vetscherad R (2013) A comparison of representations for discrete multi-criteria decision problems. Decis Support Syst 54(2):976–985
Giorgini P, Mylopoulos J, Nicchiarelli E, Sebastiani R (2002) Reasoning with Goal Models. In: Proceedings of the 21st International Conference on Conceptual Modeling (ER’02), pp 167–181. London, UK
Giorgini P, Mylopoulos J, Nicchiarelli E, Sebastiani R (2003) Formal Reasoning Techniques for Goal Models. In: Spaccapietra S, March S, Aberer K (eds) Journal on Data Semantics I, pp 1–20. Springer Berlin Heidelberg, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-39733-5_1
Gonzales-Baixauli B, Leite JCSP, Mylopoulos J (2004) Visual variability analysis for goal models. In: Proceedings of the 12th IEEE International Requirements Engineering Conference (RE’04), pp 198–207. https://doi.org/10.1109/ICRE.2004.1335677
Guizzardi G (2005) Ontological Foundations for Structural Conceptual Models. Ph.D. thesis, University of Twente
Hadar I, Reinhartz-Berger I, Kuflik T, Perini A, Ricca F, Susi A (2013) Comparing the comprehensibility of requirements models expressed in Use Case and Tropos: Results from a family of experiments. Inf Softw Technol 55(10):1823–1843
Hammond KR, Hamm RM, Grassia J, Pearson T (1987) Direct comparison of the efficacy of intuitive and analytical cognition in expert judgment. IEEE Trans Syst Man Cybern 17(5):753–770. https://doi.org/10.1109/TSMC.1987.6499282
Hmieleski KM, Corbett AC (2006) Proclivity for improvisation as a predictor of entrepreneurial intentions. J Small Bus Manag. https://doi.org/10.1111/j.1540-627X.2006.00153.x
Hopko DR, Mahadevan R, Bare RL, Hunt MK (2003) The Abbreviated Math Anxiety Scale (AMAS): Construction, Validity, and Reliability. Assessment 10(2):178–182. https://doi.org/10.1177/1073191103010002008
Horkoff J (2006) Using i* Models for Evaluation. Master’s thesis, University of Toronoto
Horkoff J, Yu ESK (2016) Interactive goal model analysis for early requirements engineering. Requir Eng 21(1):29–61. https://doi.org/10.1007/s00766-014-0209-8
Horkoff J, Yu E (2011) Comparison and evaluation of goal-oriented satisfaction analysis techniques. Requir Eng (REJ) pp 1–24
Houy C, Fettke P, Loos P (2012) Understanding understandability of conceptual models - What are we actually talking about? In: Proceedings of the 31st International Conference on Conceptual Modeling (ER 2012), vol (LNCS 7532, pp 64–77
Huysmans J, Dejaeger K, Mues C, Vanthienen J, Baesens B (2011) An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models. Decis Support Syst. https://doi.org/10.1016/j.dss.2010.12.003
Jones DR, Schkade DA (1995) Choosing and translating between problem representations. Organ Behav Hum Decis Process 61(2). https://doi.org/10.1006/obhd.1995.1017
Jošt G, Huber J, Heričko M, Polančič G (2016) An empirical investigation of intuitive understandability of process diagrams. Computer Standards and Interfaces 48:90–111. https://doi.org/10.1016/j.csi.2016.04.006
Kelton AS, Pennington RR, Tuttle BM (2010) The effects of information presentation format on judgment and decision making: A review of the information systems research. J Inf Syst. https://doi.org/10.2308/jis.2010.24.2.79
Kieras DE, Bovair S (1984) The role of a mental model in learning to operate a device. Cogn Sci 8(3):255–273. https://doi.org/10.1016/S0364-0213(84)80003-8. https://www.sciencedirect.com/science/article/pii/S0364021384800038
Kirby JR, Moore PJ, Schofield NJ (1988) Verbal and visual learning styles. Contemp Educ Psychol. https://doi.org/10.1016/0361-476X(88)90017-3
Krogstie J (2012) Model-Based Development and Evolution of Information Systems. Springer
Krogstie J, Sindre G, Jørgensen H (2006) Process models representing knowledge for action: a revised quality framework. Eur J Inf Syst 15(1):91–102. https://doi.org/10.1057/palgrave.ejis.3000598
Letier E, van Lamsweerde A (2004) Reasoning about Partial Goal Satisfaction for Requirements and Design Engineering. In: Proceedings of the 12th International Symposium on the Foundation of Software Engineering FSE-04, pp 53–62. ACM Press, Newport Beach, CA. http://www2.info.ucl.ac.be/people/eletier/
Liaskos S (2022). Replication Data for: On the Intuitive Comprehensibility of Contribution Links in Goal Models: An experimental study. https://doi.org/10.5683/SP3/T38E48
Liaskos S, McIlraith S, Sohrabi S, Mylopoulos J (2011) Representing and reasoning about preferences in requirements engineering. Requirements Engineering Journal (REJ) 16:227–249
Liaskos S, Khan SM, Litoiu M, Jungblut MD, Rogozhkin V, Mylopoulos J (2012) Behavioral adaptation of information systems through goal models. Informations Systems (IS) 37(8):767–783
Liaskos S, Khan SM, Mylopoulos J (2022) Modeling and reasoning about uncertainty in goal models: a decision-theoretic approach. Software & Systems Modeling 21:1–24. https://doi.org/10.1007/s10270-021-00968-w
Liaskos S, Dundjerovic T, Gabriel G (2018) Comparing Alternative Goal Model Visualizations for Decision Making: an Exploratory Experiment. In: Proceedings of the 33rd Annual ACM Symposium on Applied Computing (SAC’18), pp 1272–1281. Pau, France. http://www.yorku.ca/liaskos/Papers/SAC2018/Visualizations/SAC2018.pdf
Liaskos S, Jalman R, Aranda J (2012) On Eliciting Preference and Contribution Measures in Goal Models. In: Proceedings of the 20th International Requirements Engineering Conference (RE’12), pp 221–230. Chicago, IL
Liaskos S, Jalman R, Aranda J, (2012) On Eliciting Preference and Influence Measure in Goal Models. Tech rep, School of IT, York University. http://www.yorku.ca/liaskos/Docs/AHPGoals.pdf
Liaskos S, Khan SM, Soutchanski M, Mylopoulos J (2013) Modeling and Reasoning with Decision-Theoretic Goals. In: Proceedings of the 32th International Conference on Conceptual Modeling, (ER’13), pp 19–32. Hong-Kong, China
Liaskos S, Lapouchnian A, Wang Y, Yu Y, Easterbrook S (2005) Configuring Common Personal Software: a Requirements-Driven Approach. In: Proceedings of the 13th IEEE International Requirements Engineering Conference (RE’05). IEEE Computer Society, Paris, France
Liaskos S, Litoiu M, Jungblut MD, Mylopoulos J (2011) Goal-based behavioral customization of information systems. In: Mouratidis H, Rolland C (eds) Proceedings of the 23rd International Conference on Advanced Information Systems Engineering (CAiSE’11). Springer Berlin Heidelberg, London, UK, pp 77–92
Liaskos S, Mylopoulos J, Khan SM (2021) Empirically Evaluating the Semantic Qualities of Language Vocabularies. In: Ghose AK, Horkoff J, Souza VES, Parsons J, Evermann J (eds) 40th International Conference on Conceptual Modeling (ER 2021), Lecture Notes in Computer Science, vol 13011. Springer, pp 330–344. https://doi.org/10.1007/978-3-030-89022-3_26
Liaskos S, Ronse A, Zhian M (2017) Assessing the Intuitiveness of Qualitative Contribution Relationships in Goal Models: an Exploratory Experiment. In: Proceedings of the 11th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM’17), pp 466–471. http://www.yorku.ca/liaskos/Docs/ESEM17.pdf
Liaskos S, Tambosi W (2019) Factors Affecting Comprehension of Contribution Links in Goal Models: An Experiment. In: Laender AHF, Pernici B, Lim EP, de Oliveira JPM (eds) Proceedings of the 38th International Conference on Conceptual Modeling (ER’19. Springer International Publishing, Cham, pp 525–539
Luo W (2019) User choice of interactive data visualization format: The effects of cognitive style and spatial ability. Decis Support Syst. https://doi.org/10.1016/j.dss.2019.05.001
Lurie NH, Mason CH (2007) Visual representation: Implications for decision making. J Mark. https://doi.org/10.1509/jmkg.71.1.160
Ma LC, Li HL (2011) Using Gower Plots and Decision Balls to rank alternatives involving inconsistent preferences. Decis Support Syst. https://doi.org/10.1016/j.dss.2011.04.004
Maiden NAM, Pavan P, Gizikis A, Clause O, Kim H, Zhu X (2002) Making Decisions with Requirements: Integrating i* Goal Modelling and the AHP. In: Proceedings of the 8th International Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ’02). Essen, Germany
Maxwell SE, Delaney HD (2004) Designing Experiments and Analyzing Data, 2nd edn. Taylor and Francis Group, LLC, New York, MA, USA
Miettinen K (2014) Survey of methods to visualize alternatives in multiple criteria decision making problems. OR Spectrum 36(1):3–37. https://doi.org/10.1007/s00291-012-0297-0
Moody DL (2009) The “Physics" of Notations: Toward a Scientific Basis for Constructing Visual Notations in Software Engineering. IEEE Trans Softw Eng 35(6):756–779. https://doi.org/10.1109/TSE.2009.67
Morales JM, Navarro E, Sánchez P, Alonso D (2015) A controlled experiment to evaluate the under-standability of kaos and i* for modeling teleo-reactive systems. J Syst Softw 100:1–14. https://doi.org/10.1016/j.jss.2014.10.010. https://www.sciencedirect.com/science/article/pii/S0164121214002143
Morales JM, Navarro E, Sánchez P, Alonso D (2016) A family of experiments to evaluate the understandability of tristar and i* for modeling teleo-reactive systems. J Syst Softw 114:82–100. https://doi.org/10.1016/j.jss.2015.12.056. https://www.sciencedirect.com/science/article/pii/S0164121216000042
Mylopoulos J, Chung L, Liao S, Wang H, Yu E (2001) Exploring Alternatives During Requirements Analysis. IEEE Softw 18(1):92–96. https://doi.org/10.1109/52.903174
Norman DA (1983) Some Observations on Mental Models. In: Mental Models, pp 7–14. Psychology Press
Pajer S, Streit M, Torsney-Weir T, Spechtenhauser F, Möller T, Piringer H (2017) WeightLifter: Visual Weight Space Exploration for Multi-Criteria Decision Making. IEEE Trans Vis Comput Graph. https://doi.org/10.1109/TVCG.2016.2598589
Payne SJ (1991) A descriptive study of mental models. Behaviour & Information Technology 10(1):3–21. https://doi.org/10.1080/01449299108924268
Pracht WE (1990) Model visualization: Graphical support for DSS problem structuring and knowledge organization. Decis Support Syst. https://doi.org/10.1016/0167-9236(90)90011-F
Preece YR, Sharp H, Jennifer (2011) Interaction Design: beyond human-computer interaction. Wiley
Richardson A (1977) Verbalizer-visualizer: a cognitive style dimension Journal of Mental Imagery, 1, pp. 109–126
Rosnow RL, Rosenthal R (2008) Beginning Behavioral Research: A Conceptual Primer, 6th edn. Pearson Prentice Hall, NJ, USA
Saaty TL (2008) Decision making with the analytic hierarchy process. International Journal of Services Sciences (IJSSCI) 1(1):83–98
Santos A, Gómez O, Juristo N (2020) Analyzing families of experiments in SE: a systematic mapping study. IEEE Trans Softw Eng 46(5):566–583. https://doi.org/10.1109/TSE.2018.2864633
Santos M, Gralha C, Goulão, M, Araújo J, Moreira A, Cambeiro Ja (2016) What is the impact of bad layout in the understandability of social goal models? In: 2016 IEEE 24th International Requirements Engineering Conference (RE), pp 206–215 (2016). https://doi.org/10.1109/RE.2016.51
Santos M, Gralha C, Goulão M, Araújo J (2018) Increasing the Semantic Transparency of the KAOS Goal Model Concrete Syntax. In: Trujillo JC, Davis KC, Du X, Li Z, Ling TW, Li G, Lee ML (eds) Proceedings of the 37th International Conference on Conceptual Modeling (ER’18). Springer International Publishing, Cham, pp 424–439
Santos M, Gralha C, Goulão M, Araújo J, Moreira A (2018) On the Impact of Semantic Transparency on Understanding and Reviewing Social Goal Models. In: 2018 IEEE 26th International Requirements Engineering Conference (RE), pp 228–239. https://doi.org/10.1109/RE.2018.00031
Schweiger DM (1983) Is the simultaneous verbal protocol a viable method for studying managerial problem solving and decision making? Acad Manag J 26(1):185–192. http://www.jstor.org/stable/256146
Speier C (2006) The influence of information presentation formats on complex task decision-making performance. Int J Hum Comput Stud. https://doi.org/10.1016/j.ijhcs.2006.06.007
Steinle F (1997) Entering new fields: Exploratory uses of experimentation. Philosophy of Science 64:S65–S74. http://www.jstor.org/stable/188390
Stoet G (2010) PsyToolkit: A software package for programming psychological experiments using Linux. Behav Res Methods 42(4):1096–1104. https://doi.org/10.3758/BRM.42.4.1096
Stoet G (2017) PsyToolkit: A Novel Web-Based Method for Running Online Questionnaires and Reaction Time Experiments. Teach Psychol 44(1):24–31. https://doi.org/10.1177/0098628316677643
Stone DN, Schkade DA (1991) Numeric and linguistic information representation in multiattribute choice. Organ Behav Hum Decis Process. https://doi.org/10.1016/0749-5978(91)90041-Q
Teruel MA, Navarro E, López-Jaquero V, Montero F, Jaen J, González P (2012) Analyzing the understandability of requirements engineering languages for CSCW systems: A family of experiments. Inf Softw Technol 54(11):1215–1228. https://doi.org/10.1016/j.infsof.2012.06.001. https://www.sciencedirect.com/science/article/pii/S0950584912001036
Türetken O, Vanderfeesten I, Claes J (2017) Cognitive Style and Business Process Model Understanding. In: Metzger A, Persson A (eds) Advanced Information Systems Engineering Workshops. Springer International Publishing, Cham, pp 72–84
Umanath NS, Vessey I (1994) Multiattribute Data Presentation and Human Judgment: A Cognitive Fit Perspective. Decis Sci 25(5–6):795–824. https://doi.org/10.1111/j.1540-5915.1994.tb01870.x
Vance CM, Groves KS, Paik Y, Kindler H (2007) Understanding and measuring linear-nonlinear thinking style for enhanced management education and professional practice. Acad Manag Learn Educ 6:167–185. https://doi.org/10.5465/AMLE.2007.25223457
Vessey I (1991) Cognitive Fit: A Theory-Based Analysis of the Graphs Versus Tables Literature. Decis Sci. https://doi.org/10.1111/j.1540-5915.1991.tb00344.x
Young RM (1983) Surrogates and Mappings: Two Kinds of Conceptual Models for Interactive Devices. In: Mental Models. Psychology Press, pp 35–52
Yu ES (2000) GRL - Goal-oriented Requirement Language. http://www.cs.toronto.edu/km/GRL/
Yu ESK (1997) Towards Modelling and Reasoning Support for Early-Phase Requirements Engineering. In: Proceedings of the 3rd IEEE International Symposium on Requirements Engineering (RE’97). Annapolis, MD, pp 226–235
Yu E, Giorgini P, NM, Mylopoulos J (2010) Social Modeling for Requirements Engineering. MIT Press
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Compliance with Ethical Standards
Data collection has received ethics review and approval by the Human Participants Review Sub-Committee, York University’s Ethics Review Board (certificate # e2018 - 024 and subsequent renewals) and conforms to the standards of the Canadian Tri-Council Research Ethics guidelines.
Conflict of Interest
The authors declared that they have no conflict of interest.
Additional information
Communicated by: Irit Hadar.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Liaskos, S. On the intuitive comprehensibility of contribution links in goal models: an experimental study. Empir Software Eng 29, 26 (2024). https://doi.org/10.1007/s10664-023-10376-x
Accepted:
Published:
DOI: https://doi.org/10.1007/s10664-023-10376-x