Studying Conceptual Modeling Processes: A Modeling Tool, Research Observatory, and Multimodal Observation Setup

What do (non-)experienced modelers reason while conceptual modeling and how do they arrive at modeling decisions, which modeling and learning difﬁcul-ties do they face and why, and how do they overcome these difﬁculties by tailored modeling tool support are questions of relevance and importance to practicing modelers and, likewise, to conceptual modeling research. For the past 7 years, we have been designing, developing, and evaluating a modeling tool integrating a research observatory aimed at studying individual modeling processes online, in the ﬁeld, and under laboratory conditions—to contribute to a richer understandingof modeler reasoning and decision-making, to identify common modeling and learning difﬁculties, and, ultimately, to design tool support to mitigate difﬁculties and to improve assistance for (non-)experienced modelers. We present an overview of the modeling observatory and of a corresponding multimodal observation setup.

Conceptual modeling-as an activity, e.g., when constructing a data model as Entity-Relationship diagram (Chen 1976)-involves an intricate array of cognitive processes and performed actions, including goal setting, abstracting, conceptualizing, associating and contextualizing, interpreting and sense-making, evaluating and judging, anticipating and envisioning and thinking ahead, drawing and visualizing, and, in group settings, communicating, discussing, and agreeing. The activity of conceptual modeling is, therefore, at one point in time, tool-mediated and based on interacting with modeling tools supporting conceptual modelers, e.g., by way of graphical modeling editors. Performing conceptual modeling is construed as a complex task based on codified and tacit knowledge, a task that requires mastering theoretical foundations, modeling languages (such as Entity-Relationship Model, Chen 1976), modeling methods, and modeling tools, applying them to practical problems, and, while performing a modeling process, critically thinking and reflecting upon an application domain in terms of the technical languages spoken in the targeted domain-the domain's technical terminology and its use in domain-specific technical languages, their imprecision, ambiguities, and related challenges-with the intention to (re-)construct the selected aspects of organizational reality and to reshape organizational reality by (re-)presenting a new perspective and reworked terminology and language embodied and conveyed by conceptual models (Frank et al. 2014).
The process (the actual "act") of conceptual modeling has for long received limited attention in conceptual modeling research (Hoppenbrouwers et al. 2005;2006) but has recently seen increasing interest from researchers (e.g., Bera et al. 2019, Claes et al. 2015, Serral et al. 2016, Wilmont et al. 2017. How conceptual modeling is performed by modelers, how modeling processes proceed, which modeling challenges and difficulties modelers experience and why, and how to overcome these difficulties by tailored modeling support have been subject to studies on the cognitive processes and performed actions constituting conceptual modeling (e.g., Batra and Davis 1992, Bera 2011, Chaiyasut and Shanks 1994, Srinivasan and Te'eni 1995, Venable 1996. Still, further research on conceptual modeling processes is needed to understand the reasoning of modelers and their deliberations (e.g., about modeling decisions) and whether different (idealized) types of modelers can be identified, e.g., by determining the patterns of modeling processes and/or difficulties, and whether these modeler types benefit from modeling tool support tailored to overcome their difficulties. Similarly, further research is required to study how individuals learn conceptual modeling, how their learning progresses, and what their needs are for learning support and tool assistance . Studying progressively more and more individual modeling processes under varying conditions, i.e., online and in the field and the laboratory, promises to shed further light on these open questions and, ultimately, to enable us to design targeted (tool) support for modelers at different stages of their learning and mastering of conceptual modeling.
Against this background, we pursue a research program to contribute to a richer and more complete understanding of individual modeling processes and of the learning of conceptual modeling-aiming for the overarching research objective of designing, implementing, and evaluating targeted tool support for modelers at different stages of their learning and mastering of conceptual modeling. In our pursuit of this research objective, we operate on the basic assumption that modelers' individual modeling processes demand a study from multiple complementary perspectives to account for the richness and complexity of conceptual modeling as a learning and performing task-following Berger and Luckmann's inspiring insight that "the object of thought becomes progressively clearer with this accumulation of different perspectives on it" (Berger and Luckmann 1967, p. 10). Ultimately, the research program aims to progressively add to the empirical and theoretical foundation of conceptual modeling research, e.g., by identifying modeling difficulties and by developing a taxonomic theory of such difficulties as a step toward advancing the body of knowledge to offer guidance for designing and implementing targeted tailored (tool) support for conceptual modelers. As part of this research program, we configure and validate a multimodal observation setup and corresponding analysis procedure that we realize by TOOL, a web browser-based modeling tool and research observatory implementing an array of observation and analysis features.
The subsequent Sect. 2 introduces the multimodal observation setup. Section 3 demonstrates how TOOL supports the observation setup, and Sect. 4 reports on a series of laboratory observations on individual modeling processes and concludes with lessons learned.

Multimodal Observation Setup
Observing conceptual modeling processes poses methodological challenges. For example, neither the reasoning of modelers nor the corresponding deliberations on modeling decisions are immediately accessible nor directly observable. Research on conceptual modeling processes has reverted to observable aspects of modeling processes such as modelers' interactions with software tools, modelers' eye movements, or their verbalizations of their own thought processes while modeling-as verbal protocols following the "think-aloud" method (Ericsson andSimon 1993, van Someren et al. 1994). Following this trajectory, we compile and arrange multiple complementary modes of observation for a multimodal observation setup (see Fig. 1). In particular, we combine four modes of observation: 1. Audio-taping think-aloud (verbal) protocols during conceptual modeling by a subject, i.e., while working on a modeling task, to obtain insights into the subject's cognitive processes during modeling, e.g., into the modeler's reasoning and deliberations toward modeling decision and also into difficulties the modeler encounters and expresses verbally 2. Tracking modeler interactions with the modeling canvas to observe specific modeling decisions, in particular, decisions with respect to placing a new model element on the modeling canvas (e.g., a rectangular graphical symbol representing an entity type, say "Customer," in an entity-relationship diagram), to  change an existing model element, to element repositioning, to deletion of model elements, and to renaming a model element (giving it a new label) 3. Surveying subjects pre-and post-modeling to collect information on modeler demographics and to obtain self-disclosed information on modeling experience, perceived modeling difficulties, and tool usability, i.e., a post-modeling survey collects information about difficulties with using the tool's graphical editor, tool support, and overall user experience to evaluate TOOL as a design artifact 4. Videotaping subjects (e.g., from an "over-the-shoulder" perspective) to observe nonverbal cues on the individual modeling process, in particular on modeling difficulties, conveyed by body language and movement, e.g., when switching between media from the computer screen to paper and back or when interrupting the modeling flow as indicated by gestures The main rationale for combining these modes of observation lies in their complementarity : The configured setup combines observation by think-aloud protocols and video recordings with modeler-tool interaction tracking and survey data to observe a wide range of facets about individual modeling processes including identifying modeling and learning difficulties expressed verbally by the modeler during and after modeling as well as nonverbally indicated through interactions with the modeling tool and its modeling canvas as well as by body language and movements. Specifically, analyzing think-aloud protocols has shown promising results for understanding the cognitive processes of subjects working on problem-solving tasks in general and on modeling tasks in particular (e.g., Batra andDavis 1992, Srinivasan andTe'eni 1995), and we consider verbalization of thoughts as the best available means of expression for achieving insights into modelers' reasoning as our spoken language provides a rich and flexible tool to express our thinking. However, to ask subjects to think aloud is a second-best approach, warranted only because it is not possible to directly access and capture cognitive processes and, thus, modeler reasoning while modeling. Modelers may have difficulties verbalizing their reasoning while modeling (Blech et al. 2019) on principle accounts (because verbalizing one's own thoughts can be difficult) or on modeling-related accounts (e.g., because of the difficulty of finding the right words to express oneself). Nonetheless, among all possible alternative modes of observation, think-aloud verbalization promises the richest insight into quintessential non-directly observable cognitive processes of an individual modeling process.
Modelers' reasoning, their chain of thought, and line of modeling arguments will, still, not always be observable from think-aloud protocols alone, since nonverbal information such as modelers' movements and gestures entail important additional cues about their perception of, for instance, modeling challenges and difficulties. Hence, the configured observation setup complements think-aloud protocols with videotaping modelers to allow for additional visual clues, e.g., regarding interaction with pen and paper or modelers' (e.g., erratic) movements during modeling plus modeler-tool interactions. Additionally, subjects are surveyed about their individual modeling process before and after they work on a (controlled) modeling task. Following Berger and Luckmann, this observation setup combines multiple perspectives on modeling processes and supports mixed methods research designs in which open (narratives, verbal protocols, video recordings) and closed (more) standardized (tracking data, survey data) modes of observation are combined to obtain a more complete picture of the phenomenon under investigation (Creswell and Plano Clark 2018).

Modeling Tool and Research Observatory
The multimodal observation setup is realized based on the modeling tool TOOL integrating a research observatory designed for observing and analyzing individual modeling processes (Ternes 2017, Ternes and Strecker 2018. Resulting from a multi-year design science research project, the TOOL research prototype currently implements two graphical modeling editors (see Fig. 2): (1) an editor for a didactically adapted, simplified variant of the Entity-Relationship model (ERM) for data modeling and (2) an editor for a subset of the Business Process Model and Notation (BPMN 2) for business process modeling (Frank and Strecker 2021). For studying individual modeling processes under laboratory conditions, in the field, and in online settings, TOOL integrates a research observatory adapted to the multimodal observation setup (Ternes et al. 2020a;c). The modeling observatory presently features five modes of observation (as in-browser features): 1. Recording of tink-aloud (verbal) protocols via computer audio input (in addition to external audio recording used in laboratory settings; see Fig. 1) 2. Recording of the subject's computer screen to provide an additional video recording of the modeling process 3. Timed-discrete event tracking and recording of modeler interactions with the tool's modeling canvas 4. Tracking and recording of mouse pointer movement on the modeling canvas 5. Conducting and recording on-screen pre-and post-modeling surveys of subjects (as an alternative to printed surveys) The observatory also features data visualizations and analysis techniques for interpreting the recorded data: (a) Replay of the recorded modeler-tool interactions of a modeling process in real time or step by step, either as single replay or in comparison of up to four modeling processes shown concurrently over normalized time (see Fig. 3a): The step-by-step replay visually shows every step of the model construction, whereas the automatic replay shows model construction in real time, both allowing for additional visual inspection of the modeling process. (b) Replay with heat map overlays on mouse pointer position, dwell time, and mouse clicks (see Fig. 3b): Such heat maps allow insights into spatial areas of the modeling canvas in focus and into difficulties experienced with the modeling tool, e.g., indicated by uncontrolled mouse clicking. (c) Dot diagrams (see Fig. 3c), a data visualization inspired by PPMCharts (Claes et al. 2015) and dotted charts (Song and van der Aalst 2007): The vertical axis indicates the consecutively numbered model elements that are created (green circle), changed (blue circle), repositioned (gray), deleted (red circle), or relabeled (orange circle). The dot diagram visually differentiates the mentioned five types of specific modeling decisions (introduce new model element on canvas, change existing element, reposition an element, delete an element, relabel an element) and allows to clarify audio/video and tracking observations in cases when the observed modeling behavior is ambiguous. (d) Bar charts and related visualizations of survey data prepare for survey analysis, e.g., by visually representing results from closed-ended questions (see Fig. 3d). The visualizations and techniques (a-d) are used to further explore the observed situations identified as deviant or unclear in the audio and video protocol and to identify and clarify anomalous modeling behavior by manual inspection. Additionally, the research observatory is adaptable to different research designs by selecting and combining modes of observation (1-5), by tailoring surveys, and by sequencing a corresponding observation workflow. A video demonstration of the research observatory is available at https://vimeo.com/441854796/5237d3782a. From the outset run as a design science research project, two essential requirements drive the development of the TOOL research prototype: (1) platform independence and (2) ease of use and usability to the greatest extent economically viable. Further design considerations, requirements, and operating principles are outlined in Ternes (2017), Ternes and Strecker (2018), and Ternes et al. (2019;c). In a nutshell, TOOL is designed and implemented as a web application with a JavaScript-driven web browser frontend and a Java EE (Enterprise Edition)-based server backend and, as a design artifact, also serves as a research laboratory for studying (web technology) software stacks and toolchains (see Fig. 4). The current working prototype sizes up to 55,000 lines of code (LoC) in the frontend and about 32,000 LoC in the server backend. TOOL has been applied to an initial test series, to pilot studies, as well as in a series of research studies on individual modeling processes at the University of Hagen, Germany; the Universitat Politècnica de València, Spain; and the Katholieke Universiteit Leuven, Belgium (see next section). Moreover, TOOL has supplemented our teaching of an undergraduate course on modeling business information systems where students use it for their assignments.

Data Integration, Data Analysis, and Lessons Learned
A series of laboratory studies validates the multimodal observation setup in combination with the research observatory and modeling tool: For example, we have been studying individual modeling processes of experienced and non-experienced modelers to identify the modeling difficulties these modelers face while performing a data modeling task Strecker 2019, Rosenthal et al. 2020b). In a nutshell, subjects are provided with a natural language description of a data modeling task, receive instructions and an introduction to the graphical editor, and are then asked to think out loud while performing the modeling task. To achieve insights into the observed individual modeling processes, the collected data is purposefully integrated and analyzed to identify modeling difficulties and to understand modelers' reasoning: Each verbal think-aloud protocol is linked and time-synchronized to the video recording to produce an audiovisual protocol comprising the data from both modes of observation. To add structure to the data, we code these videos by systematically assigning codes to video segments/clips (following, e.g., Miles et al. 2014, pp. 81f.) using a qualitative data analysis software (see Fig. 5 for an example from a pre-test). Directly coding the audiovisual protocols rather than transcribing the verbalizations allows to benefit from the complementary angles provided by the respective mode of observation. Following problem-solving research (Newell and Simon 1972), we use the concept of cognitive breakdowns (e.g., Bera 2011) to identify modeling difficulties which modelers experience while constructing a conceptual model and, hence, use cognitive breakdown as deductive code. We define a cognitive breakdown as a cognitive difficulty which a modeler experiences when constructing a conceptual model based on a natural language description (Bera 2011, p. 4)-"when a line of thought fails" (Burton-Jones and Meso 2008, p. 768). Such a cognitive breakdown can manifest itself in a modeler explicitly verbalizing a difficulty while modeling or in interrupting or terminating a modeling activity, e.g., a modeling activity which is not completed, but instead the modeler switches to another activity (Bera 2011, p. 4). Segments in which a subject encounters a difficulty or an obstacle are initially marked with the code "cognitive breakdown," i.e., when the subject explicitly verbalizes a difficulty experienced during modeling or when the subject interrupts or terminates a modeling activity. This code is complemented with codes generally anticipated in think-aloud protocols as, e.g., talking about non-task-related issues, evaluation of the task at a meta-level, silent periods, and actions outside of the modeling tool (following, e.g., van Someren et al. 1994, p. 122). During coding, the coding scheme is complemented with emerging codes and sub-codes-allowing for refinements according to the actual behavior exhibited in the modeling processes. We supplement coding audiovisual protocols with analyzing timed modeler-tool interactions allowing us to identify peculiar situations. Segments identified as unclear in the videos are submitted for closer inspection by analyzing the recorded modeler-tool interactions in the respective time period to better understand the observed situation and to decide on assigning a code: Dot diagrams visualizing modeling processes and replays of model construction are used for further exploring situations identified as ambiguous or deviant in the audiovisual protocols. Vice versa, anomalous data in the recorded modeler-tool interactions is identified and further investigated through analyzing audiovisual protocols. Data integration is taken one step further by reviewing the pre-and post-modeling surveys and, thus, by supplementing another mode of observation (self-disclosed by the observed modeler). This coding step proved valuable especially as the individually perceived difficulties serve as indication for closer inspecting and deciding on assigning a code in the audiovisual protocols. We provide a more detailed description of the data analysis strategy and coding steps in Rosenthal and Strecker (2019, pp. 8-10).
Observations on individual modeling processes we performed so far led us to identify ten types of modeling difficulties the observed subjects faced while performing the data modeling task: The types of difficulties relate to different aspects of constructing conceptual data models, i.e., entity types, relationship types, attributes, and cardinalities. The majority of the difficulties encountered by the participants relate to modeling relationship types, i.e., deciding between modeling an entity type or a relationship type, developing sensible identifiers for relationship types, and determining cardinalities Strecker 2019, Rosenthal et al. 2020a). To overcome these modeling difficulties when formulating sensible identifiers (Breuker et al. 2016), we presently research and develop extensions to TOOL for automated modeler assistance based on natural language processing (NLP) (Manning et al. 2014) to provide tool-generated suggestions on labeling model elements at modeling time. To the best of our knowledge, this design science research is among the first to integrate a web-based data modeling tool with NLP to automatically process an arbitrary natural language description of a modeling task (in terms of its morphological structure) to identify words and phrases as suggestions for labels for model elements .
Conducting studies with experienced and non-experienced modelers demonstrates that the multimodal observation setup in combination with the research observatory opens entirely new paths for insights into conceptual modeling processes. Analyzing the collected data exemplifies that only the integration of the complementary observation modes allowed us to identify a wide range of modeling difficulties by identifying cognitive breakdowns and also to reason about the cause and type of difficulty observed: Supplementing the analysis of audiovisual protocols with analyzing tracked modeler-tool interactions enabled us to better understand ambiguous situations and to identify further deviant interactions, while reviewing the post-modeling survey proved especially valuable as the perceived difficulties served as indication for closer inspection of audiovisual protocols. Hence, our current findings suggest that combining multiple modes of observation to study individual modeling processes contributes to achieving a richer and more complete understanding of modeling processes and modeling difficulties encountered during model construction. Informed by these initial insights, we are preparing for further small-scale and large-scale studies aimed at deepening and substantiating our understanding of modeling difficulties and identifying patterns of modeling processes.
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.