Considerations about quality in model-driven engineering

Abstract

The virtue of quality is not itself a subject; it depends on a subject. In the software engineering field, quality means good software products that meet customer expectations, constraints, and requirements. Despite the numerous approaches, methods, descriptive models, and tools, that have been developed, a level of consensus has been reached by software practitioners. However, in the model-driven engineering (MDE) field, which has emerged from software engineering paradigms, quality continues to be a great challenge since the subject is not fully defined. The use of models alone is not enough to manage all of the quality issues at the modeling language level. In this work, we present the current state and some relevant considerations regarding quality in MDE, by identifying current categories in quality conception and by highlighting quality issues in real applications of the model-driven initiatives. We identified 16 categories in the definition of quality in MDE. From this identification, by applying an adaptive sampling approach, we discovered the five most influential authors for the works that propose definitions of quality. These include (in order): the OMG standards (e.g., MDA, UML, MOF, OCL, SysML), the ISO standards for software quality models (e.g., 9126 and 25,000), Krogstie, Lindland, and Moody. We also discovered families of works about quality, i.e., works that belong to the same author or topic. Seventy-three works were found with evidence of the mismatch between the academic/research field of quality evaluation of modeling languages and actual MDE practice in industry. We demonstrate that this field does not currently solve quality issues reported in industrial scenarios. The evidence of the mismatch was grouped in eight categories, four for academic/research evidence and four for industrial reports. These categories were detected based on the scope proposed in each one of the academic/research works and from the questions and issues raised by real practitioners. We then proposed a scenario to illustrate quality issues in a real information system project in which multiple modeling languages were used. For the evaluation of the quality of this MDE scenario, we chose one of the most cited and influential quality frameworks; it was detected from the information obtained in the identification of the categories about quality definition for MDE. We demonstrated that the selected framework falls short in addressing the quality issues. Finally, based on the findings, we derive eight challenges for quality evaluation in MDE projects that current quality initiatives do not address sufficiently.

Appendix A: A multiple modeling languages quality scenario

The following scenario is based on a real project from the University of Quindío (Colombia); the implementation of an information system for institutional academic quality management. This system includes all the resources, processes, technology platforms, and legal frameworks required to achieve the institutional quality accreditation certification, which is awarded by the Ministry of Education in Colombia to universities that demonstrate excellence in the exercise of their academic and research activities. The accreditation certificate is the result of an internal assessment process that was executed by members interested in the university.

With this modeling scenario, we show how current quality proposals do not integrally cover some relevant issues in MDE projects. This modeling scenario helps to identify the applicability of some of the quality works on MDE identified in Sections 2.4 and 2.5 and emerging quality issues (Section 3.4) as a consequence of using modeling languages in the development of an information system.

A quality evaluation proposal that comes from one of the primary authors identified in Section 2.5 was used to analyze this modeling context (the physics of notations proposed in Moody 2009). Even though the quality proposal meets its primary purposes in the analysis of the models and modeling languages involved, other quality issues emerge but they were not covered by the proposal. These issues influence the adoption of a model-driven initiative to manage concerns in information systems.

This information system is characterized by:

• The presence of multiple academic/administrative stakeholders from different areas of knowledge, participating collaboratively in the development of strategies for the generation/management of evidence according to the descriptive models of quality required, and the monitoring of the multiple sub-processes of quality instantiated in the university.

• The alignment with quality descriptive models that define the quality criteria. These include the self-evaluation guides issued by the National Accreditation Council (CNA)¹² under the Ministry of Education of Colombia, as well the ISO 9001–2015 standard and the Colombian technical standard NTCGP 1000: 2009. The NTCGP 1000: 2009 is a management standard directed towards the evaluation of an institution’ performance in terms of quality and social satisfaction during the delivery of services by government entities.

• The development of an organizational culture that is oriented toward the continuous improvement management of the university in the business processes. The support of this goal is the integrated management system,¹³ which is a web platform where the specification of processes, procedures, and associated institutional formats is published. The related application scenario is framed within the business process called self-assessment for the accreditation and re-accreditation of an undergraduate or graduate program.

A strategy for the collaboration between academic experts and researchers in information systems was developed for the design, construction, and deployment of the information system. Its purpose is to formulate conceptual, methodological, and technological tools that support the processes of accreditation and assurance of quality. Each group used modeling languages to represent the phenomena of interest. The panel of experts in quality specified a model for academic quality process¹⁴ using a specific variation of the Flowchart diagram (a notation selected by those responsible for the integrated management system of the University Quindío to model the processes of the organization). The group of researchers in information systems employed the proper languages of software modeling and data to conceptually support the design and implementation of software platforms for different parts of the accreditation process. The use of different modeling languages for the process of design and construction of the academic quality system favors the process specification through the contributions of the parties involved (views). Three types of models were used in the conceptual modelling of the project:

• Business process models: This part of the application design focuses on the modeling of the processes undertaken at the University of Quindío, which are oriented toward business experts and the people who interact with the processes at the university. These models are intented for users of the processes that have no prior knowledge in order to facilitate the understanding of the processes.

• Business and system models: These models focus on the design and subsequent implementation of derived software applications to support the information system of academic quality, where everything that a software system needs to fulfill customer requirements must be specified. UML models are employed using class, sequence, use case, state, and components diagrams. In addition, a proposal of stereotyped UML formulated by RUP (Kruchten 2000) is used to model business processes by applying the business modeling discipline defined in this methodological framework. Researchers with different profiles made these models: experts in accreditation and academic quality processes, experts in software engineering, senior/advanced software developers, and data experts. A model-based approach is used to produce the source code of the applications from the models made by the researchers.

• Data model: These models cover the design of the database required for the academic quality system using the core business concepts identified in the domain model made in UML (the class diagram with the most representative concepts of the business according to the business modelling discipline of the RUP). This type of design depends on the expert in data or database administrator (DBA) because of the complexity that data modeling can have.

The complexity that is inherent in the development of the academic quality system and the parties involved is the rationale for using multiple modeling languages to help fulfill the interests of each role that is in charge of the implementation of the information system at the University of Quindío. These modeling languages include:

• Flowchart: the language used for making the process flow diagrams.

• UML: the language used for the analysis and design of software.

• E/R: Models used for verifying the design of the database.

1.1 A.1 Application of multiple models

Figure 10 shows a partial view of the self-assessment process for accreditation and re-accreditation purposes of an undergraduate or graduate program. Figure 11 presents the adaptation of the flow diagram notation used in the specification of business processes for the University of Quindío. This view corresponds to the participation of the experts in the business information system and in the assurance of academic quality processes.

Fig. 10

Current self-assessment process diagram for the University of Quindío (partial view)

Fig. 11

The conventions used for the flowchart adaptation at the University of Quindío

The modeling of business processes is done by using a notation that is particularly suited for experts in institution processes. This notation prioritizes simplicity and a small number of notational constructs to represent the process components accurately. None of the quality standards used for the implementation of quality policies (ISO 9001, NTC GP 1000, CNA) requires a specific graphic language; instead, these standards grant freedom for the modeling processes to be performed autonomously at the discretion of the organization.

Figures 12, 13, 14, 15 and 16 present the conceptual models that are formulated by researchers and experts in information systems (mostly in UML) to address the various considerations associated with academic quality and the derived software platforms (publication of information related to academic quality processes, document management framed in quality contexts, document distribution of quality processes supports, and management of activities).

Fig. 12

Business modeling models (I)

Fig. 13

Business modeling models (II)

Fig. 14

System models (I)

Fig. 15

System data model (partial view)

Fig. 16

Diagram example for the rationale of an architecture decision

Due to the methodological alignment with RUP, a UML profile is used for business modelling. Then, the researchers formulate system models. The following models belong to the module of Memoranda Management System within the Context of the Information System of Institutional Accreditation.¹⁵

1.1.1 A.1.1 Business modeling models

In order to understand the organization (i.e., detect current problems, identify improvement potential, identify users, workers, parties, etc), several stereotyped UML models were employed following the RUP methodological framework (Figs. 12 and 13). Figure 12a shows the model of business use cases. This model illustrates the organization by management process areas of the university. Related business processes are identified as use cases (in light blue). For purposes of readability, they are grouped using standard UML packages. The business use case is a modeling of each business goal and its respective roles. It is used to identify the roles and different deliverables of the works performed.

The model of business use case also contains the business use cases realization (Fig. 12b) as part of the business analysis model defined in RUP. A realization of a business use case describes how the workflow is in terms of the business objects and their collaboration. A diagram of activities and a diagram of business objects are defined in the realization of a business use case.

The business process model (Fig. 12c) is a set of logically related tasks that are carried out to generate products and services. A stereotyped UML activity diagram represents this model, where the business entities that are involved in the process tasks are also identified.

The business modeling discipline of RUP considers all the things or something of value that are observable during the performing of business processes. For this, researchers used the models shown in Fig. 13. The business entity model (Fig. 13a) represents an important part of the information that is handled by business actors and business workers. The business object model (Fig. 13b) shows the relationship between the business entities associated with different business use cases and the workers associated to those cases. The model serves to show the limits of the business process considered in each business use case.

Finally, a state machine model is used to define the life cycle of the information entities at the University of Quindío. Each state considers a set of specific software features to manage the state associated with an entity at any time during the execution of the process. Figure 13c shows a sample life cycle for a communication in the context of academic quality.

1.1.2 A.1.2 System models

Once the definition of business processes has been completed, use cases are derived at the software system level by a relationship of traceability whose origin is found in automatable activities of the business process analyzed.

Figure 14a partially shows the features that are implemented for the module of memoranda management software of the information system for academic quality. Models of system classes (Fig. 14b) generate the associated source code (logical view of the application) and sequence diagrams (functional allocation of responsibilities among objects) of Fig. 14c. These diagrams (along with their associated specification) are delivered to the project developers who generate the source code in the platforms and development environments that are defined by the technical experts.

Other non-UML systems models were used to conceive and manage specific system views of the Information System of Institutional Accreditation. Figure 15 shows the Data model in the E/R notation. Due to the relational support used in the technological implementation of the modules associated with the quality system, a conceptual representation of the entities associated to the domain addressed by each module is made. This conceptual representation defines the semantics associated with the entities, the consistency constraints at the data level in order to preserve the integrity of the module once it deploys organizationally.

Additionally, as part of the process of architectural decision-making for developing software modules, models elaborated in informal notations are used to address problems associated with specific quality attributes and to facilitate the identification of architectural tactics in the management of these attributes. Figure 16 shows an example of a diagram that was developed to discuss the aspects of global integration and the consistency of the information system (taking into account the presence of multiple software modules). The aim of these diagrams is to facilitate the description of architectural alternatives in the consultation and judgment processes so that the consequences and impact of each architectural strategy formulated are easily addressed.

Finally, Fig. 17 depicts the software products obtained from the conceptual models identified by the researchers to support specific elements of the academic quality system.

Fig. 17

Examples of software products obtained from conceptual models

1.2 A.2 The first signs of quality problems

The first signs of quality problems associated with the use of multiple models and different modeling languages can be observed. The first problems can be found by analyzing the visual language used by experts and organizational stakeholders to represent the business processes of the university, since it is the self-assessment process for accreditation and re-accreditation of an undergraduate or graduate program.

The researchers decided to evaluate the graphical notation using the theory of Physics of Notations (PoN) by D.L. Moody (Moody 2009), which is the most frequently published. The application of this theory provides a scientific basis for comparison, evaluation, improvement, and construction of visual notations used in an organization. The PoN theory proposes nine principles that can be successfully used to assess visual languages of graphic modeling (Cognitive Integration, Cognitive Fit, Manageable Complexity, Perceptual Discriminability, Semiotic Clarity, Dual Coding, Graphic Economy, Visual Expressiveness, and Semantic Transparency).

The institution does not use a standard visual language for modelling its business processes. The variant of the flow chart used by the university in the modeling of its processes does not preserve the semantics that is used for this type of notation, which causes the process model to be unclear for the roles that interact with them. Thus, the application of PoN helps validate the flowchart version created in the institution by applying the principles that this theory proposes.

This type of graphic language is not suitable for modeling business processes or complex systems because of its simplicity. In these cases, it is possible to find many other languages that are also appropriate such as BPMN or UML activity diagram. However, due to the lack of knowledge about different alternatives for process modeling, the migration of these processes to other languages has not been done.

The application of the PoN principles in the flowchart diagram variant used in process modeling at the University of Quindío is presented in the following sections.

1.2.1 A.2.1 Semiotic clarity

This principle establishes a one-to-one correspondence between the semantic constructions and the graphic symbols of visual language. When there is not a one-to-one correspondence between the analyzed symbols and their respective semantics, at least one quality problem generated in the notation which is related to Symbol Deficit, Symbol Redundancy, Symbol Overload, or Symbol Excess.

Figure 18 shows the analysis of notational elements employed in the variant flowchart applied at the University of Quindío compared to the original semantic constructs from the flowchart. The simplicity that is applied at the University of Quindío for conducting the flowcharts is shown in this analysis because not all the symbols originally formulated by the notation are used. As a result, out of the 16 original notation symbols contained in the university flowchart, only 3 symbols that have the same semantic construct and another construct with a different meaning are used. This analysis found two specific anomalies regarding the principle of semiotic quality, Symbol Deficit and Symbol Excess.

Fig. 18

Principle of semiotic clarity: there should be a 1:1 correspondence between semantic constructs and graphical symbols

The Symbol Deficit anomaly found represents the lack of 13 symbols by the university in order to meet the standards of a flowchart. For the Symbol Excess problem, the use of the visual element internal connector is contrasted (Fig. 19), identifying the meaning given in the description of processes of the University of Quindío and its original semantics according to specifications of the flowcharts (ISO 1985).

Fig. 19

Comparison between the symbols used at the University of Quindío and the symbols used in the semantic construct of the flowcharts

1.2.2 A.2.2 Perceptual discriminability

This principle is related to the ease and perception with which the symbols used in a graphical notation can be distinguished from each other. Although this principle is supported by the specific adaptation of the flowchart conducted at the university, the main problem found in the analysis of perception is simplicity due to the number of symbols used. This can be seen as something that is relatively handy when making model interpretation of the business process. However, given the complexity of a business process of an organization, it is not feasible to conduct a modeling with so few symbols, since it loses too much of the useful information that provides a better understanding and proper execution of the process.

1.2.3 A.2.3 Semantic transparency

The principle of semantic transparency refers to the ease of identification of the semantic meaning of a symbol that is used in a graphical notation. This principle considers four possible classifications for the analyzed symbols of the visual language:

• Semantically perverse: When the symbol is observed, it is not easy to identify its meaning.

• Semantically opaque: When the symbol is observed, the person arbitrarily relates it to something known in order to identify its meaning.

• Semantically translucent: In order to know the meaning of the symbol, the person requires prior explanation.

• Semantically immediate: The meaning of the analyzed symbol can be identified easily without prior explanation.

The notation used for the modeling of processes at the University of Quindío identifies two semantically transparent symbols (Fig. 20—left) since they preserve the semantic construct of the flowcharts. Thanks to this, it is easy to identify their meaning (semantically immediate). However, the presence of the semantically opaque category is also evident (Fig. 20—right) because the users of the business process (when noting some of the symbols by intuition and perception) relate what they observe to any known symbol. This gives a meaning that is not generally correct. At the University of Quindío, there are symbols for start/end, and there is another symbol for referring to documents or processes.

Fig. 20

Symbols used in the university that meet the semantically immediate / semantically opaque categories

1.2.4 A.2.4 Visual expressiveness

The principle of expression evaluates the number of visual variables used and the range of values (capacity) of these variables. It considers the use of space of graphic design and the variation in the whole visual vocabulary. Table 10 presents the identified values for the variables associated with this principle for the language used in the modelling processes of the University.

Table 10 Visual variables of the flowchart notation used at the University of Quindio

The main abnormality is the lack of guidance (arrows, lines, or useful symbols) to denote the process flow of the diagram, which restricts the browsing in the business process modeled. This reduces the diagram to a top-down sequential specification. The sharp demarcation in the application of colors creates identification problems for parts of the process, which affects its cognitive assimilation.

1.2.5 A.2.5 Complexity management

This principle evaluates the ability of visual languages to present large amounts of data without overloading the human mind. This principle refers to schematic complexity, which is based on the number of elements (instances or symbols) used in the diagrams. When analyzing this principle on the models of the business processes of the university, a high level of complexity due to the high number of activities (see Fig. 10) is presented. This hinders the understanding and implementation of the process. To reduce the levels of schematic complexity in models of business processes, subprocesses are generally used to group activities. This minimizes the number of symbols used in the modeling of the process and achieves a better understanding of the workflow.

1.2.6 A.2.6 Dual coding

This principle measures the use of text and graphics that are used together to transmit information. Specifically, the use of labels (text) plays a critical role in the interpretation of business diagrams since it defines and clarifies the semantics of the processes directly on the diagrams (i.e., the correspondence with the real-world domain).

The symbols used in the modeling of business processes at the University of Quindío have text labels to help interpret the flowcharts. The graphics used are inside Excel cells, which have several adjoining cells with associated text that provide information for the people who interact with these diagrams. The main drawback of these diagrams used is their excessive emphasis on the textual representation (Fig. 10). The visual elements fulfill a decorative function instead of a reasoning and communication function about the business process itself. The interpretation and expressiveness of the process models are directly affected by the excessive simplicity of the notation. The text itself becomes the central element of each diagram.

1.2.7 A.2.7 Graphic economy

This principle states that the graphical complexity of a notation must be cognitively manageable. The number of visually distinct symbols of the notation indicates the complexity of a chart. This principle is critical to help the understanding and expressiveness of process models. The graphical notation used at the University of Quindío is too minimalist (there are only 4 symbols out of the 16 originally specified in the flowcharts). This makes it less useful for the modeling of systems or complex processes given their lack of semantic support from the specific syntax employed.

A preliminary application of the PoN method identifies the shortcomings of the modelling language that is currently used at the University of Quindío. This application highlighting its simplicity for the specification of the process models since the flowcharts do not meet the requirements for the modeling of processes and complex systems.

1.3 A.3 Limitations of the selected approach to evaluate the quality of the models of the modeling scenario

The processes for the management of academic quality are highly changing and dynamic, mainly because of regulatory updates from the authorities that govern academic quality in Colombia (the Ministry of Education and CNA). These changes affect organizations that voluntarily apply for accreditation processes, as is the case of the University of Quindío. Additionally, there are specific organizational conditions (administrative restructuring, updating of procedures, involvement of experts from different areas of knowledge, etc.) within the institution that affect the quality process models, which in turn affect the models that conceptually support the information systems generated.

The office of Planning and Development of the University of Quindío starts the exploration of a strategy of business process management using the BPM discipline with its associated notation (BPMN). To do this, in conjunction with the researchers involved in the project, a systemic approach for the selection of BPM tools (commonly known as BPM Suites) is applied. This assessment was reported in Gallego et al. (2015). Once the most suitable BPM suite for the institution was selected, the researchers formulate an initial proposal in BPMN for the business process of self-evaluation from the specification presented in Fig. 10. A model containing 14 roles, 67 activities, and 67 attachments was obtained.

The proposed model was presented to them. Both the experts and the people from the planning department had difficulty understanding the model due to the high cognitive load and information present in the diagram generated. As reported in Gallego et al. (2015), the researchers formulated an intervention to the original specification of the model to facilitate understanding by the business experts. This clearly shows the emergence of quality issues such as expressiveness, understandability, completeness, and appropriateness of the models.

From the perspective of researchers in information systems, the system models in UML and other languages (with their conceptual support) contribute to the creation of communication scenarios and documentation on which they make decisions that are related to a specific technological implementation. The modeling tools that are used support the automatic generation of source code (MDD). However, the emphasis on conceptual modeling of the different components of the information system require an extra effort for their subsequent translation into a specific platform of implementation. This is due to the particularities that must be developed in order to support the essential features of any model that formulated in the project on that platform.

Despite the considerable number of system conceptual models generated by the research group (especially the use of the UML profile for business modeling), their importance was perceived with relative apathy by the business experts at the University of Quindío. This was mainly due to the lack of alignment between the models of the information system and the specification of the models of organizational processes. Although the generation of information system platforms was delegated to the researchers because of the innovative nature of the conceptual models used to develop an information system for academic quality, the system models are limited exclusively to the use of roles for analysts, designers, and software developers. Therefore, in order to avoid suspicion and loss of confidence in the system models by the business experts and the users of the self-assessment process, the development team had to generate incremental versions of the components of the information system modeling. This produced software solutions that allowed the users and people involved in the self-evaluation process to appreciate the feasibility of the innovative proposals made by the researchers. In this case, the models contained reference information to support implementation decisions, but they were not used to automatically generate the underlying infrastructure of code (a model-based approach instead of model-driven one was used).

While models in this project played a strategic role at the organizational and conceptual support level of an information system with computational implementation, there was a decoupling between the organizational modeling and the system modeling. This caused duplication of the modeling effort and lack of mechanisms for traceability that covered the evolution of business aspects for their respective technological implementation.

In the implementation at the University of Quindío, the understandability of models was important, but there are still other questions that remain open. For example, the suitability of UML models to address organizational concerns are not covered by current modeling efforts at the University of Quindío.