Decision support in engineering design: the ELIGERE open source software platform

Abstract

In engineering design, the selection of the optimal design solution represents a critical phase for the development of successful products. In this paper, we present ELIGERE, an open source decision support system targeted at engineering design applications. It allows to rank multiple design solutions with respect to different evaluation criteria according to the evaluations provided by a group of experts. ELIGERE is composed by three main modules: (1) a distributed web application, for generation and participation to the decision making session; (2) a mathematical engine, based on the fuzzy analytical hierarchy process, to quantify the results of the decision making session according to the evaluation of the experts; (3) a relational database, to collect and store data. The most important contribution of this paper is introducing a practical and effective software tool that facilitates decision-making analysis based on the fuzzy analytical hierarchy process, thereby allowing better-informed choices on concept selection, as it has been designed with a specific focus on the engineering field. In this paper we describe the key concepts of ELIGERE and its modalities of use in several real use cases. Finally, we compare ELIGERE with the widely used general purpose decision support software based on the fuzzy analytical hierarchy process.

1 Introduction

A Group Decision Making (GDM) process typically involves the following steps: (1) identification of the decision problem; (2) analysis of the requirements; (3) establishment of goal and objectives; (4) generation of alternatives; (5) selection of evaluation criteria; (6) selection of a decision making method and/or tool; (7) evaluation of alternatives against criteria; (8) validation of the solution against the problem statement [1]. Therefore, the setup of a group decision making session requires multiple skills in: (i) decision science and operation management, to select an appropriate decision making method; (ii) information systems and software engineering, to build the infrastructure for users. This means that enabling group decision making in real academics and/or industry scenarios related to engineering design is a complex task, involving more domains of expertise than a typical team can provide.

The goal of ELIGERE is to address these issues, by providing the community an intuitive framework for group decision making in engineering design. This framework allows users to focus on solving their specific selection problem, rather than also on tiresome and time-consuming issues related to the setup of the decision making session. ELIGERE is an open source software platform which offers several features of interest in group decision making: (1) setup of the decision making session by customizing a predefined form; (2) participation to the decision making session through a web browser, so that experts can be in different physical places; (3) instant computation of the optimal solution, once that the judgments of the involved experts are available; (4) data collection of experts judgments as well as the result of the decision making session in a permanent database. The software platform is released under GNU general purpose license: its source codes are available on the ELIGERE github repository.¹ On the ELIGERE website,² the users can find an already implemented version of the framework, together with tutorials and examples.

The ELIGERE platform was presented for the first time in the 2017 IEEE International Conference on Fuzzy Systems [2]: since then, it has evolved and expanded to meet novel needs from the users. This article aims at presenting, in a broader sense, the current technical details of the platform and its modality of use. Then, we present its application to several real use cases, to show the capability of the framework. Finally, we include a comparison of ELIGERE with respect to existing platforms and we report an overall discussion.

2 Related works and main contributions

In this section, we briefly survey the research areas of: decision making methods, decision making applications in engineering design, decision support software. Then, we highlight the main contribution of this work.

2.1 Decision making methods

GDM problems requires to find the best solution from a set of n achievable alternatives ($A_j, j = 1,..,n$) according to the preferences given by a group of m users ($U_j, j = 1,..,m$) [3]. The presence of k evaluation criteria ($C_j, j = 1,..,k$) often makes it difficult for the users to accurately assess all the simultaneous possible alternatives. These problems are referred to as multiple criteria decision making (MCDM) problems [4].

In the spectrum of decision-making methods, the fuzzy set theory [5] is the more suitable in several real applications. Indeed, it considers infinite-valued logic in contrast with crisp set theory based on binary logic. As a matter of fact, fuzzy interfaces transforms linguistic variables (e.g. “strongly”, “weakly” [6]) in fuzzy numbers, can be used to avoid the uncertainty brought by numerical voting.

In addition, the literature review underlines how the most used fuzzy multi-criteria decision-making techniques are: AHP, TOPSIS, VIKOR, ANP, ELECTRE, DEMATEL, PROMETHEE and ENTROPY [7]. In this context, AHP (introduced by Saaty in [8]) is the most used approach when dealing with fuzzy numbers [7]. It allows to simplify a multiple criterion problem by decomposing it into a multilevel hierarchical structure (by evaluation of a set of criteria elements and sub-criteria elements) [9]. A discrete 9-value scale method allows comparative judgments, in term of a pairwise comparison data on elements of the hierarchical structure [10]. So, different measures are integrated into a single overall score to obtain a ranking of alternatives.

The extension of AHP with fuzzy numbers is called FAHP, where the linguist variables are expressed in triangular fuzzy numbers  [11]. In literature several approaches have proposed to derive triangular fuzzy numbers[12]. One of the most common was developed by Changs [13] using the extent analysis method.

With respect to the state-of-the-art, ELIGERE uses the FAHP with triangular fuzzy numbers and the Chang’s extent analysis method for the defuzzification process. In addition, ELIGERE employs in Chang’s method, a novel different fuzzy conversion scale and exploits a computationally efficient algorithm to compute the comparison matrix, as described in Sect. 3.

2.2 Decision making applications in engineering design

The literature on related works which use decision-making in different domains is huge. Applications are related to: finance [14], economics [15], healthcare [16], logistics [17], energy [18]. In this work, we are focused on decision-making applications in engineering design, where we can observe an increasing trend in the number of publications in the last available review focus on applications [7]. Again, the majority of papers related to engineering applications use FAHP techniques (among 217 works analysed, 102 use FAHP) [7]. The role of FAHP in engineering applications was also confirmed in a recent review that analysed all articles published on fuzzy AHP in the last three decades [19].

During the development and life-cycle of complex engineering systems, many decision making situations arise at different levels [20]. In these situations, several technical and economic factors must be taken into consideration for taking informed decisions. Within the product design and development process, a critical decision step is the selection of a design concept from a set of available alternatives: this phase is called concept selection [21]. The literature underlines the importance of the concept selection phase, indeed nearly 80% of the product cost is related to this stage [22].

Therefore, erroneous design solutions need to be minimized to diminish the overall costs. For these reasons, MCDM techniques can help the final success of a product [23]. This is particularly important for the concept selection phase of complex engineered systems whose development requires a design team with multiple expertise [24]. The literature presents many applications of decision-making methods based on AHP in the concept selection phase as for example in [25] for elevators selection or in [26] conveyor system selection.

Decision making support systems in engineering design are also useful when user studies are required to evaluate a product or a system, which might work with different settings and features. This process is called system evaluation. In this case, the decision makers (i.e., experts) are directly chosen among the population of end-users of such a product/system, to evaluate it with a different set of additional functionalities. Examples of applications using decision-making methods based on AHP in system evaluation include: evaluation of an aircraft system [27], to obtain optimum welding conditions in the submerged arc welding process [28]. In engineering design, ELIGERE supports both the phases of concept selection and system evaluation, as we can see from the example in Sect. 5. As a matter of facts, ELIGERE can be used in selecting the best design concept before the manufacturing phase, or even in testing and evaluation of a novel system, after its production. Regarding the first scenario, classic examples refer to the design and development phases of components and subsystems in complex systems. Regarding the second scenario, classic systems that need to be tested/evaluated are human–machine systems working at different modalities (e.g., surgeons evaluating different robotic surgery systems which can work at different level of autonomy; physiotherapists or patients evaluating different robotic rehabilitation systems at different level of autonomy, ...).

2.3 Decision support software

Several decision support tools have been developed from the mostly recognized decision making methods: an overview is presented in [29]. This general-purposes software can be applied in different domains (e.g., for supporting environmental planning processes in [30]). The majority of software solutions are based on AHP theory and they are primarily adopted for applications which go beyond engineering design. According to [31], the widely-used general-purposes decision support software based on AHP are: Expert Choice,³ Web-HIPRE,⁴ MakeItRational⁵ and PriEsT.⁶ These software have been applied in different domains; in this section we provide a review on their usage on engineering design applications.

Expert Choice [32] is desktop-based commercial software developed since the 80’s. Regarding engineering design applications, it has been used in concept selection (selection of a wheelchair [33]) and in system evaluation (evaluation of manufacturing systems [34]).

Web-HIPRE [35] is the first web-based and free decision support software, developed by the Helsinki University of Technology. It is the web-version of the HIPRE 3+ software, developed since the 90’s. Examples of Web-HIPRE use are related to different applications: selection of optimal municipal solid waste systems [36] and selection of optimal agricultural systems [37]).

MakeItRational [38] is a web and desktop-based commercial software for decision support. It has recently evolved in the TransparentChoice AHP Software,⁷ which offers a really simple user interface. It has been used in nuclear engineering applications, in the concept selection of heat exchangers for nuclear reactors [39], and in the comparative assessment of nuclear energy systems [40].

PriEsT [41] is an open source decision support tool developed by the University of Manchester. It has been used in the concept selection process in the ceramic industry, to help designers to predict technological evolution for novel user-friendly concepts [42]. It has also been used in system evaluation for several use cases like in the selection of a backbone infrastructure for telecommunication system [41] and in evaluation of an haptic tool for soccer fans [43].

Apart from the general-purposes software, more recent applications are related to decision making tools customized for specific domains. A great example is represented by QATCH,⁸ fully described in [44]. QATCH is a FAHP-based tool that assist the users in generating software quality models tailored to stakeholder specifications. In the context of decision making tools, ELIGERE is a web-based and open-source tool specifically born to support group decision making in engineering design. Therefore, for its development, a great effort has been made in providing user-friendly interfaces for the experts, with the possibility to include images, videos or even 3D files of the alternatives, which in most of cases are represented by industrial products: this simplify the selection and evaluation from the users. In these sense, a comparison with the main decision support software is reported in Sect. 6.

2.4 Main contributions

The main contributions of this work are:

• 1. Presentation of a decision support tool based on FAHP, which involves a novel algorithm for the computation of the pairwise comparison matrix.

• 2. User-friendly software tool specifically designed for engineering applications. It allows to generate FAHP surveys in short time, allowing experts to submit their answers via a web interface and allowing storing in a database the results of multiple FAHP surveys.

• 3. Showcasing of real-world case studies where the software has been applied to different decision-making problems, to demonstrate its effectiveness in different engineering scenarios.

• 4. Comparison of ELIGERE with other FAHP-based decision support tools.

3 The ELIGERE framework

In this section we provide the description of ELIGERE alongside its main characteristics. ELIGERE is a decision support system devoted to engineering design applications. When one concept needs to be selected among multiple design solutions, ELIGERE allows to rank the design alternatives based on given evaluation criteria. To this end, ELIGERE is based on the analytic assessment of FAHP questionnaires submitted to a panel of experts. They are asked to compare the relative importance of the given evaluation criteria and the relative preference of one alternative over another, for each evaluation criteria. Their judgments, from the mathematical point of view, are represented by fuzzy numbers which express the relative scale of importance/preference of one criterion/alternative over another. These judgments are then processed by a FAHP algorithm to rank both the criteria (preference section) and the alternatives (suitability section). A decision making session using ELIGERE involves two typologies of human actors, the admin and the users. The conceptual workflow on which ELIGERE is based is illustrated in Fig. 1, and it is described as follows.

Fig. 1

ELIGERE conceptual workflow

• The admin is the responsible of generating the FAHP questionnaire. She/he uses an automatic form which allows to generate FAHP questionnaires by simply writing the name of evaluation criteria and alternatives. Images, videos or even 3D files (in several format as.png,.jpg,.svg,.avi.,.pdf.pdf3d ...) of the alternatives might be uploaded as well. The admin gives a name and a password to each FAHP questionnaire.

• The users are the experts involved in the decision making process. They login to the FAHP questionnaire by using the name and the password, as provided by the admin. They create a personal account in which personal details as gender, age, role and experience in the specific field are requested. They fill the FAHP questionnaire on their own browser; a simple user guide is provided in the web browser with instructions on how to complete the form. When they submit the FAHP questionnaire, their answers are uploaded on the database.

• The database collects all the answers from the users. The engine, i.e. the computational module, queries the database and, when all the answers of a FAHP questionnaire are available, it processes them using to the implemented FAHP. Finally, the data processing allows to obtain the ranking of: (i) criteria; (ii) alternatives; (iii) alternatives respect to each criterion. These results are stored again on the database.

ELIGERE has been developed using the Model-view-controller (MVC) pattern architecture [45], as previous decision support tools [41]. Therefore, it articulates in presentation, business and data access layer, as illustrated in [46].

${Presentation\,\, layer}:$:

it provides a Graphical User Interface (GUI) for the users, and it contains the web client.

${Business\,\, layer}:$:

it contains the core of the software platform, i.e. the FAHP engine, as well as the web server.

${Data\, access\, layer}:$:

it stores the data from business layer in a permanent storage, i.e. a relational database.

Presentation and business layers exchange messages on the network through the HTTP protocol; indeed, business and data access layers through the TCP/IP protocol. In Fig. 2 we have reported the sequence diagram related to the data flow between clients and remote hosts. In the following, we describe the main components of ELIGERE: the FAHP engine, the dynamic web application and the relational database.

Fig. 2

ELIGERE sequence diagram related to the data flow between the clients and the remote hosts

3.1 FAHP engine

FAHP Engine is the core of ELIGERE; developed in c++ programming language, it provides the data processing based on the judgments from experts. The judgments from experts are collected in the database. Then, these data are retrieved by the FAHP engine through a MySQL function: once that connection is established, FAHP Engine and Database exchange information through the TCP/IP protocol. FAHP Engine is built upon Eigen⁹ library. FAHP engine has the objective to rank the criteria and alternatives: to this end, a custom FAHP procedure has been developed for ELIGERE.

Let us consider for simplicity just the rank of criteria, where r decision makers are asked to evaluate, in a pair-wise manner, n criteria. This means that they have to answer to $n\left( n-1\right) /2$ questions. They have just to provide the pair-wise evaluation using symbols (‘=’ equal, ‘+’ better, ‘++’ much better, ‘+++’ much much better, ‘-’ worse, ‘–’ much worse, ‘—’ much much worse). These answers are translated, from the algorithmic side, in fuzzy numbers, which in this case are considered to be triangular [2]. The pair-wise comparison matrix includes all the answers from the experts: to fill its element we can consider two vectors ${{\textbf {a}}}$ and ${{\textbf {b}}}$, with generic elements $a_i$ and $b_i$ which are given by

$$\begin{aligned}{} & {} a_i = \frac{\sum \nolimits _{j=1}^r g_{ij}}{r} \end{aligned}$$

(1)

$$\begin{aligned}{} & {} b_i = (a_i)^{-1}. \end{aligned}$$

(2)

where $g_{ij}$ is the fuzzy number associated with the $i-$th answer of the $j-$th expert (where $i = 1,\ldots , n\left( n-1\right) /2$; $j = 1,\ldots ,r$). For the basic mathematical operations with fuzzy numbers (sum, multiplication, inverse, $\dots$) refer to [2]. Now, using (1) and (2) we can compute each element $e_{ij}$ of the pair-wise comparison matrix ${{\textbf {E}}}$ as

$$\begin{aligned} {e_{i,j}=} {\left\{ \begin{array}{ll} a_{i,j-1+i \cdot \frac{i-1}{2}+ i \cdot (n-i-1)} &{} { if} j<i \\ (1,1,1) &{} { if} j=i \\ b_{i,i-1+j \cdot \frac{j-1}{2}+ j \cdot (n-j-1)} &{} { if} j>i \end{array}\right. } \end{aligned}$$

(3)

Equation (3) is a new way to build the pair-wise comparison matrix. Notice that each element $e_{i,j}$ in (3) is still a fuzzy number. Now it’s time for the defuzzification process: to this end we have used the extent analysis methodology [2], which is briefly described in the following. First, we compute the fuzzy synthetic extent value $f_i$ associated with the $i-$th criterion as

$$\begin{aligned} f_i = \sum _{j=1}^{n} e_{i,j} \odot \left[ \sum _{i=1}^{n} \sum _{j=1}^{n} e_{i,j} \right] ^{-1} \end{aligned}$$

(4)

where the elements $e_{i,j}$ are given in Eq. (3). The comparison between two fuzzy numbers $f_1$ and $f_2$ is possible only by calculating the degree of possibility Z according to a fuzzy number is greater than the other, and vice-versa, as

$$\begin{aligned} {\left\{ \begin{array}{ll} Z({f_1} \ge {f_2}) = 1 \hspace{2em} &{} \text {if}\, f_{1m} \ge f_{2m}\\ Z({f_2} \ge {f_1}) = \displaystyle \frac{f_{1l}-f_{2u}}{(f_{2m}-f_{2u})-(f_{1m}-f_{1l})} &{} \text {otherwise} \end{array}\right. }\nonumber \\ \end{aligned}$$

(5)

where $f_{\cdot l},f_{\cdot m},f_{\cdot u}$ indicate respectively the lower, medium and upper values of the triangular fuzzy numbers, while $\cdot$ indicates 1 or 2 (in this case, first and second number that we have considered). The pairwise comparison of n fuzzy synthetic extent values $f_i$ leads to 2n non-fuzzy values $Z (f_i \ge f_j)$. Finally, in Eq. (6) we report the n values of comparison between each fuzzy number $f_i$ and all the other fuzzy numbers $f_k, \text {with}\,\,\, k=1,\ldots ,n \,\,\, \text {and} \,\,\, k\ne i$

$$\begin{aligned} \begin{aligned}&Z({f_i} \ge {f_1},{f_2},\ldots ,{f_n}) \\&\quad = Z [({f_i}\ge {f_1}) \cdot ({f_i}\ge {f_2}) \cdot \ldots \cdot ({f_i}\ge {f_n})] \\&\quad = \text {min} \, Z({f_i} \ge {f_1},{f_2},\ldots ,{f_n}) \end{aligned} \end{aligned}$$

(6)

Just for simplicity, we now indicate each previous comparison value in (6) with $s'(p_i)$

$$\begin{aligned} s'(p_i)= & {} \text {min}\, Z (f_i \ge f_k), \,\,\, i=1,\ldots ,n, \,\,\, \nonumber \\{} & {} \qquad k=1,\ldots ,n, \,\,\,k\ne i \end{aligned}$$

(7)

Then, we collect all the values of (7) in a weighted vector ${{\textbf {w}}}'$ as

$$\begin{aligned} {{\textbf {w}}}' = [s'(p_1), s'(p_2),\ldots ,s'(p_n)]^T \end{aligned}$$

(8)

At the end, by normalizing (8), we obtain

$$\begin{aligned} {{\textbf {w}}} = [s(p_1), s(p_2),\ldots ,s(p_n)]^T \end{aligned}$$

(9)

which indicates the normalized weighted vector, applied in this case for the criteria.

3.2 Dynamic web application

The ELIGERE dynamic web application has two main functions:

• 1. To allow the admin in generating the FAHP questionnaire

• 2. To allow the users in filling the FAHP questionnaire

Fig. 3

Right a Front page of ELIGERE with on the top the login for (multiple) admins; on the left the login for users involved in specific FAHP questionnaires. Left b Admin web page

Fig. 4

Right a Web page (users side) corresponding to the Preference section of the FAHP questionnaire. Left b Web page (users side) corresponding to the Suitability section of the FAHP questionnaire

To do that, it has been developed using the client−server architecture: therefore, it is composed by web clients and a web server. Web clients request and visualize the web-pages of ELIGERE using classical web browser applications. They exchange information with the web server node via HTTP protocol. The communication is started by a client program which communicates with a server program for:

• 1. Retrieving a specific FAHP questionnaire;

• 2. Creating a new FAHP questionnaire, in case of admin.

The web server is represented by an Apache HTTP server.¹⁰ Through the web Server node, ELIGERE is able to (see Fig. 2):

• 1. Process HTTP requests from the users;

• 2. Dynamically generate the FAHP questionnaires using a PHP interpreter;

• 3. Retrieve data from the database;

• 4. Save data on database.

3.3 Relational database

The ELIGERE relational database, using the database management system MySQL,¹¹ allows to collect, store and make available data. It presents the following main functions:

• 1. It provides an history of past FAHP questionnaires;

• 2. It collects data from the n questionnaires (one for each user) related to the same survey;

• 3. It provides the input data to the FAHP engine;

• 4. It stores the output data from FAHP engine (outcomes of the decision making session).

4 Front end application

ELIGERE source codes can be downloaded in the github repository: https://github.com/eligere/, while the ELIGERE website (http://www.eligere.org/) contains an already implemented version of the framework, free to be used. The front page of the ELIGERE website is shown in Fig. 3a.

The process starts with the generation of the FAHP questionnaire made by the admin. The admin login to the ELIGERE website with its username and password. Once logged in, the admin views the webpage in Fig. 3b. Here, She/he can: (i) generate new FAHP questionnaires (through the Insert New Survey button); (ii) view answers of the users corresponding to specific FAHP questionnaires (through the Show Survey Data button); (iii) elaborate the results of specific questionnaires (through the Show Survey Results button).

When the admin creates a new questionnaire, she/he has to associate a password with it. The password is needed for the experts in order to login via the front page of ELIGERE website through a specific format for users (see, e.g. Fig. 3a). When logged in, the user is able to compile the Preference section (see Fig. 4a) and the Suitability section (see Fig. 4b) of the FAHP questionnaire. In particular, the alternatives in the Suitability section are presented through 2D/3D images or video for representation of the concepts/systems and with a related small description (see Fig. 4b). Notice that the Suitability section foresees a dynamic generation of web pages, one for each pairwise comparison. Once that the experts have filled this section, they have the possibility to review their answers and then submit their evaluation through the Save button. The answers of the experts are collected in a relational database, and they can be accessed by the admin through the Show Survey Data button (see Fig. 3b). When all the experts have submitted their FAHP questionnaires, the admin can elaborate the results by pushing on the Show Survey Results button (see Fig. 3b). The results are shown in Fig. 5; they include: the final ranking of the alternatives (under Final Score); the ranking of the criteria (under Section 1: Preferences) and the ranking of the alternatives with respect to each criterion separately (under Section 2: Suitability).

Research teams interested in using ELIGERE can request admin privileges to the ELIGERE developers. Each admin can create multiple FAHP questionnaires and keep the history of past questionnaires. However, an admin is not able to generate another admin. A video on the use of ELIGERE platform can be found here: https://www.youtube.com/watch?v=997ses6_b8k.

Fig. 5

Web page of the results (accessible only by admin)

Table 1 Criteria, alternatives, number of questions and experts for each use case (UC)

5 Applications

In this section we report the summary of the main use cases that our research team has carried on through ELIGERE. In particular, in Table 1, we report, for each of the seven proposed use cases, the number of criteria and alternatives and the subsequent number of questions related to the criteria and to the alternatives, number of experts involved in the decision making process. The details of the use cases, in terms of: problem statement, description of alternatives and criteria, description of users and results are collected in the Appendix 1, just to simplify the reading. Here we report just the summary. For the seven use cases, five are related to the concept selection phase of novel products/systems (two related to products in the sensing field, two related to robotic solutions in the nuclear fusion field, one related to smart garments), while two are related to the evaluation of systems working at different settings (one in the sports and one in the rehabilitation fields). The number of criteria spans from 3 to 10. Indeed, the number of alternatives spans from 2 to 6. The higher number of criteria (10 in UC3 Concept selection of a research facility to test autonomous remote operations in challenging environments, see Sect. 1) and the higher number of alternatives (6 in UC1 Concept selection of an ultrasonic sensor frame for mobile robots, see Sect. 1 was chosen to limit the maximum number of questions to be answered. Indeed, a lot of questions to be answered might compromise the reliability of the results, as they might be affected by the loss of attention during the questionnaire. Etikan et al. have reported that the time required for filling a questionnaire should not last longer than 30 to 45 min [47]. Considering that each question of the FAHP questionnaire would require from 40 to 60 s for a good response, it turns out that going above 45 questions would compromise the reliability of the result. Indeed, 10 criteria result in 45 questions, as 6 alternatives (evaluated with 3 criteria) that result again in 45 questions (15 questions for the evaluation with respect to each of the three criterion). Finally, the number of experts spans from 3 to 22: this is not very high since FAHP questionnaires require users with high experience in the field. The one with the higher number of experts is UC3 (Sect. 1) which involves a design selection process with experts in the field from all over Europe.

6 Comparison and discussion

In this section we evaluate ELIGERE by comparing it with respect to the widely-used general-purposes decision support software based on AHP, introduced in Sect. 2.3.

We derive a set of features that a decision support tool should have and, for each one, we assign a numerical score for all the tools. The features have been selected from previous works regarding the comparison of: MCDM software and MCDM AHP-based software [31, 48]. These features have been derived taking into consideration the basic criteria that a software must satisfy according to the international regulation on software quality ISO/IEC 25010:2011, i.e. functionality, usability, maintainability, portability. Although the scoring should also be formulated through pairwise comparison [48], we focus here on numerical scores. For each feature, we consider three scores: 0, if the feature is not present; 0.5, if the feature is partially present; 1, if the feature is present.

Table 2 ELIGERE scores according to the selected features

Table 3 MCDM software scores according to selected features

The considered AHP-specific features adapted from [48] and [31] are:

F1:

hierarchical model construction, i.e. the possibility to build hierarchy levels with a certain number of elements at each level;

F2:

criteria and alternatives, i.e. the possibility to include criteria and alternatives separately;

F3:

crisp/fuzzy numbers and verbal scales, i.e. the possibility to use crisp numbers (score 0.5) or fuzzy numbers (score equal to 1) associated to verbal scales;

F4:

visual aids, i.e. the possibility to include a graphical representation of criteria and alternatives (images, sketches, CAD models);

F5:

consistency analyser, i.e. the possibility to visualize inconsistency among the given pairwise judgments;

F6:

results analyser, i.e. the possibility to see the results by numerical tables, plots and graphs;

F7:

database, i.e. the possibility to store the results in a relational database;

F8:

sensitivity analyser, i.e. the possibility to see how sensitive the final rankings are to changes in input data.

F9:

user-friendly, i.e. the possibility of being simple and convenient to use, even without training (for both end-users and admin to setup the decision making session)

F10:

cost, with score equal to 0 for commercial solutions, 0.5 for free solutions, 1 for free and open-source solutions

With a total of ten features and maximum score equal to 1 for each feature, the maximum total score for a decision support system is equal to 10.

ELIGERE has been evaluated by 3 experts, two experts independently ranked the ELIGERE features. In case of disagreement between the two above experts, an independent expert stepped in. Table 2 reports the scores of ELIGERE with respect to the features, alongside with a motivation on the score. This novel software platform shows high scores with respect to: model construction and possibility to include many criteria and alternatives in a hierarchical manner, using of verbal scales associated to fuzzy numbers for dealing with the shade of judgments, usability, with simple interfaces, alternatives presented in a graphic format (2D/3D representation or videos), nice representation of the results and open-source. The main limitations arise in the lack of consistency and sensitivity analysers of the results at the current stage; therefore, these aspects will be considered in an improved version of ELIGERE.

Table 3 compares the scores obtained for ELIGERE with those obtained from the most diffused and adopted AHP-based decision making tools presented in Sect. 2.3. For these software, the scores related to the features are adapted from the evaluation presented in [31]. It is important to underline that the features have been selected so that the score is as objective as possible, indeed the scores on three levels (0, 0.5 and 1) are just related to the lack, limited presence and presence of the particular feature on the software. The total scores of the software are comparable, as they span from 6.5 (Expert Choice) to 8.5 (PriEst), with a total score of ELIGERE equal to 8. This tables underlines the strength and weakness of ELIGERE with respect to available solutions. The main strength are due to its usability and user-friendly interfaces (for the admin, in terms of model construction, storing and showing of results; for the experts, in term of questionnaires simple to fill), graphical representation of alternatives and open-source capability. The weakness, as also highlighted before, is in the lack of consistency and sensitivity analyzers. The strength of ELIGERE are particularly useful in the engineering design context, where it is really important, for making an informed decision, to show the alternatives to the experts with enhanced graphical representations. With respect to the latter point, future works will be devoted to further spreading the usability of the platform, including virtual reality experience from the users’ side as optional. The limitation of ELIGERE regarding the consistency of the answers from experts, could be partially overcome considering the target of the users that are only experts in the field.

7 Conclusions

In this work we have presented ELIGERE, a distributed software platform for supporting group decision making in engineering design. The ELIGERE infrastructure is based on several components organized in a network. From the decision making perspective, it provides a custom implementation of the FAHP, one of the most adopted MCDM methods. After a basic description of its modules, functionalities and usage, we reported several real use cases related to engineering design applications. Finally, we evaluated and compared ELIGERE with four widely-used AHP-based decision support tools, with respect to classic features that a software of this kind should have.

ELIGERE can give to design and management teams a quick, structured, analytical and efficient procedure able to take into account the high number of different skills required for the development of engineering products and services (suitable also internationally distributed teams, for its capability to be on cloud).

A Appendix

The Appendix reports the details of the applications summarized in Sect. 5. For each use case we present the problem statement, we introduce the alternatives and criteria for the decision making problem, we describe the users/experts which have been involved in the decision making session and we present the results.

1.1 A.1 Concept selection of an ultrasonic sensor frame for mobile robots

Problem statement Selecting the best concept of an ultrasonic sensors’ frame for mobile robots [2]. This system has to provide support for the sensors which give the robotic platform the functionalities of collision avoidance and environment awareness. The frame is mounted on a commercial mobile robotic platform, i.e. the omnidirectional MPO-700 from Neobotix.¹²

Alternatives and criteria Six design alternatives $A_j, j = 1, \dots , 6$ which differ from shape and sensors regulation system are evaluated with respect to (w.r.t.) three criteria: $C_1 =$ simplicity; $C_2 =$ aesthetic design; $C_3 =$ integration with the mobile robotic platform.

Users Seven experts (four computer scientists and three engineers) from the Competence Center on Cultural Heritage Digitization at the Fraunhofer Institute for Computer Graphics Research IGD (Darmstadt, GE).

Results Fig. 6 shows the results for this case study. $A_1$ is considered the best design solution among the six alternatives with a score of 25%. The most important criteria are $C_1$ and $C_3$. The $A_1$ alternative is considered the best one for these two criteria ($C_1$ with a weight of 33% and, $C_3$ with a weight of 21%).

Fig. 6

ELIGERE output for case study 1. Top row: ranking of alternatives (with the relative pictures above) w.r.t. each criterion. Bottom row: ranking of criteria, final ranking of alternatives. $C_1$: simplicity; $C_2$: aesthetic design; $C_3$: integration with the mobile robotic platform

1.2 A.2 Concept selection of a 3D body scanner

Problem statement Selecting the best concept of a 3D scanner for human body digitization [49, 50]. This system is used for instant 3D digitization of anatomical structures for design of tailored biomedical devices and for other medical applications.

Alternatives and criteria Three design alternatives of body scanning systems $A_j, j = 1, \dots , 3$ (one based on tripods, one based on poles and one based on a closed room) are evaluated w.r.t. three criteria: $C_1 =$ simplicity; $C_2 =$ aesthetic design; $C_3 =$ maintainability.

Users Three mechanical engineers working at the Department of Industrial Engineering at University of Naples Federico II (Naples, Italy).

Results Figure 7 shows the output of ELIGERE for this case study. $A_2$ is considered the best design solution among the three alternatives (with a score of 36%). The most important criteria is $C_1$ with a weight of 38% and the alternative $A_2$ is considered the best for this criteria (with a weight of 38%). This best concept today has evolved towards a commercial product: INBODY - Instant Body Scan™from BeyondShape¹³.

Fig. 7

ELIGERE output for case study 1. Top row: ranking of alternatives (with the relative pictures above) w.r.t. each criterion. Bottom row: ranking of criteria, final ranking of alternatives. $C_1$: simplicity; $C_2$: aesthetic design; $C_3$: maintainability

1.3 A.3 Concept selection of a research facility to test autonomous remote operations in challenging environments

Problem statement Selecting the best concept of a research facility to be used for testing of robotic maintenance and inspection procedures in nuclear-oriented environments, called Automated Inspection and Maintenance Test Unit (AIM-TU) [51]. This facility is based on robotic equipment able to demonstrate that remote tasks typical involved in nuclear fusion reactors (i.e., maintenance and inspection) can be automated in a reliable manner.

Alternatives and criteria Three conceptual alternatives of AIM-TU $A_j, j = 1, \dots , 3$ which differ from layout and components are evaluated w.r.t. ten criteria: $C_1 =$ functional performance; $C_2 =$ safety; $C_3 =$ environment independence; $C_4 =$ test quality; $C_5 =$ ongoing project risk; $C_6 =$ lower initial resources; $C_7 =$ space; $C_8 =$ ease of reconfiguration; $C_9 =$ versatility; $C_{10} =$ ease of use.

Users Ten engineers working at Remote Applications in Challenging Environments (RACE), United Kingdom Atomic Energy Authority (UKAEA), Culham, UK.

Results Fig. 8 shows the output of ELIGERE for this case study. $A_2$ is considered the best design solution among the three alternatives (with a score of 37%). All the considered criteria received overall similar weightings, indicating that the selected criteria are all relevant to the decision-making process.

Fig. 8

ELIGERE output for case study 1. Top row: ranking of alternatives (with the relative pictures above) w.r.t. each criterion. Middle row: ranking of criteria, final ranking of alternatives. $C_1 =$ functional performance; $C_2 =$ safety; $C_3 =$ environment independence; $C_4 =$ test quality; $C_5 =$ ongoing project risk; $C_6 =$ lower initial resources; $C_7 =$ space; $C_8 =$ ease of reconfiguration; $C_9 =$ versatility; $C_{10} =$ ease of use

1.4 A.4 Concept selection of the divertor remote maintenance strategy for the DTT machine

Problem statement Selecting the best strategy of remote maintenance for the divertor of the DTT machine [52, 53]. DTT is the next Italian facility for nuclear fusion research; the divertor is one of the major in-vessel components of the machine.

Alternatives and criteria Three strategies of divertor remote maintenance are proposed $A_1 =$ installation/removal of the divertor via the lower lateral ports; $A_2 =$ installation/removal of the divertor via the equatorial ports using an articulated manipulator [54]; $A_3 =$ installation/removal of the divertor via the equatorial ports using an articulated rail. The alternatives are evaluated w.r.t. eight criteria: $C_1 =$ design flexibility and robustness; $C_2 =$ technology readiness level; $C_3 =$ machine availability; $C_4 =$ ability to carry out heavy payloads; $C_5 =$ deployment space; $C_6 =$ manoeuvrability; $C_7 =$ technical risks; $C_8 =$ reliability.

Users Twenty-two experts from seven European Institutions working on nuclear fusion research: nineteen with background in engineering (mostly mechanics and electrics/electronics), three with background in physics.

Results Fig. 9 shows the output of ELIGERE for this case study. $A_2$ is considered as the best strategy among the three alternatives (with a score of 38%). The most important criteria are $C_1$, $C_2$, $C_7$. The DTT is an international facility that will be built at ENEA, Frascati, Italy (see, e.g. here).

Fig. 9

ELIGERE output for case study 1. Top row: ranking of alternatives (with the relative pictures above) w.r.t. each criterion. Middle row: ranking of criteria, final ranking of alternatives. $C_1 =$ design flexibility and robustness; $C_2 =$ technology readiness level; $C_3 =$ machine availability; $C_4 =$ ability to carry out heavy payloads; $C_5 =$ deployment space; $C_6 =$ manoeuvrability; $C_7 =$ technical risks; $C_8 =$ reliability

1.5 A.5 Concept selection of smart garments

Problem statement Selecting the best concept of a smart shorts for a specific sport gesture analysis, in particular used by football player for electromyographic analysis on field [55].

Alternatives and criteria Four design alternatives $A_1$ (measurement unit with two cases, textile electrode with polymer reinforcement, numerical user interface), $A_2$ (measurement unit with one case, textile electrode with polymer reinforcement, simplify user interface), $A_3$ (measurement unit with two cases, textile electrode without polymer reinforcement, simplify user interface) and $A_4$ (measurement unit with one case, textile electrode without polymer reinforcement, numerical user interface) were designed. Six criteria were defined: $C_1 =$ ergonomic; $C_2 =$ trendy; $C_3 =$ affordable; $C_4 =$ strong; $C_5 =$ performing; $C_6 =$ user friendly. These evaluation criteria were selected through the semantic space analysis of the Kansei Engineering using data collected from stakeholders.

Users Seventeen users (athletic trainers, coaches, footballers and members of the medical staff), over 90% of which with more than five years experience in the field.

Results Figure 10 shows the output of ELIGERE for this case study. $A_1$ is considered the best solution among the four concepts. Four criteria (i.e., $C_1$, $C_3$, $C_4$ and $C_5$) are considered to have the same importance.

Fig. 10

ELIGERE’s output for case study 1. Top row: ranking of alternatives (with the relative pictures above) w.r.t. each criterion. Bottom row: ranking of criteria, final ranking of alternatives. $C_1 =$ ergonomic; $C_2 =$ trendy; $C_3 =$ affordable; $C_4 =$ strong; $C_5 =$ performing; $C_6 =$ user friendly

Fig. 11

ELIGERE’s output for case study 1. Top row: ranking of alternatives (with the relative pictures above) w.r.t. each criterion. Bottom row: ranking of criteria, final ranking of alternatives. $C_1$: feeling about power transmission capability; $C_2$: feeling about comfort

Fig. 12

ELIGERE’s output for case study 1. Top row: ranking of alternatives (with the relative pictures above) w.r.t. each criterion. Bottom row: ranking of criteria, final ranking of alternatives. $C_1$: feeling of ergonomics; $C_2$: feeling of invasiveness; $C_3$: rehabilitation experience

1.6 A.6 Evaluation of sports equipment

Problem statement Selecting the best concept of foot-stretcher for ergometer, used by rowers for indoor training [56]. This concept is mounted on a commercial ergometer, i.e. the Model C from Concept2.¹⁴

Alternatives and criteria Two design alternatives $A_1$ and $A_2$ (the first with a variable interaxle distance, i.e., $I_C$, and the second with a fixed interaxle distance, i.e., $I_S$) are evaluated w.r.t. two criteria: $C_1 =$ feeling about power transmission capability; $C_2 =$ feeling about comfort. These evaluation criteria were selected with the help of elite rowers.

Users Eight rowers (three with an international background, three with a national one and two with a regional one) with different anthropometric characteristics (which cover a large range of the rowing population).

Results Fig. 11 shows the output of ELIGERE for this case study. $A_1$ is considered the best solution from the athletes’ perspective; indeed, the system with $I_C$ has obtained a score improvement of 32% with respect to the system with $I_S$. As we can see, the two criteria are considered to have more or less the same importance (i.e., 52% and 48% of weight respectively). The adjustable alternative is considered better than the standard one for all criteria (+46% for $C_1$ and +18% for $C_2$).

1.7 A.7 Evaluation of a tele-rehabilitation system

Problem statement Selecting the best working modality of a rehabilitation system for the hand. This system is based on a prototype of soft robotic glove [57, 58] and on a virtual reality interface developed using Unity3d.

Alternatives and criteria Four different rehabilitation scenarios at different level of autonomy/technology were selected: $A_1$, patient grasping a real object and supported by a physiotherapist; $A_2$ patient grasping a virtual object and supported by a physiotherapist; $A_3$ patient wearing the soft robotic glove and grasping a real object; $A_4$ patient wearing the soft robotic glove and grasping a virtual object. Three evaluation criteria for selecting the best rehabilitation scenario were selected with the support of physician: $C_1$ feeling of ergonomics; $C_2$ feeling of invasiveness; $C_3$ rehabilitation experience (in view of gamification).

Users Six people (four patients and two physiotherapists) participated to the experimental session.

Results Fig. 12 shows the output of ELIGERE for this case study. $A_3$ is considered the best rehabilitation scenario among the four alternatives (with a total score of 32%). The criteria $C_1$ and $C_2$ are considered to have the same importance (with a score equal to 35%). The $A_3$ alternative is considered the best one for these two criteria ($C_1$ with a weight of 37% and, $C_2$ with a weight of 31%).