Abstract
Acute coronary syndrome (ACS) is responsible for the obstruction of coronary arteries, resulting in the loss of lives. The onset of ACS can be determined by looking at the various signs and symptoms of a patient. However, the accuracy of ACS determination is often put into question since there exist different types of uncertainties with the signs and symptoms. Belief rule-based expert systems (BRBESs) are widely used to capture uncertain knowledge and to accomplish the task of reasoning under uncertainty by employing belief rule base and evidential reasoning. This article presents the process of developing a BRBES to determine ACS predictability. The BRBES has been validated against the data of 250 patients suffering from chest pain. It is noticed that the outputs created from the BRBES are more dependable than that of the opinion of cardiologists as well as other two expert system tools, namely artificial neural networks and support vector machine. Hence, it can be argued that the BRBES is capable of playing an important role in decision making as well as in avoiding costly laboratory investigations. A procedure to train the system, allowing its enhancement of performance, is also presented.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Cardiovascular diseases are considered as one of the leading causes of death in the industrialized countries. These diseases will become major causes of death in other countries of the world by the year 2020 (Murray and Lopez 1997). Among these, acute coronary syndrome (ACS) is a major concern. In case of Bangladesh, people with ACS are not aware of its signs and symptoms. Consequently, the incidence and prevalence of this disease are increasing significantly. The diagnosis of ACS appears to be very complex. In a country like Bangladesh, it can be noticed that people are usually recommended by the physician to perform expensive laboratory examinations such as ECG, chest X-ray, echocardiogram, and CT angiogram without initially by conducting the preliminary assessment of ACS suspicion. Usually, physicians in Bangladesh assess ACS suspicion by looking at the signs and symptoms of a person under medical treatment. Nevertheless, this procedure of ACS suspicion comprises medical misconception because clinical data cannot be measured with 100% certainty (Reason 2001). The reason for this inaccuracy or uncertainty is that both the patients and the physicians express the signs and symptoms of ACS by employing linguistic words, those are ambiguous, vague and inexact. ACS symptoms, including chest pain and breathlessness, are expressed by the patients by using linguistic words consisting of “Severe,” “Moderate” and “Little.” Thus, the existence of different categories of uncertainties is inevitable while carrying out ACS suspicion, which will be elaborated in Sect. 2.
It is, therefore, necessary to investigate the appropriate approaches, which can be employed to handle the different types of uncertainties, noticed during the assessment of ACS suspicion. Such approaches could enable the accurate assessment of ACS suspicion, and hence, the patients would avoid unnecessary as well as costly laboratory tests as (Mark et al. 1996; Weintraub et al. 1995) pointed out before. The assessment of ACS suspicion could be categorized as an instance of ill-defined problem because its measurement cannot be carried out in an accurate way. Therefore, an algorithmic solution could not be appropriate for this type of problem. The development of an expert system (ES) could be considered as appropriate in the absence of algorithmic solution of a problem, presented in this article. Therefore, expert systems (ES) are widely used in the clinical domain, facilitating the assessment of disease suspicion, its diagnosis and management and, hence, play a significant role in averting medical or clinical mistakes (Bates et al. 2001; Menachemi et al. 2007; Sim et al. 2001; Kawamoto et al. 2005). Consequently, the quality of medical healthcare systems improves significantly in the recent days (Jonsbu et al. 1993; Lin et al. 2006). Various frameworks have been evolved to construct efficient expert systems in the clinical domain with the capability of handling uncertainties. Such an expert system constructed by Shortliffe at Stanford University, which was rule-based and was used to diagnose bacterial infection (Buchanan 1984; Shortliffe 1976). This system used backward chaining as an inference procedure and used certainty factors to address uncertainties associated with incomplete knowledge. PERFEX (Herbst et al. 1992) is a rule-based expert system, containing 250 rules, in its knowledge base. There exist several other expert systems to support diagnosis, but they lack the procedures to handle different types of uncertainties, resulting in inaccurate clinical results (Piury et al. 2012; Chen et al. 2012; Issac Niwas et al. 2012; Kumar et al. 2009; Wiederhold et al. 2001; Sari et al. 2012). GDA (general discriminant analysis) and LS-SVM (least square support vector machine)-based medical expert systems also exist to diagnose lung cancer (Avci 2012). There exist other clinical domain-based expert systems to support the works of hospital emergency departments (Roukema et al. 2008; Graham et al. 2008; Kumar and Cannon 2009), intensive care units (Herbst et al. 1992; Gago et al. 2007; Mack et al. 2009), laboratories (Huang and Chen 2007; Fearn et al. 2007; Grams 1993), Bed ward (Bertsche et al. 2009) and medicine prescriptions (Lin et al. 2009). Some of these systems are knowledge-based in which domain knowledge is acquired from domain experts or medical literature, while others are non-knowledge-based which acquire large historical data (Berner and Lande 2007; Kong 2011; Spooner 2007). However, the presence of various types of uncertainties in medical domain knowledge is unavoidable and these systems are incapable of handling them. Iliad (Warner 1989) is a clinical domain-based expert system to facilitate medical decision making in differential diagnosis across the domain of internal medicine that was developed by using Bayesian probability theory. Fuzzy logic-based clinical expert systems were developed for renal transplantation assignment and renal failure disease (Yuan et al. 2002; Akgundogdu et al. 2010). However, both Bayesian probability theory and fuzzy logic are not capable of handling all types of uncertainties, especially ignorance, which can be noticed with the clinical data of ACS. In many cases ignorance may reside along with fuzziness, and therefore, both need to be treated by a single methodology.
The main components of an expert system are the knowledge base and the inference mechanism. The knowledge base can be developed by deploying different knowledge acquisition languages, consisting of PL (propositional logic), first-order predicate calculus (FOPC) and FL (fuzzy logic) (Liu et al. 1996). While forward chaining (FC) and backward chaining (BC) are deployed to construct inference mechanism (Russell and Norvig 2009), PL and FOPC are suitable to represent assertive knowledge, and hence, they are not appropriate to handle uncertain clinical knowledge, which is usually noticed in ACS. Therefore, it is inevitable to investigate a knowledge acquisition language for an expert system to assess suspicion of ACS, with the capability to capture different types of uncertainties. Likewise, the inference mechanism of such expert system should be equipped to handle uncertainties, which cannot be archived by FC and BC.
Therefore, a novel knowledge acquisition language and reasoning mechanism are necessary to handle different kinds of uncertainties existing with the signs and symptoms of ACS (Lin et al. 2006; Musen et al. 2014; Chen et al. 2013a, b). Belief rule base (BRB) can be employed to capture the uncertain clinical knowledge of ACS. Since the relationship among various signs and symptoms of ACS is nonlinear, BRB can be appropriate in this context (Liu et al. 2013; Zhou et al. 2013, 2009). Evidential reasoning (ER) can be employed as the inference engine, which has the ability to handle different types of uncertainties in an integrated framework (Yang and Singh 1994). It is interesting to note that ANN and SVM which are the two other conventional expert system tools can also address the above-mentioned uncertainties, but both systems have some flaws. These are related to dimensionality problem and handling multiple learning parameters as will be discussed in Sect. 5.
The purpose of this article is to develop belief rule-based expert system that could be employed to assess ACS suspicion. Bangladesh has been considered as the area of study to apply the BRBES.
These remaining parts of the article are organized as follows: Sect. 2 presents an overview of the signs and symptoms, associated with ACS and their uncertainties. Section 3 elaborates the framework employed to develop the BRBES. Section 4 presents the design and implementation of the expert system. Section 5 presents the results and discussion, while Sect. 6 concludes the paper.
2 Acute coronary syndrome (ACS)
The reduction of the sudden flow of blood is noticed in the heart, while a person experiences acute coronary syndrome (ACS). The disease can be experienced while a person suffers from chest pain, occurred during heart attack, at rest, or when performing light physical activity. ACS is often diagnosed in the emergency department of hospitals and is recoverable if it is diagnosed and treated immediately. However, the treatment of ACS depends on the signs, symptoms as well as the overall health condition of the patient (Cannon et al. 2001; Chai et al. 2014). This section presents the common signs and symptoms of ACS along with their associated uncertainties. The procedures to assess the suspicion of ACS are also presented.
2.1 Uncertainties of ACS signs and symptoms
Most of the cardiologists are in agreement that there is no definite reason for acute coronary syndrome (ACS) (Davidson et al. 2010). Rather it can be assessed by taking account of signs and symptoms of ACS. The symptoms consist of chest pain, arm or jaw pain, breathlessness, nausea, vomiting, sweating, dizziness and syncope (Cannon et al. 2001). However, chest pain is considered as the major symptom of ACS in both men and women (Arslanian-Engoren et al. 2006; Canto et al. 2000; DeVon and Ryan 2005; Zdzienicka et al. 2007; Davidson et al. 2010; National Collaborating Centre for Chronic Conditions 2003; Fuster and Kovacic 2014; Myers et al. 2014).
Three symptoms, namely chest pain, breathlessness and syncope, considered in this research because nausea, vomiting arm or jaw pain, sweating and dizziness are not common in every patient (Cannon et al. 2001). Breathlessness is often expressed by the patients by using linguistic terms which are subjective in nature; hence, its expression is ambiguous, vague and imprecise. Therefore, it cannot be measured with 100% accuracy. Likewise, chest pain is also expressed using linguistic terms. However, syncope is precisely defined as a transient loss of consciousness and it is expressed by the patient in terms of yes or no while asked by the physician. Therefore, in measuring the value of the three symptoms, it can be opined that breathlessness and chest pain cannot be measured with 100% certainty, while syncope can be measured with 100% certainty.
Common signs of ACS are pulse rate (heart rate), blood pressure (BP) and lung crepitation, which are observed or measured by the physician. Randomness and incompleteness are the types of uncertainties, associated with blood pressure and pulse rate. Vagueness, ignorance and randomness are the types of uncertainties, associated with the lung crepitation. There are also other signs associated with ACS such as pallor, tachycardia, raised JVP, third heart sound, quiet first heart sound and diffuse apical impulse (National Collaborating Centre for Chronic Conditions 2003), but they are not common.
2.2 ACS suspicion
The complete evaluation of the patient with ACS involves more than stating whether the syndrome is present or not; it requires consideration of the underlying abnormality of the heart, which reflects the severity of the syndrome. The diagnosis of all types of disease mainly consists of two steps, namely assessment of disease suspicion and laboratory investigation (National Collaborating Centre for Chronic Conditions 2003). Figure 1 illustrates the full diagnosis of ACS.
The spectrum of ACS [ESC guidelines] (Bassand et al. 2007)
3 Belief rule-based expert systems methodology to assess ACS suspicion
BRBESs uses the belief rule, which is the elaborated formation of conventional If-Then rule to represent the uncertain ACS clinical knowledge. Input transformation, rule activation weight calculation, belief degree update and the rule aggregation are the four inference procedures which constitute the reasoning mechanism of BRBESs (Yang et al. 2006; Kong et al. 2009). The knowledge representation and reasoning procedures along with optimal learning method are presented in this section, which are necessary to develop the BRBES to assess the ACS suspicion.
3.1 Domain knowledge representation
The knowledge representation parameters including belief degrees, rule weight and antecedent attribute weight are considered in the belief rule of a BRB. These parameters are capable of handling uncertain knowledge that exists with the ACS clinical domain. Antecedent part of a belief rule may have one or more antecedent attributes with referential values (A), while consequent part consists of only one consequent attribute with a belief structure as shown in Eq. (1).
where \(I_1\), \(I_2\), ..., \(I_k\) are the antecedent attributes of the kth rule. \(A_i^k\) (\(i=1,\ldots ,T_k\), \(k=1, \ldots , L\)) is the referential value of the ith antecedent attribute. \(O_j\) is the jth consequent reference value. \(\beta _{jk}\) (\(j=1, \ldots , N\), \(k=1, \ldots , L\)) is the degree of belief to which the consequent reference value \(O_j\) is believed to be true. If \(\sum _{j=1}^{N} \beta _{jk} = 1\) the kth rule is said to be complete; otherwise, it is incomplete. \(T_k\) is the total number of antecedent attributes used in the kth rule. L is the number of all belief rules in the BRB. An example of a belief rule by taking account of ACS is elaborated in Eq. (2).
where (Severe, 0.75), (Moderate, 0.20), (Little, 0.05) are the referential values along with belief degrees associated with the consequent attribute, which is “ACS Symptom” as elaborated in Eq. (2). The degree of belief as distributed with “Severe” is 75%, with “Moderate” is 20% and with “Little” is 5%. Since the sum of the belief degrees is 1 (0.75 + 0.20 + 0.05), the belief rule is said to be complete.
3.2 Belief rule base reasoning procedures
The reasoning or inference steps of BRBESs are presented below.
Def. Input transformation: The distribution of input data into the referential values of the antecedent attribute \(I_i\) is called input transformation as shown in Eq. (3).
H is the evaluation of the degree of belief which is set to the input value. \(A_{ij}\) (ith value) is the jth referential value of the input \(I_i\). \(\beta _{ij}\) is the degree of belief or the matching degree of the input data to the referential value \(A_{ij}\) of an antecedent attribute. The input data on the antecedent attributes are collected from the patients or from the physicians in terms of linguistic terms such as “Severe,” “Moderate” and “Little.” The degree of belief \(\epsilon _i\) is assigned from linguistic terms by considering the view of physician’s heuristics. Consequently \(\epsilon _i\) is transformed into the degree of belief associated with the various referential values \(A_{ij}\) [Severe (S), Moderate (M), Little (L)]. The utility value \(h_{ij}\) can be assigned to \(A_{ij}\). For example, “High” referential value can be assigned utility value as \(h_{i3}\) = 1.0, “Medium” as \(h_{i2}\) = 0.5 and “Low” as \(h_{i1}\) = 0. The input transformation procedure is carried out by following Eqs. (4) and (5).
The application of the above-mentioned equations can be elaborated by an example. For instance, when the input of “chest pain” attribute of ACS suspicion is found “central,” then the expert belief (\(\epsilon _i\)) for this input is acquired as 1.0. This (\(\epsilon _i\)) is to be converted into the belief degree of “chest pain” referential values by applying Eq. (4). The reason for consideration of Eq. (4) instead of Eq. (5) is that the value of expert belief (\(\epsilon _i\)) in this case is in the range of 1 \(\ge \epsilon _i \ge \) 0.5. If the value of the expert belief (\(\epsilon _i\)) is in the range of 0.5 \(\ge \epsilon _i \ge \) 0.0, then Eq. (5) should be applied. Hence, the belief degree for the referential value medium (\(h_{i2}\)) can be obtained by applying Eq. (4) as \(\beta _i2\) = (1.0 − 1.0)/(1.0 − 0.5) = 0.0, while for high (\(h_{i3}\)) it is \(\beta _{i3}\) = (1.0 − 0.0) = 1.0 and for low (\(h_{i3}\)) it is \(\beta _{i1}\) = 0.0. The transformation of the input values of ACS suspicion antecedent attributes along with expert belief degree of each input into the antecedence attributes referential values is shown in Table 1.
3.2.1 Rule activation weight calculation
The rule activation weight calculation requires the degree of matching \(\alpha _k\) of the referential value at which the belief is matched and is calculated by Eq. (6).
When the matching degrees are allotted to the referential values of the antecedent attributes of a rule, then it is said to be activated. The activation weight of such rule can be calculated by employing Eq. (7) (Yang et al. 2006).
where \(\overline{\delta _{ki}}\) is the relative weight of \(I_i\) employed in the kth rule. The rule weight of the kth rule is \(\theta _k\). The value of \(\theta _k\) is in the range between 0 and 1.
3.2.2 Rules update in BRB
There could be the case that the input data of all the antecedent attributes may not be available, which is an example of uncertainty due to ignorance. For example, ACS suspicion requires six attributes for its assessment. However, in some cases the input data related to all the attributes may not be available. In such situation, the initial belief degrees that were assigned to the consequent referential values need to be updated by Eq. (7) (Yang et al. 2006).
Here \(\overline{\beta _{ik}}\) is the original belief degree, while \(\beta _{ik}\) is the belief degree which is updated.
3.2.3 Rules aggregation using ER
The rules aggregation of BRBES inference procedures is obtained by using ER approach. This aggregation can be done by using either recursive or analytical ER approach. The analytical ER approach used in this research to reduce computational complexity (Yuan et al. 2002; Wang et al. 2006). The final conclusion C(Y) along with referential values of the consequent attribute \(O_j\) can be obtained by using Eq. (9).
where \(\beta _j\) is the belief degree related to one of the referential values of the consequent attribute, which can be calculated by employing analytical ER algorithm (Kumar et al. 2009; Kong et al. 2009) as shown in Eq. (10).
with
Training process (Yang et al. 2007)
The generated output is not a crisp or numerical value. It can be transformed into numerical value by allotting utility score to each referential value of consequent attribute (Yang et al. 2006).
where \(H(A^*)\) denotes the expected numerical value, while \(u(O_j)\) denotes the utility score of each referential value.
3.3 Optimal learning methods to train BRBES
The optimal learning model consists of finding the optimal values of the various learning parameters, such as rule weight, attribute weight and belief degrees (\(\theta _k, \delta _i, \beta _{jk}\)) in the rule of a belief rule base. Usually, the value of these parameters is acquired from the domain experts or they can be generated randomly. However, these parameters may not be optimal or 100% accurate. Therefore, the purpose of belief rule base optimal learning is to discover the optimal set of BRB learning parameters (\(\theta _k, \delta _i, \beta _{jk}\)) that will reduce the errors \(\zeta (P)\) between the BRBES results (\(y_m\)) and real system outputs (\(\overline{y}_m\)), as shown in Fig. 2. Several online and offline BRB training models can be found (Yang et al. 2007; Zhou et al. 2009, 2010). The optimal learning method to train the ACS suspicion system has been developed by following three important steps (Kong 2011) consisting of (a) construction of objective function; (b) setting constraints for the training parameters; and (c) training module to search for the optimal parameter set (\(\theta _k, \delta _i, \beta _{jk}\)). It is assumed that there are M cases in a training sample, and the input–output pairs of the M cases are (\(\overline{P}_m, \overline{y}_m )(m=1, \ldots , M\)). Details of the above steps can be found in Sect. 4.2.5.
4 BRBES to evaluate ACS suspicion
The BRBES architecture accompanied by system components is presented in this section.
4.1 System architecture
The architecture of a system refers to the fundamental organization of its various components, including input, process and output (Pressman 2005). Our BRBES adopts a three-layer architectural style, consisting of an interface layer, an application layer and a data management layer. The interface layer is designed to acquire antecedent attribute values from the ACS patients or from the physicians and to display results of the ACS suspicion. The application layer comprises inference procedures, which are input transformation, rule activation weight calculation, belief update and rule aggregation by using evidential reasoning (ER). In addition to these, the application layer also contains training module. The data management layer contains initial BRB as well as clinical facts such as signs and symptoms of ACS. The main components of the three-layered architecture of the BRBES are depicted in Fig. 3.
MySQL, an open-source relational database management system, is used at the back end to store and manipulate the initial BRB, which represents the knowledge base of the system. MySQL was chosen due to its flexibility and portability. Java 2 Platform, Standard Edition (J2SE), available in Netbean 7.2, was used to develop the user interface and the components of the application layer. Java was chosen because of its platform independence, flexibility and robustness. The system architecture at implementation level is depicted in Fig. 4.
4.2 System components
This section presents the ACS clinical facts and knowledge base, BRBESs inference method and its interface along with its training procedures.
4.2.1 ACS initial facts
ACS initial facts can be divided into two categories, namely symptoms and signs. Three symptoms consisting of chest pain (A1), Breathlessness (A2), Syncope (A3) and three signs consisting of Pulse Rate (A4), blood pressure (A5), and lung crepitation (A6) have been considered as the input parameters of the BRBES. These signs and symptoms are viewed as the antecedent attributes of the different rules of a belief rule base. The input data of these signs and symptoms are either acquired from the patient or from the physicians as shown in Table 1. This input data are then converted into the belief of the expert cardiologist. By using Eqs. (4) and (5), the expert belief is allotted over each referential value of an antecedent attribute, e.g., the input for antecedent attribute “chest pain” is collected from the patient as “Central” and expert belief against this is found as “1.” This belief is then distributed over the each referential value of this antecedent attribute as shown in Table 1.
4.2.2 BRBES knowledge base
A belief rule base schema has been formulated to build the knowledge base for the BRBES as shown in Fig. 5. A BRB can be established in four different ways (Xu et al. 2007), namely by converting expert knowledge into belief rules, by examining historical data to identify belief rules, by using available previous rule base and by creating rules randomly without any prior knowledge. By taking into account knowledge acquired from cardiologists, the initial BRB has been constructed in this article. The BRB framework, as shown in Fig. 5, encapsulates the factors, which are necessary to determine the ACS suspicion. These factors are related to signs (A4–A6) and symptoms (A1–A3) of ACS. This BRB framework represents three BRBs, namely ACS Symptoms (A7), ACS signs (A8) and ACS (A9). The “ACS Symptom BRB” consists of three antecedent attributes each with three referential values. Therefore, “ACS Symptom BRB” comprises 27 rules as represented in Table 2. The “ACS signs BRB” has three antecedent attributes (A4–A6) with three values each; hence, it consists of 27 rules as illustrated in Table 3. The “ACS Suspicion BRB” has two antecedent attributes with three referential values each; hence it comprises 9 rules as presented in Table 4. An equal rule weight (i.e., “1”) and equal antecedent attribute weight (i.e., “1”) have been considered for all the belief rules. The initial belief degrees allocated to the referential values of the consequent attribute of a belief rule has been carried out by taking opinions of two cardiologists. The referential values for each consequent attribute consist of High (H), Medium (M) and Low (L).
An instance of a belief rule as appeared in Table 2 is elaborated.
R2: IF chest pain is “H” AND Breathlessness is “H” AND Syncope is “H” THEN ACS symptom is H (0.80), M (0.20), L (0.00)
The belief degrees are embedded in each referential value of the consequent attribute as shown in the above belief rule. This belief degree is embedded by taking account of the signs and symptoms data of the patient. For example, if the survey of the symptoms such as “Breathlessness,” “Syncope” or “chest pain” of a patient with ACS appears as “High,” then the degree of belief associated with the “High” referential value of the consequent attribute “ACS Symptom” should be calculated as 1. For other referential values of the “ACS Symptom” it should be calculated as “0” for Medium as well as “0” for “Low” as illustrated in rule R1 of Table 2.
4.2.3 Inference engine using ER approach
The inference engine of this BRBES used evidential reasoning (ER) algorithm (Yang 2001; Yang and Sen 1994) as mentioned in Sect. 3.2. The procedures of inference engine comprise: firstly it reads signs and symptoms data either from patients or from the physicians; secondly, these data are then converted into matching degree by employing Eqs. (4) and (5); thirdly, the activation weight of each rule is calculated by employing Eq. (6); fourthly, in case of the presence of ignorance the belief degrees are updated by employing Eq. (7); finally, rules are aggregated by employing Eqs. (8) and (9).
4.2.4 System interface
Figures 6 and 7 illustrate the graphical user interface (GUI) of the BRBES. The figures allow the capturing of the data related to the signs and symptoms of ACS as well as the displaying of the BRBESs results.
For example, Figs. 6 and 7 illustrate the outputs for the input data (A1 = “Central,” A2 = “Medium,” A3 = “Yes,” A4 = “Normal,” A5 = “Mild,” and A6 = “Yes”). From the figures it can be noticed that the degree of belief obtained for the referential values of the consequent attribute “Symptom” of the ACS Symptom BRB is H (53.14%), M (46.85%), L (0%) for the leaf node values (A1–A3) obtained from the patient. Likewise, the belief degree for the referential values of the consequent of ACS sign BRB is H (40.52%), M (18.54%), L (40.93%) for the input value of the leaf node values (A4–A6), received from the physician. In this way, the system acquires clinical data (signs and symptoms) of ACS from both the patient and the physician. Figures 6 and 7 also illustrate the overall assessment of ACS suspicion which is H (60.10%), M (32.79%), L (7.11%). This is transformed into a numerical value by using Eq. (10), which is 76.49% as shown in Figs. 6 and 7.
4.2.5 BRBES training model
The BRBESs optimal training model is introduced, in Sect. 3.3. The total mean square error is considered as the objective function of the training model.
Since an explicit ER aggregation function is required in BRB training, analytical ER algorithm Eq. (9) is used to construct the objective function in the training model. The BRBES consists of three rule bases [ACS Symptoms (A7), ACS signs (A8), ACS (A9)] and each of them should be trained individually. Three different sets of training parameters, which are given below, have been considered for training the BRBs. The same objective function for each training round is used.
T1: Training with rule weight (\(\theta _k\)), antecedent attribute weight (\(\delta _k\)), consequent belief degrees [\(\beta _{jk} (j=1, 2, 3, k=1, \ldots , L\))], considered for A7 and A8 BRBs. However, for A9 BRB in addition to the above parameters, another additional training parameter, known as severity score, has been considered.
T2: Training by employing antecedent attribute weight and the degree of belief.
T3: Training by employing rule weight and the degree of belief.
The following constraints and initial values for each of the parameter have been considered:
-
1.
Severity scores of three reference levels \(\mu (O_j) (j=1, \ldots , 3)\):
\(1\ge \mu (O_j) (j=1, \ldots , 3) \ge 0;\)
\(\mu (O_1(\text {High}))\ge \mu (O_2(\text {Medium})) \ge \mu (O_3(\text {Low}));\)
\(\mu (O_1(\text {High}))=1, \mu (O_2(\text {Medium}))=0.5, \mu (O_3(\text {Low}))=0;\)
-
2.
Rule weights \(\theta _k (k = 1, \ldots , L):\)
\(1 \ge \theta _k (k = 1, \ldots , L) \ge 0.01\)
\(\theta _k (k = 1, \ldots , L)= 1;\)
-
3.
Antecedent attribute weights \(\delta _k, k \in \{1, \ldots , L\}\):
\(1 \ge \delta _k, k \in \{1, \ldots , L \} \ge 0 ,\)
\(\delta _k, k \in \{1, \ldots , L\} = 1;\)
-
4.
Consequent belief degrees \(\beta _{jk} (j=1, \ldots , 3, k=1, \ldots , L)\):
\(1\ge \beta _{jk} (j=1, \ldots , 3, k=1, \ldots , L)\ge 0 \text { and } 1 \ge \sum _{j=1}^{3}\beta _{jk} (k=1, \ldots , L) \ge 0.\)
The final step of the training module includes obtaining the optimal value of the training parameters by using the fmincon function available in MATLAB. The function is fed by M training datasets along with initial values of the training parameters. The output of the function consists of the set of optimal parameter values which will reduce the uncertainty associated with the parameter values used in the initial rule base. In this way, for each BRB as mentioned above, optimal values of the parameters have been obtained.
5 Results and discussion
Patients with cardiac chest pain at various hospitals located in the Chittagong District of Bangladesh have been selected as the clinical targets to assess their ACS suspicion. The data used to assess ACS suspicion are collected from the survey of signs and symptoms of 250 patients with chest pain. Usually, when patients experience chest pain, the clinician collects data on their signs and symptoms. The BRBES has used the acquired data of the patients as input data as shown in Table 5.
Column 12 of Table 5 illustrates the assessment of the suspicion of ACS, carried out by the cardiologist by taking account of the same data of the patients, which can be considered as a manual or traditional system. Column 4 presents the benchmark results which have been obtained by using appropriate laboratory investigations of the same patients. The result is considered as “1” when the patient has ACS and “0” when the patient has no ACS. For simplicity, only data of ten patients are shown in Table 5. The data on the six parameters (as shown in Fig. 5; Table 5) have been collected from 250 patients with cardiac chest pain. Column 11 of Table 5 demonstrates the percentage of ACS suspicion generated by the BRBES by taking account of the same data on the six parameters.
Utility Eq. (10) has been considered in calculating the percentage of ACS suspicion. It is interesting to observe that “Assessment of ACS Suspicion,” which is the final consequent attribute, comprises three referential values, namely “High,” “Medium” and “Low.” Since belief degree obtained each referential values from the BRBES, in order to get the overall score of the ACS suspicion in terms of percentage, utility values are assigned to the referential values. For instance, a utility value of 100% is allotted to “High,” 50% allotted to “Medium” and 0% allotted to “Low.” After allotting this utility value to the referential values, the percentage of ACS suspicion has been calculated by using Eq. (10). For instance, the final result of the ACS suspicion, which is 99.69%, can be achieved by employing belief degrees related to each referential value, such as for “High” it is 0.94, for “Medium” it is 0.04, while for “Low” it is 0.00.
The BRBES, developed in this research, has also been compared with two other conventional expert system tools, namely artificial neural networks (ANN) and support vector machine (SVM).
The receiver operating characteristic (ROC) curves are widely used to analyze the effectiveness of the diagnostic test (Body 2009). The ROC can be employed to compare the outputs of BRBES against the outputs of manual system as well as against the ANN and SVM in the light of baseline data. The size of the area under curve (AUC) is used to determine the accuracy of the outputs (Body 2009; Hanley 1988; Metz 1978; Skalská and Freylich 2006; DeLong et al. 1988; Lansky et al. 2014). Higher values of AUC determine better accuracy of the outputs. Column 11 of Table 5 shows the BRBES generated output of assessment of ACS suspicion, while column 12 shows the manual system-generated outputs by taking account of the same patients data. Column 11 of Table 6 shows the artificial neural networks (ANN) generated results of ACS suspicion, while column 12 shows the support vector machine (SVM) generated results by taking account of the same patient data.
A comparison of performance in predicting the ACS suspicion among BRBES, manual system, artificial neural networks (ANN) and support vector machine (SVM) is demonstrated in Fig. 8. The AUC of BRBES is found to be 0.974 with 95% confidence intervals (0.950–1.009), while AUC for manual system is found to be 0.900 with 95% confidence intervals (0.914–1.006). On the other hand, AUC for SVM is found to be 0.9560 with 95% confidence intervals (0.90–1.005) and the AUC for ANN is obtained as 0.9360 with 95% confidence intervals (0.89–1.003). A comparison of AUCs of the four different systems indicates that the performance of the BRBES is better than that of manual system, SVM and ANN because of its greater AUC value. Therefore, the outputs generated by the BRBES are reliable than that of outputs produced by the other three systems. Inference with BRBES is implemented using the evidential reasoning approach (ER), which can process various types of uncertain information. The learning of BRBES depends on various factors that comprises rule weight, attribute weight and belief degrees. Moreover, BRBES facilitates the multidimensionality of a problem (Zhou et al. 2010). On the other hand, ANN suffers from increasing dimensionality and it has also limited learning parameters (Patra and Bruzzone 2012). On the contrary, SVM does not have dimensionality issue but it has limited number of learning parameters which diminish the performance of the system that depends on multiple learning parameters (Wu et al. 2008; Kong et al. 2008). Therefore, the outputs produced by the BRBES are more dependable than from other methods such as ANN and SVM.
The ROC curves have been developed by employing SPSS 11.5 and also to measure the AUC values. The data of two hundred and fifty patients have been collected to validate the system, and these data have been separated into training data and the test data to carry the task of training. By employing the training data the system has been trained while by employing the test data the performance of the BRBES has been carried out. These two hundred and fifty patients input data on signs and symptoms have been used in the BRBES before training to obtain a “Pre-Trained BRB Result.” Another three sets of results for each training parameter set, namely T1, T2 and T3, were generated by the system using the test dataset. These three sets of results are denoted as “After Trained T1,” “After Trained T2” and “After Trained T3.” Based on the mentioned results, four ROC curves have been plotted using SPSS as illustrated in Fig. 9. Table 7 shows the AUC values for all the four ROC curves. The performance of T1 and T3 is better than the “Pre-Trained BRB Result” since their AUC values are greater as shown in Table 7. The AUC value of the ROC curve of T3 is the largest among all the AUC values. Therefore, the learning with rule weight and degree of belief (T3) will produce a reliable result.
6 Conclusion
A BRBES to evaluate the suspicion of ACS by taking account of its signs and symptoms is presented. The BRBES is embedded with belief rule base as well as with a novel inference procedures with the capability to handle different types of uncertainty. Eventually, such an expert system can assess the ACS suspicion with accuracy and rigor. Consequently, the application of BRBES could reduce the cost of laboratory investigation and allowing the patient to take precautionary steps in advance. It has been demonstrated that the outputs created from the BRBES are more dependable than from manual system, ANN and SVM. This BRBES can only be employed to evaluate the suspicion of ACS but should not be considered as a tool to perform the complete tasks of diagnosis. However, in future steps will be considered to improve the capability of BRBES, enabling the diagnosis of ACS likelihood. The knowledge representation parameters such as rule weight, attribute weights and degree of beliefs, associated with the consequents referential value of a rule, need to be trained by considering real clinical data and by using optimal learning methods introduced in Sect. 3. This will allow the belief rule-based expert system to be capable of learning and updating its knowledge base. Therefore, real data will be used to train the system in future research and it will be validated against the expert opinion by considering some baseline data.
References
Akgundogdu A, Kurt S, Kilic N, Ucan ON, Akalin N (2010) Diagnosis of renal failure disease using adaptive neuro-fuzzy inference system. J Med Syst 34(6):1003–1009
Arslanian-Engoren C, Patel A, Fang J, Armstrong D, Kline-Rogers E, Duvernoy CS, Eagle KA (2006) Symptoms of men and women presenting with acute coronary syndromes. Am J Cardiol 98(9):1177–1181
Avci E (2012) A new expert system for diagnosis of lung cancer: GDA-LS_SVM. J Med Syst 36(3):2005–2009
Bassand J-P, Hamm CW, Ardissino D, Boersma E, Budaj A, Fernández-Avilés F, Fox KA, Hasdai D, Ohman EM, Wallentin L, Wijns W (2007) Guidelines for the diagnosis and treatment of non-ST-segment elevation acute coronary syndromes. Eur Heart J 28(13):1598–1660
Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T (2001) Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc 8(4):299–308
Berner ES, La Lande TJ (2007) Overview of clinical decision support systems. In: Berner ES (ed) Clinical decision support systems: theory and practice, 2nd edn. Springer, New York
Bertsche T, Askoxylakis V, Habl G, Laidig F, Kaltschmidt J, Schmitt SP, Ghaderi H, Bois AZ, Milker-Zabel S, Debus J, Bardenheuer HJ, Haefeli WE (2009) Multidisciplinary pain management based on a computerized clinical decision support system in cancer pain patients. Pain 147(1–3):20–28
Body R (2009) Clinical decision rules to enable exclusion of acute coronary syndromes in Emergency Department patients with chest pain. Manchester Metropolitan University, Manchester, UK, Faculty of Health, Psychology and Social Care
Buchanan B, Shortliffe E (1984) Rule-based expert systems: the MYCIN experiments of the stanford heuristic programming project. Addison-Wesley, Reading, Massachusetts
Cannon CP, Battler A, Brindis RG, Cox JL, Ellis SG, Every NR, Flaherty JT, Harrington RA, Krumholz HM, Simoons ML, De V, Werf FJ, Weintraub WS, Mitchell KR, Morrisson SL, Brindis RG, Anderson HV, Cannom DS, Chitwood WR, Cigarroa JE, Collins-Nakai RL, Ellis SG, Gibbons RJ, Grover FL, Heidenreich PA, Khandheria BK, Knoebel SB, Krumholz HL, Malenka DJ, Mark DB, Mckay CR, Passamani ER, Radford MJ, Riner RN, Schwartz JB, Shaw RE, Shemin RJ, Van FDB, Verrier ED, Watkins MW, Phoubandith DR, Furnelli T (2001) American College of Cardiology key data elements and definitions for measuring the clinical management and outcomes of patients with acute coronary syndromes. A report of the American College of Cardiology task force on clinical data standards (acute coronary syndromes writing committee). J Am Coll Cardiol 38(7):2114–2130
Canto JG, Shlipak MG, Rogers WJ, Malmgren JA, Frederick PD, Lambrew CT, Ornato JP, Barron HV, Kiefe CI (2000) Prevalence, clinical characteristics and mortality among patients with Atypical symptoms of ACS 171 myocardial infarction presenting without chest pain. JAMA 283(24):3223–3229
Chai R, Ling SH, Hunter GP, Tran Y, Nguyen HT (2014) Brain–computer interface classifier for wheelchair commands using neural network with fuzzy particle swarm optimization. IEEE J Biomed Health Inform 18(5):1614–1624
Chen Y-W, Yang J-B, Xu D-L (2013a) Uncertain nonlinear system modeling and identification using belief rule-based systems. In: Proc. IUKM 2013, pp 209–218
Chen Y-W, Yang J-B, Xu D-L, Yang S-L (2013b) On the inference and approximation properties of belief rule based systems. Inf Sci 234:121–135
Chen HL, Yang B, Wang G, Liu J, Chen YD, Liu DY (2012) A three-stage expert system based on support vector machines for thyroid disease diagnosis. J Med Syst 36(3):1953–1963
Davidson S, Walker BR (ed), Ralston SH (ed), Colledge NR (2010) Davidsons principles and practice of medicine, 21st edn, Chapter 18. ISBN-13: 978-0-7020-3084-0
DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44(3):837–845
DeVon HA, Ryan CJ (2005) Chest pain and associated symptoms of acute coronary syndromes. J Cardiovasc Nurs 20(4):232–238
Fearn P, Regan K, Sculli F, Fajardo J, Smith B, Alli P (2007) Lessons learned from caisis: an open source, web-based system for integrating clinical practice and research. In: Proc. CBMS ’07
Fuster V, Kovacic JC (2014) Acute coronary syndromes: pathology, diagnosis, genetics, prevention, and treatment. Circ Res 114(12):1847–1851
Gago P, Silva Á, Santos MF (2007) Adaptive decision support for intensive care. Proc EPIA 2017:415–425
Graham TA, Bullard MJ, Kushniruk AW, Holroyd BR, Rowe BH (2008) Assessing the sensibility of two clinical decision support systems. J Med Syst 32(5):361–368
Grams RR (1993) Clinical laboratory test reference (CLTR). J Med Syst 17(2):59–67
Hanley JA (1988) The robustness of the “Binormal” assumptions used in fitting ROC curves. Med Decis Making 8(3):197–203
Herbst MD, Garcia EV, Cooke CD, Ezquerra NF, Folks RD, DePuey EG (1992) Myocardial ischemia detection by expert system interpretation of thallium-201 tomograms. In: Reiber JHC, van der Wall EE (eds) Cardiovascular nuclear medicine and MRI. Kluger Academic Publishers, Dordrecht, pp 77–88
Huang M-J, Chen M-Y (2007) Integrated design of the intelligent web-based Chinese Medical Diagnostic System (CMDS)—systematic development for digestive health. Expert Syst Appl 32(2):658–673
Issac Niwas S, Palanisamy P, Chibbar R, Zhang WJ (2012) An expert support system for breast cancer diagnosis using color wavelet features. J Med Syst 36(5):3091–3102
Jonsbu J, Aase O, Rollag A, Liestøl K, Erikssen J (1993) Prospective evaluation of an EDB-based diagnostic program to be used in patients admitted to hospital with acute chest pain. Eur Heart J 14(4):441–446
Kawamoto K, Houlihan CA, Balas EA, Lobach DF (2005) Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 330:765–772
Kong G (2011) An online belief rule-based group clinical decision support system, Doctor of Philosophy Thesis, Manchester Business School, University of Manchester, UK
Kong G, Xu D-L, Yang J-B (2008) Clinical decision support systems: a review on knowledge representation and inference under uncertainties. Int J Comput Intell Syst 1(2):159–167
Kong GL, Xu DL, Yang JB (2009) An evidence-adaptive belief rule-based decision support system for clinical risk assessment in emergency care. In: Proc. 23rd European Conference on Operational Research, Bonn, Germany
Kumar A, Cannon CP (2009) Acute coronary syndromes: diagnosis and management, part I. Mayo Clin Proc 84(10):917–938
Kumar KA, Singh Y, Sanyal S (2009) Hybrid approach using case-based reasoning and rule-based reasoning for domain independent clinical decision support in ICU. Expert Syst Appl 36(1):65–71
Lansky AJ, Ng VG, Meller S, Xu K, Fahy M, Feit F, Ohman EM, White HD, Mehran R, Bertrand ME, Desmet W, Hamon M, Stone GW (2014) Impact of nonculprit vessel myocardial perfusion on outcomes of patients undergoing percutaneous coronary intervention for acute coronary syndromes: analysis from the ACUITY trial (acute catheterization and urgent intervention triage strategy). JACC Cardiovasc Interv 7(3):266–275
Lin L, Hu PJ-H, Sheng ORL (2006) A decision support system for lower back pain diagnosis: uncertainty management and clinical evaluations. Decis Support Syst 42(2):1152–1169
Lin C, Lin CM, Lin B, Yang M-C (2009) A decision support system for improving doctors’ prescribing behavior. Expert Syst Appl 36(4):7975–7984
Liu TI, Singonahalli JH, Iyer NR (1996) Detection of roller bearing defects using expert system and fuzzy logic. Mech Syst Signal Process 10(5):595–614
Liu J, Chen S, Martinez L, Wang H (2013) A belief rule-based generic risk assessment framework. Decis Aid Models Disaster Manag Emergencies Atl Comput Intell Syst 7:145–169
Mack EH, Wheeler DS, Embi PJ (2009) Clinical decision support systems in the pediatric intensive care unit. Pediatr Crit Care Med 10(1):23–28
Mark DB, Talley JD, Topol EJ, Bowman L, Lam LC, Anderson KM, Jollis JG, Cleman MW, Lee KL, Aversano T, Untereker WJ, Davidson-Ray L, Califf RM (1996) Economic assessment of platelet glycoprotein IIb/IIIa inhibition for prevention of ischemic complications of high-risk coronary angioplasty. EPIC Investigators. Circulation 94(4):629–635
Menachemi N, Saunders C, Chukmaitov A, Matthews MC, Brooks RG (2007) Hospital adoption of information technologies and improved patient safety: a study of 98 hospitals in Florida. J Healthc Manag 52(6):398–410
Metz CE (1978) Basic principles of ROC analysis. Semin Nucl Med 8(4):283–298
Murray CJ, Lopez AD (1997) Alternative projections of mortality and disability by cause 19902020: global burden of disease study. Lancet 349(9064):1498–1504
Musen MA, Middleton B, Greenes RA (2014) Clinical decision-support systems. In: Shortliffe E, Cimino J (eds) Biomedical informatics. Springer, London
Myers J, de Souza CR, Borghi-Silva A, Guazzi M, Chase P, Bensimhon D, Peberdy MA, Ashley E, West E, Cahalin LP, Forman D, Arena R (2014) A neural network approach to predicting outcomes in heart failure using cardiopulmonary exercise testing. Int J Cardiol 171(2):265–269
National Collaborating Centre for Chronic Conditions (2003) chronic heart failure: national clinical guideline for diagnosis and management in primary and secondary care. national collaborating centre for chronic conditions, Royal College of Physicians (UK). ISBN 1-86016-188-X
Patra S, Bruzzone L (2012) A batch-mode active learning technique based on multiple uncertainty for SVM classifier. IEEE Geosci Remote Sens Lett 9(3):497–501
Piury J, Laita LM, Roanes-Lozano E, Hernando A, Piury-Alonso F-J, Gómez-Argüelles JM, Laita L (2012) A Gröbner bases-based rule based expert system for fibromyalgia diagnosis. RACSAM 106(2):443–456
Pressman RS (2005) Software Engineering: a Practitioners Approach, 5th edn. McGraw-Hill Series in Computer Science, pp 373–374. ISBN 0-07-365578-3
Reason J (2001) Understanding adverse events: the human factor. In: Charles V (ed) Clinical risk management: enhancing patient safety, 2nd edn. BMJ publishing house, London
Roukema J, Steyerberg EW, van der Lei J, Moll HA (2008) Randomized trial of a clinical decision support system: impact on the management of children with fever without apparent source. J Am Med Inform Assoc 15(1):107–113
Russell S, Norvig P (2009) Artificial intelligence: a modern approach, 3rd edn. Prentice Hall, Upper Saddle River , ISBN 0-13-604259-7
Sari M, Gulbandilar E, Cimbiz AJ (2012) Prediction of low back pain with two expert systems. J Med Syst 36(3):1523–1527
Shortliffe EH (1976) Computer-based medical consultations: MYCIN. Elsevier, New York
Sim I, Gorman P, Greenes RA, Haynes RB, Kaplan B, Lehmann H, Tang PC (2001) Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 8(6):527–534
Skalská H, Freylich V (2006) Web-bootstrap estimate of area under ROC curve. Austrian J Stat 35(2–3):325–330
Spooner SA (2007) Mathematical foundations of decision support systems. In: Berner ES (ed) Clinical decision support systems: theory and practice, 2nd edn. Springer, New York
Wang Y-M, Yang J-B, Xu D-L (2006) Environmental impact assessment using the evidential reasoning approach. Eur J Oper Res 174(3):1885–1913
Warner HR Jr (1989) Iliad: moving medical decision-making into new frontiers. Methods Inf Med 28(4):370–372
Weintraub WS, Mauldin PD, Becker E, Kosinski AS, King SB III (1995) A comparison of the costs of and quality of life after coronary angioplasty or coronary surgery for multivessel coronary artery disease. Results from the emory angioplasty versus surgery trial (EAST). Circulation 92(10):2831–2840
Wiederhold G, Fagan L, Shortliffe E, Perreault L (2001) Medical informatics: computer applications in health care and biomedicine, 2nd edn. Springer, New York, p 854
Wu T-K, Huang S-C, Meng Y-R (2008) Evaluation of ANN and SVM classifiers as predictors to the diagnosis of students with learning disabilities. Expert Syst Appl 34(3):1846–1856
Xu D-L, Liu J, Yang J-B, Liu G-P, Wang J, Jenkinson I, Ren J (2007) Inference and learning methodology of belief-rule-based expert system for pipeline leak detection. Expert Syst Appl 32(1):103–113
Yang J-B (2001) Rule and utility based evidential reasoning approach for multi-attribute decision analysis under uncertainties. Eur J Oper Res 131(1):31–61
Yang J-B, Sen P (1994) A general multi-level evaluation process for hybrid MADM with uncertainty. IEEE Trans Syst Man Cybern 24(10):1458–1473
Yang J-B, Singh MG (1994) An evidential reasoning approach for multiple-attribute decision making with uncertainty. IEEE Trans Syst Man Cybern 24(1):1–18
Yang JB, Liu J, Wang J, Sii H-S, Wang H-W (2006) Belief rule-base inference methodology using the evidential reasoning approach-RIMER. IEEE Trans Syst Man Cybern Part A Syst Hum 36(2):266–285
Yang JB, Liu J, Xu DL, Wang J, Wang HW (2007) Optimal learning method for training belief rule based systems. IEEE Trans Syst Man Cybern Part A Syst Hum 37(4):569–585
Yuan Y, Feldhamer S, Gafni A, Fyfe F, Ludwin D (2002) The development and evaluation of a fuzzy logic expert system for renal transplantation assignment: is this a useful tool? Eur J Oper Res 142(1):152–173
Zdzienicka J, Siudak Z, Zawiślak B, Dziewierz A, Rakowski T, Dubiel J, Dudek D (2007) Patients with non-ST-elevation myocardial infarction and without chest pain are treated less aggressively and experience higher in-hospital mortality. Kardiol Pol 65(7):769–775
Zhou Z-J, Hu C-H, Yang J-B, Xu D-L, Zhou D-H (2009) Online updating belief rule based system for pipeline leak detection under expert intervention. Expert Syst Appl 36(4):7700–7709
Zhou Z-J, Hu C-H, Yang J-B, Xu D-L, Chen M-Y, Zhou D-H (2010) A sequential learning algorithm for online constructing belief-rule-based systems. Expert Syst Appl 37(2):1790–1799
Zhou Z-G, Liu F, Jiao L-C, Zhou Z-J, Yang J-B, Gong M-G, Zhang X-P (2013) A bi-level belief rule based decision support system for diagnosis of lymph node metastasis in gastric cancer. Knowl Based Syst 54:128–136
Acknowledgements
We want to thank the Swedish Research Council (Grant No. 2014-4251) for supporting this research.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
All authors declare that they have no conflict of interest.
Human and animal rights
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Communicated by V. Loia.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Hossain, M.S., Rahaman, S., Mustafa, R. et al. A belief rule-based expert system to assess suspicion of acute coronary syndrome (ACS) under uncertainty. Soft Comput 22, 7571–7586 (2018). https://doi.org/10.1007/s00500-017-2732-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-017-2732-2