Artificial Intelligence Compared to Alvarado Scoring System Alone or Combined with Ultrasound Criteria in the Diagnosis of Acute Appendicitis

Appendices

Appendix 1

US Criteria of Acute Appendicitis

• Suggestive US examination implied direct visualization of the inflamed appendix with an outer diameter > 6 mm and/or appendicular wall thickness > 3 mm, distended, non-compressible appendicular lumen measuring up to 2 cm; with or without associated secondary signs such as right lower quadrant inflammation, intraperitoneal collection, or fluid, reflective omentum around the appendix, thickened bowel, and enlarged lymph nodes.

• Inconclusive US examination implied that the appendix was not identified but secondary ultrasound signs of appendicitis were present.

• Normal US examination implied non-visualization of the appendix and absence of secondary signs.

Appendix 2

Stages of Development of The AI-based Model

The predictive model based on AI was developed through the following steps:

• 1- Data Pre-processing

Patients’ data were preprocessed to be compatible with machine learning. The dataset was randomly divided into 224 cases (70%) for the training dataset and 95 cases (30%) for the testing dataset (Supplementary Fig. 1). Owing to the relatively low rate of negative appendectomy in the cohort studies, oversampling of the negative cases was done until a ratio of 1:1 was achieved to avoid developing a skewed model. Furthermore, splitting the data into training and testing groups was undertaken based on positive and negative diagnosis to ensure the equality in both datasets.

To overcome the overfitting of the predictive model, the training dataset was cross-validated with 5 folds. During the process of internal validation, the training dataset was randomly categorized into 5 partitions; 4 for training and one for the cross-validation.

• 2- Variables included to the model

The initial step for machine learning used all the available variables including:

• Patients’ characteristics: age, gender, marital status, obesity.

• Chronic illnesses: diabetes mellitus, hypertension, hepatitis C or B, and autoimmune diseases.

• Clinical examination parameters: history of similar pain, body temperature, anorexia, nausea, vomiting, site and duration of pain (1–2 days, 2–4 days, ≥ 5 days).

• Laboratory parameters: serum hemoglobin and total leukocyte count.

• Findings of US performed by a qualified radiologist as normal, inconclusive, or suggestive of appendicitis.

• Machine learning process

Different learning algorithms were used and the best model that showed maximum performance was chosen for the further steps of machine learning. The performance was judged based on the accuracy rate and the area under the receiver operator curve (AUC, ROC).

Dimensionality reduction was applied through Principal Component Analysis (PCA) function for more precise feature selection. PCA is a quantitatively rigorous method that generates a new set of variables called principal components. Each principal component is a linear combination of the original variables. Furthermore, the created model underwent an optimization process to minimize the prediction error. The testing dataset acted as the external validator for the created AI-based decision-making model.

Outcome of Training of The AI Decision-Making Model

During the machine learning process, seven classifier learners, including 21 different methods, were applied to the training dataset at the same training session. The most successful algorithm was Ensemble (Subspace KNN), with an accuracy of 91.1% and AUC 0.82 (Supplementary Fig. 2). All the involved machine learning algorithms are shown in Supplementary Fig. 3.

Although PCA function was applied for more accurate feature selection, no variables were excluded. Thus, the importance of the constructing variables of the AI model were ranked in order to determine how these variables were making sense to the model to accurately set its final decision (Supplementary Fig. 4). Furthermore, for optimizing the created model, Ensemble Bag method was used with 30 iterations. The minimum estimated diagnostic classification error was 0.129 (Supplementary Fig. 5).

Statistical Analysis

Categorical variables were expressed as frequencies and percentages, while continuous variables as mean ± standard deviation (SD). The diagnostic accuracy was presented as sensitivity, specificity, and accuracy. Statistical analyses were performed using IBM SPSS software (Version 25.0. Armonk, NY) and MATLAB and Statistics Toolbox Release 2020a (The MathWorks, Inc., Natick, Massachusetts, USA) for machine learning and building up the AI-based platform.

Table 2 Baseline characteristics of patients in the training and testing datasets