Introduction

Acute ischemic stroke (AIS), is the fifth leading cause of death in the United States1, and the leading cause of death in China2. AIS imposes a very heavy burden on high-income countries with the rapid development of social economy. Furthermore, the corresponding burden is increasing rapidly in low-income and middle-income countries3. The identification of risk factors associated with poor prognosis in AIS patients could assist clinicians to provide close surveillance and timely intervention for high-risk stroke patients with poor prognosis and may guide the implementation of strategies in organized stroke care provision, thereby optimizing clinical outcomes. Machine learning (ML) algorithms can efficiently identify features highly correlated to outcomes from a large number of features, outperforming traditional statistical methods4,5. Consequently, ML can be used to improve the prediction accuracy of prognosis.

ML algorithms using big data have improved and optimized the predictive performance of the prognosis prediction, as reported by prior literature6,7,8. Therefore, the goal in this retrospective study was to identify factors associated with prognosis for patients with stroke and develop an ML-based predictive model. Moreover, the proposed model was integrated in an online tool to provide clinicians and patients with visual and practical predictive assessment.

Materials and methods

Data source and collection

Retrospective data were obtained from electronic health record of the Second Affiliated Hospital of Xuzhou Medical University. Patients diagnosed with AIS from August 2017 to July 2019 were enrolled in this study cohort. Inclusion criteria were: The diagnosis of AIS met the requirements of the World Health Organization with symptom onset less than 24 h9. Exclusion criteria: (1) Incomplete clinical data. (2) Patients with severe abnormal organ function. (3) Inadequate auxiliary examinations. (4) Follow-up less than one year. The diagnosis process of AIS was completed and confirmed independently by two physicians. In situations where there was diagnostic disagreement, the final diagnosis was reviewed with a senior physician to reach a consensus. This study was approved by the Ethics Committee of the Second Affiliated Hospital of Xuzhou Medical University, and all studies were conducted in accordance with relevant guidelines/regulations and the Declaration of Helsinki, and informed consent was obtained from all participants.

Data processing and variable selection

In the data cleaning process, the information of subjects with missing value was deleted, and normality tests were performed for the continuous numerical variables to assess the presence of outliers. Subsequently, the information of subjects with outliers was deleted. Ultimately, the current study included a total of 677 subjects in the database. The data were collected by trained and qualified members of the research group using a uniformly designed questionnaire, which included living area, occupation, education level, family economic status, eating habits (alcohol intake, drinking sugary drinks, smoking, high-fat diet, etc.), lifestyle and habits, disease medication, menstrual history (female) and psychosocial factors. First, the purpose and significance of the survey were explained to the patients. On the basis of patients fully understanding, informed consent forms were signed, and the patients were asked to read the guidance on the questionnaire carefully and fill it in. Patients with reading difficulties, such as illiteracy or poor eyesight, were aided by members of the research group and the contents of the questionnaire were read to them to help them fill it in. The type and severity of stroke were evaluated by clinical symptoms, head CT, MR, and angiography results. Blood lipid, blood glucose, homocysteine, and other data were obtained by laboratory tests. Blood pressure, weight, height, time of onset, and baseline National Institutes of Health Stroke Scale (NIHSS) score were also recorded. Drug use, stroke recurrence and clinical prognosis were recorded during follow-up. Dysphagia was assess by water swallowing test according to the criteria10. Clinical prognosis was assessed according to the modified Rankin Scale (mRS) according to the results of 1-year follow-up. The mRS ≥ 3 was classified as poor prognosis and mRS < 3 was classified as good prognosis. All detection indexes were completed by the Second Hospital of Xuzhou Medical University. The flowchart of data collection is shown in Fig. 1.

Figure 1
figure 1

The flowchart of the patient selection.

Full size image

Binary categorical features were encoded with 0 and 1; for example, the gender of patients was encoded as 0 or 1 (0 = male, 1 = female). A total of 30 demographic and clinical variables were included as baseline variables for analysis. All parameter data were retrieved from inpatient electronic medical records. Univariate logistic analysis was performed to identify impactful clinical variables, and then statistically significant variables were included in multivariate logistic analysis.

Model development and performance evaluation

This dataset was randomly split into a training cohort (70%) and a validation cohort (30%). The training cohort was used to construct and perform cross-validated ML models to avoid overfitting, and the validation database was used to validate the predictive power of models.

Based on the results of multivariate logistic analysis, the identified statistically significant variables were used to construct ML models. Six ML algorithms were employed to develop predictive models based on the training cohort, the performance of all candidate models was assessed using tenfold cross-validation within the training cohort, the final model was identified according to the highest mean AUC and the performance was further validated on the validation cohort. During tenfold cross-validation, the training cohort was divided into ten sets, nine of them were used for model testing and one for model assessment. Meanwhile, radar plots were drawn and the accuracy, sensitivity and specificity were calculated to evaluate performance. Due to the “black-box” nature of ML algorithms, the Shapley Additive explanations (SHAP) metric was used to interpret the models and assist doctors understand the findings of the models. The contribution of each variable to the model prediction was evaluated by SHAP values11,12.

Statistical analysis

Continuous and categorical variables were expressed as mean ± SD and frequency, respectively. P-Values < 0.05 were considered statistically significant with 95% confidence intervals (CIs) applied for all analyses. Statistical analyses of the demographic and clinical characteristics of all included patients were performed in R (version 4.0.5, HTTPS: //www.r-project.org/). Python (version 3.8) was used to develop ML predictive models and a web risk calculator.

Ethics approval and consent to participate

This study was approved by the Ethics Committee of the Second Affiliated Hospital of Xuzhou Medical University (ethics number: [2020] 081603), and all studies were conducted in accordance with relevant guidelines/regulations, informed consent was obtained from all participants, and all studies were conducted in accordance with the Declaration of Helsinki.

Results

Characteristics of study population

A total of 713 AIS patients were included in this study, and 36 patients with missing value were deleted. Finally, 677 patients diagnosed as AIS were enrolled in the present study, and 209 patients had poor prognosis (30.9%). The differences between AIS patients with good and poor prognosis are described in Table 1. No differences in stroke occurrence between age and gender groups were observed, and there were no significant differences in the location of stroke-associated arterial blood vessels.

Table 1 Baseline characteristics of AIS patients between good and poor prognosis.
Full size table

Correlation of variables with clinical outcome

A total of 474 patients were assigned to the training cohort (70% of the total population). In the training cohort, univariable regression was used to identify fifteen significant risk factors, including Systolic Blood Pressure (SBP), Diastolic Blood Pressure (DBP), Homocysteine (HCY), Myoglobin (MB), C-reactive protein (CRP), Neuron-Specific Enolase (NSE), S100β, treatment history (thrombolysis, thrombectomy, antiplatelet, anticoagulation, statin, PPI), and complicates (dysphagia and stroke-associated pneumonia) (Table 2). Subsequently, the previously identified variables were used to determine independent risk factors using multivariate regression. The results of the multivariable analysis in the training cohort are presented in Table 2. Independent risk factors associated with prognosis included NSE, HCY, CRP, S-100β, dysphagia, and anticoagulation.

Table 2 Univariate and multivariate logistic regression analysis of risk factors for poor prognosis in AIS patients.
Full size table

Development and validation of predictive models

Based on the six significant risk factors identified through multivariable Cox regression analysis on the training cohort, the following six machine learning algorithms were used to develop predictive models: Naive Bayesian classification (NBC), extreme Gradient Boosting (XGB), Random Forest (RF), Decision Tree (DT), Gradient Boosting Machine (GBM), and Logistic Regression (LR). For internal validation, tenfold cross-validation was employed to compare the performance of all models. ROC curves were plotted, and the AUCs of all established models are illustrated in Fig. 2. Among them, the RF model exhibited the best prediction performance with an AUC of 0.931. The AUCs of the XGBboost, GBM, DT, LR, and NBC models were 0.922, 0.923, 0.923, 0.829, and 0.871, respectively. Additionally, the selected models were comprehensively evaluated by calculating their accuracy, sensitivity, and specificity, as depicted in Fig. 3. Consequently, the RF model was hereby identified as the most effective predictive model for the prognosis of AIS, displaying high accuracy (0.789), sensitivity (0.755), and specificity (0.877). The F1 score is a commonly used metric for evaluating the overall performance of classification models, especially in situations with imbalanced class distributions. The F1 score ranges between zero and one, with values closer to one indicating better model performance. In the case of the final RF model, the F1 score was 0.699. Additionally, an external cohort was used to test the model, and the result also suggested that the RF model is the optimal model to predict the prognosis, with the highest ACU of 0.908. The AUCs for each model were: 0.884 for the XGB model, 0.883 for the GBM model, 0.879 for the DT model, 0.882 for the LR model, and 0.860 for the NBC model (Fig. 4).

Figure 2
figure 2

The results of tenfold cross-validation in the training cohort.

Full size image
Figure 3
figure 3

Performance of different models.

Full size image
Figure 4
figure 4

The results of the validation in the validation cohort.

Full size image

Model interpretation

This study analyzed the independent validation set in the RF model through the SHAP package, as shown in Fig. 5. In the final RF model, feature importance rankings of six predictors are shown in Fig. 5A. Based on the SHAP summary plots for poor prognosis in AIS patients, the related features ranked from highest to lowest importance were NSE, HCY, S-100β, dysphagia, CRP and anticoagulation. Figure 5B showed the distribution of the contribution of each feature to the model output. The influence of feature values on the results is represented by colors, with each dot representing a case in each row, red dots representing larger feature values, and blue representing lower ones. The smaller the feature value, the corresponding SHAP value is less than zero7, indicating a negative impact. On the other hand, the larger the feature value, the corresponding SHAP value is greater than zero, indicating a positive impact. The region with the widest distribution is NSE, indicating that it has the greatest impact.

Figure 5
figure 5

SHAP for the RF model. (A) importance ranking of features. (B) the distribution of the impacts of each feature on the model output.

Full size image

Online application for prognosis prediction

Based on the RF model, an easy-to-use online calculator was built to predict the prognosis of AIS patients, which can be obtained at https://mlmedicine-prog-prog-stroke-ysnamt.streamlitapp.com/. By entering the patient's clinical data, doctors and patients can obtain the estimated probability of poor prognosis immediately. Readers can also use these detailed parameter settings to reproduce the proposed model in Python. The model_parameter_settings.txt can be downloaded from the following publicly available GitHub repositories. (https://github.com/Wu-Shi-Nan/Acute-ischemic-stroke/blob/main/model_parameter_settings.txt).

Discussion

Stroke is the second most common cause of disability and the leading cause of death among adults13, imposing a substantial burden on families and society in terms of disease and medical impact. Furthermore, this burden has consistently increased over the past three decades14,15. Consequently, there is an urgent need to research early and accurate prediction of adverse outcomes in patients with AIS, to facilitate precise clinical management and surveillance.

According to the present study, dysphagia was identified as a significant cause of poor prognosis in patients with stroke. Dysphagia is a common complication after a stroke16. Martino et al. reported that dysphagia occurs in 37% to 78% of patients with stroke and increases the risk for pneumonia 3–11 times in patients with confirmed aspiration17. Additionally, previous literature has proved that dysphagia is the most important cause of post-stroke pneumonia18,19,20. Stroke inhibits immunological responses through the activation of the autonomic nervous system and stress axis, contributing to the development of stroke-associated pneumonia, further exacerbating the condition21.

The present findings showed that CRP has a negative effect on the prognosis in stroke patients. CRP is an inflammatory response indicator, which can be significantly elevated during the acute phase of an inflammatory response. Meanwhile, stroke and infection are strongly intertwined. Many studies have reported that impaired immune response occurs in stroke patients, resulting in increased susceptibility to infections22. It has been previously reported that the presence of infection during acute and subacute phases of stroke is a common predictor of a poorer prognosis23. Higher CRP values as associated with worse the infection24. Increased c-reactive protein is related to a poorer prognosis regarding the course of stroke23.

Meanwhile, AIS patients are prone to develop inflammatory response, and the immune-inflammatory response is an essential process post-stroke25. Xie et al. showed that the interaction between the activation of coagulation and the inflammatory response leads to the consumption of coagulation factors. Additionally, patients with a higher international normalized ratio had more serious strokes, and the increased international normalized ratio was an independent predictor of one-year all-cause mortality26. Anticoagulation increases the international normalized ratio and bleeding risk, thus increasing the incidence of poor prognosis. Bautista et al. suggested that warfarin use at baseline was associated with mortality27, which is consistent with the findings of this study.

S100β and NSE were positively associated with poor prognosis in the present findings. S-100β, a member of a family of Ca + binding proteins, is a small acidic calcium-binding protein and abundantly expressed in the nervous system, mostly in astrocytes and several neuronal populations28. Clinically, it is considered to be a serum marker of cerebral damage and hypoxia for assessing neurological prognosis29. NSE, a dimeric isoenzyme of the glycolytic enzyme enolase, is involved in the glycolytic pathway. It is abundantly present in neurons and cells of neuroendocrine origin and used as a serum marker for neuronal loss, and its highest activity is found in brain tissue cells30,31. S100β and NSE are markers of glial cells32. Due to stroke, ischemia and hypoxia occur in the brain, leading to rupture of the membrane structure of the neurons, and the process of acute stroke causes a large release of S100β and NSE into the blood. The degree of neurological impairment can be assessed to some extent by S100β and NSE23,33. Previous studies have reported that increased levels of NSE and S100B are positively correlated with poor outcomes of stroke patients34,35,36, which is consistent with our findings. NSE and S100β were also included in the presented predictive model to evaluate the prognosis of patients with stroke. Meanwhile, NSE ranked first among all predictors based on SHAP analysis. In addition, HCY was related to a poor outcome in stroke patients, and the relationship between hyperhomocysteinemia and poor outcome in stroke patients has been previously reported37. When stroke patients presented with inflammatory response, cellular injury and necrosis would occur. Furthermore, as a response to this reaction, adenosine triphosphate would be released into the extracellular space, and hydrolyzed into adenosine. Adenosine has anti-inflammatory and tissue-protective properties38. However, high HCY could reduce activities and protein content of electron transport chain components, hence lowering mitochondrial energy metabolism and decreasing adenosine triphosphate production39. Thereby, for patients with high HCY, the anti-inflammatory and tissue protection in AIS patients were worse, and the prognosis was poorer accordingly.

An RF model was developed in this study to predict the risk of poor prognosis in AIS patients, which showed excellent performance compared with other ML models. The factors NSE, HCY, CRP, S-100β, anticoagulation, and dysphagia were identified as independent factors for poor prognosis. This study has laid a foundation for timely and accurate prediction of poor prognosis risk in order to further organize stroke care. Finally, poor prognosis rates for AIS patients can be easily obtained by entering corresponding clinical features in the developed online calculator. As shown in Fig. 6, the probability is calculated online quickly (Probability of poor prognosis = 9.4%, low-risk group). The wide application of smart devices nowadays makes the developed tool greatly convenient to use.

Figure 6
figure 6

The web calculator for predicting prognosis of AIS patients.

Full size image

Previous studies have attempted to use ML algorithms to evaluate the prognosis of stroke5,40,41,42,43. Peng et al. leveraged a cohort of 423 patients for predicting 30-day mortality in spontaneous intracerebral hemorrhage patients42. Bacchi et al. designed models using LR, RF, DT, and artificial neural networks to predict in-hospital mortality43. Chen et al. used five ML techniques, including CatBoost, XGB, Boosting Decision Tree, RF, and AdaBoost, to investigate the 90-day poor prognosis in patients with transient ischemic attack or minor stroke. The factors associated with poor prognosis were identified, and the CatBoost model (AUC = 0.839) had the best predictive performance40. However, their study had a relatively short predictive window time. Heo et al. explored the applicability of ML algorithms to predict long-term prognosis in AIS patients. They reported that the deep neural network with an AUC of 0.888 can improve the prediction of long-term outcomes, but the predictive variables were not identified5.

Compared to other studies, the proposed predictive model has the following advantages. First, a comprehensive range of ML algorithms were tested, including NBC, XGB, RF, DT, GBM, and LR. This comprehensive approach allows exploration of different modeling techniques and selecting the most suitable algorithm for the specific prediction task. Second, several significant risk factors were identified through the analysis, such as NSE, HCY, CRP, S-100β, anticoagulation, and dysphagia, which are independently associated with unfavorable outcomes in AIS patients. The variables mentioned above are mostly readily available patient information, which makes this model potentially suitable for assisting clinical applications. Proper use of the model can provide valuable insights into the factors influencing prognosis and aids in clinical decision-making. Third, this study employed a relatively long prediction window, which extends the observation period and captures a broader range of patient outcomes. This reduces the impact of short-term fluctuations and random variability, enhancing the reliability and stability of the predictions. Fourth, the SHAP values were used to interpret the model, providing insights into the contribution and importance of each feature in the prediction process. This enhances the model transparency and interpretability, allowing for better understanding and trust in the results. Fifth, a web-based platform that visualizes the predictive model and provides a dynamic calculator for easy and practical usage was developed. This user-friendly interface enhances the accessibility and usability of the proposed model, making it more convenient for clinical applications. Overall, the predictive model stands out by utilizing clinical samples combined with various ML algorithms, identifying significant risk factors, employing a long prediction window, providing model interpretation, and offering a user-friendly platform. These advantages contribute to its potential clinical utility and effectiveness in predicting AIS outcomes.

There are several limitations present in this study. First, this is a retrospective study, which introduces the potential for selection bias and limits the establishment of causal relationships. Second, the sample size was not large enough, which may affect the generalization ability of the findings, and the data collection was limited to a single medical center, which may restrict the external validity of the results. Despite the good predictive power of our ML model, external validation in a completely different patient cohort is unavailable. Third, it is recommended to compare the predictive model with another prognostic system, such as the Acute Stroke Registry and Analysis of Lausanne score, to provide a more comprehensive evaluation and enhance the persuasiveness of the findings44. Finally, it should be noted that the NIHSS score, which stands for National Institutes of Health Stroke Scale, is a widely predictor of outcomes in AIS predictive models45. The NIHSS score upon admission was found to be a significant factor in the development of unfavorable outcomes. It is important to acknowledge that the absence of the NIHSS score data in the final predictive model might have limited its performance. A prospective multiple-center cohort could be designed to enhance the credibility of the results in the future.

Conclusion

In summary, this paper established machine learning models to predict the prognosis of acute ischemic stroke. This tool can assist clinicians and patients to predict prognoses and formulate management strategies.