Introduction

In 2019, an estimated 37 million individuals, or 11.3% of the United States (U.S.) population, has diabetes [1]. Many persons with diabetes have additional complications, including neuropathy and peripheral arterial disease, which increase the risk of developing foot ulcers [2, 3]. Between 15–25% of persons with diabetes develop a diabetic foot ulcer (DFU) during their lifetime [4]. Furthermore, a DFU is the greatest risk factor for lower extremity amputation; persons with type 2 diabetes (compared to those without) have a ten-fold higher rate of amputation [5]. The development of a DFU or amputation results in a significant decrease in patients’ quality of life and productivity due to a reduction in physical, social and employment activities [6]. Treatment costs range between $18,600 to $35,100 per DFU, with the U.S. spending $60 billion annually on lower extremity-related care for patients with diabetes [7, 8]. Hence, DFU development or amputation is associated with significant morbidity, mortality, decreased quality of life and productivity, and substantial health care costs.

Since the 1990s, a large body of research has focused on developing tools that predict risk of DFU or amputation with the goal of identifying persons at high-risk who may benefit from early prevention and intervention [8]. Multiple recent systematic reviews (SRs) have described these prediction tools and their performance characteristics [9,10,11]. This review of reviews was conducted (as part of a larger review) by the Department of Veterans Affairs (VA) Evidence Synthesis Program at the request of the VA National Clinical Orthotic and Prosthetic Program Office. The goal was to identify tools that predict DFU development or amputation and summarize their performance (prognostic accuracy) to help inform clinical practice and policy in the VA Health Care System. Hence, we also evaluated the usability of the tool in busy outpatient clinics. Given the high prevalence of DFUs and amputations in the U.S. and worldwide, risk stratification and prevention would be of great benefit to individuals, health care systems, and society.

Methods

Overview

This review of reviews focused on the performance characteristics (e.g., accuracy, external validation, and clinical applicability) of tools that predict development of a new DFU (first or subsequent). It also provides a detailed description of the clinical applicability and usability of top-performing prediction tools based on additional data abstraction from the original studies describing tool development and validation. This review was registered in PROSPERO (http://www.crd.york.ac.uk/PROSPERO/; #CRD42021287645).

Data sources and search strategy

We searched for peer-reviewed English language SRs from inception to January 13, 2023 in MEDLINE, Embase, and the Cochrane Database of Systematic Reviews. A search filter was applied to limit results to reviews and a language restriction (English) was also used. To supplement the database search, we reviewed reference lists of relevant SRs and sought input for additional studies from our clinical experts or Technical Expert Panel members. Appendix Table 1 shows the detailed search strategy.

Study selection of eligible reviews

At least two reviewers (A.L., A.K., and C.S.) independently screened titles and abstracts of all identified studies using Distiller SR (Evidence Partners, Ottawa, Canada). Articles included by any reviewer were moved to full-text review. At full-text review, at least two individuals decided independently on inclusion/exclusion; disagreements or inconsistencies were resolved by discussion and input from a third reviewer. Pre-specified eligibility criteria included SRs that evaluated tools predicting DFU (primary or subsequent) or amputation in adults (≥ 18 years of age) with diabetes. Studies were excluded if they were not peer-reviewed full-text SRs or included non-eligible populations such as children.

Data extraction and quality assessment

Data from all eligible SRs was abstracted by one reviewer (A.L.) and confirmed by a second reviewer (C.S.) using a standardized data extraction form developed with input from review team members based on a priori inclusion, exclusion criteria, as well as populations, interventions and outcomes of interest. This data extraction form was pilot tested before abstraction. We abstracted data on SR title and authors, funding sources, SR characteristics, search dates and strategy, population characteristics, outcome, number of studies and models predicting outcomes of DFU development or amputation, SR limitations, and SR authors’ conclusions on top-performing tools. When information from the review was missing, we accessed the primary publication. Since the included SRs were narrative, we did not identify discrepant or overlapping data. Extracted data was cross-checked by a third reviewer (A.K.) for quality assurance but no formal kappa for agreement was calculated because of the limited number of studies for which data extraction was performed. Risk of bias (ROB) was assessed using the Risk of Bias in Systematic Review (ROBIS) tool [12] and is provided in Appendix Table 2. ROB was assessed by one reviewer, confirmed by a second, and disagreements were resolved by discussions or a third reviewer.

Data synthesis and analysis

Data was synthesized by one reviewer and confirmed by the second using an electronic form. There were no disagreements. Given the heterogeneity in populations, the varying inclusion/exclusion criteria across reviews and differences in SR methodology, we provided a qualitative or narrative summary of the best performing prediction tools from the included SRs. There is currently no GRADE guidance on assessment of certainty for risk prediction models. The GRADE prognosis project group is currently developing definitive guidance for application to clinical prediction models [13]. We primarily relied on the review authors’ assessments and reporting of prediction tool performance. For tools with good prognostic accuracy, we retrieved and reviewed the primary studies that reported on the original tool development. We categorized tools based on inclusion of a specified prediction horizon, as (i) risk classification systems if they predicted level of risk without a specified time horizon, or (ii) risk prediction models if they predicted risk over a specified time horizon. For risk prediction models, prognostication could be subjective (e.g., high risk at 2 years), relative (e.g., twofold increased risk at 2 years), or absolute (e.g., 6% rate of DFU development at 2 years). Since prediction over a specified time horizon is important for clinical decision-making, including interventions and referral to specialists, we only included risk prediction models for further consideration.

For the top performing risk prediction models, we abstracted prognostic accuracy measures (calibration and discrimination) reported by original studies of tool development (internal/external validation) and the Beulens et al. SR (external validation) [9]. An independent search for all external validation studies for tools was not conducted. Calibration was evaluated using calibration slope and the observed/predicted ratios. Discrimination was evaluated using a C statistic or area under the curve for the receiver operating curve (AUC-ROC) [14]. A C statistic value of 0.5–0.6 was rated as poor, 0.6–0.7 as fair, 0.7–0.8 as good, and ≥ 0.8 as excellent discrimination. A formal risk of bias assessment or evaluation of the primary studies was not performed.

Finally, to determine clinical relevance and usability, we analyzed the number of variables included in each model, ease of obtaining these variables in the busy clinical practice setting, feasibility of score calculation, and categorization of risk.

Results

Of 1,495 unique citations, 131 articles underwent full-text review of which 30 reviews were deemed eligible for the larger report (Fig. 1) [9,10,11]. Of these, 3 SRs were eligible for this review of reviews (tools that predict risk of DFU or amputation). Two SRs were rated low ROB [9, 11] and one was rated moderate ROB [10]. Detailed characteristics and conclusions of the 3 SRs are provided in Table 1. Across these three SRs, there was heterogeneity in the populations, models, and outcomes of the included studies, and the SRs reached different conclusions. We first provide results from eligible SRs, then describe performance characteristics of the top performing prediction models (distinguishing models used for risk classification versus risk prediction) and finally we reviewed usability characteristics.

Fig. 1
figure 1

Literature Flow Diagram. *Search through January 13, 2023

Full size image
Table 1 Overview of systematic reviews evaluating, for patients with diabetes, risk prediction tools for diabetic foot ulcer development or amputation
Full size table

Systematic reviews of prediction tools

The SR by Beulens et al. (low ROB) identified tools that predicted DFU or amputation risk in patients with type 2 diabetes (without a DFU at baseline) with ≥ 1-year follow-up [9]. Beulens et al. identified 21 studies of 34 risk prediction models predicting neuropathy, DFU, or amputation. The commonly used prediction horizons were 1 year and 10 years. The authors also conducted an external validation study of 13 models predicting DFU or amputation using a Dutch cohort of community-dwelling adults with type 2 diabetes mellitus (mean age 67 years, 53% male, 4.1% with a history of DFU or amputation) seen in a primary care clinic (n = 7,624) using a 5-year follow-up period. In this external validation cohort, 485 (6.4%) developed a new DFU and 70 (0.9%) underwent amputation during the 5-year follow-up. Among individuals with no history of DFU or amputation (n = 7309; 95.9% of entire cohort), 265 (3.6%) developed a DFU and 28 (0.4%) underwent amputation over 5 years. In contrast, among individuals with a prior DFU or amputation (n = 315), 220 (69.8%) developed a DFU and 42 (13.3%) underwent amputation over 5 years.

Based on the external validation results, the authors identified top-performing models for:

  1. (i)

    Predicting new DFU at 5 years: The Boyko [15], PODUS 2015 [16], and Martins-Mendes (original and simplified) [17] models performed well with good to excellent discrimination. Calibration plots for the Martins-Mendes models (original and simplified) demonstrated good agreement between observed and predicted rates in the lower quintiles of predicted risk, but observed risks exceeded predicted risks in the higher quintiles. No calibration plots were presented for the models by Boyko or PODUS 2015.

  2. (ii)

    Predicting amputation at 5 years: The Martins-Mendes models (original and simplified) performed well with good to excellent discrimination (C statistic 0.81 and 0.78, respectively). Calibration plots for these models for amputation showed results similar to their performance for DFU prediction, i.e., good agreement for amputation prediction between observed and predicted risks in the lower quintiles of predicted risk, but observed risks exceeded predicted risks in the higher quintiles of predicted risk.

The authors concluded that using a combined endpoint of DFU or amputation prediction, the models by Boyko, PODUS 2015, and Martins-Mendes showed good performance and may be applicable for use in clinical practice.

The SR by Fernandez-Torres et al. [10] (moderate ROB) identified clinician-assessment tools for measuring diabetic foot disease related variables which included neuropathy and ulceration risk, and DFU-related variables which included amputation risk, healing, infection assessment, and measurement, applicable to patients with diabetes (type 1 or 2). Studies were excluded if tools did not include psychometric properties in their development or did not provide any measurement properties that met the consensus-based standards for the selection of health measurement instruments (COSMIN) criteria. This SR identified 29 studies of 39 clinician-assessment tools validated for the assessment of diabetic foot disease and DFU-related variables. Prediction horizons were not reported. Thus, measures of calibration and discrimination or absolute risks of diabetic foot disease or DFU related outcomes over a specified time horizon were not reported. ROB of included studies was not reported. Of the 10 scales assessing ulceration risk, the authors identified the Queensland High Risk Foot Form scale (QHRFF) as a valid and reliable instrument for assessing risk of developing a DFU. However, the authors also stated that the psychometric characteristics of QHRFF did not have sufficient strength, because the QHRFF validation study was conducted in only 22 subjects.

The SR by Monteiro-Soares et al. [11] (low ROB) identified risk stratification systems for predicting DFU and identified 13 studies evaluating 5 models. The authors stated that the quality of evidence for these systems was low, as little validation of their predictive ability had been performed. Hence, the authors concluded that the best method for assessment of risk stratification was not immediately apparent.

Re-classification and performance characteristics of prediction tools

Based on the results and conclusions of the 3 SRs described above, we identified 5 recommended tools to predict DFU or amputation risk: Boyko et al., Martins-Mendes et al. (simplified and original), PODUS 2015, and QHRFF [9,10,11]. We additionally identified an updated model for PODUS 2015 – PODUS 2020 [8] from a review of the reference lists of SRs. Hence, in total we prioritized 6 tools for further review. For these, we reviewed original studies outlining tool development and ultimately categorized tools as risk classification systems or risk prediction models [8, 15,16,17,18,19,20] described in Appendix Tables 3 and 4. We determined that PODUS 2015 and QHRFF are best categorized as risk classification systems as they do not specify the time horizon for prediction, hence we excluded these from further consideration. We describe below prognostic accuracy of the following risk prediction models: Boyko, Martins-Mendes (original and simplified), and PODUS 2020. All 4 risk prediction models predict DFU; the 2 Martins-Mendes models also predict amputation. The models by Boyko and Martins-Mendes categorize risk subjectively, in contrast to PODUS 2020 which predicts absolute risk.

Prognostic accuracy

All 4 risk prediction models have been externally validated. The Beulens et al. SR externally validated the models by Boyko and Martins-Mendes [9]. PODUS 2020 was externally validated by the study team [8]. Prognostic accuracy (calibration and discrimination) in validation studies for these models is described in Table 2.

Table 2 Performance characteristics of recommended risk prediction models for diabetic foot ulcer development or amputation
Full size table

Discrimination

In external validation studies, discrimination was good to excellent for all 4 models (Boyko, Martins-Mendes (original and simplified), and PODUS 2020) predicting DFU, and the 2 models (Martins-Mendes) predicting amputation [9].

Calibration

In external validation studies, calibration plots for the models by Martins-Mendes (for DFU and amputation prediction) and PODUS 2020 (for DFU prediction) showed good agreement between observed and predicted absolute risks in the lower quintiles of predicted risk, but observed risk exceeded predicted risk in the higher quintiles [9].

Usability characteristics of prediction tools

Table 3 and Appendix Table 3 describe variables included, score calculation, and score interpretation for the 4 recommended risk prediction models. The 4 models include 2 to 7 variables which can be obtained by history or chart review (prior DFU, prior amputation, and diabetes complications), physical exam (neuropathy, peripheral arterial disease [PAD], fungal infection, and physical impairment), diagnostic testing in the clinic (visual acuity), and laboratory tests (microbiology to assess for onychomycosis or tinea pedis, and HbA1c). The models by Boyko and Martins-Mendes (original or simplified) require a calculator to determine risk score, but PODUS 2020 score is a simple addition (not requiring a calculator). The models by Boyko and Martins-Mendes (original or simplified) provide a subjective assessment of DFU or amputation risk, but PODUS 2020 quantifies absolute average DFU risk at 2 years of follow-up.

Table 3 Variables included in risk prediction models for diabetic foot ulcer development or amputation
Full size table

Discussion

Development of a DFU or amputation has severe consequences for the individual and healthcare system [21]. The identification of persons at a high absolute risk for DFU or amputation over a specified time horizon can aid in targeted monitoring and focused prevention efforts, whilst reducing unnecessary resource expenditure on low-risk persons. In this review of reviews, we found 3 SRs describing the performance characteristics of tools predicting DFU development or amputation [9,10,11]. The SR by Monteiro-Soares et al., published prior to the development of 3 of the 4 top-performing models, did not identify any well-performing tools [11]. The 2 more recent SRs (2021 and 2020), included studies published in PubMed and EMBASE databases over similar time frames. However, these 2 SRs differed in their search criteria, included study populations, and necessity for tool/model validation and likely due to these differences, the 2 SRs identified different tools and reached different conclusions. Thus, in contrast to the low ROB SR by Beulens et al. which identified risk prediction models with a specified time horizon for prediction [9], the moderate ROB SR by Fernandez-Torres et al. mostly identified risk classification systems without a prediction time horizon [10]. Since tools that do not provide a time horizon for risk prediction are less useful for shared clinical decision making, we excluded these from further consideration.

Based on the results of the SRs, we investigated the performance of 4 models predicting DFU development (Boyko, Martins-Mendes [original and simplified], PODUS 2020), and 2 models predicting amputation (Martins-Mendes [original and simplified]) over a specified time horizon [8, 15, 17]. For the models by Boyko and Martins-Mendes, the time frame for risk prediction varied not only between models, but also for the same model between internal and external validation studies [9]. Additionally, categorization of risk, and thresholds used to define levels of risk varied across models without clear rationale. However, all 4 models had good prognostic accuracy, with PODUS 2020 performing best [8, 9].

Clinical usability is a critical consideration for successful widespread adoption of a risk prediction model. Given the time constraints in primary care clinics, the ideal prediction model includes evaluation of a few readily obtainable variables that permit accurate, reproducible, and understandable risk calculation to clinicians and patients. The 4 models included 2 to 7 variables. The models by Boyko and Martins-Mendes include many variables, or variables that take time or resources to obtain. In addition, these tools also require a calculator for risk calculation. These factors decrease the likelihood of widespread use of the Boyko and Martins-Mendes models. In contrast, PODUS 2020 consists of 3 binary variables (neuropathy [1 point], absence of any pedal pulse [1 point], and prior history of DFU or amputation [2 points]) that can be easily measured in the clinic and the scoring is summative [8]. In the PODUS 2020 developmental cohort, 8.5% of all persons had a prior history of DFU or amputation. The risk of DFU at 2 years with scores of 0, 1, 2, 3, and 4 were 2.4%, 6%, 14%, 29.2%, and 51.1%, respectively, with 5.2% of all persons developing a DFU at 2 years. PODUS 2020 has also been externally validated in an independent cohort, in whom 3.9% of persons developed DFU at 2 years. Based on their findings, PODUS 2020 authors concluded that individuals with a score of ≥ 1 (predicted 2-year rate of DFU ≥ 6%) would benefit from preventative treatment [8]. Based on our review, the PODUS 2020 model has the most favorable prognostic accuracy and feasibility characteristics to predict primary or subsequent DFU development. Furthermore, the absolute quantification of DFU risk provided by PODUS 2020 is more easily understood by clinicians and patients, and potentially more informative for shared clinical decision making than models that do not provide a time horizon or categorize individuals by subjective risk categories.

We identified several limitations in the models. For PODUS 2020, prognostic accuracy depends on the ability of clinicians to accurately use a 10 g monofilament and assess for palpable pedal pulses; skills which may vary based on clinicians’ specialty and experience [18]. All studies assessed one-time tool use to predict DFU development or amputation at specified time horizons. In the absence of studies on how risks for DFU change over time, appropriate rescreening intervals for any model are unknown. Most models did not assess if race or ethnicity predicted DFU development, hence the accuracy of these models in predicting DFU in different populations is unknown. Despite the relative simplicity of PODUS 2020, use of this tool in the primary care clinic setting may be challenging as patients and clinicians have competing health care priorities and there are limited time and resources to address them. Lastly, whether tool deployment will better guide referral, subsequent monitoring, or initiation or continuation of treatment to reduce risk of DFU or amputation is unknown.

A limitation of our review is that our search only includedstudies published in English which may have excluded potentially relevant SRs, however, we did not have geographical or date limitations on the search strategy or study eligibility. A strength of our review was that we were inclusive with search, abstraction, analysis, and critique criteria of SRs and we additionally reviewed primary studies of tool development to qualitatively describe clinical feasibility of prognostically accurate models.

Future research

All current models predicting DFU development include a history of DFU as a risk factor. However, most individuals seen in primary care do not have a history of DFU and are likely at much lower absolute risk of DFU development or amputation. Future studies should develop and validate models to predict development of first DFU. Prior to widespread use, models should be externally validated in the intended healthcare setting to ensure site-specific prognostic accuracy and feasibility. In theory, risk prediction tools identify high-risk individuals for targeted early preventive interventions. Future research should also focus on determining appropriate triage decisions for level of identified risk, and whether such decisions result in improved health outcomes. Additionally, future research is needed to identify DFU risk in racial and/or ethnic minorities who may be at increased DFU risk or have limited access to healthcare. Lastly, research is needed to determine appropriate rescreening intervals for the risk prediction tools and associated incremental benefits and harms of these strategies.

Conclusions

Four well-performing models discriminate the risk of developing primary or subsequent DFU or amputation in adults with diabetes who are ulcer-free at baseline. Of these, PODUS 2020 has the most favorable prognostic accuracy and is feasible to use in the primary care clinic setting. The PODUS 2020 score predicts average absolute rate of DFU development at 2-years. The rescreening interval and the type or effectiveness of interventions in response to prediction scores to decrease DFU or amputation are unknown.