FormalPara Key Summary Points

Stratifying patients with skin and soft tissue infections (SSTI) based on their risk for methicillin-resistant Staphylococcus aureus (MRSA) has the potential to improve anti-MRSA antibiotic prescribing and promote good patient and public health outcomes.

This study evaluated the predictive performance of various MRSA risk assessment approaches based on previously developed MRSA SSTI risk assessment tools in an international cohort of patients.

The MRSA risk score, with or without inclusion of nationally reported MRSA surveillance percentage, was significantly more predictive of MRSA than the national surveillance percentage alone.

An MRSA risk score of zero had high predictive value for a non-MRSA infection, demonstrating that this tool could help avoid unnecessary empiric MRSA coverage in low-acuity patients.

Introduction

Skin and soft tissue infections (SSTI) are among the most common infections encountered in community and healthcare settings. [1] Worldwide, bacterial skin diseases, which includes cellulitis and abscesses, accounted for nearly 267 million new cases and contributed to an increase in years lived with disability. [2] A primary SSTI antibiotic treatment consideration is whether methicillin-resistant Staphylococcus aureus (MRSA) coverage is necessary. The rate of MRSA SSTI increased at an alarming rate in the early 2000s in the USA, likely contributing to increased prescription of MRSA-active antibiotics for skin infections [3,4,5]. Increasing importance of MRSA among SSTI has also been reported worldwide. However, MRSA prevalence differs between regions and countries [6,7,8,9,10]. Prevalence of MRSA ranges from less than 1% in some European countries to upwards of 60% in certain parts of South America, Asia, and the USA [11,12,13]. With approximately 50% MRSA prevalence in many regions, local surveillance data and antibiograms become less useful for clinical decision-making. Consequently, SSTI prescribing practices vary widely, whereby both under- and over-prescribing of MRSA-active antibiotics occurs frequently [5, 14,15,16,17]. This antibiotic misuse has important implications for both patient outcomes and public health.

The ubiquitous nature of skin infections coupled with the challenges associated with empiric antibiotic selection underscore the critical need for improved clinical decision support tools to facilitate antimicrobial stewardship [18]. One potential approach to address this need is individual patient risk assessment to identify those at low or high risk of having MRSA SSTI. Numerous MRSA skin infection risk factors have been reported in the literature [19,20,21,22,23,24,25,26,27,28,29,30]. To aid in clinical application of these risk factors, we previously developed a risk scoring tool that combined data from previous literature with individual patient data from a single health system in southeastern Michigan [31]. Given the widespread variability in MRSA prevalence and regional differences in MRSA epidemiology, the objectives of this study were to validate this risk scoring tool in an international cohort.

Methods

Study Design and Population

This was a multicenter, observational, prospective cohort study of adults with SSTI presenting to seven international hospital or clinic sites from July 2016 to March 2018. Sites included seven university-affiliated clinics and/or hospitals from China, Italy, Mexico, Russia, Singapore, and the UK. Sites were selected to be as geographically diverse as possible and included investigators who regularly treat patients with SSTI and volunteered to participate in the study as an investigator. Patients who did not have a culture obtained, those with osteomyelitis and/or septic arthritis, an SSTI secondary to bite wounds, odontogenic infection, and those who were pregnant or prisoners were excluded. There were no missing data given the prospective nature of the data collection. This study was approved by the Wayne State University institutional review board and waiver of informed consent was granted. The study was also approved by a human subject research committee at each study site when necessary, waiver of informed consent granted, and was conducted in accordance with the principles of the Declaration of Helsinki of 1964, and it's later amendments.

Patient Data Elements and Collection

Patients with SSTI were identified for inclusion by clinician investigators at each study site. Patient demographic and clinical data were obtained from patient interviews and health records and electronically entered into a structured data collection form within Research Electronic Data Capture (REDCap, Vanderbilt University) [32]. SSTI definition and type was determined by the treating clinician and investigator at the time of study inclusion. Data included social history, past medical history, and comorbid conditions previously identified as MRSA risk factors, including those used to compute our previously derived MRSA SSTI risk score [31]. Tissue or wound cultures obtained as part of routine care and results that were available at study enrollment were collected.

Individual Patient MRSA Risk Assessment

The MRSA risk for each patient was quantified using our previously published MRSA risk score [31]. Individual MRSA risk scores were computed as follows: MRSA infection/colonization history (2 points), hospitalization in the previous year (1 point), antibiotic exposure in the previous 6 months (1 point), chronic kidney disease (1 point), intravenous drug use (1 point), human immunodeficiency virus (HIV) infection or acquired immunodeficiency syndrome (AIDS) (1 point), diabetes with obesity (1 point). Fagan’s nomogram and the likelihood ratio of the MRSA scores from the derivation cohort were then used to convert national (country-level) surveillance MRSA percentage (prior probability) into an estimated MRSA probability for each included patient (posterior probability) [33]. Country-level surveillance MRSA percentages were obtained from the European Centre for Disease Prevention and Control and World Health Organization [11, 12].

Data Analysis

The primary analysis evaluated the predictive performance of three MRSA risk assessment approaches: (1) national surveillance MRSA percentage alone, (2) MRSA risk score alone, and (3) MRSA probability estimate (combination of national surveillance and risk score). Predictive performance of each approach was quantified by area under the receiver operating characteristics curve (aROC) along with a 95% confidence interval. The aROC of each risk assessment approach was compared using the Hanley and McNeil method [34].

The secondary analyses evaluated the ability of candidate MRSA risk score thresholds to identify patients at lowest and highest MRSA risk. Observed performance characteristics, including aROC, sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV) were computed with 95% confidence intervals for each MRSA risk score threshold with at least 15 patients (e.g., 15 patients with a risk score ≥ 4). Estimated NPV and PPV for each threshold was also computed across hypothetical MRSA prevalence scenarios that were different from what was observed in the international cohort using conditional probability equations [35].

All statistical tests were two-sided; P values ≤ 0.05 were considered statistically significant. Analyses were performed using SPSS Statistics, IBM SPSS software, version 26.0 (IBM Corp., Armonk, NY). No formal sample size calculation was used and a convenience sample of patients was sought.

Results

Two hundred three patients across seven international sites were included. A full description of the cohort is displayed in Supplementary Table 1. The majority of patients (59.6%) were male and either white (82.3%) or Asian (12.3%). The median (IQR) age was 54 (35–67) years. The most common infection types were wound (28.6%), abscess (25.1%), and cellulitis with abscess (20.7%). The most common MRSA risk factors observed were systemic antibiotic exposure in the previous 6 months (45.8%), hospitalization in the previous year (42.4%), diabetes mellitus (30.5%), obesity (23.2%), diabetes with obesity (11.8%), previous abscess (15.8%), MRSA colonization or infection history (12.3%), and intravenous drug use (11.3%). Overall, one-third (33%) of patients had culture-positive MRSA SSTI. Nationally reported surveillance MRSA percentages and observed MRSA percentages at each study site are shown in Fig. 1. Observed MRSA percentages ranged from 10% in Beijing, China to 58.8% in Mexico City, Mexico. The number of patients from each site also ranged from 10 patients in Beijing, China to 50 patients from both St. Petersburg, Russia and Winchester, UK, respectively. In general, study sites with lower national surveillance MRSA percentages saw correspondingly lower observed MRSA percentages. The MRSA risk score distribution and observed MRSA percentages by risk score values are shown in Fig. 2. The majority (58.2%) of patients had a risk score of 0 or 1. The median (IQR) MRSA risk score of the entire cohort was 1 (0–2). As the MRSA risk score increased, the proportion of patients with MRSA increased significantly (P < 0.001).

Fig. 1
figure 1

National surveillance and observed methicillin-resistant Staphylococcus aureus percentage by study site

Full size image
Fig. 2
figure 2

MRSA risk score distribution and observed MRSA percentage by risk score value

Full size image

The predictive performances of the national surveillance MRSA percentage, the MRSA risk score, and the MRSA probability estimate (surveillance MRSA percentage adjusted by the MRSA risk score) are presented in Table 1. The predictive performance of surveillance MRSA percentage was significantly better than random chance, with an AUC (95% CI) of 0.646 (0.569–0.722). The MRSA risk score performed significantly better than surveillance percentage, with an AUC (95% CI) of 0.748 (0.678–0.819) (P = 0.016). The MRSA probability estimate was also significantly more predictive relative to surveillance percentage, with an AUC (95% CI) of 0.781 (0.716–0.845) (P < 0.001). However, no significant difference in aROC was observed between the MRSA risk score and MRSA probability estimate.

Table 1 Predictive performance comparisons of surveillance MRSA percentage with the MRSA risk score and estimated MRSA probability
Full size table

The performance characteristics of various risk score thresholds for predicting MRSA are presented in Table 2. Overall predictive performance, as measured by aROC, was maximized at a risk score threshold ≥ 2 (AUC 0.733, 95% CI 0.658–0.808). Sensitivity and negative predictive value were maximized at a risk score threshold of ≥ 1 (sensitivity 91.0%, 95% CI 84.2–97.9; NPV 89.8%, 95% CI 82.1–97.5). Ten percent of subjects with a risk score of 0 had MRSA (Fig. 2). Sensitivity and negative predictive value progressively decreased as risk score thresholds increased. Specificity and positive predictive value were maximized at a risk score threshold of ≥ 4 (specificity 94.9%, 95% CI 91.1–98.6; PPV 63.2%, 95% CI 41.5–84.8). Positive predictive value of risk score thresholds of ≥ 2, ≥ 3, and ≥ 4 were similar and ranged from 54.1% to 63.2%. Among patients with a risk score ≥ 2, 57.6% had MRSA (Fig. 2).

Table 2 Performance characteristic of various MRSA risk score thresholds
Full size table

The estimated positive and negative predictive values of an MRSA risk score ≥ 1 across various hypothetical MRSA prevalence settings are displayed in Table 3. The estimated NPV of an MRSA score ≥ 1 (i.e., a risk score of 0) was in excess of 90% when MRSA prevalence was 30% or less. When MRSA prevalence was > 50%, the NPV of a risk score ≥ 1 was below 80%. The estimated PPV of an MRSA score ≥ 1 was poor until MRSA prevalence was very high (> ~70%). The estimated positive and negative predictive values of MRSA risk score thresholds are displayed in Supplementary Table 2. These thresholds generally had better estimated PPV than a threshold of ≥ 1 but all candidate thresholds had a PPV ≤ 76% at an MRSA prevalence of 50%.

Table 3 Estimated positive and negative predictive values of an MRSA risk score ≥ 1 across hypothetical MRSA prevalence settings
Full size table

Discussion

This study sought to internationally validate an MRSA risk scoring tool developed from previously published literature and individual patient data from a single US health system [31]. The MRSA risk score demonstrated fair predictive performance in an international cohort derived from seven study sites. The aROC score (95% CI) of 0.748 (0.678–0.819) was greater than the original derivation cohort 0.601 (95% CI 0.521–0.681). The score was significantly more predictive of MRSA relative to national surveillance MRSA percentages alone. Using Fagan’s nomogram, the score was also able to generate individual patient-predicted MRSA probabilities from the MRSA surveillance percentages. These MRSA probability estimates also demonstrated fair predictive performance and was a significant improvement from MRSA surveillance percentages alone. Collectively, these approaches demonstrate that the use of individual patient characteristics can better predict MRSA SSTI than the use of general surveillance data alone. This demonstrates the potential for similar approaches to be used in clinical practice when a suitable risk stratification model is available. This could include electronic health record-based clinical decision support tools incorporating Fagan’s nomogram or simple bedside scoring systems such as the MRSA score.

No significant difference in predictive performances between the MRSA probability estimates and the MRSA risk scores was observed. This is a noteworthy finding in the context of the widely varying MRSA prevalence observed in our study and in clinical practice. Unlike the probability estimate, the MRSA risk score only accounts for individual patient clinical characteristics and does not account for national MRSA prevalence. Knowledge of local MRSA prevalence would theoretically improve the ability to predict MRSA presence. Although there was no significant difference, the aROC for estimated MRSA probability was numerically greater than the aROC for the MRSA risk score. It is unclear whether this convenience sample was large enough to exclude a meaningful difference in aROC but, at minimum, it appears that accounting for national MRSA prevalence did not reduce the ability of the risk score to predict MRSA. This finding indicates that the MRSA risk score may be reasonably applied without knowledge of MRSA prevalence in settings where it is either unknown, or computation of estimated MRSA probability estimate is difficult or impossible.

Despite less than optimal predictive performance, these risk scoring approaches were able to identify patients at lowest and highest risk of MRSA infection comparably to other risk scores [36]. Methicillin-resistant S. aureus was observed in 10% of patients with a risk score of zero and nearly 60% in those with a risk score of two or more. An MRSA risk score of zero had an NPV in excess of 90% even when hypothetical MRSA prevalence was 30%, and an NPV in excess of 80% up to a hypothetical MRSA prevalence of 50%. This hypothetical MRSA prevalence range encompasses all but two of the study sites from this international cohort, suggesting this MRSA prevalence range is common. Risk of MRSA, along with patient clinical status and disease severity, may be used to better inform diagnostic and antibiotic treatment decisions. Clinicians may be able to avoid MRSA coverage in low-acuity patients with an MRSA risk score of zero. The majority of patients in this and similar studies are at low MRSA risk and this could greatly reduce the number of patients receiving unnecessary MRSA coverage. [16, 36] Such an approach would prevent unnecessary MRSA coverage in approximately 25% of the patients in this study. It would only result in failure to cover MRSA when necessary in 3% of the patients in this study. Although failing to provide in vitro active therapy is never desirable, studies suggest that a 3% rate of under-prescribing is less frequent than estimates from a cohort of patients with abscess receiving antibiotics [5].

These findings should be interpreted in the context of several considerations and limitations. Although this study comprehensively collected MRSA risk factors through health records and patient interviews, it is possible that some clinical data or risk factors were missed or were present too infrequently to assess (e.g., recent travel from high-prevalence area). Prospective MRSA colonization screenings were not conducted at all study sites and thus, it is possible some colonized patients were considered non-colonized [37]. However, MRSA infection/colonization history was strongly associated with MRSA SSTI in this study, thus suggesting this potential non-differential misclassification bias did not substantially alter the findings. It is also important to note that although inclusion of multiple study sites across six countries promotes external validity of the findings, the generalizability is not without several limitations. Firstly, this study focused primarily on purulent SSTI to facilitate organism identification. Therefore, these findings may not apply to patients with non-purulent skin infections. Secondly, the majority of patients were also managed in various hospital settings rather than clinics, thereby potentially limiting applicability in the ambulatory and community settings. It is unclear what proportion of these infections were healthcare-associated versus community acquired. Patient management differences across sites could also introduce selection bias into the findings. Finally, the limited patient numbers from some study sites may have limited the ability to fully control for site effects when deriving an adjusted risk score. Despite this, the study sites that were controlled for did not result in exclusion of risk factors from the initial risk score, suggesting that site effect would not substantially change the most important risk factors. Lastly, although country-level surveillance MRSA percentages were obtained from the European Centre for Disease Prevention and Control and World Health Organization, this may not be the best method to estimate true local prevalence and it is unclear if more granular local data would improve or impede the ability of these methods to predict MRSA.

Conclusions

In conclusion, an MRSA SSTI risk assessment with an individual patient approach using previously published MRSA risk factors with or without knowledge of national MRSA surveillance percentage was significantly more predictive of MRSA SSTI than MRSA surveillance percentages alone. Such risk assessment approaches may be simply administered at the point of care (risk score) or through an electronic clinical decision support application (risk score or MRSA probability estimate). It is unclear at this time how such risk scoring approaches would affect antibiotic prescribing patterns or patient outcomes in SSTI. Future research to evaluate this or other MRSA risk scoring approaches in SSTI antimicrobial stewardship should be conducted prior to widespread clinical implementation.