Introduction

Hospital-acquired infections (HAIs) present a substantial problem for healthcare providers, as these infections occur with relatively high frequency [1] and each case may result in considerable damage [2]. There has been a recent increase in the frequency of published studies analyzing the incremental cost of HAIs at the hospital level, with many studies reporting quantified estimates. The driving force behind this recent increase in the number of cost of HAI (COHAI)-related research is the rationale that at least 20% of HAIs have been shown to be preventable [3].

In this way, COHAI research serves the following two purposes. Firstly, the quantification of COHAI can highlight the necessity of HAI control measures. As the containment of rising healthcare costs is a common goal among industrialized nations, the prevention of HAIs could be expected to curtail unnecessary costs. Secondly, COHAI quantifications can contribute to elucidating the savings effect necessary in the calculation of cost-effectiveness and cost-savings of HAI control measures. The rising importance of investigating the economic evaluations of HAI measures has been accelerated by the decision of Medicare in the US to withhold reimbursement to healthcare providers for preventable errors, which includes HAIs. Recent guidelines permit researchers to utilize COHAI estimates from existing published literature when evaluating the economic impacts of control measures [2], and some studies have been previously shown to use existing estimates [4].

In recent years, discussions concerning the external validity of economic evaluations have gained momentum within medical circles, with the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) using the term “transferability”. The ISPOR task force has acknowledged that cost-effectiveness results may not be directly applied to other jurisdictions due to major differences with the context and circumstances in which the original results were produced, and that, when applying results to other jurisdictions, it is vitally important to account for differences in unit costs and clinical practice variations [5]. It has also been suggested that the high level of diversity in economic evaluations are due primarily to dissimilarities in unit costs, followed by variations in clinical practice [6]. Furthermore, the BMJ checklist [7], various national guidelines [5], two seminal texts for economic evaluation research [8, 9], and other researchers [10] have also emphasized the importance of reporting both resource consumption quantities and unit costs separately in order to improve the transferability of economic research.

The aims of this study were to conduct a systematic review of published studies that have produced COHAI estimates and to evaluate the quality of these estimates from the perspective of transferability. Also, we sought to investigate if studies evaluated as possessing high transferability are correspondingly well regarded by other researchers by analyzing the number of citations.

Methods

Study selection

We conducted a systematic review of published articles that had produced estimates of incremental COHAI (not including length of stay). This systematic review was conducted according to the general principles of the Cochrane Collaboration framework. The selection criteria for candidate studies were those that produced original COHAI estimates that were published in English from 1980 to 2006. We excluded review papers, editorials, meta-analyses, and decision analyses. The following keywords were used in a MEDLINE search to extract candidate publications: [“economics”(Subheading) OR “Hospital Costs”(MeSH)] AND [“Cross Infection”(MeSH) OR “Surgical Wound Infection”(MeSH) OR “Bacteremia”(MeSH) OR “Bacterial Infections”(MeSH) OR “Sepsis”(MeSH) OR “Staphylococcal Infections”(MeSH) OR “Pseudomonas Infections”(MeSH) OR “Pneumonia”(MeSH) OR “Urinary Tract Infections”(MeSH)]. Studies that fulfilled the aforementioned criteria were then subjected to a two-step review process: first, an abstract review, followed by a full-paper review, before the decision to include them in the final assessment was made. The initial abstract review was conducted to identify: (1) studies that produced original cost estimates for treatment instead of those that used published cost estimates and (2) studies focusing on healthcare-acquired infections versus those that focused on or included community-acquired infections. Studies included in the subsequent full-paper review were those that had produced original estimates of costs of treating HAIs and those that could not be properly evaluated on the basis of the abstract alone. In the full-paper review, the estimates of the COHAI were evaluated, and the transferability of each paper was investigated. Both the abstract review and the full-literature review were conducted independently by two evaluators (HF and JL) whom possess backgrounds in economic evaluations and infectious diseases. Non-congruent evaluations were discussed before decisions were made. Additionally, we included publications cited within the extracted publications that produced original cost estimates of COHAI.

Study assessment

All selected papers were assessed for: (1) estimates of cost attributable to HAI, (2) transparency of cost estimates, (3) costing methods, and (4) record of the number of citations between publication and the year 2009. The citation record, obtained from the ISI Web of Science, was used to investigate if studies evaluated as possessing high transferability are correspondingly well regarded by other researchers. The transparency of cost estimates and costing methods were evaluated based on the two evaluation axes as described below.

Evaluation criteria for transferability: evaluation axis 1

The first axis of evaluation assesses the clarification of the scope of costing, i.e., the level of transparency on the reporting of items included in the estimates and how the estimates were calculated. Without clarifying the scope of costing, such as detailed description of costing components and data for both quantity and unit price of resources in each cost component, readers would be unable to judge the potential applicability to their own analyses. Therefore, our first axis assesses whether estimates have clarified scopes of costing, and we have established a hierarchy of four levels of transparency as follows:

  • Level A: all components of costs were described and data for both quantity and unit price of resources were reported for each component.

  • Level B: all components of costs were described and data for costs in each component were reported. This included studies that used graphical presentations of the aforementioned data.

  • Level C: all components of costs were described but data for costs in each component were not reported.

  • Level D: only the scope of costing was described but the components of costs were not described. For example, studies that only reported terms such as “hospital stay” or “direct costs” without further exposition were evaluated at Level D.

When there were studies that used two or more costing methods to produce COHAI estimates, we categorized these studies into the higher level of costing method.

Evaluation criteria for transferability: evaluation axis 2

However, differences in the methods used to calculate unit costs in the first evaluation axis may result in large variations in COHAI estimates, thereby, affecting the transferability of these estimates. Therefore, the second axis of evaluation was used to identify costing methodologies. The optimal choice of costing methods is the use of micro-costing or quasi-micro-costing, i.e., activity-based costing, in which the measurement of actual resource consumption is attempted. The second costing method involves the use of relative value units (RVUs). The next costing method uses charge data based on the ratio of costs to charges (RCCs). The difference with the use of RVUs is that RCCs utilize cost data at the hospital, medical department, and diagnostic categories (such as the diagnosis-related group [DRG]) levels, using ratios against charge data at each level, and is, therefore, intrinsically reliant on charge data. In general, cost estimates based on the use of RVUs are considered to be more accurate than those based on RCCs, as the former directly applies relative cost information on each item consumed [11]. The fourth costing method involves the use of unmodified charge data. As there is a strong political dimension in the determination of charge data, the external validity of such estimates to different contexts is drastically reduced. The final category of the evaluation of the method of cost estimation includes studies that offer no information to readers about the methodology used.

Statistical analysis

In recent years, discussions concerning the external validity of economic evaluations have gained momentum in healthcare research. As such, changes in proportions of the transferability of COHAI estimates by year of publication were analyzed using Fisher’s exact test. Also, COHAI estimates provide a foundation for the quantification of additional resources attributable to HAIs, and as these estimates can be used in the calculations of cost-effectiveness and cost-savings analyses of infection control measures, there are many chances for such studies to be cited by other researchers. With the purpose of verifying whether studies that have been assessed as having a higher level of transferability are more frequently used by other researchers, we studied the post-publication number of citations per year for each paper, calculated the median of this citation record categorized by the two evaluation axes of transferability, and compared them using a Kruskal–Wallis test.

Results

The search of MEDLINE using our criteria returned 3,069 distinct results, and after conducting abstract and full-paper reviews, we identified 79 studies that estimated incremental COHAI. In addition, we identified 110 other publications cited by these 79 studies. Of these 110 publications, ten fulfilled our criteria for inclusion in our analysis. There were, therefore, a total of 89 publications included in our final assessment [12–100].

The characteristics of the studies included in the sample are shown in Table 1. With regard to the types of infections targeted in these studies, there were 28 COHAI estimates that targeted surgical site infections (SSI), 19 estimates for bloodstream infections (BSI), 12 estimates for pneumonia and ventilator-associated pneumonia (VAP), ten estimates for urinary tract infection (UTI), five estimates for respiratory tract infection (RTI), and 39 estimates for unspecified HAIs. Forty-one of the studies were conducted in the US, six studies in Turkey, five studies in the UK and France, and four in Taiwan. There were ten studies in the 1980s, 21 publications in the 1990s, and a steep increase to 58 studies from 2000 to 2006. Clarification of the scope of costing was graded from Levels A to D. There was only one study graded at Level A (1.1%), 35 studies graded at Level B (39.3%), 16 studies graded at Level C (18.0%), and 37 studies graded at Level D (41.6%). Variations in costing methods showed that the most frequent method was the use of charges as proxy, with 30 studies (33.7%). This was followed by 28 studies with unknown methods of costing (31.5%), 18 studies that used actual costs (20.2%), 11 studies with RCCs (12.4%), and two studies with RVUs (2.2%). There were three studies that used multiple costing methods to produce COHAI estimates, consisting of one study using actual costs and charges, one study using actual costs and an unknown method, and the final study using charges and an unknown method. The first two studies were categorized as actual costs, while the third was included with the studies that utilized charges.

Table 1 Characteristics of papers which estimated the additive cost of hospital-acquired infection (HAI) (n = 89)
Full size table

The results of the evaluation of the transferability of the COHAI by infection route are shown in Table 2. As shown in the table, there are very large variations observed in the cost estimates among the publications. These variations could be attributed to differences in the items included in costing, clinical practice patterns for HAIs, unit costs, difference between actual costs and charges, and cost estimation methods. As there is, therefore, no real discernable meaning to the aggregation of COHAI estimates by infection type, this study presents the estimates by publication, country of origin, type of hospital used in the sample, type of patients used in the sample, additional cost estimates to HAI, clarification of the scope of costing, and costing methods. The only study that reported both quantities of resources consumed and unit costs for each costing component used for the cost estimates was Coello et al. [12].

Table 2 Impact of hospital-acquired infection on hospital cost and results of the assessment of transferability
Full size table

The clarification of scope and costing methods by year of publication are shown in Table 3. As there was only one study [12] that reported both the quantities of resources consumed and unit costs and, therefore, could be evaluated as Level A, this study was included together with Level B studies. We observed no change and an increase from the 1990s to post-2000 in the proportions of studies that were evaluated as A/B or C, respectively. Contemporaneously, the proportion of studies evaluated as Level D, in which the cost estimation methods were unclear, had decreased in the same time period. However, we did not find a statistical significance associated with this change in proportions. The use of charge data was categorized separately in costing methods. We observed a trend in which the proportion of studies that utilized actual costing or RVU methods, and not charge data, had increased since the 1980s to the present. There was a reduction of studies that used charge data to estimate costs from the 1980s to the 1990s, and after 2000, these comprised approximately 50% of reports. On the other hand, studies which used unknown cost estimation methods had reduced in 2000. The changes in cost estimation methods by year of publication have been shown to be marginally significant (P = 0.039).

Table 3 Number of published papers by the clarification of scope and costing method by publication year
Full size table

The post-publication numbers of citations per year by evaluation of transferability are shown in Table 4. With respect to the clarification of scope, there was no statistical difference (P = 0.278) in the median number of citations among studies evaluated at Level A or B (median = 2.86), Level C (median = 3.93), and Level D (median = 3.83). With regard to costing methods, however, we found a significant difference (P = 0.025) in the number of citations between studies that used cost estimates without charge data (median = 4.87), cost estimates with charge data (median = 3.48), and unknown methods (median = 2.45).

Table 4 Times of citations per passed year by the clarification of scope and costing method by publication year
Full size table

Discussion

As the concept of transferability is still burgeoning, there is a possibility that the existence of COHAI estimates with little or no transferability may detrimentally influence the decision-making process in developing infection control measures. The following three conclusions were derived from this study: (1) there is a large degree of variation in COHAI estimates among publications, (2) there is a large degree of variation in the transferability of these COHAI estimates among publications, and (3) there is no significant difference in citation frequency between COHAI estimates with high transferability and COHAI estimates with low transferability. In order to scientifically implement infection control measures, it is necessary to conduct cost-effectiveness and cost-saving analyses on various control measure programs. However, as this study shows the existence of variations in these estimates, on which the calculated incremental cost-effectiveness ratios (ICERs) for infection control programs are heavily dependent, this may incorporate citation bias. In the current situation where tests for the transferability of estimates are not required [2], analysts are freely allowed to cite COHAI data. As such, there is the danger that the credibility of ICERs calculated for these infection control programs may be severely compromised. As such, we feel that, in order to ensure an appropriate contribution to the infection control program decision-making process, it has become essential for researchers who estimate COHAI, analysts who use COHAI estimates for decision-making, as well as relevant journal reviewers and editors to recognize the importance of a transferability paradigm.

Various publications have previously identified barriers to the external validity of economic evaluations. Sculpher et al. [6] and O’Brien [101] identified the following factors that hinder generalizability based on data from previous studies: (1) clinical factors, (2) healthcare system factors, and (3) patient factors. Clinical factors represent the problems associated with variations among healthcare providers with regard to processes of care. As variations in clinical practice are affected by country, community, and individuals [102], economic evaluations of practice should not be directly extrapolated to other jurisdictions. However, by using the evaluation axis of the clarification of the scope of costing that we have proposed, it becomes possible to judge whether there is sufficient information necessary to evaluate the transferability of the estimates. For example, in the case of COHAI clarification of scope, descriptions in detail of the medical resource consumption amounts by processes of care (e.g., the different mean values or standard errors of the groups) between infected and uninfected groups allows readers to comparatively evaluate the process of care in their own institution with the process of care in the institution where the original evaluation was produced. It, therefore, is possible to adjust for the intrinsic differences and allow for transferability to a reader’s own institution. Healthcare system factors could affect both unit costs and clinical practice. In our evaluation axis of the clarification of the scope of costing, therefore, we included not only resource consumption amounts in detail, but the unit costs as well. By specifying the unit costs used by the sample institution, readers can compare and appropriately adjust for the differences with their own institution, thereby, increasing the study’s usefulness as a reference or source of applicable data. Patient factors may also influence COHAI estimates, particularly in severity biases or selection biases. While this represents an important issue in COHAI research, it is distinct from transferability, which is the focus of this paper.

In general, it is desirable for cost estimates to fulfill the Level A criteria of providing precise reports of unit costs and resource loading amounts. However, in the field of cost of illness estimates, we believe that it is more realistic to accept estimates that attain the standards of Level B, where all components of costs were described and data for costs in each component were reported. This is because providing precise data on unit costs and resource amounts used for all medications, tests, and manpower involved in medical treatment would be prohibitively extensive. Also, even in cases where medication costs are summarized in a consolidated amount, and not broken down by individual items, the inclusion of detailed information on medications would severely hamper efforts to describe unit costs and resource amounts. On the other hand, if the cost items were not sufficiently clear, other researchers would be unable to evaluate transferability. Therefore, it is essential to provide the clarification of Level B. While only one of the studies in our sample, that of Coello et al. [12], fulfilled the standards of Level A, 40% of the papers sampled were able to fulfill Level B standards, which may point to its practical usability. When elucidating the details of cost estimates, in cases subject to the limitations of space, the use of journal web sites should be maximized [103].

With regard to costing methods, it is advisable for estimates published in international journals to be based on either actual costing or RVUs. According to Shwartz et al. [11], by providing the means of individual patient cost estimates by each DRG category based on RCCs categorized by department, the errors between cost estimates based on the RVU method would be minimized, and, therefore, may secure an acceptable level of accuracy. However, as COHAI are focused on the individual costs per patient, and not per department, the accuracy of the RCC method may not be sufficient. As such, the use of RCCs and charge data to calculate estimates could be thought to be appropriate for domestic journals. According to Howard et al., while estimates based on RCCs are acceptable [104], charges are determined politically and, therefore, have high usability for other researchers in the same country using the same payment systems. On the other hand, descriptions of costing methods used were markedly scarcely, as studies whose costing methods could not be determined comprised 31.8% of the sample, which highlights a serious issue in the quality of economic evaluations. The simple reporting of “costs” without further exposition would make it difficult for readers to determine if actual costing or charges had been used. As providing the necessary details would be easy on the part of the reporting analyst, this would appear to be a negative effect caused by their insufficient recognition. After 2000, while there was a reduction in the proportion of publications with an unknown method of costing, the complete elimination of this type of study may be the most important step to increase the overall quality of economic evaluation studies.

While we observed a low citation level for studies that scored highly on the clarification of the scope of costing evaluation axis, there was a statistically higher citation level for studies that used cost estimates without charge data, which were evidently usable to other researchers. As discussions on the topic of transferability has recently started to gain momentum [5, 6, 10, 101], it is likely that its importance was not previously well understood among researchers. On the other hand, costing methods had been addressed by Finkler in his forward-thinking paper in a high-impact medical journal [105], as well as emphasized by Gold et al. [9] in their economic evaluation study, which may have had a large impact on improving the costing methods of researchers.

While there have been previous review studies focused on COHAI, this is the first paper to address the issues of the possibility of the utilization of COHAI estimates from the point of view of other researchers. Some researchers had reviewed COHAI publications and calculated the mean additional costs by infection type [2, 106]. While these studies used this data to appeal for the necessity for infection control measures, taking into account the differences among target samples, estimation methods, items included in costing, unit costs, and clinical practice leads to the conclusion that there is very little meaning in the simple aggregation and comparison among existing COHAI studies. These reviews, therefore, did not provide enough information to allow for other researchers to identify which COHAI estimates could be used in their own analyses. Few review studies have elucidated the estimation methods in COHAI. In this study, we present information to allow other researchers to identify the transferability of each paper, as well as a guide to assist researchers who intend to conduct economic evaluation studies for infection controls and decision-makers for infection control policies. In addition, our evaluation axis has a further characteristic in that it has a framework that allows for the four-layered evaluation of the degree of clarification of the scope of costing for each economic evaluation study. We believe that this will assist other researchers in gaining a greater understanding of transferability.

This evaluation of transferability was conducted based on the reported methods in each study in the sample. However, there is the possibility that the description in the methods had been simplified due to space limitations, resulting in the study being evaluated at a lower level than in reality. However, while this possibility may exist, we are convinced of the appropriateness of basing our evaluation on the reported information in this study, as other readers would only have access to the reported information. The current environment of many medical journals is such that there is a trend towards simplifying studies for the convenience of readers. However, in the case of economic evaluations, information regarding transferability would influence the value of the research, and should, therefore, be reported in greater detail.

While the Cochrane Collaboration generally advocates the use of multiple databases for searches, Sassi et al. have argued that the use of MEDLINE is the key source for reviewers of economic evaluations, and searches of other databases produces “limited additional yield” [107]. We have, therefore, decided to utilize the MEDLINE database solely in this review. Furthermore, a citation search using the ISI Web of Science database only revealed duplicates of the papers that were already identified by the MEDLINE search, thereby, supporting the use of MEDLINE as the sole database for review analysis searches. We also only included English language studies in our review. However, because of the large number of studies evaluated, we believe that our results of the evaluation of the transferability of COHAI reflect the actual stature.

As this study is an evaluation of transferability based on judgment criteria, a study which scores highly in this investigation will not necessarily score highly according to the evaluation framework for economic evaluation research developed by Drummond et al. [8]. An important component of Drummond et al.’s framework is the inclusion of an estimation sample in the evaluation sample. However, this would seem to be more related to internal validity than external validity. In contrast, the framework used in this study would result in a favorable evaluation of a study if the estimation sample has a high level of transparency, even if it did not include important costing components. Therefore, our framework alone is insufficient for evaluating the quality of economic evaluation studies, and should be used in conjunction with the guidelines as presented by Drummond et al.’s group.

This study evaluated the transferability of published studies that estimated COHAI, and the evaluation axes used here are applicable and encouraged for future use in general economic evaluation research. By improving the transferability of economic evaluations, it becomes possible to apply knowledge from each country and institution to a researcher’s own country and institution, and increase the efficiency and effectiveness of related decision-making processes.