Background

Key performance indicators (KPIs) are used to monitor performance in key areas of a service. These KPIs are implemented at hospital level and are used to identify where organisational performance is meeting desired standards, and where performance requires improvement. They enable the public, service users, and healthcare providers alike to have reliable information on current and desired standards in healthcare services [1]. However, the feasibility of collecting the relevant minimum data set ((MDS) the core data identified as the minimum required to measure performance for a KPI) elements is always a limitation of performance monitoring using KPIs. For example, in a pilot feasibility analysis of four potential Emergency Department (ED) KPIs, approximately half of the relevant MDS items were missing in the patient records [2].

The reporting burden of capturing the relevant MDS elements should not outweigh the value of information when using KPIs for performance monitoring [1]. Preferably, data should be integrated into service delivery, and, where additional data are required that are not currently part of service delivery, there should be an economic evaluation to determine the cost of collecting all the relevant MDS elements [1]. KPIs are often used in the measurement of costs (or benefits) of data collection, and there is, therefore, a need for a systematic review which synthesises and coheres the evidence base regarding economic analyses of hospital data collection for performance monitoring purposes.

The aim of this systematic review was to synthesise the evidence base concerning the costs of hospital data collection for performance monitoring using KPIs and to identify hospital data collection systems that have proven to be cost-minimising. The review also aimed to identify published studies that addressed the benefits of hospital data collection for performance monitoring and to summarise the methods used to evaluate hospital data collection for performance monitoring purposes.

Methods

Research objectives

A systematic review was carried out to identify all published economic analysis and costing studies regarding hospital data collection for performance monitoring. The review was conducted using the methods detailed in the published protocol of the review [3].

Electronic search

The electronic databases used to search for relevant publications included MEDLINE (1946 to May week 4 2014), Embase (1974 to May week 2 2014), and CINAHL (1937 to date) via the Ovid interface. The electronic search strategies were created specifically for each database using relevant index and free text terms. The full search strategy used can be found in the Appendix. We also limited the search to English language publications.

Searching other resources

Additional efforts were made to identify eligible studies by cross-referencing from the reference lists of major publications on the subject and published government reports [3]. We also made additional efforts to identify potential studies relevant to the topic from a ‘grey literature’ (theses, internal reports, non-peer reviewed journals) database (OpenGrey - system for information on grey literature in Europe).

Inclusion criteria

The inclusion criteria were specified based on the type of study conducted, the population involved with data collection, and the intervention that was adopted to record and collect data.

The studies that were included in the review were economic evaluations and cost or feasibility studies that examined hospital level data collection for performance monitoring purposes using KPIs. The definition of KPIs refers to clinical and quality-of-care indicators such as time to treatment and time to initial assessment. Both types of KPI were accepted for inclusion into the review.

There were two broad categories of participants that were involved with data collection, including health professionals, such as doctors and nurses, and non-clinical staff, such as administrators and managers. All studies that used patient reported data were excluded for the purposes of this review.

All types of interventions (as defined by the authors) that collect and record data for the purpose of monitoring performance were included in the review.

Study selection

Two researchers (BG and CJ) independently screened the titles and abstracts of the identified studies and assessed the inclusion of studies using specific inclusion and exclusion criteria (see Table 1) [3]. All studies that analysed costs and benefits/effects of data collection for performance monitoring were included in the review. One researcher (CJ) read the papers in full and re-assessed them for inclusion. If there were discrepancies with study selection decisions, and the two investigators who independently screened the potentially eligible studies could not reach a consensus, we planned to resolve the disagreement through discussion and consultation with a third investigator (AW) [3].

Table 1 Inclusion of studies using specific inclusion and exclusion criteria
Full size table

Data extraction

The purpose of this review was to report the hospital costs and benefits associated with data collection. Information concerning the costs and benefits of data collection were extracted regardless of whether any formal KPIs were implemented at the hospital in the study. One author extracted the data using a tailored data collection form as previously described [3]. The data extraction form was based on a checklist developed by the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) [4]. The data extracted incorporated (i) author, year of publication, and country; (ii) intervention; (iii) study population and setting; (iv) methodology and study design; (v) resources; (vi) costs; and (vii) results. The results were then summarised as part of a narrative synthesis.

Quality assessment

All studies were assessed for their reporting quality. The quality assessment (QA) was based on the CHEERS checklist [4] and was incorporated into the data extraction tables. Because this systematic review was not reviewing typical medical interventions, such as improved screening strategies or new drugs, it was decided that it was unnecessary to assess each study using the full CHEERS checklist. Therefore, with careful consideration, aspects of the CHEERS checklist were included as part of the QA. Table 2 lists the criteria chosen to assess the studies. The results of the QA were tabulated as part of the data extraction process. A description regarding the quality of reporting is presented below.

Table 2 Quality assessment criteria
Full size table

Results

Results of the search

The electronic search yielded a total of 1,130 publications, once duplicates were removed, and five papers were identified through other sources. After reviewing the titles and abstracts of the publications, we retrieved 22 full-text versions of publications for possible inclusion into the review. Once the full-text versions were examined, we excluded a further 14 publications. The selection process resulted in the identification of eight relevant studies and is summarised using a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [5] (flow diagram presented in Figure 1) (Table 3).

Figure 1
figure 1

Study PRISMA flow diagram.

Full size image
Table 3 Data extraction and quality assessment results
Full size table

Description of studies

Eight studies published between 1975 and 2013 were identified for inclusion in this review. The studies retrieved were conducted in the USA (6), the UK (1), and Belgium (1). All studies were conducted in a secondary care (hospital) setting; however, the studies were set in different hospital departments including the Emergency Department, Surgery, and Medicine. Six of the included studies focused their analyses on computerised data collection systems, and the remaining two focused on information usage in order to create performance indicators.

Four studies focused on the costs and benefits of data collection using information communication technology (ICT) compared to normal practice (hand-written patient records) [6-9]. Two studies examined how already implemented ICT systems can be improved to collect better quality data [9,10], and the two remaining studies [11,12] investigated how information collected by ICT can be analysed for further performance monitoring purposes.

Costs

The included studies reported costs in various ways. Two studies [6,9] calculated the costs based on the differences between the computerised system and the manual system. Two studies [11,12] reported the differences in the total costs of a hospital-acquired medical event before and after the adoption of 1) basic electronic medical records (2011 paper) and 2) a quality indicator (2013 paper). The study by Klimt et al. [7] estimated the costs of transcribing by taking into account the average length of a patient record, true transcriber costs (salary, benefits), and productivity. Tierney et al. [8] reported total costs including bed costs, test costs, drug costs, and other costs; however, no more information about the types of costs included in ‘other’ were reported in the published study. Willems et al. [10] measured the financial impact of a specifically designed follow-up programme for prophylactic antibiotic use by analysing the costs of drug usage before and after implementation. Barnes et al. [13] reported changes in the average subsequent hospital care revenue per trauma service admission rather than focusing on costs. Of the eight included studies identified, five reported start-up costs or associated IT costs in their analysis [7-9,11,12]. However, it must be noted that technology has advanced rapidly since the mid-1980s and has become relatively less expensive over time. For example, the first computers implemented in business were very expensive and could cost thousands, however today’s computers, which are more powerful and efficient, are installed for as little as a few hundred pounds. Therefore, it is not possible to compare the costs of data collected by systems that were made more than 30 years ago with systems produced only a few years ago.

Effects of interventions

The intervention of interest is data collection for performance monitoring purposes and the effects this intervention may have in terms of cost benefits, service delivery benefits, and patient benefits.

All eight studies included in this review report or, at least, discuss the benefits of improved data collection in hospitals. Six studies [7,8,10-13] discuss the potential benefits that may be gained by both the hospital and the patient and the remaining two papers [6,9] report the benefits generated to the hospital only.

Overall, the studies show that it is difficult to measure the benefits of improved data collection systems. Three studies measured benefits by using hospital length of stay. Tierney et al. [8] found that hospital length of stay was reduced by 0.89 days when the computerised system was implemented. Barnes et al. [13] also measured hospital length of stay but found no difference. Encinosa and Bae [12] measured patient benefits in terms of reduced 30-day stay for heart attack, heart failure, and pneumonia. Encinosa and Bae [12] measured the benefits by quantifying the amount of averted adverse drug events occurring in hospital. Three studies [7,10,13] infer the potential benefits to the patient from the results. For example, Barnes et al. [13] assumed that improved documentation results in higher quality-of-care, and Willems et al. [10] stated that, due to the intervention, ‘more appropriate drug administration is likely to have a beneficial effect on antimicrobial resistance, rates of adverse drug events, length of stay in hospital and mortality rates’. One paper by Philp et al. [9] states that ‘patient benefits can only be inferred, but not proven’ and do not make any speculations as to what these might include.

Quality assessment

Overall, the reporting quality of the studies included in this review is mixed. All studies clearly reported the intervention that was being evaluated and what it was being compared against. The objectives of each study were also clear and easy for the reader to follow. The setting and target population were also stated transparently. However, other areas were reported with varying degrees of success.

The costs reported by Klimt et al. [7] were detailed, reporting those associated with the day-to-day running of the Dictaphone-transcribing technology and its set-up. Tierney et al. [8] also reported start-up costs of the microcomputer workstation intervention, but this was stated as an approximation and lacked sufficient detail. Four studies [6,9,11,12] were not transparent in their reporting of what costs where included in their analyses. For example, Encinosa and Bae [12] reported that they include ‘all hospital costs except physician and laboratory costs’ [12]; however, they are not explicit in stating what costs have been included and also they do not provide any justifications as to why they did not include physician and laboratory costs.

The measurements of benefits vary throughout the studies included in the review. Four studies [8,11-13] attempted to measure the benefits through more commonly used units such as hospital length of stay and the probability of death or readmission. On the whole, these studies clearly stated the outcome measures of interest; however justifications and explanations concerning the reasons behind their chosen measurement unit are not reported clearly. Holloway et al. [6] gave a detailed description of why they chose to measure effectiveness on the basis of completeness, timeliness, and operability. In contrast, Klimt et al. [7], who also commented on the effectiveness of the Dictaphone system through benefits such as completeness, timeliness, and operability, did not provide reasons for their choice of effectiveness measurements. Philp et al. [9] focused on the quantification of benefits to the hospital by eliciting nurses’ preferences. Finally, one paper by Willems et al. [10] did not provide a measure of effectiveness and instead stated that the benefits associated with data collection can only be speculated from their results.

The results reported by the studies are clear, but they are limited in their generalisability. Whilst, overall, the results from all of the studies, largely, present a positive effect from the installation of improved data collection systems, the lack of agreement regarding how to measure benefits best and what costs should be included makes it difficult to draw any firm conclusions.

Discussion

Main findings

The feasibility of collecting the relevant MDS elements is always a limitation of performance monitoring using KPIs. This systematic review aimed to synthesise the evidence base concerning the costs of hospital data collection for performance monitoring using KPIs and to identify hospital data collection systems that have proven to be cost-minimising. The main finding of this review is that the evidence base is limited regarding the impact of data capture for performance monitoring purposes and how the data is collected, recorded, and used in a hospital setting. Overall, the studies identified and included in this review weakly indicate that the collection of hospital administrative data and improvements in recording data (installation of computerised systems) can be cost-saving and potentially provide benefits to both hospital management and patients [7-13]. The review also summarises how economic evaluations of data collection systems measure the associated economic costs and benefits. The methods utilised are inconsistent throughout the studies included in this review but that may be due to the lack of research completed in this area.

Unsurprisingly, the outcomes of the review also suggest that the costs and benefits associated with data collection are largely driven by the advance of ICT. The progression of technology, to some extent, is reflected by the year in which the study was performed. The four earliest included studies focused on the costs and benefits of data collection using ICT compared to normal practice (hand-written patient records) [6-9]. Two, more recent, studies discuss how already implemented ICT systems can be improved to collect better quality data [10,13]. The remaining two included studies discussed how the information collected by ICT can be analysed for further performance monitoring purposes [11,12]. A surprising finding of the review is that there seems to be a distinct lack of studies that evaluate ICT data collection systems.

It is worth stating that more general studies have focussed on the costs (and benefits) of data collection processes, for example, Hillestad et al. [14], Kaushal et al. [15], and Himmelstein et al. [16]a.

On a final note, it is interesting that Tierney et al. [8] and Philp et al. [9] draw attention to staff attitudes in regards to the implementation of new data collection systems. Tierney et al. [8] state that ‘systems can only affect costs and quality of care if physicians use them, which will only happen if “costs” are minimised and offset by perceived benefits’. It is, therefore, an imperative that studies take into account the usage of these systems by physicians and hospital staff since that alone may determine the cost-effectiveness of such a system.

Limitations

Firstly, there is currently no reference standard when reporting the costs associated with each intervention. Some of the included studies in this review [8,9] fail to clearly report the individual costs included in their analysis making it difficult for the reviewer to fully understand how the costs were calculated. Other included studies are not transparent when reporting the setup costs associated with an intervention such as the installation and maintenance of IT systems [6,8,10] bringing into question the validity of the costing results. Another important issue concerns the potential for publication bias since almost all hospitals must be engaged in this activity and have some estimate of the cost of their data collection.

It is also important to mention that none of the studies that analysed computerised technology extrapolated the cost results over time to take into account the depreciation of technology, future maintenance costs, or the cost of upgrading the system.

Secondly, the studies highlight the difficulty in measuring the level of quality of care associated with data collection and, as a result, the studies are not consistent in their reporting of this. As mentioned above, one study measures benefits using averted adverse drug events, and three studies measure quality of care using hospital length of stay with some success, but the remaining studies admit that they can only speculate the potential expected benefits of improved quality of care via data collection [9].. As a result, four studies [9-12] acknowledge and recommend that more research is completed to explore the impact of data collection on quality of care.

Conclusions

Given the limitations of this systematic review, it is difficult to conclude whether improvements in data collection systems can save money, increase quality of care, and assist performance monitoring of hospitals. Nevertheless, the results are positive and hint that data collection improvements may lead to cost savings and derive benefits for both the hospital and patient. The review has also highlighted that there is no standard reference of how to measure the benefits and costs associated with data collection; however, it is suggested that studies work towards being more transparent when reporting the methods used and the results obtained. Overall, there is a need for more research regarding the costs and benefits associated with the installation or improvement of data collection systems for performance monitoring purposes.

Endnote

aWe thank an anonymous referee for this observation.