Background

Healthcare-associated infections (HAIs) have become a globally recognized public health issue. The occurrence of HAIs, on the one hand, affects the recovery of patients, prolongs hospital stays, increases patients’ financial burden, and poses a threat to their physical health and life safety, and on the other, it leads to a higher incidence of medical disputes, reduces hospital bed turnover rates, and wastes a significant amount of healthcare resources [1]. Hand hygiene (HH) is the most basic, direct, affordable, and effective method for reducing HAIs [2]. The global hand hygiene compliance (HHC) rate among HCPs in healthcare facilities is reportedly suboptimal, ranging from 20 to 40%, which falls significantly short of the HH standards (90–95%) for HCPs during medical activities established by the World Health Organization (WHO). The first global strategy for infection prevention and control (IPC) placed special emphasis on the requirement for continual surveillance of HH indicators to facilitate prompt feedback [3]. Additionally, it recommended that monitoring the HHC of HCPs in healthcare facilities is a crucial element of HH promotion programs [4]. The inconsistent distribution of regional healthcare resources globally has resulted in varying risks of HAIs among different disease groups within healthcare facilities. A critical issue when it comes to monitoring HH is selecting the most efficient monitoring tool for a particular situation, taking into account the available workforce, healthcare resources, acceptable HAI rates, and cost-effectiveness.

Currently, the most widely accepted approaches for HH monitoring around the globe are direct observation and indirect observation [5]. Direct observation, that is, “trained observers directly observing HHC” is still regarded as the “gold standard” for HHC monitoring and is the one of the most reliable measures of evaluating HHC [6, 7]. This monitoring method is fairly straightforward and can be implemented regardless of the size or existing structure of the hospital. However, it is prone to the Hawthorne effect and is overly tedious and time-consuming, making it difficult to obtain accurate results. Also, this approach makes it challenging to conduct large-scale surveys [8]. The Hawthorne effect refers to people who deliberately change some behavior or verbal effects when they realize that they are being noticed or observed. The higher the Hawthorne effect in HH monitoring, the greater the indication of bias in the results for HH. As the Hawthorne effect increases, it diminishes the accuracy of reflecting actual HH compliance and amplifies bias levels [9]. Indirect monitoring includes but is not limited to estimating the consumption of paper towels, hand sanitizers, and soap, among other products, for maintaining HH or assessing the needs based on the nursing operation database. These methods require less time and resources compared with direct observation but have some biases, such as a lack of evaluation of patient factors and workload. In recent years, with the continuous development of technology, intelligent monitoring systems have been widely used in healthcare facilities and are considered potential alternative solutions for measuring and improving HHC. Previous studies documented the usage of electronic badges with alcohol vapor sensors to monitor whether HCPs practiced HH [10]. Some researchers have used smart rings worn on the fingers to monitor HHC. Compared with the most common manual paper-based direct observation method, using information systems for HHC monitoring has prominent advantages such as sustainability, reduced Hawthorne effect, saved human resources, improved data collection efficiency, and increased data traceability. Further investigations should be performed to evaluate the practicality, cost, acceptability, and cost-effectiveness of HHC monitoring information systems, even though they provide a novel way of accurately monitoring HHC.

This study was conducted to compare various health economic indicators of electronic system monitoring (ESM) and manual paper-based monitoring (MPM) for HHC and to utilize the results to furnish healthcare institutions with varying levels of resources and infection risk with HHC monitoring evidence-based advice.

Methods

Study design

A before and after study in 40 clinical departments with 4,524 HCPs of Zhongnan Hospital of Wuhan University, China, was conducted from November 2022 to January 2023 (MPM implementation phase) and March 2023 to May 2023 (ESM implementation phase; Fig. 1).

Fig. 1
figure 1

Flow chart comparing MPM and ESM

Full size image

Participants

Participants were derived from forty clinical departments with 4,524 HCPs of the hospital.

ESM group

The ESM group used of the ESM as a hand hygiene monitoring tool (March 2023 to May 2023). ESM was accomplished using a self-developed app (patent number: 202230575169.5, REDACTED) embedded in nursing operation mobile including the following functions: HHC monitoring, automatic statistics, inquiry, analysis, reminders, and so on. The main process is the infection monitor holding the information equipment, can monitor four medical staff at the same time, in turn in the observed personnel department, name, occupational type, hand health timing and indications, the system automatically generate hand hygiene compliance data, realize hand hygiene compliance monitoring, automatic statistics, query, analysis, remind, and other functions.

MPM group

The MPM group used of the MPM as a hand hygiene monitoring tool (November 2022 to January 2023). MPM refers to the direct manual paper-based monitoring in hand hygiene compliance monitoring. The main process is observation by 2 trained infection monitors, and the observation time is 20 min, and up to 3 medical staff at the same time. The infection monitor should prepare paper monitoring forms in advance, including the observation date, start and end time, observation place (clinical departments), the observed name of medical staff, occupational type, hand hygiene timing, hand hygiene indications, etc. All observations were recorded and filled in manually. The data and results are calculated through manual analysis.

Outcomes

Outcomes included cost-effectiveness, cost-efficiency, Hawthorne effect and indirect cost-benefit analysis. This study has been approved by the Clinical Research Ethics Committee of Zhongnan Hospital of Wuhan University ( approval number: 2023136 K).

Basic data

Basic data included information concerning beds and HCPs of 40 clinical departments. In addition, these 40 clinical departments were divided into three levels of high risk (11 clinical departments), medium risk (23 clinical departments), and low risk (six clinical departments) according to the previous literature [11,12,13] (sTable 1).

Costs

The cost of ESM included the cost of software, electricity, and labor cost. ① the cost of developing software (C1): the development cost of the software was 50,000 CNY, the use cycle is at least three years; our study lasted three months, and the average payment per department was 31.25 CNY; ② the cost of electricity (C2): a full charge required one degree of electricity, and according to the power supply standard of the hospital, the cost of a full charge of the device was 0.8 CNY [14], where the charging time of device was once a month; ③ labor cost (C3): The labor cost is the time cost of hand hygiene monitoring, including the time for providers to wash their hands. Labor cost = labor time×the hourly labor cost. Monitors are responsible for manually recording their monitoring time on paper forms during the designated period, ensuring compliance with WHO requirements. According to the investigation and statistics of Wuhan Municipal Bureau of Human Resources and Social Security, the average monthly salary of employees in 2022 was 8,845 CNY [15], and the hourly labor cost was about 40 CNY.

The cost of MPM included the cost of paper and the labor cost. ① cost of paper (C4): the cost of an A4 paper was estimated to be 0.04 CNY. According to the design of WHO HHC questionnaire [16], it took two sheets of paper to conduct an MPM; ② labor cost (C3): The manual monitoring time required by the MPM monitoring mode is derived from the automatie statistics of the electronic system once the monitor initiates and completes the monitoring tasks. The costs and calculation of the two monitoring methods are summarized in Table 1.

Table 1 Cost calculation of two monitoring methods
Full size table

ESM directly generates results such as hand hygiene compliance through the system, reducing the time of recording and calculating data through manual approach. The labor cost, being the most expensive among all costs, is significantly reduced by ESM, resulting in a more favorable total cost compared to MPM.

HHC Monitoring effectiveness

HHC includes hand wash and alcohol-based hand rub. HHC means that HCPs adhere to the established HH guidelines and the steps, time, and scope of HH are in line with regulations. The purpose of evaluating the effectiveness of hand hygiene monitoring is to compare the hand hygiene compliance of medical staff under the two monitoring methods. HHC cost-effectiveness was quantified by monitoring HH action and opportunity by respective part-time IPCPs from clinical departments and based on Eq. 1 [17]:

$$\begin{aligned} &\text{HHC}\;\text{monitoring}\;\text{effectiveness}\;\\&=\;\frac{\text{HH}\;\text{action}}{\text{HH}\;\text{opportunity}}\times\;100\% \end{aligned}$$
(1)

HHC Monitoring Efficiency

Cost-efficiency was evaluated by time consumed on HHC monitoring and investigated by 10 full-time IPCPs from departments of IPC and nursing. HHC was quantified through ESM by directly inputting the observed number of HH actions, and the system automatically generated the result of HHC and uploaded the same to the computer. The use of MPM required the observers to record the observed and actual HH moments in a paper observation sheet and calculate the results manually. HHC monitoring efficiency (Eq. 2) was estimated as the time spent for each monitoring moment:

$$\begin{aligned} &\text{HHC}\;\text{monitoring}\;\text{efficiency}\;\\&=\;\frac{\text{The}\;\text{number}\;\text{of}\;\text{monitoring}\;\text{moments}}{\text{Total}\;\text{time}\;\text{consumed}}\times100\% \end{aligned}$$
(2)

Hawthorne effect

The Hawthorne effect refers to a change in behavior as a motivational response to the interest, care, or attention received through observation and assessment [18]. The Hawthorne effect may deliberately increase HH when the observer perceives that he/she is being observed during HH monitoring. The purpose of evaluating the Hawthorne effect is to compare the influence of the Hawthorne effect brought by the two monitoring methods. We believe that the main difference between the two monitoring methods is that MPM requires the infection monitor to hold a paper observation sheet during the observation process and manually record the observed data in the paper observation sheet. When using ESM, the monitoring device held by the infection monitor is the original operating phone used in clinical nursing, and the hand hygiene monitoring function is nested within it. Therefore, HCPs have no way to know the purpose of the infection monitor holding the device. In this study, the HHC during the period when HCPs were not observed, i.e., the unobservation period of different risk departments was derived from previous references (sTable 2). The extent of the Hawthorne effect was calculated using Eq. 3 [19]:

$$\begin{aligned}\text{Hawthorne}\;\text{effect}\;=\;\frac{\text{The}\;\text{number}\;\text{of}\;\text{HHC}\left(\text{during}\;\text{the}\;\text{observation}\;\text{period}-\text{during}\;\text{the}\;\text{unobservation}\;\text{period}\right)}{\text{The}\;\text{number}\;\text{of}\;\text{HHC}\;\text{during}\;\text{the}\;\text{observation}\;\text{period}}\times100\%\end{aligned}$$
(3)

Indirect benefit

Indirect benefit was evaluated in terms of disease burden due to HAIs during the monitoring period. HAI rates were estimated as the number of new infections in the total number of hospitalized patients in a certain period of time (Eq. 4):

$$\begin{aligned}&\text{HAI}\;\text{rates}\;=\\&\;\frac{\text{The}\;\text{number}\;\text{of}\;\text{new}\;\text{infections}\;\text{in}\;\text{the}\;\text{same}\;\text{period}}{\text{Total}\;\text{number}\;\text{of}\;\text{hospitalized}\;\text{patients}}\times100\%\end{aligned}$$
(4)

Data on HAI rates in this study were exported from the hospital information system of hospital. HAI rates without HH of different risk departments were estimated from references (sTable 3). The average disease burden per case of HAIs was 39,800 CNY estimated from references [20].

Outcomes

Cost-effectiveness ratio, cost-efficiency ratio, Hawthorne effect, and indirect cost–benefit ratio are the main outcomes and target effects and based on Eq. 5 to 9[21]:

$$\text{Cost}-\text{effectiveness}\;\text{ratio}\;=\;\frac{\text{Cost}}{\text{Effectiveness}}\times100\%$$
(5)
$$\text{Cost}-\text{efficiency}\;\text{ratio}\;=\;\frac{\text{Cost}}{\text{Efficiency}}\times100\%$$
(6)
$$\mathrm{Cost}-\mathrm{benefit}\;\mathrm{ratio}\left(\mathrm{CBR}\right)=\frac{\mathrm{Benefit}}{\mathrm{Cost}}\times100\%$$
(7)
$$\begin{aligned}\text{Benefit}\;=\;\frac{(\text{HAI}\;\text{rates}\;\text{during}\;\text{the}\;\text{survey}\;\text{period}-\text{HAI}\;\text{rates}\;\text{witout}\;\text{HH})\times\text{Disease}\;\text{burden}\;\text{due}\;\text{to}\;\text{HAIs}}{\text{Cost}}\times100\%\end{aligned}$$
(8)
$$\begin{aligned}&\text{Incremental}\;\text{cost}-\text{effectiveness}\;\text{ratio}\\&=\frac{\text{Cost}_{\text{MPM}}-{\text{Cost}}_{\text{ESM}}}{{\text{Effectiveness}}_{\text{MPM}}-{\text{Effectiveness}}_{\text{ESM}}}\times100\%\end{aligned}$$
(9)

Data and analyses

Cost calculation was recorded into Microsoft Excel by two research members, and the formula was set in Excel and calculated to avoid any computational errors. SPSS (version 16, SPSS Inc) software was used for statistical analysis, and the chi-square test was used to detect any statistical differences within and between groups. Based on the previous studies and combined with the data characteristics of this study, the decision tree model [22] was used to analyze cost-effectiveness. The decision node of the decision tree model was HHC monitoring, and the branches of the opportunity node were ESM and MPM. The nodes of these two branches are divided into HHC and its lack thereof, i.e., no-HHC. The decision tree model was performed using TreeAge Pro 2022 (Fig. 2).

Fig. 2
figure 2

Decision tree model

Full size image

Results

Comparison of cost-effectiveness of the two monitoring methods among different departments

The total cost spent on ESM for the 40 departments (17,702.92 CNY) was 4,123.76 CNY lower than that of MPM (21,826.68 CNY). The HHC of MPM (80.16%) was higher than that of ESM (69.82%), and the difference was statistically significant (p < 0.01; Table 2). The HHC of ESM in high-, medium-, and low-risk departments was 61.33%, 76.82%, and 56.59%, respectively (p < 0.01). The HHC of MPM in high-, medium-, and low-risk departments was 78.79%, 83.42%, and 65.85%, respectively (p < 0.01).

Table 2 Cost-effectiveness analysis and incremental cost-effectiveness analysis of the two monitoring methods among the different risk departments
Full size table

In high- and medium-risk departments, the cost-effectiveness ratio of ESM (7,977.90 CNY and 13,794.60 CNY, respectively) was lower than that of MPM (9,039.61 CNY and 14,549.05 CNY, respectively), indicating that the average cost of ESM was lower when improving HHC. On the contrary, in low-risk departments, the cost-effectiveness ratio of ESM (3,910.77 CNY) was higher than that of MPM (3,899.06 CNY), indicating that the average cost of ESM was higher. Compared with ESM, the incremental cost of MPM in all departments was 4,123.76 CNY, the incremental effectiveness was 10.34%, and the incremental cost-effectiveness ratio was 39,881.62 CNY. Every 1% increase in the HHC compliance of MPM was found to result in a 39,881.62 CNY reduction in cost. However, in low-risk departments, every 1% increase in the HHC of MPM was noted to result in a 3,828.29 CNY reduction in cost. As shown in Fig. 3, both ESM and MPM are undominated strategies. Our tornado analysis revealed that the cost of MPM was the only factor that had the largest impact of the overall strategy (Fig. 4).

Fig. 3
figure 3

Cost-effectiveness analysis of using the two monitoring methods among the different risk departments Undominated (An undominated strategy indicates that when there are multiple strategies to choose from, the dominant strategy is better than the others)

Full size image
Fig. 4
figure 4

One-way sensitivity analysis on incremental cost

Full size image

The cost-effectiveness acceptability curve (CEAC, Fig. 5) illustrates that when the cost of all departments, namely, high-risk departments, medium-risk departments, and low-risk departments was 40,000 CNY, 15,000 CNY to 20,000 CNY and 5,000 CNY to 10,000 CNY, respectively, the choice of ESM had a higher cost-effectiveness.

Fig. 5
figure 5

Cost-effectiveness acceptability curve of using the two monitoring methods among the different risk departments

Full size image

Comparison of cost-efficiency analysis of using two monitoring methods among different departments

Between the two monitoring methods, the efficiency of ESM (48.11%) in all departments was higher than that of MPM (14.20%), and the difference was statistically significant (p < 0.01). The cost-efficiency ratio of MPM in all departments (155,775.56 CNY) was higher than that of ESM (36,796.76 CNY), indicating that ESM was efficient and low-cost. In high-risk departments, the maximum gap in the cost-efficiency ratio of the MPM and ESM were 72,013.35 CNY and 8,858.02 CNY, respectively (Table 3).

Table 3 Cost efficiency analysis of using two monitoring methods among the different risk departments
Full size table

Comparison of the extent of Hawthorne effect of using two monitoring methods among different departments

The extent of Hawthorne effect of MPM of HHC in all departments (43.99%) was higher than that of ESM (35.69%), and the difference between the two monitoring methods was noted to be statistically significant (p < 0.01). Among departments with different risk levels, the extent of Hawthorne effect of MPM (59.45%) and ESM (47.90%) in high-risk departments were all higher than those in other departments, and no statistical difference existed between the two monitoring methods in high-risk departments (p = 0.940). The extent of Hawthorne effect of MPM (41.46%) and ESM (36.20%) in medium-risk departments was the lowest, and there existed a statistical difference between the two monitoring methods (p < 0.01; Table 4).

Table 4 Hawthorne effect analysis of using the two monitoring methods among different departments
Full size table

Comparison of cost-benefit analysis of the two monitoring methods among different departments

When ESM was used as the HHC monitoring approach, the HAI rates (1.39%) in all departments were higher than that when MPM was used (1.34%), but no statistical difference was observed (p = 0.562). When comparing the CBR of MPM and ESM, the average CBR of ESM in all departments (2,722.59 CNY) was higher than that of MPM (2,454.37 CNY), indicating that in the current analysis, the CBR of ESM was better than MPM. Among departments with different risk levels, The average CBR of MPM (3,305.45 CNY) was 304.51 CNY higher than that corresponding to ESM (3,000.94 CNY), indicating that it is more cost-benefit to use MPM in medium-risk departments (p < 0.05; Table 5).

Table 5 Cost-benefit analysis of using the two monitoring methods among the different risk departments
Full size table

Discussion

The WHO Hand Hygiene Research Agenda for 2023–2030 underscores the need to advance HH research and increase the effectiveness of HHC monitoring through the use of information technology in the next two decades [23]. HHC monitoring is widely recognized as a highly challenging task that requires expertise and training, as this determines the accuracy of the monitoring results, and it also demands human resources and time investment, as this determines the completeness of the monitoring and the timeliness of feedback. This study marked the initial examination of a comprehensive health economic analysis to evaluate the utilization of ESM and MPM in HHC monitoring. The findings revealed that ESM outperformed MPM in terms of cost-effectiveness, cost-efficiency, cost–benefit, and the extent of Hawthorne effect.

In terms of cost-effectiveness, the total cost of ESM was lower than that of MPM. Particularly in high- and medium-risk departments, the average cost of improving HHC using ESM was lower than that of MPM. In line with this study is the observation that most commonly used information technology systems for HHC motoring can significantly improve HHC among HCPs (OR = 3.06, p < 0.001) [24]. With continued improvement of electronic monitoring systems, combining electronic monitoring with observational methods may provide the best information as part of a multimodal strategy to improve and sustain HHC rates among HCPs. Moreover, this study further distinguished the cost-effectiveness of clinical departments of different risk levels. The high-risk settings such as intensive care units has the stricter regulations for HAI prevention and control with an increasing HH opportunities moments. Thus the cost-effective advantage of ESM is more significant [25]. However, in low-risk departments, MPM resulted in higher cost effectiveness [26]. Undeniably, the disadvantages of electronic monitoring in combination with direct observation include, among others, the cost of installation [8]. Hence, MPM were recommended to be applied in low-risk medical departments.

As for cost-efficiency, ESM was noted to be superior to MPM, especially in high- and medium-risk departments, where their efficiency advantage was even more pronounced. We theorized that this could be attributed to the higher-risk divisions, such as the intensive care unit, executing more intrusive procedures, accommodating patients with weakened immune systems, and consuming a great amount of resources in the hospital. The staff in this department were short-handed (with a nurse-to-patient ratio of 2.5 to 3:1 [27]), and there were shortages of equipment, drugs, and other resources, all of which put an extra strain on HHC, making it necessary to keep a close eye on personnel and time expenditure. ESM could help managers better understand the implementation, promptly identify problems and non-compliant behavior, and avoid waste and misuse of resources in high-risk departments. Moreover, through analysis of HH data, resource allocation could be optimized, and work efficiency can be improved [28]. This study’s analysis of cost-efficiency is consistent with the actual clinical demands. ESM could simultaneously monitor the HHC of multiple individuals online, and the data could be automatically analyzed, provided with feedback, and traced, thereby saving a significant amount of workforce and time costs. Kardaś-Słoma [10] also remarked that ESM can automatically collect data, improving the accuracy and timeliness of monitoring. The data collected through an ESM can be further analyzed and used for trend prediction. This analysis can help identify hotspots and high-risk periods for HH issues, enabling targeted preventive measures to improve HHC and correctness.

The Hawthorne effect, which refers to the alteration of behavior on being aware that one is being observed, was chosen as an evaluation indicator for monitoring biases. ESM, being discreetly conducted on nursing mobile devices, offers a certain level of secrecy when conducting compliance monitoring. On the other hand, MPM relies on paper records, making the Hawthorne effect more pronounced. Therefore, ESM was more effective in controlling the Hawthorne effect. Interestingly, it was found that HHC was lower under ESM compared with that under MPM, which may be attributed to a weaker Hawthorne effect associated with ESM. Casaroto’s research [28] in the ICU context demonstrated an HHC rate of 56.3% under MPM and 51.0% under ESM, with the Hawthorne effect being the influencing factor. Another research group [5] demonstrated a downward trend in HHC rates among HCPs after the introduction of intelligent monitoring systems. This phenomenon mainly occurs when direct observation and ESM are used simultaneously to measure HHC. Direct observation produces the Hawthorne effect, resulting in higher measurement results compared with those obtained through intelligent monitoring systems. These findings suggest that information technology-enabled monitoring provides a more accurate reflection of the actual situation, enabling the establishment of more precise baseline data for subsequent HHC interventions. Electronic HHC motoring systems can monitor HHC on all work shifts without a Hawthorne effect and provide significantly more data regarding HHC [4].

Finally, a comparison of the occurrence of HAIs during the application of the two monitoring methods was conducted. Although no significant difference was observed in the occurrence of HAIs between ESM and MPM during the study period, the cost-effectiveness of ESM outweighed that of MPM when considering the cost and the burden of patient diseases caused by HAIs. Likewise, Salinas-Escudero [29] observed that within one month of implementing ESM, the number of infections decreased by 46–79 individuals, resulting in cost savings of $308,927 to $546,795 for preventing HAIs.

However, it is important to note some limitations of this study. First, a large number of studies focused on using electronic monitoring systems to monitor HHC, including application-assisted direct observation, camera-assisted observation, sensor-assisted observation, and real-time locating system [4]. This study focused on utilizing electronic monitoring methods for data recording and statistical analysis to reduce manpower costs, without conducting a cost-benefit analysis of other electronic monitoring methods. Hence, the findings cannot be extrapolated to other electronic monitoring systems. However, we intend to conduct an exploratory health economic evaluation in terms of tools for monitoring HHC and provide a reference for future research. Second, the cost of ESM did not consider the potential repair costs resulting from device damage in the later stages. Long-term prospective observational studies are also being conducted to provide a comprehensive health economic assessment of the long-term application of ESM in HHC. Thirdly, the evaluation of HAIs in this study is influenced by various factors beyond hand hygiene that are difficult to control. An additional limitation of our research pertains to its design, where participants were categorized into the “ESM group”” and the “MPM group”. It is important to note that the “ESM group”, while part of the study, was not a pure intervention group, as it also underwent manual HH monitoring. The constraints inherent in our design methodology have diminished the level of evidence we can present. Nonetheless, the exploration of potential efficacy and efficiency, even with a less robust design, offers valuable insights and serves as a foundation for future research endeavors that aim to achieve a higher standard of evidence.

Conclusion

In conclusion, for high-risk departments with multiple HH moments and higher requirements for HHC monitoring frequency and coverage, ESM exhibits notable advantages over MPM in terms of cost-effectiveness, cost-efficiency, cost–benefit, and the Hawthorne effect.