Background

Computerized decision support systems (CDSSs) have the potential to improve quality of care [1, 2]. To the extent that their content is guideline-based, they can support physicians in adhering to the best-available evidence as presented in clinical guidelines, and ultimately lead to improved patient outcomes. Evidence on their uptake and effectiveness, however, is thus far limited and is usually retrieved from small-scale academic-driven settings, in which CDSSs aimed at a limited number of specific decision points are tested [3–7].

In the Netherlands, a CDSS for primary care - NHGDoc - was developed in 2006, as a collaborative effort between the Dutch college of General Practitioners (NHG) [8] and ExpertDoc BV [9], a private company. NHGDoc covers multiple domains of care and provides a variety of patient-specific advices during patient consultation for both general practitioners (GPs) and practice nurses (PNs). Its content is based on the NHG guidelines, the national prevailing guidelines for general practice. The system is gradually being implemented at a large scale and is currently available to approximately 65 % of all Dutch general practices [9].

To evaluate the uptake of NHGDoc and its effectiveness in terms of improving quality of primary care, a large-scale cluster randomised controlled trial (RCT) was conducted within 231 general practices in the Netherlands. More specifically, this two-arm cluster RCT with a follow-up period of 1 year, assessed the effectiveness of NHGDoc with respect to improving the process of medical care (e.g. prescription behavior) as well as patient outcomes (e.g. hospital admissions), which is described in detail in the Methods section [10]. Preliminary analyses reveal that no effects were found on the identified outcome measures [11].

Delivering a complex intervention such as a CDSS in a trial setting does not guarantee that the target group is actually exposed to the intervention as planned. Particularly, in cluster RCTs exposure to the intervention may vary considerably between participating practices [12]. Also, participants’ experiences with the intervention may affect exposure and may help explain the results of a trial. A process evaluation can shed light on the factors/mechanisms responsible for the outcomes of a trial [12, 13].

Therefore, we conducted a process evaluation alongside the trial. The aim of this study was to investigate the exposure to and experiences with our CDSS quality improvement intervention, in order to gain insight into the factors contributing to the intervention’s impact. Results of the process evaluation can provide insight in the feasibility and reproducibility of the intervention [14] and can be used to improve the implementation of NHGDoc as well as similar CDSS interventions.

Methods

Description of the intervention

The CDSS NHGDoc

NHGDoc is a CDSS integrated within the electronic health record system (EHRS) and based on the NHG guidelines, the prevailing guidelines for general practice in the Netherlands [8]. It provides GPs and PNs evidence-based and patient-specific advices during consultation in terms of patient data registration, drug prescription and management.

Basic functions of NHGDoc

When the GP or PN opens a patient file in the EHRS, anonymous patient and performance data are sent to the NHGDoc server. The patient and performance data are compared to the digitized guideline recommendations and in case of a discrepancy between current and advised care, an alert will be sent back to the GP or PN. Also, users have the option to ask or provide feedback from/to ExpertDoc, the organization that has developed and maintains NHGDoc, about the received alert. See [15] for a detailed description of the NHGDoc system.

Personalization functions of NHGDoc

Aside from the basic functions (alerts and feedback), NHGDoc allows the user to adapt the decision support to meet their personal preferences in two different ways. By using alert settings, users can adjust the preferences of the alerts to match their personal needs. They can choose to switch alerts on and off on demand at several levels: the system, the modules (NHGDoc domains), the types of alerts (patient data registration, management, drug prescription), and the patients. The reporting settings allow users to request specific reports with respect to the number and types of alerts they have received per domain within a specific period of time (per year, per month, per week of per day). See [15] for a detailed description of the NHGDoc system.

To evaluate the effects of NHGDoc on quality of care we conducted a cluster randomized controlled trial [10]. All general practices in the Netherlands that, at the time of recruitment had NHGDoc at their disposal, were invited to participate in the study (n = approx. 1.100). Our power analysis revealed that we needed to include at least 122 general practices to detect an effect on our main primary outcome measures. A comprehensive recruitment plan was conducted, consisting of a combination of direct invitations via postal mailings as well as emails, combined with announcements of the evaluation study in several relevant Dutch journals, websites, newsletters and through social media (i.e. Facebook, Twitter and LinkedIn) [10]. A total of 231 general practices gave their consent to participate in the trial. 115 practices were randomly assigned to the control arm, whereas 116 practices were allocated to the intervention arm.

Regular NHGDoc modules (control arm)

General practices assigned to the control arm received the regular NHGDoc decision support [10]. These practices received decision support with respect to all modules (NHG guidelines) that – at the onset of the trial – had already been integrated into NHGDoc. These were cardiovascular risk management, asthma/COPD, diabetes mellitus type II, thyroid disorders, viral hepatitis and other liver diseases, atrial fibrillation and subfertility.

The NHGDoc heart failure module (intervention arm)

General practices allocated to the intervention arm received the same decision support modules as the control arm, extended with the NHGDoc module on heart failure [10]. The NHGDoc module on heart failure is directly derived from the NHG guideline on heart failure [16]. It consists of three types of alerts:

  1. 1.

    Alerts on heart failure in terms of registering patient data;

  2. 2.

    Alerts on heart failure in terms of prescribing (or adjusting the dose of) drugs;

  3. 3.

    Alerts on heart failure in terms of (paying attention to) management aspects.

    See [15] for a detailed description of the module.

At the onset of the trial the NHGDoc heart failure module was activated in the intervention group. Other than that, no activities and time investments were required. General practices and the staff working within these practices were blinded to group allocation. They were even unaware of participating in an intervention trial with a concurrent control group. The trial did not show a significant effect on the identified primary outcome measures (prescribing of ACE inhibitors/angiotensin II; prescribing of beta-blocker; prescribing of diuretics) [11].

Data collection and study measures

The need for ethical approval for the NHGDoc evaluation study was waived by the research ethics committee of the Radboud university medical center. Data for the process evaluation were collected among all participating practices in both the intervention (n = 115) and control arm (n = 116) - whenever applicable - in two different ways: by collecting data from the NHGDoc server and by conducting a questionnaire. Informed written consent was obtained from all practices: each general practice filled out an online registration form in which they agreed with the design of the study and the use of NHGDoc server data as described on the website (www.nhgdoc-evaluatie.nl) and gave consent on behalf of all practice staff (GPs and PNs). Responding to the questionnaire was voluntary and again information was provided on handling the data in terms of confidentiality and anonymity.

We used the process evaluation framework as described by Hulscher et al. [12] whenever applicable. Besides from describing the intervention itself, this framework recommends to describe the actual exposure to the intervention as well as the experiences with the interventions [12].

NHGDoc server

We extracted the following data from the NHGDoc server to measure exposure to the intervention in both study groups: NHGDoc activity (send NHGDoc requests), switched off NHGDoc modules including heart failure (intervention group only), opening of alerts including for heart failure patients (intervention group only). Data were collected continuously during the trial and were sent to our research institute (IQ healthcare) on a monthly base [10].

Questionnaire

At the end of the trial an electronic questionnaire (see Additional file 1) was sent to all participating practices (n = 231). From each general practice we invited one GP and one PN (if applicable) to fill out the questionnaire. Aside from demographic and professional characteristics (e.g., age, gender, number of hours worked weekly), exposure to and experiences with the CDSS intervention were measured.

Exposure was measured with six statements about the level of use of the NHGDoc system as a whole; the alert function; the feedback function; the My NHGDoc function, and more specifically the alerts settings and reporting settings. Responses were rated on a 5-point Likert scale (ranging from 1. not at all to 5. very much).

Experiences with the intervention were measured in two different ways: by measuring attitudes towards CDSSs in general and by measuring perceived barriers and suggested interventions to using NHGDoc. Prior to developing the questionnaire, we conducted a qualitative focus group study to identify the range of barriers that GPs and PNs perceive in using NHGDoc or similar CDSSs in practice. Three focus group sessions were conducted in which 24 primary care practitioners (PCPs) participated (general practitioners, general practitioners in training and practice nurses), varying from 7 to 9 per session. In each focus group, barriers to using CDSSs were discussed using a semi-structured literature-based topic list. Two researchers independently performed thematic content analysis using the software program Atlas.ti 7.0.

This resulted in a framework of barriers, which is described elsewhere in detail [15]. In general, three groups of barriers emerged, related to 1) the users’ knowledge of the system, 2) the users’ evaluation of features of the system (source and content, output, and functionality), and 3) the interaction of the system with external factors (patient-related and environmental factors). Whereas the focus group study was conducted to identity all relevant barriers to using CDSSs, the survey study aimed to validate and quantify these findings.

Eight statements were included to measure the attitudes towards CDSSs in general, based on literature on attitudes towards using clinical practice guidelines [17, 18], complemented with results from the focus group study [15]. A 5-point Likert scale was used to rate the extent of agreement with the statements (ranging from 1. strongly disagree to 5. strongly agree).

To measure the perceived barriers three groups of barriers were included, based on the framework that emerged from the focus group study [15].

  • Knowledge-related barriers were measured with six statements referring to knowledge about the (specific functions of the) system (e.g. ‘I am aware of the option to ask for or to provide feedback in NHGDoc’).

  • Barriers related to the evaluation of the features of the CDSS were measured with 11 statements (e.g. ‘I believe the loading of alerts takes too long’).

  • External barriers interacting with the CDSS were measured with 6 statements (e.g. ‘Using NHGDoc has a negative effect on patient-doctor communication’).

The knowledge-related barriers were scored on a 2-point scale (yes/no). All other statements were rated a 5-point scale (ranging from 1. strongly disagree to 5. strongly agree), with the option ‘not applicable’ (6.) added to the response scale. Suggested interventions to improve CDSS usage were also discussed and will be described elsewhere.

Finally, the intended ‘blinding’ in terms of the topic of the intervention was checked among the participants by formulating the following open question ‘Do you have any idea on which clinical topic this study focused on?’.

Data analysis

Data from the NHGDoc server measuring exposure to NHGDoc were analyzed using descriptive statistics. To analyze exposure as measured by the questionnaire we categorized the scores 1 and 2 to indicate a low level of use, we coded 3 as a moderate level of use, and combined the scores 4 and 5 to reflect an intermediate to high level of use.

To analyze the attitudes towards CDSSs in general we categorized the scores 4 and 5 (agree/strongly agree) indicating agreement; the scores 3 to refer to a neutral attitude, and the scores 1 and 2 (strongly disagree/disagree) to indicate disagreement.

The knowledge-related barriers were calculated by the percentage of respondents that indicated not to be aware of NHGDoc or its specific functions (score 1). For the second and third group of barriers (attitude-related and external barriers) the barrier statements that were stated positively rather than negatively, were first recoded, so that a higher score indicated a higher level of perceived barriers. Next, the scores 4 and 5 were combined to indicate the percentage users that indicate a barrier to be applicable.

Results

Description of study sample

The 231 participating practices were an acceptable reflection of the population of Dutch GP practices, with solo practices only being slightly overrepresented [19].

Description of sample NHGDoc server

Almost all recruited practices (229/231 = 99 %) generated data into the NHGDoc server database; two were eliminated from the trial due to technical problems. Because of this high rate, the background characteristics are comparable to the study population.

Description of sample of questionnaire

Of the 231 practices 50 % responded to the survey (see Table 1). The responding practices were equally distributed between the intervention group and control group. Most practices concerned solo practices (57 %) and used the EHRS MicroHIS X (56 %), which is comparable to our study population (resp. 56 % and 58 %).

Table 1 Characteristics of the responding GPs and PNs and their practices compared to the total study population
Full size table

From the group of responding GPs (49 %) relatively more men (69 %) were included compared to our total study population (69 vs. 53 %). The majority of the participating GPs were aged between 55 and 64 years (46 %) (no comparison possible). Ninety-eight percent of the responding PNs (37 %) were women (98 %), a little more compared to our study population (93 %). Most of them were aged between 45 and 54 years of age (no comparison possible).

Exposure to the CDSS intervention

NHGDoc server

Figure 1 indicates the level of NHGDoc activity in participating practices illustrated by the number of requests sent per week. As can be seen, there are three large negative spikes during the trial year.

Fig. 1
figure 1

NHGDoc activity of the participating practices during the year of the trial. Q quarter

Full size image

The NHGDoc alerts concerning heart failure patients were opened in only 0.24 % of the cases. This was 0.35 % for the total patient population. No trend towards increased use was detected during the trial period.

Data from the NHGDoc server also revealed that less than 2 % of the participating primary care providers used the personalization functions of NHGDoc to switch off one or more NHGDoc domains. With respect to the specific Heart failure domain this percentage was less than 1 %.

Questionnaire

Table 2 shows the reported use of the system NHGDoc and its specific functions, as identified by the questionnaire. As can be seen contrasting subgroups can be identified: approximately half of the GPs (52 %) and 42 % of the PNs reported to either never or rarely use the system whereas almost 10 % of the GPs and 20 % of the PNs use it very often or all the time.

Table 2 Reported use of NHGDoc and its specific functions
Full size table

From the basic functions half of the GPs (51 %) and 40 % of the PNs reported never or rarely to use the alert functions. Thirty-eight percent of both groups indicated to use the alert functions sometimes. The feedback function demonstrated a lower level of use with 88 % of the GPs and 86 % of the GPs never or rarely using it. Less than 2 % of both groups indicated to use the feedback option often or all the time.

With respect to the personalization functions of My NHGDoc, the majority of both GPs (72 %) and PNs (65 %) reported to never or rarely use this function. Relatively more PNs as compared to GPs used it often or all the time (18 vs. 5 %). Eighty-two percent of the target group never or rarely used the alert settings; for the reporting settings this figure was 85 %.

Experiences with the NHGDoc intervention

Attitude towards CDSSs in general

As can be seen in Fig. 2, 80 % of the GPs and 67 % of the PNs agreed to the statement that CDSSs are useful sources of advice. In addition, 85 % of the GPs and 73 % of the PNs indicated that they considered CDSSs as useful tools to improve the uptake of guidelines and about three-third of both groups (78 and 71 %) believed that using CDSSs results in improved patient care. Eighty-nine percent of the GPs and 79 % of the PNs believed that CDSSs are relevant for different user groups.

Fig. 2
figure 2

GPs’ (n = 112) and PNs’ (n = 52) ratings on statements measuring the attitude towards CDSSs in general

Full size image

Sixty percent of the GPs and 25 % of the PNs indicated that the use of CDSSs is difficult to integrate in daily practice. About a third of both groups (36 % of GPs and 31 % of PNs) was afraid that using CDSSs leads to ‘cookbook medicine’. One fifth of both groups (21 and 19 %) reported to lack knowledge and skills to properly use CDSSs. Thirteen percent of the GPs and 6 % of the PNs indicated to have overall resistance to working with CDSSs.

Experiences with NHGDoc: perceived barriers

Table 3 presents the three groups of barriers as reported by the respondents of the questionnaire.

Table 3 Number and percentage of GPs and PNs that perceive barriers to using NHGDoc
Full size table

Knowledge-related barriers

As can be seen in Table 3 2 % of the GPs and 18 % of the PNs were not aware of the existence of NHGDoc at all. Of those who did, almost all respondents were aware of the fact that the CDSS sends alerts to users. Fourty-one percent of the GPs and 55 % of the PNs, however, were not aware of the existence of the feedback option. Particularly, a lack of knowledge regarding the option to use the MyNHGDoc personalization functions was found: 31 % of the GPs and 40 % of the PNs were not aware of this function. This particularly applied to the reporting setting options (51 % GPs and 70 % of PNs), but also to the alert setting options (46 % GPs and 65 % of PNs).

Barriers related to the evaluation of the features of NHGDoc

From the group of barriers related to the evaluation of the features of the CDSS, particularly the subgroup functionality was considered relevant. In both groups a lack of learning capacity of the system (80 and 75 %), a too high intensity of the alerts (40 and 32 %) and a lack of responsiveness of the system (33 and 38 %) were often perceived as barriers. From the subgroup source and content of the CDSS the relevance of the alert content for individual users was often perceived as a barrier in both groups (58 and 62 %).

External barriers interacting with the CDSS

From the group of external barriers interacting with the CDSS, particularly environmental factors were often perceived as barriers. Sixty percent of the GPs indicated the limited time available to be a barrier, whereas only 16 % of the PNs considered this to be a barrier. Also, 60 % of GPs reported that using the CDSS required too much extra work, as compared to 27 % of the PNs. From the patient-related factors, particularly a negative effect on patient communication was considered as a barrier by 26 % of the GPs and 22 % of the PNs.

Blinding

Finally, the blinding check showed that none of the respondents was aware of the fact that heart failure was the focus of the intervention.

Discussion

This process evaluation revealed that the target group was not exposed to the intervention as planned as the use of NHGDoc and its specific heart failure module was limited. Although (potential) users had a positive attitude towards CDSSs in general, several barriers to using NHGDoc could have hampered the uptake and effectiveness of the intervention. The large share of users that was insufficiently aware of the functions of NHGDoc, as well as the proportion that finds the decision support not useful or not relevant and difficult to integrate into daily practice, indicates that both the CDSS intervention itself as well as its implementation could be improved. These findings are in line with the lack of changes in outcomes of the CDSS intervention.

The actual exposure to NHGDoc and the heart failure module was low, with about half of the (potential) users reporting to either never or rarely use the system and less than 1 % opening alerts. Despite a few negative spikes in NHGDoc activity, which are presumably related to vacation periods of the staff, the activity was quite stable throughout the trial and technical problems did not seem to be a major issue. Also, the low level of exposure to NHGDoc was not related to users deliberately switching of one or more modules of NHGDoc. In interpreting the number of alerts in our study, it is important to realize that the different modules do not result in different alerts each. Rather, the advices are combined in one alert, consisting of an alert window with different tab pages for all relevant domains. Moreover, prior studies have also found high levels of ignoring of alerts among their users [20–22]. Nonetheless, our results suggest that other factors may have prevented our study sample from using NHGDoc as well.

Although the target group of (potential) users had a positive attitude towards CDSSs in general, several barriers were reported that could have affected the uptake and effectiveness of the intervention. The most perceived barriers to using NHGDoc were a lack of learning capacity of the system, the limited time available and the additional work it requires using the CDSS, irrelevant alerts, too high intensity of alerts and insufficient knowledge regarding the system. These findings are consistent with our focus group study and confirm that barriers to using NHGDoc were found within all three main groups of barriers in our previously developed framework of barriers to using CDSSs [15].

With respect to the knowledge-related barriers we found that the target group was insufficiently aware of NHGDoc and its functions, particularly the advanced personalization functions. At the onset of the trial the NHGDoc module on heart failure was activated in the intervention practices, without any further instructions. We deliberately chose for this method, as we aimed at both practitioners and patients being blinded to group allocation, which is important in protecting against bias [23, 24]. Moreover, since all participating practices already had NHGDoc at their disposal, we presumed a certain level of basic knowledge about NHGDoc. However, our findings show a serious lack of knowledge about the system and demonstrate the importance of a proper introduction of a CDSS intervention among its target group. This is particularly challenging when implementing CDSSs at a larger scale, outside the academic setting [15].

Aside from improving the implementation of CDSS interventions among the target group, our results indicate that in our study the CDSS intervention itself also exhibits some shortcomings, often referred to as the ‘intervention failure’ [25, 26]. With regard to the features of the system itself, a perceived lack of learning capacity of the system, irrelevant alerts and a too high intensity of (or a too low of a threshold for) alerts were most often perceived as barriers. It is precisely these types of barriers that often result in ‘alert fatigue’, and the ignoring of alerts [22, 27]. They may be less relevant for the more common CDSSs aimed at a limited number of decision points and may particularly apply in the dynamic and complex primary care setting in which different types of PCPs work with CDSSs covering multiple disease areas [15]. Increasing the system’s flexibility and learning capacity in order to be able to adapt the decision support to meet the varying needs of different users may then become increasingly important.

External barriers interacting with the CDSS were also experienced as barriers which has been found in other studies as well [27–32]. As much as 60 % of the GPs from our sample reported the limited time available and the additional work it requires to use the CDSS as barriers to using it. Obviously, many GPs find it difficult to integrate the use of CDSSs into daily practice. However, we found that these barriers only apply to 16–27 % of the PNs. This is probably related to the nature of their work activities, which contains a larger share of patient data registration, as well as to the way their patient consultation is scheduled. To reduce integration problems it could be helpful to rearrange GPs’ patient consultation as well, with more preparation time before actual patient consultation [15]. Adequate training about how to properly use the system as well as providing the target group with decision support that is tailored to the specific and varying needs of each user, could also contribute in solving these integration problems.

One of the limitations of this study is the seemingly late conduction of this barrier study. However, it should be noted that the NHGDoc system already existed for 5 years at the start of the trial and several small user-satisfaction studies had been conducted among users. Moreover, implementing (CDSS) interventions at a larger scale outside the academic setting, seems to elicit different types of barriers, that cannot be identified in a preceding barrier study among early adopters [15, 33]. Implementing and further developing the system, while conducting prospective formative process evaluations at several stages of implementation, seems therefore necessary. It allows interventions to be disseminated and adopted more successfully at each subsequent later stage.

A major strength of this study is that we collected data using several methods and targeting different types of PCPs. Moreover, our study demonstrates high content validity as the questionnaire used in this study is based on qualitative study findings from the same group [15]. However, whereas the samples of the two approaches used (questionnaire, log-data from NHGDoc-server) were quite comparable in terms of background characteristics, selection bias may have occurred with respect to use and attitudes in the self-selected questionnaire. It may well be that responding PCPs used NHGDoc more often and had more positive attitudes towards NHGDoc compared to non-responders. The difference between reported use of NHGDoc in the questionnaire and data from the NHGDoc server points in that direction as well. This should be taken into account in interpreting our findings.

Few thorough process evaluations have been conducted in the field of CDSSs, particularly regarding multiple-domain covering CDSSs implemented at a larger scale. This study used multiple methods, targeting different types of PCPs, and with data collected within the intervention group as well as the control group. Future studies on CDSSs may benefit from thorough evaluations like this one to determine the factors that facilitate or hinder the implementation. Whereas some of the suggested strategies, such as increasing user flexibility, may particularly be helpful in implementing multiple-domain covering CDSSs at a large scale, other strategies, such as involving the end-users in designing the CDSS, also apply to the more common CDSS aimed at a limited number of decision points.

Conclusions

This process evaluation has identified some factors that might be responsible for the lack of changes in outcomes of the CDSS intervention, as well as useful strategies to improve its implementation. Results indicate that both the CDSS intervention itself as well as its implementation could be improved. Although users are generally positive about CDSSs, a large share of them is insufficiently aware of the functions of NHGDoc and finds the decision support not always useful or relevant and difficult to integrate into daily practice. In designing (similar) CDSS interventions we suggest to more intensely involve the end-users and increase the system’s flexibility and learning capacity. To improve implementation a proper introduction of a CDSS among its target group including adequate training is advocated.