Efficiency of Clinical Decision Support Systems Improves with Experience
- 1.3k Downloads
Efficiency, or the resources spent while performing a specific task, is widely regarded as one the determinants of usability. In this study, the authors hypothesize that having a group of users perform a similar task over a prolonged period of time will lead to improvements in efficiency of that task. This study was performed in the domain of decision-supported medication reviews. Data was gathered during a randomized controlled trial. Three expert teams consisting of an independent physician and an independent pharmacist conducted 150 computerized medication reviews on patients in 13 general practices located in Amsterdam, the Netherlands. Results were analyzed with a linear mixed model. A fixed effects test on the linear mixed model showed a significant difference in the time required to conduct medication reviews over time; F(31.145) = 14.043, p < .001. The average time in minutes required to conduct medication reviews up to the first quartile was M = 20.42 (SD = 9.00), while the time from the third quartile up was M = 9.81 (SD = 6.13). This leads the authors to conclude that the amount of time users needed to perform similar tasks decreased significantly as they gained experience over time.
KeywordsClinical decision support Medication review Efficiency Usability
Efficiency, or the resources spent while performing a specific task, is widely regarded as one the determinants of usability . The experience users have with software has been shown to influence their attitudes towards prolonged use of it [2, 3]. However, only limited longitudinal research measuring the effects of prolonged software use on efficiency has been conducted. Some studies, in this regard, have indicated that determinants common in conventional usability research differ for experienced users [4, 5].
This lack of longitudinal research on efficiency needs to be addressed. In this study, this research interest is tested in the domain of decision-supported medication reviews. The authors hypothesize that having a group of users perform a similar task over a prolonged period of time will lead to improvements in the efficiency of that task.
Efficiency in usability
Efficiency is being regarded as one of the main aspects determining software applications’ usability; it has been included in all major definitions of the concept [1, 6]. Users gain experience through prolonged exposure to a system. This experience has been shown to be a major determinant on their attitudes towards accepting technology [2, 3, 7, 8].
The commonly used definition proposed by the International Standards Organization (ISO) defines usability as ‘the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use’ . In this context, efficiency exists of ‘resources expended in relation to the accuracy and completeness with which users achieve goals’. Theoretically the notion of efficiency can encompass any kind of resource, including money or knowledge, but in practice the concept is usually limited to indicate the amount of time spent on a certain task.
Most usability studies use experimental methods to determine systems’ effectiveness, efficiency, and satisfaction. The effect of temporality, or the degree to which these factors change over time, is largely overlooked . At the same time, however, aspects dependent on temporality are accepted as major determinants: a common definition proposed by Nielsen emphasizes the importance of learnability and memorability of user interfaces . Learnability encompasses the ease with which users can accomplish basic tasks when they first encounter a design, while memorability concerns the degree to which they can reestablish proficiency when reusing it. Thus, testing of usability through experiments often entails nothing but testing of learnability, measuring factors of novice rather than experienced users . In experimental studies, in which participants are unfamiliar with the method or the user interface, a drop in efficiency can reasonably be expected .
Efficiency in clinical decision support systems
In literature, there is consensus that usability has a significant impact on users’ adoption of clinical decision support systems . However, many clinical decision support systems have not been shown to improve efficiency. While systems’ improvements in effectiveness are well-documented, studies on efficiency are lacking. A recent systematic literature review found that “contamination of clinicians in the control group, […] evaluation periods that were too brief to demonstrate an effect on efficiency, and small clinician sample sizes” made it impossible to generalize on efficiency consequences .
A study synthesizing features of proven effective clinical decision support systems found mixed results regarding efficiency: while embedding systems in physicians’ workflows was associated with improved effectiveness, advice presented within computerized physician order entry (CPOE) systems had negative correlations regarding success . The authors explained this apparent anomaly by subscribing to the oft-mentioned ‘alert fatigue’ found in clinical decision support systems. In many systems, users are presented with a multitude of warnings and suggestions, forcing them to ignore the majority [13, 14].
Following the above-mentioned considerations, the authors hypothesize that users grow more proficient performing similar tasks over time. The effects of experience on applying the application effectively and memorizing its functionality are assumed to lead to gradual improvements in efficiency.
This objective is investigated in the domain of clinical decision support systems, which is explained in more detail in the adjacent section. The research question reads: Does the time physicians and pharmacists use to systematically optimize the medical records of polypharmacy patients decrease as they gain experience?
A secondary interest is how users allocate their time during the task. This information can be used to improve decision support systems’ workflow and user interface.
Domain: Decision-supported medication reviews
The chronic use of multiple medications, while often indicated in older patients, has been shown to lead to adverse drug reactions. A comprehensive Dutch study showed that 5.6% of Dutch hospital admissions are due to medication-related problems . Appropriate prescribing in older people is challenging, because of age-related changes in pharmacokinetics and pharmacodynamics, multimorbidity and polypharmacy [16, 17].
Clinical medication reviews are oft-mentioned approaches to improve this situation. Clinical medication reviews are ‘structured, critical examination[s] of the patient’s medicines with the objective of reaching an agreement with the patient about treatment, optimizing the impact of medicines, and minimizing the number of drug related problems’.
Several clinical medication review methods have been developed to aid physicians and pharmacists in optimizing their prescriptions for these polypharmacy patients [18, 19]. Implementations of these methods as decision support systems have been proven to improve the quality of prescriptions in research settings [9, 19].
A major disadvantage of these methods, however, is that physicians need more time to perform them than they do to perform their usual care methods. In a 2009 study in which the Polypharmacy Optimization Method (POM) was tested, Drenth-van Maanen et al. found that performing a medication review with the structured method took more time (16.7 min) than performing one without (8 min) . A more recent study, in which a decision support system facilitating the Systematic Tool to Reduce Inappropriate Prescribing (STRIP) was validated, Meulendijk et al. found that the average time physicians needed to optimize a patient’s health record with the STRIP method was 24 min, while the time needed without any structured approach was 13 min .
In this domain, the efficiency of any structured method is of major importance. So far, research has shown that effectiveness in this domain can improve with structured methods, but longitudinal research reporting on their efficiency is lacking.
The data was collected within the intervention arm of a randomized controlled trial . This trial aimed at testing whether or not the use of clinical medication reviews can reduce inappropriate drug use of older people. The randomized controlled trial included 500 patients from 25 practices in Amsterdam, the Netherlands. They were eligible for inclusion if they were 65 years or older, suffered from pre-specified geriatric symptoms in general practice, and chronically used at least one prescribed drug. The primary outcomes were quality of life and presence of geriatric symptoms, with secondary outcomes focusing on costs, feasibility, number of drug-related problems, adherence, and satisfaction. The results have not been published yet. A more detailed description of the trial’s design and intended outcomes can be found in Willeboordse et al. .
Patients of 13 intervention practices received clinical medication reviews, while those in the other practices served as a control group. The data for this study was collected from the clinical decision support system that was used to perform the intervention.
Operationalization of the concept of efficiency in the domain of structured medication reviews
Time (in seconds)
in relation to the accuracy
Having responded to all pieces of generated advice
Having finished all steps of the STRIP medication review method
with which users
Teams of one independent physician and one independent pharmacist
For the randomized controlled trial, five expert teams, each consisting of an independent physician and an independent pharmacist, were recruited through a convenience sample. As one team had a variable composition, and another team performed only five medication reviews, three teams’ data were used. Of these three teams, all participants were female. Two of the physicians were general practitioners; one was a nursing home physician.
The participating teams were tasked with conducting structured medication reviews for fifty patients included in the intervention arm of the trial. This assessment was carried out according to the STRIP, which is the preferred method according to the Dutch multidisciplinary guideline on polypharmacy in elderly patients . In the trial, it was facilitated by the clinical decision support system described in the next section. The medication reviews allowed the teams to identify underprescribing, overtreatment, clinical interactions, contra-indications, and inappropriate dosages in patients’ health records. They were then able to adjust the health records by prescribing new medications, removing superfluous ones, or altering dosages.
The clinical decision support system that was used to answer the research question is the STRIP Assistant (STRIPA), a web application that facilitates the structured pharmacotherapeutical analysis of the STRIP method . STRIPA displays a single patient’s health record at a time, showing his or her diseases, medications, symptoms, and relevant measurements or observations. Users are guided through the analysis steps to optimize a patient’s medication; first, they manually assign drugs to the diseases for which they have been prescribed; second, they add any missing medication for which there is an indication; third, they eliminate superfluous ineffective drugs or drugs for which there is no appropriate indication; fourth, they check for any relevant clinical interactions between drugs; finally, they readjust dosages if necessary. In all these steps, except the first one, STRIPA generates advice based on (inter)national guidelines, most notably the START/STOPP criteria and guidelines for medication safety recommended by the Royal Dutch Pharmacists Association [23, 24]. The pieces of advice arrived at through these guidelines are used to alert users of anomalies and suggest appropriate actions in a non-intrusive fashion. A more detailed description of the software’s architecture, design, and functioning is described in Meulendijk et al. . A video demonstrating the use of the STRIP Assistant can be viewed here: http://videodemo.stripa.eu/english/ .
A sample log showing the addition of a medication (lactulose) to a disease (obstipation) as part of an underprescribing intervention
User interface phase:
November 1st, 2013, 15:00:01
1. Medication: lactulose 667 mg/ml, apply once daily.
2. Disease: chronic obstipation
Clickstream analysis was used to gain insight into the recorded logs of users’ actions. A clickstream is an “electronic record of a user’s activity on the Internet” and is an often used method for gaining insight into people’s behavior with web applications [26, 27].
For the statistical analysis, a linear mixed model with fixed effects was adopted. In this model, teams served as subjects, with repeated measurements being the medication reviews they performed. To account for changes in difficulty of health records, the numbers of patients’ diseases and medications were incorporated into the model as covariates. Additionally, to account for the differences in the number of advices, the number of recommendations generated by the system per review was incorporated as a covariate.
The expert teams conducted a total of 150 analyses for patients in the intervention group. All analyses were performed in the period of November 2013 to November 2014. On average, it took them 14.6 min (SD = 7.98) to complete a medication review, ranging from 2.1 to 42.7 min.
On average, the cases consisted of 13.4 (SD = 5.48) diagnosed diseases to which the users assigned 8.0 (SD = 4.66) medications. These medications comprised both ones patients already used, and new ones that the users prescribed.
Summary of users’ responses to generated recommendations
Average number of followed advices
8.80 % (M = 0.48, SD = 1.05)
13.69 % (M = 0.92, SD = 0.90)
25.00 % (M = 1.64, SD = 1.54)
16.16 % (M = 1.01, SD = 1.28)
Average number of rejected advices
47.62 % (M = 2.60, SD = 2.12)
28.27 % (M = 1.90, SD = 1.30)
20.43 % (M = 1.34, SD = 1.56)
31.20 % (M = 1.95, SD = 1.76)
Average number of ignored advices
43.58 % (M = 2.38, SD = 2.84)
59,04 % (M = 3.90, SD = 2.97)
54,57 % (M = 3.58, SD = 3.65)
52.64 % (M = 3.29, SD = 3.22)
Summary of users’ time allocation per step
Average time spent on medication assignment
44.47 % (M = 4.55, SD = 3.56)
53.00 % (M = 7.25, SD = 5.75)
40.74 % (M = 6.05, SD = 4.56)
46.16 % (M = 5.95, SD = 4.81)
Average time spent on underprescribing
43.79 % (M = 4.48, SD = 3.56)
36.04 % (M = 4.93, SD = 3.48)
44.44 % (M = 6.60, SD = 5.23)
41.43 % (M = 5.34, SD = 4.24)
Average time spent on overtreatment
5.47 % (M = 0.56, SD = 0.91)
4.24 % (M = 0.58, SD = 0.81)
5.72 % (M = 0.85, SD = 1.56)
5.20 % (M = 0.67, SD = 1.14)
Average time spent on clinical interactions
4.30 % (M = 0.44, SD = 0.93)
5.53 % (M = 0.77, SD = 1.32)
5.86 % (M = 0.87, SD = 1.70)
5.43 % (M = 0.70, SD = 1.35)
Average time spent on dosage adjustment
1.95 % (M = 0.20, SD = 0.46)
1.10 % (M = 0.15, SD = 0.22)
2.23 % (M = 0.48, SD = 2.07)
1.78 % (M = 0.23, SD = 1.23)
In this paper, the results of a longitudinal study on the effects of experience on physicians’ and pharmacists’ efficiency of performing decision-supported medication reviews were reported. In line with the authors’ hypothesis, the amount of time users needed to perform similar tasks decreased significantly as they gained experience.
Earlier validations of the STRIP method showed significant increases in effectiveness over usual care approaches, but decreases in efficiency [9, 19]. In the latest validation of the method embedded in a decision support system, the authors hypothesized that “experiments in which more gradual changes to the method are applied may result in improvements in both effectiveness and efficiency” . This assumption was in line with the view proposed by Nilsson & Følstad of effectiveness and efficiency as conflicting requirements . The results of this study confirm the hypothesis that users’ familiarity with an application and their experience with performing similar tasks leads to increases in efficiency.
Limited research has been done into the effects of temporality on usability in general, and efficiency in particular. Mendoza and Novick reported, though, that frustration levels of participants using unfamiliar software decreased over time, leading them to remark that “factors such as features being hard to find and operators committing slips and mistakes really are the principal causes of severe frustration” . They report that with increased familiarity these frustrations decrease. It can be reasonably expected that, in a similar fashion, time spent on repetitive tasks decreases.
In fields related to usability, temporality has been recognized as being influential in shaping people’s motivations towards using software. In a user experience study, temporality appeared to be a determinant in the changing motivations of people who use software for a prolonged period of time: “prolonged experiences became increasingly more tied to aspects reflecting how the product becomes meaningful in one’s life” . Experience has been shown to be a determinant in users’ attitudes towards acceptance of technology [2, 3, 7].
Practical implications in STRIPA
The clickstream analysis pinpointed which steps of the STRIP process were least efficient and could be improved in the software application. In an attempt to decrease the workload of the first, medication assignment, step, knowledge discovery methods were employed. Based on historical data, the application was improved to automatically assign medications to appropriate diseases. Additionally, medication menus were redesigned to allow for quicker selection of probable choices. Clinical analysis of the recommendations that were most often rejected or ignored by users led to improvements in the clinical rule base which powers the decision support system.
This paper reports the results of a longitudinal study on the effects of experience on physicians’ and pharmacists’ efficiency of performing decision-supported medication reviews. Corresponding with the authors’ hypothesis, the amount of time users needed to perform similar tasks decreased significantly as they gained experience.
Limitations of this study include the limited number of participants and the fact that the analyses were performed unsupervised. The participants were recruited as a convenience sample and are not representative of the population of clinical decision support systems users.
The effectiveness of the medication reviews that the users performed was not measured. Further research should reveal any changes in effectiveness over time, and if this is in any way correlated with the measured improvement in efficiency.
Compliance with ethical standards
Conflict of interest
The authors declare that they have no conflict of interest.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee (Medical Ethics Committee of the VU University Medical Centre Amsterdam, reference 2011/408) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.
Informed consent was obtained from all individual participants included in the study.
- 1.International Organization for Standardization (ISO). ISO/IEC 25010:2011: Systems and software engineering -- Systems and software Quality Requirements and Evaluation (SQuaRE) -- System and software quality models. Standard.; 2011.Google Scholar
- 3.Venkatesh, V., Morris, M. G., Davis, G. B., and Davis, F. D., User acceptance of information technology: Toward a unified view. MIS Q. 27(3):425–478, 2003.Google Scholar
- 4.Mendoza, V., Novick, D.G., Usability over time. In: Proceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive information. p. 151–158. 2005.Google Scholar
- 5.Karapanos, E., Zimmerman, J., Forlizzi, J., Martens, J.B., User experience over time. In: Modeling Users’ Experiences with Interactive Systems. p. 57–83. Berlin Heidelberg; Springer, 2013.Google Scholar
- 6.Nielsen, J., Usability engineering. Elsevier, 1994.Google Scholar
- 8.Meulendijk, M. C., Spruit, M. R., Drenth-van Maanen, A. C., Numans, M., Brinkkemper, S., and Jansen, P. A. F., General practitioners’ attitudes towards decision-supported prescribing: an analysis of the Dutch primary care sector. Health Informatics J. 19(4):247–263, 2013.CrossRefPubMedGoogle Scholar
- 9.Meulendijk, M.C., Spruit, M.R., Drenth-van, Maanen, A.C., Numans, M.E., Brinkkemper, S., Jansen, P.A.F., et al. Computerized decision support improves m edication review effectiveness: An experiment evaluating the STRIP Assistant’s usability. Drugs Aging. 2015.Google Scholar
- 10.Croll, J., The Impact of Usability on Clinician Acceptance of a Health Information System. PhD Thesis. Queensland University of Technology, Centre for Learning Innovation, Faculty of Education. 2009.Google Scholar
- 12.Roshanov, P.S., Fernandes, N., Wilczynski, J.M., Hemens, B.J., You, J.J., Handler, S.M., et al. Features of effective computerised clinical decision support systems: Meta-regression of 162 randomised trials. Br. Med. J. 346, 2013Google Scholar
- 14.Van der Sijs, H., Drug safety alerting in computerized physician order entry: unraveling and counteracting alert fatigue. PhD Thesis. Erasmus MC: University Medical Center Rotterdam. 2009.Google Scholar
- 20.Willeboordse, F., Hugtenburg, J.G., van Dijk, L., Bosmans, J.E., de Vries, O.J., Schellevis, F.G., et al. Opti-Med: the effectiveness of optimised clinical medication reviews in older people with ‘geriatric giants’ in general practice; study protocol of a cluster randomised controlled trial. BMC Geriatr. 14(1), 2014.Google Scholar
- 21.Dutch College of General Practitioners (‘Nederlands Huisartsengenootschap’ in Dutch). Multidisciplinary guideline Polypharmacy in elderly (‘Multidisciplinaire richtlijn Polyfarmacie bij ouderen’ in Dutch). Guideline. Utrecht, 2012.Google Scholar
- 22.Meulendijk, M.C., Spruit, M.R., Jansen, P.A.F., Numans, M.E., Brinkkemper, S., STRIPA: A Rule-Based Decision Support System for Medication Reviews in Primary Care. In: ECIS 2015. 2015.Google Scholar
- 23.O’Mahony, D., O’Sullivan, D., Byrne, S., O’Connor, M.N., Ryan, C., Gallagher, P., STOPP/START criteria for potentially inappropriate prescribing in older people: version 2. Age Ageing. 2014.Google Scholar
- 28.Nilsson, E.G., Følstad, A., Effectiveness and efficiency as conflicting requirements in designing emergency mission reporting. I-UxSED. p. 20–25. 2012Google Scholar
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.