Abstract
Measurement feedback systems (MFS) can help guide treatment and improve clinical outcomes. Studies of MFS are heterogeneous both in execution and results, and the effects of MFS seem restricted by limited attention to process and implementation elements and by limited adoption by health professionals. The current systematic review mapped the use of process and implementation elements in MFS studies. An overview of therapists’ use of and attitudes toward MFS is provided. Three-level meta-analyses were used to test theoretically informed process and implementation elements as moderators of the effects of MFS. Hypotheses and general propositions from Clinical Performance Feedback Intervention Theory (CP-FIT) were used to organize the elements of the studies and were used as moderator variables. Previous studies on MFS interventions have had a limited focus on implementation efforts and process elements that may increase the effects of MFS and their use among therapists. Efforts have sparsely been made to reduce barriers to MFS use, and several studies have reported limited engagement with MFS among therapists. Therapists’ attitudes toward MFS, feedback, or standardized measures were heterogeneously reported, making data synthesis challenging. Identified process and implementation elements were not significantly associated with effect sizes in the studies and the results did not support the propositions of CP-FIT. The lack of statistically significant associations may be due to limited reporting of details about process and implementation aspects. More research designed to test hypotheses regarding process and implementation elements is needed to improve the use and effects of MFS. Future studies should aspire to report findings in a manner that allows for an understanding of the implementation process and therapists’ adoption of these systems.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Patients often leave therapy for mental health disorders without experiencing symptom relief or better functioning (Warren et al., 2010; Wolpert, 2016). Concomitantly, the magnitude of the effect of therapy has not increased in clinical trials over the last decades (Weisz et al., 2017), illustrating the need for new approaches to enhance treatment effects. One such approach is the use of measurement feedback systems (MFS) to guide therapy by informing therapists and/or others involved about changes for patients. In several studies, MFS have had positive effects on clinical outcomes (e.g., Amble et al., 2014; Brattland et al., 2018), and recent meta-analyses have presented small positive overall effect estimates (de Jong et al., 2021; Rognstad et al., 2022). Still, the effect of MFS seems hampered by a lack of attention to the process and implementation elements necessary for its success (Bickman et al., 2016; Kendrick et al., 2016).
Feedback-informed therapy involves the continuous and systematic collection of information about the patient. The use of standardized measures throughout the course of treatment is meant to improve care by ensuring better informed patients, health care providers, and/or other stakeholders. Progress, or lack thereof, can be tracked throughout treatment, which helps clinicians evaluate, adjust, and tailor the treatment course to the individual patient. Still, the implementation of MFS can be complicated.
Several systematic reviews point out that there is substantial heterogeneity in the results from MFS studies (Gondek et al., 2016; Ishaque et al., 2019; Rognstad et al., 2022). This is likely due in part to implementation issues and therapists’ reception of MFS interventions (Kendrick et al., 2016). In a narrative review, Lewis et al. (2019) present a 10-point research agenda for measurement-based care (MBC), which is a form of care that relies on MFS to inform decision making. They point to the need for the development of a criterion standard method for monitoring fidelity to MFS and understanding elements that facilitate fidelity. Identifying important process and implementation elements that facilitate fidelity to MFS can inform the development of fidelity standards and assessments. Degree of implementation has been shown to moderate the effectiveness of MFS, and a dose–response effect has been found showing a larger treatment effect in individual cases where the MFS was more frequently used (Bickman et al., 2016). Brattland et al. (2018) reported increased effects of MFS over time and proposed that this might be due to continued implementation efforts such as regular MFS training and supervision throughout the trial period of about 4 years. Also, based on a systematic review of qualitative studies, Brown et al. (2019) developed a comprehensive theoretical framework to describe how clinical performance feedback systems such as MFS work as interventions. This Clinical Performance Feedback Intervention Theory (CP-FIT) encompasses processes and mechanisms related to implementing and using MFS theorized as determinants of success. CP-FIT provides a framework for a theory-informed review of elements that are important for the delivery and implementation of MFS.
Clinical Performance Feedback Intervention Theory
CP-FIT was developed to describe causal pathways in audit and feedback processes. It proposes that feedback is a cyclical process comprising goal setting; data collection and analysis; feedback; recipients’ interaction with feedback; and perception and acceptance of the feedback resulting in intention, behavior, and clinical performance improvement (Brown et al., 2019). CP-FIT further states that feedback will be less effective if any of these individual processes fail.
CP-FIT identifies variables related to the feedback, recipients, and context that influence how successful feedback interventions are. The theory suggests that feedback needs to be clinically meaningful to the recipients, automatically collected and analyzed, actively delivered, easily comprehended, and target goals within the recipient’s control. Also, feedback recipients should feel ownership and control over the interventions, as well as have a positive view of feedback. Further, CP-FIT suggests that feedback interventions should be implemented with leadership and opportunities to discuss feedback with peers and that they should not be time and energy-consuming.
Three general propositions are put forth by CP-FIT to summarize 42 more specific hypotheses. The three general propositions are: (1) Capacity limitations—less taxing feedback interventions will work better due to therapists’ and organizations’ finite capacity; (2) Identity and culture—feedback interventions better aligned with therapists’ and organizations’ beliefs about patient care will be more successful; and (3) Behavioral induction—feedback interventions that directly support clinical behavior will be more effective.
Practice, Process, and Implementation Elements
Different approaches to identifying common practice elements have been adopted to identify potentially active ingredients and mechanisms of change in treatment programs (Chorpita et al., 2005; Helland et al., 2022; Kjøbli et al., 2023; Kvamme et al., 2022; Leijten et al., 2019). A “practice element” is any “discrete clinical technique or strategy used as part of a larger intervention plan” (Chorpita et al., 2005). Identifying practice elements can help with “unboxing” interventions and implementations and, through different methods, with finding the elements that cause change or are highly influential (Engell et al., 2023).
In addition to identifying effective practice elements in MFS, the circumstances needed for MFS interventions to be effectively used should be mapped. Whereas a practice element is what people do, a process element is how people do things or how things unfold or emerge (Engell et al., 2023). Implementation elements, however, are what makes people do what things, and implementation elements can include discrete implementation strategies (Powell et al., 2015), implementation determinants (Nilsen & Bernhardson, 2019), implementation competencies (Metz et al., 2021), or any other relevant part of the implementation processes (Engell et al., 2023). While process elements tend to describe delivery forms and contexts, implementation elements tend to describe facilitating actions such as ongoing training or technical support, and motivations and states such as attitudes towards the intervention or readiness for implementation. Implementation elements can also be the implementation competencies such as abilities to interact with service providers, communicate research findings, improve research-practice partnerships (Metz et al., 2021) or competence in implementation facilitation, context assessment or knowledge of implementation theories, models, and frameworks (Bührmann et al., 2022). Such aspects of interventions and implementations may be necessary or catalyzing conditions for effects to occur or emerge (Engell et al., 2023). This study will focus on process and implementation elements.
Process and Implementation of MFS
Therapists are often reluctant to engage with feedback data due to both philosophical reasons (e.g., belief that assessment disrupts the flow of treatment or that the measures do not address relevant constructs) and practical implementation barriers (Cooper et al., 2021). Receiving negative feedback can be intimidating and may cause therapists to lose optimism (Lambert et al., 2019). In a systematic review, the most common barriers to therapists’ use of feedback were the perceptions that the relevance of feedback is limited, data collection and synthesis are time-consuming, and MFS is intrusive in the context of clinical practice (Gelkopf et al., 2022).
MFS have been implemented in a variety of contexts and frameworks. The systems themselves are also heterogeneous. Each system comprises various elements, with, for instance, different surveys for patients and differences in the presentation of the collected information to the therapists. Lyon et al. (2016) describe several capabilities and characteristics of existing MFSs, thus providing a useful mapping of commonalities and differences in existing systems. However, there is limited research on how specific elements/components are associated with effective implementation and use of MFS. A system’s capabilities are no guarantee that it will be used—merely granting access does not mean the feedback is read. Further, reading the feedback does not guarantee understanding or clinical use of the information. Identifying the elements that can reduce barriers and increase use is important. Some aspects in this regard will be found outside of the system per se, e.g., in support systems for understanding the data, for collection of the data, ease of use, and efforts to build a culture for feedback use.
Process Elements of MFS
Computer-based systems for data collection and analysis could ease the burden on service providers. Systems that gather data outside of therapy sessions may be less disruptive to therapy, and the brevity of the measures may reduce stress for respondents. Automatic data synthesis, including scoring systems and individual graphs, can facilitate interpretation and use (Gelkopf et al., 2022). Easy access to data presented in an understandable fashion should increase use and effects; elaborate logins and unexplained raw data may produce difficulties in access and reduce intentions to use the data. MFS can provide therapists with scores on the scales used, graphs illustrating change over time, typologies of patients, and warning signals for not-on-track patients. The accuracy of predictions will increase if they are based on a comparison with normative patient progress data.
Implementation Elements of MFS
Implementers prioritize training, guidance, standby support, and manuals for the use and interpretation of the feedback data to different degrees. Such resources may help therapists understand how to use the data and its benefits. Well-informed therapists are also more likely to explain what MFS do and the reasons for using them to patients, who report uncertainty about their purpose and usage when encountering MFS (Börjesson & Boström, 2020). Studies differ in terms of whether the therapists participated voluntarily or whether the MFS was mandatory for the entire institution, both of which are likely to influence “ownership” and adherence. Feedback could be accompanied by discussion groups or supervision to help therapists understand data and revise treatment plans. Creating a “culture of feedback,” where opinions about the process and the outcome of services are welcomed and likely to have an impact on the nature and quality of services, may increase the use of feedback (Miller et al., 2016). MFS range from those that provide therapists with feedback data to those that provide suggestions for resolving identified problems. One meta-analysis indicated an additional effect in a subgroup of not-on-track patients from adding therapy suggestions through Clinical Support Tools (CST) to the MFS (Lambert et al., 2018).
In summary, a wealth of different strategies may be applied to improve the adoption and effects of MFS. More efficient systems may be created by identifying the elements that contribute, and those which do not contribute, to desired therapy outcomes. A systematic review and meta-analyses of theoretically informed process and implementation elements in MFS are thus needed to explore explanations for the heterogeneity in effects and large variations in feedback use and identify elements likely to improve the implementation and effectiveness of MFS.
The Current Review
The current review aimed to identify process and implementation elements in the MFS literature to provide an overview of the state of the field. Furthermore, we tested the impact of these elements on the effect of MFS by applying them as moderators in a meta-analysis. We also intended to map therapist use of feedback and therapist attitudes toward feedback use and apply them as moderators for the effect of the MFS.
Three central propositions of the CP-FIT informed our operationalization of process and implementation elements in MFS, i.e., capacity limitation, identity and culture, and behavioral induction. We hypothesized that measures taken to limit how taxing the systems are on clinicians (capacity limitation), ensure positive attitudes and a sense of ownership of the feedback (identity and culture), and support clinicians in comprehension and initiation of alternative treatment plans in response to feedback (behavioral induction) would increase use and effects of MFS. Identified elements were used to consider whether the prerequisites for well-functioning feedback systems according to each of CP-FIT’s 42 specific hypotheses were fulfilled. The scores on each hypothesis were then summarized in the three central propositions of the model.
Method
The current article is based on an update and secondary analyses of a review preregistered in PROSPERO (Code CRD42021240379) and follows the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines (Page et al., 2021). The review included both cluster-randomized and individually randomized studies with control conditions that were similar to the active condition except for the MFS tested. Participants were in treatment for common mental health problems (depression; mixed anxiety and depression; and specific anxiety disorders, such as generalized anxiety disorder (GAD), phobias, obsessive–compulsive disorder (OCD), panic disorder, and post-traumatic stress disorder (PTSD)) in any primary care, out- or inpatient therapy, or multidisciplinary mental health care setting.
Inclusion Criteria
-
Randomized controlled trials, both cluster-randomized and randomized at the level of participants
-
Participants in treatment with common mental health disorders, with the majority of participants having a diagnosis or clinical assessments indicating such a problem
-
Any age groups; studies with child and youth populations, as well as adult patient populations
-
Any studies of interventions where patient outcome data were given to therapists, patients, or both, on a regular basis for the duration of therapy
-
Any primary care, out- or inpatient therapy, multidisciplinary mental health care, or other psychological therapy settings
-
Studies where subsets of the data may qualify (fulfill criteria 1–4), e.g., three-armed RCTs where a portion of participants are relevant and can be extracted
Exclusion Criteria
-
Non-randomized design, including comparisons of assumed similar groups treated at different time periods, or benchmark studies
-
Studies comparing MFS to other treatment options besides treatment as usual (TAU)
-
Studies where the intervention arm also included other manual-based or otherwise defined interventions not available to both the intervention and control groups
-
Studies of group therapy or couples’ therapy, and studies with more than 10% of the sample in drug/alcohol treatment or with dementia, learning disorders, or psychosis
Information Sources and Search
The review updated the search from Rognstad et al. (2022) by conducting new searches in the Cochrane Central Registry of Controlled Trials (CENTRAL), American Psychology Associations’ PsychoInfo, Ovid MEDLINE and Epub Ahead of Print, In-Process, In-Data-Review & Other Non-Indexed Citations, Daily and Versions, and Excerpta Medica Database (EMBASE). The last search was done on January 31, 2023. The search terms and strategy are provided in Online Appendix 1.
All references were added to the Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia; available at www.covidence.org). Abstracts and full texts were screened independently by two reviewers. When disagreement occurred within a pair of reviewers, full-text articles were obtained, and disagreement was discussed until a consensus was reached.
Data Extraction and Risk of Bias
All data extraction was done by two of the reviewing authors independently for each study. Agreements were reached in meetings between the reviewers to produce a final data set. Predefined process and implementation elements were extracted, along with outcome data (post-treatment measurement means, standard deviations, and the number of respondents). Any outcome data on quality of life or functioning, mental health symptoms or mental health measures were included.
Risk of bias was also assessed independently by two reviewers and discussions were held to reach a consensus. Risk of bias was assessed in accordance with Cochrane Collaboration’s risk of bias tool (Sterne et al., 2019) for the studies that could provide outcome data suitable for use in the meta-analysis.
Coding Interventions
A coding manual inspired by the methods used by Engell et al. (2020) was created to identify the process and implementation elements included in the interventions. The coding manual included discrete implementation strategies per the ERIC taxonomy (Powell et al., 2015) and additional process and implementation elements from CP-FIT (Brown et al., 2019). A list of the a priori implementation elements is provided in Online Appendix 2. Further, the manual was developed through a data-driven process where articles were reviewed, and candidate process and implementation elements not covered in ERIC or CP-FIT were noted until no new elements emerged. Using SPSS, all discrete process and implementation elements were listed as variables and coded as present or absent based on information provided in the articles. In addition, study characteristics and data such as sample sizes, effect sizes, and variance measures were coded. All articles were coded by two of the reviewers in parallel. Any discrepancies were discussed in meetings between the reviewers until a consensus was reached. Two of the coders had prior experience in coding elements and effects from similar projects and participated in all meetings where discrepancies were discussed. As we found that several of the elements we searched for were not reported in many of the articles, corresponding authors for the included studies were contacted via email with a survey. Twelve authors completed the survey, which led to a more complete dataset.
Analysis
Primarily, we present a descriptive overview of the reporting of the process and implementation elements. The frequency of the presence of the elements in the codebook is reported to indicate the extent to which they were used in the MFS studies.
Effect sizes were calculated for all outcome measurements where reporting allowed for it. When available, post-treatment measurement means, standard deviation, and number of respondents were used to calculate Cohen’s d and variance of this effect size. Alternatively, reported effect sizes were converted into Cohen’s d. All effect sizes that indicated a treatment outcome, either in symptoms/mental health or quality of life/functioning, were included, resulting in 67 effect sizes from 30 studies, based on a total sample of 13,807 participants. A three-level analysis in accordance with the procedure described by Assink and Wibbelink (2016) allowing for the use of several outcome measures in each study was applied, as implemented in a shiny app (https://github.com/ToreWentzel-Larsen/threelevel).
Analyses were done using identified process and implementation elements as potential moderators. The process and implementation elements were used as indicators to test the hypotheses proposed by CP-FIT. Some of the elements were used for more than one of the hypotheses. Based on these indicators, studies were scored dichotomously as either meeting the premise of the hypotheses or not. In accordance with the CP-FIT, these hypothesis variables were grouped under the three general propositions of the model regarding what would govern the effect of feedback interventions: capacity limitations, identity and culture, and behavioral induction. Capacity limitations consisted of elements related to ease of use, automatization of collection and synthesizing of data, and immediate access to the data presentations. Identity and culture consisted of elements such as whether the therapists volunteered to participate and the presence of a “local advocate” or “champion.” Behavioral induction consisted of whether the feedback data were used in supervision or discussion groups, the presence of a clinical support tool (CST) or other devices to indicate potential actions warranted by the feedback data, the presence of a typology of patient progress, marking of not-on-track patients, lights or color coding, or messages to induce action from the therapists in response to the data. In the final step, more granular moderator analyses were performed in which data on the 42 more detailed hypotheses proposed by CP-FIT were applied as moderators.
Any descriptions of the degree of therapists’ use of feedback and therapists’ attitudes toward feedback were noted. An overview of these findings is provided in the results section. We planned to apply therapists’ use of feedback and therapists’ attitudes toward feedback as moderators for effect and, if possible, consider a mediating role for these variables between process and implementation variables and clinical outcomes. However, the review process revealed a lack of reporting in this regard that we considered quantifiable.
Results
Study Selection and Characteristics
Figure 1 shows the flow diagram of the search and inclusion process (PRISMA flow diagram). The literature search updated the findings of Rognstad et al. (2022), resulting in 241 new abstracts, but after two reviewers independently screened all abstracts and four full-text articles, no further studies were included. Thus, the final sample consisted of the same 39 studies.
The 39 included studies were mainly from the United States (k = 22) and Europe (k = 16), and most studied the effects of either the Outcome Questionnaire (OQ; k = 13 studies) or the Partners for Change Outcome Management System (PCOMS; k = 12 studies).
Risk of Bias in Identified Studies
All studies that allowed for the calculation of effect sizes were evaluated for risk of bias. All studies had high risk of bias due to the lack of blinding of personnel and participants (performance bias), while risk of bias was generally low or unclear in other areas (sequence generation, allocation concealment, blinding of outcome assessors, incomplete data/attrition bias, selective outcome reporting). Details on the within and across studies risk of bias can be found in Online Appendices 3 and 4.
Descriptive Data
Data on the process and implementation elements were extracted from 39 studies. Outcome data could be calculated or extracted for 30 of the included studies. Any effect size indicating a treatment outcome, either in symptoms/mental health or quality of life/functioning, was included. This resulted in 67 effect sizes.
Goals
A few studies indicated that they had engaged therapists in tailoring the design of the MFS (k = 5) and that the MFS had been tailored or adapted to the context (k = 5). Therapists did not seem to be involved in the choice of questionnaires or interviews for the feedback data. Most of the measures in the MFS were related to patients’ functioning/life circumstances during the previous week (k = 29) and/or symptoms/psychological distress (k = 25), while quite a few involved measures of alliance (k = 18). It is unclear whether the therapists’ sense of clinically relevant information informed the choice of these measures in any of the projects.
Data Collection and Analysis
There has been a tendency towards more automatization in the field in recent years. Thirteen of the studies had an electronic solution to gather data for the feedback system, and all of these were published post-2012. Fifteen of the studies reported that therapists were responsible for collecting the data; in twelve studies they did not have this responsibility, while the rest of the studies did not report on this. In about a third of the studies, data had to be transferred from paper surveys to electronic devices (k = 13).
Display
Most of the systems display graphs for patient progress (k = 31). Several of the systems gave indications of patients that were not-on-track (k = 24) or other typologies of patients (k = 3) e.g., by indicating patients above or below clinical cut-off scores.
Delivery
Feedback was given to therapists in a multitude of ways, either e-mailed (k = 3), on paper (k = 16), added to case records (k = 4), in a web application (9), or via an unreported delivery method.
Recipient Variables
In 14 studies, it was clearly stated that the therapists voluntarily participated in the feedback project. In one study, five sites allowed therapists to opt into the project, while it was obligatory at a sixth site. In the remaining studies, MFS was implemented at the institutional level (k = 10), or the studies did not provide any information on this.
Implementation Process
Only a few studies reported having a formal implementation plan k = (3) or that they had assessed readiness and/or identified barriers and facilitators for the implementation of MFS (k = 7). Likewise, few studies reported that they had provided training to leaders in implementation leadership or in supporting therapists’ use of MFS (k = 2).
Over half the studies described giving the feedback recipients training in using the MFS (k = 25) and reported providing training in the interpretation of the feedback reports (k = 25). Procedures for the use of the system or interpretation of the data were also manualized in most of the studies (k = 28). Half of the studies reported having technical assistance available for the users of the MFS (k = 20), and in six instances there was a local advocate/champion for the project.
Co-interventions
Only one study reported that the feedback data was also used for broader staff evaluation purposes. Four of the studies also implemented the clinical support tools (CST) associated with the Outcome Questionnaire and seven other studies had other systems to suggest or define suitable therapeutic action in response to feedback results.
Organization and Team Characteristics
Eight studies reported facilitating clinical supervision informed by feedback data, and in seven studies, feedback data was used in collegial group discussions. Two studies reported that training was provided to leaders in implementation leadership or in supporting therapists’ use of MFS.
Therapists’ Use of Feedback
Twelve studies did not contain information on how therapists used the system and another nine reported explicitly that they had no procedures for monitoring use. In the latter group, some indicated an impression of ambivalence and lack of commitment (e.g., Hansson et al., 2013).
The remaining studies had either some form of quantifiable data or commented more qualitatively on the therapists’ feedback use (or lack thereof). We did not consider it meaningful to use these levels of therapists’ use as a moderator of effect as the studies reported this in very different ways—post-hoc self-reports, data tracking, surveys for clinicians to evaluate the MFS, or observations and comments from the researchers. Some of the researchers’ comments gave the impression that feedback was heavily underused, as the articles reported, for example, that “several indications that counselors’ adherence to the PCOMS intervention was low” (Cooper, M 2021, p. 29).
Lutz et al. (2021) measured the therapist-rated usefulness of feedback, therapist-rated negative effects of feedback, and therapist evaluation of the feedback system, but not therapist use of the MFS directly. In a related vein, Trudeau (2000) asked clinicians to complete the Provider Satisfaction Questionnaire and received both positive evaluations of the feedback system and reports of feelings that it was a waste of time. Kellybrew-Miller (2015) monitored adherence through integrity checklists after each session and 78% of the integrity checklists were completed correctly in the MFS condition, indicating correct implementation.
Several of the studies had post-hoc self-reports from therapists about feedback use and the level of commitment from therapists varied considerably. In two studies, about half of the therapists indicated that they had not used the feedback, while in several other studies about a third had no interaction with the feedback (k = 3). More use was found in two studies, namely 72% in Connolly Gibbons et al. (2015) and 89% in Chang et al. (2012). In other studies, therapists reported a degree of use, as in Lambert et al. (2001) where therapists generally reported using the feedback “at least to a moderate degree” (p. 62). In Amble et al. (2014, 2015), the therapists reported that they used the feedback to a large extent (4.47 on a scale of 1–5). Similarly, the therapists in McClintock et al. (2017) were asked how frequently they discussed feedback with clients and reported a mean rating of 4.67 on a 5-point Likert scale (1 = never, 5 = always).
Three studies had data to calculate an implementation index indicating the degree to which therapists used MFS (Bickman et al., 2016 [rural] [urban]; Janse et al., 2020). Bickman et al. (2016) used these data to provide evidence that increased use also increased effect. In Janse et al. (2020), cases with a high implementation index had significantly shorter treatment and a tendency towards larger symptom reduction (albeit not statistically significant) compared to those with a low implementation index. Two other studies also used implementation or therapist use as a moderator in their analysis. Although they did not find a significant beneficial effect of the feedback in their full sample, de Jong et al. (2012) found that not-on-track cases had a significant positive effect when therapists indicated that they used the feedback. Brattland et al. (2018) tested a Time × Condition interaction and found MFS to have increasingly more effect with longer implementation time. In other words, the difference in post-treatment distress between conditions became larger for each month of the 4-year-long trial in favor of the MFS condition compared to TAU (Brattland et al., 2018).
Therapists’ Attitudes Towards Measurement and Feedback Systems
Most studies presented no data on therapists’ attitudes towards measurement or feedback systems, nor did they report anything about readiness for or user involvement prior to the implementation of the MFS intervention. Primarily, we were interested in data on therapists’ attitudes measured before the project started. Therapists’ attitudes were reported in two of the studies. McClintock et al. (2017) measured therapists’ beliefs about the effectiveness of therapy with or without MFS. In this study, the therapists believed that MFS-bolstered therapy would be more effective than TAU (4.50 vs. 3.83 on a 5-point Likert scale). Beliefs about MFS’ increased effectiveness were not used as a moderator in the study. de Jong et al. (2012) measured both perceived validity and commitment to using feedback. The correlation between perceived validity and commitment to using feedback was strong (r = .70). Surprisingly, a significant interaction was found between commitment and feedback in a negative direction, indicating that higher commitment led to a slower rate of change among the patients in cases where the therapist received feedback.
Lutz et al. (2021) gathered responses to the survey Therapist Attitude Toward and Confidence in Using Feedback after the project ended. This indicator of overall attitude to the use of MFS was significantly associated with better treatment outcomes on most of the outcome measures in the study. Lutz et al. (2015) found a larger effect size when therapists were satisfied with the MFS project than when they were not.
Several studies had post hoc surveys or interviews with the therapists to gauge satisfaction or perceived usefulness of the intervention (k = 8), while others only provided anecdotal or general statements about acceptance (k = 2). We deemed that the measures and interviews used were too heterogeneous for a comparison or summary to be sensible.
Process and Implementation Elements as Moderators of MFS Effects
A significant overall effect size estimate for clinical outcomes was found favoring MFS over TAU (d = 0.14, 95% CI [0.08–0.21], p < 0.001). The data could indicate the presence/absence of 16 of CP-FIT’s 42 hypotheses. These were summed up in the three general CP-FIT propositions, and moderator analyses were performed to investigate whether variables related to CP-FIT’s three general propositions adjusted the effect of MFS. None of the groupings of variables that represented the three general propositions of CP-FIT significantly moderated MFS effects: Capacity limitations (slope = − 0.013, 95% CI [− 0.033 to 0.007], p = 0.185); Identity and culture (slope = − 0.014, 95% CI [− 0.04 to 0.012], p = 0.284); Behavioral induction (slope = − 0.028, 95% CI [− 0.058 to 0.002], p = 0.067). The moderation of MFS effects from the more specific CP-FIT hypotheses was also tested separately, but these were non-significant (see Online Appendix 5 for a complete list of CP-FIT hypotheses for which our data set had variance that allowed for testing of moderation).
Heterogeneity and Risk of Reporting Bias
A modified Egger test (Egger et al., 1997; Marengo & Montag, 2020) was performed to investigate the heterogeneity among effect sizes. Significant heterogeneity was found (Q = 113.327, df = 65, p < 0.001), and a likelihood ratio test showed significant variance on the between-study level (SE = 0.12, p < 0.001). This suggests that between-study characteristics are likely to impact the overall effect estimate synthesizing effect sizes comparing MFS to TAU. The current review was initiated based on the proposition that much of the observed heterogeneity in the studies may be due to clinical differences and variation in implementation, as much as potential publication bias.
Discussion
This review systematically mapped theoretically informed process and implementation elements of MFS across the literature (Brown et al., 2019; Engell et al., 2023). Identified elements included aspects related to data collection and analysis, display, delivery, recipient variables, context, co-interventions, and organization and team characteristics. We found only a few studies reporting that therapists were involved in tailoring the design of feedback systems. The measures used in feedback systems were mainly related to patients’ functioning, life circumstances, symptoms, psychological distress, and alliance, but it is unclear whether end users or therapists had any influence on the choice of outcome data. The application of user-centered design principles has been proposed to close the “research–practice gap” (Lyon & Koerner, 2016) and this is clearly lacking in most MFS studies thus far.
Data collection was done quite evenly via surveys distributed electronically or in pen and paper format, and in a large part of the studies, the therapists themselves were responsible for the data collection. In one study, the use of tablet surveys rather than pen-and-paper surveys was associated post hoc with greater implementation and positive effects of MFS (Bickman et al., 2016). Most systems displayed graphs for patient progress and indicated patients who were not on track.
Therapists voluntarily participated in feedback projects in a minority of the studies, and there was limited reporting on implementation plans, readiness assessments, and training for leaders in implementing feedback systems. In a clear minority of studies, feedback data were used for clinical supervision or group discussions. Overall, the findings highlight the need for more research on the implementation and effectiveness of feedback systems in mental health services. Despite searching in the studies for different implementation strategies, determinants, competencies, and other relevant parts of the implementation processes, most of the implementation elements in the identified articles reflect common implementation strategies (Waltz et al., 2015). One exception was therapist attitudes towards MFS which was reported in two studies. However, data from only two studies did not allow statistical testing of how attitudes were associated with effects.
Neither the moderator analyses used to test CP-FIT’s three general propositions nor those used to test the more specific hypotheses of the model had any significant results. In fact, the Behavioural induction proposition was close to significant in the opposite direction than proposed by the model. Most of the studies provided evidence of effect, or lack thereof, for different systems without identifying the factors likely to influence success. Although pragmatism may have its place, going forward, there is, as Kurt Lewin (1943) famously put it, “nothing as practical as a good theory.” CP-FIT is one potential framework for predicting the effects of MFS, but our data do not support the three general propositions in CP-FIT. However, this may be explained by the lack of attention to process and implementation and/or lack of reporting in the included studies, indicated by the low frequencies of relevant variables reported. Many rational ideas and strategies have been applied in the field that may help overcome barriers reported by health professionals and hypothesized by feedback theory. Still, so far, they appear to be underused, and we could not statistically connect these with the clinical effects of the interventions.
Some of the hypotheses in CP-FIT may not apply to MFS implementation, and specific functions of MFS interventions may influence therapists in more ways than one. For example, automated data collection should reduce the burden on the therapists and hence reduce problems related to capacity limitation. Yet it may also be that by demanding therapists to be “hands-on” in collecting and analyzing data, engagement with feedback data increases as therapists are already closely involved. As such, those demands may function as a nudge toward feedback engagement, a common strategy to change behavior (Yoong et al., 2020). The processes and elements hypothesized by CP-FIT to predict feedback use and success are based on a comprehensive systematic review of qualitative research on feedback interventions (Brown et al., 2019). However, these hypotheses may have to be tested experimentally to provide clearer answers regarding their relevance in MFS interventions.
Fidelity would be a key implementation outcome (Lewis et al., 2019), but the current review finds fidelity monitoring to be scarce and lacking standard reporting methods in the field. Therapists’ use of MFS was only reported in a few of the examined studies. The studies that provided information about use indicated that about one-third to half of the therapists did not engage with the available feedback. This might be expected as self-report surveys from therapists have indicated that even those who acknowledge the potential benefits of MFS may perceive the barriers to outweigh the positives (Chung & Buchanan, 2019). In different ways, a few of the studies were able to show that increased use was associated with better outcomes (Bickman et al., 2011, 2016; de Jong et al., 2012; Janse et al., 2020). Also, one study observed significant improvement in the MFS condition over time and proposed that this may be explained by continuous implementation efforts (Brattland et al., 2018). Therapists’ attitudes toward MFS, feedback, or standardized measures were very heterogeneously reported, and thus did not allow for data synthesis. Amble et al. (2014) showed high self-reported use by therapists who mostly self-recruited to the MFS project and higher effect sizes (Cohen’s d = .32) compared to the overall effect estimate of the current meta-analysis (Cohen’s d = .14).
We found the degree of implementation to be quite rarely reported, which did not permit comparison or data synthesis across studies. As Bickman et al. (2016) pointed out, a minimum for MFS studies should be to calculate an implementation index reporting the proportion of surveys answered by the patients and the proportion of feedback reports looked at by therapists. Appropriate measures of the degree of implementation can help exclude type III errors where one interprets study results as evidence of lack of effect despite attempting to measure something that did not exist (Dobson & Cook, 1980), in this case, the effect from assumed MFS use in cases lacking sufficient implementation of the system. Even though the implementation index proposed by Bickman et al. (2016) is a crude measure of implementation that only shows the potential amount of feedback the therapist could have used, both questionnaire completion and viewing of feedback must be at some reasonable level before implementation can be successful. Still, this would only indicate success in three of the first steps in the CP-FIT cycle (2. Data collection and analysis, 3. Feedback, and 4. Interaction). Following CP-FIT, one would also warrant measures of implementation elements such as perception, verification, and acceptance of the feedback, and subsequent intentions based on this reception. For this, it could be useful to gather documentation of when the therapists have discussed the feedback (Lewis et al., 2019) either with the clients, colleagues, or supervisors. In the current review, acceptance proved sparsely reported in most studies, and those who did often reported more general attitudes to measurement or satisfaction with the research project. We suggest that MFS researchers also attempt to gather data on the frequency of clinical decisions that are informed by the feedback data, as this will give an indication of the overall success of the implementation of a feedback system, from goal setting, data collection, therapist and client reception, intentions, to therapeutical behavior.
Only one of the identified studies (Bickman et al., 2016) reported feedback data being used for staff performance evaluation or other quality registry purposes. This will likely change as precision medicine and artificial intelligence are already making their way into psychotherapy clinics (Flemotomos et al., 2022; Imel et al., 2019). The same data may be used for MFS and as administrative tools and, as such, become an integrated part of clinics. This may facilitate the creation of a “culture of feedback” (Miller et al., 2016) where user opinions are welcomed and impact the services provided. On the other hand, it may also cause resistance from healthcare providers who experience discomfort and worry that the data will be used for top-down control and limit service accessibility (Moran et al., 2012). To facilitate the former effect and reduce the latter, efforts may need to be made to increase therapists’ ownership over MFS. An increased focus on process and implementation elements may be necessary to increase compatibility and appeal and to get therapists on board, for example by engaging them in tailoring MFS to their needs and providing appropriate training and supervision in MFS use. Therapists will be more likely to use MFS if they are drawn to the information gained rather than pushed by external pressures (Hatfield & Ogles, 2004).
Limitations
The quality of the data from original studies influences the quality of any systematic review. As the current review set out to investigate topics that were not the central tenet of the literature reviewed, there are clear limitations. The degree of implementation, as well as therapists’ use and attitudes, were only reported in a small fraction of the identified studies. It was difficult to test all CP-FIT propositions as the original studies may have underreported implementation details and were not designed to test CP-FIT. In most cases, they were not designed to test any hypotheses about process or implementation. Several aspects of CP-FIT could not be tested, as the design or reporting in the original studies did not prioritize this.
Although 12 of the corresponding authors answered the survey we sent out to get a more complete data set, it is still unclear whether low frequencies of several variables are due to underreporting or their absence in the studies. The field of implementation science and the reporting standards it has produced are still young (see Rudd et al., 2020 for a review of reporting standards), and some of the studies date back to a time when, in general, publications devoted little space to implementation strategies and details. Also, in the newer studies, implementation aspects were likely not the focus of articles reporting outcomes from MFS interventions.
Implications
Future studies on MFS should focus more on the processual aspects of MFS and implementation, both to increase the reach and effectiveness of MFS and to uncover the mechanisms for MFS’ success. Lack of reporting of process and implementation elements is a potential problem, and the mechanisms behind effective MFS interventions may be better charted if this improves in future studies.
It should be noted that several single studies pointed to increased use of and commitment to using MFS as being advantageous for clinical outcomes. This is likely to be promoted through well-planned and well-conducted implementation, even though the current review fails to link these elements to better outcomes. Future studies should aim to empirically test hypotheses regarding the process and implementation elements that are likely to influence the use of MFS as we currently cannot provide clear answers.
References
References to Studies Included in this Review
Amble, I., Gude, T., Stubdal, S., Andersen, B. J., & Wampold, B. E. (2014). The effect of implementing the Outcome Questionnaire-45.2 feedback system in Norway: A multisite randomized clinical trial in a naturalistic setting. Psychotherapy Research, 25(6), 669–677. https://doi.org/10.1080/10503307.2014.928756
Bastiaansen, J. A., Ornée, D. A., Meurs, M., & Oldehinkel, A. J. (2020). An evaluation of the efficacy of two add-on ecological momentary intervention modules for depression in a pragmatic randomized controlled trial (ZELF-i). Psychological Medicine, 52(13), 1–10. https://doi.org/10.1017/S0033291720004845
Bickman, L., Douglas, S. R., De Andrade, A. R. V., Tomlinson, M., Gleacher, A., Olin, S., & Hoagwood, K. (2016). Implementing a measurement feedback system: A tale of two sites. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 410–425. https://doi.org/10.1007/s10488-015-0647-8
Bickman, L., Kelley, S. D., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services, 62(12), 1423–1429. https://doi.org/10.1176/appi.ps.002052011
Bovendeerd, B., de Jong, K., de Groot, E., Moerbeek, M., & de Keijser, J. (2021). Enhancing the effect of psychotherapy through systematic client feedback in outpatient mental healthcare: A cluster randomized trial. Psychotherapy Research, 32(6), 710–722. https://doi.org/10.1080/10503307.2021.2015637
Brattland, H., Koksvik, J. M., Burkeland, O., Gråwe, R. W., Klöckner, C., Linaker, O. M., & Iversen, V. C. (2018). The effects of routine outcome monitoring (ROM) on therapy outcomes in the course of an implementation process: A randomized clinical trial. Journal of Counseling Psychology, 65(5), 641–652. https://doi.org/10.1037/cou0000286
Chang, T. E., Jing, Y., Yeung, A. S., Brenneman, S. K., Kalsekar, I., Hebden, T., McQuade, R., Baer, L., Kurlander, J. L., & Watkins, A. K. (2012). Effect of communicating depression severity on physician prescribing patterns: Findings from the Clinical Outcomes in Measurement-based Treatment (COMET) trial. General Hospital Psychiatry, 34(2), 105–112. https://doi.org/10.1016/j.genhosppsych.2011.12.003
Connolly Gibbons, M. B., Kurtz, J. E., Thompson, D. L., Mack, R. A., Lee, J. K., Rothbard, A., Eisen, S. V., Gallop, R., & Crits-Christoph, P. (2015). The effectiveness of clinician feedback in the treatment of depression in the community mental health system. Journal of Consulting and Clinical Psychology, 83(4), 748–759. https://doi.org/10.1037/a0039302
Cooper, M., Duncan, B., Golden, S., & Toth, K. (2021a). Systematic client feedback in therapy for children with psychological difficulties: Pilot cluster randomised controlled trial. Counselling Psychology Quarterly, 34(1), 21–36. https://doi.org/10.1080/09515070.2019.1647142
de Jong, K., Timman, R., Hakkaart-Van Roijen, L., Vermeulen, P., Kooiman, K., Passchier, J., & Busschbach, J. V. (2014). The effect of outcome monitoring feedback to clinicians and patients in short and long-term psychotherapy: A randomized controlled trial. Psychotherapy Research, 24(6), 629–639. https://doi.org/10.1080/10503307.2013.871079
de Jong, K., van Sluis, P., Nugter, M. A., Heiser, W. J., & Spinhoven, P. (2012). Understanding the differential impact of outcome monitoring: Therapist variables that moderate feedback effects in a randomized clinical trial. Psychotherapy Research, 22(4), 464–474. https://doi.org/10.1080/10503307.2012.673023
Delgadillo, J., McMillan, D., Gilbody, S., de Jong, K., Lucock, M., Lutz, W., Rubel, J., Aguirre, E., & Ali, S. (2021). Cost-effectiveness of feedback-informed psychological treatment: Evidence from the IAPT-FIT trial. Behaviour Research and Therapy, 142, 103873. https://doi.org/10.1016/j.brat.2021.103873
Duncan, B. L., Reese, R. J., Lengerich, A. J., DeSantis, B., Comeau, C. V., & Johnson-Esparza, Y. (2021). Measurement-based care in integrated health care: A randomized clinical trial. Families, Systems, & Health, 39(2), 259–268. https://doi.org/10.1037/fsh0000608
Errázuriz, P., & Zilcha-Mano, S. (2018). In psychotherapy with severe patients discouraging news may be worse than no news: The impact of providing feedback to therapists on psychotherapy outcome, session attendance, and the alliance. Journal of Consulting and Clinical Psychology, 86(2), 125–139. https://doi.org/10.1037/ccp0000277
Garland, A. (2017). The effectiveness of utilizing the treatment support measure for treatment planning in youth mental health services. Brigham Young University.
Hansson, H., Rundberg, J., Österling, A., Öjehagen, A., & Berglund, M. (2013). Intervention with feedback using Outcome Questionnaire 45 (OQ-45) in a Swedish psychiatric outpatient population. A randomized controlled trial. Nordic Journal of Psychiatry, 67(4), 274–281. https://doi.org/10.3109/08039488.2012.736534
Harris, M. W. (2011). Providing patient progress information and clinical support tools to therapists: Effects on patients at risk for treatment failure. Brigham Young University.
Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K. L., & Tuttle, K. C. (2006). The therapeutic effects of providing patient progress information to therapists and patients. Psychotherapy Research, 14(3), 308–327. https://doi.org/10.1093/ptr/kph027
Janse, P. D., de Jong, K., Veerkamp, C., van Dijk, M. K., Hutschemaekers, G. J., & Verbraak, M. J. (2020). The effect of feedback-informed cognitive behavioral therapy on treatment outcome: A randomized controlled trial. Journal of Consulting and Clinical Psychology, 88(9), 818–828. https://doi.org/10.1037/ccp0000549
Kellybrew-Miller, A. (2015). The impact of systematic client feedback on client outcomes in a community mental health center. University of Central Arkansas.
Kendrick, T., Stuart, B., Leydon, G. M., Geraghty, A. W., Yao, L., Ryves, R., & Lewis, G. (2017). Patient-reported outcome measures for monitoring primary care patients with depression: PROMDEP feasibility randomised trial. British Medical Journal Open, 7(3), e015266. https://doi.org/10.1136/bmjopen-2016-015266
Lambert, M. J., Whipple, J. L., Smart, D. W., Vermeersch, D. A., Nielsen, S. L., & Hawkins, E. J. (2001). The effects of providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced? Psychotherapy Research, 11(1), 49–68. https://doi.org/10.1080/713663852
Lester, M. C. (2012). The effectiveness of client feedback measures with adolescents in an acute psychiatric inpatient setting. University of Central Arkansas.
Lutz, W., Deisenhofer, A.-K., Rubel, J., Bennemann, B., Giesemann, J., Poster, K., & Schwartz, B. (2021). Prospective evaluation of a clinical decision support system in psychological therapy. Journal of Consulting and Clinical Psychology, 90(1), 90–106. https://doi.org/10.1037/ccp0000642
Lutz, W., Rubel, J., Schiefele, A. K., Zimmermann, D., Böhnke, J. R., & Wittmann, W. W. (2015). Feedback and therapist effects in the context of treatment outcome and treatment length. Psychotherapy Research, 25(6), 647–660. https://doi.org/10.1080/10503307.2015.1053553
McClintock, A. S., Perlman, M. R., McCarrick, S. M., Anderson, T., & Himawan, L. (2017). Enhancing psychotherapy process with common factors feedback: A randomized, clinical trial. Journal of Counseling Psychology, 64(3), 247–260. https://doi.org/10.1037/cou0000188
Melendez, G. (2002). Effects of feedback to the therapist on child clinical outcomes: A patient-oriented approach to demonstrating effective practice. Ohio University.
Murphy, K. P., Rashleigh, C. M., & Timulak, L. (2012). The relationship between progress feedback and therapeutic outcome in student counselling: A randomised control trial. Counselling Psychology Quarterly, 25(1), 1–18. https://doi.org/10.1080/09515070.2012.662349
Ogles, B. M., Carlston, D., Hatfield, D., Melendez, G., Dowell, K., & Fields, S. A. (2006). The role of fidelity and feedback in the wraparound approach. Journal of Child and Family Studies, 15(1), 114–128. https://doi.org/10.1007/s10826-005-9008-7
Probst, T., Lambert, M. J., Dahlbender, R. W., Loew, T. H., & Tritt, K. (2014). Providing patient progress feedback and clinical support tools to therapists: Is the therapeutic process of patients on-track to recovery enhanced in psychosomatic in-patient therapy under the conditions of routine practice? Journal of Psychosomatic Research, 76(6), 477–484. https://doi.org/10.1016/j.jpsychores.2014.03.010
Reese, R. J., Norsworthy, L. A., & Rowlands, S. R. (2009). Does a continuous feedback system improve psychotherapy outcome? Psychotherapy: Theory, Research, Practice, Training, 46(4), 418–431. https://doi.org/10.1037/a0017901
Rise, M. B., Eriksen, L., Grimstad, H., & Steinsbekk, A. (2016). The long-term effect on mental health symptoms and patient activation of using patient feedback scales in mental health out-patient treatment. A randomised controlled trial. Patient Education and Counseling, 99(1), 164–168. https://doi.org/10.1016/j.pec.2015.07.016
Schöttke, H., Unrath, M., & Uhlmann, C. (2019). The effect of patient progress feedback on psychotherapy outcome. Verhaltenstherapie, 32, 116–126. https://doi.org/10.1159/000503765
She, Z., Duncan, B. L., Reese, R. J., Sun, Q., Shi, Y., Jiang, G., Wu, C., & Clements, A. L. (2018). Client feedback in China: A randomized clinical trial in a college counseling center. Journal of Counseling Psychology, 65(6), 727–737. https://doi.org/10.1037/cou0000300
Simon, W., Lambert, M. J., Harris, M. W., Busath, G., & Vazquez, A. (2012). Providing patient progress information and clinical support tools to therapists: Effects on patients at risk of treatment failure. Psychotherapy Research, 22(6), 638–647. https://doi.org/10.1080/10503307.2012.698918
Trudeau, L. S. (2000). Effects of a clinical feedback system on client and therapist outcomes in a rural community mental health center. Iowa State University.
Whipple, J. L., Lambert, M. J., Vermeersch, D. A., Smart, D. W., Nielsen, S. L., & Hawkins, E. J. (2003). Improving the effects of psychotherapy: The use of early identification of treatment and problem-solving strategies in routine practice. Journal of Counseling Psychology, 50(1), 59–68. https://doi.org/10.1037/0022-0167.50.1.59
Additional References
Amble, I., Gude, T., Ulvenes, P., Stubdal, S., & Wampold, B. E. (2015). How and when feedback works in psychotherapy: Is it the signal? Psychotherapy Research, 26(5), 545–555. https://doi.org/10.1080/10503307.2015.1053552
Assink, M., & Wibbelink, C. J. (2016). Fitting three-level meta-analytic models in R: A step-by-step tutorial. The Quantitative Methods for Psychology, 12(3), 154–174. https://doi.org/10.20982/tqmp.12.3.p154
Brown, B., Gude, W. T., Blakeman, T., van der Veer, S. N., Ivers, N., Francis, J. J., Lorencatto, F., Presseau, J., Peek, N., & Daker-White, G. (2019). Clinical Performance Feedback Intervention Theory (CP-FIT): A new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implementation Science, 14(1), 1–25. https://doi.org/10.1186/s13012-019-0883-5
Börjesson, S., & Boström, P. K. (2020). “I want to know what it is used for”: Clients’ perspectives on completing a routine outcome measure (ROM) while undergoing psychotherapy. Psychotherapy Research, 30(3), 337–347. https://doi.org/10.1080/10503307.2019.1630780
Bührmann, L., Driessen, P., Metz, A., Burke, K., Bartley, L., Varsi, C., & Albers, B. (2022). Knowledge and attitudes of Implementation Support Practitioners—Findings from a systematic integrative review. PLoS ONE, 17(5), e0267533. https://doi.org/10.1371/journal.pone.0267533
Chorpita, B. F., Daleiden, E. L., & Weisz, J. R. (2005). Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Mental Health Services Research, 7(1), 5–20. https://doi.org/10.1007/s11020-005-1962-6
Chung, J., & Buchanan, B. (2019). A self-report survey: Australian clinicians’ attitudes towards progress monitoring measures. Australian Psychologist, 54(1), 3–12. https://doi.org/10.1111/ap.12352
Cooper, L. D., Murphy, H. G., Delk, L. A., Fraire, M. G., Van Kirk, N., Sullivan, C. P., Waldron, J. C., Halliburton, A. E., Schiefelbein, F., & Gatto, A. (2021b). Implementing routine outcome monitoring in a psychology training clinic: A case study of a process model. Training and Education in Professional Psychology, 15(2), 87–96. https://doi.org/10.1037/tep0000298
de Jong, K., Conijn, J. M., Gallagher, R. A., Reshetnikova, A. S., Heij, M., & Lutz, M. C. (2021). Using progress feedback to improve outcomes and reduce drop-out, treatment duration, and deterioration: A multilevel meta-analysis. Clinical Psychology Review, 85, 102002. https://doi.org/10.1016/j.cpr.2021.102002
Dobson, D., & Cook, T. J. (1980). Avoiding type III error in program evaluation: Results from a field experiment. Evaluation and Program Planning, 3(4), 269–276. https://doi.org/10.1016/0149-7189(80)90042-7
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629
Engell, T., Kirkøen, B., Hammerstrøm, K. T., Kornør, H., Ludvigsen, K. H., & Hagen, K. A. (2020). Common elements of practice, process and implementation in out-of-school-time academic interventions for at-risk children: A systematic review. Prevention Science, 21(4), 545–556. https://doi.org/10.1007/s11121-020-01091-w
Engell, T., Stadnick, N. A., Aarons, G. A., & Barnett, M. L. (2023). Common elements approaches to implementation research and practice: Methods and integration with intervention science. Global Implementation Research and Applications, 3(1), 1–15. https://doi.org/10.1007/s43477-023-00077-4
Flemotomos, N., Martinez, V. R., Chen, Z., Singla, K., Ardulov, V., Peri, R., Caperton, D. D., Gibson, J., Tanana, M. J., Van Epps, J., & Georgiou, P. (2022). Automated evaluation of psychotherapy skills using speech and language technologies. Behavior Research Methods, 54(2), 690–711.
Gelkopf, M., Mazor, Y., & Roe, D. (2022). A systematic review of patient-reported outcome measurement (PROM) and provider assessment in mental health: Goals, implementation, setting, measurement characteristics and barriers. International Journal for Quality in Health Care, 34(Supplement_1), ii13–ii27.
Gondek, D., Edbrooke-Childs, J., Fink, E., Deighton, J., & Wolpert, M. (2016). Feedback from outcome measures and treatment effectiveness, treatment efficiency, and collaborative practice: A systematic review. Administration and Policy in Mental Health and Mental Health Services Research, 43, 325–343. https://doi.org/10.1007/s10488-015-0710-5
Hatfield, D. R., & Ogles, B. M. (2004). The use of outcome measures by psychologists in clinical practice. Professional Psychology: Research and Practice, 35(5), 485–491. https://doi.org/10.1037/0735-7028.35.5.485
Helland, S. S., Mellblom, A., Kjøbli, J., Wentzel-Larsen, T., Espenes, K., Engell, T., & Kirkøen, B. (2022). Elements in mental health interventions associated with effects on emotion regulation in adolescents: A meta-analysis. Administration and Policy in Mental Health and Mental Health Services Research, 49(6), 1004–1018. https://doi.org/10.1007/s10488-022-01213-2
Imel, Z. E., Pace, B. T., Soma, C. S., Tanana, M., Hirsch, T., Gibson, J., Georgiou, P., Narayanan, S., & Atkins, D. C. (2019). Design feasibility of an automated, machine-learning based feedback system for motivational interviewing. Psychotherapy, 56(2), 318–328. https://doi.org/10.1037/pst0000221
Ishaque, S., Karnon, J., Chen, G., Nair, R., & Salter, A. B. (2019). A systematic review of randomised controlled trials evaluating the use of patient-reported outcome measures (PROMs). Quality of Life Research, 28, 567–592. https://doi.org/10.1007/s11136-018-2016-z
Kendrick, T., El-Gohary, M., Stuart, B., Gilbody, S., Churchill, R., Aiken, L., Bhattacharya, A., Gimson, A., Bruett, A. L., Moore, M., & de Jong, K. (2016). Routine use of patient reported outcome measures (PROMs) for improving treatment of common mental health disorders in adults. Cochrane Database of Systematic Reviews, 7(7), CD011119.
Kjøbli, J., Melendez-Torres, G., Gardner, F., Backhaus, S., Linnerud, S., & Leijten, P. (2023). Research review: Effects of parenting programs for children’s conduct problems on children’s emotional problems—A network meta-analysis. Journal of Child Psychology and Psychiatry, 64(3), 348–356. https://doi.org/10.1111/jcpp.13697
Kvamme, L. S., Keles, S., Nes, R. B., Vaskinn, L., Waaler, P. M., Wentzel-Larsen, T., & Kjøbli, J. (2022). Common practice elements in treatment programs for adolescents with externalizing and internalizing problems: A meta-analysis. Residential Treatment for Children & Youth. https://doi.org/10.1080/0886571X.2022.2111397
Lambert, M. J., Whipple, J. L., & Kleinstäuber, M. (2018). Collecting and delivering progress feedback: A meta-analysis of routine outcome monitoring. Psychotherapy, 55(4), 520. https://doi.org/10.1037/pst0000167
Lambert, M. J., Whipple, J. L., & Kleinstäuber, M. (2019). Collecting and delivering client feedback. In J. C. Norcross & M. J. Lambert (Eds.), Psychotherapy relationships that work: Evidence-based therapist contributions (pp. 580–630). Oxford University. https://doi.org/10.1093/med-psych/9780190843953.003.0017
Leijten, P., Gardner, F., Melendez-Torres, G., Van Aar, J., Hutchings, J., Schulz, S., Knerr, W., & Overbeek, G. (2019). Meta-analyses: Key parenting program components for disruptive child behavior. Journal of the American Academy of Child & Adolescent Psychiatry, 58(2), 180–190. https://doi.org/10.1016/j.jaac.2018.07.900
Lewin, K. (1943). Psychology and the process of group living. Journal of Social Psychology, 17, 113–131.
Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., Hoffman, M., Scott, K., Lyon, A., Douglas, S., & Kroenke, K. (2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76(3), 324–335. https://doi.org/10.1001/jamapsychiatry.2018.3329
Lyon, A. R., & Koerner, K. (2016). User-centered design for psychosocial intervention development and implementation. Clinical Psychology, 23(2), 180–200. https://doi.org/10.1111/cpsp.12154
Lyon, A. R., Lewis, C. C., Boyd, M. R., Hendrix, E., & Liu, F. (2016). Capabilities and characteristics of digital measurement feedback systems: Results from a comprehensive review. Administration and Policy in Mental Health and Mental Health Services Research, 43, 441–466. https://doi.org/10.1007/s10488-016-0719-4
Marengo, D., & Montag, C. (2020). Digital phenotyping of big five personality via Facebook data mining: A meta-analysis. Digital Psychology, 1(1), 52–64. https://doi.org/10.24989/dp.v1i1.1823
Metz, A., Albers, B., Bruke, K., Bartley, L., Louison, L., Ward, C., & Farley, A. (2021). Implementation practice in human service systems: Understanding the principles and competencies of professionals who support implementation. Human Service Organizations: Management, Leadership & Governance. https://doi.org/10.1080/23303131.2021.1895401
Miller, S. D., Bargmann, S., Chow, D., Seidel, J., & Maeschalck, C. (2016). Feedback-informed treatment (FIT): Improving the outcome of psychotherapy one person at a time. In W. O’Donohue & A. Maragakis (Eds.), Quality improvement in behavioral health (pp. 247–262). Springer International Publishing/Springer Nature. https://doi.org/10.1007/978-3-319-26209-3_16
Moran, P., Kelesidi, K., Guglani, S., Davidson, S., & Ford, T. (2012). What do parents and carers think about routine outcome measures and their use? A focus group study of CAMHS attenders. Clinical Child Psychology and Psychiatry, 17(1), 65–79. https://doi.org/10.1177/1359104510391859
Nilsen, P., & Bernhardson, S. (2019). Context matters in implementation science: A scoping review of determinant frameworks that describe contextual determinants for implementation outcomes. BMC Health Services Research, 19(1), 189. https://doi.org/10.1186/s12913-019-4015-3
Østergård, O. K., Randa, H., & Hougaard, E. (2020). The effect of using the Partners for Change Outcome Management System as feedback tool in psychotherapy—A systematic review and meta-analysis. Psychotherapy Research, 30(2), 195–212. https://doi.org/10.1080/10503307.2018.1517949
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hrobjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. International Journal of Surgery, 88, 105906. https://doi.org/10.1136/bmj.n71
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10(1), 1–14. https://doi.org/10.1186/s13012-015-0209-1
Rognstad, K., Wentzel-Larsen, T., Neumer, S.-P., & Kjøbli, J. (2022). A systematic review and meta-analysis of measurement feedback systems in treatment for common mental health disorders. Administration and Policy in Mental Health and Mental Health Services Research, 50(2), 269–282. https://doi.org/10.1007/s10488-022-01236-9
Rudd, B. N., Davis, M., & Beidas, R. S. (2020). Integrating implementation science in clinical research to maximize public health impact: A call for the reporting and alignment of implementation strategy use with implementation outcomes in clinical research. Implementation Science, 15(1), 103. https://doi.org/10.1186/s13012-020-01060-5
Sterne, J. A., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H-Y., Corbett, M. S., Eldridge, S. M., Emberson, J. R., Hernán, M. A., Hopewell, S., Hróbartsson, A., Junquiera, D. R., Jüni, P., Kirkham, J. J., Lasserson, T., Li, T., ... Higgins, J. P. (2019). RoB 2: A revised tool for assessing risk of bias in randomised trials. BMJ. https://doi.org/10.1136/bmj.l4898
Waltz, T. J., Powell, B. J., Matthieu, M. M., Damschroder, L. J., Chinman, M. J., Smith, J. L., Proctor, E. K., & Kirchner, J. E. (2015). Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: Results from the Expert Recommendations for Implementing Change (ERIC) study. Implementation Science, 10, 1–8. https://doi.org/10.1186/s13012-015-0295-0
Warren, J. S., Nelson, P. L., Mondragon, S. A., Baldwin, S. A., & Burlingame, G. M. (2010). Youth psychotherapy change trajectories and outcomes in usual care: Community mental health versus managed care settings. Journal of Consulting and Clinical Psychology, 78(2), 144–155. https://doi.org/10.1037/a0018544
Weisz, J. R., Kuppens, S., Ng, M. Y., Eckshtain, D., Ugueto, A. M., Vaughn-Coaxum, R., Jensen-Doss, A., Hawley, K. M., Marchette, L. S. K., Chu, B. C., Weersing, V. R., & Fordwood, S. R. (2017). What five decades of research tells us about the effects of youth psychological therapy: A multilevel meta-analysis and implications for science and practice. American Psychologist, 72(2), 79–117. https://doi.org/10.1037/a0040360
Wolpert, M. (2016). Failure is an option. The Lancet Psychiatry, 3(6), 510–512. https://doi.org/10.1016/S2215-0366(16)30075-X
Yoong, S. L., Hall, A., Stacey, F., Grady, A., Sutherland, R., Wyse, R., Anderson, A., Nathan, N., & Wolfenden, L. (2020). Nudge strategies to improve healthcare providers’ implementation of evidence-based guidelines, policies and practices: A systematic review of trials included within Cochrane systematic reviews. Implementation Science: IS, 15(1), 1–50. https://doi.org/10.1186/s13012-020-01011-0
Funding
Open access funding provided by University of Oslo (incl Oslo University Hospital).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Rognstad, K., Engell, T., Fjermestad, K. et al. Process and Implementation Elements of Measurement Feedback Systems: A Systematic Review. Adm Policy Ment Health (2023). https://doi.org/10.1007/s10488-023-01325-3
Accepted:
Published:
DOI: https://doi.org/10.1007/s10488-023-01325-3