Attitudes Towards the Use of Routine Outcome Monitoring of Psychological Therapies Among Mental Health Providers: The EBPAS–ROM

Abstract

Implementation of routine outcome monitoring (ROM) in mental health care is progressing slowly. Knowledge about factors influencing ROM implementation, including health providers’ attitudes towards ROM, is necessary. Based on a survey of 662 psychologists and nurses, this article describes (1) the development of a short instrument measuring provider attitudes towards ROM, derived from the Evidence-based Practice Attitude Scale (EBPAS), and (2) how attitudinal domains relate to clinicians’ current use of standardized instruments for treatment evaluation. The EBPAS–ROM showed concurrent validity in predicting aspects important for the implementation of ROM, including perceived limitations and the value of organizational support.

Efforts to integrate science and practice in mental health care increasingly involve the implementation of routine outcome monitoring (ROM). ROM involves the systematic evaluation of patient progress throughout the course of treatment, using standardized outcome measures to elicit client feedback on mental health status and treatment outcomes as an integral part of the clinical service provided (Lambert and Harmon 2018; Lambert 2007; Wampold 2015). The term ROM is often used interchangeable with the term measurement feedback systems (MFSs) which emphasize the use of systems to provide feedback from those accessing services (Bickman 2008; Bickman et al. 2016a, b), as well as other related terms like Feedback Informed Therapy (Miller et al. 2015). Despite terminology differences, they all share common intentions with regard to using standardized measures to assess and improve patient outcomes as part of a measurement based care delivery approach in routine mental and behavioral health care (Bickman et al. 2016a, b; Edbrooke-Childs et al. 2016; Lewis et al. 2019; Scott and Lewis 2015; The Joint Commission 2018). From the patient, health provider and service manager points of view, access to feedback on patient progress should be of vital interest. For instance, with regard to quality improvements, such feedback can guide decisions about continuing or reconsidering the courses of individual treatments, as well as guide the distribution of resources (e.g., which treatments should be delivered in which doses to which patients and the scaling of training and support needed for the therapist delivering the treatments) (Bickman 2008; Lambert 2007; Wampold 2015). ROM has shown to improve client outcomes in numerous studies, especially for patients who are off-track or not responding to treatment as expected (e.g., Amble et al. 2015; Bickman et al. 2011, 2016a, b; Brattland et al. 2018; Carlier et al. 2012; De Jong et al. 2014; Lambert and Harmon 2018; Lambert et al. 2018; Lambert et al. 2001, 2007; Shimokawa et al. 2010; Simon et al. 2012). However, despite decades of literature supporting the use of ROM, the actual use of ROM in routine mental health care settings remains low, and implementation efforts meet with obstacles (Boswell et al. 2015; Goldman and Seybolt 2015; Hall et al. 2014; Ionita and Fitzpatrick 2014; Lambert and Harmon 2018; Lewis et al. 2019; Sharples et al. 2017; Wampold 2015). Additionally, a recent Cochrane Review called for more research to be able to support the use of ROM (Kendrick et al. 2016). This implies that more knowledge regarding implementation factors that play a role in the quality and use of ROM are both important and needed.

The attitudes and willingness of health care providers to use ROM have been suggested as an important explanatory factor for the low ROM implementation rate, but little research has been done on this topic (as also discussed by, for instance, Kaiser et al. 2018; Norman et al. 2014). Several behavioral change theories, including the theory of planned behavior (Ajzen 1991) and the diffusion of innovation theory (Rogers 2003), have suggested that attitudes are an important precursor of behavior. A review by Eccles et al. (2006) also found a positive relationship between self-reported intentions and subsequent behavior among health professionals (Eccles et al. 2006), supporting the relevance of studying provider attitudes in relation to implementation mechanisms including the potential mediational role of interventions (Moullin et al. 2018). As clinicians are the end-users of ROM, their attitudes, motivation, and intentions to use ROM are important to consider, as these may predict both the actual use of the feedback, quality of treatment and client progress. The existing literature on provider attitudes and views towards ROM suggests that providers report both benefits and concerns regarding its use (e.g., Boswell et al. 2015; Edbrooke-Childs et al. 2016; Gleacher et al. 2016; Hall et al. 2014; Hatfield and Ogles 2004; Ionita et al. 2016; Miller et al. 2015; Norman et al. 2014; Overington et al. 2015; Sharples et al. 2017; Stasiak et al. 2013; Wolpert et al. 2016). Additionally, not all clinicians use the feedback from ROM even when it is provided to them (Garland et al. 2003; Ross et al. 2016). Given the complexities with regard to both provider attitudes and the multilevel challenges associated with implementation (Aarons et al. 2011), there is a need for a better understanding of the range of clinicians’ attitudes and perceptions of ROM and how such attitudes might relate to the implementation and use of ROM in clinical settings.

Barriers to ROM have been suggested to stem from both practical (e.g., financial burden, time, administration, training, turnover) and philosophical (e.g., clinical utility, relevance, professional concern) levels (Hatfield and Ogles 2004; Boswell et al. 2015). A study by Sharples et al. (2017) using semistructured interviews and focusing on clinicians’ attitudes, facilitators and barriers to implementing ROM identified training, practical experience, and ongoing support as crucial facilitators of the use of ROM at both the individual clinic level and the individual clinical session level. The same study also highlighted the balance between the mandatory use and a consistent use of ROM, showing that clinicians reported struggling with standardized use of ROM in sessions when it was not seen as appropriate. Another study using semistructured interviews identified that providers thought ROM could help in the monitoring of, reflection on and evaluation of progress, while also perceiving disadvantages mainly concerning time and effort, concerns about how information would be used, and fears about therapists being evaluated (Norman et al. 2014). An online survey study among Austrian psychotherapists (Kaiser et al. 2018) found that prior knowledge and experience with monitoring was associated with more positive attitudes, while concerns included its administration and increased work burden. A reluctance to use ROM has also been suggested to stem from clinicians placing more value on their own clinical judgment rather than on the information that is provided by ROM (Hall et al. 2014; Hatfield and Ogles 2004), views that ROM can depersonalize and objectify themes that are essentially subjective (Kaiser et al. 2018; Norman et al. 2014; Wolpert et al. 2016), different needs of different stakeholders such as clinicians and administrators (Boswell et al. 2015), fears that ROM might interfere with forming a therapeutic alliance (Youn et al. 2012) and practical concerns regarding administration and efforts required for data collection or information technology systems (Bickman et al. 2016a, b; Boyce et al. 2014; Gleacher et al. 2016). Reported facilitators include training and organizational support (Edbrooke-Childs et al. 2016; Gleacher et al. 2016; Overington et al. 2015; Persons et al. 2016; Sharples et al. 2017), the role of leadership (Gleacher et al. 2016; Lambert and Harmon 2018) and prior experience with outcome monitoring (Kaiser et al. 2018). While Kaiser et al. (2018) found no significant association between monitoring attitudes and the demographic variables age or years of experience, other research relevant to implementation science suggest differences with regard to both attitudes to adopt new evidence-based interventions in general and attitudes to ROM and demographic variables (e.g., Aarons et al. 2012; Okamura et al. 2018). For instance, Hatfield and Ogles (2004) found earlier career clinicians to be more likely to use outcome measures while Okamura et al. (2018) found that younger therapist had more positive attitudes in attitudinal domains related to organizational support to using an intervention. An ultimate goal of this line of efforts to describe provider attitudes in relation to the implementation of ROM is to increase our understanding of factors likely to facilitate the successful sustained use of ROM, with the potential to increase the quality of mental health care services and better patient outcomes. Of importance here, this work can inform the development and/or tailoring of effective implementation strategies. However, theory and strategy development rely on a proper measurement and conceptualization of important implementation constructs and factors that may predict implementation outcomes, including therapist attitudes towards the adoption of new interventions (Wisdom et al. 2014; Lewis et al. 2015; Martinez et al. 2014; Chaudoir et al. 2013; Chor et al. 2015; Proctor et al. 2011; Glasgow and Riley 2013). Several reviews have raised concerns that many implementation measurement constructs exhibit weak psychometric properties (Chaudoir et al. 2013; Chor et al. 2015; Lewis et al. 2015). This can cast doubt on study findings and ultimately the development and testing of implementation strategies that facilitate the uptake and sustained use of both ROM and other empirically supported interventions. The identification and development of instruments with sound psychometric properties has thus become a high priority in the field of implementation science (Lewis et al. 2015). One of the instruments that was highlighted as psychometrically sound in the preliminary results from the Society for Implementation Research Collaboration (SIRC) Instrument Review Project (Lewis et al. 2015) was the Evidence-based Practice Attitude Scale (EBPAS; Aarons 2004). The current article employs the EBPAS in its extended 50-item version (Aarons et al. 2012) as a template for developing a measure of attitudes to ROM. The EBPAS-50 assesses a broad range of mental health and social service providers’ attitudes towards adopting evidence-based interventions (e.g., perceived limitations or burdens associated with the adoption of new practices and the appeal of new practices). The EBPAS-50 was developed based on theories of attitudes, dissemination and implementation in mental health, and consultations with mental health service providers and researchers (Aarons et al. 2012).

Given the need for brief, pragmatic and psychometrically strong instruments for measuring provider attitudes in relation to specific practices/interventions (e.g., ROM) instead of interventions more generally (Moullin et al. 2018), the present study aimed to adapt the previously validated EBPAS-50 to focus specifically on attitudes towards the adoption of ROM. Furthermore, we aimed to explore how the ROM attitudinal domains were associated with current use of standardized instruments as a means of treatment evaluation (e.g. treatment planning, ongoing evaluation during the course of treatment and assessment of treatment effects), which can be seen as some of the central elements of ROM. As a secondary aim, associations of demographic variables with EBPAS–ROM scale scores were investigated. As adapted from well-known and prior validated instruments for assessing provider attitudes (Aarons 2004; Aarons et al. 2012; Rye et al. 2017), we expected the adapted EBPAS–ROM instrument to show good psychometric properties and to have a broad scope while simultaneously being relatively short and pragmatic and having concurrent validity.

Methods

Measures and Assessment

Conceptualization

We specified the instructions of the rephrased EBPAS-50 instrument measuring attitudes towards ROM as; “The following questions concerns your attitudes towards systematically using routine outcome measures to get feedback on patient’s problems and changes throughout the course of treatment. Routine outcome measures refer to standardized instruments assessing mental health status, in which health personnel or patients report their current status regarding common mental health issues. The instruments can be administered either on paper or through web- or software support systems”.

Demographic

The demographic variables included gender, age, highest level of education, profession, number of years worked in substance abuse and/or mental health service and whether the respondent was currently working as a clinician (yes/no).

Current Use of Standardized Instruments

Clinicians’ current use of standardized instruments as the means of treatment planning and evaluation included the following questions: “How often do you use standardized tests and measurements when planning your clinical work?”, “How often do you use standardized questionnaires as part of monitoring treatment responses?” and “How often do you use standardized questionnaires as part of evaluating treatment effects?” Responses were provided on a 5-point Likert scale ranging from “very seldom/never” to “very often/always”.

Attitudes Towards ROM

Attitudes towards ROM were measured with a rephrased version of the Evidence-based Practice Attitude Scale-50 (EBPAS-50) (Aarons 2004; Aarons et al. 2012), adapted and translated to Norwegian for the present study. The original EBPAS-50 assesses mental health and social service provider attitudes towards adopting evidence-based practices (Aarons 2004; Aarons et al. 2012). For the present study, the questions were edited and framed to ask about attitudes towards adopting ROM. The 50 items are grouped into 12 subscales (the names of the subscales are here provided in italics): (1) the likelihood of adopting ROM based on requirements to do so by a supervisor, agency or state, (2) the intuitive appeal of adopting ROM, (3) openness to new practices, (4) the perceived divergence of providers’ usual practice from research-based or academically developed interventions, (5) limitations of outcome measures and their inability to address client needs, (6) ROM fit with the values and needs of both the client and clinician, (7) negative perceptions of monitoring, (8) balance between perceptions of clinical skills and science, (9) the time and administrative burden of learning ROM, (10) job security related to using and learning ROM, (11) perceived organizational support for adoption of ROM, and (12) positive perceptions of receiving feedback. Responses to each item are provided on a 5-point Likert scale (from 0 indicating “not at all” to 4 indicating “to a very great extent”). To reduce response biases, 15 items belonging to 5 subscales (divergence, limitations, monitoring, balance and burden) are negatively framed and are reverse-scored before computing the total score. The total score represents a respondent’s global attitude towards adopting ROM, with a higher total score indicating a more positive global attitude.

Translation Procedure

The Norwegian translation of the EBPAS-50 was conducted by the first author (MR) and then back-translated by a professional (Rye et al. 2017). The translational adaption to a focus on ROM was conducted parallel to the translation of the EBPAS-50 Norwegian version. The procedure included several consensus discussions between authors MR and IS, and between MR and the original EBPAS-50 author (GAA). It was also piloted in a sample of clinicians and students from both clinical and Ph.D. programme in psychology, who provided comments regarding readability prior to finalization, as described in Rye et al. (2017).

Procedure and Sample

Representing two of the main types of professionals in Norwegian mental health care settings (Statistics Norway 2019), psychologists and nurses were recruited by invitation emails distributed by the Norwegian Psychological Association to half of their members (Sample 1, n  = 3654) and by the Norwegian Nurses Organization’s suborganization for nurses in mental health and substance abuse, to all of their members (Sample 2, n = 1436). In addition, the survey was announced on the internet sites of these organizations. The invitation email contained information about the study, as well as a web link providing access to the survey. Completion of the survey was accepted as a consent to participate in the survey. SurveyMonkey online software was used to collect data from May to July 2014 for Sample 1, and from February to March 2015 for Sample 2. One and two reminders were sent to Samples 2 and 1, respectively. The two samples had the opportunity to participate in random drawings for one iPad mini and two psychology or nursing handbooks, respectively, as incentives for participation.

A total of 734 psychologists and psychology students (a 20.1% response rate for Sample 1) and 360 nurses (a 25.1% response rate) for Sample 2) completed the survey (N = 1094). The survey included an adapted version of the Evidence-based Practice Attitude Scale-50 (EBPAS-50) where all items were rephrased to ask about attitudes towards ROM. Subjects not completing any of the 50 items from the EBPAS-50–ROM version were excluded, as were those with missing data for whole subscales, > 1 item on a 3-item scale or > 2 items on a 4-item scale (n = 300). Thus, the final sample included data from 794 respondents. Students and providers not working as clinicians were excluded from the current analyses, given our focus on practitioners as end-users in clinical service settings (n = 662).

Statistical Analyses

Confirmatory factor analyses (CFAs) for item reduction evaluations were conducted in Mplus v8.0. The model specification was based on the original subscales of the EBPAS-50. The subscales rephrased to ask directly about attitudes towards ROM were retained (requirements, appeal, limitations, fit, burden, job security, organizational support). Three subscales that were not edited to focus on ROM for the present survey were excluded from further analysis, as their item content was considered outside the scope of the present study. These included the openness subscale, which asks about attitudes towards the adoption of research-based therapies or interventions more generally; the divergence subscale, with a focus on manualized therapies and academically developed interventions in relation to one’s usual practice; and the balance subscale, with a focus on clinical experience and competence in relation to science. Based on their perceived relevance for the implementation of ROM, the monitoring and feedback subscales were retained, although the wording did not ask directly about ROM.

The Norwegian sample was divided using the first sample to identify a short ROM version (n = 333) and the second sample to validate its factor structure (n = 329). The parameters were estimated with the full information maximum likelihood procedure (FIML). Robust standard errors (MLR) were requested in order to accommodate nonnormal item distributions. To assess model fit, the following indices were used: χ², root mean square error of approximation (RMSEA), standardized root mean error (SRMR) and comparative fit indices (CFIs). In accordance with Hu and Bentler’s cutoff recommendations (1999), RMSEA values close to .06, SRMR values close to .08 and CFI values close to 0.95 indicate acceptable model fit. Building upon our previous work developing the shortened EBPAS-36 (Rye et al. 2017), subscales containing four or more items were shortened, with items being retained based on the combined evaluation of the following criteria: (1) items with the highest factor loadings, (2) evaluations of modification indices, (3) items that were conceptually similar or added unique information.

Correlational analysis and regression analysis were conducted in SPSS v25. To allow analysis of a complete dataset, missing EBPAS-50–ROM version items were imputed using the expectation maximization (EM) method. Values were imputed separately for sets of items belonging to each subscale, following the exclusion of missing data as described in the “Procedure and Sample” section. Bivariate associations were calculated as Pearson’s correlation coefficients.

Regression models were built to examine the attitudinal domains’ predictive value for clinicians’ reported use of standardized instruments as a means of treatment planning and evaluation. Data were checked for influential cases according to Cook’s distance criteria, with no values with a Cook’s distance greater than 1. The regression models were adjusted for the demographic variables gender, age and years of experience. In the first model, the regression analysis was run separately for each subscale adjusted only for the demographic variables. In the second model, all subscales were entered together with the demographic variables to assess the predictive value of each subscale when also adjusted for the other subscales. In the third model, the total scale score representing global attitudes towards adopting ROM was entered, together with the demographic variables.

Results

Samples

Descriptive data of the two samples are provided in Table 1. The majority of participants were women. The nurses were older than the psychologists, and had more years of clinical experience.

Table 1 Demographic characteristics

Acceptability

Among the 1047 respondents, 73% had complete data on all of the EBPAS-50–ROM items. Of respondents who provided answers to the EBPAS-50–ROM items, no single item was left unanswered by more than 2.7% of the respondents, with items 42 and 44 being those omitted most frequently.

Item Reduction

Based on the reduction criteria, one item in each of the four-item subscales (appeal, monitoring, and burden) and four items in each of the seven-item subscales (limitations and fit) were removed. The final EBPAS–ROM version containing 27 items is presented in Table 2.

Table 2 EBPAS–ROM item standardized factor loadings, means and standard deviations

Item 9 from the appeal subscale (“intuitively appealing”) was removed due to having the lowest factor loading and content overlap with the other items in the subscale. In the limitations subscale, the two lowest loading items were removed. In addition, item 24 (“develop a strong working alliance”) was removed due to content overlap with item 23 (“truly connecting with your clients”), while item 27 (“families with multiple problems”) was removed due to content overlap with item 26 (“clients with multiple problems”). In the fit subscale, one item was removed due to having the lowest factor loading.

Item 19 (“had a say in which outcome measure”) was removed despite having a high factor loading due to content overlap and less universality than item 20 (“had a say in how to use”). Item 22 (“Fit with your treatment philosophy”) was removed due to content overlap and a lower factor loading than item 21 (“Fit with your clinical approach”). Special attention was paid to item 16 (“clients wanted it”) and item 17 (“knew more about how your client liked it”), as their content is in line with the underlying construct of the scale, but the items had quite similar factor loadings and content. A consensus was reached to retain item 16, as the item was evaluated to be more readable than item 17. Item 33 from the monitoring subscale (“I do not need to be monitored”) was removed despite having a high factor loading due to content overlap and not referring to one’s work situation as the other items. Finally, item 39 from the burden subscale (“Can’t meet other obligations”) was removed due to having the lowest factor loading. The instrument and its scoring instructions can be found in Additional files 1–3 (validated in the Norwegian language and including English wording).

Subscale Correlations and Internal Consistency

The correlation coefficients (Pearson’s r) between the nine EBPAS–ROM subscales are presented in Table 3. The highest correlations were between the appeal and fit subscales (r = .58), the job security and organizational support subscales (r = .43), the limitations and burden subscales (r = .40) and the appeal and organizational support subscales (r = .39), all in the expected directions. The internal consistency of the EBPAS–ROM is presented in Table 3. The total scale Cronbachʼs alpha (α) was good (.85), and subscales α ranged from adequate to excellent (.70 to .93).

Table 3 EBPAS–ROM subscale intercorrelations and internal consistency

Confirmatory Factor Analyses

CFAs modeling the subscale structure of the EBPAS–ROM was performed. The absolute model fit was significant (χ² = 552.52, p < .001). However, the model fit was adequate in terms of low misspecification (RMSEA = .053, 90% C.I. [0.046, 0.059]; SRMR = .06) and good in terms of incremental fit (CFI = .93, TLI = .92). The standardized factor loadings ranged from .42 to .98 and were statistically significant (all p-levels < .001).

Concurrent Validity of the EBPAS–ROM

Regression analyses with the reported utilization of standardized outcome measures as dependent variables [use of standardized instruments for (1) treatment planning, (2) evaluation of ongoing therapy, and (3) evaluation of effect of therapy] were performed to investigate the concurrent validity of the EBPAS–ROM. The results of the regression analyses are presented in Table 4. The results from model 1, where each EBPAS–ROM subscale was entered separately with the demographic variables, showed that five of the subscales (requirements, appeal, limitations, organizational support and feedback) significantly predicted all of the dependent variables when adjusted for demographic variables. Here, higher scores on the requirement, appeal, organizational support and feedback subscales predicted greater use of standardized instruments, while a higher score on the limitations subscale predicted less use of standardized instruments as a means of treatment planning, the evaluation of ongoing therapy and evaluation of therapy effects. In addition, the monitoring subscale predicted the use of standardized instruments for treatment planning and the evaluation of effects, with a higher score predicting less use for the stated purposes. Furthermore, a higher score on the job security subscale predicted greater use of standardized instruments for the evaluation of ongoing therapy and the therapy effects. Finally, in the first model, a higher score on the burden subscale predicted less use of standardized instruments for evaluation of treatment effects.

Table 4 Regression with EBPAS–ROM predicting use of standardized instruments

The results from model 2, where all EBPAS–ROM subscales and the demographic variables were entered simultaneously, showed that the limitations subscale predicted the use of standardized instruments of the stated purposes when adjusted for the other subscales and the demographic variables. A higher limitations score predicted less use of standardized instruments for treatment planning, the evaluation of ongoing therapy and the evaluation of treatment effects. In addition, a higher score on the organizational support subscale predicted greater use of standardized instruments as the means of evaluation of ongoing therapy and the evaluation of treatment effect when adjusted for the other subscales and demographic variables, while a higher score on the job security subscale predicted greater use of standardized instruments for the evaluation of treatment effect. Finally, in model 2, a higher score on the fit subscale predicted less use of standardized instruments for treatment planning, while a higher score on the burden subscale predicted greater use for the purpose of treatment planning.

In model 3, a higher score on the total EBPAS–ROM scale representing respondent global attitudes towards the adoption of ROM significantly predicted all dependent variables (the use of standardized instruments for both treatment planning, evaluation of ongoing therapy and evaluation of effect of therapy) when adjusted for the demographic variables.

Regarding the demographic predictors, gender, age and years of experience significantly predicted treatment planning in model 1. Females reported greater use than males of standardized instruments for treatment planning, older clinicians reported less use than younger clinicians, and more experienced clinicians reported less use than less experienced clinicians. In models 2 and 3, only years of experience significantly predicted treatment planning, with more experienced clinicians reporting less use of standardized instruments than less experienced clinicians.

Discussion

The slow implementation rate of ROM implies that more knowledge is needed of important implementation factors, including mental health care providers’ attitudes towards adopting ROM and how these attitudes are to be measured. This article presents the EBPAS–ROM instrument for measuring provider attitudes towards the use of ROM, adapted from the well-known and previously validated EBPAS-50 and -36 instruments assessing providers’ attitudes towards the adoption of new practices (Aarons 2004; Aarons et al. 2012; Rye et al. 2017). We also investigated associations among ROM attitudinal domains and clinicians’ reported use of standardized instruments for treatment planning and evaluation. The EBPAS–ROM measures 9 domains corresponding to 9 out of the 12 original subscales from the EBPAS-50 and -36. The internal consistency of the subscales was adequate to excellent. Compared with the EBPAS-50 subscales, most subscales of the EBPAS–ROM had slightly lower internal consistencies, which can occur when the number of items per subscale is reduced (Rye et al. 2017). Taken together, the instrument is still broad in scope with a potential lower burden of administration. Furthermore, it is linked to both theory and practice, as it is adapted from well-known instruments developed from implementation theories and consultations with mental health care providers (Aarons 2004; Aarons et al. 2012; Rye et al. 2017), and it is psychometrically strong, which are all important aspects of a pragmatic measure (see Glasgow and Riley 2013; Lewis et al. 2015; Rye et al. 2017). Concurrent validity was supported as all of the EBPAS–ROM subscales, as well as the total scale, were associated with respondents’ reported use of standardized instruments for treatment planning and evaluation, either independently after adjusting for demographic variables (models 1 and 3) or when adjusted for the other subscales (model 2). More positive global attitudes towards the adoption of ROM predicted greater use of standardized instruments for all of the stated purposes, implying that global attitudes of clinicians are an important factor to take into account when planning the implementation of ROM.

Specifically, the perceived limitations of the use of ROM related to it being too narrowly focused, not being suitable for patients with multiple problems and hindering the connection between therapist and patient, predicted less reported use of standardized instruments for treatment planning, the evaluation of ongoing treatment and the evaluation of effects. Next, the experience of more organizational support, such as training, ongoing support and educational credits, predicted both greater reported use of standardized instruments for the evaluation of ongoing treatment and, marginally, the evaluation of treatment effects. Taken together, these findings concerning the perceived limitations of ROM and the value of training and organizational support can be a seen as reflecting the distinct needs of providers, representing, on one hand, clinical utility and professional concern and, on the other hand, the more administrative and practical needs. This distinction parallels the division between practical and philosophical attitudinal barriers to ROM (Hatfield and Ogles 2004; Boswell et al. 2015). We consider this distinction important for implementation and educational efforts since, as Boyce et al. (2014) also discusses, professional concerns may endure even after the successful provision of implementation strategies such as proper practical training and administrative support. Consequently, the sustained use of ROM in clinical practice settings might be hindered.

The value of training and ongoing organizational support during an implementation process are recognized as important facilitators of the implementation of ROM. Even brief training has been found to improve provider attitudes towards the use of ROM, especially when practical training in one’s daily work setting is provided (Edbrooke-Childs et al. 2016). However, training issues are not limited to implementation settings. Wampold (2015) highlights that the use of ROM also has the potential to be used together with various skill assessments, thus allowing training to be directed towards therapist areas in need of development. This also implies the potential for ROM to be more generally utilized for therapeutic, provider, and organizational growth. For instance, early career training approaches that include ROM maybe used in improving educational training settings as well as the clinical practice and supervision of students and novices. Explicit ROM-training during students’ education and clinical practice should be evaluated as a strategy, as it may facilitate future clinicians’ competent use of ROM as well as therapeutic competence overall (Overington et al. 2015).

Furthermore, valuing job security predicted greater use of standardized instruments for the evaluation of treatment effects, perhaps indicating that this knowledge is considered a mark of important competence and quality of therapeutic work. Finally, both the fit and the burden subscale predicted the reported use of standardized instruments, but in unexpected directions. Here, addressing the fit with ROM and one’s clinical approach and knowing that one’s clients wanted to use ROM, co-occurred with less use of instruments for treatment planning,—while more concern about paperwork, administrative burden and having little time to learn new tasks predicted greater use of standardized instruments for treatment planning. While it is beyond the scope of this article to be able to describe the reasons for these findings, it is possible that the use of standardized instruments as a means of planning ones clinical work is considered helpful with complex clinical cases or in a busy, highly scheduled work environment. This explanation is consistent with Sharples et al. (2017), describing ROM as a potential valuable tool in busy clinics where time for reflection is limited. The explanation could also be the other way around, that the finding is related to the use of outcome measures for some people being associated with the experience of lack of time and administrative burden (see for instance Boswell et al. 2015; Norman et al. 2014). Furthermore, Gleacher et al. (2016) found that a clinic with higher implementation success actually reported a higher ratio of barriers to facilitators compared to a clinic with less successful implementation. This finding could reflect an increased work effort associated with integrating ROM as a new practice and the experience of “implementation fatigue” when new work requirements are introduced in a short period of time (Chung et al. 2017; Gleacher et al. 2016). To alleviate this and support successful adoption of ROM, leadership that is sensitive to the extended work demands and that makes efforts to relieve anticipated barriers on behalf of the providers (e.g., through, for instance, engagement, technical support and appointing local champions) should be provided (Gleacher et al. 2016; Lambert and Harmon 2018). This reflects a need for organizations to foster positive leadership approaches to develop organizational environments that are inclusive and motivational in the process of adopting new practices, while simultaneously adjusting to providers’ total work situation and the presence or absence of their professional concerns (Rye et al. 2019). For the fit subscale, potential causal mechanisms might be subtle, but one conceivable understanding might have to do with the feeling of increased autonomy or individual decisions about the “intention to use” (Moullin et al. 2018), leading clinicians not to use such instruments in cases where it is not perceived to be appropriate.

Although the demographic predictors were not of our primary interest, it is worth noting that both clinicians with more years of clinical experience and older respondents reported less use of standardized instruments for treatment planning. Also, in model 1, females reported more use of standardized instruments for treatment planning than males. Although not directly comparable, these findings are interesting compared to other findings showing younger or earlier career clinicians to report more use of ROM than older clinicians (Hatfield and Ogles 2004), while Kaiser et al. (2018) reported no significant association between monitoring attitudes and both gender and years of experience. However, in the original EBPAS development study, more positive EBP attitude scale scores were found for intern compared to permanent staff (Aarons 2004). More research on the therapist demographic variables for implementation of ROM is considered important for future research.

Limitations and Future Directions

Several limitations should be considered. First, three subscales from the original EBPAS-50 and -36 were left out of the analysis, as they were not rephrased to focus on ROM for the survey study and were perceived to be beyond the scope of the present study. In omitting these subscales, one might argue that important aspects relevant to the adoption of ROM were lost. Future studies should examine how these other subscales might also be tailored for ROM. It is also possible that not employing these scales has had effects with regard to the original factor structure of the instrument. While the present study adhered to the original subscales from the EBPAS-50 and -36, future research should continue to explore instrument factor structure, validity and practicality, adhering to the need for psychometrically strong and pragmatic instruments in implementation science. Furthermore, using the instrument in real world implementation initiatives and with several time points might provide more knowledge regarding the predictive validity of the instrument, and allow investigation of the impact of various factors on the success or failure of implementation, as well as the links between attitudes and actual behavior. A second limitation concerns the low response rate, which is a usual problem in web-based surveys (see Kaiser et al. 2018; Van Horn et al. 2009). The surveys were sent out to all members of the participating organizations, as it was difficult to filter out for instance different work site categories or people who were retired. This limitation might have contributed to the low response rate, as some organization members did not consider the surveys relevant to their work setting. Another limitations concerning the present study include that multiple testing might have increased the risk of type 1 errors. Also, the multitude and complexities of factors involved are underscored by the explained variances between 4 and 16%, indicating that many other factors in addition to those we have studied also contribute to the clinicians’ reported use of standardized instruments. Furthermore, the concept of ROM includes many components in addition to having patients fill out a form on a regular basis. Other components include the feedback of patient status and progress back to the therapist and the interpretation of simple scores and subscales (Lambert and Harmon 2018; Wampold 2015). Respondents putting weight on different components of ROM or having different understandings of what ROM entails may have affected how participants responded. Rather than exploring ROM as a unified construct, future research efforts might benefit from a proper operationalization of core components of the processes of assessing treatment outcomes (Lewis et al. 2019) and thus how attitudes relate to these components, for instance to increase our understanding of tailored implementation strategies.

Conclusion

The EBPAS–ROM has good psychometric properties. It lives up to the intentions of its predecessors, the EBPAS-50 and -36, to be a short and pragmatic instrument that can be used for applied and research purposes for both understanding implementation factors and planning implementation strategies. This adaptation of a prior scale is consistent with the development of new implementation measures that can be tailored to focus on specific practices that are being implemented in a given health setting (Moullin et al. 2018). The present study provides insight into how specific attitudes towards ROM represented by different subscales differentially co-occurred with different aspects of the reported use of standardized instruments. This study also demonstrated that having more positive global attitudes towards the adoption of ROM significantly predicts greater use of standardized instruments for treatment planning and evaluation. These findings suggest that the attitudes of clinicians can be an important factor to consider and address when planning the implementation of ROM.