When the Right Measure Doesn’t Exist: A Novel Motivational Interviewing Tool for Community Programs

Abstract

This report describes the reliability and validity of the Motivational Interviewing Evaluation Rubric (MIER). The MIER is a novel MI evaluation tool for use in supervision with community-based providers. In the MIER, there are 15 items targeting the following sub-dimensions: Spirit (four items), Process (six items), and Skills (five items). Two raters engaged in 10 h of training and practice on the use of the MIER. Data were session audio-files from youth justice program providers. Inter-rater reliability results showed good performance for the measure. Specifically, 2-way mixed Intraclass Coefficients were excellent, ranging from 0.81 to 0.92 for sub-dimension scores, and Kappa ratings per item were fair to substantial (k =  0.35 to 0.87). Internal consistency reliability for Spirit, Process, and Skills scores ranged from a = .72 to 0.78. In concurrent validity analyses, MIER sub-dimension scores were positively and moderately correlated with relevant Motivational Interviewing Treatment Integrity (4.2) criterion scores (rs = .45 to 0.67; p < .005). The MIER is a promising observational measure for guiding supervision in community youth programs.

This is a preview of subscription content, access via your institution.

Fig. 1

References

  1. Cicchetti, V. (1994). Guidelines, criteria and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290.

    Article  Google Scholar 

  2. Cohen, J. (1968). Weighed kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220.

    Article  Google Scholar 

  3. Cronbach, L. J. (1970). Essentials of psychological testing (3rd ed.). New York: Harper & Row.

    Google Scholar 

  4. Cushing, C. C., Jensen, C. D., Miller, M. B., & Leffingwell, T. R. (2014). Meta-analysis of motivational interviewing for adolescent health behavior: Efficacy beyond substance use. Journal of Consulting and Clinical Psychology, 82, 1212–1218.

    Article  Google Scholar 

  5. Fu, S. S., Roth, C., Battaglia, C. T., Nelson, D. B., Farmer, M. M., Do, T., … Zillich, A. J. (2015). Training primary care clinicians in motivational interviewing: a comparison of two models. Patient Education and Counseling, 98, 61–68.

    Article  Google Scholar 

  6. Landis, J. R., & Koch, G. G. (1977). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 33, 363–374.

    Article  Google Scholar 

  7. Leeman, J., Calancie, L., Hartman, M. A., Escoffery, C. T., Herrmann, A. K., Tague, L. E., … Samuel-Hodge, C. (2015). What strategies are used to build practitioners’ capacity to implement community-based interventions and are they effective? A systematic review. Implementation Science, 10, 80.

    Article  Google Scholar 

  8. Lundahl, B., & Burke, B. L. (2009). The effectiveness and applicability of motivational interviewing: A practice-friendly review of four meta-analyses. Journal of Clinical Psychology, 65, 1232–1245.

    Article  Google Scholar 

  9. Madson, M. B., & Campbell, T. C. (2006). Measures of fidelity in motivational enhancement: A systematic review. Journal of Substance Abuse Treatment, 31, 67–73.

    Article  Google Scholar 

  10. Madson, M. B., Campbell, T. C., Barrett, D. E., Brondino, M. J., & Melchert, T. P. (2005). Development of the motivational interviewing supervision and training scale. Psychology of Addictive Behaviors, 19, 303–310.

    Article  Google Scholar 

  11. Madson, M. B., Loignon, A. C., & Lane, C. (2009). Training in motivational interviewing: A systematic review. Journal of Substance Abuse Treatment, 36, 101–109.

    Article  Google Scholar 

  12. Madson, M. B., Shumacker, J. A., Baer, J. S., & Martino, S. (2016). Motivational interviewing for substance use: Mapping out the next generation of research. Journal of Substance Abuse Treatment, 65, 1–5.

    Article  Google Scholar 

  13. Manuel, J. K., Moyers, T. B., & Houck, J. M. (2011). Motivational interviewing: A review of coding systems. Poster session presented at the Motivational Interviewing Network of Trainers (MINT), Fort Wayne, IN.

  14. Martino, S., Ball, S. A., Nich, C., Frankforter, T. L., & Carroll, K. M. (2008). Community program therapist adherence and competence in motivational enhancement therapy. Drug and Alcohol Dependence, 96(1–2), 37–48.

    Article  Google Scholar 

  15. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46.

    Article  Google Scholar 

  16. Miller, W. R., & Rollnick, S. (2013). Motivational interviewing: Helping people change (3rd ed.). New York: Guilford Press.

    Google Scholar 

  17. Miller, W. R., Yahne, C. E., Moyers, T. B., Martinez, J., & Pirritano, M. (2004). A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting and Clinical Psychology, 72, 1050–1062.

    Article  Google Scholar 

  18. Moyers, T. B., Manuel, J. K., & Ernst, D. (2014). Motivational interviewing treatment integrity coding manual 4.1. Unpublished Manual.

  19. Moyers, T., Martin, T., Catley, D., Harris, K. J., & Ahluwalia, J. S. (2003). Assessing the integrity of motivational interviewing interventions: Reliability of the motivational interviewing skills code. Behavioural and Cognitive Psychotherapy, 31, 177–184.

    Article  Google Scholar 

  20. Moyers, T. B., Martin, T., Manuel, J. K., Hendrickson, S. L., & Miller, W. R. (2005). Assessing competence in the use of motivational interviewing. Journal of Substance Abuse Treatment, 28, 19–26.

    Article  Google Scholar 

  21. Moyers, T. B., Martin, T., Manuel, J. K., Miller, W. R., & Ernst, D. (2010). Revised global scales: Motivational interviewing treatment integrity 3.1.1 (MITI 3.1.1) [Unpublished manuscript]. Center on Alcoholism, Substance Abuse and Addictions (CASAA), University of New Mexico.

  22. Moyers, T. B., Rowell, L. N., Manuel, J. K., Ernst, D., & Houck, J. M. (2016). The Motivational Interviewing Treatment Integrity Code (MITI 4): Rationale, preliminary reliability and validity. Journal of Substance Abuse Treatment, 65, 36–42.

    Article  Google Scholar 

  23. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.

    Google Scholar 

  24. Nuro, K. F., Maccarelli, L., Martino, S., Ball, S. A., Baker, S. M., Rounsaville, B. J., & Carroll, K. M. (2005). Yale adherence and competence scale (YACSII) guidelines. West Haven, CT: Yale University.

    Google Scholar 

  25. Pecukonis, E., Greeno, E., Hodorowicz, M., Park, H., Ting, L., Moyers, T., … Wirt, C. (2016). Teaching motivational interviewing to child welfare social work students using live supervision and standardized clients: A randomized controlled trial. Journal of the Society for Social Work and Research, 7(3), 479–505.

    Article  Google Scholar 

  26. Stanhope, V., Tennille, J., Bohrman, C., & Hamovitch, E. (2016). Motivational interviewing: Creating a leadership role for social work in the era of healthcare reform. Social Work in Public Health, 31, 474–480.

    Article  Google Scholar 

Download references

Acknowledgements

We would like to specifically acknowledge Elizabeth Zarate and Jada Nyankson for their heartfelt work on this project, along with the staff at Southwest Key Programs Milwaukee. We would also like to acknowledge those who supported the initial work, Hillary England and DeAna Swan, and to Cassandra Harper and Justin Walthers who helped with project management and assisted with data collection, respectively. Lastly, thanks to Matthea Marquart for introducing Dr. Magill to our community-based work.

Funding

Dr. Magill was paid a consultation fee by Southwest Key Programs Inc.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Johanna Creswell Báez.

Ethics declarations

Conflicts of interest

Dr. Baez was employed at Southwest Key Programs Inc. for part of this research project and Rachel Galanis was a Clinical Director at Southwest Key Programs Inc. during this research project.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Báez, J.C., Galanis, R. & Magill, M. When the Right Measure Doesn’t Exist: A Novel Motivational Interviewing Tool for Community Programs. Child Adolesc Soc Work J 37, 195–205 (2020). https://doi.org/10.1007/s10560-020-00650-y

Download citation

Keywords

  • Fidelity monitoring
  • Community based programs
  • Implementation science
  • Motivational Interviewing
  • Psychometric