Skip to main content

Advertisement

SpringerLink
Log in

Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention

  • Original Paper
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Careful fidelity monitoring and feedback are critical to implementing effective interventions. A wide range of procedures exist to assess fidelity; most are derived from observational assessments (Schoenwald and Garland, Psycholog Assess 25:146–156, 2013). However, these fidelity measures are resource intensive for research teams in efficacy/effectiveness trials, and are often unattainable or unmanageable for the host organization to rate when the program is implemented on a large scale. We present a first step towards automated processing of linguistic patterns in fidelity monitoring of a behavioral intervention using an innovative mixed methods approach to fidelity assessment that uses rule-based, computational linguistics to overcome major resource burdens. Data come from an effectiveness trial of the Familias Unidas intervention, an evidence-based, family-centered preventive intervention found to be efficacious in reducing conduct problems, substance use and HIV sexual risk behaviors among Hispanic youth. This computational approach focuses on “joining,” which measures the quality of the working alliance of the facilitator with the family. Quantitative assessments of reliability are provided. Kappa scores between a human rater and a machine rater for the new method for measuring joining reached 0.83. Early findings suggest that this approach can reduce the high cost of fidelity measurement and the time delay between fidelity assessment and feedback to facilitators; it also has the potential for improving the quality of intervention fidelity ratings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health, 38(1), 4–23. doi:10.1007/s10488-010-0327-7.

    Article  PubMed Central  PubMed  Google Scholar 

  • Alexander, J., Pugh, C., Parsons, B., & Sexton, T. (2000). Family functional therapy. In D. S. Elliot (Ed.), Blueprints for violence prevention: Book three (pp. 445–453). Boulder: Center for the Study and Prevention of Violence.

    Google Scholar 

  • Allen, J. D., Linnan, L. A., & Emmons, K. M. (2012). Fidelity and its relationship to impementation effectiveness, adaptation, and dissemination. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and implementation in health: Translating science to practice (pp. 281–304). New York: Oxford University Press.

    Chapter  Google Scholar 

  • Bahrammirzaee, A. (2010). A comparative survey of artificial intelligence applications in finance: Artificial neural networks, expert system and hybrid intelligent systems. Neural Computing and Applications, 19(8), 1165–1195. doi:10.1007/s00521-010-0362-z.

    Article  Google Scholar 

  • Barber, J. P., Gallop, R., Crits-Christoph, P., Frank, A., Thase, M. E., Weiss, R. D., et al. (2006). The role of therapist adherence, therapist competence, and the alliance in predicting outcome of individual drug counseling: Results from the NIDA collaborative cocaine treatment study. Psychotherapy Research, 16, 240–299.

    Article  Google Scholar 

  • Brown, C. H., Kellam, S. G., Kaupert, S., Muthén, B. O., Wang, W., Muthén, L., et al. (2012). Partnerships for the design, conduct, and analysis of effectiveness, and implementation research: Experiences of the prevention science and methodology group. Administration and Policy in Mental Health, 39(4), 301–316. doi:10.1007/s10488-011-0387-3.

    Article  PubMed Central  PubMed  Google Scholar 

  • Brown, C. H., Mohr, D. C., Gallo, C. G., Mader, C., Palinkas, L. A., Wingood, G., et al. (2013). A computational future for preventing HIV in minority communities: How advanced technology can improve implementation of effective programs. Journal of Acquired Immune Deficiency Syndromes, 63(1), S72–S84.

    Article  PubMed Central  PubMed  Google Scholar 

  • Busch, A. M., Kanter, J. W., Callaghan, G. M., Baruch, D. E., Weeks, C. E., & Berlin, K. S. (2009). A micro-process analysis of functional analytic psychotherapy’s mechanism of change. Behavior Therapy, 40(3), 280–290. doi:10.1016/j.beth.2008.07.003.

    Article  PubMed  Google Scholar 

  • Carletta, J. (1996). Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2), 249–254.

    Google Scholar 

  • Chambers, D. A. (2008). Advancing the science of implementation: A workshop summary. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 3–10. doi:10.1007/s10488-007-0146-7.

    Article  PubMed  Google Scholar 

  • Chambers, D. (2012). The interactive systems framework for dissemination and implementation: Enhancing the opportunity for implementation science. American Journal of Community Psychology, 50(3–4), 282–284. doi:10.1007/s10464-012-9528-4.

    Article  PubMed  Google Scholar 

  • Chinman, M., Hunter, S., Ebener, P., Paddock, S., Stillman, L., Imm, P., et al. (2008). The getting to outcomes demonstration and evaluation: An illustration of the prevention support system. American Journal of Community Psychology, 41(3–4), 206–224. doi:10.1007/s10464-008-9163-2.

    Article  PubMed Central  PubMed  Google Scholar 

  • De Cooman, G., & Zaffalon, M. (2004). Updating beliefs with incomplete observations. Artificial Intelligence, 159(1–2), 75–125.

    Article  Google Scholar 

  • Durlak, J., & DuPre, E. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3), 327–350. doi:10.1007/s10464-008-9165-0.

    Article  PubMed  Google Scholar 

  • Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. (No. FMHI Publication #231). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute.

    Google Scholar 

  • Gallo, C. G., Aist, G., Allen, J. F., de Beaumont, W., Coria, S., Gegg-Harrison, W., & Swift, M. (2007). Continuous understanding in a multimodal dialogue corpus. Proceedings of DECALOG: The 2007 Workshop on the Semantics and Pragmatics of Dialogue, Trento (pp. 75–82).

  • Gallo, C. G., Jaeger, T. F., & Furth, K. (2010). A database for the exploration of Spanish planning. The Seventh International Conference on Language Resources and Evaluation (LREC). Valletta, Malta.

  • Hanson, R., Gros, K., Davidson, T., Barr, S., Cohen, J., Deblinger, E., & Ruggiero, K. (2013). National trainers’ perspectives on challenges to implementation of an empirically-supported mental health treatment. Administration and Policy in Mental Health and Mental Health Services Research, Online First, 1–13. doi:10.1007/s10488-013-0492-6.

  • Henggeler, S. W., & Borduin, C. M. (1990). Family therapy and beyond: A multisystemic approach to treating the behavior problems of children and adolescents. Pacific Grove: Brooks/Cole.

    Google Scholar 

  • Henggeler, S. W., Schoenwald, S. K., Borduin, C. M., Rowland, M. D., & Cunningham, P. B. (1998). Multisystemic treatment of antisocial behavior in children and adolescents. Treatment manuals for practitioners. New York: Guilford Press.

    Google Scholar 

  • Henggeler, S. W., Schoenwald, S. K., Liao, J. G., Letourneau, E. J., & Edwards, D. L. (2002). Transporting efficacious treatments to field settings: The link between supervisory practices and therapist fidelity in MST programs. Journal of Clinical Child and Adolescent Psychology: The Official Journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53, 31(2), 155–167.

    Article  Google Scholar 

  • Herschell, A. D. (2010). Fidelity in the field: Developing infrastructure and fine-tuning measurement. Clinical Psychology: Science and Practice, 17(3), 253–257. doi:10.1111/j.1468-2850.2010.01216.x.

    Google Scholar 

  • Hogue, A., Dauber, S., Chinchilla, P., Fried, A., Henderson, C., Inclan, J., et al. (2008). Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment, 35(2), 137–147.

    Article  PubMed Central  PubMed  Google Scholar 

  • Holmes, J. N., & Holmes, W. J. (2002). Speech synthesis and recognition 2e (HBK) (2nd ed.). New York: Taylor & Francis.

    Google Scholar 

  • Huth, M., & Ryan, M. (2004). Logic in computer science: Modelling and reasoning about systems. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Inoue, M., Ogihara, M., Hanada, R., & Furuyama, N. (2010). Utility of gestural cues in indexing semantic miscommunication. Future Information Technology (FutureTech), 2010 5th International Conference on Future Information Technology (pp. 1–6).

  • Kellam, S. G., Mackenzie, A. C. L., Brown, C. H., Poduska, J. M., Petras, H., & Wilcox, H. C. (2011). The good behavior game and the future of prevention and treatment. Addiction Science and Clinical Practice, 6(1), 73–84.

    PubMed Central  PubMed  Google Scholar 

  • Landsverk, J., Brown, C. H., Chamberlain, P., Palinkas, L., Rolls Reutz, J., & Horwitz, S. M. (2012). Design and analysis in dissemination and implementation research. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and implementation research in health: Translating science to practice (pp. 225–260). London: Oxford University Press.

    Chapter  Google Scholar 

  • Liddle, H. (2002). Multidimensional family therapy (MDFT) for adolescent cannabis users. Rockville: Center for Substance Abuse Treatment, Substance Abuse and Mental Health Services Administration.

    Google Scholar 

  • Liddle, H. A., Rowe, C. L., Gonzalez, A., Henderson, C. E., Dakof, G. A., & Greenbaum, P. E. (2006). Changing provider practices, program environment, and improving outcomes by transporting multidimensional family therapy to an adolescent drug treatment setting. American Journal on Addictions, 15(Suppl), 102–112.

    Article  PubMed  Google Scholar 

  • Lopez, A. (2008). Statistical machine translation. ACM Computing Surveys (CSUR), 40(3), 8.

    Article  Google Scholar 

  • Minuchin, S. (1974). Families and family therapy. Cambridge: Harvard University Press.

    Google Scholar 

  • Minuchin, S., & Fishman, C. H. (1981). Family therapy techniques. Cambridge: Harvard University Press.

    Google Scholar 

  • Orlinsky, D. E., Ronnestad, M. H., & Willutski, U. (2004). In Lambert M. J. (Ed.), Fifty years of psychotherapy process-outcome research: Continuity and change [Handbook of psychotherapy and behaviour change] (5th ed.). New York: John Wiley & Sons.

  • Overholser, J. C. (1995). Elements of the socratic method: IV. Disavowal of knowledge. Psychotherapy: Theory, research, practice. Training, 32(2), 283–292.

    Google Scholar 

  • Palinkas, L. A., Aarons, G. A., Horwitz, S. M., Chamberlain, P., Hulburt, M., & Landsverk, J. (2011). Mixed method designs in implementation research. Administration & Policy in Mental Health and Mental Health Services, 38(1), 44–53.

    Article  Google Scholar 

  • Pantin, H., Coatsworth, J. D., Feaster, D. J., Newman, F. L., Briones, E., Prado, G., et al. (2003). Familias unidas: The efficacy of an intervention to promote parental investment in Hispanic immigrant families. Prevention Science, 4(3), 189–201.

    Article  PubMed  Google Scholar 

  • Pantin, H., Prado, G., Lopez, B., Huang, S., Tapia, M. I., Schwartz, S. J., et al. (2009). A randomized controlled trial of familias unidas for Hispanic adolescents with behavior problems. Psychosomatic Medicine, 71(9), 987–995. doi:10.1097/PSY.0b013e3181bb2913.

    Article  PubMed Central  PubMed  Google Scholar 

  • Pantin, H., Schwartz, S. J., Sullivan, S., Prado, G., & Szapocznik, J. (2004). Ecodevelopmental HIV prevention programs for Hispanic adolescents. American Journal of Orthopsychiatry, 74(4), 545–558.

    Article  PubMed Central  PubMed  Google Scholar 

  • Poduska, J., Kellam, S. G., Brown, C. H., Ford, C., Windham, A., Keegan, N., et al. (2009). Study protocol for a group randomized controlled trial of a classroom-based intervention aimed at preventing early risk factors for drug abuse: Integrating effectiveness and implementation research. Implementation Science, 4, 56. doi:10.1186/1748-5908-4-56.

    Article  PubMed Central  PubMed  Google Scholar 

  • Popescu, A., Etzioni, O., & Kautz, H. (2003). Towards a theory of natural language interfaces to databases. Proceedings of the 8th International Conference on Intelligent User Interfaces, Miami (pp. 149–157).

  • Prado, G., Huang, S., Cordova, D., Malcolm, S., Estrada, Y., Cano, N., et al. (2013). Ecodevelopmental and intrapersonal moderators of a family based preventive intervention for hispanic youth: A latent profile analysis. Prevention Science: The Official Journal of the Society for Prevention Research, 14(3), 290–299. doi:10.1007/s11121-012-0326-x.

    Article  Google Scholar 

  • Prado, G., & Pantin, H. (2011). Reducing substance use and HIV health disparities among Hispanic youth in the USA: The Familias Unidas program of research. Intervencion Psicosocial, 20(1), 63–73. doi:10.5093/in2011v20n1a6.

    Article  PubMed Central  PubMed  Google Scholar 

  • Prado, G., Pantin, H., Briones, E., Schwartz, S. J., Feaster, D., Huang, S., et al. (2007). A randomized controlled trial of a parent-centered intervention in preventing substance use and HIV risk behaviors in Hispanic adolescents. Journal of Consulting and Clinical Psychology, 75(6), 914–926. doi:10.1037/0022-006X.75.6.914.

    Article  PubMed Central  PubMed  Google Scholar 

  • Prado, G., Pantin, H., Huang, S., Cordova, D., Tapia, M. I., Velazquez, M. R., et al. (2012). Effects of a family intervention in reducing HIV risk behaviors among high-risk Hispanic adolescents: A randomized controlled trial. Archives of Pediatric and Adolescent Medicine, 166(2), 127–133. doi:10.1001/archpediatrics.2011.189.

    Article  Google Scholar 

  • Prado, G., Pantin, H., Schwartz, S. J., Schwartz, S. J., Lupei, N. S., & Szapocznik, J. (2006). Predictors of engagement and retention into a parent-centered, ecodevelopmental HIV preventive intervention for Hispanic adolescents and their families. Journal of Pediatric Psychology, 31(9), 874–890. doi:10.1093/jpepsy/jsj046.

    Article  PubMed  Google Scholar 

  • Prinz, R. J., Sanders, M. R., Shapiro, C. J., Whitaker, D. J., & Lutzker, J. R. (2009). Population-based prevention of child maltreatment: The US triple p system population trial. Prevention Science, 10(1), 1–12. doi:10.1007/s11121-009-0123-3.

    Article  PubMed Central  PubMed  Google Scholar 

  • Real, K., & Poole, M. S. (2005). Innovation implementation: Conceptualization and measurement in organizational research. In R. Woodman, W. Pasmore & A. B. Shani (Eds.), Research in organizational change and development (15th ed., pp. 63–134). Online: Emerald Group Publishing Limited. doi:10.1016/S0897-3016(04)15003-9.

  • Robbins, M. S., Feaster, D. J., Horigian, V. E., Puccinelli, M. J., Henderson, C., & Szapocznik, J. (2011). Therapist adherence in brief strategic family therapy for adolescent drug abusers. Journal of Consulting and Clinical Psychology, 79(1), 43–53. doi:10.1037/a0022146.

    Article  PubMed Central  PubMed  Google Scholar 

  • Rowe, C., Rigter, H., Henderson, C., Gantner, A., Mos, K., Nielsen, P., et al. (2013). Implementation fidelity of multidimensional family therapy in an international trial. Journal of Substance Abuse Treatment, 44(4), 391–399.

    Article  PubMed  Google Scholar 

  • Schoenwald, S. K., Chapman, J. E., Kelleher, K., Hoagwood, K. E., Landsverk, J., Stevens, J., et al. (2008). A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs). Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 84–97.

    Article  PubMed  Google Scholar 

  • Schoenwald, S. K., & Garland, A. F. (2013). A review of treatment adherence measurement methods. Psychological Assessment, 25, 146–156. doi:10.1037/a0029715.

    Article  PubMed Central  PubMed  Google Scholar 

  • Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health, 38(1), 32–43. doi:10.1007/s10488-010-0321-0.

    Article  PubMed Central  PubMed  Google Scholar 

  • Suarez-Morales, L., Matthews, J., Martino, S., Ball, S. A., Rosa, C., Farentinos, C., et al. (2007). Issues in designing and implementing a Spanish-language multi-site clinical trial. American Journal on Addictions, 16(3), 206–215.

    Article  PubMed Central  PubMed  Google Scholar 

  • Szapocznik, J., & Coatsworth, J. D. (1999). An ecodevelopmental framework for organizing the influences on drug abuse: A developmental model of risk and protection. Drug abuse: Origins & interventions. Washington: American Psychological Association. doi:10.1037/10341-014.

    Google Scholar 

  • Szapocznik, J., Hervis, O. E., & Schwartz, S. (2003). Brief strategic family therapy for adolescent drug abuse. (NIH publication no. 03-4751; NIDA Therapy Manuals for Drug Addiction Series ed. Rockville: National Institute of Drug Abuse.

    Google Scholar 

  • Wall, L., Christiansen, T., & Orwant, J. (2000). Programming perl (3rd ed.). Sebastopol: O’Reilly Media.

    Google Scholar 

Download references

Acknowledgments

The content of this paper is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies. We acknowledge support from the Center for Computational Science at the University of Miami (CG, MO, CHB) and the National Institute on Drug Abuse (NIDA) for this work through the Center for Prevention Implementation Methodology for Drug Abuse and Sex Risk Behavior, P30DA027828 (HP, JV, GP, MO, CHB) and Familias Unidas Stage III Study: Preventing Substance Abuse in Hispanic Youth, R01DA025192 (HP, GP, MT). This work was supported by National Institute of Allergy and Infectious Diseases Grant P30AI073961 (GP, HB, MT) and Centers for Disease Control and Prevention Grant U01PS0000671 (GP, HP, MT). The content of this paper is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.

Author information

Authors and Affiliations

  1. Department of Psychiatry and Behavioral Sciences, Fienberg School of Medicine, Northwestern University, 680 N. Lake Shore Drive, Suite 1400, Chicago, IL, 60611, USA

    Carlos Gallo, Juan Villamar, Gracelyn Cruden & C. Hendricks Brown

  2. Department of Public Health Sciences, University of Miami Miller School of Medicine, Miami, FL, 33136, USA

    Hilda Pantin, Guillermo Prado & Maria Tapia

  3. Center for Computational Sciences, University of Miami Miller School of Medicine, Miami, FL, 33136, USA

    Mitsunori Ogihara

Authors
  1. Carlos Gallo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hilda Pantin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Juan Villamar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guillermo Prado
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Maria Tapia
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mitsunori Ogihara
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Gracelyn Cruden
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. C. Hendricks Brown
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carlos Gallo.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gallo, C., Pantin, H., Villamar, J. et al. Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention. Adm Policy Ment Health 42, 574–585 (2015). https://doi.org/10.1007/s10488-014-0538-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-014-0538-4

Keywords

Search

Navigation