Measuring Use of Evidence Based Psychotherapy for Posttraumatic Stress Disorder in a Large National Healthcare System

  • Shira Maguen
  • Erin Madden
  • Olga V. Patterson
  • Scott L. DuVall
  • Lizabeth A. Goldstein
  • Kristine Burkman
  • Brian Shiner
Original Report


To derive a method of identifying use of evidence-based psychotherapy (EBP) for post-traumatic stress disorder (PTSD), we used clinical note text from national Veterans Health Administration (VHA) medical records. Using natural language processing, we developed machine-learning algorithms to classify note text on a large scale in an observational study of Iraq and Afghanistan veterans with PTSD and one post-deployment psychotherapy visit by 8/5/15 (N = 255,968). PTSD visits were linked to 8.1 million psychotherapy notes. Annotators labeled 3467 randomly-selected psychotherapy notes (kappa = 0.88) to indicate receipt of EBP. We met our performance targets of overall classification accuracy (0.92); 20.2% of veterans received ≥ one session of EBP over the study period. Our method can assist with identifying EBP use and studying EBP-associated outcomes in routine clinical practice.


Posttraumatic stress disorder Natural language processing Evidence-based medicine Health services utilization Psychotherapy 



We would like to thank Corinne Halls and the VINCI services clinical annotation team for performing manual chart review. We would also like to thank Callan Lujan for her assistance with this manuscript.


This work was funded by Department of Defense Grant W81XWH-15-1-0038 (Maguen) and VA grant HSR&D CDA11-263 (Shiner).

Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. Baharudin, B., Lee, L. H., & Khan, K. A. (2010). Review of machine learning algorithms for text-documents classification. Journal of Advances in Information Technology, 1(1), 4–20. Scholar
  2. Batuwita, R., & Palade, V. (2013). Class imbalance learning methods for support vector machines. In Imbalanced learning (pp. 83–99). Hoboken, NJ: Wiley.CrossRefGoogle Scholar
  3. Chang, C. C., & Lin, C. J. (2011). LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3), 1–27. Scholar
  4. Chard, K. M., Schumm, J. A., Owens, G. P., & Cottingham, S. M. (2010). A comparison of OEF and OIF veterans and Vietnam veterans receiving cognitive processing therapy. Journal of Traumatic Stress, 23(1), 25–32. Scholar
  5. Cornia, R., Patterson, O. V., Ginter, T., & DuVall, S. L. (2014). Rapid NLP development with Leo. In AMIA Annual Symposium Proceedings. (p. 1356).Google Scholar
  6. Cully, J. A., Tolpin, L., Henderson, L., Jimenez, D., Kunik, M. E., & Peterson, L. A. (2008). Psychotherapy in the Veterans Health Administration: Missed opportunities?. Psychological Services, 5(4), 320–331. Scholar
  7. D’Avolio, L. W., Nguyen, T. M., Farwell, W. R., Chen, Y., Fitzmeyer, F., Harris, O. M., & Fiore, L. D. (2010). Evaluation of a generalizable approach to clinical information retrieval using the automated retrieval console (ARC). Journal of the American Medical Informatics Association, 17(4), 375–382. Scholar
  8. Dreyer, N. A., Tunis, S. R., Berger, M., Ollendorf, D., Mattox, P., & Glikich, R. (2010). Why observational studies should be among the tools used in comparative effectiveness research. Health Affairs, 29(10), 1818–1825. Scholar
  9. Ferrucci, D., & Lally, A. (2004). UIMA: An architectural approach to unstructured information processing in the corporate research environment. Natural Language Engineering, 10(3–4), 327–348. Scholar
  10. Haagen, J. F., Smid, G. E., Knipscheer, J. W., & Kleber, R. J. (2015). The efficacy of recommended treatments for veterans with PTSD: A metaregression analysis. Clinical Psychology Review, 40, 184–194. Scholar
  11. Harpaz-Rotem, I., & Rosenheck, R. A. (2011). Serving those who served: Retention of newly returning veterans from Iraq and Afghanistan in mental health treatment. Psychiatric Services, 62(1), 22–27. Scholar
  12. Hebenstreit, C. L., Madden, E., Koo, K. H., & Maguen, S. (2015). Minimally adequate mental health care and latent classes of PTSD symptoms in female Iraq and Afghanistan veterans. Psychiatric Services, 65(12), 1414–1419.Google Scholar
  13. Hsu, C. W., & Lin, C. J. (2002). A comparison of methods for multi-class support vector machines. IEEE Transactions on Neural Networks, 13, 415–425. Scholar
  14. Karlin, B. E., Ruzek, J. I., Chard, K. M., Eftekhari, A., Monson, C. M., Hembree, E. A., … Foa, E. B. (2010). Dissemination of evidence-based psychological treatments for posttraumatic stress disorder in the veterans health administration. Journal of Traumatic Stress, 23(6), 663–673. Scholar
  15. Kehle-Forbes, S. M., Meis, L. A., Spoont, M. R., & Polusny, M. A. (2016). Treatment initiation and dropout from prolonged exposure and cognitive processing therapy in a VA outpatient clinic. Psychological Trauma: Theory, Research, Practice, and Policy, 8(1), 107–114. Scholar
  16. Kussman, M. J. (2008). VHA Handbook 1160.01: Uniform mental helath services in VA medical centers and clinic. Accessed March 1, 2010, from
  17. Lu, M. W., Plagge, J. M., Marsiglio, M. C., & Dobscha, S. K. (2016). Clinician documentation on receipt of trauma-focused evidence-based psychotherapies in a VA PTSD clinic. Journal of Behavioral Health Services & Research, 43(1), 71–87. Scholar
  18. Maguen, S., Madden, E., Neylan, T. C., Cohen, B. E., Bertenthal, D., & Seal, K. H. (2014). Timing of mental health treatment and PTSD symptom improvement among Iraq and Afghanistan veterans. Psychiatric Services, 65(12), 1414–1419. CrossRefPubMedGoogle Scholar
  19. Meystre, S. M., Savova, G. K., Kipper-Schuler, K. C., & Hurdle, J. F. (2008). Extracting information from textual documents in the electronic health record: A review of recent research. Yearbook of Medical Informatics, 35, 128–144.Google Scholar
  20. Monson, C. M., Schnurr, P. P., Resick, P. A., Friedman, M. J., Young-Xu, Y., & Stevens, S. P. (2006). Cognitive processing therapy for veterans with military-related posttraumatic stress disorder. Journal Consulting and Clinical Psychology, 74(5), 898–907. Scholar
  21. Mott, J. M., Mondragon, S., Hundt, N. E., Beason-Smith, M., Grady, R. H., & Teng, E. J. (2014). Characteristics of U.S. veterans who begin and complete prolonged exposure and cognitive processing therapy for PTSD. Journal of Traumatic Stress, 27(3), 265–273. Scholar
  22. Resick, P. A., Wachen, J. S., Dondanville, K. A., Pruiksma, K. E., Yarvis, J. S., Peterson, A. L., … The STRONG STAR Consortium. (2017). Effect of group vs individual cognitive processing therapy in active-duty military seeking treatment for posttraumatic stress disorder: A randomized clinical trial. JAMA Psychiatry, 74(1), 28–36. Scholar
  23. Resick, P. A., Wachen, J. S., Mintz, J., Young-McCaughan, S., Roache, J. D., Borah, A. M., … Peterson, A. L. (2015). A randomized clinical trial of group cognitive processing therapy compared with group present-centered therapy for PTSD among active duty military personnel. Journal of Consulting and Clinical Psychology, 83(6), 1058–1068. Scholar
  24. Rosen, C. S., Eftekhari, A., Crowley, J. J., Smith, B. N., Kuhn, E., Trent, L., … Ruzek, J. I. (2017). Maintenance and reach of exposure psychotherapy for posttraumatic stress disorder 18 months after training. Journal of Traumatic Stress, 30(1), 63–70. Scholar
  25. Rosen, C. S., Matthieu, M. M., Stirman, S. W., Cook, J. M., Landes, S., Bernardy, N. C., … Watts, B. V. (2016). A review of studies on the system-wide implementation of evidence-based psychotherapies for posttraumatic stress disorder in the Veterans Health Administration. Administration and Policy in Mental Health and Mental Health Services Research, 43(6), 957–977. Scholar
  26. Schnurr, P. P., Friedman, M. J., Engel, C. C., Foa, E. B., Shea, M. T., Chow, B. K., … Bernardy, N. (2007). Cognitive behavioral therapy for posttraumatic stress disorder in women: A randomized controlled trial. JAMA Psychaitry, 297(8), 820–830. Scholar
  27. Schnurr, P. P., & Lunney, C. A. (2015). Differential effects of prolonged exposure on posttraumatic stress disorder symptoms in female veterans. Journal of Consulting Clinical Psychology, 83(6), 1154–1160. Scholar
  28. Seal, K. H., Bertenthal, D., Miner, C. R., Sen, S., & Marmar, C. (2007). Bringing the war back home: Mental health disorders among 103,788 US veterans returning from Iraq and Afghanistan seen at Department of Veterans Affairs facilities. Archives of Internal Medicine, 167(5), 476–482. Scholar
  29. Seal, K. H., Maguen, S., Bertenthal, D., Batki, S. L., Striebel, J., Stein, M. B., Madden, E., & Neylan, T. C. (2016). Observational evidence for buprenorphine’s impact on posttraumatic stress symptoms in Veterans with chronic pain and opioid use disorder. Journal of Clinical Psychiatry, 77(9), 1182–1188. Scholar
  30. Seal, K. H., Maguen, S., Cohen, B., Gima, K. S., Metzler, T. J., Ren, L., … Marmar, C. R. (2010). VA mental health services utilization in Iraq and Afghanistan veterans in the first year of receiving new mental health diagnoses. Journal of Traumatic Stress, 23(1), 5–16. Scholar
  31. Shiner, B., D’Avolio, L. W., Nguyen, T. M., Zayed, M. H., Young-Xu, Y., Desai, R. A., … Watts, B. V. (2013). Measuring use of evidence based psychotherapy for posttraumatic stress disorder. Administration and Policy in Mental Health and Mental Health Services Research, 40(4), 311–318. Scholar
  32. Shiner, B., D’Avolio, L. W., Nguyen, T. N., Zayed, M. H., Watts, B. V., & Fiore, L. (2012). Automated classification of psychotherapy note text: Implications for quality assessment in PTSD care. Journal of Evaluation in Clinical Practice, 18(3), 698–701. Scholar
  33. Shiner, B., Westgate, C. L., Bernary, N. C., Schnurr, P. P., & Watts, B. V. (2017). Anticonvulsant medication use in veterans with posttraumatic stress disorder. Journal of Clinical Psychiatry, 78(5), e545–e552. Scholar
  34. Shiner, B., Westgate, C. L., Simiola, V., Thompson, R., Schnurr, P. P., & Cook, J. M. Measuring use of evidence-based psychotherapy for PTSD in VA residential treatment settings with clinician survey and electronic medical record templates. Military Medicine (in press).Google Scholar
  35. Smith, E. R., Porter, K. E., Messina, M. G., Beyer, J. A., Defever, M. E., Foa, E. B., & Rauch, S. A. (2015). Prolonged exposure for PTSD in a Veteran group: A pilot effectiveness study. Journal of Anxiety Disorders, 30, 23–27. Scholar
  36. Spoont, M. R., Murdoch, M., Hodges, J., & Nugent, S. (2010). Treatment receipt by veterans after a PTSD diagnosis in PTSD, mental health, or general medical clinics. Psychiatric Services, 61(1), 58–63. Scholar
  37. Tuerk, P. W., Yoder, M., Grubaugh, A., Myrick, H., Hamner, M., & Acierno, R. (2011). Prolonged exposure therapy for combat-related posttraumatic stress disorder: An examination of treatment effectiveness for veterans of the wars in Afghanistan and Iraq. Journal of Anxiety Disorders, 25(3), 397–403. Scholar
  38. Voelkel, E., Pukay-Martin, N. D., Walter, K. H., & Chard, K. M. (2015). Effectiveness of cognitive processing therapy for male and female U.S. Veterans with and without military sexual trauma. Journal of Traumatic Stress, 28(3), 174–182. Scholar
  39. Yoder, M., Tuerk, P. W., Price, M., Grubaugh, A. L., Strachan, M., Myrick, H., & Acierno, R. (2012). Prolonged exposure therapy for combat-related posttraumatic stress disorder: Comparing outcomes for Veterans of different wars. Psychological Services, 9(1), 16–25. Scholar
  40. Zubkoff, L., Carpenter-Song, E., Shiner, B., Ronconi, J. M., & Watts, B. V. (2016). Clinicians’ perception of patient readiness for treatment: An emerging theme in implementation science. Administration and Policy in Mental Health and Mental Health Services Research, 43(2), 250–258. Scholar

Copyright information

© This is a U.S. Government work and not under copyright protection in the US; foreign copyright protection may apply 2018

Authors and Affiliations

  1. 1.San Francisco VA Health Care SystemSan FranciscoUSA
  2. 2.University of California, San Francisco, School of MedicineSan FranciscoUSA
  3. 3.VA Salt Lake City Health Care SystemSalt Lake CityUSA
  4. 4.University of Utah, School of MedicineSalt Lake CityUSA
  5. 5.White River Junction VA Medical CenterWhite River JunctionUSA
  6. 6.Geisel School of Medicine at DartmouthHanoverUSA
  7. 7.Executive DivisionNational Center for Posttraumatic Stress DisorderWhite River JunctionUSA
  8. 8.Department of Mental HealthSan Francisco VA Health Care SystemSan FranciscoUSA

Personalised recommendations