The AAPS Journal

, 21:40 | Cite as

A Receiver Operating Characteristic Framework for Non-adherence Detection Using Drug Concentration Thresholds—Application to Simulated Risperidone Data in Schizophrenic Patients

  • Carlos Pérez-RuixoEmail author
  • Bart Remmerie
  • Juan José Peréz-Ruixo
  • An Vermeulen
Research Article


Non-adherence to antipsychotic medication is a primary factor in disease relapse in schizophrenic patients. We sought to evaluate if plasma concentrations of the antipsychotic risperidone can be used as a predictor of treatment adherence and to identify the optimal plasma concentration threshold to reliably distinguish between adherent and non-adherent patients. A population pharmacokinetic model was used to simulate plasma risperidone steady-state trough concentrations in 1000 virtual patients, where 60% of the patients were 100% adherent to their medication, while 40% of the patients were non-adherent to their medication. The probability of adherence was assessed by receiver operating characteristic (ROC) analysis on Ctrough. The area under the ROC curve (AUCROC) was used to identify the optimal Ctrough threshold. Single vs multiple Ctrough at steady state was also evaluated. After a single risperidone Ctrough measurement, the AUCROC (95% CI) was estimated to be 0.71 (0.69–0.72) and the optimal Ctrough threshold accounting for the lowest number of adherent and non-adherent misclassifications was estimated to be 11.9 ng/mL. After multiple Ctrough measurements, the AUCROC (95% CI) increased up to 0.85 (0.84–0.87) for three Ctrough measurements. The optimal probability threshold to reliably discriminate between adherent and non-adherent patients was estimated to be 0.51. Using this model which is reflective of typical adherence to antipsychotic medication, we found that three consecutive steady-state Ctrough measurements are needed for an accurate and precise diagnostic test to discriminate between patients who are adherent or non-adherent to treatment.


diagnostic test drug adherence drug plasma concentrations population PK model–based simulation ROC analysis 



Stacey E. Shehin, Ph.D. (PRA Health Sciences) provided medical writing assistance, which was funded by Janssen Research & Development. Harry Ma, Ph.D. (Janssen Global Services) provided additional editorial support. Portions of this study have been previously presented at the Population Approach Group of Europe’s 26th Annual Scientific Meeting, June 6–9, 2017, Budapest, Hungary. All authors meet ICMJE criteria, had full access to the study data, and take responsibility for integrity of the data.


This study was funded by Janssen Research & Development, LLC.

Compliance with Ethical Standards

Conflict of interest

All authors are employees of Janssen Research & Development, LLC, and may hold stock options or shares in the company.


  1. 1.
    World Health Organization. Mental Health, Schizophrenia. 2017. Available from: Accessed 13 Nov 2018.
  2. 2.
    Keating D, McWilliams S, Schneider I, Hynes C, Cousins G, Strawbridge J, et al. Pharmacological guidelines for schizophrenia: a systematic review and comparison of recommendations for the first episode. BMJ Open. 2017;7(1):e013881.PubMedPubMedCentralGoogle Scholar
  3. 3.
    Lacro JP, Dunn LB, Dolder CR, Leckband SG, Jeste DV. Prevalence of and risk factors for medication nonadherence in patients with schizophrenia: a comprehensive review of recent literature. J Clin Psychiatry. 2002;63(10):892–909.PubMedGoogle Scholar
  4. 4.
    Acosta FJ, Hernandez JL, Pereira J, Herrera J, Rodriguez CJ. Medication adherence in schizophrenia. World J Psychiatry. 2012;2(5):74–82.PubMedPubMedCentralGoogle Scholar
  5. 5.
    Novick D, Haro JM, Suarez D, Perez V, Dittmann RW, Haddad PM. Predictors and clinical consequences of non-adherence with antipsychotic medication in the outpatient treatment of schizophrenia. Psychiatry Res. 2010;176(2–3):109–13.PubMedGoogle Scholar
  6. 6.
    Higashi K, Medic G, Littlewood KJ, Diez T, Granstrom O, De Hert M. Medication adherence in schizophrenia: factors influencing adherence and consequences of nonadherence, a systematic literature review. Ther Adv Psychopharmacol. 2013;3(4):200–18.PubMedPubMedCentralGoogle Scholar
  7. 7.
    Velligan DI, Weiden PJ, Sajatovic M, Scott J, Carpenter D, Ross R, et al. The expert consensus guideline series: adherence problems in patients with serious and persistent mental illness. J Clin Psychiatry. 2009;70(Suppl 4):1–46 quiz 7-8.PubMedGoogle Scholar
  8. 8.
    Karve S, Cleves MA, Helm M, Hudson TJ, West DS, Martin BC. Good and poor adherence: optimal cut-point for adherence measures using administrative claims data. Curr Med Res Opin. 2009;25(9):2303–10.PubMedGoogle Scholar
  9. 9.
    Velligan DI, Lam Y-WF, Glahn DC, Barrett JA, Maples NJ, Ereshefsky L, et al. Defining and assessing adherence to oral antipsychotics: a review of the literature. Schizophr Bull. 2006;32(4):724–42.PubMedPubMedCentralGoogle Scholar
  10. 10.
    Horvitz-Lennon M, Mattke S, Predmore Z, Howes OD. The role of antipsychotic plasma levels in the treatment of schizophrenia. Am J Psychiatry. 2017;174(5):421–6.PubMedGoogle Scholar
  11. 11.
    Korell J, Green B, Remmerie B, Vermeulen A. Determination of plasma concentration reference ranges for risperidone and paliperidone. CPT Pharmacometrics Syst Pharmacol. 2017;6(9):589–95.PubMedPubMedCentralGoogle Scholar
  12. 12.
    Suzuki Y, Kawasaki K, Sato Y, Tokimatsu I, Itoh H, Hiramatsu K, et al. Is peak concentration needed in therapeutic drug monitoring of vancomycin? A pharmacokinetic-pharmacodynamic analysis in patients with methicillin-resistant Staphylococcus aureus pneumonia. Chemotherapy. 2012;58(4):308–12.PubMedGoogle Scholar
  13. 13.
    Olivares-Morales A, Hatley OJ, Turner D, Galetin A, Aarons L, Rostami-Hodjegan A. The use of ROC analysis for the qualitative prediction of human oral bioavailability from animal data. Pharm Res. 2014;31(3):720–30.PubMedGoogle Scholar
  14. 14.
    Clements JD, Perez Ruixo JJ, Gibbs JP, Doshi S, Perez Ruixo C, Melhem M. Receiver operating characteristic analysis and clinical trial simulation to inform dose titration cecisions. CPT: Pharmacometrics Systems Pharmacol. 2018;7(11):771–9.Google Scholar
  15. 15.
    Hiligsmann M, Rabenda V, Gathon H-J, Ethgen O, Reginster J-Y. Potential clinical and economic impact of nonadherence with osteoporosis medications. Calcif Tissue Int. 2010;86(3):202–10.PubMedGoogle Scholar
  16. 16.
    Chang DS, Friedman DS, Frazier T, Plyler R, Boland MV. Development and validation of a predictive model for nonadherence with once-daily glaucoma medications. Ophthalmology. 2013;120(7):1396–402.PubMedPubMedCentralGoogle Scholar
  17. 17.
    Florkowski CM. Sensitivity, specificity, receiver-operating characteristic (ROC) curves and likelihood ratios: communicating the performance of diagnostic tests. Clin Biochem Rev. 2008;29(Suppl 1):S83–7.PubMedPubMedCentralGoogle Scholar
  18. 18.
    Park SH, Goo JM, Jo CH. Receiver operating characteristic (ROC) curve: practical review for radiologists. Korean J Radiol. 2004;5(1):11–8.PubMedPubMedCentralGoogle Scholar
  19. 19.
    Vermeulen A, Piotrovsky V, Ludwig EA. Population pharmacokinetics of risperidone and 9-hydroxyrisperidone in patients with acute episodes associated with bipolar I disorder. J Pharmacokinet Pharmacodyn. 2007;34(2):183–206.PubMedGoogle Scholar
  20. 20.
    Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993;39(4):561–77.PubMedGoogle Scholar
  21. 21.
    Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978;8(4):283–98.PubMedGoogle Scholar
  22. 22.
    Swets JA. Indices of discrimination or diagnostic accuracy: their ROCs and implied models. Psychol Bull. 1986;99(1):100–17.PubMedGoogle Scholar
  23. 23.
    Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36.PubMedGoogle Scholar
  24. 24.
    Boyd JC. Statistical analysis and presentation of data. In: Price CP, editor. Evidence-based laboratory medicine; principles, practice and outcomes. 2nd ed. Washington DC, USA: AACC Press; 2007. p. 113–40.Google Scholar
  25. 25.
    Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc. 1927;22(158):209–12.Google Scholar
  26. 26.
    McNeil BJ, Keller E, Adelstein SJ. Primer on certain elements of medical decision making. N Engl J Med. 1975;293(5):211–5.PubMedGoogle Scholar
  27. 27.
    Lusted LB. Signal detectability and medical decision-making. Science. 1971;171(3977):1217–9.PubMedGoogle Scholar
  28. 28.
    Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3(1):32–5.PubMedGoogle Scholar
  29. 29.
    Altman DG, Bland JM. Diagnostic tests 2: predictive values. BMJ. 1994;309(6947):102.PubMedPubMedCentralGoogle Scholar
  30. 30.
    NONMEM. 7.1.0 Users Guides. Icon Development Solutions: Ellicott City, MD; 1989-2009.Google Scholar
  31. 31.
    R Development Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2012.Google Scholar
  32. 32.
    El Khouli RH, Macura KJ, Barker PB, Habba MR, Jacobs MA, Bluemke DA. Relationship of temporal resolution to diagnostic performance for dynamic contrast enhanced MRI of the breast. J Magn Reson Imaging. 2009;30(5):999–1004.PubMedPubMedCentralGoogle Scholar
  33. 33.
    Ludemann L, Grieger W, Wurm R, Wust P, Zimmer C. Glioma assessment using quantitative blood volume maps generated by T1-weighted dynamic contrast-enhanced magnetic resonance imaging: a receiver operating characteristic study. Acta Radiol. 2006;47(3):303–10.PubMedGoogle Scholar
  34. 34.
    Mannaert E, Vermeulen A, Remmerie B, Bouhours P, Levron JC. Pharmacokinetic profile of long-acting injectable risperidone at steady-state: comparison with oral administration. Encéphale. 2005;31(5 Pt 1):609–15.PubMedGoogle Scholar
  35. 35.
    Melkote R, Singh A, Vermeulen A, Remmerie B, Savitz A. Relationship between antipsychotic blood levels and treatment failure during the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) study. Schizophr Res. 2018;201:324–8.PubMedGoogle Scholar
  36. 36.
    Morrison A, Stauffer ME, Kaufman AS. Relationship between adherence rate threshold and drug ‘forgiveness’. Clin Pharmacokinet. 2017;56(12):1435–40.PubMedGoogle Scholar
  37. 37.
    Kane JM. Tools to assess negative symptoms in schizophrenia. J Clin Psychiatry. 2013;74(6):e12-e.Google Scholar
  38. 38.
    Fellows K, Stoneking CJ, Ramanathan M. Bayesian population modeling of drug dosing adherence. J Pharmacokinet Pharmacodyn. 2015;42(5):515–25.PubMedGoogle Scholar

Copyright information

© American Association of Pharmaceutical Scientists 2019

Authors and Affiliations

  • Carlos Pérez-Ruixo
    • 1
    Email author
  • Bart Remmerie
    • 1
  • Juan José Peréz-Ruixo
    • 1
  • An Vermeulen
    • 2
  1. 1.Janssen Research & DevelopmentClinical Pharmacology & PharmacometricsBeerseBelgium
  2. 2.Janssen Research & DevelopmentQuantitative Sciences ConsultingBeerseBelgium

Personalised recommendations