Abstract
Background
An adequate supply of mental health (MH) professionals is necessary to provide timely access to MH services. Veterans Health Administration (VHA) continues to prioritize the expansion of the MH workforce to meet increasing demand for services.
Objective
Validated staffing models are essential to ensure timely access to care, to plan for future demand, to ensure delivery of high-quality care, and to balance the demands of fiscal responsibility and strategic priorities.
Design
Longitudinal retrospective cohort of VHA outpatient psychiatry, fiscal years 2016–2021.
Participants
Outpatient VHA psychiatrists.
Main Measures
Quarterly outpatient staff-to-patient ratios (SPRs), defined as the number of full-time equivalent clinically assigned providers per 1000 veterans receiving outpatient MH care, were calculated. Longitudinal recursive partitioning models were created to identify optimal cut-offs for the outpatient psychiatry SPR associated with success on VHA’s measures of quality, access, and satisfaction.
Key Results
Among outpatient psychiatry staff, the root node identified an outpatient SPR of 1.09 for overall performance (p < 0.001). For metrics associated with Population Coverage, a root node identified an SPR of 1.36 (p < 0.001). Metrics associated with continuity of care and satisfaction were associated with a root node of 1.10 and 1.07 (p < 0.001), respectively. In all analyses, the lowest SPRs were associated with the lowest group performance on VHA MH metrics of interest.
Conclusions
Establishing validated staffing models associated with high-quality MH care is critical given the national psychiatry shortage and increasing demand for services. Analyses support VHA’s current recommended minimum outpatient psychiatry-specific SPR of 1.22 as a reasonable target to provide high-quality care, access, and satisfaction.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Across America, the growing demand for mental health (MH) care far outpaces the current supply of MH providers.1,2,3 Specific to psychiatry, current provider deficits have been well documented and are projected to increase.4,5,6 The availability of an adequate workforce is critical to deliver safe and effective psychiatric care. With increasing demand and a limited supply of psychiatrists, it is vital to have clear staffing expectations to maintain efficient quality and access to care, while maintaining fiscal responsibility and stewardship.7,8
Within the Veterans Health Administration (VHA), between fiscal years (FY) 2006 and 2021, the number of outpatient MH visits increased from 10.7 to 20.0 million, and the proportion of veterans served by VHA who receive MH services increased from 20 to 28%. To meet the growing demand for MH services during this time, VHA increased overall MH staffing from 6923 to 18,606 full-time equivalents (FTE) and psychiatry-specific staffing from 1869 to 3439 FTE during this same period.
In an effort to align supply and demand with the optimal delivery of clinical care, numerous MH staffing models have been developed.9,10,11,12,13,14,15,16 While varied models have been proposed, no clinical or single industry standard identifies expected psychiatry staffing levels in private or public healthcare systems. In response to a 2012 US Department of Veterans Affairs Office of the Inspector General report,17 VHA established a recommended minimum outpatient MH staffing model focusing on the number of providers needed to treat a defined population. This recommendation (7.72 FTE per 1000 outpatient veterans) was calculated by averaging the overall outpatient staff-to-patient ratio (SPR) at that time. VHA subsequently established a psychiatry SPR of 1.22 FTE per 1000 veterans treated in outpatient MH by using the average outpatient psychiatry SPR at the time. To support these recommendations, VHA established internal dashboards to track SPRs at the facility level and formally demonstrated associations between outpatient SPR and MH performance.18,19,20
While greater MH staffing has been associated with decreased suicidal behavior among facility patients,21,22,23 the relationship between staffing levels and other outcomes, such as depression, has been mixed.24,25 In VHA, higher outpatient SPRs have been associated with better performance on VHA measures of quality, access, and satisfaction.18,19,20 Yet, with a national staff shortage, there are significant supply–demand challenges necessitating the need to optimize resources in a clinical and fiscally responsible manner. Ongoing research evaluating staffing models that optimally meet patient demand and provide positive MH outcomes given supply and fiscal limitations remains critical.
Optimal psychiatry SPRs that distinguish higher performing from lower performing VHA facilities have not been evaluated. To evaluate existing VHA psychiatry SPR recommendations, we conducted a series of exploratory analyses utilizing longitudinal binary recursive modeling to determine the optimal SPR associated with overall quality, access, and satisfaction. A series of decision tree models were developed using outpatient psychiatry SPR as the variable of interest. Optimal psychiatry SPR cut-offs have not been systematically evaluated, and this analytic approach can tell us whether the current VHA recommendation established in 2012 is consistent with statistically derived cut-offs for optimal quality, access, and satisfaction.
METHODS
Study Population and Data
This longitudinal study uses quarterly outpatient staffing data from FY 2016 through FY 2021. SPRs from each VHA Medical Center beginning the 1st quarter (Q1) of fiscal year (FY) 2016 (October, 2015) through the 4th quarter (Q4) of FY 2021 (September, 2021) were calculated.
Measures
Staff-to-Patient Ratio
For each of the 24 quarters, outpatient psychiatry SPRs were calcuated.18,19 As calculated, psychiatry SPR does not reflect total FTE available at the medical center, but instead, the time psychiatrists spent in direct outpatient care; non-direct patient care time, such as program oversight, time spent in research or education, and time spent providing inpatient psychiatric services, is not included in the outpatient SPR.
MH Strategic Analytics for Improvement and Learning (MH SAIL) Domain Score
The Mental Health Strategic Analytics for Improvement and Learning (MH SAIL) Domain score is a meta-composite of three MH composite measures created by the VHA for internal evaluation.26,27 The three composite metrics consist of over 30 individual measures considered to assess (a) the proportion of veterans with identified mental disorders who receive MH services (termed Population Coverage), (b) the proportion of veterans receiving a treatment episode or coordinated, proactive care (termed Continuity of Care), and (c) ratings of veteran and provider opinions regarding access, quality, and coordination of care (termed Experience of Care). For each performance year (PY: FYQ4-FY + 1Q3), the individual metrics and the MH SAIL composite scores are standardized to the start of year facility distribution (mean = 0, in units of standard deviation) to assess individual facility change from baseline through the PY. For this study, analyses focused on MH SAIL Domain and the three composite measures, Population Coverage, Continuity of Care, and Experience of Care. See Table 1 in the Appendix for a description of the individual metrics comprising each MH SAIL composite for 2021 and score interpretation.
Statistical Analyses
Traditional parametric analyses do not offer straightforward interpretation methods regarding staffing needed for optimal performance. Thus, we utilize unbiased random regression tree models to identify psychiatry SPRs associated with MH SAIL performance. Hypothesis testing through regression or factorial designs requires established groups or an a priori determination of group designation. In contrast, machine learning models, such as classification and regression trees (CART), offer computationally intensive, non-parametric statistical methods that have been shown to be advantageous when relationships are not necessarily linear.28,29,30 While CART-based algorithms function by maximizing a statistical criterion over all possible predictors and split points simultaneously, tree algorithm models developed by Horton, Hornik, and Zeileis31 correct the problem of selection bias and overfitting by choosing predictors for splitting based upon a series of tests identifying statistically significant associations between the responses and the predictors utilizing the conditional inference tree (cTree) framework.30 These methods30 have been extended to include mixed and random effect expectation maximization (REEM) models associated with longitudinal data.32,33,34 Fu and Simonoff32 subsequently combined the REEM tree algorithm with the conditional inference approach (REEMcTree) to recursively test the null hypothesis between random input and response variables by selecting the binary split based upon the highest p-value with the response variable.
To evaluate the relationships between outpatient psychiatry SPR and MH SAIL performance, we developed a series of unbiased conditional inference trees (REEMcTree). Use of the automated unbiased conditional inference tree modeling analytic approach will tell us which SPR cut-offs are associated with statistically significant group differences in MH SAIL performance. All REEMcTree models are fitted using a maximum likelihood, random effect intercept model allowing for variations across facilities. Analyses were completed in R 4.2.135 utilizing psych, party, and ggplot2 packages.36,37,38,39 REEMcTree coding is available at http://people.stern.nyu.edu/jsimonof/unbiasedREEM/.
RESULTS
Psychiatry Staffing
For the 24 quarters between FY16Q1 and FY2021Q4, the number of onboard psychiatrists increased from 3209 to 3439 staff. Despite ongoing hiring psychiatry across the 6 years, SPR remained generally flat across time (Fig. 1), as the number of unique veterans receiving MH services increased from 1.63 million to nearly 1.84 million over this period. Quarterly SPRs and MH SAIL performance are available in Table 1 in the Appendix.
Psychiatry Staffing Ratio
The recursive model for overall MH SAIL Domain performance identified an outpatient psychiatry SPR of 1.09 as the initial split (Fig. 2). The average MH SAIL Domain performance was.02 for sites below the 1.09 average, and 0.18 for sites above the average. The initial root node for optimal performance on the Population Coverage domain (see Fig. 1 in the Appendix) was an outpatient psychiatry SPR of 1.36 with lower SPRs associated with decreased performance (leaf node 2: mean = − 0.06) compared to sites with psychiatry SPRs above this level (leaf node 3: mean = 0.08). Unbiased conditional decision tree for Continuity of Care identified a single psychiatry-specific SPR associated with group performance differences of 1.10 (see Fig. 2 in the Appendix). Facilities below this level averaged 0.11 on MH SAIL Continuity of Care while facilities above this level averaged 0.27. The decision tree model identified a single SPR split at 1.07 FTE per 1000 outpatient MH patients with sites averaging Experience of Care performance of − 0.02 below this SPR and sites averaging 0.14 above this SPR (see Fig. 3 in the Appendix).
CONCLUSION
Numerous MH staffing models have been developed9,10,11,12,13,14,15,16 to balance provider supply with clinical demand. Recognizing the critical need for MH staff and the clear relationship between supply and demand, congressional legislation has demonstrated ongoing interest in the VHA staffing model and the ratio of MH providers to veterans receiving care in the VHA. In response to a 2012 OIG report, VHA recommended a minimum overall outpatient MH staffing ratio of 7.72 and an outpatient psychiatry SPR of 1.22. In the current study, we identified optimal outpatient psychiatry SPRs associated with VHA measures of MH quality, access, and satisfaction utilizing non-parametric binary recursive models. Overall, the current study found that higher outpatient psychiatry SPRs are associated with higher MH SAIL performance. Optimal SPRs varied for the individual MH SAIL composites, which is not unexpected. In MH SAIL, Population Coverage is specifically designed to assess overall access to care utilizing a population health approach (i.e., engagement of known population in care), whereas the Continuity of Care domain assesses the ongoing ability of those engaged in care to receive care.26,27 Given the complementary, but unique demands of care within each domain, the staffing levels needed to meet those demands are unique to each domain. Regardless, results were consistent across MH SAIL domains, supporting 1.22 as a reasonable balance between the models and the VHA recommendation as minimum psychiatry SPR benchmark.
The principle of supply and demand is commonly applied to healthcare access. In VHA, efforts to increase supply (i.e., hiring additional providers, increasing daily productivity expectations, increasing scheduling utilization efficiency, etc.) have been offset, paradoxically, by further demand increases (i.e., more veterans requesting care). This paradox is not unexpected for two reasons. First, the lack of growth in psychiatry SPR despite increased VHA hiring is consistent with the MH services dilemma outlined by Riessman40 who observed that if the supply of a MH service is increased to meet the growing demand, this will in turn lead to even further increase in demand. This paradox has been the experience across the VHA, particularly at facilities with lower SPR levels. This increased demand is thought to reflect the insufficient supply–demand balance now having additional resources. Second, while VHA has implemented initiatives to increase provider efficiency, such as scheduling procedures and bookable time expectations, Jevons’ Paradox41 asserts that increasing the efficiency with which a resource is available tends to increase, rather than decrease, the rate of consumption of that resource. While originally conceptualized in relation to material resource utilization, this observation is consistent with recent VHA experiences in which access initiatives have resulted in minimal access gains due to the offset of increased patient demand for and utilization of those services. Similar to the observation by Riessman,40 increased clinic efficiency addressing insufficient supply–demand balance when optimized has resulted in minimal access gains, as the unmet, historical demand quickly utilizes available resources. Taking these observations into account, the pace of hiring or supply enhancement must aggressively stay ahead of the corresponding increases in patient care demand.
This exploratory analysis utilizing binary recursive modeling is not without limitation. First, performance on MH SAIL is variable across the SPR spectrum. There are low performing facilities with high SPR levels and high performing facilities with low SPR levels, highlighting that staffing alone is not sufficient for MH SAIL success. Factors such as leadership and administrative decisions, including program alignment, team-based collaboration, and provider burnout, among others, likely contribute to MH SAIL variety. Second, MH SAIL is specifically designed to be fluid in that new measures are added, existing measures are updated, and stable measures associated with program success are removed upon annual review. While the decision tree modeling process independently considers individual SPR and MH SAIL, optimal SPR for any given year may vary slightly as measures of quality, access, and satisfaction are added or removed. Third, moderating covariates were not utilized in the decision tree models, as we focused on the overall psychiatry SPR recommendation. Covariates, such as hospital complexity, staff-mix42, case-mix, and rurality, likely impact supply–demand balance and MH SAIL performance. Currently, while VHA policy does not recommend a minimum staffing ratio based upon potential covariates, additional research will be necessary to refine optimal resource alignment. Finally, this longitudinal analysis consists of staffing and performance data collected across pre- and post-COVID pandemic periods, which is another potential covariate moderating SPRs. VHA shifted rapidly to telehealth services in an effort to maintain safety for veterans and staff,43 and VHA MH SAIL metrics were updated to include the expanded telehealth delivery services authorized by the Centers for Medicare and Medicaid Services (CMS). As with additional covariates, there may be internal and external contributors to optimal performance during any given performance year, which requires additional research consideration.
Broadly, there are two ways to increase the outpatient SPR. First, focusing on the numerator, you can increase SPR by adding more staff either through hiring or by administrative decisions, such as shifting psychiatrists from non-outpatient settings (e.g., acute inpatient units or residential units) or by decreasing psychiatrist administrative, research, or teaching availability. Second, focusing on the denominator, outpatient SPRs can be increased by reducing the demand or number of patients who utilize MH care. Factors limiting utilization may be attributed to internal (e.g., eliminating services, increasing barriers to engagement, or restricting eligibility) or external (e.g., reducing patient treatment-seeking patterns associated with a national pandemic) causes. VHA wants to enhance (i.e., increase) utilization and access across the spectrum of care; thus, increasing the SPR numerator by shifting staff resources or decreasing the denominator by reducing the volume of patients does not align with VHA goals.
Supply and demand models designed to optimize care must evaluate opportunities to optimize the supply and address the demand. To increase the supply of providers, the first strategy typically focuses on hiring. In 2017, VHA established the Mental Health Hiring Initiative (MHHI) to hire 1000 net new MH providers across the enterprise. The MHHI has since developed into a Sustainment Initiative focused on enhancing hiring efficiencies across the enterprise. Between 2016 and 2021, VHA increased psychiatry FTE by 187 FTE (5.75%). Hiring efforts to date have included establishing dedicated recruiters, increasing advertising of positions, increasing trainee and fellowship opportunities, increasing the number of available debt reduction programs, increasing salary pay scales, and leveraging telehealth and alternate work schedules. Recent congressional legislation requiring VHA MH staffing models and annual reports on underserved and critical hiring sites, use of alternative work schedules, and increased trainee/resident funding will continue to prioritize MH staffing levels and the need for ongoing hiring across the VHA. Additional strategies in VHA have included expanding MH support services through hiring Peer Support Specialists, refining processes for referral to ensure the right care at the right time is available, utilizing stepped care and care management practices, and maximizing services supported by advanced practice providers, including clinical pharmacists and nurse practitioners, among others.
Strategies for managing demand typically focus on the volume of patients requesting care. Recent congressional legislation has increased veteran and caregiver eligibility for MH services. Demand management strategies, such as making a program difficult to engage, limiting hours of availability, and restricting eligibility, can reduce service demand, yet these efforts are clinically inappropriate. In contrast, there are clinically appropriate strategies for managing demand. For example, VHA developed medical decision algorithms to identify veterans receiving psychiatric management who are medically stable and ready for discharge back to lower level of care for ongoing medication management. Clinical decision tools are available to prioritize outreach to at-risk veterans, which may subsequently reduce increased service needs due to a crisis. Additionally, VHA has moved to train MH staff in time-limited, evidence-based psychotherapy protocols and has emphasized the delivery of an episode of care where appropriate. While many veterans may require intensive, life-long psychiatric management, many veterans may have their MH needs met through a variety of strategies, such as Peer Support Services, problem focused, brief intervention in Primary Care Mental Health Integration, utilization of group practices, including drop-in group medical appointments, and increasing asynchronous care delivery. Hiring is certainly a strategy for increasing the supply of available providers, yet hiring alone will not solve the supply–demand imbalance. Strategies to both increase provider availability and manage growing demand in a clinically efficient manner will be critical for maximizing veteran MH care.
The outpatient psychiatry SPR utilized in the VHA could certainly be applied to non-VHA settings. While the VHA healthcare system includes a broad array of integrated physical, social, and MH resources not common to private healthcare systems, the core principle of calculating the necessary resources (supply) needed to meet a known clinical need (demand) is universal—the provision of MH care is dependent upon staff availability to meet the patient demand. VHA methodology integrates caseload calculations,9 needs-based calculations,13,14 and ratio consieration.12,16 In contrast to previous studies where ratios are reported as full-time equivalent (FTE) providers per 100,000 health organization members,12,16 the VHA SPR focuses on the number of providers directly needed to treat MH outpatients (per 1000), which is more conducive to facility-level planning and supports efforts to ensure adequacy of MH services for patients who engage. Additional provider time necessary for program administration and non-direct clinical duties, such as research and education training, is over and above the direct care SPR. While non-VHA systems may not have the volume, care complexity, or fiscal and logistical resources of the VHA, all systems must work to optimize supply limitations and demand expectations. Future research utilizing non-VHA healthcare systems would need to confirm or refute this assertion.
References
Health Resources and Services Administration. National projections of supply and demand for selected behavioral health practitioners: 2013–2025. 2016. Accessed September 1, 2022 at https://bhw.hrsa.gov/sites/default/files/bureau-health-workforce/data-research/behavioral-health-2013-2025.pdf.
Hubbard K. Many states face shortage of mental health providers. U.S. News & World Report. June 10, 2021. Accessed September 1, 2022 at https://www.usnews.com/news/best-states/articles/2021-06-10/northeastern-states-have-fewest-mental-health-provider-shortages.
Weiner S. A growing psychiatrist shortage and an enormous demand for mental health services. Association of American Medical Colleges. August 9, 2022. Accessed September 1, 2022 at https://www.aamc.org/news-insights/growing-psychiatrist-shortage-enormous-demand-mental-health-services.
Bishop T, Seirup JK, Pincus HA, Ross JS. Population of US practicing psychiatrists declined, 2003–13, which may help explain poor access to mental health care. Health Aff. 2016;35(7): 1271–1277.
Association of American Medical Colleges. The complexities of physician supply and demand: projections from 2017 to 2032. 2019. Accessed September 1, 2022 at https://aamc-black.global.ssl.fastly.net/production/media/filer_public/31/13/3113ee5c-a038-4c16-89af-294a69826650/2019_update_-_the_complexities_of_physician_supply_and_demand_-_projections_from_2017-2032.pdf.
National Council for Mental Wellbeing. The psychiatric shortage: causes and solutions. 2018. Accessed September 1, 2022 at https://www.thenationalcouncil.org/wp-content/uploads/2022/02/Revised-Final-Access-Paper.pdf.
Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff. 2008;27(3):759-769.
Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med. 2014;12(6):573-576.
Baker JG. A spreadsheet method for calculating maximum caseload and intake capacity for CMHC psychiatrists. Psychiatr Serv. 1997;12: 1578-1581.
Bhaskara SM. Setting benchmarks and determining psychiatric workloads in community mental health programs. Psychiatr Serv. 1999;50:695-697.
Burke BT, Miller BF, Proser M, Petterson SM, Bazemore AW, Goplerud E, Phillips RL. A needs-based method for estimating the behavioral health staff needs of community health centers. BMC Health Serv Res. 2013;13: 245-257.
Dial TH, Bergsten C, Haviland MG, Pincus MG. Psychiatrist and nonphysician mental health provider staffing levels in health maintenance organizations. Am J Psychiatry. 1998;155;405-408.
Faulkner LR, Goldman CR. Estimating psychiatric manpower requirements based on patients’ needs. Psychiatr Serv. 1997;48:666-670.
Goldman CR, Faulkner LR, Breeding KA. A method for estimating psychiatrist staffing needs in community mental health programs. Hosp Community Psychiatry. 1994;45(4):333-337.
Morrison JA. Staffing model based on workload in three VA mental health clinics. Psychiatr Serv. 1998; 49: 1091-1093.
Scheffler R, Ivey SL. Mental health staffing in managed care organizations: a case study. Psychiatr Serv. 1998;49: 1303-1308.
US Department of Veterans Affairs, Office of Inspector General. Healthcare inspection: alleged mental health access and treatment issues at a VA medical Center (Report No. 11–03021–133). 2012. Accessed September 1, 2022 at https://www.va.gov/oig/pubs/VAOIG-11-03021-133.pdf.
Boden MT, Smith CA, Klocek JW, Trafton JA. Mental Health Treatment Quality, Access, and Satisfaction: Optimizing Staffing in an Era of Fiscal Accountability. Psychiatr Serv. 2018;70(3);168-175. doi: https://doi.org/10.1176/appi.ps.201800229.
Boden M, Smith CA, Trafton JA. Investigation of Population-Based Mental Health Staffing and Efficiency-Based Mental Health Productivity Using an Information-Theoretic Approach. PLoS One. 2021;16(8): e0256268. doi: https://doi.org/10.1371/journal.pone.0256268
Smith CA, Boden MT, Trafton JA. Outpatient Provider Staffing Ratios: Binary Recursive Models Associated with Quality, Access, and Satisfaction. Psychol Serv. 2021. Advance online publication. doi:https://doi.org/10.1037/ser0000449.
Atkinson JA, Page A, Heffernan M, McDonnell G, Prodan A, Campos B, Meadows G, Hickie IB. The impact of strengthening mental health services to prevent suicidal behavior. Aust N Z J Psychiatry. 2019;53(7):642-650.
Katz IR, Kemp JE, Blow FC, McCarthy JF, Bossarte RM. Changes in Suicide Rates and in Mental Health Staffing in the Veterans Health Administration, 2005-2009. Psychiatr Serv. 2013;64: 620-625. doi: https://doi.org/10.1176/appi.ps.201200253
Feyman, Y, Figueroa, SM, Yuan, Y, Price ME, Kabdiyeva A, Nebeker JR, Ward MC, Shafer PR, Pizer SD, Strombotne KL. Effect of Mental Health Staffing Inputs on Suicide-Related Events. Health Serv Res. 2022; 1- 8. doi:https://doi.org/10.1111/1475-6773.14064.
Levine DS, McCarthy JF, Cornwell B, Brockmann L, Pfeiffer PN. Primary Care–Mental Health Integration in the VA Health System: Associations Between Provider Staffing and Quality of Depression Care. Psychiatr Serv. 2017;68(5):476-481. doi:https://doi.org/10.1176/appi.ps.201600186
Cooper DC, Helfrich CD, Thielke SM, Trivedi RB, Nelson KM, Reiber GE, Eugenio EC, Beaver K, Nugent-Carney J, Fan VS. Association Between Mental Health Staffing Level and Primary Care-Mental Health Integration Level on Provision of Depression Care in Veteran’s Affairs Medical Facilities. Adm Policy Ment Health. 2018; 45:131-141. doi: https://doi.org/10.1007/s10488-016-0775-9
Lemke S, Boden MT, Kearney LK, Krahn DD, Neuman MJ, Schmidt EM, Trafton JA. Measurement-based management of mental health quality and access in VHA: SAIL mental health domain. Psychol Serv. 2017;14(1): 1-12.
Schmidt EM, Krahn DD, McGuire MH, Tavakoli S, Wright DM, Solares HE, Lemke S, Trafton J. Using organizational and clinical performance data to increase the value of mental health care. Psychol Serv. 2017;14(1): 13-22.
Merkle EC, Shaffer VA. Binary recursive partitioning: background, methods, and application to psychology. Br J Math Stat Psychol. 2011;64:161-181.
Strobl C, Malley J, Tutz G. An Introduction to Recursive Partitioning: Rationale, Application, and Characteristics of Classification and Regression Trees, Bagging and Random Forests. Psychol Methods. 2009;14(4): 334-348. doi:https://doi.org/10.1037/a0016973.
Breiman L, Friedman JH, Olshen, RA, Stone, CJ. Classification and regression trees. Boca Raton, FL: Chapman & Hall/CRC Press; 1984.
Hothorn T, Hornik K, Zeileis A. Unbiased recursive partitioning: a conditional inference framework. J Comput Graph Stat. 2006;15:651-674.
Fu W, Simonoff JS. Unbiased regression tress for longitudinal and clustered data. Comput Stat Data Anal. 2015;88:53-74.
Kundu MG, Harezlak J. Regression trees for longitudinal data with baseline covariates. Biostatist Epidemiol. 2019; 3(1): 1-22. doi:https://doi.org/10.1080/24709360.2018.1557797
Sela RJ, Simonoff JS. RE-EM trees: a data mining approach for longitudinal and clustered data. Mach Learn. 2012; 86: 169-207.
R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3–900051–07–0, http://www.R-project.org. 2019.
Revelle W. psych: Procedures for Personality and Psychological Research. 2015. Northwestern University, Evanston. R package version 1.5.8.
Hothorn T, Hornik K, Strobl C, Zeileis A. party: A Laboratory for Recursive Partytioning. 2019. R package version 1.3–3.
Sela RJ, Simonoff JS. REEMtree: Regression Trees with Random Effects for Longitudinal (Panel) Data. 2011. R package version 0.90.3.
Wickham H. ggplot2: Elegant Graphics for Data Analysis. 2016. Springer-Verlag New York.
Riessman CK. (1970). The supply-demand dilemma in community mental health centers. Am J Orthopsychiatry. 1970;40: 858-869.
Jevons WS. The Coal Question. 1865. Access August 28, 2022 at https://oll.libertyfund.org/title/jevons-the-coal-question.
Stefos T, Burgess JF, Cohen JP, Lehner L, Moran E. Dynamics of the Mental Health Workforce: Investigating the Composition of Physicians and Other Health Providers. Health Care Manag Sci 2012;15:373-384. doi:https://doi.org/10.1007/s10729-012-9203-1
Rosen CS, Morland LA, Glassman LH, et al. Virtual Mental Health Care in the Veterans Health Administration’s Immediate Response to Coronavirus Disease-19. Am Psychol. 2021;76(1):26-38. doi:https://doi.org/10.1037/amp0000751
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have do not have a conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Smith, C., Boden, M. & Trafton, J. Veterans Health Administration Outpatient Psychiatry Staffing Model: Longitudinal Analysis on Mental Health Performance. J GEN INTERN MED 38 (Suppl 3), 814–820 (2023). https://doi.org/10.1007/s11606-023-08119-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11606-023-08119-1