Skip to main content

Advertisement

SpringerLink
Log in

Does balancing site characteristics result in balanced population characteristics in a cluster-randomized controlled trial?

  • Published:
Health Services and Outcomes Research Methodology Aims and scope Submit manuscript

Abstract

Background

Intervention trials with nested designs seek to balance sites randomized regarding key site characteristics. Among the goals of such site-level balancing is to accrue patient-level equivalence among treatment arms. We investigated patient-level equivalence in a cluster randomized controlled trial, which balanced study waves on site-level characteristics.

Methods

The Behavioral Health Interdisciplinary Program—Collaborative Chronic Care Model project utilized a stepped wedge design to stagger implementation of an evidence-based team-oriented mental health patient management system at 9 Veteran Affairs Medical Centers. Study sites were balanced on eight site-level characteristics over time (3 balanced waves [consecutive time periods] with 3 sites per wave) to minimize trend. Sites were balanced on selected site-level characteristics but not on patient-level variables. We explored internal differences in patient demographics across the three study waves. Eligible patients had at least two visits to a participating mental health clinic in the prior year and did not have a diagnosis of dementia (n = 5,596).

Results

We found modest but statistically significant inter-site differences in age, marital status, ethnicity, service-related disability, mental health hospitalizations, and selected diagnoses by study wave. Although many of the differences in patient demographics by study wave were statistically significant, only a few results were practically meaningful as measured by effect size. A bipolar diagnosis (49.0%, 21.0%, 17.0% in waves 1–3, respectively; Cramer’s V = 0.3124) and Hispanic ethnicity (2.9%, 29.6%, 2.0% in waves 1–3, respectively; Cramer’s V = 0.3949) resulted in differences that were considered a ‘moderate’ effect size. The number of patient characteristics that were both statistically and meaningfully different by study wave among all possible site assignments was comparable to the 34 most balanced site assignments identified in our balancing algorithm.

Conclusions

Using a balancing algorithm to reduce imbalance among site characteristics across time periods did not appear to negatively affect the balance of patient characteristics across sites over time. A site-level balancing algorithm that includes characteristics with a direct relationship to relevant patient-level factors may improve the overall balance across key elements of the study, and aide in the interpretation of results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Availability of data and materials

The datasets generated and/or analyzed during the current study contained protected VA data; as such, these datasets are not publicly available; however, a limited dataset is available from the corresponding author on reasonable request.

Abbreviations

BHIP-CCM:

Behavioral Health Interdisciplinary Program—Chronic Care Model.

SWD:

Stepped wedge design.

VA:

Veterans Affairs.

VAMC:

Veteran Affairs Medical Center.

References

  1. Donner, A., Klar, N.: Design and Analysis of Cluster Randomization Trials in Health Research, 1st edn. Oxford University Press Inc., New York (2000)

    Google Scholar 

  2. Brown, C.A., Lilford, R.J.: The stepped wedge trial design: a systematic review. BMC Med Res Methodol 6, 54 (2006)

    Article  Google Scholar 

  3. Silber, J.H., Rosenbaum, P.R., Trudeau, M.E., Even-Shoshan, O., Chen, W., Zhang, X., et al.: Multivariate matching and bias reduction in the surgical outcomes study. Med Care 39(10), 1048–1064 (2001)

    Article  CAS  Google Scholar 

  4. Blackford, J.U.: Propensity scores: method for matching on multiple variables in down syndrome research. Intellect Dev Disabil 47(5), 348–357 (2009)

    Article  Google Scholar 

  5. Zhan, C., Miller, M.R.: Excess length of stay, charges, and mortality attributable to medical injuries during hospitalization. JAMA 290(14), 1868–1874 (2003)

    Article  CAS  Google Scholar 

  6. Karkouti, K., Beattie, W.S., Dattilo, K.M., McCluskey, S.A., Ghannam, M., Hamdy, A., et al.: A propensity score case-control comparison of aprotinin and tranexamic acid in high-transfusion-risk cardiac surgery. Transfusion 46(3), 327–338 (2006)

    Article  CAS  Google Scholar 

  7. Lew, R.A., Miller, C.J., Kim, B., Wu, H., Stolzmann, K., Bauer, M.S.: A method to reduce imbalance for site-level randomized stepped wedge implementation trial designs. Implement Sci 14(1), 46 (2019)

    Article  Google Scholar 

  8. Rosenbaum, P.R., Rubin, D.B.: The bias due to incomplete matching. Biometrics 41(1), 103–116 (1985)

    Article  CAS  Google Scholar 

  9. Ming, K., Rosenbaum, P.R.: Substantial gains in bias reduction from matching with a variable number of controls. Biometrics 56(1), 118–124 (2000)

    Article  CAS  Google Scholar 

  10. Bauer, M.S., Miller, C., Kim, B., Lew, R., Weaver, K., Coldwell, C., et al.: Partnering with health system operations leadership to develop a controlled implementation trial. Implement Sci 11, 22 (2016)

    Article  Google Scholar 

  11. Bauer, M.S., Miller, C.J., Kim, B., Lew, R., Stolzmann, K., Sullivan, J., et al.: Effectiveness of Implementing a Collaborative Chronic Care Model for Clinician Teams on Patient Outcomes and Health Status in Mental Health: A Randomized Clinical Trial. JAMA Netw Open 2(3), e190230 (2019)

    Article  Google Scholar 

  12. Miller, C.J., Grogan-Kaylor, A., Perron, B.E., Kilbourne, A.M., Woltmann, E., Bauer, M.S.: Collaborative chronic care models for mental health conditions: cumulative meta-analysis and metaregression to guide future research and implementation. Med Care 51(10), 922–930 (2013)

    Article  Google Scholar 

  13. Woltmann, E., Grogan-Kaylor, A., Perron, B., Georges, H., Kilbourne, A.M., Bauer, M.S.: Comparative effectiveness of collaborative chronic care models for mental health conditions across primary, specialty, and behavioral health care settings: systematic review and meta-analysis. Am. J. Psychiatry 169(8), 790–804 (2012)

    Article  Google Scholar 

  14. Silber, J.H., Rosenbaum, P.R., Ross, R.N., Ludwig, J.M., Wang, W., Niknam, B.A., et al.: A hospital-specific template for benchmarking its cost and quality. Health Serv Res 49(5), 1475–1497 (2014)

    Article  Google Scholar 

  15. Zubizarreta, J.R., Reinke, C.E., Kelz, R.R., Silber, J.H., Rosenbaum, P.R.: Matching for Several Sparse Nominal Variables in a Case-Control Study of Readmission Following Surgery. Am Stat 65(4), 229–238 (2011)

    Article  Google Scholar 

  16. US Department of Veterans Affairs.: 172VA10P2: VHA Corporate Data Warehouse – VA. 79 FR 4377; Accessed August 15, 2018

  17. Effect Size: Statistics Solutions: Advancement Through Clarity; 2013 [Available from: http://www.statisticssolutions.com/academic-solutions/resources/directory-of-statistical-analyses/effect-size/

  18. Levine, T.R.H.: C.R. Eta squared, partial eta squared, and misreporting of effect size in communication research. Hum Commun Res 28, 612–625 (2002)

    Article  Google Scholar 

  19. Cohen, J.: Statistical power and analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum Associates, Inc., Hillsdale (1988)

    Google Scholar 

  20. Wennberg, J.E.: Dealing with medical practice variations: a proposal for action. Health Aff. (Millwood) 3(2), 6–32 (1984)

    Article  CAS  Google Scholar 

  21. Luft, H.S.: Becoming accountable-opportunities and obstacles for ACOs. N. Engl. J. Med. 363(15), 1389–1391 (2010)

    Article  CAS  Google Scholar 

  22. Kilbourne, A.M., Neumann, M.S., Pincus, H.A., Bauer, M.S., Stall, R.: Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implement Sci 2, 42 (2007)

    Article  Google Scholar 

  23. Shortell, S.M., Wu, F.M., Lewis, V.A., Colla, C.H., Fisher, E.S.: A taxonomy of accountable care organizations for policy and practice. Health Serv Res 49(6), 1883–1899 (2014)

    PubMed  PubMed Central  Google Scholar 

  24. Fihn, S.D., Francis, J., Clancy, C., Nielson, C., Nelson, K., Rumsfeld, J., et al.: Insights from advanced analytics at the Veterans Health Administration. Health Aff. (Millwood) 33(7), 1203–1211 (2014)

    Article  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

QUERI 15–289.

Author information

Authors and Affiliations

  1. VA Boston Healthcare System, 150 South Huntington Avenue (Mail Stop 152M), 02130, Boston, MA, USA

    Kelly Stolzmann, Robert A. Lew, Christopher J. Miller, Bo Kim, Hongsheng Wu & Mark S. Bauer

  2. VA Boston Healthcare System, Center for Healthcare Organization and Implementation Research, Boston, USA

    Kelly Stolzmann, Robert A. Lew, Christopher J. Miller, Bo Kim & Mark S. Bauer

  3. VA Boston Healthcare System, Massachusetts Veterans Epidemiology Research and Information Center, Boston, USA

    Robert A. Lew & Hongsheng Wu

  4. Department of Psychiatry, Harvard Medical School, Boston, USA

    Christopher J. Miller, Bo Kim & Mark S. Bauer

  5. Wentworth Institute of Technology, Boston, USA

    Hongsheng Wu

Authors
  1. Kelly Stolzmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert A. Lew
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christopher J. Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bo Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hongsheng Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mark S. Bauer
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

RL, CM and BK made substantial contributions to the conception and design of the work. RL and HU made substantial contributions on the acquisition, analysis, and interpretation of the data. MB oversaw all aspects of the conception and design of the study and was a major contributor in writing the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Kelly Stolzmann.

Ethics declarations

Ethics approval and consent to participate

The overall study was reviewed and approved by the VA Central IRB, Project Number 15−08.

Consent for publication

Not applicable.

Conflict of interest

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Trial registration: Hybrid Collaborative Care Randomized Program Evaluation (BHIP-CCM).

NCT02543840 (September 7, 2015), https://clinicaltrials.gov/ct2/show/NCT02543840.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stolzmann, K., Lew, R.A., Miller, C.J. et al. Does balancing site characteristics result in balanced population characteristics in a cluster-randomized controlled trial?. Health Serv Outcomes Res Method 22, 469–478 (2022). https://doi.org/10.1007/s10742-022-00271-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10742-022-00271-1

Keywords

Search

Navigation