Skip to main content
Log in

Modeling variability in treatment effects for cluster randomized controlled trials using by-variable smooth functions in a generalized additive mixed model

Behavior Research Methods Aims and scope Submit manuscript

Cite this article


Variability in treatment effects is common in intervention studies using cluster randomized controlled trial (C-RCT) designs. Such variability is often examined in multilevel modeling (MLM) to understand how treatment effects (TRT) differ based on the level of a covariate (COV), called TRT \(\times \) COV. In detecting TRT \(\times \) COV effects using MLM, relationships between covariates and outcomes are assumed to vary across clusters linearly. However, this linearity assumption may not hold in all applications and an incorrect assumption may lead to biased statistical inference about TRT \(\times \) COV effects. In this study, we present generalized additive mixed model (GAMM) specifications in which cluster-specific functional relationships between covariates and outcomes can be modeled using by-variable smooth functions. In addition, the implementation for GAMM specifications is explained using the mgcv R package (Wood, 2021). The usefulness of the GAMM specifications is illustrated using intervention data from a C-RCT. Results of simulation studies showed that parameters and by-variable smooth functions were recovered well in various multilevel designs and the misspecification of the relationship between covariates and outcomes led to biased estimates of TRT \(\times \) COV effects. Furthermore, this study evaluated the extent to which the GAMM can be treated as an alternative model to MLM in the presence of a linear relationship.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5


  1. We use a notation of a by-variable smooth function as f(x)factor following notations in GAMM (e.g., Wood, 2017, p. 326), which means that a smooth function \(f_{h}\) (where h is an index for a smooth function) is conditional on a factor.

  2. Because a parametric random intercept is considered in Equation 2, the unconditional GAMM is the same as the unconditional MLM.

  3. We use the abbreviation RMSEI to distinguish it from the root mean squared error of an estimator (RMSE) in the simulation study.

  4. In this study, we did not consider relative bias due to its scaling artifact. When true parameters are close to 0 (e.g., \(\tau _{00}=\)0.032, 0.067, and 0.257 and basis coefficients for smooth functions in the simulation study), relative bias will be inflated by the true parameters close to 0 in the denominator even for small differences from the true parameters.

  5. Exceptional cases include bias of \(\widehat{\gamma }_{00}\) and \(\widehat{\gamma }_{02}\) with respect to \(n_{j}\) and \(\widehat{\sigma }^{2}\) with respect to J in MLM; and bias of \(\widehat{\gamma }_{00}\) with respect to J and \(n_{j}\), \(\widehat{\gamma }_{02}\) with respect to J, and \(\widehat{\sigma }^{2}\) with respect to \(n_{j}\) in GAMM.

  6. For a single replication of GAMM estimation under GAMM as a data-generating model in a simulation condition with \(J=80\), about an hour was required on a laptop computer with a 2.8 GHz Intel Core i7 CPU and 16 GB of RAM.


Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Sun-Joo Cho.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cho, SJ., Preacher, K.J., Yaremych, H.E. et al. Modeling variability in treatment effects for cluster randomized controlled trials using by-variable smooth functions in a generalized additive mixed model. Behav Res (2023).

Download citation

  • Accepted:

  • Published:

  • DOI: