Abstract
Objectives
Due to time and financial limitations, most randomized controlled trials (RCTs) are conducted employing non-random sampling techniques. Although valuable, when the unique characteristics of a non-random sample unknowingly interact with the treatment, the results of the RCT could become biased. Nevertheless, the amount of bias remains unexamined.
Methods
The current study evaluated if non-random sampling techniques could bias the slope coefficients of an RCT when an interaction exists between the treatment and a characteristic in the population using two simulation analyses.
Results
The results suggested that the sampling distributions of slope coefficients from an RCT — across random specifications — expand drastically when (1) an interaction between the treatment and a characteristic in the population exists and (2) the non-random sample has unique scores on that characteristic.
Conclusions
Considering these findings, four recommendations are made for scholars currently or intending to conduct a RCT employing non-random sampling techniques.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
“Tr” represents treatment, “M” represents the moderator, and “Y” represents the dependent variable. Although not represented in the equation, this data specification generates an unmoderated slope coefficient of 1.00 between the treatment and the dependent variable.
The population was demarcated into treatment and control cases (250,000 each) and a random sample of 250 cases were selected from the treatment cases in the population and a random sample of 250 cases were selected from the control cases in the population.
“Tr” represents treatment, “M” represents the moderator, and “Qk” represents the specified quartile.
These values were specified to constraint the simulation analysis to potentially realistic situations (Morris et al., 2019). Nevertheless, all of the R-code is provided to allow you to edit the constraints of the simulation analysis. It should be cautioned, however, that increasing the number of populations substantively alters the time required to complete the simulation analysis.
“Tr” represents treatment, “M” represents the moderator, and “Y” represents the dependent variable. U(n, [−5, 5]) indicates that n(1) value was drawn from a uniform distribution ranging between −5 and 5.
References
Ahlin, E. M. (2015). Conducting randomized controlled trials with offenders in an administrative setting. American Journal of Evaluation, 36(2), 164–178.
Andrews, D. A., &Dowden, C. (2006). Risk principle of case classification in correctional treatment: A meta-analytic investigation. International Journal of Offender Therapy and Comparative Criminology, 50(1), 88–100.
Berk, R. A. (2005). Randomized experiments as the bronze standard. Journal of Experimental Criminology, 1(4), 417–433.
Bickman, L., &Reich, S. M. (2008). Randomized controlled trials. What counts as credible evidence in applied research and evaluation practice?, 51.
Bonta, J., &Andrews, D. A. (2016). The psychology of criminal conduct. Taylor &Francis.
Braga, A. A., Weisburd, D. L., Waring, E. J., Mazerolle, L. G., Spelman, W., &Gajewski, F. (1999). Problem-oriented policing in violent crime places: A randomized controlled experiment. Criminology, 37(3), 541–580.
Campbell, D., &Stanley, J. (1963). Experimental and quasi-experimental designs for research. Rand-McNally.
Davies, T., &Bowers, K. J. (2018). Street networks and crime. In The Oxford handbook of environmental criminology.
Ellenberg, J. H. (1994). Selection bias in observational and experimental studies. Statistics in Medicine, 13(5-7), 557–567.
Groff, E. R., &Lockwood, B. (2014). Criminogenic facilities and crime across street segments in Philadelphia: Uncovering evidence about the spatial extent of facility influence. Journal of Research in Crime and Delinquency, 51(3), 277–314.
Hinkle, J. C., Weisburd, D., Telep, C. W., &Petersen, K. (2020). Problem-oriented policing for reducing crime and disorder: An updated systematic review and meta-analysis. Campbell Systematic Reviews, 16(2), e1089.
Krauss, A. (2018). Why all randomised controlled trials produce biased results. Annals of Medicine, 50(4), 312–322.
Law, A. M. (2015). Simulation modeling and analysis (Vol. 5). McGraw-Hill.
Lewis, P. W. A., &McKenzie, E. (1988). Simulation methodology for statisticians, operations analysts, and engineers (Vol. 1). CRC press.
Lipsey, M. W., &Cullen, F. T. (2007). The effectiveness of correctional rehabilitation: A review of systematic reviews. Annual Review of Law and Social Science., 3, 297–320.
Lowenkamp, C. T., Latessa, E. J., &Holsinger, A. M. (2006). The risk principle in action: What have we learned from 13,676 offenders and 97 correctional programs? Crime &Delinquency, 52(1), 77–93.
Lum, C., &Mazerolle, L. (2014). History of randomized controlled experiments in criminal justice. In G. Bruinsma &D. Weisburd (Eds.), Encyclopedia of criminology and criminal justice. Springer. https://doi.org/10.1007/978-1-4614-5690-2_252.
Maria, A. (1997, December). Introduction to modeling and simulation. In Proceedings of the 29th conference on Winter simulation (pp. 7-13).
Martinson, R. (1974). What works?-Questions and answers about prison reform. The Public Interest, 35, 22.
Morris, T. P., White, I. R., &Crowther, M. J. (2019). Using simulation studies to evaluate statistical methods. Statistics in Medicine, 38(11), 2074–2102.
NIMH Collaborative Data Synthesis for Adolescent Depression Trials Study Team including, Perrino, T., Howe, G., Sperling, A., Beardslee, W., Sandler, I., et al. (2013). Advancing science through collaborative data sharing and synthesis. Perspectives on Psychological Science, 8(4), 433–444.
Sampson, R. J. (2010). Gold standard myths: Observations on the experimental turn in quantitative criminology. Journal of Quantitative Criminology, 26(4), 489–500.
Shadish, W. R., Cook, T. D., &Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Singleton Jr., R. A., &Straits, B. C. (2010). Approaches to social research. Oxford Univ. Press.
Wang, R., &Ware, J. H. (2013). Detecting moderator effects using subgroup analyses. Prevention Science, 14(2), 111–120.
Weisburd, D., &Eck, J. E. (2004). What can police do to reduce crime, disorder, and fear? The Annals of the American Academy of Political and Social Science, 593(1), 42–65.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
ESM 1
(r 29 kb)
Rights and permissions
About this article
Cite this article
Silver, I.A., Kelsay, J.D. The moderating effects of population characteristics: a potential biasing factor when employing non-random samples to conduct experimental research. J Exp Criminol 19, 107–118 (2023). https://doi.org/10.1007/s11292-021-09478-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11292-021-09478-7