Statistics in Biosciences

, Volume 8, Issue 1, pp 77–98 | Cite as

Permutation Testing for Treatment–Covariate Interactions and Subgroup Identification

  • Jared C. Foster
  • Bin Nan
  • Lei Shen
  • Niko Kaciroti
  • Jeremy M. G. Taylor


We consider the problem of using permutation-based methods to test for treatment–covariate interactions from randomized clinical trial data. Testing for interactions is common in the field of personalized medicine, as subgroups with enhanced treatment effects arise when treatment-by-covariate interactions exist. Asymptotic tests can often be performed for simple models, but in many cases, more complex methods are used to identify subgroups, and non-standard test statistics proposed, and asymptotic results may be difficult to obtain. In such cases, it is natural to consider permutation-based tests, which shuffle selected parts of the data in order to remove one or more associations of interest; however, in the case of interactions, it is generally not possible to remove only the associations of interest by simple permutations of the data. We propose a number of alternative permutation-based methods, designed to remove only the associations of interest, but preserving other associations. These methods estimate the interaction term in a model, then create data that “looks like” the original data except that the interaction term has been permuted. The proposed methods are shown to outperform traditional permutation methods in a simulation study. In addition, the proposed methods are illustrated using data from a randomized clinical trial of patients with hypertension.


Permutation tests Treatment–covariate interactions Subgroup analysis Personalized medicine 



This research was partially supported by a Grant from Eli Lilly, Grant DMS-1007590 from the National Science Foundation, Grants CA083654 and AG036802 from the National Institutes of Health (NIH), and the Intramural Research Program of the NIH, Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD). We also utilized the high-performance computational capabilities of the Biowulf Linux cluster at NIH, Bethesda, MD. (


  1. 1.
    Assmann SF, Pocock SJ, Enos LE, Kasten LE (2000) Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet 355(9209):1064–1069CrossRefGoogle Scholar
  2. 2.
    Brookes ST, Whitley E, Peters TJ, Mulheran PA, Egger M, Davey Smith G (2001) Subgroup analyses in randomised controlled trials: quantifying the risks of false-positives and false-negatives. Health Technol Assess (Winchester, England) 5(33):1–56Google Scholar
  3. 3.
    Bůžková P, Lumley T, Rice K (2011) Permutation and parametric bootstrap tests for gene-gene and gene-environment interactions. Ann Human Genet 75(1):36–45. doi: 10.1111/j.1469-1809.2010.00572.x CrossRefGoogle Scholar
  4. 4.
    Cai T, Tian L, Wong PH, Wei LJ (2011) Analysis of randomized comparative clinical trial data for personalized treatment selections. Biostatistics 12(2):270–282CrossRefGoogle Scholar
  5. 5.
    Edgington ES (1986) Randomization tests. Marcel Dekker Inc., New YorkzbMATHGoogle Scholar
  6. 6.
    Foster JC, Taylor JMG, Ruberg SJ (2011) Subgroup identification from randomized clinical trial data. Stat Med 30(24):2867–2880MathSciNetCrossRefGoogle Scholar
  7. 7.
    Good P (2000) Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer, BerlinCrossRefzbMATHGoogle Scholar
  8. 8.
    Julius S, Nesbitt SD, Egan BM, Weber MA, Michelson EL, Kaciroti N, Black HR, Grimm RH, Messerli FH, Oparil S, Schork MA (2006) Feasibility of treating prehypertension with an angiotensin-receptor blocker. New Engl J Med 354(16):1685–1697CrossRefGoogle Scholar
  9. 9.
    Lipkovich I, Dmitrienko A, Denne J, Enas G (2011) Subgroup identification based on differential effect search—a recursive partitioning method for establishing response to treatment in patient subpopulations. Stat Med 30(21):2601–2621MathSciNetGoogle Scholar
  10. 10.
    Peto R, Collins R, Gray RN (1995) Large-scale randomized evidence: large, simple trials and overviews of trials. J Clin Epidemiol 48(1):23–40CrossRefGoogle Scholar
  11. 11.
    Potthoff RF, Peterson BL, George SL (2001) Detecting treatment-by-centre interaction in multi-centre clinical trials. Stat Med 20(2):193–213CrossRefGoogle Scholar
  12. 12.
    Ruberg SJ, Chen L, Wang Y (2010) The mean does not mean as much anymore: finding sub-groups for tailored therapeutics. Clin Trials (London, England) 7(5):574–583CrossRefGoogle Scholar
  13. 13.
    Simon N, Tibshirani R (2012) A permutation approach to testing interactions in many dimensions. arXiv:1206.6519v1
  14. 14.
    Yusuf S, Wittes J, Probstfield J, Tyroler HA (1991) Analysis and interpretation of treatment effects in subgroups of patients in randomized clinical trials. JAMA 266(1):93–98CrossRefGoogle Scholar
  15. 15.
    Zhang B, Tsiatis AA, Laber EB, Davidian M (2012) A robust method for estimating optimal treatment regimes. Biometrics 68(4):1010–1018MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© International Chinese Statistical Association (outside the US) 2015

Authors and Affiliations

  • Jared C. Foster
    • 1
  • Bin Nan
    • 2
  • Lei Shen
    • 3
  • Niko Kaciroti
    • 2
  • Jeremy M. G. Taylor
    • 2
  1. 1.Biostatistics and Bioinformatics Branch, Division of Intramural Population Health Research, Eunice Kennedy Shriver National Institute of Child Health and Human DevelopmentNational Institutes of HealthBethesdaUSA
  2. 2.Department of BiostatisticsUniversity of MichiganAnn ArborUSA
  3. 3.Global Statistical Sciences, Advanced AnalyticsEli LillyIndianapolisUSA

Personalised recommendations