Machine Learning

, Volume 52, Issue 1–2, pp 91–118 | Cite as

Consensus Clustering: A Resampling-Based Method for Class Discovery and Visualization of Gene Expression Microarray Data

  • Stefano Monti
  • Pablo Tamayo
  • Jill Mesirov
  • Todd Golub


In this paper we present a new methodology of class discovery and clustering validation tailored to the task of analyzing gene expression data. The method can best be thought of as an analysis approach, to guide and assist in the use of any of a wide range of available clustering algorithms. We call the new methodology consensus clustering, and in conjunction with resampling techniques, it provides for a method to represent the consensus across multiple runs of a clustering algorithm and to assess the stability of the discovered clusters. The method can also be used to represent the consensus over multiple runs of a clustering algorithm with random restart (such as K-means, model-based Bayesian clustering, SOM, etc.), so as to account for its sensitivity to the initial conditions. Finally, it provides for a visualization tool to inspect cluster number, membership, and boundaries. We present the results of our experiments on both simulated data and real gene expression data aimed at evaluating the effectiveness of the methodology in discovering biologically meaningful clusters.

unsupervised learning class discovery model selection gene expression microarrays 


  1. Banfield, J., & Raftery, A. E. (1993). Model-based Gaussian and non-Gaussian clustering. Biometrics, 49, 803–821.Google Scholar
  2. Bar-Joseph, Z., Demaine, E. D., Gifford, D. K., Hamel, A. M., Jaakkola, T. S., & Srebro, N. (2002). K-ary clustering with optimal leaf ordering for gene expression data. Bioinformatics, to appear.Google Scholar
  3. Ben-Hur, A., Elisseeff, A., & Guyon, I. (2002). A stability based method for discovering structure in clustered data. In Pacific Symposium on Biocomputing 2002, vol. 7, pp. 6–17, Lihue, Hawaii.Google Scholar
  4. Bhattacharjee, A., Richards, W. G., Staunton, J., Li, C., Monti, S., Vasa, P., Ladd, C., Beheshti, J., Bueno, R., Gillette, M., Loda, M., Weber, G., Mark, E. J., Lander, E. S., Wong, W., Johnson, B. E., Golub, T. R., Sugarbaker, D. J., & Meyerson, M. (2001). Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinomas sub-classes. In Proceedings of the National Academy of Sciences, 98:24, 13790–13795.Google Scholar
  5. Bock, H. (1985). On some significance tests in cluster analysis. Journal of Classification, 2, 77–108.Google Scholar
  6. Cheeseman, P., & Stutz, J. (1996), Bayesian classification (AutoClass): Theory and results. In U. M. Fayyad, G. Piatetsky-Shapiro, P. Smyth, & R. Uthurasamy (Eds.), Advances in Knowledge Discovery and Data Mining, pp. 153–180, MIT Press.Google Scholar
  7. Chickering, D. M., & Heckerman, D. (1997). Efficient approximation for the marginal likelihood of Bayesian networks with hidden variables. Machine Learning, 29, 181–212.Google Scholar
  8. Cowell, F. A. (1995). Measuring Inequality. New York: Prentice Hall.Google Scholar
  9. Duda, R. O., & Hart, P. E. (1973). Pattern Classification and Scene Analysis. John Wiley & Sons.Google Scholar
  10. Dudoit, S., & Fridlyand, J. (2002). A prediction-based resampling method for estimating the number of clusters in a dataset. Genome Biology, 3:7,1–21.Google Scholar
  11. Efron, B., & Tibshirani, R. J. (1994). An Introduction to the Bootstrap, No. 57 in Monographs on Statistics and Applied Probability. CRC Press.Google Scholar
  12. Eisen, M. B., Spellman, P. T., Brown, P. O., & Botstein, D. (1998). Cluster analysis and display of genome-wide expression patterns. Proceedings of the National Academy of Sciences, 95, 14863–14868.Google Scholar
  13. Golub, T. R., Slonim, D. K., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J. P., Coller, H., Loh, M., Downing, J., Caligiuri, M., Bloomfield, C., & Lander, E. (1999). Molecular classification of cancer: Class discovery and class prediction by gene expression. Science, 286:5439, 531–537.Google Scholar
  14. Hartigan, J. A. (1978). Asymptotic distributions for clustering criteria. Annals of Statistics, 6:1, 117–131.Google Scholar
  15. Hastie, T., Tibshirani, R., & Friedman, J. (2001). The Elements of Statistical Learning, Statistics. New York: Springer.Google Scholar
  16. Hubert, L., & Arabie, P. (1985). Comparing partitions. Journal of Classification, 2, 193–218.Google Scholar
  17. Jain, A. K., & Dubes, R. C. (1988). Algorithms for Clustering Data. Englewood Cliffs, NJ: Prentice Hall.Google Scholar
  18. Jain, A. K., & Moreau, J. (1988). Bootstrap techniques in cluster analysis. Pattern Recognition, 20, 547–568.Google Scholar
  19. Kass, R. E. & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795.Google Scholar
  20. Kohonen, T. (1990). The self-organizing map. Proceedings of the IEEE, 78:9, 1464–1480.Google Scholar
  21. Kohonen, T. (1997). Self-Organizing Maps, Information Sciences. Springer.Google Scholar
  22. Levine, E., & Domany, E. (2001). Resampling method for unsupervised estimation of cluster validity. Neural Computation, 13:11, 2573–2593.Google Scholar
  23. Milligan, G., & Cooper, M. (1985). An examination of procedures for determining the number of clusters in a data set. Psyochometrika, 50, 159–179.Google Scholar
  24. Milligan, G. & Cooper, M. (1986). Astudy of the comparability of external criteria for hierarchical cluster analysis. Multivariate Behavioral Research, 21, 441–458.Google Scholar
  25. Pomeroy, S., Tamayo, P., Gaasenbeek, M., Angelo, L. M. S. M., McLaughlin, M. E., Kim, J. Y., Goumnerova, L. C., Black, P. M., Lau, C., Allen, J. C., Zagzag, D., Olson, J. M., Curran, T., Wetmore, C., Biegel, J. A., Poggio, T., Mukherjee, S., Rifkin, A., Califano, G., Stolovitzky, D. N., Louis, J. P., Mesirov, E. S., Lander, R., & Golub, T. R. (2002). Gene expression-based classification and outcome prediction of central nervous system embryonal tumors. Nature, 415:6870, 436–442.Google Scholar
  26. Ramaswamy, S., Tamayo, P., Rifkin, R., Mukherjee, S., Yeang, C.-H., Angelo, M., Ladd, C., Reich, M., Latulippe, E., Mesirov, J. P., Poggio, T., Gerald, W., Loda, M., Lander, E. S., & Golub, T. R. (2001). Multi-class cancer diagnosis using tumor gene expression signatures. Proceedings of the National Academy of Sciences, 98:26, 15149–15154.Google Scholar
  27. Ramoni, M., Sebastiani, P., & Kohane, I. S. (2002). Cluster analysis of gene expression dynamics. In Proceedings of the National Academy of Sciences, 99:14, 9121–9126.Google Scholar
  28. Slonim, D. K., Tamayo, P., Mesirov, J. P., Golub, T. R., & Lander, E. S. (2000). Class prediction and discovery using gene expression data. In RECOMB 2000: The Fourth Annual International Conference on Research in Computational Molecular Biology (pp. 263–272), Tokyo, Japan.Google Scholar
  29. Su, A. I., Cooke, M. P., Ching, K. A., Hakak, Y., Walker, J. R., Wiltshire, T., Orth, A. P., Vega, R. G., Sapinoso, L. M., Moqrich, A., Patapoutian, A., Hampton, G. M., Schultz, P. G., & Hogenesch, J. B. (2002). Large-scale analysis of the human and mouse transcriptomes. Proceedings of the National Academy of Sciences, 99:7, 4465–447.Google Scholar
  30. Tamayo, P., Slonim, D., Mesirov, J., Zhu, Q., Kitareewan, S., Dmitrovsky, E., Lander, E. S., & Godlub, T. R. (1999), Interpreting patterns of gene expression with self-organizing maps: Methods and application to hematopoietic differentiation. Proceedings of the National Academy of Sciences, 96, 2907–2912.Google Scholar
  31. Tibshirani, R., Walther, G., Botstein, D., & Brown, P. (2001a). Cluster validation by prediction strength. Unpub-lished manuscript ( Scholar
  32. Tibshirani, R., Walther, G., & Hastie, T. (2001b). Estimating the number of clusters in a dataset via the gap statistic. Journal of the Royal Statistical Society B, 63:2, 411–423.Google Scholar
  33. Titterington, D., Smith, A., & Makov, U. (1985). Statistical Analysis of Finite Mixture Distributions. New York: Wiley.Google Scholar
  34. Todd Golub et. al. (2002). GeneCluster 2.0. genecluster2/gc2.html.Google Scholar
  35. West, M. (2002). Bayesian factor regression models in the Large p, Small n Paradigm. Bayesian Statistics, 7,to appear.Google Scholar
  36. West, M., Blanchette, C., Dressman, H., Huang, E., Ishida, S., Spang, R., Zuzan, H., Olson Jr., J. A., Marks, J. R., & Nevins, J. R. (2001). Predicting the clinical status of human breast cancer by using gene expression profiles. Proceedings of the National Academy of Sciences, 98:20, 11462–11467.Google Scholar
  37. Yeoh, E.-J., Ross, M. E., Shurtleff, S. A., Williams, W. K., Patel, D., Mahfouz, R., Behm, F. G., Raimondi, S. C., Relling, M. V., Patel, A., Cheng, C., Campana, D., Wilkins, D., Zhou, X., Li, J., Liu, H., Pui, C.-H., Evans, W. E., Naeve, C., Wong, L., & Downing, J. R. (2002). Classification, subtype discovery, and prediction of outcome in pediatric acute lymphoblastic leukemia by gene expression profiling. Cancer Cell, 1:2.Google Scholar
  38. Yeung, K. Y., Fraley, C., Murua, A., Raftery, A. E., & Ruzzo, W. L. (2001a). Model-based clustering and data transformations for gene expression data. Bioinformatics, 17:10, 977–987.Google Scholar
  39. Yeung, K. Y., Haynor, D. R., & Ruzzo, W. L. (2001b) Validating clustering for gene expression data. Bioinformatics, 17:4.Google Scholar

Copyright information

© Kluwer Academic Publishers 2003

Authors and Affiliations

  • Stefano Monti
    • 1
  • Pablo Tamayo
    • 1
  • Jill Mesirov
    • 1
  • Todd Golub
    • 1
  1. 1.Whitehead Institute/MIT Center for Genome ResearchCambridgeUSA

Personalised recommendations