A case study on the detailed reproducibility of a Human Cell Atlas project

  • Kui Hua
  • Xuegong ZhangEmail author



Reproducibility is a defining feature of a scientific discovery. Reproducibility can be at different levels for different types of study. The purpose of the Human Cell Atlas (HCA) project is to build maps of molecular signatures of all human cell types and states to serve as references for future discoveries. Constructing such a complex reference atlas must involve the assembly and aggregation of data from multiple labs, probably generated with different technologies. It has much higher requirements on reproducibility than individual research projects. To add another layer of complexity, the bioinformatics procedures involved for single-cell data have high flexibility and diversity. There are many factors in the processing and analysis of single-cell RNA-seq data that can shape the final results in different ways.


To study what levels of reproducibility can be reached in current practices, we conducted a detailed reproduction study for a well-documented recent publication on the atlas of human blood dendritic cells as an example to break down the bioinformatics steps and factors that are crucial for the reproducibility at different levels.


We found that the major scientific discovery can be well reproduced after some efforts, but there are also some differences in some details that may cause uncertainty in the future reference. This study provides a detailed case observation on the on-going discussions of the type of standards the HCA community should take when releasing data and publications to guarantee the reproducibility and reliability of the future atlas.


Current practices of releasing data and publications may not be adequate to guarantee the reproducibility of HCA. We propose building more stringent guidelines and standards on the information that needs to be provided along with publications for projects that evolved in the HCA program.


Human Cell Atlas reproducibility single cell bioinformatics 



We thank Nir Hacohen, Alexandra-Chloé Villani and Orit Rozenblatt-Rosen (authors of the original paper) for their helpful discussion. This work is supported by CZI Human Cell Atlas Pilot Project and the National Natural Science Foundation of China (Nos. 61673231 and 61721003).

Supplementary material

40484_2018_164_MOESM1_ESM.pdf (6.2 mb)
A case study on the detailed reproducibility of a human cell atlas project


  1. 1.
    Watson, J. D. (1990) The human genome project: past, present, and future. Science, 248, 44–49CrossRefGoogle Scholar
  2. 2.
    Collins, F. S., Morgan, M. and Patrinos, A. (2003) The Human Genome Project: lessons from large-scale biology. Science, 300, 286–290CrossRefGoogle Scholar
  3. 3.
    Gibbs, R. A., Belmont, J. W., Hardenbol, P., Willis, T. D., Yu, F., Zhang, H., Zeng, C., Matsuda, I., Fukushima, Y., Macer, D. R., et al. (2003) The International HapMap Project. Nature, 426, 789–796CrossRefGoogle Scholar
  4. 4.
    Feingold, E. A., Good, P. J., Guyer, M. S., Kamholz, S., Liefer, L., Wetterstrand, K., Collins, F. S., Gingeras, T. R., Kampa, D., Sekinger, E. A. et al. (2004) The ENCODE (ENCyclopedia of DNA Elements) project. Science, 306, 636–640CrossRefGoogle Scholar
  5. 5.
    Haines, J. L., Hauser, M. A., Schmidt, S., Scott, W. K., Olson, L. M., Gallins, P., Spencer, K. L., Kwan, S. Y., Noureddine, M., Gilbert, J. R., et al. (2005) Complement factor H variant increases the risk of age-related macular degeneration. Science, 308, 419–421CrossRefGoogle Scholar
  6. 6.
    The 1000 Genomes Project Consortium. (2010) A map of human genome variation from population-scale sequencing. Nature, 467, 1061–1073Google Scholar
  7. 7.
    The 1000 Genomes Project Consortium. (2012) An integrated map of genetic variation from 1092 human genomes. Nature, 491, 56–65Google Scholar
  8. 8.
    Kellis, M., Wold, B., Snyder, M. P., Bernstein, B. E., Kundaje, A., Marinov, G. K., Ward, L. D., Birney, E., Crawford, G. E., Dekker, J., et al. (2014) Defining functional DNA elements in the human genome. Proc. Natl. Acad. Sci. USA, 111, 6131–6138CrossRefGoogle Scholar
  9. 9.
    The Human Cell Atlas Meeting Participants. (2017) The Human Cell Atlas. eLife, 6, e27041Google Scholar
  10. 10.
    Rozenblatt-Rosen, O., Stubbington, M. J. T., Regev, A. and Teichmann, S. A. (2017) The Human Cell Atlas: from vision to reality. Nature, 550, 451–453CrossRefGoogle Scholar
  11. 11.
    Svensson, V., Vento-Tormo, R. and Teichmann, S. A. (2018) Exponential scaling of single-cell RNA-seq in the past decade. Nat. Protoc., 13, 599–604CrossRefGoogle Scholar
  12. 12.
    Cusanovich, D. A., Daza, R., Adey, A., Pliner, H. A., Christiansen, L., Gunderson, K. L., Steemers, F. J., Trapnell, C. and Shendure, J. (2015) Multiplex single cell profiling of chromatin accessibility by combinatorial cellular indexing. Science, 348, 910–914CrossRefGoogle Scholar
  13. 13.
    Nagano, T., Lubling, Y., Stevens, T. J., Schoenfelder, S., Yaffe, E., Dean, W., Laue, E. D., Tanay, A. and Fraser, P. (2013) Single-cell Hi-C reveals cell-to-cell variability in chromosome structure. Nature, 502, 59–64CrossRefGoogle Scholar
  14. 14.
    Zenobi, R. (2013) Single-cell metabolomics: analytical and biological perspectives. Science, 342, 1243259CrossRefGoogle Scholar
  15. 15.
    Crosetto, N., Bienko, M. and van Oudenaarden, A. (2015) Spatially resolved transcriptomics and beyond. Nat. Rev. Genet., 16, 57–66CrossRefGoogle Scholar
  16. 16.
    Lein, E., Borm, L. E. and Linnarsson, S. (2017) The promise of spatial transcriptomics for neuroscience in the era of molecular cell typing. Science, 358, 64–69CrossRefGoogle Scholar
  17. 17.
    Zhong, S., Zhang, S., Fan, X., Wu, Q., Yan, L., Dong, J., Zhang, H., Li, L., Sun, L., Pan, N., et al. (2018) A single-cell RNA-seq survey of the developmental landscape of the human prefrontal cortex. Nature, 555, 524–528CrossRefGoogle Scholar
  18. 18.
    Muraro, M. J., Dharmadhikari G., Grün, D., Groen, N., Dielen, T., Jansen, E., van Gurp, L., Engelse, M. A., Carlotti, F., de Koning, E. J. P. et al. (2016) A single-cell transcriptome atlas of the human pancreas. Cell Syst., 3, 385–394 e3CrossRefGoogle Scholar
  19. 19.
    Darmanis, S., Sloan, S. A., Zhang, Y., Enge, M., Caneda, C., Shuer, L. M., Hayden Gephart, M. G., Barres, B. A. and Quake, S. R. (2015) A survey of human brain transcriptome diversity at the single cell level. Proc. Natl. Acad. Sci. USA, 112, 7285–7290CrossRefGoogle Scholar
  20. 20.
    Macosko, E. Z., Basu, A., Satija, R., Nemesh, J., Shekhar, K., Goldman, M., Tirosh, I., Bialas, A. R., Kamitaki, N., Martersteck, E. M., et al. (2015) Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets. Cell, 161, 1202–1214CrossRefGoogle Scholar
  21. 21.
    Stubbington, M. J. T., Rozenblatt-Rosen, O., Regev, A. and Teichmann, S. A. (2017) Single-cell transcriptomics to explore the immune system in health and disease. Science, 358, 58–63CrossRefGoogle Scholar
  22. 22.
    Jaitin, D. A., Kenigsberg, E., Keren-Shaul, H., Elefant, N., Paul, F., Zaretsky, I., Mildner, A., Cohen, N., Jung, S., Tanay, A., et al. (2014) Massively parallel single-cell RNA-seq for marker-free decomposition of tissues into cell types. Science, 343, 776–779CrossRefGoogle Scholar
  23. 23.
    Villani, A. C., Satija, R., Reynolds, G., Sarkizova, S., Shekhar, K., Fletcher, J., Griesbeck, M., Butler, A., Zheng, S., Lazo, S., et al. (2017) Single-cell RNA-seq reveals new types of human blood dendritic cells, monocytes, and progenitors. Science, 356, eaah4573CrossRefGoogle Scholar
  24. 24.
    Data Coordination–Human Cell Atlas (
  25. 25.
    Butler, A., Hoffman, P., Smibert, P., Papalexi, E. and Satija, R. (2018) Integrating single-cell transcriptomic data across different conditions, technologies, and species. Nat. Biotechnol., 36, 411–420CrossRefGoogle Scholar
  26. 26.
    van der Maaten, L. and Hinton, G. (2008) Visualizing data using t- SNE. J. Mach. Learn. Res., 9, 2579–2605.Google Scholar
  27. 27.
    Wattenberg, F. V. M. and Johnson, I. (2016) How to use t-SNE effectively. Distill CrossRefGoogle Scholar
  28. 28.
    Yuansheng Zhou, T. O. S. (2018) Using global t-SNE to preserve inter-cluster data structure. bioRxiv, Doi:
  29. 29.
    Kobak, D. and Berens, P. (2018) The art of using t-SNE for singlecell transcriptomics. bioRxiv, Doi: Google Scholar
  30. 30.
    Baker, M. (2016) Is there a reproducibility crisis? Nature. 533, 452–454CrossRefGoogle Scholar
  31. 31.
    Berg, J. (2018) Progress on reproducibility. Science, 359, 9CrossRefGoogle Scholar
  32. 32.
    Stark, P. B. (2018) Before reproducibility must come preproducibility. Nature, 557, 613CrossRefGoogle Scholar

Copyright information

© Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.MOE Key Laboratory of Bioinformatics Division and Center for Synthetic & System BiologyBNRISTBeijingChina
  2. 2.Department of AutomationTsinghua UniversityBeijingChina
  3. 3.School of Life SciencesTsinghua UniversityBeijingChina

Personalised recommendations