Abstract
Background
Reproducibility is a defining feature of a scientific discovery. Reproducibility can be at different levels for different types of study. The purpose of the Human Cell Atlas (HCA) project is to build maps of molecular signatures of all human cell types and states to serve as references for future discoveries. Constructing such a complex reference atlas must involve the assembly and aggregation of data from multiple labs, probably generated with different technologies. It has much higher requirements on reproducibility than individual research projects. To add another layer of complexity, the bioinformatics procedures involved for single-cell data have high flexibility and diversity. There are many factors in the processing and analysis of single-cell RNA-seq data that can shape the final results in different ways.
Methods
To study what levels of reproducibility can be reached in current practices, we conducted a detailed reproduction study for a well-documented recent publication on the atlas of human blood dendritic cells as an example to break down the bioinformatics steps and factors that are crucial for the reproducibility at different levels.
Results
We found that the major scientific discovery can be well reproduced after some efforts, but there are also some differences in some details that may cause uncertainty in the future reference. This study provides a detailed case observation on the on-going discussions of the type of standards the HCA community should take when releasing data and publications to guarantee the reproducibility and reliability of the future atlas.
Conclusion
Current practices of releasing data and publications may not be adequate to guarantee the reproducibility of HCA. We propose building more stringent guidelines and standards on the information that needs to be provided along with publications for projects that evolved in the HCA program.
Article PDF
References
Watson, J. D. (1990) The human genome project: past, present, and future. Science, 248, 44–49
Collins, F. S., Morgan, M. and Patrinos, A. (2003) The Human Genome Project: lessons from large-scale biology. Science, 300, 286–290
Gibbs, R. A., Belmont, J. W., Hardenbol, P., Willis, T. D., Yu, F., Zhang, H., Zeng, C., Matsuda, I., Fukushima, Y., Macer, D. R., et al. (2003) The International HapMap Project. Nature, 426, 789–796
Feingold, E. A., Good, P. J., Guyer, M. S., Kamholz, S., Liefer, L., Wetterstrand, K., Collins, F. S., Gingeras, T. R., Kampa, D., Sekinger, E. A. et al. (2004) The ENCODE (ENCyclopedia of DNA Elements) project. Science, 306, 636–640
Haines, J. L., Hauser, M. A., Schmidt, S., Scott, W. K., Olson, L. M., Gallins, P., Spencer, K. L., Kwan, S. Y., Noureddine, M., Gilbert, J. R., et al. (2005) Complement factor H variant increases the risk of age-related macular degeneration. Science, 308, 419–421
The 1000 Genomes Project Consortium. (2010) A map of human genome variation from population-scale sequencing. Nature, 467, 1061–1073
The 1000 Genomes Project Consortium. (2012) An integrated map of genetic variation from 1092 human genomes. Nature, 491, 56–65
Kellis, M., Wold, B., Snyder, M. P., Bernstein, B. E., Kundaje, A., Marinov, G. K., Ward, L. D., Birney, E., Crawford, G. E., Dekker, J., et al. (2014) Defining functional DNA elements in the human genome. Proc. Natl. Acad. Sci. USA, 111, 6131–6138
The Human Cell Atlas Meeting Participants. (2017) The Human Cell Atlas. eLife, 6, e27041
Rozenblatt-Rosen, O., Stubbington, M. J. T., Regev, A. and Teichmann, S. A. (2017) The Human Cell Atlas: from vision to reality. Nature, 550, 451–453
Svensson, V., Vento-Tormo, R. and Teichmann, S. A. (2018) Exponential scaling of single-cell RNA-seq in the past decade. Nat. Protoc., 13, 599–604
Cusanovich, D. A., Daza, R., Adey, A., Pliner, H. A., Christiansen, L., Gunderson, K. L., Steemers, F. J., Trapnell, C. and Shendure, J. (2015) Multiplex single cell profiling of chromatin accessibility by combinatorial cellular indexing. Science, 348, 910–914
Nagano, T., Lubling, Y., Stevens, T. J., Schoenfelder, S., Yaffe, E., Dean, W., Laue, E. D., Tanay, A. and Fraser, P. (2013) Single-cell Hi-C reveals cell-to-cell variability in chromosome structure. Nature, 502, 59–64
Zenobi, R. (2013) Single-cell metabolomics: analytical and biological perspectives. Science, 342, 1243259
Crosetto, N., Bienko, M. and van Oudenaarden, A. (2015) Spatially resolved transcriptomics and beyond. Nat. Rev. Genet., 16, 57–66
Lein, E., Borm, L. E. and Linnarsson, S. (2017) The promise of spatial transcriptomics for neuroscience in the era of molecular cell typing. Science, 358, 64–69
Zhong, S., Zhang, S., Fan, X., Wu, Q., Yan, L., Dong, J., Zhang, H., Li, L., Sun, L., Pan, N., et al. (2018) A single-cell RNA-seq survey of the developmental landscape of the human prefrontal cortex. Nature, 555, 524–528
Muraro, M. J., Dharmadhikari G., Grün, D., Groen, N., Dielen, T., Jansen, E., van Gurp, L., Engelse, M. A., Carlotti, F., de Koning, E. J. P. et al. (2016) A single-cell transcriptome atlas of the human pancreas. Cell Syst., 3, 385–394 e3
Darmanis, S., Sloan, S. A., Zhang, Y., Enge, M., Caneda, C., Shuer, L. M., Hayden Gephart, M. G., Barres, B. A. and Quake, S. R. (2015) A survey of human brain transcriptome diversity at the single cell level. Proc. Natl. Acad. Sci. USA, 112, 7285–7290
Macosko, E. Z., Basu, A., Satija, R., Nemesh, J., Shekhar, K., Goldman, M., Tirosh, I., Bialas, A. R., Kamitaki, N., Martersteck, E. M., et al. (2015) Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets. Cell, 161, 1202–1214
Stubbington, M. J. T., Rozenblatt-Rosen, O., Regev, A. and Teichmann, S. A. (2017) Single-cell transcriptomics to explore the immune system in health and disease. Science, 358, 58–63
Jaitin, D. A., Kenigsberg, E., Keren-Shaul, H., Elefant, N., Paul, F., Zaretsky, I., Mildner, A., Cohen, N., Jung, S., Tanay, A., et al. (2014) Massively parallel single-cell RNA-seq for marker-free decomposition of tissues into cell types. Science, 343, 776–779
Villani, A. C., Satija, R., Reynolds, G., Sarkizova, S., Shekhar, K., Fletcher, J., Griesbeck, M., Butler, A., Zheng, S., Lazo, S., et al. (2017) Single-cell RNA-seq reveals new types of human blood dendritic cells, monocytes, and progenitors. Science, 356, eaah4573
Data Coordination–Human Cell Atlas (https://doi.org/www.humancellatlas.org/data-sharing)
Butler, A., Hoffman, P., Smibert, P., Papalexi, E. and Satija, R. (2018) Integrating single-cell transcriptomic data across different conditions, technologies, and species. Nat. Biotechnol., 36, 411–420
van der Maaten, L. and Hinton, G. (2008) Visualizing data using t- SNE. J. Mach. Learn. Res., 9, 2579–2605.
Wattenberg, F. V. M. and Johnson, I. (2016) How to use t-SNE effectively. Distillhttps://doi.org/10.23915/distill.00002
Yuansheng Zhou, T. O. S. (2018) Using global t-SNE to preserve inter-cluster data structure. bioRxiv, Doi: https://doi.org/10.1101/331611
Kobak, D. and Berens, P. (2018) The art of using t-SNE for singlecell transcriptomics. bioRxiv, Doi: https://doi.org/10.1101/453449
Baker, M. (2016) Is there a reproducibility crisis? Nature. 533, 452–454
Berg, J. (2018) Progress on reproducibility. Science, 359, 9
Stark, P. B. (2018) Before reproducibility must come preproducibility. Nature, 557, 613
Acknowledgements
We thank Nir Hacohen, Alexandra-Chloé Villani and Orit Rozenblatt-Rosen (authors of the original paper) for their helpful discussion. This work is supported by CZI Human Cell Atlas Pilot Project and the National Natural Science Foundation of China (Nos. 61673231 and 61721003).
Author information
Authors and Affiliations
Corresponding author
Additional information
Author summary: The Human Cell Atlas (HCA) project aims to build a comprehensive set of “Google Maps” for all cell types of a healthy human body. As a reference for future studies, it is important to guarantee that all maps are reproducible by third-party labs at high fidelity. Building the cell atlas or its parts involves complex bioinformatics procedures besides the bench work. Subtle differences in the bioinformatics processing may cause big differences in the resulted maps, but many current publications paid less attention to reporting the details of bioinformatics processing than the bench protocols. To study how the reproducibility can be reached under current practices, we conducted a detailed case study of a recent cell atlas work. The experiment provides observations helpful for safeguarding the reproducibility of the future HCA projects.
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Hua, K., Zhang, X. A case study on the detailed reproducibility of a Human Cell Atlas project. Quant Biol 7, 162–169 (2019). https://doi.org/10.1007/s40484-018-0164-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40484-018-0164-3