The Architecture of SciDB

  • Michael Stonebraker
  • Paul Brown
  • Alex Poliakov
  • Suchi Raman
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6809)

Abstract

SciDB is an open-source analytical database oriented toward the data management needs of scientists. As such it mixes statistical and linear algebra operations with data management ones, using a natural nested multidimensional array data model. We have been working on the code for two years, most recently with the help of venture capital backing. Release 11.06 (June 2011) is downloadable from our website (SciDB.org).

This paper presents the main design decisions of SciDB. It focuses on our decisions concerning a high-level, SQL-like query language, the issues facing our query optimizer and executor and efficient storage management for arrays. The paper also discusses implementation of features not usually present in DBMSs, including version control, uncertainty and provenance.

Keywords

scientific data management multi-dimensional array statistics linear algebra 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
  2. 2.
    Becla, J., Lim, K.-T.: Report from the First Workshop on Extremely Large Databases. Data Science Journal 7 (2008)Google Scholar
  3. 3.
    Szalay, A.: Private communicationGoogle Scholar
  4. 4.
    Branco, M., Cameron, D., Gaidioz, B., Garonne, V., Koblitz, B., Lassnig, M., Rocha, R., Salgado, P., Wenaus, T.: Managing ATLAS data on a petabyte-scale with DQ2. Journal of Physics: Conference Series 119 (2008)Google Scholar
  5. 5.
    Szalay, A.: The Sloan Digital Sky Survey and Beyond. In: SIGMOD Record (June 2008)Google Scholar
  6. 6.
    Cudre-Mauroux, P., et al.: A Demonstration of SciDB: a Science-oriented DBMS. VLDB 2(2), 1534–1537 (2009)Google Scholar
  7. 7.
    Becla, J., Lim, K.-T.: Report from the Second Workshop on Extremely Large Databases, http://www-conf.slac.stanford.edu/xldb08/, http://www.jstage.jst.go.jp/article/dsj/7/0/1/_pdf
  8. 8.
    Becla, J., Lim, K.-T.: Report from the Third Workshop on Extremely Large Databases, http://www-conf.slac.stanford.edu/xldb09/
  9. 9.
    Becla, J., Lim, K.-T.: Report from the Fourth Workshop on Extremely Large Databases, http://www-conf.slac.stanford.edu/xldb10/
  10. 10.
    Cudre-Maroux, P., et al.: SS-DB: A Standard Science DBMS Benchmark (submitted for publication)Google Scholar
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
    Stonebraker, M., Rowe, L.A., Hirohama, M.: The Implementation of POSTGRES. IEEE Transactions on Knowledge and Data Engineering 2(1), 125–142 (1990)CrossRefGoogle Scholar
  16. 16.
  17. 17.
  18. 18.
    Sarawagi, S., Stonebraker, M.: Efficient organization of large multidimensional arrays. In: ICDE, pp. 328–336 (1994), citeseer.ist.psu.edu/article/sarawagi94efficient.html
  19. 19.
    Soroush, E., et al.: ArrayStore: A Storage Manager for Complex Parallel Array Processing. In: Proc. 2011 SIGMOD Conference (2011)Google Scholar
  20. 20.
    Seering, A., et al.: Efficient Versioning for Scientific Arrays (submitted for publication)Google Scholar
  21. 21.
    Mutsuzaki, M., Theobald, M., de Keijzer, A., Widom, J., Agrawal, P., Benjelloun, O., Das Sarma, A., Murthy, R., Sugihara, T.: Trio-One: Layering Uncertainty and Lineage on a Conventional DBMS. In: Proceedings of the 2007 CIDR Conference, Asilomar, CA (January 2007)Google Scholar
  22. 22.
    Wu, E., et al.: The SciDB Provenance System (in preparation) Google Scholar
  23. 23.
    Cohen, J., et al.: Mad Skills: New Analysis Practices for Big Data. In: Proc. 2009 VLDB ConferenceGoogle Scholar
  24. 24.
  25. 25.
  26. 26.
    van Ballegooij, A., Cornacchia, R., de Vries, A.P., Kersten, M.L.: Distribution Rules for Array Database Queries. In: Andersen, K.V., Debenham, J., Wagner, R. (eds.) DEXA 2005. LNCS, vol. 3588, pp. 55–64. Springer, Heidelberg (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Michael Stonebraker
    • 1
  • Paul Brown
    • 1
  • Alex Poliakov
    • 1
  • Suchi Raman
    • 1
  1. 1.Paradigm4, Inc.WalthamUSA

Personalised recommendations