Distributed and Parallel Databases

, Volume 30, Issue 5–6, pp 371–384 | Cite as

Data challenges of time domain astronomy

  • Matthew J. GrahamEmail author
  • S. G. Djorgovski
  • Ashish Mahabal
  • Ciro Donalek
  • Andrew Drake
  • Giuseppe Longo


Astronomy has been at the forefront of the development of the techniques and methodologies of data intensive science for over a decade with large sky surveys and distributed efforts such as the Virtual Observatory. However, it faces a new data deluge with the next generation of synoptic sky surveys which are opening up the time domain for discovery and exploration. This brings both new scientific opportunities and fresh challenges, in terms of data rates from robotic telescopes and exponential complexity in linked data, but also for data mining algorithms used in classification and decision making. In this paper, we describe how an informatics-based approach—part of the so-called “fourth paradigm” of scientific discovery—is emerging to deal with these. We review our experiences with the Palomar-Quest and Catalina Real-Time Transient Sky Surveys; in particular, addressing the issue of the heterogeneity of data associated with transient astronomical events (and other sensor networks) and how to manage and analyze it.


Astronomy Time domain Virtual observatory Classification 



This work has been supported in part by the National Science Foundation grants AST-0407448, CNS-0540369, AST-0834235, AST-0909182 and IIS-1118041; the National Aeronautics and Space Administration grant 08-AISR08-0085; and by the Ajax and Fishbein Family Foundations. We are thankful to numerous colleagues in the VO and Astroinformatics community, and to the members of the DPOSS, PQ, and CRTS survey teams, for many useful discussions and interactions through the years. We thank the anonymous referees for their useful comments.


  1. 1.
    Gray, J., Szalay, A.: 2020 computing: science in an exponential world. Nature 440, 23 (2006) CrossRefGoogle Scholar
  2. 2.
    York, D.G., et al. (the SDSS team): The Sloan Digital Sky Survey: technical summary. Astron. J. 120, 1579 (2000) CrossRefGoogle Scholar
  3. 3.
    Djorgovski, S.G., Gal, R., Odewahn, S., de Carvalho, R., Brunner, R., Longo, G., Scaramella, R.: The Palomar Digital Sky Survey (DPOSS). In: Wide Field Surveys in Cosmology, p. 89. Editions Frontieres, Gif sur Yvette (1998) Google Scholar
  4. 4.
    Skrutskie, M., et al. (the 2MASS team): The Two Micron All Sky Survey (2MASS). Astron. J. 131, 1163 (2006) CrossRefGoogle Scholar
  5. 5.
    Djorgovski, S.G., Mahabal, A.A., Drake, A.J., Graham, M.J., Donalek, C.: Sky Surveys, Planets, Stars, and Stellar Systems. Springer, Berlin (2012, in press) Google Scholar
  6. 6.
    Brunner, R.J., Djorgovski, S.G., Szalay, A.S.: Virtual Observatories of the Future. Astronomical Society of the Pacific, San Francisco (2001) Google Scholar
  7. 7.
    National Research Council: New Worlds, New Horizons in Astronomy and Astrophysics, Decadal Survey of Astronomy and Astrophysics Comm. The National Academies Press, Washington (2010) Google Scholar
  8. 8.
    Large Synoptic Sky Survey.
  9. 9.
    Square Kilometer Array.
  10. 10.
    Drake, A.J., et al.: First results from the Catalina Real-Time Transient Survey. Astrophys. J. 696, 870 (2009) CrossRefGoogle Scholar
  11. 11.
    Djorgovski, S.G., et al.: The Catalina Real-Time Transient Survey (CRTS), The First Year of MAXI: Monitoring Variable X-ray Sources. JAXA Special Publ., Tokyo (2011) Google Scholar
  12. 12.
    Mahabal, A.A., et al.: Discovery, classification, and scientific exploration of transient events from the Catalina Real-Time Transient Survey. Bull. Astron. Soc. India 39, 387–408 (2011) Google Scholar
  13. 13.
    Djorgovski, S.G., Mahabal, A., Drake, A., Graham, M., Donalek, C., Williams, R.: Exploring the time domain with Synoptic Sky Surveys. In: Proc. IAU Symp. 285, New Horizons in Time Domain Astronomy, p. 141. Cambridge University Press, Cambridge (2012) Google Scholar
  14. 14.
    Djorgovski, S.G., et al.: Some pattern recognition challenges in data-intensive astronomy. In: Proc. 18th Intl. Conf. on Pattern Recognition, p. 856. IEEE Press, New York (2006) Google Scholar
  15. 15.
    Djorgovski, S.G., Donalek, C., Mahabal, A., Moghaddam, B., Turmon, M., Graham, M., Drake, A., Sharma, N., Chen, Y.: Towards an automated classification of transient events in Synoptic Sky Surveys. In: Proc. CIDU 2011 Conf. (ASA), p. 174 (2011) Google Scholar
  16. 16.
    Donalek, C., Mahabal, A., Djorgovski, S.G., Marney, S., Drake, A., Graham, M., Glikman, E., Williams, R.: New approaches to object classification in Synoptic Sky Surveys. In: AIP Conf. Proc., vol. 1082, p. 252 (2008) CrossRefGoogle Scholar
  17. 17.
  18. 18.
    Cornwell, T.J., van Diepen, G.: Scaling mount exaflop: from the pathfinders to the square kilometre array. In: Proc. SPIE, vol. 7016 (2008) Google Scholar
  19. 19.
    International Virtual Observatory Alliance.
  20. 20.
    Ochsenbein, F., et al.: IVOA recommendation: VOTable format definition version 1.2 (2011). arXiv:1110.0524
  21. 21.
    Graham, M.J., Morris, D., Rixon, G.: IVOA recommendation: VOSpace specification version 1.15 (2011). arXiv:1110.0508
  22. 22.
    Seaman, R., et al.: IVOA recommendation: sky event reporting metadata version 2.0 (2011). arXiv:1110.0523
  23. 23.
    Djorgovski, S.G., et al.: The Palomar-Quest Digital Synoptic Sky Survey. Astron. Nachr. 329, 263 (2008) CrossRefGoogle Scholar
  24. 24.
    Williams, R.D., Djorgovski, S.G., Drake, A.J., Graham, M.J., Mahabal, A.: Skyalert: real-time astronomy for you and your robots. In: Astronomical Data Analysis Software and Systems XVIII, p. 115. Astronomical Society of the Pacific, San Francisco (2009) Google Scholar
  25. 25.
    Philip, N.S., Mahabal, A., Abraham, S., Williams, R., Djorgovski, S.G., Drake, A., Donald, C., Graham, M.: Classification by boosting differences in input vectors. In: Proc. International Workshop on Stellar Spectra Libraries, Ast. Soc. of India Conf. Ser. (ASICS) (2012, in press) Google Scholar
  26. 26.
    Budavari, T.: Probabilistic cross-identification of cosmic events. Astrophys. J. 736, 155–159 (2011) CrossRefGoogle Scholar
  27. 27.
    Palomar-Quest Synoptic Sky Survey data release 1.
  28. 28.
    Kunszt, P.Z., Szalay, A.S., Thakar, A.R.: The hierarchical triangular mesh, mining the sky. In: Proceedings of the MPA/ESO/MPE Workshop Held at Garching, Germany, July 31–August 4 2000, p. 631. Springer, Berlin (2001). Google Scholar
  29. 29.
    Gorski, K.M., et al.: HEALPix: a framework for high-resolution discretization and fast analysis of data distributed on the sphere. Astrophys. J. 622, 759 (2005) CrossRefGoogle Scholar
  30. 30.
    Gray, J., Nieto-Santisteban, M., Szalay, A.S.: The Zones algorithm for finding points-near-a-point or cross-matching spatial datasets (2006). arXiv:cs/0701171
  31. 31.
    Koposov, S., Bartunov, O.: Q3C, Quad Tree Cube—the new sky-indexing concept for huge astronomical catalogues and its realization for main astronomical queries (cone search and Xmatch) in open source database PostgreSQL. In: Astronomical Data Analysis Software and Systems XV, p. 735. Astronomical Society of the Pacific, San Francisco (2006) Google Scholar
  32. 32.
    Richards, J.W., et al.: On machine-learned classification of variable stars with sparse and noisy time-series data. Astrophys. J. 733, 10 (2011) CrossRefGoogle Scholar
  33. 33.
    Schmidt, M., Lipson, H.: Distilling free-form natural laws from experimental data. Science 324(5923), 81–85 (2012) CrossRefGoogle Scholar
  34. 34.
    Graczyk, D., Eyer, L.: The light curve statistical moments analysis: the identification of eclipsing binaries. Acta Astron. 60, 109 (2010) Google Scholar
  35. 35.
    Edelson, R.A., Krolik, J.H.: The discrete correlation function—a new method for analyzing unevenly sampled variability data. Astrophys. J. 333, 646 (1988) CrossRefGoogle Scholar
  36. 36.
    Simonetti, J.H., Cordes, J.M., Heeschen, D.S.: Flicker of extragalactic radio sources at two frequencies. Astrophys. J. 296, 46 (1985) CrossRefGoogle Scholar
  37. 37.
    Jackson, B., et al.: An algorithm for the optimal partitioning of data on an interval. IEEE Signal Process. Lett. 12, 105 (2005) CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  • Matthew J. Graham
    • 1
    Email author
  • S. G. Djorgovski
    • 1
  • Ashish Mahabal
    • 1
  • Ciro Donalek
    • 1
  • Andrew Drake
    • 1
  • Giuseppe Longo
    • 2
  1. 1.California Institute of TechnologyPasadenaUSA
  2. 2.Department of PhysicsUniversity Federico IINaplesItaly

Personalised recommendations