Advertisement

SkyQuery: An Implementation of a Parallel Probabilistic Join Engine for Cross-Identification of Multiple Astronomical Databases

  • László Dobos
  • Tamás Budavári
  • Nolan Li
  • Alexander S. Szalay
  • István Csabai
Conference paper
  • 1.3k Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7338)

Abstract

Multi-wavelength astronomical studies require cross-identification of detections of the same celestial objects in multiple catalogs based on spherical coordinates and other properties. Because of the large data volumes and spherical geometry, the symmetric N-way association of astronomical detections is a computationally intensive problem, even when sophisticated indexing schemes are used to exclude obviously false candidates. Legacy astronomical catalogs already contain detections of more than a hundred million objects while ongoing and future surveys will produce catalogs of billions of objects with multiple detections of each at different times. One time, pair-wise cross-identification of these large catalogs is not sufficient for many astronomical scenarios. Consequently, a novel system is necessary that can cross-identify multiple catalogs on-demand, efficiently and reliably. In this paper, we present our solution based on a cluster of commodity servers and ordinary relational databases. The cross-identification problems are formulated in a language based on SQL, but extended with special clauses. These special queries are partitioned spatially by coordinate ranges and compiled into a complex workflow of ordinary SQL queries. Workflows are then executed in a parallel framework using a cluster of servers hosting identical mirrors of the same data sets.

Keywords

probabilistic join query optimization and languages astronomical catalogs workflow computational statistics 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [BPD1]
    Boch, T.: Pineau, F.X., Derriere, S.: CDS xMatch service documentation (2011), http://cdsxmatch.u-strasbg.fr
  2. [Bud1]
    Budavári, T., Malik, T., Szalay, A.S., Thakar, A.R., Gray, J.: Proceedings of the Conference Astronomical Data Analysis Software and Systems XII, vol. 295, p. 31 (2003)Google Scholar
  3. [Bud2]
    Budavári, T., Szalay, A.S., Gray, J., et al.: Proceedings of the Conference Astronomical Data Analysis Software and Systems XIII, vol. 314, p. 177 (2004)Google Scholar
  4. [Bud3]
    Budavári, T., Szalay, A.S.: Astrophysical Journal 679, 301 (2008)CrossRefGoogle Scholar
  5. [Bud4]
    Budavári, T., Szalay, A.S., Fekete, G.: Publications of the Astronomical Society of the Pacific, vol. 122, p. 1375 (2010)Google Scholar
  6. [Fek]
    Fekete, G., Szalay, A.S., Gray, J.: Proceedings of the Conference Astronomical Data Analysis Software and Systems XIII, vol. 314, p. 289 (2004)Google Scholar
  7. [Gor]
    Górski, K.M., Hivon, E., Banday, A.J., et al.: Astrophysical Journal 622, 759 (2005)CrossRefGoogle Scholar
  8. [Gra]
    Gray, J., Szalay, A., Budavari, T., et al.: arXiv:cs/0701172 (2007)Google Scholar
  9. [OM1]
    O’Mullane, W., Gray, J., Li, N., et al.: Proceedings of the Conference Astronomical Data Analysis Software and Systems XIII, vol. 314, p. 372 (2004)Google Scholar
  10. [Or1]
    Ortiz, I., Lusted, J., Dowler, P., et al.: arXiv:1110.0503 (2011)Google Scholar
  11. [TP1]
    Taghizadeh-Popp, M.: Publications of the Astronomical Society of the Pacific, vol. 122, p. 976 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • László Dobos
    • 1
    • 2
  • Tamás Budavári
    • 2
  • Nolan Li
    • 2
  • Alexander S. Szalay
    • 2
  • István Csabai
    • 1
  1. 1.Department of Physics of Complex SystemsEötvös Loránd UniversityBudapestHungary
  2. 2.Department of Physics & AstronomyThe Johns Hopkins UniversityBaltimoreUSA

Personalised recommendations