Skip to main content
SpringerLink
Log in

HX-MATCH: In-Memory Cross-Matching Algorithm for Astronomical Big Data

  • Conference paper
  • First Online:
Advances in Spatial and Temporal Databases (SSTD 2017)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10411))

Included in the following conference series:

Abstract

The advanced progress in telescope facilities is continuously generating observation images containing billions of objects. Cross-match is a fundamental operation in astronomical data processing which enables astronomers to identify and correlate objects belonging to different observations in order to make new scientific achievements by studying the temporal evolution of the sources or combining physical properties. Comparing such vast amount of astronomical catalogs with low latency is a serious challenge. In this demonstration, we propose HX-MATCH, a new cross-matching algorithm based on Healpix and showcase an in-memory distributed framework where astronomers can compare large datasets.

This work has made use of data from the European Space Agency (ESA) mission Gaia (https://www.cosmos.esa.int/gaia), processed by the Gaia Data Processing and Analysis Consortium (DPAC, https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has been provided by national institutions, in particular the institutions participating in the Gaia Multilateral Agreement.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. ADQL. http://www.ivoa.net/documents/latest/ADQL.html

  2. ALADIN. http://aladin.u-strasbg.fr/

  3. GAIA. https://www.cosmos.esa.int/web/gaia/dr1

  4. Armbrust, M., et al.: Spark SQL: relational data processing in spark. In: Proceedings of the 2015 SIGMOD International Conference on Management of Data (2015)

    Google Scholar 

  5. Brahem, M.A., et al.: AstroSpark: towards a distributed data server for big data in astronomy. SIGSPATIAL Ph.D. Symposium (2016)

    Google Scholar 

  6. Eldawy, A., Mokbel, M.F.: Spatialhadoop: a mapreduce framework for spatial data. In: 2015 IEEE 31st International Conference on Data Engineering (ICDE). IEEE (2015)

    Google Scholar 

  7. Gorski, K.M., et al.: HEALPix: a framework for high-resolution discretization and fast analysis of data distributed on the sphere. Astrophys. J. 622(2), 759 (2005)

    Article  Google Scholar 

  8. Xie, D., et al.: Simba: efficient in-memory spatial analytics. In: Proceedings of the 2016 International Conference on Management of Data. ACM (2016)

    Google Scholar 

  9. Zhao, Q., Sun, J., Yu, C., Cui, C., Lv, L., Xiao, J.: A paralleled large-scale astronomical cross-matching function. In: Hua, A., Chang, S.-L. (eds.) ICA3PP 2009. LNCS, vol. 5574, pp. 604–614. Springer, Heidelberg (2009). doi:10.1007/978-3-642-03095-6_57

    Chapter  Google Scholar 

Download references

Acknowledgments

This work is partly founded by the CNES (Centre National d’Etudes Spatiales). We would like to thank Frederic Arenou at Paris Observatory, and Veronique Valette at CNES for their cooperation.

Author information

Authors and Affiliations

  1. DAVID Laboratory, University Versailles St Quentin, Paris Saclay University, Versailles, France

    Mariem Brahem, Karine Zeitouni & Laurent Yeh

Authors
  1. Mariem Brahem
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Karine Zeitouni
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Laurent Yeh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mariem Brahem .

Editor information

Editors and Affiliations

  1. Institute of Computer Science, Heidelberg University, Heidelberg, Germany

    Michael Gertz

  2. George Mason University, Fairfax, Virginia, USA

    Matthias Renz

  3. University of Queensland, Brisbane, Queensland, Australia

    Xiaofang Zhou

  4. ESRI, University of Minnesota, Minneapolis, Minnesota, USA

    Erik Hoel

  5. Auburn University, Auburn, Alabama, USA

    Wei-Shinn Ku

  6. Free University of Berlin, Dahlem, Berlin, Germany

    Agnes Voisard

  7. Microsoft, Redmond, Washington, USA

    Chengyang Zhang

  8. California State University, Sacramento, California, USA

    Haiquan Chen

  9. LinkedIn, Sunnyvale, California, USA

    Liang Tang

  10. University of North Texas, Denton, Texas, USA

    Yan Huang

  11. Virginia Tech, Falls Church, Virginia, USA

    Chang-Tien Lu

  12. Oracle, Redwood Shores, California, USA

    Siva Ravada

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Brahem, M., Zeitouni, K., Yeh, L. (2017). HX-MATCH: In-Memory Cross-Matching Algorithm for Astronomical Big Data. In: Gertz, M., et al. Advances in Spatial and Temporal Databases. SSTD 2017. Lecture Notes in Computer Science(), vol 10411. Springer, Cham. https://doi.org/10.1007/978-3-319-64367-0_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-64367-0_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-64366-3

  • Online ISBN: 978-3-319-64367-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation