Abstract
In time-domain astronomy, we need to use the relational database to manage star catalog data. With the development of sky survey technology, the size of star catalog data is larger, and the speed of data generation is faster. So, in this paper, we make a systematic and comprehensive introduction to process the data in time-domain astronomy, and valuable research questions are detailed. Then, we list candidate systems usually used in astronomy and point out the advantages and disadvantages of these systems. In addition, we present the key techniques needed to deal with astronomical data. Finally, we summarize the challenges faced by the design of our database prototype.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mongodb. http://www.mongodb.org/
Oceanbase. https://github.com/alibaba/oceanbase/tree/master/oceanbase_0.4
Spark. http://spark-project.org/
Zone project. http://research.microsoft.com/apps/pubs/default.aspx?id=64524
Ameri, P., Lutz, R., Latzko, T., Meyer, J.: Management of meteorological mass data with mongodb. In: Einviroinfo (2014)
Boncz, P., Grust, T., Van Keulen, M., Manegold, S., Rittinger, J., Teubner, J.: MonetDB/XQuery: a fast XQuery processor powered by a relational engine. In: SIGMOD, pp. 479–490 (2006)
Bryant, R.E., Katz, R.H., Lazowska, E.D.: Bigdata computing: creating revolutionary breakthroughs in commerce, science, and society (2008)
Cui, C., et al.: Astronomy research in big-data era. Chin. Sci. Bull. 60(z1), 445–449 (2015). (in Chinese)
Idreos, S., Groffen, F.E., Nes, N.J., Manegold, S., Mullender, K.S., Kersten, M.L.: MonetDB: two decades of research in column-oriented database architectures. IEEE Data Eng. Bull. 35(1), 40–45 (2012)
Manegold, S., Kersten, M.L., Boncz, P.: Database architecture evolution: mammals flourished long before dinosaurs became extinct. Proc. VLDB Endow. 2(2), 1648–1653 (2009)
Wan, M.: Column store for GWAC: a high cadence high density large-scale astronomical light curve pipeline and distributed shared-nothing database. Publ. Astron. Soc. Pac. 128(969), 114501 (2016)
Naimi, A.I., Westreich, D.J.: Big data: a revolution that will transform how we live, work, and think. Information 17(1), 181–183 (2014)
Shanahan, J.G., Dai, L.: Large scale distributed data science using apache spark. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2015)
Stonebraker, M., et al.: Requirements for science data bases and SciDB. In: CIDR (2009)
Szalay, A.S., Blakeley, J.A., Szalay, A.S., Blakeley, J.A.: Gray’s laws: database-centric computing in science (2009)
Wang, S., Zhao, Y., Luo, Q., Wu, C., Xv, Y.: Accelerating in-memory cross match of astronomical catalogs. In: IEEE International Conference on Escience, pp. 326–333 (2013)
Yang, X., et al.: A fast cross-identification algorithm for searching optical transient sources. Astron. Res. Technol. 10(3), 273–282 (2013)
Zaharia, M., et al.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Usenix Conference on Networked Systems Design and Implementation, pp. 141–146 (2012)
Zhao, B., Luo, Q., Wu, C.: Parallelizing astronomical source extraction on the GPU. In: IEEE International Conference on Escience, pp. 88–97 (2013)
Zhao, Y., Luo, Q., Wang, S., Wu, C.: Accelerating astronomical image subtraction on heterogeneous processors. In: IEEE International Conference on Escience, pp. 70–77 (2013)
Acknowledgement
This research was partially supported by the grants from the National Key Research and Development Program of China (No. 2016YFB1000602, 2016YFB1000603); the Natural Science Foundation of China (No. 91646203, 61532016, 61532010, 61379050, 61762082); the Fundamental Research Funds for the Central Universities, the Research Funds of Renmin University (No. 11XNL010); and the Science and Technology Opening up Cooperation project of Henan Province (172106000077).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, C., Meng, X., Du, Z., Duan, Z., Du, Y. (2019). Data Management in Time-Domain Astronomy: Requirements and Challenges. In: Li, J., Meng, X., Zhang, Y., Cui, W., Du, Z. (eds) Big Scientific Data Management. BigSDM 2018. Lecture Notes in Computer Science(), vol 11473. Springer, Cham. https://doi.org/10.1007/978-3-030-28061-1_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-28061-1_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-28060-4
Online ISBN: 978-3-030-28061-1
eBook Packages: Computer ScienceComputer Science (R0)