An Efficient Parallel Framework to Analyze Astronomical Sky Survey Data

Abstract

Big data has been an important analysis method anywhere we turn today. We hold broad recognition of the value of data, and products obtained through analyzing it. There are multiple steps to the data analysis pipeline, which can be abstracted as a framework provides universal parallel high-performance data analysis. Based on ray, this paper proposed a parallel framework written in Python with an interface to aggregate and analyze homogeneous astronomical sky survey time series data. As such, we can achieve parallel training and analysis only by defining the customized analyze functions, decision module and I/O interfaces, while the framework is able to manage the pipeline such as data fetching, saving, parallel job scheduling and load balancing. Meanwhile, the data scientists can focus on the analysis procedure and save the time speeding this program up. We tested out the framework on synthetic data with raw files and HBase entries as data sources and result formats, reduced the analyze cost for scientists not familiar with parallel programming while needs to handle a mass of data. We integrate time series anomaly detection algorithms with our parallel dispatching module to achieve high-performance data processing frameworks. Experimental results on synthetic astronomical sky survey time series data show that our model achieves good speed up ratio in executing analysis programs.