Hydrological stream data pipeline framework based on IoTDB
- 73 Downloads
With the increasing amount of hydrological data in Chuhe river basin, the traditional relational database has been unable to meet the needs of users, which not only makes it difficult to achieve low latency and high throughput in the real-time transmission of hydrological data, but also causes the phenomenon of long time or even system crash when querying large amount of annual water-level data. To solve this problem, this paper proposes a stream data pipeline framework based on timeseries databases IoTDB and Kafka, which can provide services for hydrological early warning and anomaly detection researchers. Based on the hydrological sensor data of Chuhe river, the processing scenarios of sensor stream data are set and compared with other NoSQL (HBase, MongoDB, RiakTS and Redis) in different scenarios. The performance and workload of different NoSQL in this data pipeline are tested. Finally, it is docked with Flink real-time stream data processing platform and compared with other data pipelines. The experimental results show that the stream data pipeline composed of IoTDB, Kafka and Flink is outstanding in data acquisition, transmission, incremental query and data analysis.
KeywordsIoTDB NoSQL Kafka Stream data Data pipeline Real-time processing
This work was partly supported by the 2018 Jiangsu Province Key Research and Development Program (Modern Agriculture) Project under Grant No. 20195013812, 2017 Jiangsu Province Postdoctoral Research Funding Project under Grant No. 1701020C, 2017 Six Talent Peaks Endorsement Project of Jiangsu under Grant No. XYDXX- 078, the Fundamental Research Funds for the Central Universities under Grant No. 2013B01814.
- 1.Tang E, Fan Y (2017) Performance comparison between five NoSQL databases. In: International conference on cloud computing & big data. IEEEGoogle Scholar
- 2.Kang L, Deolalikar V, Pradhan N (2015) Big data gathering and mining pipelines for CRM using open-source. In: IEEE international conference on big dataGoogle Scholar
- 5.Nazeer H, Iqbal W, Bokhari F (2017) Real-time text analytics pipeline using open-source big data tools. arXiv:1712.04344v1
- 7.Nguyen CN, Kim JS, Hwang S (2016) KOHA: building a kafka-based distributed queue system on the fly in a Hadoop cluster. Foundations and applications of self* systems. In: IEEE International Workshops on IEEEGoogle Scholar
- 8.Yi M, Ting X, Shao-Bin L (2017) Research on NoSQL distributed big data mining method in complex attribute environment. Sci Technol EngGoogle Scholar
- 12.Al-Sakran A, Qattous H, Hijjawi M (2018) A proposed performance evaluation of NoSQL databases in the field of IoT. In: The 8th international conference on computer science and information technology (CSIT 2018). IEEE Computer SocietyGoogle Scholar
- 14.Feng Y, Peng Z, Sheng G, Yong C (2019) Intelligent Chuhe system based on the new generation of big data processing engine Flink. Water Resour Prot 2:90–94Google Scholar