The Research of Data Blood Relationship Analysis on Metadata
In the process of continuous expansion of data and continuous expansion of the system, various data relations and data forms form crisscross connections, forming an extremely complex network diagram. If there is an error in the data, how do we quickly lock the cause of the problem? How do we find out which entities are affected by the implications or changes of the problem? These issues create challenges and pressures for large-scale, enterprise-level data platforms. The paper proposes to use data blood analysis to solve the relationship among tens of millions of tables. To get this kind of more underlying blood information, we need to add embedded parts to the execution engine, which will be fed into the blood relationship collection system using push mode when the job is executed. The paper is to implement field level blood relationship analysis in the data warehouse of China Commercial bank on the architecture of Teradata, and separated it from the ETL process and made it into a single part. By parsing multiple ETL jobs, we get a number of mapping relationship of atoms, and atoms and relationships make up the molecules that form the blood relationship network we need. This experimental scheme can be simply embedded into the data platform by eliminating the complexity of the system and achieving a separate component structure. The blood relationship can be conducted any time and temporary scripts and error logic of related data will have no data pollution on the data blood relationship.
KeywordsNetwork diagram of data relations Quickly lock Enterprise-level data Data blood analysis Blood relationship network
The paper is sponsored by fund (fund id: JELRGBDT201707, National Natural Science Foundation of China 61662002, 61463003 & 11865002). Fenfen Guan, Yongping Gao & Jun Zhang are the corresponding authors.
- 1.Chen, X.-J.: Research and realization of the college scientific research management system based on SSH frame. Electron. Des. Eng. 8, 8–12 (2011)Google Scholar
- 2.Liang, L., Yi, C., Yang, X., et al.: Computer Applications and Software, vol. 29, p. 283 (2012)Google Scholar
- 3.Sankaradass, V., Arputharaj, K.: An intelligent recommendation system for web user personalization with fuzzy temporal association rules. Eur. J. Sci. Res. 51(1), 88–96 (2011). ISSN 1450-216XGoogle Scholar
- 4.Zhang, Y., Liu, X.: Application of “consanguinity analysis” technology in information system of commercial banks. Inf. Technol. Inf. 6, 147–149 (2017)Google Scholar
- 5.Heng, X.: Research and practice of metadata management system in electric power enterprises. Autom. Instrum. 4, 101–107 (2017)Google Scholar
- 6.Huang, B., Peng, Y.: An efficient scalable metadata management method in cloud computing. Comput. Eng. Des. 9, 1147–1154 (2014)Google Scholar
- 7.Pesic, M., Schonenberg, M.H., Van der Aalst, W.M.P.: Declare: full support for loosely-structured processes. In: Proceedings of the 11th IEEE International Enterprise Distributed Object Computing Conference, pp. 287–298. IEEE Computer Society, Washington, D.C. (2007)Google Scholar
- 8.Peng, J.: Research on deep aggregation of characteristic resources based on metadata ontology. Libr. J. 11, 82–89 (2016)Google Scholar