Coherent quality management for big data systems: a dynamic approach for stochastic time consistency
- 60 Downloads
Big data systems for reinforcement learning have often exhibited problems (e.g., failures or errors) when their components involve stochastic nature with the continuous control actions of reliability and quality. The complexity of big data systems and their stochastic features raise the challenge of uncertainty. This article proposes a dynamic coherent quality measure focusing on an axiomatic framework by characterizing the probability of critical errors that can be used to evaluate if the conveyed information of big data interacts efficiently with the integrated system (i.e., system of systems) to achieve desired performance. Herein, we consider two new measures that compute the higher-than-expected error,—that is, the tail error and its conditional expectation of the excessive error (conditional tail error)—as a quality measure of a big data system. We illustrate several properties (that suffice stochastic time-invariance) of the proposed dynamic coherent quality measure for a big data system. We apply the proposed measures in an empirical study with three wavelet-based big data systems in monitoring and forecasting electricity demand to conduct the reliability and quality management in terms of minimizing decision-making errors. Performance of using our approach in the assessment illustrates its superiority and confirms the efficiency and robustness of the proposed method.
KeywordsBig data Dynamic coherent measure Optimal decision Quality management Time consistency
JEL ClassificationC02 C10 C63
The authors would like to thank the three anonymous reviewers and the guest editor for providing valuable comments. This work was supported in part by the Ministry of Science and Technology (MOST) under Grant 106-2221-E-009-006 and Grant 106-2221-E-009-049-MY2, in part by the “Aiming for the Top University Program” of National Chiao Tung University and the Ministry of Education, Taiwan, and in part by Academia Sinica AS-105-TP-A07 and Ministry of Economic Affairs (MOEA) 106-EC-17-A-24-0619.
- Chen, Y., & Sun, E. (2015). Jump detection and noise separation with singular wavelet method for high-frequency data. Working paper of KEDGE BS.Google Scholar
- Chen, Y., & Sun, E. (2018). Chapter 8: Automated business analytics for artificial intelligence in big data \(@\)x 4.0 era. In M. Dehmer & F. Emmert-Streib (Eds.), Frontiers in Data Science (pp. 223–251). Boca Raton: CRC Press.Google Scholar
- Chen, Y., Sun, E., & Yu, M. (2015). Improving model performance with the integrated wavelet denoising method. Studies in Nonlinear Dynamics and Econometrics, 19(4), 445–467.Google Scholar
- Chen, Y., Sun, E., & Yu, M. (2017). Risk assessment with wavelet feature engineering for high-frequency portfolio trading. Computational Economics. https://doi.org/10.1007/s10614-017-9711-7.
- Deichmann, J., Roggendorf, M., & Wee, D. (2015). McKinsey quarterly november: Preparing IT systems and organizations for the Internet of Things. McKinsey & Company.Google Scholar
- Hazen, B., Boone, C., Ezell, J., & Jones-Farmer, J. (2014). Data quality for data science, predictive analytics, and big data in supply chain management: An introduction to the problem and suggestions for research and applications. International Journal of Production Economics, 154, 72–80.CrossRefGoogle Scholar
- O’Neill, P., Sohal, A., & Teng, W. (2015). Quality management approaches and their impact on firms’ financial performance—An Australian study. International Journal of Production Economics. https://doi.org/10.1016/j.ijpe.2015.07.015i.
- Sun, E., Chen, Y., & Yu, M. (2015). Generalized optimal wavelet decomposing algorithm for big financial data. International Journal of Production Economics, 165, 161–177.Google Scholar