Statistical Process Control Charts as a Tool for Analyzing Big Data

  • Peihua QiuEmail author
Part of the Contributions to Statistics book series (CONTRIB.STAT.)


Big data often take the form of data streams with observations of certain processes collected sequentially over time. Among many different purposes, one common task to collect and analyze big data is to monitor the longitudinal performance/status of the related processes. To this end, statistical process control (SPC) charts could be a useful tool, although conventional SPC charts need to be modified properly in some cases. In this paper, we introduce some basic SPC charts and some of their modifications, and describe how these charts can be used for monitoring different types of processes. Among many potential applications, dynamic disease screening and profile/image monitoring will be discussed in some detail.


Curve data Data stream Images Longitudinal data Monitoring Profiles Sequential process Surveillance 



This research is supported in part by a US National Science Foundation grant. The author thanks the invitation of the book editor Professor Ejaz Ahmed, and the review of an anonymous referee.


  1. 1.
    Albers, W., Kallenberg, W.C.M.: Empirical nonparametric control charts: estimation effects and corrections. J. Appl. Stat. 31, 345–360 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Amin, R.W., Widmaier, O.: Sign control charts with variable sampling intervals. Commun. Stat. Theory Meth. 28, 1961–1985 (1999)CrossRefzbMATHGoogle Scholar
  3. 3.
    Apley, D.W., Lee, H.C.: Design of exponentially weighted moving average control charts for autocorrelated processes with model uncertainty. Technometrics 45, 187–198 (2003)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Bakir, S.T.: Distribution-free quality control charts based on signed-rank-like statistics. Commun. Stat. Theory Meth. 35, 743–757 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Capizzi, G., Masarotto, G.: A least angle regression control chart for multidimensional data. Technometrics 53, 285–296 (2011)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Chakraborti, S., van der Laan, P., Bakir, S.T.: Nonparametric control charts: an overview and some results. J. Qual. Technol. 33, 304–315 (2001)Google Scholar
  7. 7.
    Chakraborti, S., Qiu, P., Mukherjee, A. (eds.): Special issue on nonparametric statistical process control charts. Qual. Reliab. Eng. Int. 31, 1–151 (2015)Google Scholar
  8. 8.
    Champ, C.W., Woodall, W.H.: Exact results for Shewhart control charts with supplementary runs rules. Technometrics 29, 393–399 (1987)CrossRefzbMATHGoogle Scholar
  9. 9.
    Chatterjee, S., Qiu, P.: Distribution-free cumulative sum control charts using bootstrap- based control limits. Ann. Appl. Stat. 3, 349–369 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Chicken, E., Pignatiello, J.J. Jr., Simpson, J.R.: Statistical process monitoring of nonlinear profiles using wavelets. J. Qual. Technol. 41, 198–212 (1998)Google Scholar
  11. 11.
    Chiu, D., Guillaud, M., Cox, D., Follen, M., MacAulay, C.: Quality assurance system using statistical process control: an implementation for image cytometry. Cell. Oncol. 26, 101–117 (2004)Google Scholar
  12. 12.
    Crosier, R.B.: Multivariate generalizations of cumulative sum quality-control schemes. Technometrics 30, 291–303 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Crowder, S.V.: Design of exponentially weighted moving average schemes. J. Qual. Technol. 21, 155–162 (1989)Google Scholar
  14. 14.
    Cupples, L.A., et al.: The Framingham heart study 100K SNP genome-wide association study resource: overview of 17 phenotype working group reports. BMC Med. Genet. 8, S1 (2007)CrossRefGoogle Scholar
  15. 15.
    Ding, Y., Zeng, L., Zhou, S.: Phase I analysis for monitoring nonlinear profiles in manufacturing processes. J. Qual. Technol. 38, 199–216 (2006)Google Scholar
  16. 16.
    Hawkins, D.M.: Multivariate quality control based on regression-adjusted variables. Technometrics 33, 61–75 (1991)Google Scholar
  17. 17.
    Hawkins, D.M., Deng, Q.: A nonparametric change-point control chart. J. Qual. Technol. 42, 165–173 (2010)Google Scholar
  18. 18.
    Hawkins, D.M., Olwell, D.H.: Cumulative Sum Charts and Charting for Quality Improvement. Springer, New York (1998)CrossRefzbMATHGoogle Scholar
  19. 19.
    Hawkins, D.M., Qiu, P., Kang, C.W.: The changepoint model for statistical process control. J. Qual. Technol. 35, 355–366 (2003)Google Scholar
  20. 20.
    Healy, J.D.: A note on multivariate CUSUM procedures. Technometrics 29, 409–412 (1987)CrossRefGoogle Scholar
  21. 21.
    Jensen, W.A., Birch, J.B.: Profile monitoring via nonlinear mixed models. J. Qual. Technol. 41, 18–34 (2009)Google Scholar
  22. 22.
    Jensen, W.A., Birch, J.B., Woodall, W.H.: Monitoring correlation within linear profiles using mixed models. J. Qual. Technol. 40, 167–183 (2008)Google Scholar
  23. 23.
    Jin, J., Shi, J.: Feature-preserving data compression of stamping tonnage information using wavelets. Technometrics 41, 327–339 (1999)CrossRefGoogle Scholar
  24. 24.
    Kang, L., Albin, S.L.: On-line monitoring when the process yields a linear profile. J. Qual. Technol. 32, 418–426 (2000)Google Scholar
  25. 25.
    Kim, K., Mahmoud, M.A., Woodall, W.H.: On the monitoring of linear profiles. J. Qual. Technol. 35, 317–328 (2003)Google Scholar
  26. 26.
    Li, J., Qiu, P.: Nonparametric dynamic screening system for monitoring correlated longitudinal data. IIE Trans. 48, 772–786 (2016)CrossRefGoogle Scholar
  27. 27.
    Liu, R.Y.: Control charts for multivariate processes. J. Am. Stat. Assoc. 90, 1380–1387 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Lowry, C.A., Woodall, W.H., Champ, C.W., Rigdon, S.E.: A multivariate exponentially weighted moving average control chart. Technometrics 34, 46–53 (1992)CrossRefzbMATHGoogle Scholar
  29. 29.
    Lu, C.W., Reynolds, M.R. Jr.: Control charts for monitoring the mean variance of autocorrelated processes. J. Qual. Technol. 31, 259–274 (1999)Google Scholar
  30. 30.
    Martin, H., Priscila, L.: The world’s technological capacity to store, communicate, and compute information. Science 332, 60–65 (2011)CrossRefGoogle Scholar
  31. 31.
    Page, E.S.: Continuous inspection scheme. Biometrika 41, 100–115 (1954)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Paynabar, K., Jin, J., Pacella, M.: Analysis of multichannel nonlinear profiles using uncorrelated multilinear principal component analysis with applications in fault detection and diagnosis. IIE Trans. 45, 1235–1247 (2013)CrossRefGoogle Scholar
  33. 33.
    Paynabar, K., Qiu, P., Zou, C.: A change point approach for phase-I analysis in multivariate profile monitoring and diagnosis. Technometrics 58, 191–204 (2016)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Qiu, P.: Image Processing and Jump Regression Analysis. Wiley, New York (2005)CrossRefzbMATHGoogle Scholar
  35. 35.
    Qiu, P.: Distribution-free multivariate process control based on log-linear modeling. IIE Trans. 40, 664–677 (2008)CrossRefGoogle Scholar
  36. 36.
    Qiu, P.: Introduction to Statistical Process Control. Chapman Hall/CRC, Boca Raton, FL (2014)Google Scholar
  37. 37.
    Qiu, P., Hawkins, D.M.: A rank based multivariate CUSUM procedure. Technometrics 43, 120–132 (2001)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Qiu, P., Hawkins, D.M.: A nonparametric multivariate CUSUM procedure for detecting shifts in all directions. J. R. Stat. Soc. -D (The Statistician) 52, 151–164 (2003)Google Scholar
  39. 39.
    Qiu, P., Li, Z.: On nonparametric statistical process control of univariate processes. Technometrics 53, 390–405 (2011)MathSciNetCrossRefGoogle Scholar
  40. 40.
    Qiu, P., Xiang, D.: Univariate dynamic screening system: an approach for identifying individuals with irregular longitudinal behavior. Technometrics 56, 248–260 (2014)MathSciNetCrossRefGoogle Scholar
  41. 41.
    Qiu, P., Xiang, D.: Surveillance of cardiovascular diseases using a multivariate dynamic screening system. Stat. Med. 34, 2204–2221 (2015)MathSciNetCrossRefGoogle Scholar
  42. 42.
    Qiu, P., Xing, C.: On nonparametric image registration. Technometrics 55, 174–188 (2013)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Qiu, P., Zou, C.: Control chart for monitoring nonparametric profiles with arbitrary design. Stat. Sin. 20, 1655–1682 (2010)MathSciNetzbMATHGoogle Scholar
  44. 44.
    Qiu, P., Zou, C., Wang, Z.: Nonparametric profile monitoring by mixed effects modeling (with discussions). Technometrics 52, 265–293 (2010)MathSciNetCrossRefGoogle Scholar
  45. 45.
    Reichman, O.J., Jones, M.B., Schildhauer, M.P.: Challenges and opportunities of open data in ecology. Science 331, 703–705 (2011)CrossRefGoogle Scholar
  46. 46.
    Roberts, S.V.: Control chart tests based on geometric moving averages. Technometrics 1, 239–250 (1959)CrossRefGoogle Scholar
  47. 47.
    Ross, G.J., Tasoulis, D.K., Adams, N.M.: Nonparametric monitoring of data streams for changes in location and scale. Technometrics 53, 379–389 (2011)MathSciNetCrossRefGoogle Scholar
  48. 48.
    Shewhart, W.A.: Economic Control of Quality of Manufactured Product. D. Van Nostrand Company, New York (1931)Google Scholar
  49. 49.
    Timmer, D.H., Pignatiello, J., Longnecker, M.: The development and evaluation of CUSUM-based control charts for an AR(1) process. IIE Trans. 30, 525–534 (1998)Google Scholar
  50. 50.
    Tong, L.-I., Wang, C.-H., Huang, C.-L.: Monitoring defects in IC fabrication using a Hotelling T2 control chart. IEEE Trans. Semicond. Manuf. 18, 140–147 (2005)CrossRefGoogle Scholar
  51. 51.
    Tracy, N.D., Young, J.C., Mason, R.L.: Multivariate control charts for individual observations. J. Qual. Technol. 24, 88–95 (1992)Google Scholar
  52. 52.
    Wang, K., Jiang, W.: High-dimensional process monitoring and fault isolation via variable selection. J. Qual. Technol. 41, 247–258 (2009)Google Scholar
  53. 53.
    Woodall, W.H., Ncube, M.M.: Multivariate CUSUM quality-control procedures. Technometrics 27, 285–292 (1985)MathSciNetCrossRefzbMATHGoogle Scholar
  54. 54.
    Xing, C., Qiu, P.: Intensity-based image registration by nonparametric local smoothing. IEEE Trans. Pattern Anal. Mach. Intell. 33, 2081–2092 (2011)CrossRefGoogle Scholar
  55. 55.
    Yashchin, E.: Statistical control schemes: methods, applications and generalizations. Int. Stat. Rev. 61, 41–66 (1993)CrossRefGoogle Scholar
  56. 56.
    Zhang, J., Kang, Y., Yang, Y., Qiu, P.: Statistical monitoring of the hand, foot and mouth disease in China. Biometrics 71, 841–850 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  57. 57.
    Zhang, J., Qiu, P., Chen, X.: Statistical monitoring-based alarming systems in modeling the AIDS epidemic in the US, 1985–2011. Curr. HIV Res. 14, 130–137 (2016)CrossRefGoogle Scholar
  58. 58.
    Zou, C., Qiu, P.: Multivariate statistical process control using LASSO. J. Am. Stat. Assoc. 104, 1586–1596 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  59. 59.
    Zou, C., Tsung, F.: Likelihood ratio-based distribution-free EWMA control charts. J. Qual. Technol. 42, 1–23 (2010)Google Scholar
  60. 60.
    Zou, C., Tsung, F.: A multivariate sign EWMA control chart. Technometrics 53, 84–97 (2011)MathSciNetCrossRefGoogle Scholar
  61. 61.
    Zou, C., Zhang, Y., Wang, Z.: Control chart based on change-point model for monitoring linear profiles. IIE Trans. 38, 1093–1103 (2006)CrossRefGoogle Scholar
  62. 62.
    Zou, C., Tsung, F., Wang, Z.: Monitoring general linear profiles using multivariate EWMA schemes. Technometrics 49, 395–408 (2007)MathSciNetCrossRefGoogle Scholar
  63. 63.
    Zou, C., Tsung, F., Wang, Z.: Monitoring profiles based on nonparametric regression methods. Technometrics 50, 512–526 (2008)MathSciNetCrossRefGoogle Scholar
  64. 64.
    Zou, C., Qiu, P., Hawkins, D.: Nonparametric control chart for monitoring profiles using change point formulation and adaptive smoothing. Stat. Sin. 19, 1337–1357 (2009)MathSciNetzbMATHGoogle Scholar
  65. 65.
    Zou, C., Jiang, W., Wang, Z., Zi, X.: An efficient on-line monitoring method for high-dimensional data streams. Technometrics 57, 374–387 (2015)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of BiostatisticsUniversity of FloridaGainesvilleUSA

Personalised recommendations