Skip to main content
SpringerLink
Log in
Book cover

Big and Complex Data Analysis pp 123–138Cite as

Statistical Process Control Charts as a Tool for Analyzing Big Data

Part of the Contributions to Statistics book series (CONTRIB.STAT.)

Abstract

Big data often take the form of data streams with observations of certain processes collected sequentially over time. Among many different purposes, one common task to collect and analyze big data is to monitor the longitudinal performance/status of the related processes. To this end, statistical process control (SPC) charts could be a useful tool, although conventional SPC charts need to be modified properly in some cases. In this paper, we introduce some basic SPC charts and some of their modifications, and describe how these charts can be used for monitoring different types of processes. Among many potential applications, dynamic disease screening and profile/image monitoring will be discussed in some detail.

Keywords

  • Curve data
  • Data stream
  • Images
  • Longitudinal data
  • Monitoring
  • Profiles
  • Sequential process
  • Surveillance

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   139.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Albers, W., Kallenberg, W.C.M.: Empirical nonparametric control charts: estimation effects and corrections. J. Appl. Stat. 31, 345–360 (2004)

    CrossRef  MathSciNet  MATH  Google Scholar 

  2. Amin, R.W., Widmaier, O.: Sign control charts with variable sampling intervals. Commun. Stat. Theory Meth. 28, 1961–1985 (1999)

    CrossRef  MATH  Google Scholar 

  3. Apley, D.W., Lee, H.C.: Design of exponentially weighted moving average control charts for autocorrelated processes with model uncertainty. Technometrics 45, 187–198 (2003)

    CrossRef  MathSciNet  Google Scholar 

  4. Bakir, S.T.: Distribution-free quality control charts based on signed-rank-like statistics. Commun. Stat. Theory Meth. 35, 743–757 (2006)

    CrossRef  MathSciNet  MATH  Google Scholar 

  5. Capizzi, G., Masarotto, G.: A least angle regression control chart for multidimensional data. Technometrics 53, 285–296 (2011)

    CrossRef  MathSciNet  Google Scholar 

  6. Chakraborti, S., van der Laan, P., Bakir, S.T.: Nonparametric control charts: an overview and some results. J. Qual. Technol. 33, 304–315 (2001)

    Google Scholar 

  7. Chakraborti, S., Qiu, P., Mukherjee, A. (eds.): Special issue on nonparametric statistical process control charts. Qual. Reliab. Eng. Int. 31, 1–151 (2015)

    Google Scholar 

  8. Champ, C.W., Woodall, W.H.: Exact results for Shewhart control charts with supplementary runs rules. Technometrics 29, 393–399 (1987)

    CrossRef  MATH  Google Scholar 

  9. Chatterjee, S., Qiu, P.: Distribution-free cumulative sum control charts using bootstrap- based control limits. Ann. Appl. Stat. 3, 349–369 (2009)

    CrossRef  MathSciNet  MATH  Google Scholar 

  10. Chicken, E., Pignatiello, J.J. Jr., Simpson, J.R.: Statistical process monitoring of nonlinear profiles using wavelets. J. Qual. Technol. 41, 198–212 (1998)

    Google Scholar 

  11. Chiu, D., Guillaud, M., Cox, D., Follen, M., MacAulay, C.: Quality assurance system using statistical process control: an implementation for image cytometry. Cell. Oncol. 26, 101–117 (2004)

    Google Scholar 

  12. Crosier, R.B.: Multivariate generalizations of cumulative sum quality-control schemes. Technometrics 30, 291–303 (1988)

    CrossRef  MathSciNet  MATH  Google Scholar 

  13. Crowder, S.V.: Design of exponentially weighted moving average schemes. J. Qual. Technol. 21, 155–162 (1989)

    Google Scholar 

  14. Cupples, L.A., et al.: The Framingham heart study 100K SNP genome-wide association study resource: overview of 17 phenotype working group reports. BMC Med. Genet. 8, S1 (2007)

    CrossRef  Google Scholar 

  15. Ding, Y., Zeng, L., Zhou, S.: Phase I analysis for monitoring nonlinear profiles in manufacturing processes. J. Qual. Technol. 38, 199–216 (2006)

    Google Scholar 

  16. Hawkins, D.M.: Multivariate quality control based on regression-adjusted variables. Technometrics 33, 61–75 (1991)

    Google Scholar 

  17. Hawkins, D.M., Deng, Q.: A nonparametric change-point control chart. J. Qual. Technol. 42, 165–173 (2010)

    Google Scholar 

  18. Hawkins, D.M., Olwell, D.H.: Cumulative Sum Charts and Charting for Quality Improvement. Springer, New York (1998)

    CrossRef  MATH  Google Scholar 

  19. Hawkins, D.M., Qiu, P., Kang, C.W.: The changepoint model for statistical process control. J. Qual. Technol. 35, 355–366 (2003)

    Google Scholar 

  20. Healy, J.D.: A note on multivariate CUSUM procedures. Technometrics 29, 409–412 (1987)

    CrossRef  Google Scholar 

  21. Jensen, W.A., Birch, J.B.: Profile monitoring via nonlinear mixed models. J. Qual. Technol. 41, 18–34 (2009)

    Google Scholar 

  22. Jensen, W.A., Birch, J.B., Woodall, W.H.: Monitoring correlation within linear profiles using mixed models. J. Qual. Technol. 40, 167–183 (2008)

    Google Scholar 

  23. Jin, J., Shi, J.: Feature-preserving data compression of stamping tonnage information using wavelets. Technometrics 41, 327–339 (1999)

    CrossRef  Google Scholar 

  24. Kang, L., Albin, S.L.: On-line monitoring when the process yields a linear profile. J. Qual. Technol. 32, 418–426 (2000)

    Google Scholar 

  25. Kim, K., Mahmoud, M.A., Woodall, W.H.: On the monitoring of linear profiles. J. Qual. Technol. 35, 317–328 (2003)

    Google Scholar 

  26. Li, J., Qiu, P.: Nonparametric dynamic screening system for monitoring correlated longitudinal data. IIE Trans. 48, 772–786 (2016)

    CrossRef  Google Scholar 

  27. Liu, R.Y.: Control charts for multivariate processes. J. Am. Stat. Assoc. 90, 1380–1387 (1995)

    CrossRef  MathSciNet  MATH  Google Scholar 

  28. Lowry, C.A., Woodall, W.H., Champ, C.W., Rigdon, S.E.: A multivariate exponentially weighted moving average control chart. Technometrics 34, 46–53 (1992)

    CrossRef  MATH  Google Scholar 

  29. Lu, C.W., Reynolds, M.R. Jr.: Control charts for monitoring the mean variance of autocorrelated processes. J. Qual. Technol. 31, 259–274 (1999)

    Google Scholar 

  30. Martin, H., Priscila, L.: The world’s technological capacity to store, communicate, and compute information. Science 332, 60–65 (2011)

    CrossRef  Google Scholar 

  31. Page, E.S.: Continuous inspection scheme. Biometrika 41, 100–115 (1954)

    CrossRef  MathSciNet  MATH  Google Scholar 

  32. Paynabar, K., Jin, J., Pacella, M.: Analysis of multichannel nonlinear profiles using uncorrelated multilinear principal component analysis with applications in fault detection and diagnosis. IIE Trans. 45, 1235–1247 (2013)

    CrossRef  Google Scholar 

  33. Paynabar, K., Qiu, P., Zou, C.: A change point approach for phase-I analysis in multivariate profile monitoring and diagnosis. Technometrics 58, 191–204 (2016)

    CrossRef  MathSciNet  Google Scholar 

  34. Qiu, P.: Image Processing and Jump Regression Analysis. Wiley, New York (2005)

    CrossRef  MATH  Google Scholar 

  35. Qiu, P.: Distribution-free multivariate process control based on log-linear modeling. IIE Trans. 40, 664–677 (2008)

    CrossRef  Google Scholar 

  36. Qiu, P.: Introduction to Statistical Process Control. Chapman Hall/CRC, Boca Raton, FL (2014)

    Google Scholar 

  37. Qiu, P., Hawkins, D.M.: A rank based multivariate CUSUM procedure. Technometrics 43, 120–132 (2001)

    CrossRef  MathSciNet  Google Scholar 

  38. Qiu, P., Hawkins, D.M.: A nonparametric multivariate CUSUM procedure for detecting shifts in all directions. J. R. Stat. Soc. -D (The Statistician) 52, 151–164 (2003)

    Google Scholar 

  39. Qiu, P., Li, Z.: On nonparametric statistical process control of univariate processes. Technometrics 53, 390–405 (2011)

    CrossRef  MathSciNet  Google Scholar 

  40. Qiu, P., Xiang, D.: Univariate dynamic screening system: an approach for identifying individuals with irregular longitudinal behavior. Technometrics 56, 248–260 (2014)

    CrossRef  MathSciNet  Google Scholar 

  41. Qiu, P., Xiang, D.: Surveillance of cardiovascular diseases using a multivariate dynamic screening system. Stat. Med. 34, 2204–2221 (2015)

    CrossRef  MathSciNet  Google Scholar 

  42. Qiu, P., Xing, C.: On nonparametric image registration. Technometrics 55, 174–188 (2013)

    CrossRef  MathSciNet  Google Scholar 

  43. Qiu, P., Zou, C.: Control chart for monitoring nonparametric profiles with arbitrary design. Stat. Sin. 20, 1655–1682 (2010)

    MathSciNet  MATH  Google Scholar 

  44. Qiu, P., Zou, C., Wang, Z.: Nonparametric profile monitoring by mixed effects modeling (with discussions). Technometrics 52, 265–293 (2010)

    CrossRef  MathSciNet  Google Scholar 

  45. Reichman, O.J., Jones, M.B., Schildhauer, M.P.: Challenges and opportunities of open data in ecology. Science 331, 703–705 (2011)

    CrossRef  Google Scholar 

  46. Roberts, S.V.: Control chart tests based on geometric moving averages. Technometrics 1, 239–250 (1959)

    CrossRef  Google Scholar 

  47. Ross, G.J., Tasoulis, D.K., Adams, N.M.: Nonparametric monitoring of data streams for changes in location and scale. Technometrics 53, 379–389 (2011)

    CrossRef  MathSciNet  Google Scholar 

  48. Shewhart, W.A.: Economic Control of Quality of Manufactured Product. D. Van Nostrand Company, New York (1931)

    Google Scholar 

  49. Timmer, D.H., Pignatiello, J., Longnecker, M.: The development and evaluation of CUSUM-based control charts for an AR(1) process. IIE Trans. 30, 525–534 (1998)

    Google Scholar 

  50. Tong, L.-I., Wang, C.-H., Huang, C.-L.: Monitoring defects in IC fabrication using a Hotelling T2 control chart. IEEE Trans. Semicond. Manuf. 18, 140–147 (2005)

    CrossRef  Google Scholar 

  51. Tracy, N.D., Young, J.C., Mason, R.L.: Multivariate control charts for individual observations. J. Qual. Technol. 24, 88–95 (1992)

    Google Scholar 

  52. Wang, K., Jiang, W.: High-dimensional process monitoring and fault isolation via variable selection. J. Qual. Technol. 41, 247–258 (2009)

    Google Scholar 

  53. Woodall, W.H., Ncube, M.M.: Multivariate CUSUM quality-control procedures. Technometrics 27, 285–292 (1985)

    CrossRef  MathSciNet  MATH  Google Scholar 

  54. Xing, C., Qiu, P.: Intensity-based image registration by nonparametric local smoothing. IEEE Trans. Pattern Anal. Mach. Intell. 33, 2081–2092 (2011)

    CrossRef  Google Scholar 

  55. Yashchin, E.: Statistical control schemes: methods, applications and generalizations. Int. Stat. Rev. 61, 41–66 (1993)

    CrossRef  Google Scholar 

  56. Zhang, J., Kang, Y., Yang, Y., Qiu, P.: Statistical monitoring of the hand, foot and mouth disease in China. Biometrics 71, 841–850 (2015)

    CrossRef  MathSciNet  MATH  Google Scholar 

  57. Zhang, J., Qiu, P., Chen, X.: Statistical monitoring-based alarming systems in modeling the AIDS epidemic in the US, 1985–2011. Curr. HIV Res. 14, 130–137 (2016)

    CrossRef  Google Scholar 

  58. Zou, C., Qiu, P.: Multivariate statistical process control using LASSO. J. Am. Stat. Assoc. 104, 1586–1596 (2009)

    CrossRef  MathSciNet  MATH  Google Scholar 

  59. Zou, C., Tsung, F.: Likelihood ratio-based distribution-free EWMA control charts. J. Qual. Technol. 42, 1–23 (2010)

    Google Scholar 

  60. Zou, C., Tsung, F.: A multivariate sign EWMA control chart. Technometrics 53, 84–97 (2011)

    CrossRef  MathSciNet  Google Scholar 

  61. Zou, C., Zhang, Y., Wang, Z.: Control chart based on change-point model for monitoring linear profiles. IIE Trans. 38, 1093–1103 (2006)

    CrossRef  Google Scholar 

  62. Zou, C., Tsung, F., Wang, Z.: Monitoring general linear profiles using multivariate EWMA schemes. Technometrics 49, 395–408 (2007)

    CrossRef  MathSciNet  Google Scholar 

  63. Zou, C., Tsung, F., Wang, Z.: Monitoring profiles based on nonparametric regression methods. Technometrics 50, 512–526 (2008)

    CrossRef  MathSciNet  Google Scholar 

  64. Zou, C., Qiu, P., Hawkins, D.: Nonparametric control chart for monitoring profiles using change point formulation and adaptive smoothing. Stat. Sin. 19, 1337–1357 (2009)

    MathSciNet  MATH  Google Scholar 

  65. Zou, C., Jiang, W., Wang, Z., Zi, X.: An efficient on-line monitoring method for high-dimensional data streams. Technometrics 57, 374–387 (2015)

    CrossRef  MathSciNet  Google Scholar 

Download references

Acknowledgements

This research is supported in part by a US National Science Foundation grant. The author thanks the invitation of the book editor Professor Ejaz Ahmed, and the review of an anonymous referee.

Author information

Authors and Affiliations

  1. Department of Biostatistics, University of Florida, Gainesville, FL, USA

    Peihua Qiu

Authors
  1. Peihua Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peihua Qiu .

Editor information

Editors and Affiliations

  1. Department of Mathematics & Statistics, Brock University, St. Catherines, Ontario, Canada

    S. Ejaz Ahmed

Rights and permissions

Reprints and Permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Qiu, P. (2017). Statistical Process Control Charts as a Tool for Analyzing Big Data. In: Ahmed, S. (eds) Big and Complex Data Analysis. Contributions to Statistics. Springer, Cham. https://doi.org/10.1007/978-3-319-41573-4_7

Download citation

search

Navigation