Some Applications of the EM Algorithm to Analyzing Incomplete Time Series Data

  • R. H. Shumway
Part of the Lecture Notes in Statistics book series (LNS, volume 25)


One may encounter incompletely specified time series data in several distinct forms: (1) observations in time or space may be irregularly observed or (2) the underlying time series model may be incompletely observed, as in the case where one observes only the sum of a signal and a noise process. Maximum likelihood estimators for parameters in these missing data problems can be developed in a simple, heuristically appealing form by utilizing the EM (expectation-maximization) algorithm proposed by Dempster, et al. (1977) and others. Furthermore, the conditional expectations computed as a by-product of applying the algorithm are the empirical Bayes (in the sense of Efron and Morris (1973), (1975)) estimators for the unobserved components. The EM algorithm is reviewed here within the time series context and applied to (i) the parameter estimation and smoothing problem for missing data state-space models and (ii) linear estimation (deconvolution) in a frequency domain regression model.


Discrete Fourier Transform Maximum Likelihood Estimator Conditional Expectation Noise Process Noise Spectrum 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Anderson, T.W. (1971). The Statistical Analysis of Time Series, Wiley, New York.MATHGoogle Scholar
  2. Anderson, T.W. (1978). Repeated measurements on autoregressive processes. J. Amer. Statist. Assoc., 73, 371–378.MathSciNetGoogle Scholar
  3. Backus, G. and Gilbert, F. (1970). Uniqueness in the inversion of inaccurate gross earth data. Trans, Phil. R. Soc. 266A, 123–192.MathSciNetCrossRefGoogle Scholar
  4. Bloomfield, P. (1970). Spectral Analysis with randomly missing observations. Jour. Roy. Statist. Soc., Ser. B, 32, 369–380.MathSciNetMATHGoogle Scholar
  5. Borpujari, A. (1977). An empirical Bayes approach for estimating the mean of N stationary time series. J. Amer. Statist. Assoc., 72, 397–402.MathSciNetMATHGoogle Scholar
  6. Box, G.E. P. and Jenkins, G.M. (1970). Time Series Analysis, Forecasting, and Control, Holden Day, San Francisco.MATHGoogle Scholar
  7. Boyles, R.A. (1980). Convergence results for the EM algorithm. Technical Report No. 13, Division of Statistics, University of California, Davis.Google Scholar
  8. — (1983). On the convergence of the EM algorithm. To appear, J. Royal Statist. Soc., Ser. B, 44.Google Scholar
  9. Brillinger, D.R. (1975). Time Series Data Analysis and Theory. Holt, Reinhart and Winston, New York.MATHGoogle Scholar
  10. — (1980). Analysis of variance under time series models. P.R. Krishnaiah ed., Handbook of Statistics, Vol. 1, 237-278. North Holland, New York.Google Scholar
  11. Dempster, A.P., Laird, N.M. and Rubin, D.B. (1977). Maximum likelihood from incomplete data via the EM algorithm. J. of the Royal Statist. Soc, Ser. B, 39, 1–38.MathSciNetMATHGoogle Scholar
  12. Deregowski, S.M. (1971). Optimum digital filtering and inverse filtering in the frequency domain. Geophys. Prosp., 19, 729–768.CrossRefGoogle Scholar
  13. Donoho, D.L. (1981). On minimum entropy convolution. In D.F. Findley, ed., Applied Time Series Analysis II, 565–608. Academic Press, New York.Google Scholar
  14. Draper, N.R. and Van Nostrand, C. (1979). Ridge regression and James-Stein estimation: Review and Comments. Technometrics, 21, 451–566.MathSciNetMATHGoogle Scholar
  15. Dunsmuir, W. (1981). Estimation for stationary time series when data are irregularly spaced or missing. In Applied Time Series II, D.F. Findley, ed., Academic Press, New York.Google Scholar
  16. Dunsmuir, W. and Robinson, P.M. (1981). Estimation of time series models in the presence of missing data. J. Amer. Statist. Assoc., 76, 560–568.MATHGoogle Scholar
  17. Efron, B. and Morris, C. (1973). Stein’s estimation rule and its competitors — An empirical Bayes approach. J. Amer. Statist. Assoc., 68, 117–130.MathSciNetMATHGoogle Scholar
  18. Efron, B. and Morris, C. (1975). Data analysis using Stein’s estimator and its generalizations. J. Amer. Statist. Assoc., 70, 311–319.MATHGoogle Scholar
  19. Goodrich, R.L. and Caines, P.E. (1979). Linear system identification from nonstationary cross-sectional data. IEEE Trans. on Aut. Control AC-24, 403–411.MathSciNetCrossRefGoogle Scholar
  20. Gupta, N.K. and Mehra, R.K. (1974). Computational aspects of maximum likelihood estimation and reduction in sensitivity function calculations. IEEE Trans. on Aut. Control AC-19, 774–783.MathSciNetCrossRefGoogle Scholar
  21. Hannan, E.J. (1970). Multiple Time Series. Wiley, New York.MATHCrossRefGoogle Scholar
  22. Harrison, P.J. and Stevens, C.F. (1976). Bayesian forecasting. Jour. Royal Statist. Soc., B, 38, 205–247.MathSciNetMATHGoogle Scholar
  23. Hartley, H.O. and Hocking, R.R. (1971). The analysis of incomplete data. Biometrics, 14, 174–194.CrossRefGoogle Scholar
  24. Harvey, A.C. and Phillips, G.D.A. (1979). Maximum likelihood estimation of regression models with autoregressive-moving average disturbances. Biometrika, 66, 49–58.MathSciNetMATHGoogle Scholar
  25. Harvey, A.C. and McKenzie, C.R. (1983). Missing observations in dynamic econometric models. Presented at Office of Naval Research Sponsored Symposium on Time Series Analysis of Irregularly Spaced Data. Feb. 10-13.Google Scholar
  26. Hoerl, A.E. and Kennard, R.W. (1970). Ridge regression: Biased estimation for some non-orthogonal problems. Technometrics, 12, 55–62.MATHGoogle Scholar
  27. Hosoya, Y. (1979). Efficient estimation of a model with an autoregressive signal with white noise. Tech. Report No. 37, Dept. of Statistics, Stanford University.Google Scholar
  28. Hunt, B.R. (1971). Biased estimation for nonparametric identification of linear systems. Math. Biosci., 10, 215–237.MathSciNetMATHCrossRefGoogle Scholar
  29. Hunt, H.R. (1972). Deconvolution of linear systems by constrained regression and its relationship to the Wiener theory. IEEE Trans. Aut. Control, Oct., 703-705.Google Scholar
  30. Jazwinski, A.H. (1970). Stochastic Processes and Filtering Theory. Academic Press, New York.MATHGoogle Scholar
  31. Jones, R.H. (1962). Spectral analysis with regularly missed observations. Ann. Math. Statist., 33, 455–461.MathSciNetMATHCrossRefGoogle Scholar
  32. Jones, R.H. (1966). Exponential smoothing for multivariate time series. J. Roy. Statist. Soc., Ser. B, 1, 241–251.Google Scholar
  33. Jones, R.H. (1980). Maximum likelihood fitting of ARMA models to time series with missing observations. Technometrics, 22, 389–395.MathSciNetMATHGoogle Scholar
  34. Kalman, R.E. (1960). A new approach to linear filtering and prediction problems. Trans. ASME J. of Basic Eng., 8, 35–45.CrossRefGoogle Scholar
  35. Kalman, R.E. and Bucy, R.S. (1961). New results in linear filtering and prediction theory. Trans. ASME J. of Basic Eng., 83, 95–108.MathSciNetCrossRefGoogle Scholar
  36. Kendall, M.G. (1973). Time Series. Hafner Press, New York.Google Scholar
  37. Kitagawa, G. and Gersch, W. (1982). A smoothness priors approach to the modeling of time series with trend and seasonality. Submitted.Google Scholar
  38. Ledolter, J. (1979). A recursive approach to parameter estimation in regression and time series problems. Comm. Statist. Theor. Meth., A8, 1227–1245.MathSciNetMATHCrossRefGoogle Scholar
  39. Meltzer, A., Goodman, C., Langwell, K., Cosler, J., Baghelai, C., and Bobula, J. (1980). Develop Physician and Physician Extender Data Bases. G-155, Final Report, Applied Management Sciences, Inc., Silver Spring, MD 20910.Google Scholar
  40. Murray, G.D. (1977). Contribution to discussion of paper by Dempster, Laird and Rubin. J. Roy. Statist. Soc., Ser. B, 39, 27–28.Google Scholar
  41. Oldenburg, D.W. (1981). A comprehensive solution to the linear deconvolution problem. Geophys. J. R. Astr. Soc., 65, 331–357.CrossRefGoogle Scholar
  42. Oldenburg, D.W., Levy, S. and Whittall, K.P. (1981). Wavelet estimation and deconvolution. Geophysics, 46, 1528–1542.Google Scholar
  43. Orchard, T. and Woodbury, M.A. (1972). A missing information principle: Theory and applications. Proc. 6th Berkeley Symp. on Math. Statist. and Prob., 1, 697–715.MathSciNetGoogle Scholar
  44. Pagano, A. (1974). Estimation of models of autoregressive signal plus white noise. Ann. Statist., 2, 99–108.MathSciNetMATHCrossRefGoogle Scholar
  45. Parzen, E. (1961). Spectral analysis of asymptotically stationary time series. Bull. I.S.I., 33rd Session, Paris.Google Scholar
  46. Parzen, E. (1963). On spectral analysis with missing observations and amplitude modulation. Sankhya, Series A, 25, 383–392.MathSciNetMATHGoogle Scholar
  47. Parzen, E. (1967). Time series analysis for models of signal plus white noise. Spectral Analysis of Time Series, B. Harris, ed., 233-258. WileyGoogle Scholar
  48. Robinson, P.M. (1979). Distributed lag approximation to linear time-invariant systems. Ann. of Statist., 7, 507–515.MATHCrossRefGoogle Scholar
  49. Scheinok, P.S. (1965). Spectral analysis with randomly missed observations: The binomial case. Ann. Math. Statist., 36, 971–977.MathSciNetMATHCrossRefGoogle Scholar
  50. Shumway, R.H. and Dean, W.C. (1968). Best linear unbiased estimation for multivariate stationary processes. Technometrics, 10, 523–534.MATHGoogle Scholar
  51. Shumway, R.H. (1970). Applied regression and analysis of variance for stationary time series. J. Amer. Statist. Assoc., 65, 1527–1546.MATHGoogle Scholar
  52. Shumway, R.H. and Blandford, R.R. (1978). On detecting and estimating multiple arrivals from underground nuclear explosions. Seismic Data Analysis Center Report No. SDAC-TR-77-8. Teledyne-Geotech, P.O. Box 334, Alexandria, Virginia 22313.Google Scholar
  53. Shumway, R.H., Olsen, D.E. and Levy, L.J. (1981). Estimation and tests of hypotheses for the initial mean and covariance in the Kalman filter model. Comm. of Statist., Theor. & Meth., A10(16), 1625–1641.MathSciNetMATHCrossRefGoogle Scholar
  54. Shumway, R.H. and Stoffer, D.S. (1982a). An approach to time series smoothing and forecasting using the EM algorithm. Jour. Time Series Anal., 3, 253–264.MATHCrossRefGoogle Scholar
  55. — (1982b). An algorithm for parameter estimation and smoothing in space-time models with missing data. Final Report P.O. 82-ABA-1198, NOAA Pacific Environmental Group, National Marine Fisheries, Monterey, CA.Google Scholar
  56. Stoffer, D.S. (1982). Estimation of parameters in a linear dynamic system with missing observations. Ph.D. dissertation, University of California, Davis.Google Scholar
  57. Tan, Suan-Boon (1979). Maximum likelihood estimation in autoregressive processes with missing data. Ph.D. thesis, Univ. of Pittsburgh.Google Scholar
  58. Wahba, G. (1969). Estimation of the coefficients in a multidimensional distributed lag model. Econometrica, 37, 398–407.MathSciNetMATHCrossRefGoogle Scholar
  59. Webster, G.M. (1978). Deconvolution. Geophysics Reprint Series, No. 1, Society of Exploration Geophysics, P.O. Box 3098, Tulsa, Oklahoma 74101.Google Scholar
  60. Wiggins, R.A. (1978). Minimum entropy deconvolution. Geoexploration, 16, 21–35.CrossRefGoogle Scholar
  61. Wu, C.F. (1983). On the convergence properties of the EM algorithm. Ann. Statist., 11, 95–103.MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1984

Authors and Affiliations

  • R. H. Shumway
    • 1
  1. 1.Division of StatisticsUniversity of CaliforniaDavisUSA

Personalised recommendations