Abstract
State-space systems arise in many applications in signal processing and data assimilation. In this context, the main goal is to estimate online the state of the system based on available observations, the so-called filtering problem. Standard filtering solutions are computed recursively as successive cycles of alternating time-update (forecast) and observation-update (analysis) steps. This path is however not the only recursive way to compute the filtering solution. In this context, new one-step-ahead smoothing (OSAS)-like filtering schemes have been introduced, reversing the order of the observation-update and the time-update steps. These involve two Bayesian-like update steps based on the same (present) observation: one for smoothing the previous state and one for analyzing the present one. These include new variants of Kalman filters (KF-OSAS), particle filters (PF-OSAS) and ensemble Kalman filters (EnKF-OSAS), depending on the size and the linear-Gaussian character of the underlying state-space system. While the standard KF and KF-OSAS provide the same (exact) estimator, the use of the same data twice in the estimation process generally leads to improved trade-off between estimation quality and computational burden for the PF-OSAS and EnKF-OSAS. This chapter offers a comprehensive presentation of the OSAS-like filtering algorithms, reviewing their derivations, detailing algorithmic and practical differences and similarities with their classical counterparts, and discussing their relevance for both small- and large-dimensional applications.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Without loss of generality, throughout the chapter PF refers to the particular bootstrap algorithm of Gordon et al. (1993), which is commonly used in geophysics problems. Sequential Importance Sampling (SIS) or Sampling Importance Resampling (SIR) algorithms will designate the other importance sampling (IS)-based algorithms.
- 2.
Without loss of generality, the term \(\mathbf{g}(\mathbf{m})\) in any modeling equation \(\mathbf{d} = \mathbf{g}(\mathbf{m}) + { noise}\), will be called operator, whereas the whole equation will be called model, i.e., equations in (1) refer to as dynamical and observational models; ditto for those in (25).
- 3.
Notice that (13) is simply an “averaging” of \(p(\mathbf{y}_n | \mathbf{x}_{n})\) w.r.t. \(p(\mathbf{x}_n | \mathbf{x}_{n-1})\) to obtain \(p(\mathbf{y}_n | \mathbf{x}_{n-1})\), i.e., \(p(\mathbf{y}_n | \mathbf{x}_{n-1}) {\mathop {=}\limits ^{(13)}} \mathbb {E}_{p(\mathbf{x}_n |\mathbf{x}_{n-1})} [p(\mathbf{y}_n | \mathbf{x}_n)]\).
- 4.
Recall that the OSAS-like filter reverts the order of the time-update and observation-update steps (i.e., step (12) then (14)), involving the use of the new likelihood before the new transition density. Thus, for the sake of consistency, we revert the conventional order of the models in the new system.
- 5.
Notice that another OSAS-like KF has been introduced in Ait-El-Fquih and Hoteit (2015), but is left out in this chapter. Its difference with the KF-OSAS lies in the fact that (i) it does not follow from the generic filter (12)–(15), but from splitting the state vector using the variational Bayesian approach, and (ii) it only computes an approximation of the analysis and smoothing pdfs.
- 6.
Without abuse of language, we use iid throughout the chapter even though this is true only asymptotically (in M). Indeed, with finite M, the particles are identically distributed (id) from the associated density, but not independent.
- 7.
The term \(p(\mathbf{y}_n |\mathbf{x}_{n-1}^{a,m})\) in (42) is not a probability density function, but the value of this function at point \(\mathbf{y}_n\) (i.e., the observed data).
- 8.
Steps W and S commute in SIR-CID since the weights do not depend on the new particles \(\mathbf{z}_n^{m}\).
- 9.
It follows from (54) that \(\mathbf{y}_n - \mathbf{y}_{n}^{f,m} = \mathbf{y}_n - \tilde{\mathbf{v}}_n^m - \tilde{\mathbf{h}}_{n-1} (\mathbf{x}_{n-1}^{a,m})\), suggesting that the Kalman correction in (53) is based on observations perturbed with \(-\tilde{\mathbf{v}}_n^{m} \sim \mathcal{N}(\mathbf{0}, \tilde{\mathbf{R}}_n)\). Similarly to the classical EnKF, perturbing the observations for each ensemble member enables matching with the error statistics in the KF-OSAS smoothing step when the state-space system is linear-Gaussian.
- 10.
Note that members \({\boldsymbol{\upsilon }}_{n}^m\) correspond to (28) and their covariance, \(\mathbf{P}_{{\boldsymbol{\upsilon }}_n}\), is an ensemble-based approximation of the form (29) of the innovation matrix \(\mathbf{V}_n\). One could also perturb the observations by \(\mathbf{v}_n^{m}\) instead of \(\tilde{\mathbf{v}}_n^{m}\) (i.e., \(\mathbf{y}_{n}^{m}\) become \(\mathbf{y}_{n}^{m} = \mathbf{y}_n - \mathbf{v}_n^{m}\)); in this case the members \({\boldsymbol{\upsilon }}_{n}^m\) become \({\boldsymbol{\upsilon }}_{n}^m = \mathbf{y}_{n}^m - \mathbf{H}_n \mathbf{x}_{n}^{f,m}\) (which correspond to the form (33) of the innovation) and their covariance, \(\mathbf{P}_{{\boldsymbol{\upsilon }}_n}\), corresponds to the form (34) of \(\mathbf{V}_n\).
- 11.
References
Ades M, van Leeuwen PJ (2013) An exploration of the equivalent weights particle filter. Q J R Meteorol Soc 139:820–840
Ait-El-Fquih B, Desbouvries F (2006) Kalman filtering in triplet Markov chains. IEEE Trans Signal Process 54(8):2957–2963
Ait-El-Fquih B, Desbouvries F (2011) Fixed-interval Kalman smoothing algorithms in singular state-space systems. J Signal Process Syst 65(3):469–478
Ait-El-Fquih B, Hoteit I (2015) Fast Kalman-like filtering in large-dimensional linear and Gaussian state-space models. IEEE Trans Signal Process 63(21):5853–5867
Ait-El-Fquih B, Hoteit I (2016) A variational Bayesian multiple particle filtering scheme for large-dimensional systems. IEEE Trans Signal Process 64(20):5409–5422
Ait-El-Fquih B, Hoteit I (2018) An efficient state-parameter filtering scheme combining ensemble Kalman and particle filters. Mon Weather Rev 146:871–887
Ait-El-Fquih B, Hoteit I (2020) A particle filter-based adaptive inflation scheme for the ensemble Kalman filter. Q J R Meteorol Soc 146:922–937
Ait-El-Fquih B, Gharamti ME, Hoteit I (2016) A Bayesian consistent dual ensemble Kalman filter for state-parameter estimation in subsurface hydrology. Hydrol Earth Syst Sci 20:3289–3307
Ait-El-Fquih B, Hoteit I (2015) An efficient multiple particle filter based on the variational Bayesian approach. In: Proceedings of the IEEE international ISSPIT symposium
Aksoy A, Zhang F, Nielsen-Gammon JW (2006) Ensemble-based simultaneous state and parameter estimation with MM5. Geophys Res Lett 33:L12801
Anderson JL (2001) An ensemble adjustment Kalman filter for data assimilation. Mon Weather Rev 129:2884–2903
Anderson JL (2009) Ensemble Kalman filters for large geophysical applications. IEEE Control Syst Mag 29(3):66–82
Anderson BDO, Moore JB (1979) Optimal filtering. Prentice Hall, Englewood Cliffs, New Jersey
Annan JD, Lunt DJ, Hargreaves JC, Valdes PJ (2005) Parameter estimation in an atmospheric GCM using the ensemble Kalman filter. Nonlinear Process Geophys 12:363–371
Arulampalam MS, Maskell S, Gordon N, Clapp T (2002) A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Trans Signal Process 50(2):174–188
Asch M, Bocquet M, Nodet M (2016) Data assimilation: methods, algorithms, and applications. SIAM
Bar-Shalom Y, Li X, Kirubarajan T (2001) Estimation with applications to tracking and navigation. Wiley, New York
Bellsky T, Berwald J, Mitchell L (2014) Nonglobal parameter estimation using local ensemble Kalman filtering. Mon Weather Rev 142:2150–2164
Bishop CH, Etherton BJ, Majumdar SJ (2001) Adaptive sampling with the ensemble transform Kalman Filter. Part I: Theoretical aspects. Mon Weather Rev 129:420–436
Burgers G, van Leeuwen PJ, Evensen G (1998) Analysis scheme in the ensemble Kalman filter. Mon Weather Rev 126:1719–1724
Cappé O, Moulines E, Rydén T (2005) Inference in hidden Markov models. Springer
Carrassi A, Bocquet M, Bertino L, Evensen G (2018) Data assimilation in the geosciences: an overview of methods, issues, and perspectives. In: Wiley interdisciplinary reviews: climate change, p e535
Chen Y, Zhang D (2006) Data assimilation for transient flow in geologic formations via ensemble Kalman filter. Adv Water Resour 29:1107–1122
Chui CK, Chen G (1999) Kalman filtering with real-time applications. Springer, Berlin, DE
Crisan D, Doucet A (2002) A survey on convergence results on particle filtering methods for practitioners. IEEE Trans Signal Process 50(3):736–746
Desbouvries F, Petetin Y, Ait-El-Fquih B (2011) Direct, prediction- and smoothing-based Kalman and particle filter algorithms. Signal Process 91(8):2064–2077
Desbouvries F, Ait-El-Fquih B (2008) Direct, prediction-based and smoothing-based particle filter algorithms. In: Proceedings of the international conference on IASC
Djuric PM, Bugallo MF (2013) Particle filtering for high-dimensional systems. In: Proceedings of the IEEE international workshop on CAMSAP
Djuric PM, Zhang J, Ghirmai T, Huang Y, Kotecha JH (2002) Applications of particle filtering to communications: a review. In: Proceedings of the European conference on EUSIPCO
Doucet A, Godsill SJ, Andrieu C (2000) On sequential monte Carlo sampling methods for Bayesian filtering. Stat Comput 10:197–208
Doucet A, de Freitas N, Gordon N (eds) (2001) Sequential Monte Carlo methods in practice. Statistics for engineering and information science. Springer, New York
Dreano D, Tandeo P, Pulido M, Ait-El-Fquih B, Chonavel T, Hoteit I (2017) Estimating model-error covariances in nonlinear state-space models using Kalman smoothing and the expectation-maximization algorithm. Q J R Meteorol Soc 143:1877–1885
Dunne SC, Entekhabi D, Njoku EG (2007) Impact of multiresolution active and passive microwave measurements on soil moisture estimation using the ensemble Kalman smoother. IEEE Trans Geosci Remote Sens 45(4):1016–1028
Durrant-Whyte H, Bailey T (2006) Simultaneous localisation and mapping (SLAM): Part I: The essential algorithms. IEEE Robot Autom Mag 13:99–110
Ephraim Y, Merhav N (2002) Hidden Markov processes. IEEE Trans Inf Theory 48(6):1518–1569
Evensen G (1994) Sequential data assimilation with nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J Geophys Res 99(C5):143–162
Evensen G (2006) Data assimilation: the ensemble Kalman filter. Springer, New York
Evensen G, van Leeuwen PJ (2000) An ensemble Kalman smoother for nonlinear dynamics. Mon Weather Rev 128(6):1852–1867
Frei M, Künsch H (2013) Bridging the ensemble Kalman and particle filters. Biometrika 100:781–800
Furrer R, Bengtsson T (2007) Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants. J Multivar Anal 98(2):227–255
Gharamti ME, Valstar J, Hoteit I (2014) An adaptive hybrid EnKF-OI scheme for efficient state-parameter estimation of reactive contaminant transport models. Adv Water Resour 71:1–15
Gharamti ME, Kadoura A, Valstar J, Sun S, Hoteit I (2014) Constraining a compositional flow model with flow-chemical data using an ensemble-based Kalman filter. Water Resour Res 50:2444–2467
Gharamti ME, Ait-El-Fquih B, Hoteit I (2015) An iterative ensemble Kalman filter with one-step-ahead smoothing for state-parameters estimation of contaminant transport models. J Hydrol 527:442–457
Gordon NJ, Salmond DJ, Smith AFM (1993) Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEE Proc F 140:107–113
Gustafsson F (2010) Particle filter theory and practice with positioning applications. IEEE Trans Aerosp Electron Syst Mag 25(7):53–82
Harvey AC (1989) Forecasting, structural time series models and the Kalman filter. Cambridge University Press
Haykin S, Huber K, Chen Z (2004) Bayesian sequential state estimation for MIMO wireless communications. Proc IEEE 92(3):439–454
Hendricks Franssen HJ, Kinzelbach W (2008) Real-time groundwater flow modeling with the ensemble Kalman filter: joint estimation of states and parameters and the filter inbreeding problem. Water Resour Res 44(9)
Hoteit I et al (2020) Towards an end-to-end analysis and prediction system for weather, climate, and marine applications in the red sea. Bull Am Meteorol Soc. https://doi.org/10.1175/BAMS-D-19-0005.1
Hoteit I, Pham D-T, Blum J (2002) A simplified reduced order Kalman filtering and application to altimetric data assimilation in tropical pacific. J Mar Syst 36(1–2):101–127
Hoteit I, Pham D-T, Triantafyllou G, Korres G (2008) A new approximate solution of the optimal nonlinear filter for data assimilation in meteorology and oceanography. Mon Weather Rev 136(1):317–334
Hoteit I, Luo X, Pham D-T (2012) Particle Kalman filtering: a nonlinear Bayesian framework for ensemble Kalman filters. Mon Weather Rev 140:528–542
Hoteit I, Pham D-T, Gharamti ME, Luo X (2015) Mitigating observation perturbation sampling errors in the stochastic EnKF. Mon Weather Rev 143:2918–2936
Hoteit I, Luo X, Bocquet M, Köhl A, Ait-El-Fquih B (2018) New frontiers in operational oceanography. In: Data assimilation in oceanography: current status and new directions. GODAE Ocean View, pp 465–512
Houtekamer PL, Mitchell HL (1998) Data assimilation using an ensemble Kalman filter technique. Mon Weather Rev 126:796–811
Houtekamer PL, Mitchell HL (2005) Ensemble Kalman filtering. Q J R Meteorol Soc 131:3269–3289
Hunt BR, Kostelich EJ, Szunyogh I (2007) Efficient data assimilation for spatiotemporal chaos: a local ensemble transform Kalman filter. Physica D 230:112–126
Husz ZL, Wallace AM, Green PR (2011) Tracking with a hierarchical partitioned particle filter and movement modelling. IEEE Trans Syst Man Cybern B Cybern 41(6):1571–1584
Jazwinski AH (1970) Stochastic processes and filtering theory. Mathematics in science and engineering, vol 64. Academic Press, San Diego
Kailath T, Sayed AH, Hassibi B (2000) Linear estimation. Prentice Hall information and system sciences series. Prentice Hall, Upper Saddle River, NJ
Kalman RE (1960) A new approach to linear filtering and prediction problems. Trans ASME J Basic Eng Ser D 82(1):35–45
Khaki M, Ait-El-Fquih B, Hoteit I (2020) Calibrating land hydrological models and enhancing their forecasting skills using an ensemble Kalman filter with one-step-ahead smoothing. J Hydrol 584
Kivman GA (2003) Sequential parameter estimation for stochastic systems. Nonlinear Process Geophys 10(3):253–259
Künsch H (2001) State space and hidden Markov models. In: Barndorff-Nielsen OE, Cox DR, Klüppelberg C (eds) Complex stochastic systems. CRC Press, pp 109–173
Lamberti R, Petetin Y, Desbouvries F, Septier F (2017) Independent resampling sequential Monte Carlo algorithms. IEEE Trans Signal Process 65(20):5318–5333
Le Dimet F-X, Talagrand O (1986) Variational algorithms for analysis and assimilation of meteorogical observations: theoretical aspects. Tellus 38A:97–110
Liu JS, Chen R (1998) Sequential Monte Carlo methods for dynamic systems. J Am Stat Assoc 93(443):1032–1044
Liu B, Ait-El-Fquih B, Hoteit I (2015) Efficient Kernel-based ensemble Gaussian mixture filtering. Mon Weather Rev 144:781–800
Mandel J, Beezley JD, Coen JL, Minjeong K (2009) Data assimilation for wildland fires. IEEE Control Syst Mag 29(3):47–65
Moradkhani H, Sorooshian S, Gupta HV, Houser PR (2005) Dual state-parameter estimation of hydrological models using ensemble Kalman filter. Adv Water Resour 28(2):135–147
Morzfelda M, Tub X, Atkins E, Chorina AJ (2012) A random map implementation of implicit filters. J Comput Phys 231:2049–2066
Pham D-T (2001) Stochastic methods for sequential data assimilation in strongly nonlinear systems. Mon Weather Rev 129:1194–1207
Pham D-T, Verron J, Rouband MC (1998) Singular evolutive Kalman filter with EOF initialization for data assimilation in oceanography. J Mar Syst 16(3–4):323–340
Pitt MK, Shephard N (1999) Filtering via simulation: auxiliary particle filter. J Am Stat Assoc 94(446):550–599
Raboudi NF, Ait-El-Fquih B, Subramanian AC, Hoteit I (2020) Enhancing ensemble data assimilation into one-way-coupled models with one-step-ahead smoothing. Quart J R Meteorol Soc. https://doi.org/10.1002/qj.3916
Raboudi NF, Ait-El-Fquih B, Hoteit I (2018) Ensemble Kalman filtering with one-step-ahead smoothing. Mon Weather Rev 146:561–581
Raboudi NF, Ait-El-Fquih B, Dawson C, Hoteit I (2019) Combining hybrid and one-step-ahead smoothing for efficient/short-range storm surge forecasting with an ensemble Kalman filter. Mon Weather Rev 147:3283–3300
Rasmussen J, Madsen H, Jensen KH, Refsgaard JC (2015) Data assimilation in integrated hydrological modeling using ensemble Kalman filtering: evaluating the effect of ensemble size and localization on filter performance. Hydrol Earth Syst Sci 19:2999–3013
Rubin DB (1988) Using the SIR algorithm to simulate posterior distributions. In: Bernardo JM, DeGroot MH, Lindley DV, Smith AFM (eds) Bayesian statistics, vol 3. Oxford University Press, pp 395–402
Septier F, Peters GW (2015) An overview of recent advances in Monte-Carlo methods for Bayesian filtering in high-dimensional spaces. In: Peters GW, Matsui T (eds) Theoretical aspects of spatial-temporal modeling. Briefs - JSS research series in statistics. Springer
Sherman S (1955) A theorem on convex sets with applications. Ann Math Stat 26:763–767
Snyder C, Bengtsson T, Bickel P, Anderson J (2008) Obstacles to high-dimensional particle filtering. Mon Weather Rev 136(12):4629–4640
Spiller ET, Budhiraja A, Ide K, Jones CK (2008) Modified particle filter methods for assimilating Lagrangian data into a point-vortex model. Phys D 237(10):1498–1506
Stordal AS, Karlsen HA, Naevdal G, Skaug HJ, Valles B (2011) Bridging the ensemble Kalman filter and particle filters: the adaptive Gaussian mixture filter. Comput Geosci 15(2):293–305
Subramanian A, Hoteit I, Cornuelle B, Song H (2012) Linear vs. nonlinear filtering with scale selective corrections for balanced dynamics in a simple atmospheric model. J Atmos Sci 69:3405–3419
Tippett M, Anderson J, Bishop C, Hamill T, Whitaker J (2003) Ensemble square root filters. Mon Weather Rev 131(7):1485–1490
van Leeuwen PJ (2009) Particle filtering in geophysical systems. Mon Weather Rev 137(12):4089–4114
van Trees HL (1968) Detection, estimation, and modulation theory: Part I. Wiley, New York
Wen XH, Chen WH (2007) Real-time reservoir updating using ensemble Kalman filter: the confirming approach. Soc Pet Eng 11:431–442
Whitaker JS, Hamill TS (2002) Ensemble data assimilation without perturbed observations. Mon Weather Rev 130:1913–1924
Whitaker JS, Hamill HT, Wei X, Song Y, Toth Z (2008) Ensemble data assimilation with the NCEP global forecast system. Mon Weather Rev 136:463–481
Yardim C, Michalopoulou Z-H, Gerstoft P (2011) An overview of sequential Bayesian filtering in ocean acoustics. IEEE J Oceanic Eng 36(1):71–89
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix
Appendix
Property 1
(Hierarchical sampling Ait-El-Fquih et al. 2016) Assuming that one can sample from \(p(\mathbf{x}_1)\) and \(p(\mathbf{x}_{2}|\mathbf{x}_{1})\), then a sample, \(\mathbf{x}_2^{*}\), from \(p(\mathbf{x}_2)\) can be drawn as follows:
-
1.
\(\mathbf{x}_1^{*} \sim p(\mathbf{x}_1)\);
-
2.
\(\mathbf{x}_2^{*} \sim p(\mathbf{x}_2|\mathbf{x}_1^{*})\).
Property 2
(Conditional sampling Ait-El-Fquih et al. 2016) Consider a Gaussian pdf, \(p(\mathbf{x} , \mathbf{y})\), with \(\mathbf{P}_{xy}\) and \(\mathbf{P}_{y}\) denoting the cross-covariance of \(\mathbf{x}\) and \(\mathbf{y}\) and the covariance of \(\mathbf{y}\), respectively. Then a sample, \(\mathbf{x}^{*}\), from \(p(\mathbf{x}|\mathbf{y})\), can be drawn as follows:
-
1.
\((\tilde{\mathbf{x}} , \tilde{\mathbf{y}}) \sim p(\mathbf{x} , \mathbf{y})\);
-
2.
\(\mathbf{x}^{*} = \tilde{\mathbf{x}} + \mathbf{P}_{xy} \mathbf{P}_y^{-1} [\mathbf{y} - \tilde{\mathbf{y}}]\).
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Ait-El-Fquih, B., Hoteit, I. (2022). Filtering with One-Step-Ahead Smoothing for Efficient Data Assimilation. In: Park, S.K., Xu, L. (eds) Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications (Vol. IV). Springer, Cham. https://doi.org/10.1007/978-3-030-77722-7_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-77722-7_3
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77721-0
Online ISBN: 978-3-030-77722-7
eBook Packages: Earth and Environmental ScienceEarth and Environmental Science (R0)