Skip to main content
SpringerLink
Log in

A Tabu Search implementation for adaptive localization in ensemble-based methods

  • Methodologies and Application
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

In this paper, we propose an efficient and practical implementation of the ensemble Kalman filter based on adaptive localization via Tabu search. The proposed method works as follows: during assimilation steps, observed components from the model state are split into two groups: the training set and the validation set, after which analysis states are obtained by using the training data, while posterior errors are estimated by means of the validation set. These steps are repeated for a fixed number of iterations and based on a Tabu search implementation, for each model component an optimal radius of influence is estimated. Experimental tests are performed by using the Lorenz 96 model which mimics the chaotic behaviour of the atmosphere. We assess the accuracy of the proposed method by contrasting its numerical results with those obtained by reference filters from the specialized literature such as the local ensemble transform Kalman filter and the ensemble Kalman filter based on modified Cholesky decomposition. Besides, numerical simulations are enriched by using different ensemble sizes, radii of influences (where appropriate), and inflation factors. The results reveal that, for all configurations, the proposed adaptive localization-based filter can improve the accuracy as well as the convergence of ensemble-based methods in the context of sequential data assimilation methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Anderson E, Bai Z, Dongarra J, Greenbaum A, McKenney A, Du Croz J, Hammerling S, Demmel J, Bischof C, Sorensen D (1990) Lapack: a portable linear algebra library for high-performance computers. In: Proceedings of the 1990 ACM/IEEE conference on supercomputing, IEEE Computer Society Press, pp 2–11

  • Arlot S, Celisse A et al (2010) A survey of cross-validation procedures for model selection. Stat Surv 4:40–79

    Article  MathSciNet  MATH  Google Scholar 

  • Askarzadeh A, dos Santos Coelho L, Klein CE, Mariani VC (2016) A population-based simulated annealing algorithm for global optimization. In: 2016 IEEE international conference on systems, man, and cybernetics (SMC), IEEE, pp 004626–004633

  • Bell N, Garland M (2009) Implementing sparse matrix-vector multiplication on throughput-oriented processors. In: Proceedings of the conference on high performance computing networking, storage and analysis, ACM, p 18

  • Bickel PJ, Levina E (2008) Covariance regularization by thresholding. Ann Stat 36:2577–2604

    Article  MathSciNet  MATH  Google Scholar 

  • Bickel PJ, Levina E (2008) Regularized estimation of large covariance matrices. Ann Stat 36(1):199–227

    Article  MathSciNet  MATH  Google Scholar 

  • Bishop CH, Hodyss D (2011) Adaptive ensemble covariance localization in ensemble 4d-var state estimation. Mon Weather Rev 139(4):1241–1255

    Article  Google Scholar 

  • Bishop CH, Toth Z (1999) Ensemble transformation and adaptive observations. J Atmos Sci 56(11):1748–1765

    Article  Google Scholar 

  • Bouttier C, Babando O, Gadat S, Gerchinovitz S, Laporte S, Nicol F (2017) Adaptive simulated annealing with homogenization for aircraft trajectory optimization. In: Operations research proceedings 2015, Springer, pp 569–574

  • Ceberio J, Mendiburu A, Lozano JA (2016) A note on the boltzmann distribution and the linear ordering problem. In: Conference of the Spanish association for artificial intelligence, Springer, pp 441–446

  • Chang S-Y, Saha A (2016) Application of 3d var Kalman filter in a three-dimensional subsurface contaminant transport model for a continuous pollutant source. In: Proceedings of the 2013 national conference on advances in environmental science and technology, Springer, pp 97–104

  • Evensen G (2003) The ensemble Kalman filter: theoretical formulation and practical implementation. Ocean Dyn 53(4):343–367

    Article  Google Scholar 

  • Evensen G (2006) Data assimilation: the ensemble Kalman filter. Springer, New York

    MATH  Google Scholar 

  • Fertig EJ, Harlim J, Hunt BR (2007) A comparative study of 4d-var and a 4d ensemble Kalman filter: perfect model simulations with lorenz-96. Tellus A 59(1):96–100

    Article  Google Scholar 

  • Fu H, Wu X, Li W, Xie Y, Han G, Zhang S (2016) Reconstruction of typhoon structure using 3-dimensional doppler radar radial velocity data with the multigrid analysis: a case study in an idealized simulation context. Adv Meteorol 2016:1–10

    Article  Google Scholar 

  • Furrer R, Genton MG, Nychka D (2006) Covariance tapering for interpolation of large spatial datasets. J Comput Gr Stat 15(3):502–523

    Article  MathSciNet  Google Scholar 

  • Gillijns S, Mendoza OB, Chandrasekar J, De Moor BLR, Bernstein DS, Ridley A (2006) What is the ensemble Kalman filter and how well does it work? In: American control conference, 2006, pages 6 pp –, June 2006

  • Glover F, Laguna M (2013) Tabu Search. Springer, Berlin

    MATH  Google Scholar 

  • Golnari PA, Malik S (2017) Evaluating matrix representations for error-tolerant computing. In: 2017 design, automation and test in Europe conference and exhibition (DATE), IEEE, pp 1659–1662

  • Gottwald GA, Melbourne I (2005) Testing for chaos in deterministic systems with noise. Physica D 212(1):100–110

    Article  MathSciNet  MATH  Google Scholar 

  • Grewal MS (2011) Kalman filtering. In: International Encyclopedia of statistical science, Springer, pp 705–708

  • Greybush SJ, Kalnay E, Miyoshi T, Ide K, Hunt BR (2011) Balance and ensemble Kalman filter localization techniques. Mon Weather Rev 139(2):511–522

    Article  Google Scholar 

  • Gu Y, Wylie BK, Boyte SP, Picotte J, Howard DM, Smith K, Nelson KJ (2016) An optimal sample data usage strategy to minimize overfitting and underfitting effects in regression tree models based on remotely-sensed data. Remote Sens 8(11):943

    Article  Google Scholar 

  • Hamill TM, Whitaker JS, Snyder C (2001) Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon Weather Rev 129(11):2776–2790

    Article  Google Scholar 

  • Houtekamer PL, Mitchell HL (2005) Ensemble Kalman filtering. Q J R Meteorol Soc 131(613):3269–3289

    Article  Google Scholar 

  • Hunt BR, Kostelich EJ, Szunyogh I (2007) Efficient data assimilation for spatiotemporal chaos: a local ensemble transform Kalman filter. Physica D 230(1):112–126

    Article  MathSciNet  MATH  Google Scholar 

  • Kang J-S, Kalnay E, Liu J, Fung I, Miyoshi T, Ide K (2011) Variable localization in an ensemble Kalman filter: application to the carbon cycle data assimilation. J Geophys Res Atmos 116(D9):D09110

    Google Scholar 

  • Karimi A, Paul MR (2010) Extensive chaos in the lorenz-96 model. Chaos Interdiscip J Nonlinear Sci 20(4):043105

    Article  Google Scholar 

  • Kaufman CG, Schervish MJ, Nychka DW (2008) Covariance tapering for likelihood-based estimation in large spatial data sets. J Am Stat Assoc 103(484):1545–1555

    Article  MathSciNet  MATH  Google Scholar 

  • Lorenz EN (2005) Designing chaotic models. J Atmos Sci 62(5):1574–1587

    Article  MathSciNet  Google Scholar 

  • Lou Z, Reinitz J (2016) Parallel simulated annealing using an adaptive resampling interval. Parallel Comput 53:23–31

    Article  MathSciNet  Google Scholar 

  • Luo X, Bhakta T, Nædal G et al (2017) Data driven adaptive localization with applications to ensemble-based 4d seismic history matching. In: SPE Bergen one day seminar. Society of Petroleum Engineers

  • Mattern JP, Song H, Edwards CA, Moore AM, Fiechter J (2017) Data assimilation of physical and chlorophyll a observations in the california current system using two biogeochemical models. Ocean Model 109:55–71

    Article  Google Scholar 

  • Merrill D, Garland M (2016) Merge-based sparse matrix-vector multiplication (spmv) using the CSR storage format. In: Proceedings of the 21st ACM SIGPLAN symposium on principles and practice of parallel programming, ACM, p 43

  • Miyoshi T (2011) The Gaussian approach to adaptive covariance inflation and its implementation with the local ensemble transform Kalman filter. Mon Weather Rev 139(5):1519–1535

    Article  Google Scholar 

  • Nerger L, Schulte S, Bunse-Gerstner A (2014) On the influence of model nonlinearity and localization on ensemble Kalman smoothing. Q J R Meteorol Soc 140(684):2249–2259

    Article  Google Scholar 

  • Nino Ruiz ED, Sandu A (2016) A derivative-free trust region framework for variational data assimilation. J Comput Appl Math 293:164–179

    Article  MathSciNet  MATH  Google Scholar 

  • Nino Ruiz ED, Sandu A, Anderson J (2015) An efficient implementation of the ensemble Kalman filter based on an iterative Sherman–Morrison formula. Stat Comput 25(3):561–577

    Article  MathSciNet  MATH  Google Scholar 

  • Niño ED (2012) Samods and sagamods: novel algorithms based on the automata theory for the multiobjective optimization of combinatorial problems. Int J Artif Intell 8(S12):147–165

    Google Scholar 

  • Niño ED, Ardila CJ, Barrios A, Donoso Y (2011) Mods: a novel metaheuristic of deterministic swapping for the multi-objective optimization of combinatorials problems. Constraints 9:1

    Google Scholar 

  • Nino-Ruiz ED, Ardila C, Capacho R (2017) Local search methods for the solution of implicit inverse problems. Soft Comput, 1–14

  • Nino-Ruiz ED, Sandu A (2017) Efficient parallel implementation of DDDAS inference using an ensemble Kalman filter with shrinkage covariance matrix estimation. Cluster Comput. https://doi.org/10.1007/s10586-017-1407-1

  • Nino-Ruiz ED, Sandu A, Deng X (2015) A parallel ensemble Kalman filter implementation based on modified Cholesky decomposition. In: Proceedings of the 6th workshop on latest advances in scalable algorithms for large-scale systems, ACM, p 4

  • Nino-Ruiz ED, Sandu A, Deng X (2017) A parallel implementation of the ensemble Kalman filter based on modified Cholesky decomposition. J Comput Sci

  • Obrien RM (2007) A caution regarding rules of thumb for variance inflation factors. Qual Quant 41(5):673–690

    Article  Google Scholar 

  • Ott E, Hunt BR, Szunyogh I, Zimin AV, Kostelich EJ, Corazza M, Kalnay E, Patil DJ, Yorke JA (2004) A local ensemble Kalman filter for atmospheric data assimilation. Tellus A 56(5):415–428

    Article  Google Scholar 

  • Ott E, Hunt B, Szunyogh I, Zimin AV, Kostelich EJ, Corazza M, Kalnay E, Patil DJ, Yorke JA (2008) A local ensemble transform Kalman filter data assimilation system for the NCEP global model. Tellus A 60(1):113–130

    Article  Google Scholar 

  • Petra CG, Zavala VM, Nino-Ruiz ED, Anitescu M (2016) A high-performance computing framework for analyzing the economic impacts of wind correlation. Electr Power Syst Res 141:372–380

    Article  Google Scholar 

  • Prasad VS, Johny CJ, Sodhi JS (2016) Impact of 3d Var GSI-ENKF hybrid data assimilation system. J Earth Syst Sci 125(8):1509–1521

    Article  Google Scholar 

  • Refaeilzadeh P, Tang L, Liu H (2009) Cross-validation. In: Encyclopedia of database systems, Springer, pp 532–538

  • Rothman AJ, Levina E, Zhu J (2009) Generalized thresholding of large covariance matrices. J Am Stat Assoc 104(485):177–186

    Article  MathSciNet  MATH  Google Scholar 

  • Shahvari O, Logendran R (2017) An enhanced tabu search algorithm to minimize a bi-criteria objective in batching and scheduling problems on unrelated-parallel machines with desired lower bounds on batch sizes. Comput Oper Res 77:154–176

    Article  MathSciNet  MATH  Google Scholar 

  • Soto M, Sevaux M, Reinholz A, Rossi A (2017) Multiple neighborhood search, tabu search and ejection chains for the multi-depot open vehicle routing problem. Comput Ind Eng 107:211–222

    Article  Google Scholar 

  • Stein ML (2014) Limitations on low rank approximations for covariance matrices of spatial data. Spat Stat 8:1–19

    Article  MathSciNet  Google Scholar 

  • Tippett MK, Anderson JL, Bishop CH, Hamill TM, Whitaker JS (2003) Ensemble square root filters. Mon Weather Rev 131(7):1485–1490

    Article  Google Scholar 

  • Tony Cai T, Yuan M et al (2012) Adaptive covariance matrix estimation through block thresholding. Ann Stat 40(4):2014–2042

    Article  MathSciNet  MATH  Google Scholar 

  • Yu W, Yin P, Liu T, Yang L, Liu X (2016) Study on the optimal mode of foreign investment based on simulated annealing algorithm. J Comput Theor Nanosci 13(5):3019–3023

    Article  Google Scholar 

  • Zupanski M (2009) Theoretical and practical issues of ensemble data assimilation in weather and climate. In: Park SK, Xu L (eds) Data assimilation for atmospheric, oceanic and hydrologic applications. Springer, Berlin, pp 67–84

    Chapter  Google Scholar 

Download references

Acknowledgements

This work was supported by the Applied Math and Computer Science Laboratory (AML-CS) at Universidad del Norte, BAQ, COL.

Author information

Authors and Affiliations

  1. Applied Math and Computer Science Laboratory (AML-CS), Department of Computer Science, Universidad del Norte, Barranquilla, 080001, Colombia

    Elias D. Nino-Ruiz & Luis E. Morales-Retat

Authors
  1. Elias D. Nino-Ruiz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luis E. Morales-Retat
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elias D. Nino-Ruiz.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Human and animals rights

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Communicated by V. Loia.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nino-Ruiz, E.D., Morales-Retat, L.E. A Tabu Search implementation for adaptive localization in ensemble-based methods. Soft Comput 23, 5519–5535 (2019). https://doi.org/10.1007/s00500-018-3210-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-018-3210-1

Keywords

Search

Navigation