Abstract
In fitting data with a spline, finding the optimal placement of knots can significantly improve the quality of the fit. However, the challenging high-dimensional and non-convex optimization problem associated with completely free knot placement has been a major roadblock in using this approach. We present a method that uses particle swarm optimization (PSO) combined with model selection to address this challenge. The problem of overfitting due to knot clustering that accompanies free knot placement is mitigated in this method by explicit regularization, resulting in a significantly improved performance on highly noisy data. The principal design choices available in the method are delineated and a statistically rigorous study of their effect on performance is carried out using simulated data and a wide variety of benchmark functions. Our results demonstrate that PSO-based free knot placement leads to a viable and flexible adaptive spline fitting approach that allows the fitting of both smooth and non-smooth functions.
Similar content being viewed by others
References
Akaike H (1998) Information theory and an extension of the maximum likelihood principle. In: Selected papers of Hirotugu Akaike. Springer, Berlin, pp 199–213
Bratton D, Kennedy J, (2007) Defining a standard for particle swarm optimization. In: Swarm intelligence symposium (SIS). IEEE, pp 120–127
Burchard HG (1974) Splines (with optimal knots) are better. Appl Anal 3(4):309–319
Claeskens G, Krivobokova T, Opsomer JD (2009) Asymptotic properties of penalized spline estimators. Biometrika 96(3):529–544
Craven P, Wahba G (1978) Smoothing noisy data with spline functions. Numer Math 31(4):377–403
Curry HB, Schoenberg IJ (1947) On spline distributions and their limits—the polya distribution functions. Bull Am Math Soc 53(11):1114–1114
de Boor C (1972) On calculating with b-splines. J Approx Theory 6(1):50–62
de Boor C (2001) A practical guide to splines (applied mathematical sciences). Springer, Berlin
Denison D, Mallick B, Smith A (1998) Automatic Bayesian curve fitting. J R Stat Soc Ser B (Stat Methodol) 60(2):333–350
DiMatteo I, Genovese CR, Kass RE (2001) Bayesian curve-fitting with free-knot splines. Biometrika 88(4):1055–1071
Donoho DL, Johnstone IM (1995) Adapting to unknown smoothness via wavelet shrinkage. J Am Stat Assoc 90(432):1200–1224
Eilers PH, Marx BD (1996) Flexible smoothing with b-splines and penalties. Stat Sci 11:89–102
Eilers PH, Marx BD, Durbán M (2015) Twenty years of p-splines. SORT: Stat Oper Res Trans 39(2):0149–186
Engelbrecht AP (2005) Fundamentals of computational swarm intelligence, vol 1. Wiley, Chichester
Friedman JH (1991) Multivariate adaptive regression splines. Ann Stat 19(1):1–67
Friedman JH, Silverman BW (1989) Flexible parsimonious smoothing and additive modeling. Technometrics 31(1):3–21
Gálvez A, Iglesias A (2011) Efficient particle swarm optimization approach for data fitting with free knot b-splines. Comput Aided Des 43(12):1683–1692
Goepp V, Bouaziz O, Nuel G (2018) Spline regression with automatic knot selection. arXiv preprint arXiv:1808.01770
Golub GH, Heath M, Wahba G (1979) Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21(2):215–223
Green PJ (1995) Reversible jump markov chain monte carlo computation and bayesian model determination. Biometrika 82(4):711–732
Härdle W (1990) Applied nonparametric regression, no. 19. Cambridge University Press, Cambridge
Huo X, Duncan M, Levi O, Buckheit J, Chen S, Donoho D, Johnstone I, (2000). http://statweb.stanford.edu/~wavelab/Wavelab_850/AboutWaveLab.pdf
Jupp DL (1978) Approximation to data by splines with free knots. SIAM J Numer Anal 15(2):328–343
Kang H, Chen F, Li Y, Deng J, Yang Z (2015) Knot calculation for spline fitting via sparse optimization. Comput Aided Des 58:179–188
Kennedy J (1999) Small worlds and mega-minds: effects of neighborhood topology on particle swarm performance. In: Proceedings of the 1999 congress on evolutionary computation-CEC99 (Cat. No. 99TH8406), vol 3. IEEE, pp 1931–1938
Kennedy J, Eberhart RC (1995) Particle swarm optimization. In: Proceedings of the IEEE international conference on neural networks, Perth, WA, Australia, vol 4. IEEE, p 1942
Krivobokova T (2013) Smoothing parameter selection in two frameworks for penalized splines. J R Stat Soc Ser B (Stat Methodol) 75(4):725–741
Krivobokova T, Crainiceanu CM, Kauermann G (2008) Fast adaptive penalized splines. J Comput Graph Stat 17(1):1–20
Lee T (2002) On algorithms for ordinary least squares regression spline fitting: a comparative study. J Stat Comput Simul 72(8):647–663
Leung C (2015) Estimation of unmodeled gravitational wave transients with spline regression and particle swarm optimization. SIAM Undergraduate Research Online (SIURO) 8
Li M, Yan Y (2006) Bayesian adaptive penalized splines. J Acad Bus Econ 2:129–141
Lindstrom MJ (1999) Penalized estimation of free-knot splines. J Comput Graph Stat 8(2):333–352
Liu Z, Guo W (2010) Data driven adaptive spline smoothing. Stat Sin 20:1143–1163
Luo Z, Wahba G (1997) Hybrid adaptive splines. J Am Stat Assoc 92(437):107–116
Luo J, Kang H, Yang Z (2019) Knot calculation for spline fitting based on the unimodality property. Comput Aided Geom Des 73:54–69
Lyche T, Mørken K (1988) A data-reduction strategy for splines with applications to the approximation of functions and data. IMA J Numer Anal 8(2):185–208
Matlab Release (2018b) The MathWorks, Inc., Natick, MA, United States. http://www.mathworks.com/
Mitchell M (1998) An introduction to genetic algorithms by Melanie Mitchell. MIT Press, Cambridge
Miyata S, Shen X (2003) Adaptive free-knot splines. J Comput Graph Stat 12(1):197–213
Mohanty SD (2012) Particle swarm optimization and regression analysis I. Astron Rev 7(2):29–35
Mohanty SD (2017) Spline based search method for unmodeled transient gravitational wave chirps. Phys Rev D 96:102008. https://doi.org/10.1103/PhysRevD.96.102008
Mohanty SD (2018) Swarm intelligence methods for statistical regression. Chapman and Hall/CRC, London
Normandin ME, Mohanty SD, Weerathunga TS (2018) Particle swarm optimization based search for gravitational waves from compact binary coalescences: performance improvements. Phys Rev D 98(4):044029
Park H, Lee J-H (2007) B-spline curve fitting based on adaptive curve refinement using dominant points. Comput Aided Des 39(6):439–451
Pittman J (2002) Adaptive splines and genetic algorithms. J Comput Graph Stat 11(3):615–638
R Core Team, R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria (2019). https://www.R-project.org/
Ramsay J, Silverman BW (1997) Functional data analysis. Springer series in statistics. Springer, New York
Reinsch CH (1967) Smoothing by spline functions. Numer Math 10(3):177–183
Ruppert D (2002) Selecting the number of knots for penalized splines. J Comput Graph Stat 11(4):735–757
Ruppert D, Carroll RJ (2000) Theory and methods: spatially-adaptive penalties for spline fitting. Austr N Z J Stat 42(2):205–223
Ruppert D, Wand MP, Carroll RJ (2003) Semiparametric regression, vol 12. Cambridge University Press, Cambridge
Scheipl F, Kneib T (2009) Locally adaptive bayesian p-splines with a normal-exponential-gamma prior. Comput Stat Data Anal 53(10):3533–3552
Smith PL (1982) Curve fitting and modeling with splines using statistical variable selection techniques. Technical report, NASA, Langley Research Center, Hampton, VA, report NASA, 166034
Solis FJ, Wets RJ-B (1981) Minimization by random search techniques. Math Oper Res 6(1):19–30
Stone CJ, Hansen MH, Kooperberg C, Truong YK et al (1997) Polynomial splines and their tensor products in extended linear modeling: 1994 Wald memorial lecture. Ann Stat 25(4):1371–1470
Storlie CB, Bondell HD, Reich BJ (2010) A locally adaptive penalty for estimation of functions with varying roughness. J Comput Graph Stat 19(3):569–589
Ülker E, Arslan A (2009) Automatic knot adjustment using an artificial immune system for b-spline curve approximation. Inf Sci 179(10):1483–1494
Wahba G (2002) In discussion Wavelet shrinkage: Asymptopia? with discussion and a reply by Donoho DL, Johnstone IM, Kerkyacharian G, Picard D, J R Stat Soc Ser B 57: 545–564
Wahba G (1990) Spline models for observational data, vol 59. SIAM, Philadelphia
Wand MP (2000) A comparison of regression spline smoothing procedures. Comput Stat 15(4):443–462
Wang X, Du P, Shen J (2013) Smoothing splines with varying smoothing parameter. Biometrika 100(4):955–970
Wegman EJ, Wright IW (1983) Splines in statistics. J Am Stat Assoc 78(382):351–365
Wold S (1974) Spline functions in data analysis. Technometrics 16(1):1–11
Yang L, Hong Y (2017) Adaptive penalized splines for data smoothing. Comput Stat Data Anal 108:70–83
Yoshimoto F, Harada T, Yoshimoto Y (2003) Data fitting with a spline using a real-coded genetic algorithm. Comput Aided Des 35(8):751–760
Zhou S, Shen X (2001) Spatially adaptive regression splines and accurate knot selection schemes. J Am Stat Assoc 96(453):247–259
Acknowledgements
The contribution of S.D.M. to this paper was supported by National Science Foundation (NSF) Grant PHY-1505861. The contribution of E.F. to this paper was supported by NSF Grant PHY-1757830. We acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin (www.tacc.utexas.edu) for providing HPC resources that have contributed to the research results reported within this paper.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Mohanty, S.D., Fahnestock, E. Adaptive spline fitting with particle swarm optimization. Comput Stat 36, 155–191 (2021). https://doi.org/10.1007/s00180-020-01022-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00180-020-01022-x