1 Introduction

Nonlinear dynamical systems are ubiquitous in various fields, encompassing engineering, physics, and biology, exhibiting intricate behaviors such as bifurcations, limit cycles, and chaos [1, 2]. Accurate parameter estimation plays a vital role in effectively modeling these systems [3,4,5,6]. Trust region optimization, a powerful technique for solving nonlinear optimization problems, has shown successful applications in diverse domains [7]. In this article, we propose a method for parameter estimation in three classic nonlinear dynamical systems: the van der Pol oscillator, Damped oscillator, and Lorenz system, utilizing trust region optimization.

The article provides an overview of each system, highlighting their applications in modeling real-world phenomena, and emphasizes the challenges associated with accurately estimating their parameters due to the inherent nonlinearity. It is structured into several sections, commencing with a review of trust region optimization and an examination of prior work in parameter estimation of nonlinear systems. Subsequent sections elaborate on the proposed method for parameter estimation in each system. Finally, the article provides experimental results to validate the proposed method and concludes by highlighting the importance of carefully evaluating the performance of estimated models.

2 Earlier approaches

Parameter estimation of nonlinear dynamical systems is a crucial problem encountered in diverse disciplines, including engineering, physics, and biology [8, 9]. Numerous optimization algorithms have been proposed for this task, such as the Nelder–Mead algorithm, Levenberg–Marquardt algorithm, and genetic algorithms [10,11,12]. However, these methods may suffer from limited global convergence guarantees and sensitivity to initial parameters and optimization settings.

Trust region optimization, on the other hand, has emerged as a powerful technique for addressing nonlinear optimization problems. It is known for providing global convergence guarantees and fast convergence speeds [13]. In recent years, trust region optimization has been successfully applied to parameter estimation of nonlinear systems [14,15,16]. For instance, Ardenghi et al. [16] proposed a trust region optimization algorithm for parameter estimation in the field of biotechnology. Similarly, Zhang et al. [17] introduced a Differential Evolution algorithm-based parameter estimation approach for chaotic systems, surpassing the performance of genetic algorithms and particle swarm optimization in terms of estimation accuracy and convergence speed.

Regarding the Lorenz system, numerous optimization algorithms have been developed for parameter estimation. Cheng et al. [18] proposed a trust region optimization algorithm for parameter estimation in a nonlinear model of an energy storage system, outperforming the Levenberg–Marquardt algorithm and genetic algorithms in terms of estimation accuracy and convergence speed. Similarly, Zheng et al. [19] presented a particle swarm optimization algorithm for parameter estimation in the Lorenz system, achieving superior estimation accuracy and convergence speed compared to other methods such as genetic algorithms and simulated annealing. Furthermore, Lazzús et al. [20] introduced a hybrid optimization algorithm that combines the differential evolution algorithm and the particle swarm optimization algorithm for parameter estimation in the Lorenz system, surpassing other optimization methods in terms of estimation accuracy and convergence speed [21, 22].

In summary, trust region optimization demonstrates promise as a technique for parameter estimation in nonlinear systems, and its successful application has been observed across various domains [23].

3 Materials and methods

Consider a system of ordinary differential equations for state variables, denoted as \({{\textbf{x}}}(t)\), accompanied by parameter estimation challenges [6]. The dynamics of the system are described by the differential equation

$$\begin{aligned} {\dot{{{\textbf{x}}}}}(t) = {{\textbf{F}}}({{\textbf{x}}},t,\theta ), \end{aligned}$$
(1)

where \({{\textbf{x}}}(t) \in {\mathbb {R}}^{n}\) is the state variable, the initial conditions are \({{\textbf{x}}}(0) = {{\textbf{x}}}_{0}\), \(\theta =(\theta _{1},\theta _{2}, \ldots ,\theta _{p}) \in {\mathbb {R}}^{p}\) are the unknown parameters, and \({{\textbf{F}}}: {\mathbb {R}}^{n+p} \rightarrow {\mathbb {R}}^{n}\) is a known vector function. Furthermore, measurements \(\eta _{ij}\) for the state variables or system capacities are available and can be expressed as

$$\begin{aligned} \eta _{ij}= g_{ij}(x(t_{j}),\theta )+\varepsilon _{ij}, \end{aligned}$$
(2)

where \(t_{j}\) denotes the measurement time, \(j=1,2,\ldots ,k\), and \(\varepsilon _{ij}\) is the measurement error.

3.1 Optimization problem

The goal of parameter estimation is to find the values of the unknown parameters \(\theta \) that minimize the discrepancy between the predicted values and the observed data. This objective is achieved by minimizing a suitable objective function that considers the measurement errors \(\eta _{ij}\). One commonly used objective function is the weighted \(l_{2}\) norm of the measurement errors, given by:

$$\begin{aligned} J(\theta ) = \sum _{i,j} \sigma _{ij}^{-2}\varepsilon _{ij}^{2}= \sum _{ij} \sigma _{ij}^{-2}[\eta _{ij}-g_{ij}(x(t_{j}),\theta )]^{2}, \end{aligned}$$
(3)

where \(\sigma _{ij}^{2}\) represents the variance of the measurement errors. The measurement errors are assumed to be independent and follow a Gaussian distribution with zero mean. To address this problem, Trust region optimization algorithms are employed to identify the parameter vector \(\theta \) and trajectory x that minimize the objective function.

3.2 Trust-region optimization

The trust region method solves the parameter estimation problem by iteratively minimizing the objective function within a trust region around the current estimate of the parameter values. At each iteration, a quadratic model is used to approximate the objective function within the trust region, and the optimal step size is computed by solving a constrained optimization problem. The trust region is then updated based on the relative success of previous iterations, and the process is repeated until convergence. The trust region method ensures that the step size is within the trust region and that the objective function is decreasing at each iteration. The mathematical form of the trust region algorithm for parameter estimation in ODE systems is given in Algorithm 1. More mathematical details about the trust region can be found in any standard numerical optimization books [10, 24].

Algorithm 1
figure b

Trust Region Algorithm for Parameter Estimation in ODE Systems

Full size image

In this algorithm, \(g_k\) and \(H_k\) are the gradients and Hessian matrix of the objective function evaluated at \(\theta _k\), and \(\Delta _{\max }\) is the maximum trust region radius. The ratio \(\rho _k\) measures the relative decrease in the objective function between the current and proposed parameter values.

4 Results

In the field of time series analysis, a common objective is to identify an equation that effectively describes the dynamic behavior of a given set of observed variables, denoted as x. This equation should capture both deterministic and stochastic aspects of the system. One approach to achieving this goal is through the use of stochastic differential equations (SDEs) [25].

An SDE of the form

$$\begin{aligned} \frac{\text {d}x}{\text {d}t}=f(x)+g(x)\zeta (t) \end{aligned}$$
(4)

can be used to describe the dynamics of x, where f and g are functions of x, and \(\zeta (t)\) represents noise. Equation (4) specifies how the rate of change of x depends on the current value of x. The deterministic component of the equation is captured by f, which determines the average rate of change of x over time. The stochastic component of the equation is captured by \(g^{2}\), which determines the fluctuation around the average value of x.

When g is constant, the strength of the noise is the same for all values of x, and is referred to as additive or state-independent noise. In contrast, when g depends on x, the strength of the noise varies with the instantaneous value of x, and is referred to as multiplicative or state-dependent noise [26].

We use uncorrelated noise \(\zeta (t)\) with the following properties

$$\begin{aligned} \langle \zeta (t)\rangle= & {} 0\nonumber \\ \langle \zeta (t)\zeta (t^\prime )\rangle= & {} \delta (t-t^\prime ) \end{aligned}$$
(5)

where \(\langle \cdot \rangle \) denotes the time average which is introduced in the equation of motion as follows:

In all cases, the systems were subject to additive noise of different intensities. The effect of white as well as coloured noise [27, 28] is separately investigated as follows.

White Gaussian noise with constant power spectral density across all frequencies, when added to the true data simulates random measurement error or other sources of uncertainty. This affects the observed data and can impact the parameter estimation process. Clearly, the parameter estimation can deteriorate with noise intensity. Colored noise, on the other hand, has a specific frequency distribution with different power levels at different frequencies, and this introduces additional complexity and variability in the observed data, potentially making the parameter estimation more challenging.

We use both white noise and pink (1/f) noise in this work. The accuracy of parameter estimation is judged by the root mean squared error (RMSE)

$$\begin{aligned} RMSE = \sqrt{\frac{1}{N}\sum _{i=1}^{N}(y_i-{\hat{y}}_i)^2} \end{aligned}$$
(6)

N being the total number of samples, \(y_i\) the true value and \({\hat{y}}_i\) the predicted value of sample i.

In order to evaluate the robustness and accuracy of the optimization algorithms, we performed simulations on various systems of different levels of complexity. In each case, we generated noisy data by simulating the system using known parameters and adding Gaussian noise to the output. The optimization algorithms were then applied to estimate the system’s parameters from the noisy data. This process was repeated 10 times for both Gaussian noise and colored (pink) noise, and the average values of the estimated parameters, as well as the RMSE, were computed. The obtained results are consistent with the SINDY methods described by Brunton et al. [29].

In the following examples, we demonstrate the application of the methods described in Sect. 3.2 to identify the governing equations from noisy data. We begin with simple systems to illustrate the effectiveness of the approach, including a comparison between a two-dimensional linear and nonlinear damped oscillator. We also investigate a three-dimensional stable linear system. Subsequently, we examine the van der Pol oscillator in the second example, and finally, we explore the chaotic dynamics of the Lorenz system in the third example.

4.1 Example: simple illustrative systems

4.1.1 Example 1a: two-dimensional damped oscillator (linear vs. nonlinear)

In this example, we investigate a two-dimensional damped harmonic oscillator with linear dynamics described by Eq. (7):

$$\begin{aligned} \frac{\text {d}x}{\text {d}t}=ax+by\nonumber \\ \frac{\text {d}y}{\text {d}t}=cx+dy \end{aligned}$$
(7)

where x, and y represent the variables that describe the state of the system, and a, b, c, d are the parameters of the system with true parameter values with \(a=-0.1\), \(b=2\), \(c=-2\), and \(d=-0.1\).

Fig. 1
figure 1

In Linear damped harmonic oscillator, the trust-region optimization accurately reproduces the dynamics on left panel and the phase portrait on right panel at different levels of gaussian noise (0.0001, 0.001, 0.01, 0.1) with initial conditions \((x_{1},x_{2})=(2.0, 0.0)\)

Full size image
Table 1 Estimated parameters and accuracy at different noise levels for linear system 7
Full size table
Fig. 2
figure 2

The identified system accurately captures the dynamics of the two-dimensional damped harmonic oscillator with cubic dynamics. The solid colored lines represent the true dynamics of the system, while the dashed lines indicate the learned dynamics. The phase portrait demonstrates the precise reproduction of the system’s behavior

Full size image

Figure 1 illustrates the accuracy of the trust-region optimization in reproducing the dynamics and phase portrait of the linear damped harmonic oscillator under different levels of Gaussian noise (0.0001, 0.001, 0.01, 0.1). The initial conditions for the system are set as \((x_{1},x_{2}) = (2.0, 0.0)\).

The estimated parameters for the linear system at each noise level are presented in Table 1. The optimized parameters closely match the true parameter values, indicating the effectiveness of the trust region algorithm in parameter estimation. The accuracy of the estimated trajectories is further quantified by calculating the root mean squared error values, as shown in Table 1.

We also explore the behavior of the damped harmonic oscillator with cubic dynamics, as given by Eq. (8):

$$\begin{aligned} \frac{\text {d}x}{\text {d}t}=ax^{3}+by^{3}\nonumber \\ \frac{\text {d}x}{\text {d}t}=cx^{3}+dy^{3} \end{aligned}$$
(8)

Using the same noise levels, we estimate the parameters for the system with cubic dynamics (Table 3). The estimated parameter values and the corresponding true trajectories are depicted in Fig. 2. The results are summarized in Table 3, which shows the estimated parameters and the RMSE values.

Table 2 Comparison of additive Gaussian and colored noise on parameter estimation and accuracy
Full size table
Table 3 Estimated parameters and accuracy at different noise level for cubic dynamics 8
Full size table

Finally, we compare the effect of Gaussian and colored (pink) noise on parameter estimation and solution accuracy. Tables 2 and 4 presents the average estimated parameter values, and RMSE for linear and nonlinear systems respectively. The estimates obtained with colored noise slightly differ from those obtained with Gaussian noise, indicating a potential bias introduced by the colored noise. The RMSE values quantify the accuracy of the solution, with colored noise achieving comparable accuracy to Gaussian noise.

In summary, the trust-region optimization technique accurately reproduces the dynamics and phase portrait of the two-dimensional damped harmonic oscillator with linear and cubic dynamics, even in the presence of different levels of Gaussian noise. The estimated parameter values closely match the true values, and the optimized trajectories capture the system’s behavior with high accuracy. The comparison between Gaussian and colored noise highlights their similar impact on parameter estimation and solution accuracy.

4.1.2 Example: three-dimensional linear system

In this example, we consider a three-dimensional linear system and its approximation. The dynamics of the system are described by the following equations (Eq. 9):

$$\begin{aligned} \frac{\text {d}x}{\text {d}t}= & {} p_{1}x+p_{2}y\nonumber \\ \frac{\text {d}y}{\text {d}t}= & {} p_{3}x+p_{4}y\nonumber \\ \frac{\text {d}z}{\text {d}t}= & {} p_{5} z \end{aligned}$$
(9)

where x, y, and z represent the variables that describe the state of the system, and \(p_{1}\), \(p_{2}\), \(p_{3}\), \(p_{4}\), \(p_{5}\) are the parameters of the system. The true parameter values used in our simulations are \(p_{1}=-0.1\), \(p_{2}=-2\), \(p_{3}=2\), \(p_{4}=-0.1\), and \(p_{5}=-0.3\). Gaussian noise with different levels (0.0001, 0.001, 0.01, and 0.1) is added to the system trajectories to account for variability.

Table 4 Comparison of additive Gaussian and colored noise on parameter estimation and accuracy
Full size table
Table 5 Estimated parameters and accuracy at different noise levels for three-dimensional linear system (9)
Full size table

We apply the trust region method to estimate the parameters of the three-dimensional linear system from the noisy data. Table 5 presents the estimated parameter values at different noise levels. The optimized parameters closely approximate the true parameter values, indicating the effectiveness of the trust region optimization approach.

The accuracy of the estimated trajectories is evaluated using the root mean squared error as shown in Table 5. The lower RMSE values indicate better accuracy in capturing the dynamics of the system.

Furthermore, we compare the effect of noise characteristics on the parameter estimation and solution accuracy using Gaussian and colored (pink) noise. Table 6 presents the estimated parameter values, and RMSE for both noise types. The estimated parameters show slight variations between the two noise types. The RMSE values are slightly lower for colored (pink) noise compared to Gaussian noise, indicating improved solution accuracy.

Trust region optimization approach effectively reproduces the dynamics of the three-dimensional linear system as shown in Fig. 3, even in the presence of different levels of Gaussian noise. The estimated parameter values closely match the true values, and the optimized trajectories accurately capture the system’s behavior. The comparison between Gaussian and colored (pink) noise highlights the influence of noise characteristics on parameter estimation and solution accuracy, with colored noise yielding slightly improved accuracy. These findings demonstrate the robustness and versatility of our approach in handling various noise scenarios.

4.2 Test problem: van der Pol oscillator

The van der Pol oscillator is a non-linear second-order differential equation that describes the behavior of a damped oscillator. Widely used as a test problem in the field of dynamical systems, the equation are

$$\begin{aligned} \frac{\text {d}^2x}{\text {d}t^2}-\mu (1-x^2)\frac{\text {d}x}{\text {d}t}+x=0 \end{aligned}$$
(10)

where x represents the displacement of the oscillator, t is time, and \(\mu \) is a parameter that controls the nonlinearity [30]. This can be rewritten as a pair of coupled first-order equations,

$$\begin{aligned} \frac{\text {d}x_1}{\text {d}t}= & {} x_2 \nonumber \\ \frac{\text {d}x_2}{\text {d}t}= & {} \mu (1 - x_1^2)x_2 - x_1. \end{aligned}$$
(11)
Fig. 3
figure 3

The trust-region optimization technique successfully reproduces the dynamics of the 3-D Linear system in the left panel, as well as the corresponding phase portrait in the right panel, even when subjected to different levels of Gaussian noise. The initial conditions for the system are set as \((y_{1}, y_{2}, y_{3}) = (0.0, 2.0, 1.0)\)

Full size image

The parameter value is set to \(\mu = 1.5\) in our simulation and the ordinary differential equations (ODEs) are solved in Python using the Scipy.integrate package, employing the odeint function. We initialize the system with \([x_{1_{0}} \hspace{0.1cm} x_{2_{0}}]^{T} = [1.0 \hspace{0.2cm} 0.0]^{T}\) and choose a time-step size of \(\delta t = 0.01\). Gaussian noise is then added to the simulated data, with standard deviations of [0.0001, 0.001, 0.01, 0.1] corresponding to different noise levels.

To estimate the parameters of the van der Pol oscillator from the noisy data, we apply a trust region optimization algorithm. The initial parameter value is set to \(\mu = 1.35\), and the trust region radius is set to 0.1 with a tolerance of \(10^{-6}\). The estimated parameter values are close to the true values, as shown in Table 7. Figure 4 illustrates the accuracy of the estimated trajectories and phase portraits compared to the true trajectories and noisy data.

The accuracy of the parameter estimation is further assessed using the root mean squared error (RMSE), as shown in Table 7. Higher noise levels result in increased RMSE values, indicating reduced accuracy in the estimated trajectories.

Fig. 4
figure 4

In van der Pol oscillator, the trust-region optimization accurately reproduces the trajectories on the left side, and phase portraits on the right side. The initial condition is set as \([x_{1_{0}} \hspace{0.1cm} x_{2_{0}}]^{T} = [1.0 \hspace{0.2cm} 0.0]^{T}\). we compared the resulting trajectories with the true trajectories and the noisy data

Full size image

Furthermore, we compare the effect of noise characteristics on the parameter estimation and solution accuracy by considering Gaussian and colored (pink) noise. The results are presented in Table 8. The estimated value of \(\mu \) is slightly higher when using colored noise (1.4991) compared to Gaussian noise (1.4984), indicating a small bias introduced by colored noise. The solution accuracy, measured by RMSE, is improved when using colored noise compared to Gaussian noise, with lower error values observed for colored noise.

In conclusion, our parameter estimation approach successfully captures the dynamics of the van der Pol oscillator, even in the presence of noise. The choice of noise characteristics has an impact on the accuracy of the estimation, with colored (pink) noise resulting in slightly improved solution accuracy compared to Gaussian noise. These findings highlight the importance of considering noise characteristics in parameter estimation tasks and demonstrate the effectiveness of our trust region optimization approach in handling different noise scenarios.

4.3 Test problem: Lorenz system

The Lorenz system is a set of three non-linear ordinary differential equations that were first studied by Edward Lorenz in the 1960s [31]. It has since become a well-known example in the field of chaos theory. The system describes the evolution of three variables x, y, and z over time, and is given by the equations:

$$\begin{aligned} \frac{\text {d}x}{\text {d}t}&= \sigma (y - x)\nonumber \\ \frac{\text {d}y}{\text {d}t}&= x(\rho - z) - y\nonumber \\ \frac{\text {d}z}{\text {d}t}&= xy - \beta z \end{aligned}$$
(12)

where \(\sigma \), \(\rho \), and \(\beta \) are parameters that determine the behavior of the system. The Lorenz system exhibits chaotic behavior, meaning that even small changes in the initial conditions can lead to significantly different trajectories.

To evaluate the performance of our parameter estimation approach, we conducted simulations using the Lorenz system as the underlying model. We collected data by integrating the system equations over a time interval of \(t = 0\) to \(t = 25\), with a time-step size of \(\Delta t = 0.01\). The true parameters of the Lorenz system were set to \(\sigma = 10.0\), \(\rho = 28.0\), and \(\beta = 8/3\). Gaussian noise was added to the true trajectory at different levels (0.0001, 0.001, 0.01, and 0.1) to generate noisy data.

Table 6 Comparison of additive Gaussian and colored noise on parameter estimation and accuracy
Full size table
Table 7 Estimated parameters and accuracy at different noise levels
Full size table
Table 8 Comparison of additive Gaussian and colored noise on parameter estimation and accuracy
Full size table
Fig. 5
figure 5

We observe the dynamic paths of the Lorenz system, specifically focusing on the case where measurements of both position (x) and velocity (\(\dot{x}\)) are affected by noise. The true trajectories of the system is depicted in blue (solid lines), while the estimated trajectories, obtained through trust-region optimization, is illustrated by dashed red arrows

Full size image
Fig. 6
figure 6

We compare the true phase portrait of the Lorenz systems, spanning from time \(t=0\) to \(t=25\), with the initial condition \([x_{0} \hspace{0.1cm} y_{0} \hspace{0.1cm} z_{0}]^{T} = [-8 \hspace{0.1cm} 7 \hspace{0.1cm} 27]^{T}\), to the phase portrait of the identified systems at different levels of gaussian noise. This allows us to assess how accurately the identified systems capture the dynamics of the original system

Full size image

We employed a trust region optimization approach to estimate the parameters of the Lorenz system from the noisy data. The trust region method accurately captured the underlying dynamics of the system, as illustrated in Fig. 5. Additionally, the phase portraits of the identified systems, shown in Fig. 6, closely matched the true dynamics of the Lorenz system.

Table 9 Estimated parameters and accuracy at different noise levels for Lorenz system
Full size table

The estimated parameter values at different noise levels are presented in Table 9. Despite the presence of noise, the estimated parameters were close to the true values. The accuracy of the estimation was further evaluated using the root mean squared error (RMSE) metrics, as shown in Table 9. Higher noise levels led to increased RMSE values, indicating reduced accuracy in the estimated solution. However, even with relatively high noise levels, the estimated trajectories still captured the underlying dynamics of the Lorenz system.

Table 10 Comparison of additive Gaussian and colored noise on parameter estimation and accuracy
Full size table

To assess the impact of noise characteristics on parameter estimation and solution accuracy, we compared the results obtained using Gaussian noise and colored (pink) noise. The estimated parameter values and error metrics are presented in Table 10. We observed slight variations in the estimated parameter values between the two noise types, suggesting that noise characteristics influenced the estimation process. The accuracy of the estimated solution was slightly lower for Gaussian noise compared to colored (pink) noise, as indicated by higher RMSE values.

In conclusion, our parameter estimation approach successfully captured the dynamics of the Lorenz system, even in the presence of noise. The choice of noise characteristics had an impact on the accuracy of the estimation, with Gaussian noise resulting in slightly lower solution accuracy compared to colored (pink) noise. These findings emphasize the importance of considering noise characteristics in parameter estimation tasks and highlight the effectiveness of our trust region optimization approach in handling different noise scenarios.

5 Conclusion

In this study, we have presented a trust region optimization algorithm for effective parameter estimation in the Van der Pol oscillator, Damped oscillator, and Lorenz system. Our algorithm demonstrates robust performance in estimating parameters for a wide range of models, including highly nonlinear and non-convex systems. By applying the algorithm to the van der Pol, Damped oscillator, and Lorenz systems, we have successfully illustrated its capability to accurately estimate model parameters even in the presence of noise.

Furthermore, we extended our analysis by incorporating colored noise (pink noise) in addition to Gaussian noise. We observed that the choice of noise type had an impact on the accuracy of the estimation. With the presence of colored noise, the estimated trajectories deviated slightly more from the true trajectories compared to Gaussian noise. However, even with relatively high noise levels, the estimated trajectories still captured the underlying dynamics of the systems. This highlights the algorithm’s robustness and effectiveness in dealing with different noise characteristics.

It is important to note that the trust region algorithm is sensitive to the choice of initial parameter values and the size of the trust region. Careful selection of these parameters is crucial to ensure convergence to the correct parameter values. Additionally, accounting for noise in the estimation process is essential to achieve accurate results. Our algorithm effectively incorporates noise and provides reliable parameter estimates even in the presence of noise.

In summary, our proposed trust region optimization algorithm serves as a powerful tool for parameter estimation in nonlinear systems. Its potential impact extends to enhancing the understanding and control of complex real-world systems. The ability to handle different noise characteristics, as demonstrated through the inclusion of colored noise, further strengthens the algorithm’s applicability in real-world scenarios.