1 Introduction

The number of ship crews engaged in coastal shipping is aging, and there has been a shortage of manpower in recent years [1]. One of the solutions to this problem is the autonomous ship. Significant studies have been conducted across the globe on this topic to solve the various challenges associated with autonomous ships. For instance, automatic berthing/unberthing experiments using actual vessels were successfully conducted in the 1980s in Japan, and research on automatic berthing/unberthing has a long research history [2].

Several issues need to be resolved in establishing this automatic berthing/unberthing technology. For instance, establishing an accurate maneuvering motion model, establishing optimal berthing paths, and developing automatic berthing/unberthing algorithms [3,4,5].

To achieve automatic berthing and unberthing, accurate state estimation must be performed within the control period. Developing a real-time accurate state estimation method is one of the challenges. State estimation in this research refers to the estimation of states in a system from observed values obtained using sensors, filters, and other methods.

To avoid damage to the ship and pier when a vessel contacts the pier during berthing, the speed at which the vessel hits the pier, i.e., the berthing speed, should be sufficiently slow [6]. Roubos conducted a statistical study of berthing speed in Rotterdam. The results showed that the average berthing speed in 555 examples of common merchant vessels, including container ships, tankers, and bulk carriers, was approximately 4 \(\mathrm {[cm/s]}\) [7]. In addition, some ports have established the lateral (sway) velocity as a rule. For instance, at the Kinjo Wharf in the Port of Nagoya in Japan, large passenger ships are required to berth at no more than 8 \(\mathrm {[cm/s]}\) [8]. As described above, it is essential for safety to estimate the ship states with the highest possible accuracy to keep the berthing speed smaller than the criteria, even when performing automatic berthing. A commonly used method for estimating states in real time is the Kalman filter  [9, 10]. The Kalman filter is a kind of sequential Bayesian filter proposed by Kalman. It has proven to be optimal in the sense that it minimizes the sum of squares of the estimation error within the assumptions of noise normality and linearity of the state equation [11]. The Kalman filter has been applied for state estimation in various engineering fields, and many applications of nonlinear Kalman filters can be applied to even nonlinear systems for vehicles [12, 13], space crafts  [14,15,16], and robots [17]. It is worth noting that the extended Kalman filter (EKF), one of the nonlinear Kalman filter algorithms, was implemented on the Apollo spacecraft for a manned flight to the moon and used to accurately calculate the spacecraft’s position [18].

In naval architecture and ocean engineering fields, there have been many studies on state estimation using nonlinear Kalman filters. For instance, Balchen used the EKF for state estimation using the dynamic positioning system (DPS) [19]. Sorensen also used Kalman filter for DPS [20]. Abkowitz optimized the maneuvering model using Kalman filter [21]. Fossen et al. used EKF to estimate the maneuvering motion model [22]. Ochiai applied EKF to estimate the attitude of the ship, future position, and future velocity using the data of global positioning system (GPS) and automatic radar plotting aids [23]. Miyoshi used EKF to estimate ship position from GPS data for tracking control. They also used EKF to estimate sway velocity, which is not actually observed and verified its performance in a real ship test [24]. S.Fossen and T.I. Fossen estimated the ship’s position and speed from AIS data using EKF [25]. T.I. Fossen also calculated course over ground and speed over ground from Global Navigation Satellite System data using EKF [26]. However, many studies using nonlinear filters other than EKF have been conducted. For instance, Subchan et al. estimated the hydrodynamic forces and moments from the experimental results of free-running tests using an unscented Kalman filter (UKF). The estimated hydrodynamic forces and moments were then used to estimate the hydrodynamic derivatives included in the maneuvering equation of motion  [27]. Deng et al. designed a model predictive controller for the DPS system that simultaneously estimates states and determines input control using a UKF-based observer [28].

Several nonlinear Kalman filters, including EKF, have been proposed, and the user must select a suitable filter among them according to the user’s purpose. In addition, there have been many studies comparing the performance among filters. For instance, Konatowski compared the estimation accuracy of EKF and UKF for constant velocity and complex motion systems. The results showed that UKF exhibited better estimation accuracy [29]. In addition, Konatowski conducted a comparison of the estimation accuracy of the EKF, UKF, and particle filter (PF) using a simulation of an object moving in a two-dimensional Cartesian coordinate system. Consequently, he showed that PF exhibited the best estimation accuracy [30]. In addition, Chen et al. compared the estimation accuracy when using the EKF, UKF, PF, and linear minimum mean square error filter as filters applied to the tracking filter [31]. The tracking filter is an algorithm that estimates the accurate value of the target’s position and velocity based on the observed values of the target object, including the observation errors obtained from the sensors. Thus, many studies have compared the performance of nonlinear filters. Several studies on the implementation of Kalman filters have also been conducted. For instance, Nishimura presented the algorithms for evaluating the effect of errors due to modeling errors in the Linear Kalman filter [32]. However, there has not been any attempt to discuss guidelines for which filter to use in each case. This research assumed the implementation of a nonlinear Kalman filter with a complex motion model applied to an actual ship system. It differs from other papers in that it examines possible problems in implementing such a filter and provides guidelines for filter selection.

The contributions of this study are summarized as follows.

  • This study provides guidelines to implement UKF and EnKF to guide users in state estimation.

  • The following three aspects are examined for their impact on filter estimation accuracy for the implementation.

    1. 1.

      Effect of modeling error

    2. 2.

      Effect of observation noise

    3. 3.

      Effect of type of maneuvering motion

  • Verifying the real-time performance of each filtering method

2 Notation

The notation used in this paper is explained as follows. First, \(\mathbb {R}^n\) denotes a Euclidean space of dimension n. The hat (\(\hat{\ }\)) denotes the estimated value, and the superscript minus (\(^-\)) denotes the prior estimate. The superscript minus denotes prior estimates. A diagonal matrix \(\varvec{A} \in \mathbb {R}^{n \times n}\) is denoted as \(\varvec{A}=\textrm{diag}(a_{11},a_{22},\ldots ,a_{nn})\). Here, \(a_{ij}\) represents the elements of the i rows and j columns of the matrix. In addition, the superscript \(\textrm{T}\) denotes the transpose of the matrix. Vectors are column vectors unless otherwise stated.

3 Overview of the Kalman filter

This section outlines the nonlinear Kalman filter used in this study. First, we consider the following discrete nonlinear system:

$$\begin{aligned} \left\{ \begin{aligned} \varvec{x}(k+1)&=\varvec{f}(\varvec{x}(k), \varvec{u}(k)) + \varvec{v}(k) \\ \varvec{y}(k)&=\varvec{H}\varvec{x}(k) + \varvec{w}(k). \end{aligned}\right. \end{aligned}$$
(1)

\(\varvec{f}(\varvec{x}(k), \varvec{u}(k)):\mathbb {R}^n \times \mathbb {R}^l \rightarrow \mathbb {R}^n\) is the nonlinear system model. \(\varvec{H} \in \mathbb {R}^{m \times n}\)is output matrix. Here, k denotes the number of steps at a given time \(t_k\), denoted as \(t_k=\Delta tk, (k=0,1,2,\ldots )\); \(\Delta t\) denotes the time interval of one time step; \(\varvec{x} \in \mathbb {R}^n\) is the state vector; \(\varvec{u} \in \mathbb {R}^l\) is the input vector; \(\varvec{y} \in \mathbb {R}^m\) is the observation vector; \(\varvec{v} \in \mathbb {R}^n\) is the system noise according to \(\varvec{v}\sim \mathcal {N}\left( \varvec{0}, \textbf{Q}(k)\right)\); \(\varvec{w} \in \mathbb {R}^m\) is the additive observation noise according to \(\varvec{w}\sim \mathcal {N}\left( \varvec{0}, \textbf{R}(k)\right)\). The covariance matrix \(\textbf{Q}, \textbf{R}\) are defined as follows:

$$\begin{aligned} \begin{aligned} E\left[ \textbf{v}(i) \textbf{v}^{T}(j)\right]&=\delta _{i j} \textbf{Q}(I) \\ E\left[ \textbf{w}(i) \textbf{w}^{T}(j)\right]&=\delta _{i j} \textbf{R}(I), \quad \forall i, j \\ E\left[ \textbf{v}(i) \textbf{w}^{T}(j)\right]&=\textbf{0}. \end{aligned} \end{aligned}$$
(2)

Here, \(\delta _{i j}\) is the Kronecker delta.

3.1 UKF [33,34,35]

\(\texttt {UKF}\) is a nonlinear Kalman filter that uses a nonlinear transformation called the “U transform.” This transformation method performs a nonlinear transformation of sample points called sigma points and calculates the expected value and variance from the sample mean of the transformed sigma points. Using this transformation, \(\texttt {UKF}\) has the advantage that, unlike \(\texttt {EKF}\), it does not require local linearization using the Jacobian matrix [33]. The following is the computational procedure of \(\texttt {UKF}\).

First, \(2n+1\) sigma points \(\varvec{\chi }_i \in \mathbb {R}^{n}\) are determined using the state estimate \(\hat{\varvec{x}}(k-1)\) at \(k-1\) and the error covariance matrix \(\varvec{P}(k-1)\in \mathbb {R}^{n \times n}\):

$$\begin{aligned} \left\{ \begin{aligned} \varvec{\chi }_{0}(k-1)&=\hat{\varvec{x}}(k-1) \\ \varvec{\chi }_{i}(k-1)&=\hat{\varvec{x}}(k-1)+(\sqrt{(n+\kappa ) \varvec{P}(k-1)})_{i}, \\for\ i&=1,2,\ldots ,n \\ \varvec{\chi }_{i+n}(k-1)&=\hat{\varvec{x}}(k-1)-(\sqrt{(n+\kappa ) \varvec{P}(k-1})_{i}., \\for\ i&=1,2,\ldots ,n \\ \end{aligned} \right. \end{aligned}$$
(3)

\((\sqrt{\varvec{P}(k-1)})_{i}\in \mathbb {R}^{n}\) denotes the i-th column of the square root matrix of the error covariance matrix \(\varvec{P}(k-1)\). Here, the square root matrix refers to the matrix \(\varvec{B} \in \mathbb {R}^{n \times n}\) that satisfies the relation \(\varvec{A}=\varvec{B}\varvec{B}\) for some positive definite symmetric matrix \(\varvec{A}\). This \(\varvec{B}\) is written as \(\varvec{B}=\sqrt{\varvec{A}}\). For each sigma point, the weight \(W_i\in \mathbb {R} (i=0,1,2,....,2n)\) are determined as follows:

$$\begin{aligned} \left\{ \begin{aligned} W_{0}&=\frac{\kappa }{n+\kappa } \\ W_{i}&=\frac{1}{2(n+\kappa )}. \end{aligned} \right. \end{aligned}$$
(4)

Here, \(\kappa \in \mathbb {R}\) is the design parameter for approximating the statistical moment \(\mu _{r}\) of second or higher order. The statistical moment \(\mu _{r}\) is defined in Eq.(5) and represents the r-order moment around the mean \(\mu\):

$$\begin{aligned} \mu _{r}=E\left[ (X_p-\mu )^{r}\right] . \end{aligned}$$
(5)

where \(X_p\) is the random variable for the states. Next, the sigma points are updated using the system model \(\varvec{f}(\varvec{x}(k), \varvec{u}(k))\):

$$\begin{aligned} \begin{aligned} \varvec{\chi }^-_{i}(k)=&\varvec{f}\left( \varvec{\chi }_{i}(k-1), \varvec{u}(k)\right) . \\ \end{aligned} \end{aligned}$$
(6)

By this operation, the prior state estimate \(\hat{\varvec{x}}^-(k)\) and prior error covariance matrix \(\varvec{P}^-(k)\) are obtained from the updated sigma points as follows:

$$\begin{aligned} \left\{ \begin{aligned} \hat{\varvec{x}}^-(k)=&\sum _{i=0}^{2 n} W_{i} \varvec{\chi }^-_{i}(k) \\ \varvec{P}^-(k)=&\sum _{i=0}^{2 n} W_{i}\left\{ \varvec{\chi }^-_{i}(k)-\hat{\varvec{x}}^-(k)\right\} \left\{ \varvec{\chi }^-_{i}(k) -\hat{\varvec{x}}^-(k)\right\} ^{\textrm{T}} \\ {}&+ \textbf{Q}(k). \end{aligned} \right. \end{aligned}$$
(7)

However, the sigma points are updated using the prior error covariance matrix \(\varvec{P}^-\) as follows:

$$\begin{aligned} \left\{ \begin{aligned} \varvec{\chi }^-_{0}(k)&=\hat{\varvec{x}}^-(k) \\ \varvec{\chi }^-_{i}(k)&=\hat{\varvec{x}}^-(k)+\sqrt{(n+\kappa ) \varvec{P}^-(k)}_{i} \\ \varvec{\chi }^-_{i+n}(k)&=\hat{\varvec{x}}^-(k)-\sqrt{(n+\kappa ) \varvec{P}^-(k)}_{i}. \\ \end{aligned} \right. \end{aligned}$$
(8)

Update the output sigma point \(\varvec{\Upsilon }^-_i(k) \in \mathbb {R}^{m}\) from the updated sigma point:

$$\begin{aligned} \varvec{\Upsilon }^-_i(k) = \varvec{H}\varvec{\chi }^-_{i}(k). \end{aligned}$$
(9)

From these output sigma points, a prior output estimate \(\hat{\varvec{y}}^-(k)\) is obtained:

$$\begin{aligned} \hat{\varvec{y}}^-(k)= \sum _{i=0}^{2 n} W_{i} \varvec{\Upsilon }^-_{i}(k). \end{aligned}$$
(10)

The prior output error covariance matrix \(\varvec{P}^-_{y y}(k)\in \mathbb {R}^{m \times m}\) is obtained from the output sigma points and prior output estimates \(\hat{\varvec{y}}^-\):

$$\begin{aligned} \begin{aligned} \varvec{P}^-_{y y}(k)=&\sum _{i=0}^{2 n} W_{i}\left\{ \varvec{\Upsilon }^-_{i}(k)-\hat{\varvec{y}}^-(k)\right\} \left\{ \varvec{\Upsilon }^-_{i}(k)-\hat{\varvec{y}}^-(k)\right\} ^{\textrm{T}} \\&+ \textbf{R}(k). \end{aligned} \end{aligned}$$
(11)

The prior state output error covariance matrix \(\varvec{P}^-_{x y}(k)\in \mathbb {R}^{m \times n}\) is obtained as follows:

$$\begin{aligned} \varvec{P}_{x y}^-(k)=\sum _{i=0}^{2 n} W_{i}\left\{ \varvec{\chi }_{i}(k)-\hat{\varvec{x}}^-(k)\right\} \left\{ \varvec{\Upsilon }^-_{i}(k)-\hat{\varvec{y}}^-(k)\right\} ^{\textrm{T}}. \end{aligned}$$
(12)

The Kalman gain \(\varvec{G}(k)\in \mathbb {R}^{n \times m}\) is obtained from \(\varvec{P}_{y y}^-(k)\) and \(\varvec{P}^-_{x y}(k)\):

$$\begin{aligned} \varvec{G}(k)= \frac{\varvec{P}^-_{x y}(k)}{\varvec{P}^-_{y y}(k)}. \end{aligned}$$
(13)

The state estimate \(\hat{\varvec{x}}(k)\) is obtained using the above Kalman gain \(\varvec{G}(k)\), prior state estimate \(\hat{\varvec{x}}^-(k)\), prior output estimate \(\hat{\varvec{y}}^-(k)\), and observables \(\varvec{y}(k)\), as shown in Eq.(14). The posterior error covariance matrix \(\varvec{P}(k)\in \mathbb {R}^{n \times n}\) is also calculated using the prior estimation error covariance matrix \(\varvec{P}^-(k)\), Kalman gain \(\varvec{G}(k)\), prior state output error covariance matrix \(\varvec{P}_{x y}^-(k)\), and measurement values \(\varvec{y}(k)\), as shown in Eq.(14):

$$\begin{aligned} \left\{ \begin{aligned} \hat{\varvec{x}}(k)&= \hat{\varvec{x}}^-(k) + \varvec{G}(k) (\varvec{y}(k)- \hat{\varvec{y}}^-(k))\\ \varvec{P}(k)&= \varvec{P}^-(k) - \varvec{G}(k)(\varvec{P}_{x y}^-(k)). \end{aligned} \right. \end{aligned}$$
(14)

3.2 EnKF [36,37,38]

One of the nonlinear Kalman filters, EnKF, is a filter that uses the probability of the system state distribution held by a set called the ensemble members \(\left\{ \varvec{x}^{(i)}(k)\right\} _{i=1}^{N}\). The method then updates the ensemble members using the update rule of the Kalman filter each time an observation value is obtained. This method, like UKF, does not require the computation of a Jacobian matrix and has the advantage that it can be applied to non-differentiable state-space models. The following summarizes the computation steps of EnKF. First, N sets of ensemble members \(\varvec{x}^{(i)}(k-1) \in \mathbb {R}^n\ (i=1,2,\ldots ,N)\) are updated based on the system model. Moreover, \(\varvec{v}^{(i)}(k-1) \in \mathbb {R}^n(i=1,2,\ldots ,N)\) represents the ensemble member of the system noise:

$$\begin{aligned} \left\{ \begin{aligned} \varvec{x}^{(i)}(k-1)=\varvec{f}(\varvec{x}^{(i)}(k-1), \varvec{u}(k-1), \varvec{v}^{(i)}(k-1)),\\ \quad \varvec{v}^{(i)}(k-1) \sim \mathcal {N}\left( 0, \textbf{Q}(k-1)\right) . \end{aligned} \right. \end{aligned}$$
(15)

Here, the system model \(\varvec{f}\) is treated as Eq.(1) for convenience because it gives disturbance when updating ensemble members in EnKF:

$$\begin{aligned} \varvec{f}(\varvec{x}(k), \varvec{u}(k), \varvec{v}(k))=\varvec{f}(\varvec{x}(k), \varvec{u}(k)) + \varvec{v}(k). \end{aligned}$$
(16)

Next, the sample variance-covariance matrix \(\hat{\varvec{V}}(k-1)\) of the state variable vector \(\varvec{x}^{(i)-}(k-1)\) and the sample variance-covariance matrix \(\hat{\textbf{R}}(k-1)\) of the observed noise are calculated from the updated ensemble members:

$$\begin{aligned} \left\{ \begin{gathered} \varvec{x}^{(i)-}(k-1)=\varvec{x}^{(i)}(k-1)-\frac{1}{N} \sum _{j=1}^{N} \varvec{x}^{(j)}(k-1) \\ \hat{\varvec{V}}(k-1)=\frac{1}{N-1} \sum _{j=1}^{N} \varvec{x}^{(j)-}(k-1)\varvec{x}^{(j)-}(k-1)^{T} \\ \hat{\textbf{R}}(k-1)=\frac{1}{N-1} \sum _{j=1}^{N} \varvec{w}^{(j)}(k-1) \varvec{w}^{(j)}(k-1)^{T}. \end{gathered} \right. \end{aligned}$$
(17)

Next, the Kalman gain \(\varvec{G}(k)\) is obtained from the sample variance-covariance matrix \(\hat{\varvec{V}}(k-1)\), the sample variance-covariance matrix of the observed noise \(\hat{\textbf{R}}(k-1)\), and the matrix representing the observed state \(\varvec{H}\):

$$\begin{aligned} \begin{aligned} \varvec{G}(k)=\hat{\varvec{V}}(k-1) \varvec{H}^{\textbf{T}}(\varvec{H} \hat{\varvec{V}}(k-1) \varvec{H}^{\textrm{T}}+\hat{\textbf{R}}(k-1))^{-1}. \end{aligned} \end{aligned}$$
(18)

Finally, the ensemble members are updated through the Kalman filter update rule:

$$\begin{aligned} \begin{aligned} \varvec{x}^{(i)}(k)=\varvec{x}^{(i)}(k-1)&+\varvec{G}(k)(\varvec{y}(k)\\ {}&+\varvec{w}^{(i)}(k)-\varvec{H} \varvec{x}^{(i)}(k-1)). \end{aligned} \end{aligned}$$
(19)

4 System model

In this study, we used the maneuvering modeling group (MMG) model as the system model, representing the surge-sway-yaw maneuvering motion [39]. The model used in this study is for a vessel with a single propeller and twin rudders. Space-fixed coordinate systems \(\textrm{O}-{XY}\) and ship fixed coordinate systems are used \(\textrm{o}-{xy}\), as shown in Fig. 1. The origin of ship fixed coordinate systems \(\textrm{o}\) is on midship. Here, \(\textrm{O}-{XY}\) represents a right-handed coordinate system, and Z-axis is the positive downward; \(\textrm{o}-{xy}\) is a right-handed coordinate system, and z-axis is the positive downward. U denotes the ship’s speed, and \(u,\ v_m\) denotes the speed in the surge and sway directions, respectively; \(\psi\) and r denote the angle and angular velocity in the yaw direction, respectively; \(\beta\) denotes the drift angle. \(U_T,\gamma _T\) denotes the wind speed and direction in the true direction, and \(U_A\), \(\gamma _{\textrm{A}}\) denotes the relative wind speed and direction.

Fig. 1
figure 1

Coordinate systems

Full size image

The MMG model used is shown in Eq.(20). In the following, the dots represent differentiation with respect to time t:

$$\begin{aligned} \begin{aligned} \left( m+m_{x}\right) \dot{u}-(m+m_{y})v_\textrm{m}r+m x_{G} r^{2}&=X \\ \left( m+m_{y}\right) \dot{v}_\textrm{m}+m x_{G} \dot{r}+\left( m+m_{x}\right) u r&=Y \\ \left( I_{z z}+m x_{\textrm{G}}^{2}+J_{z z}\right) \dot{r}+m x_{G}(\dot{v}_\textrm{m}+u r)&=N. \end{aligned} \end{aligned}$$
(20)

Here, \(x_G\) denotes the longitudinal coordinate of the center of the ship’s gravity; m denotes the ship’s mass and \(m_x,\ m_y\) denotes the added mass of the x-axis direction and y-axis direction; \(I_{zz}, J_{zz}\) denote the moment of inertia and added moment of inertia in the yaw direction, respectively; \(X,\ Y\) denote the hydrodynamic forces at the midship in x and y axis, respectively; N is the force moment around the z axis. In addition, the elements of the hydrodynamic forces to be considered are represented as follows:

$$\begin{aligned} \begin{aligned} X&=X_\textrm{H}+X_\textrm{P}+X_\textrm{R}\\ Y&=Y_\textrm{H}+Y_\textrm{P}+Y_\textrm{R}\\ N&=N_\textrm{H}+N_\textrm{P}+N_\textrm{R}. \end{aligned} \end{aligned}$$
(21)

Here, the subscripts \(\textrm{H}\), \(\textrm{R}\), and \(\textrm{P}\) denote the hull, rudder, and propeller, respectively. For the hydrodynamic forces and moment acting on the hull, we employ the Yoshimura’s low speed maneuvering model shown in Eq.(22) [40]:

$$\begin{aligned} \begin{aligned} X_{H}=&\left( \frac{\rho }{2}\right) L d\\ {}&\left[ \begin{array}{l} \left\{ X_{0(F)}^{\prime }+\left( X_{0(A)}^{\prime }-X_{0(F)}^{\prime }\right) (\beta / \pi )\right\} u U \\ +X_{v r}^{\prime } L \cdot v_m r \end{array}\right] \\ Y_{H}=&\left( \frac{\rho }{2}\right) L d\\ {}&\left[ \begin{array}{l} Y_{v}^{\prime } v_m|u|+Y_{r}^{\prime }L \cdot r u \\ \left. -\left( \frac{C_{D}}{L}\right) \displaystyle \int _{-L / 2}^{L / 2}\left| v_m+C_{r Y} r x\right| \left( v_m+C_{r Y} r x\right) dx\right. ] \end{array}\right] \\ N_{H}=&\left( \frac{\rho }{2}\right) L^{2} d\\ {}&\left[ \begin{array}{l} N_{v}^{\prime } v_m u+N_{r}^{\prime } L \cdot r|u| \\ -\left( \frac{C_{D}}{L^{2}}\right) \displaystyle \int _{-L / 2}^{L / 2}\left| v_m+C_{r Y} r x\right| \left( v_m+C_{r Y} r x\right) x dx \end{array}\right] . \end{aligned} \end{aligned}$$
(22)

Here, \(\rho\) is the water density; L is the length between perpendicular of the ship; d is the draft; \(X_{0(\textrm{A})}^{\prime },\ X_{0(\textrm{F})}^{\prime }\) are the resistance coefficient in the forward and backward directions, respectively; \(C_D\) is the drag coefficient; \(C_{r Y},\ C_{r N}\) are the modification coefficients for the sway direction force and yaw moments, respectively; \(X^{\prime }_{v r},\ Y^{\prime }_{v},\ Y^{\prime }_{r},\ N^{\prime }_{v},\ N^{\prime }_{r},\) are non-dimensionalized hydrodynamic derivatives. The states \(\varvec{x}\), control input vector \(\varvec{u}\), and measurement vector \(\varvec{y}\) are as follows. The non-dimensionalized values are represented by superscript primes, as in \(X_{0(\textrm{A})}^{\prime }\). The added mass term included in the original Yoshimura’s low speed model is transferred to the left side of Eq.(20):

$$\begin{aligned} \begin{aligned} \varvec{x}&\equiv (x, u, y, v, \psi , r)^{\textrm{T}} \in \mathbb {R}^{6}\\ \varvec{u}&\equiv (n, \delta _s, \delta _p,)^{\textrm{T}} \in \mathbb {R}^{3}\\ \varvec{y}&\equiv (x, u, y, v, \psi , r)^{\textrm{T}} \in \mathbb {R}^{6}. \end{aligned} \end{aligned}$$
(23)

Here, n is the number of propeller revolutions; \(\delta _s\) and \(\delta _p\) are the starboard and port rudder angles, respectively.

The MMG model estimate the time derivative \(\dot{\varvec{x}}\) by solving \(\varvec{g}(\varvec{x},\varvec{u}):\mathbb {R}^6 \times \mathbb {R}^3 \rightarrow \mathbb {R}^6\) expressed in Eq.(20):

$$\begin{aligned} \dot{\varvec{x}}=\varvec{g}(\varvec{x},\varvec{u}). \end{aligned}$$
(24)

To use the Kalman filter in discrete time, we used the following discretized state transitions. Let \(\varvec{x}(k)\) denote the state at k steps and \(\varvec{u}(k)\) denote the control input at k steps:

$$\begin{aligned} \varvec{x}(k) +\varvec{g}(\varvec{x}(k),\varvec{u}(k))\Delta t =\varvec{f}(\varvec{x}(k),\varvec{u}(k)). \end{aligned}$$
(25)

Here, \(\varvec{f}(\varvec{x}(k),\varvec{u}(k))\) in Eq.(25) is applied to the system model in Eq.(1).

In this study, \(\varvec{H}\) is a unitary matrix (\(\varvec{I}\)) representing the observed model since the elements of the observed and estimated states coincide, as follows:

$$\begin{aligned} \varvec{H} = \varvec{I} \in \mathbb {R}^{6 \times 6}. \end{aligned}$$
(26)

5 Investigation of factors affecting the accuracy of filtering

As mentioned in the introduction, the choice of the Kalman filter determines the accuracy of the estimation of state quantities. In this section, these factors are outlined and their importance is discussed. In this study, we examine the following three factors.

  1. 1.

    Modeling error on estimation accuracy

  2. 2.

    Observation noise

  3. 3.

    Type of maneuvering motion

For some parameters of the Kalman filter, first, the initial value of \(\varvec{x}(0)\) was taken as the initial value of the observed value \(\varvec{y}(0)\). Thus, it was defined as in Eq. (27):

$$\begin{aligned} \varvec{x}(0)=\varvec{y}(0). \end{aligned}$$
(27)

The covariance matrix \(\textbf{Q}\) of the system noise, the covariance matrix \(\textbf{R}\) of the observed noise, and the initial value of the error covariance matrix \(\textbf{P}(0)\) are identical to align conditions other than items for verification. The parameter \(\kappa\) of \(\texttt {UKF}\) is also used with \(\kappa =3\). \(\textbf{Q}\in \mathbb {R}^{6 \times 6}, \textbf{R}\in \mathbb {R}^{6 \times 6}, \textbf{P}(0)\in \mathbb {R}^{6 \times 6}\) is a diagonal matrix. The covariance matrix of the system noise \(\textbf{Q}\) is as follows:

$$\begin{aligned} \textbf{Q}=\varvec{0}. \end{aligned}$$
(28)

The covariance matrix of the observed noise \(\textbf{R}\) can be obtained from the system’s performance specification of the onboard sensor of the system. Since this study was based on simulation, it was determined by referencing the measured data of onboard sensors. The value of \(\textbf{P}(0)\) is excessively small, the Kalman gain may become too small [41], and to avoid this, we adopt a unit matrix for \(\textbf{P}(0)\).

$$\begin{aligned} \textbf{R}&={{\textrm{diag}}}(5.0 \times 10^{-2},\ldots ,5.0 \times 10^{-2}). \end{aligned}$$
(29)
$$\begin{aligned} \textbf{P}(0)&= \varvec{I} \in \mathbb {R}^{6 \times 6}. \end{aligned}$$
(30)

The number of ensemble members was set to \(N=100\) as the parameter for EnKF. Furthermore, to compare the filtering performance, we used the linear Kalman filter (KF) with the persistence prediction model to perform the filtering time. The detail of the algorithms can be found in the literature [9, 10]. In this study, KF was applied to the linear state-space model, as shown in Eq.(31):

$$\begin{aligned} \begin{aligned} \varvec{x}(k+1)&= \varvec{A}\varvec{x}(k) + \varvec{v}(k)\\ \varvec{y}(k)&= \varvec{C}\varvec{x}(k) + \varvec{w}(k). \end{aligned} \end{aligned}$$
(31)

Here, \(\varvec{A}\) is \(\varvec{A}=\varvec{I} \in \mathbb {R}^{6 \times 6}\) and \(\varvec{C}\) is \(\varvec{C}=\varvec{I} \in \mathbb {R}^{6 \times 6}\). The parameter of KF was tuned separately by trial and error for better filtering performance.

5.1 Effect of modeling error on estimation accuracy

In this subsection, we investigate the effect of modeling errors on filtering using the Kalman filter. It is impossible to construct a motion model that estimates motion completely without error. Hence, modeling errors always exist. In this study, we did not consider wind in the system model but as a modeling error.

First, two models are prepared for comparison. The first is the MMG model using Eq. (21). The second is the MMG model mentioned above with the effect of wind, as shown below. The system model used for estimation does not incorporate a wind model. Instead, the impact of wind is treated as a modeling error, and the wind is only taken into account in the observations for filtering. The equation of state used in this study is as follows:

$$\begin{aligned} \begin{aligned} X&=X_\textrm{H}+X_\textrm{P}+X_\textrm{R}+\underline{X_\textrm{A}}\\ Y&=Y_\textrm{H}+Y_\textrm{P}+Y_\textrm{R}+\underline{Y_\textrm{A}}\\ N&=N_\textrm{H}+N_\textrm{P}+N_\textrm{R}+\underline{N_\textrm{A}}. \end{aligned} \end{aligned}$$
(32)

Here, the subscript \(\textrm{A}\) represents the effect of wind. The force due to wind was obtained using the Fujiwara’s equation [42] shown in Eq.(34):

$$\begin{aligned} \left\{ \begin{aligned}&X_{\textrm{A}}=(1 / 2) \rho _{\textrm{A}} U_{\textrm{A}}^{2} A_{\textrm{T}} \cdot C_{X}\\ {}&Y_{\textrm{A}}=(1 / 2) \rho _{\textrm{A}} U_{\textrm{A}}^{2} A_{\textrm{L}} \cdot C_{Y}\\&N_{\textrm{A}}=(1 / 2) \rho _{\textrm{A}} U_{\textrm{A}}^{2} A_{\textrm{L}} L_{\textrm{OA}} \cdot C_{N}, \end{aligned} \right. \end{aligned}$$
(33)

\(\text {where}\)

$$\begin{aligned} \left\{ \begin{aligned}&C_{X}=X_{0}+X_{1} \cos \left( 2 \pi -\gamma _{\textrm{A}}\right) +X_{3} \cos 3\left( 2 \pi -\gamma _{\textrm{A}}\right) \\&+X_{5} \cos 5\left( 2 \pi -\gamma _{\textrm{A}}\right) \\&C_{Y}=Y_{1} \sin \left( 2 \pi -\gamma _{\textrm{A}}\right) +Y_{3} \sin 3\left( 2 \pi -\gamma _{\textrm{A}}\right) \\&+Y_{5}\sin 5\left( 2 \pi -\gamma _{\textrm{A}}\right) \\&C_{N}=N_{1} \sin \left( 2 \pi -\gamma _{\textrm{A}}\right) +N_{2} \sin 2\left( 2 \pi -\gamma _{\textrm{A}}\right) \\&+N_{3} \sin 3\left( 2 \pi -\gamma _{\textrm{A}}\right) . \end{aligned} \right. \end{aligned}$$
(34)

Here, \(\rho _{\textrm{A}}\) is the density of the air; \(A_T,\ A_L\) are the transverse projected and lateral projected areas, respectively; \(L_{\textrm{OA}}\) is the overall length of the ship; \(X_{i},\ Y_{i},\ N_{i}\) are the wind pressure coefficients derived using the regression equation [42]. This equation is based on wind tunnel experimental data of many scaled models with the geometric parameters of the ships as explanatory variables. The model that takes wind into account is referred to as the w/ ‘wind model. The model that does not take wind into account is referred to as the without-wind model. An investigation of the effect of modeling errors is conducted as follows:

  • First, time-series data are generated using the w/ wind model for three different wind speeds. White noise, according to \(\varvec{w} \sim \mathcal {N}(\varvec{0},\textbf{R})\), is applied to the time-series data as the observation noise. These generated data were used for filtering. \(\Delta t\) of the data is \(0.1 \mathrm {[s]}\).

  • In order to take the effect of wind as the modeling error, we apply the without-wind model to the system models of UKF and EnKF. The data are filtered for the three different wind speeds generated above.

  • We use UKF and EnKF to calculate the root mean square error (RMSE) between the result of each filtered state and the true value. The definition of RMSE is given in Eq. (35):

    $$\begin{aligned} \textrm{RMSE}=\sqrt{\frac{1}{N} \sum _{i=1}^{N}\left( y_{i}-x_{i}\right) ^{2}}. \end{aligned}$$
    (35)

    Here, N is the number of data; \(y_{i}\) is the filtered value; \(x_{i}\) is the true value. The true values are the time-series data before white noise is added.

Wind direction and speed were observed from the data on June 10, 2021, at Inukai Pond on the Osaka University campus. Here, the wind is assumed to be a variable in time but constant in space. Figure 2 shows the time series of wind direction and speed. Table 1 presents the average wind speed \(\bar{U}_T\) and the average wind direction \(\bar{\gamma }_T\).

Table 1 Average wind speed and direction
Full size table
Fig. 2
figure 2

Example of time series of wind velocity and direction

Full size image

For the time-series data of wind direction and speed applied to the simulation, only the wind speed was multiplied by 1, 0.5, and 0 to produce three different data with average wind speeds of \(\bar{U}_T=1.64\ \mathrm {[m/s]}\), \(\bar{U}_T=0.82\ \mathrm {[m/s]}\), and \(\bar{U}_T=0.0\ \mathrm {[m/s]}\), respectively. When the wind speed is \(\bar{U}_T=0.0\ \mathrm {[m/s]}\), the with-wind and without models completely coincide, and the modeling error increases as the average wind speed increases. First, the modeling errors of the with-wind and without models were compared by the trajectory of the turning test. The results shown in Fig. 3 are consistent with the with-wind and without models when the average wind speed is \(\bar{U}_T=1.64\ \mathrm {[m/s]}\).

Fig. 3
figure 3

Comparison of the w/ wind model and w/o wind model trajectory and time series

Full size image

Figure 3 shows that the with-wind model trajectory drifts more downwind than the without-wind model results.

Table 2 System model and model to be estimated (different wind speeds are applied)
Full size table

Next, we present the filtering results. Filtering is performed using UKF, EnKF, and KF under each condition shown in Table 2. Figure 4 shows the comparison of RMSE in each state.

Fig. 4
figure 4

Comparison of RMSE in wind condition (upper tile: UKF result, middle tile: EnKF result, and lower tile: KF result)

Full size image

As shown in Fig. 4, the estimation accuracy became worse as the modeling error increased for UKF and EnKF. In particular, the estimation accuracy rapidly worsened for EnKF due to modeling error. These results suggest that EnKF is relatively more easily affected by the modeling error of the system model than UKF. The covariance matrix of the system noise was set to \(\textbf{Q}=\varvec{0}\) in this validation. Therefore, the effect of setting of \(\textbf{Q}\) significantly affected the estimation accuracy, especially in EnKF that applied the ensemble member of the system noise to the state as a disturbance. Next, we show the filtering results for UKF and EnKF when the system noise covariance matrix \(\textbf{Q}\) is set to \(\textbf{Q}\ne \varvec{0}\). The covariance matrix \(\textbf{Q}_\textrm{UKF} \in \mathbb {R}^{6 \times 6}\) of the system noise in UKF was obtained by trial and error for better filtering performance as follows:

$$\begin{aligned} \textbf{Q}_\textrm{UKF}={{\textrm{diag}}}(1.0 \times 10^{-4},\ldots ,1.0 \times 10^{-4}). \end{aligned}$$
(36)

The covariance matrix \(\textbf{Q}_\textrm{EnKF} \in \mathbb {R}^{6 \times 6}\) of the system noise in EnKF is as follows:

$$\begin{aligned} \textbf{Q}_\textrm{EnKF}={{\textrm{diag}}}(1.0 \times 10^{-2},\ldots ,1.0 \times 10^{-2}). \end{aligned}$$
(37)

Here, the filtering was performed using UKF and EnKF under the conditions shown in Table 3. The filtering results for each obtained state and RMSE were compared for each filter. Figure 5 shows the obtained RMSE for each state. In addition, the estimated time-series data were those at \(\bar{U}_T=1.64 \mathrm {[m/s]}\) with the largest modeling error.

Table 3 System model and model to be estimated (different \(\textbf{Q}\) are applied to system model)
Full size table
Fig. 5
figure 5

Comparison of RMSE in wind condition ( \(\textbf{Q}=0\) and \(\textbf{Q}\ne 0\), upper tile: UKF result, lower tile: EnKF result)

Full size image

Figure 5 shows that the estimation improved when the system noise was considered. In particular, some states in EnKF significantly improve estimation accuracy. In other words, tuning the ensemble members of the system noise can improve the estimation accuracy. For UKF, tuning the system noise covariance matrix also improved the estimation accuracy. However, the difference was not as remarkable as in the case of EnKF. Therefore, when using UKF for state estimation, it is more important than for EnKF to reduce the modeling error of the system model as much as possible.

5.2 Effect of observation noise on estimation accuracy

Observed values by sensors always include observation noise. When using the Kalman filter, it is necessary to input parameters, such as the covariance matrix of the noise for the observed noise \(\varvec{w}\) and the system noise \(\varvec{v}\). In addition, several studies have been conducted on how to determine these parameters when implementing the Kalman filter  [41]. However, they are often determined by trial and error. In this section, we investigate the effect of the magnitude of the observation noise on the filtering accuracy. To achieve this, we implemented the following method.

  • The time-series data are generated using the same procedure as in Sect. 5.1. Here, the time-series data were generated using the without-wind model. To verify the effect of the observed noise, we prepared data with three types of noise applied to the covariance matrix of the observed noise: \(\textbf{R}={\text {diag}}(0.05^2,\ldots , 0.05^2)\), \(\textbf{R}={\text {diag}}(0.10^2,\ldots , 0.10^2)\), and \(\textbf{R}={\text {diag}}(0.20^2,\ldots , 0.20^2)\).

  • Filtering is performed on the three types of data using UKF and EnKF. At this time, in order to align the conditions, the covariance matrix of the observed noise used in UKF and the ensemble members of the observed noise used in EnKF are the same regardless of the data type. The system model applied to each filter is the without-wind model.

  • Using UKF and EnKF, we calculate the RMSE between the result of each filtered state and the true value.

Table 4 System model and model to be estimated (different \(\textbf{R}\) are applied)
Full size table

Using the above method, we performed the filtering using UKF, EnKF, and KF under the conditions shown in Table 4. The filtering results for each obtained state and RMSE were compared for each filter. Figure 6 shows the obtained RMSE for each state.

Fig. 6
figure 6

Comparison of RMSE under different observation noise conditions (upper tile: UKF result, middle tile: EnKF result, and lower tile: KF result)

Full size image

As shown in Fig. 6, the RMSE obtained using UKF and EnKF showed that the observation accuracy tends to be worse as the variance of the observation noise increases. However, the observation noise did not affect the estimation accuracy significantly as the modeling error. Therefore, it is essential to consider the effects of observation noise for accurate state estimation but not the extent of modeling errors in the system model.

5.3 Effect of type of maneuvers on estimation accuracy

In this section, we investigate the effect of the type of maneuvers on the estimation accuracy of UKF and EnKF. Thus, we generate the following two types of maneuvering motions as time-series data.

  1. 1.

    Zig-zag motion

  2. 2.

    Random motion

As mentioned in the introduction, Konatowski compared the estimation accuracy of EKF and UKF for constant velocity and complex motion systems [29]. For the type of maneuvering motion, as long as the Zig-zag maneuvers are given input with periodic motion, the final result is a periodic solution as long as it does not diverge. The motion is broadly considered as stationary, periodic, and aperiodic solutions. For simplicity (Fig. 7), no verification for step response will be performed at this time. We examine the Zig–zag and random maneuvers to compare periodic and aperiodic motions. Here, the propeller revolution was set to be constant. The rudder angle was given as Zig-zag maneuvers and random maneuvers. Here, the random maneuvers were created by giving the rudder angle as a uniform random number (\(\delta _s\) is \([-35,105]\) \(\delta _p\) is \([-105,35]\)). Figure 8 shows an example of rudder angle time series and the trajectory.

Fig. 7
figure 7

Time series of rudder angle and trajectory(upper panel: rudder angle of Zig-zag motion, lower panel: trajectory of Zig-zag motion)

Full size image
Fig. 8
figure 8

Time series of rudder angle and trajectory( upper panel: rudder angle of random motion, lower panel: trajectory of random motion)

Full size image

We used the following procedures to examine the effect of the type of maneuvers.

  • The same procedure as in Sect. 5.2 is used to create the time-series data. At this time, the two types of inputs shown in Fig. 8 are used. The system model is the without-wind model described above. The filtering data are created by applying white noise of \(\varvec{w} \sim \mathcal {N}(\varvec{0},\textbf{R})\) to the time-series data.

  • Filtering is performed on the two types of data using UKF and EnKF. In order to align the conditions, the covariance matrix \(\textbf{R}\) of the observed noise used in UKF and the variance of the ensemble members of the noise used in EnKF were the same, regardless of the data type.

  • UKF and EnKF are used to calculate the RMSE between the result of each filtered state and the true value.

Figure 9 compares the obtained filtering results of each state and RMSE for each filter.

Fig. 9
figure 9

Comparison of RMSE in different maneuvering motion conditions (upper tile: UKF result, middle tile: EnKF result, and lower tile: KF result)

Full size image

As shown in Fig. 9, there was no significant difference in the estimation accuracy of each state between different types of maneuvers in the estimation using EnKF and UKF. Thus, it can be considered that the difference in the type of maneuvers has little effect on the estimation accuracy. From the dynamical perspective, the periodic or aperiodic motion did not have a significant difference in estimation accuracy.

5.4 Comparison of computation time

Real-time state estimation is important for real-time control. Therefore, we verify the real-time performance of the filtering algorithm by comparing the computational time of UKF and EnKF. The sample data to be filtered are numerically simulated Zig-zag maneuvers explained in Sect. 5.3 was performed. Ten datasets with different applied noises were prepared, and the average computational time was obtained for each filter. The computational time per step (\(0.1 \mathrm {[s]}\)) is obtained to verify the real-time performance of the filtering algorithm and compare the computational time for each filter. Table 5 presents the computer specification used in this study.

Table 5 Computer specifications
Full size table

The comparison of computational time is shown in Fig. 10.

Fig. 10
figure 10

Comparison of computational time between EnKF, UKF, and KF

Full size image

As shown in Fig. 10, all filters took less than \(0.1 \mathrm {[s]}\) per step to filter. As a result, all filters had real-time performance. The computational time for EnKF depends on the number of ensemble members, which was about four times longer than that of UKF.

5.5 Guideline for filtering

From the results of Sects. 5.1 to 5.4, we obtained the following guideline for filter selection.

  • Modeling error:

    Modeling error had a significant effect on estimation accuracy for UKF and EnKF. In UKF and EnKF, it was possible to improve the estimation accuracy by tuning the covariance matrix and the ensemble members of the system noise given to EnKF. In particular, EnKF showed significant improvement. It can also happen that KF is better than UKF and EnKF if the modeling error is high. However, the accuracies of UKF and EnKF were better than that of the KF in all other cases.

  • Covariance matrix:

    It is essential to choose the appropriate covariance matrix for the observed noise because it affects the accuracy of the UKF and EnKF estimates. However, the importance was not as large as the modeling error. Therefore, even if the covariance matrix of the observed noise is not chosen carefully, it is possible to obtain an acceptable estimation.

  • Periodic or aperiodic motion:

    We could not confirm that filtering performance was affected by periodic or aperiodic motion. Thus, there is no need to select between UKF and EnKF for different maneuverings.

  • Computation time:

    For both filters, the computational time per step was less than \(0.1 \ \mathrm {[s]}\), the observation period. Therefore, there was a real-time performance in this calculation. EnKF took about four times longer to compute than UKF. Thus, it is appropriate to use UKF when real-time performance is required under severe conditions.

Since UKF and EnKF do not have extreme differences in estimation accuracy, it is essential to select the appropriate filter for the computational time and purpose.

5.6 Discussion

To further improve the estimation accuracy in the future, it is important to keep the modeling error of the system model sufficiently small when using UKF and EnKF. Several attempts have been recently made to improve the performance of maneuvering motion models in ship maneuvering fields. For instance, Araki et al. estimated the maneuvering model by utilizing several experimental fluid dynamic systems and computational fluid dynamics free-running trials. They employed the EKF and the constrained least square method with the generalized reduced gradient algorithm [43]. Jian et al. identified the Abkowitz model from the simulation dataset using least square support vector machines [44]. Serge and Guedes developed an identification algorithm for ship maneuvering mathematical models using the classic genetic algorithm [45]. Wakita et al. proposed a system identification method for generating low speed maneuvering models using neural networks [46]. Miyauchi et al. estimated the parameters of the MMG model from the trajectories of free-running tests using an optimization algorithm, the covariance matrix adaptation evolution strategy (CMA-ES). The results show that the model is more consistent with the results of free-running tests than with the parameters obtained from captive model tests [47]. Based on these studies, it is possible that a motion model with improved performance could be applied to the nonlinear Kalman filter to provide a more accurate state estimation. Furthermore, from a robustness perspective, we consider it important to develop algorithms that account for the sudden deterioration of observation noise due to instrument failure. To deal with this problem, for example, the development of sensor fusion approaches would be useful. Currently, there is a lot of research on sensor fusion, and a sensor fusion technique using UKF has been proposed. For example, a study on sensor fusion technology using UKF for aircraft was conducted by Majeed and Kar [48]. They are also developing a sensor fusion algorithm using a nonlinear Kalman filter for ships.

6 Conclusion

In this study, we investigated the performance of UKF and EnKF filters to provide guidance. We examined the effect of the type of maneuvers, observation noise, and modeling error on the estimation accuracy of UKF and EnKF. The results showed that the modeling error of the system model has a significant effect on the estimation accuracy of EnKF and UKF. There was no significant difference in estimation accuracy between the different types of observation noise and the type of maneuvers. Thus, since berthing motion is considered to be similar to random motion, both filters are considered to be applicable during berthing. In addition, the time required for filtering was measured to verify the online capability of the system, and real-time filtering was possible for EnKF and UKF. However, in this study, EnKF took approximately four times longer to compute than UKF. Therefore, we obtained that EnKF and UKF should be used separately depending on the required computational time and purpose.