1 Introduction

Pan–tilt camera systems are frequently utilized in 3D visual tracking. For instance, they are used by autonomous vehicles or robots that have a vision head [2, 5, 20, 27]. Such vision systems have to solve two problems: real-time processing of visual data and inducing the correct tracking motion of the camera. Solving the problem of efficient visual information processing is a significant challenge and constitutes the majority of the research. Data acquired by an RGB-D camera need to be processed in real time [3], and thus the image must be segmented, target features must be identified and extracted, useful information must be structured into control data, and finally, these data must be delivered to the system component that requires them. Those calculations and data handling actions are time-consuming. The majority of the research focuses on their speed up using time-efficient algorithms, usually through parallelization of calculations [24] or faster hardware. The task of servopositioning of the camera supporting neck is considered separately. Control issues and very high control accuracy do not lie in the mainstream of research. The main goal is to keep the object within the image.

Practical experience shows that tracking and recognition are coupled. Image recognition performance depends on proper camera tracking and vice versa. Once we can keep the tracked object within the same sector of the image frame, we may limit the search and analysis space of the whole frame, what speeds up calculations. On the contrary, if we fail to achieve image processing within the system sampling period, the information about the target will be lost for one or more samples, what may cause the servomechanism tracking degradation or even induce losing the tracked object out of sight. Therefore, camera image processing and servo tracking tasks are closely coupled, and the correlation works in both directions. Better camera tracking operation speeds up visual data processing and prevents losing the object from sight. Fast visual data evaluation prevents sampling losses and tracking problems. As the tracked object can be located quite far away from the head and thus the cameras, even a small improvement in tracking accuracy may help.

One may find different control strategies applied in pan–tilt servoing. Classical PID-based feedback control is the primary choice. When the servomotor is overrated and fast, just simple P (proportional) version might be satisfactory, especially when the steady-state error is allowed [44]. However, most frequently full PID version is used [22, 41], sometimes modified by factors similar to those used in process control, such as derivative filtering [44] or lead-lag compensation [40]. Feedback control is improved by feedforward decoupling [27] in situations when disturbance measurement is available. A similar task can be solved alternately by active disturbance rejection control [5]. If accurate control is required, feedback control is replaced by more advanced control strategies [4]. Once system nonlinearities affect control performance, the multi-regional PID strategies can be applied, such as fuzzy PID [28].

Interesting, although quite different, idea uses synergetic control [18] and the analytical design of aggregated regulators (ADAR) methodology [38]. The use of linear quadratic regulation (LQR) [14, 33], improved augmented LQR (IALQR) [29] or linear quadratic Gaussian (LQG) control [31] forms the first steps towards the use of the model predictive control (MPC) techniques. Although MPC strategy is quite frequently, especially recently, used in robotic applications [10, 11, 16, 17, 21, 26], its application to a pan–tilt tracking system is rare [6, 34, 43].

Visual trackers can use various algorithms to evaluate and extrapolate tracked object trajectories. Kalman filters are the most obvious choices [30, 39]. However, the subject literature is much richer [36]. Interestingly, deep coupling between both tasks is not popular, although quite obvious. MPC strategy needs to have a setpoint trajectory over the prediction horizon. And yet the tracking algorithm allows us to evaluate it. The relationship works in the opposite direction as well. Accurate tracking and keeping the tracked target in the same position of the camera image enables improving image recognition algorithm minimizing tracking evaluation time, and minimizing the possible effect of losing a sample due to the prolonged calculations. It is important to stress that if the tracked object is at all times in the center of the image frame, the whole procedure gives the best results.

Following the above observations, the proposed research fills the observed gap. It focuses on efficient, predictive pan–tilt camera servomechanism (neck-MPC). The aim of the neck-MPC is high target-object tracking accuracy within a camera image frame. The application of predictive control algorithms, such as DMC and GPC in the pan–tilt tracking system enables practical realization of coupling between the servomechanism and image recognition. The DMC and GPC control rules represent a relatively simple predictive approach. However, their superiority lies not only in an embedded model or optimization, but in the repetitive operation over a horizon. It is especially important how the setpoint is formulated. The PID-type controller uses only the current setpoint and current control error in its rule, even if the future setpoint trajectory is known. Predictive control enables the incorporation of the knowledge about the future setpoint trajectory in the evaluation of the current manipulated variable. We acknowledge the fact that the more sophisticated is the control algorithm, the better is its performance. However, its robustness, sustainability and the number of engineers that can maintain it generally diminishes. Therefore, a simple approach, such as DMC or GPC, is taken into account in the present research. High tracking performance improves visual data processing, generating a beneficial synergy effect.

The paper is organized as follows. Section 2 presents the theoretical aspects of this research, i.e., the system kinematics (Sect. 2.1), reference PID control (Sect. 2.2), and the servomotor predictive control strategy (Sect. 2.3). The developed neck-MPC is validated with Matlab, the Gazebo simulator and the real Velma robot, what is described in Sect. 3. Section 4 summarizes the results of the work, draws conclusions, and presents the identified open issues for further research.

2 Pan–tilt control system

This section describes two main elements of the proposed contribution, i.e., the controlled pan–tilt system and the applied control strategy.

2.1 Neck kinematics

The relevant part of the kinematic structure of the Velma robot [35] equipped with a pan-tilt (neck) mechanism is presented in Fig. 1. Table 1 contains the respective D-H parameters [7].

Fig. 1
figure 1

Coordinate frames assigned according to Denavit–Hartenberg convention to the links of the Velma robot between the base coordinate frame 0 and the camera frame C; the left and right arms are marked symbolically and only their end-effector frames \(E_{l}\) and \(E_{r}\) are presented

Full size image

The following frames are relevant to the considerations: 0 – base coordinate frame, 1 – robot torso frame, 2 – pan–tilt first link frame, 3 – pan-tilt second link frame, C – camera frame, G – goal frame (the tracked object). Homogeneous matrices represent those frames, e.g., \(^{0}_{C}\varvec{\mathcal {T}}\) – camera with respect to the base. A homogeneous matrix contains the orientation submatrix, e.g., \(^{0}_{C}\mathcal {R}\), and the position vector, e.g., \(^{0}_{C}\varvec{\mathcal {P}}\).

Table 1 Denavit–Hartenberg parameters of the kinematic structure between the base and the neck of the Velma robot
Full size table

The task of the visual servo is to track the object of interest G in such a way that the location of its image will be approximately at the center of the camera imaging plane. For that to happen, the versor of the camera optical axis \(^{C}_{C}z\) and the vector from the camera to the center of the object \(^{C}_{G}\varvec{\mathcal {P}}\) should coincide. In such a situation, the vector cross-product of those two vectors should be 0. If this is not the case, the vector cross-product defines the axis of rotation about which the camera should be rotated so that the image of the object will become centered in the camera image plane. If \(^{C}_{G}\varvec{\mathcal {P}}\) is normalized the length of the resulting vector is equal to \(\left\| ^{C}_{C}z \times \frac{^{C}_{G}\varvec{\mathcal {P}}}{\left\| ^{C}_{G}\varvec{\mathcal {P}} \right\| } \right\| = \sin \phi \), where \(\phi \) is the required angle of rotation. If this value is nonzero, the versor of the axis of rotation \(\kappa \) has to be derived

$$\begin{aligned} \varvec{\kappa } = \left[ \begin{array}{c} \kappa _{x} \\ \kappa _{y} \\ \kappa _{z} \end{array} \right] = \left[ \begin{array}{c} - \frac{^{C}_{G}\varvec{\mathcal {P}}_{y}}{\sqrt{^{C}_{G}\varvec{\mathcal {P}}_{x}^{2}+\,^{C}_{G}\varvec{\mathcal {P}}_{y}^{2}}} \\ \frac{^{C}_{G}\varvec{\mathcal {P}}_{x}}{\sqrt{^{C}_{G}\varvec{\mathcal {P}}_{x}^{2}+\,^{C}_{G}\varvec{\mathcal {P}}_{y}^{2}}} \\ 0 \end{array} \right] . \end{aligned}$$
(1)

If \(\sin \phi =0\), no camera motion is required, thus versor \(\kappa \) is irrelevant. Hence, if the motion is required, the orientation of the camera has to change by rotating about \(\kappa \) by \(\phi \), i.e., the desired rotation is [7]:

$$\begin{aligned} \begin{array}{l} \varvec{\mathcal {R}}_{\kappa ,\phi } = \left[ \begin{array}{c} \kappa _{x}\kappa _{x} (1-\text{ c }\phi )+\text{ c }\phi \\ \kappa _{x}\kappa _{y} (1-\text{ c }\phi )+\kappa _{z}\text{ s }\phi \\ \kappa _{x}\kappa _{z} (1-\text{ c }\phi )-\kappa _{y}\text{ s }\phi \\ \end{array} \right| \\ \left. \begin{array}{c|c} \kappa _{y}\kappa _{x} (1-\text{ c }\phi )-\kappa _{z}\text{ s }\phi &{} \kappa _{z}\kappa _{x} (1-\text{ c }\phi )+\kappa _{y}\text{ s }\phi \\ \kappa _{y}\kappa _{y} (1-\text{ c }\phi )+\text{ c }\phi &{} \kappa _{z}\kappa _{y} (1-\text{ c }\phi )-\kappa _{x}\text{ s }\phi \\ \kappa _{y}\kappa _{z} (1-\text{ c }\phi )+\kappa _{x}\text{ s }\phi &{} \kappa _{z}\kappa _{z} (1-\text{ c }\phi )+\text{ c }\phi \\ \end{array} \right] , \end{array} \end{aligned}$$
(2)

where here and further \(\text{ s }\) stands for \(\sin \) and \(\text{ c }\) for \(\cos \). The desired orientation of the camera with respect to the torso is: \(^{1}_{C}\varvec{\mathcal {R}}(\varphi _{c}, \psi _{c}) \varvec{\mathcal {R}}_{\kappa ,\phi }\). This has 3-DOF, while the neck has only 2-DOF; thus an artificial angle \(\gamma \) is introduced:

$$\begin{aligned} ^{1}_{C}\varvec{\mathcal {R}}(\theta _{2},\theta _{3}) \varvec{\mathcal {R}}_{\kappa ,\phi } = \; ^{1}_{C}\varvec{\mathcal {R}}(\theta _{2}^{\textrm{SP}},\theta _{3}^\textrm{SP}) \varvec{\mathcal {R}}_{\,_{C}z,\gamma } \end{aligned}$$
(3)

where \(\theta _{2}\) and \(\theta _{3}\) are the current value of the neck joint angles, while superscript \(\textrm{SP}\) represents the desired value (set-point) of the angles.

The value of \(^{1}_{C}\varvec{\mathcal {R}}(\theta _{2},\theta _{3})\) is produced by solving the direct kinematics problem:

$$\begin{aligned} ^{1}_{C}\varvec{\mathcal {R}}(\theta _{2},\theta _{3}) = \left[ \begin{array}{ccc} \text{ s }\theta _{2} &{} \text{ c }\theta _{2}\text{ c }\theta _{3} &{} -\text{ c }\theta _{2}\text{ s }\theta _{3} \\ -\text{ c }\theta _{2} &{} \text{ s }\theta _{2}\text{ c }\theta _{3} &{} -\text{ s }\theta _{2}\text{ s }\theta _{3} \\ 0 &{} \text{ s }\theta _{3} &{} \text{ c }\theta _{3} \\ \end{array} \right] \end{aligned}$$
(4)

using the current encoder readings. The value of \(\varvec{\mathcal {R}}_{\kappa ,\phi }\) is computed based on (2). Hence

$$\begin{aligned} ^{1}_{C}\varvec{\mathcal {R}}(\theta _{2},\theta _{3}) \varvec{\mathcal {R}}_{\kappa ,\phi } = \left[ \begin{array}{ccc} r_{11} &{} r_{12} &{} r_{13} \\ r_{21} &{} r_{22} &{} r_{23} \\ r_{31} &{} r_{32} &{} r_{33} \end{array} \right] \end{aligned}$$
(5)

is a known matrix. Thus the desired orientation is:

$$\begin{aligned} \begin{array}{l} ^{1}_{C}\varvec{\mathcal {R}}(\theta _{2}^{\textrm{SP}},\theta _{3}^{\textrm{SP}}) = \, ^{1}_{C}\varvec{\mathcal {R}}(\theta _{2},\theta _{3}) \varvec{\mathcal {R}}_{\kappa ,\phi } \varvec{\mathcal {R}}_{_{C}z, \gamma }^{-1} \\ = \; \left[ \begin{array}{ccc} r_{11}\text{ c }\gamma - r_{12}\text{ s }\gamma &{} r_{11}\text{ s }\gamma + r_{12}\text{ c }\gamma &{} r_{13} \\ r_{21}\text{ c }\gamma - r_{22}\text{ s }\gamma &{} r_{21}\text{ s }\gamma + r_{22}\text{ c }\gamma &{} r_{23} \\ r_{31}\text{ c }\gamma - r_{32}\text{ s }\gamma &{} r_{31}\text{ s }\gamma + r_{32}\text{ c }\gamma &{} r_{33} \end{array} \right] . \end{array} \end{aligned}$$
(6)

Equating (6) and (4) and using their last columns, as they do not depend on \(\gamma \), the inverse kinematics problem for the neck is solved:

$$\begin{aligned} \begin{array}{lcl} \theta _{3}^{\textrm{SP}} = \theta _{\textrm{tilt}}^{\textrm{SP}} &{} = &{} \text{ Atan2 } (\pm \sqrt{r_{13}^{2} + r_{23}^{2}}, r_{33}),\\ \theta _{2}^{\textrm{SP}} = \theta _{\textrm{pan}}^{\textrm{SP}} &{} = &{} \text{ Atan2 } (r_{23}, r_{13}) \text{ or } \\ \theta _{2}^{\textrm{SP}} = \theta _{\textrm{pan}}^{\textrm{SP}} &{} = &{} \text{ Atan2 } (-r_{23}, -r_{13}). \end{array} \end{aligned}$$
(7)

It is important to note that equations (7) assume that \(sin(\theta _{3})\ne 0\). Fortunately, the angle \(\theta _{3}\) never equals to \(0+k\pi \), \(k=0,1,2,\ldots \), due to the kinematics limitations. In other words – it is physically impossible for the camera to be pointed vertically upwards.

2.2 PID – benchmark control strategy

A PID algorithm has also been formulated to check whether an MPC algorithm such as DMC would give better results than this standard controller. In continuous time this algorithm generates new values of the control signal according to the following formula:

$$\begin{aligned} \varvec{{\hat{\theta }}}(t)=\varvec{{\hat{\theta }}}(0)+K_{\textrm{G}} \Big ({\varvec{e}}(t)+\frac{1}{T_\textrm{I}}\int ^t_0{\varvec{e}}(\uptau )\textrm{d} \uptau +T_\textrm{D}\frac{\textrm{d}{\varvec{e}}(t)}{\textrm{d}t}\Big ) \end{aligned}$$
(8)

where \(\varvec{{\hat{\theta }}}(0)\) is the initial value of control for the integral part, \({\varvec{e}}\) is the control error, \(K_{\textrm{G}}\) is the gain, \(T_\textrm{I}\) is the time constant of the the integral part and \(T_\textrm{D}\) is the time constant of the derivative part. Vectors in the formulation (8) have the following form: \(\varvec{{\hat{\theta }}}(t)=[{\hat{\theta }}_{\textrm{pan}}(t)\,{\hat{\theta }}_{\textrm{tilt}}(t)]^{\textrm{T}}\), \(\varvec{{\hat{\theta }}}(0)=[{\hat{\theta }}_{\textrm{pan}}(0)\,{\hat{\theta }}_{\textrm{tilt}}(0)]^{\textrm{T}}\), \({\varvec{e}}=[{\varvec{e}}_{\textrm{pan}}\,{\varvec{e}}_{\textrm{tilt}}]^{\textrm{T}}\). The control error is calculated as follows:

$$\begin{aligned} {\varvec{e}}(t)=\varvec{\theta }^{\textrm{SP}}(t)-\varvec{\theta }(t) \end{aligned}$$
(9)

Formula (8) can be rewritten into discrete time (k) form, with the sampling period \(T_\textrm{s}\):

$$\begin{aligned} \varvec{{\hat{\theta }}}(k)=a_2{\varvec{e}}(k-2)+a_1{\varvec{e}}(k-1)+a_0{\varvec{e}}(k) +\varvec{{\hat{\theta }}}(k-1) \end{aligned}$$
(10)

where

$$\begin{aligned} a_0&=K_{\textrm{G}}\left( 1+\frac{T_\textrm{s}}{2T_\textrm{I}}+\frac{T_\textrm{D}}{T_\textrm{s}}\right) \end{aligned}$$
(11)
$$\begin{aligned} a_1&=K_{\textrm{G}}\left( \frac{T_\textrm{s}}{2T_\textrm{I}}-2\frac{T_\textrm{D}}{T_\textrm{s}}-1\right) \end{aligned}$$
(12)
$$\begin{aligned} a_2&=K_{\textrm{G}}\frac{T_\textrm{D}}{T_\textrm{s}} \end{aligned}$$
(13)

Vectors in the discrete-time notation presented in (10) have analogous form to the respective vectors in continuous time notation.

2.3 Predictive pan–tilt servoing

Model predictive control (MPC) constitutes the focal element of the system. The considered predictive pan–tilt camera system control is realized in a hierarchical control structure with the MPC delivering joint angle setpoints for the low level servomotors controlling the joint motors. Figure 2 presents the pan–tilt control system neck-MPC(\(\varvec{\theta }\)) tracking the target in the pan \(\theta _\textrm{pan}\left( k \right) \) and tilt \(\theta _\textrm{tilt}\left( k \right) \) joint angle spaces.

The neck-MPC controller operates in a direct mode with a high sampling frequency of 500 Hz. It directly controls the pan and tilt joint angles. Its setpoint is generated by the vision system, which acquires the image of the target. Visual object pose estimation operates with a longer sampling rate of 20 Hz. This setpoint value over the 50-ms interval is interpolated into the 2  ms sampling of the pan–tilt servomechanism. Therefore, the dynamic pan–tilt controller has enough time to realize each setpoint signal. Moreover, some time is left for setpoint signal shaping and smoothing during interpolation.

Fig. 2
figure 2

Pan–tilt neck-MPC(\(\varvec{\theta }\)) control in the joint space

Full size image

MPC controllers work with the sampling frequency of 500 Hz (denoted by time instant index k). They obtain their setpoint with 20 Hz (denoted by time instant index \(\tau \)), which is further interpolated from 20 Hz to 500 Hz. The employed MPC algorithm and the proposed modifications, i.e., the internal model, cost function and constraints, are described below. The schematic diagram of the implementation procedures of the proposed approach is sketched in Fig. 3.

Fig. 3
figure 3

Diagram of the procedures necessary to implement the MPC approach

Full size image

2.3.1 Model predictive control—general formulation

The process input, i.e., manipulated variable (MV), is denoted by u and the output, controlled variable (CV), is denoted by y. The vector of decision variables calculated at each discrete sampling moment (\(k=0,1,2,\ldots \)) by the MPC [37] is

$$\begin{aligned}{} & {} \triangle {\varvec{u}}(k)=\left[ \triangle u(k|k) \ \triangle u(k+1|k)\ldots \right. \nonumber \\{} & {} \quad \left. \triangle u(k+N_{\textrm{u}}-1|k)\right] ^{\textrm{T}}, \end{aligned}$$
(14)

where \(N_{\textrm{u}}\) is the control horizon, i.e., the number of calculated future control increments defined as backward differences, i.e., \(\triangle u(k|k)=u(k|k)-u(k-1)\) and \(\triangle u(k+p|k)=u(k+p|k)-u(k+p-1|k)\) for \(p=1,\ldots ,N_{\textrm{u}}-1\). For \(p\ge N_{\textrm{u}}\) it is assumed that the manipulated variable is constant, i.e., \(u(k+p|k)=u(k+N_{\textrm{u}}-1|k)\). The decision variables of MPC (14) are calculated by solving the optimization problem:

$$\begin{aligned}&\min \limits _{\triangle {\varvec{u}}(k)}\Bigg \{\sum _{p=1}^{N}\psi _p (y^{\textrm{SP}}(k+p|k)-{\bar{y}}(k+p|k))^{2}+\nonumber \\&\qquad \qquad +\sum _{p=0}^{N_{\textrm{u}}-1}\lambda _p(\triangle u(k+p|k))^{2}\Bigg \} \end{aligned}$$
(15)

subject to

$$\begin{aligned}{} & {} u^{\min } \le u(k+p|k)\le u^{\max },\ p=0,\ldots ,N_{\textrm{u}}-1,\\{} & {} \quad -\triangle u^{\max } \le \triangle u(k+p|k)\le \triangle u^{\max },\\{} & {} \quad p=0,\ldots ,N_{\textrm{u}}-1,\\{} & {} \quad y^{\min } \le {\bar{y}}(k+p|k)\le y^{\max },\ p=1,\ldots ,N,\\{} & {} \quad {\bar{y}}(k+p|k)=f_{\textrm{model}}(\cdot ),\ p=1,\ldots ,N. \end{aligned}$$

The role of the first part of the MPC cost function is to minimize the predicted control errors over the prediction horizon N. The setpoint (CV demand) and predicted values of the CV for the future sampling instant \(k+p\) known or calculated at the current moment k are denoted by \(y^{\textrm{SP}}(k+p|k)\) and \({\bar{y}}(k+p|k)\). The predictions \({\bar{y}}(k+p|k)\) are calculated on-line using the model of the process, described by the general function \(f_{\textrm{model}}(\cdot )\).

The role of the second part of the cost function is to eliminate excessive MV changes. In general, the constraints may be imposed on

  • future excessive values of the manipulated variable (over the control horizon), limited by the minimal and maximal allowed values \(u^{\min }\) and \(u^{\max }\),

  • future MV changes with its maximal value denoted as \(\triangle u^{\max }\) and

  • predicted values of the controlled variable (over the prediction horizon), limited by the minimal and maximal allowed values \(y^{\min }\) and \(y^{\max }\).

Although the whole sequence of decision variables (14) is calculated at each sampling, only its first component is applied to the process. Measurement of the process output is updated during the next sampling period \(k+1\), and the procedure is repeated. Appropriate stability proofs exist and can be found in the literature [1, 23]. The considered MPC optimization task (15) is common in industrial practice [9].

2.3.2 Applied MPC formulation

The optimization problem (15) can be rewritten for the neck into the following form:

$$\begin{aligned}&\min \limits _{\triangle \varvec{{\hat{\varTheta }}}(k)} \Bigg \{ J(k)=\left\| \varvec{\varTheta }^{\textrm{SP}}(k)-\bar{\varvec{\varTheta }}(k) \right\| ^{2}_{\varvec{\psi }}+\left\| \triangle \hat{\varvec{\varTheta }}(k)\right\| ^{2}_{\varvec{\lambda }}\Bigg \}\nonumber \\&\text {subject to} \end{aligned}$$
(16)
$$\begin{aligned}&-\hat{\varvec{\varTheta }}^{\max } \le \hat{\varvec{\varTheta }}(k)\le \hat{\varvec{\varTheta }}^{\max }\nonumber \\&-\triangle \hat{\varvec{\varTheta }}^{\max } \le \triangle \hat{\varvec{\varTheta }}(k)\le \triangle \hat{\varvec{\varTheta }}^{\max } \end{aligned}$$
(17)

where

$$\begin{aligned}&\varvec{\varTheta }^{\textrm{SP}}(k)=\left[ \begin{array} {c} \varvec{\theta }^{\textrm{SP}}(k+1|k)\\ \vdots \\ \varvec{\theta }^{\textrm{SP}}(k+N|k) \end{array} \right] , \bar{\varvec{\varTheta }}(k)=\left[ \begin{array} {c} \bar{\varvec{\theta }}(k+1|k)\\ \vdots \\ \bar{\varvec{\theta }}(k+N|k) \end{array} \right] , \\&\triangle \hat{\varvec{\varTheta }}^{\max }=\left[ \begin{array} {c} \triangle \varvec{{\hat{\theta }}}^{\max }\\ \vdots \\ \triangle \varvec{{\hat{\theta }}}^{\max } \end{array} \right] , \triangle \hat{\varvec{\varTheta }}(k)=\left[ \begin{array}{c} \triangle \varvec{{\hat{\theta }}}(k|k)\\ \vdots \\ \triangle \varvec{{\hat{\theta }}}(k+N_{\textsf{u}}-1|k) \end{array} \right] ,\\&\hat{\varvec{\varTheta }}^{\max }=\left[ \begin{array} {c} \varvec{{\hat{\theta }}}^{\max }\\ \vdots \\ \varvec{{\hat{\theta }}}^{\max } \end{array} \right] , \hat{\varvec{\varTheta }}(k)=\left[ \begin{array}{c} \varvec{{\hat{\theta }}}(k|k)\\ \vdots \\ \varvec{{\hat{\theta }}}(k+N_{\textsf{u}}-1|k) \end{array} \right] . \end{aligned}$$

The optimization task (16) minimizes the control error for the estimated target pan and tilt positions as well as excessive changes of their next positions. It is subject to a single constraint. The joints can be positioned between \(\pm 90^{\circ }\), i.e., \(\pm 1.5708\) radians.

The first element of the predicted target estimated positions for both joints \(\bar{\varvec{\varTheta }}(k)\) can be expanded into \(\bar{\varvec{\theta }}(k+1|k)=[{\bar{\theta }}_{\textrm{pan}}(k+1|k)\;\; {\bar{\theta }}_{\textrm{tilt}}(k+1|k)]^\textrm{T}\) and the last element into \(\bar{\varvec{\theta }}(k+N|k)=[{\bar{\theta }}_{\textrm{pan}} (k+N|k)\;\;{\bar{\theta }}_{\textrm{tilt}}(k+N|k)]^\textrm{T}\). Similarly, all elements of \(\varvec{\theta }^{\textrm{SP}}(k)\), \(\hat{\varvec{\theta }}(k)\), \(\triangle \hat{\varvec{\theta }}(k)\), \(\hat{\varvec{\theta }}^{\max }\) and \(\triangle \hat{\varvec{\theta }}^{\max }\) are described, where \(\hat{\varvec{\theta }}^{\max }=[1.5708\,\,1.5708]^\textrm{T}\) and \(\triangle \hat{\varvec{\theta }}^{\max }=[0.002\,\,0.002]^\textrm{T}\). The matrix weighting the errors \(\varvec{\varPsi }=\textrm{diag}(\varvec{\psi }_1,\ldots ,\varvec{\psi }_N)\), where \(\varvec{\psi }_i=\left[ \psi _\textrm{pan}\,\,\psi _\textrm{tilt} \right] ^\textrm{T}\), while the matrix weighting the excessive changes of control \(\varvec{\varLambda }=\textrm{diag}(\varvec{\lambda }_0,\ldots ,\varvec{\lambda }_{N_\textrm{u}-1})\), where \(\varvec{\lambda }_i=\left[ \lambda _\textrm{pan}\,\,\lambda _\textrm{tilt}\right] ^\textrm{T}\). By manipulating the values of parameters inside the matrix \(\varvec{\varPsi }\) an improvement of control performance can be obtained [25]. The result may be significantly better, when the parameters of matrix \(\varvec{\varLambda }\) are changed. However, this approach is less effective, if the setpoint trajectory constantly changes, as in our case.

For the considered robot neck two MPC algorithms are used: DMC and GPC.

DMC (Dynamic Matrix Control) algorithm uses a step response model to calculate the prediction of the controlled variables. First, the step response coefficients are gathered from a dynamic model of the process. Next, the step responses are used to create the following matrix

$$\begin{aligned} {\varvec{m}}_p=\left[ \begin{array}{cc} m_p^{\textrm{pan},\textrm{pan}} &{} m_p^{\textrm{pan},\textrm{tilt}}\\ m_p^{\textrm{tilt},\textrm{pan}} &{} m_p^{\textrm{tilt},\textrm{tilt}} \end{array} \right] \end{aligned}$$
(18)

for \(p=1,\ldots ,D\), where D denotes the horizon of sampling intervals needed for the output to stabilize. The step response matrix \(\varvec{m}_p\), in (18), is then used to formulate the following dynamic matrix

$$\begin{aligned} {\varvec{M}}=\left[ \begin{array}{cccc} {\varvec{m}}_{1} &{} {\varvec{0}} &{} \ldots &{} {\varvec{0}}\\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ {\varvec{m}}_{N} &{} {\varvec{m}}_{N-1} &{} \ldots &{} {\varvec{m}}_{N-N_{\textrm{u}}+1} \end{array} \right] \end{aligned}$$
(19)

and also the past dynamic matrix

$$\begin{aligned} {\varvec{M}}^{\textrm{p}}=\left[ \begin{array}{ccc} {\varvec{m}}_{2}-{\varvec{m}}_{1} &{} \ldots &{} {\varvec{m}}_{D}-{\varvec{m}}_{D-1}\\ \vdots &{} \ddots &{} \vdots \\ {\varvec{m}}_{N+1}-{\varvec{m}}_{1} &{} \ldots &{} {\varvec{m}}_{N+D-1}-{\varvec{m}}_{D-1} \end{array} \right] \end{aligned}$$
(20)

which are of dimensionality \(2N \times 2N_{\textrm{u}}\) and \(2N \times 2(D-1)\), respectively. Finally, the vector of the predicted controlled variables is calculated as follows

$$\begin{aligned} \hat{\bar{\varvec{\varTheta }}}(k)={\varvec{M}}\triangle \hat{\varvec{\varTheta }}(k) +\varvec{\varTheta }(k)+{\varvec{M}}^{\textrm{p}}(k)\triangle \hat{\varvec{\varTheta }}^{\textrm{p}}(k) \end{aligned}$$
(21)

where the vectors

$$\begin{aligned} {\varvec{\varTheta }}(k)=\left[ \begin{array}{c} {\varvec{\theta }}(k)\\ \vdots \\ {\varvec{\theta }}(k) \end{array} \right] , \ \triangle \hat{\varvec{\varTheta }}^{\textrm{p}}(k)=\left[ \begin{array}{c} \triangle \hat{\varvec{\theta }}(k-1)\\ \vdots \\ \triangle \hat{\varvec{\theta }}(k-(D-1)) \end{array} \right] \nonumber \\ \end{aligned}$$
(22)

are of length 2N and \(2(D-1)\), respectively. The minimized MPC cost function (16), with the use of the prediction equation (21), can be rewritten as

$$\begin{aligned}&J_{\textrm{DMC}}(k)=\Big \Vert \varvec{\varTheta }^{\textrm{SP}} (k)-{\varvec{M}}\triangle \hat{\varvec{\varTheta }}(k)-\varvec{\varTheta }(k)\nonumber \\&\quad -{\varvec{M}}^{\textrm{p}}(k)\triangle \hat{\varvec{\varTheta }}^{\textrm{p}}(k)\Big \Vert ^{2}_{\varvec{\varPsi }} +\left\| \triangle \hat{\varvec{\varTheta }}(k)\right\| ^{2}_{\varvec{\varLambda }} \end{aligned}$$
(23)

The cost function in (23) is quadratic with respect to \(\triangle \hat{\varvec{\varTheta }}\). After differentiating it, we obtain

$$\begin{aligned}&\frac{\textrm{d}J_{\textrm{DMC}}(k)}{\textrm{d}\triangle \hat{\varvec{\varTheta }}(k)}=-2{\varvec{M}}^{T}\varvec{\varPsi } \Big (\varvec{\varTheta }^{\textrm{SP}}(k)-{\varvec{M}}\triangle \hat{\varvec{\varTheta }}(k)-\varvec{\varTheta }(k)\nonumber \\&\quad -{\varvec{M}}^{\textrm{p}}\triangle \hat{\varvec{\varTheta }}^ {\textrm{p}}(k)\Big )+2\varvec{\varLambda }\triangle \hat{\varvec{\varTheta }}(k) \end{aligned}$$
(24)

The optimal solution vector is found by making the gradient vector calculated by equation (24) equal to zero and as a result the following formula is obtained

$$\begin{aligned} \triangle \hat{\varvec{\varTheta }}(k)={\varvec{K}}(\varvec{\varTheta }^{\textrm{SP}}(k) -\varvec{\varTheta }(k)-{\varvec{M}}^{\textrm{p}}\triangle \hat{\varvec{\varTheta }}^{\textrm{p}}(k)) \end{aligned}$$
(25)

where

$$\begin{aligned} {\varvec{K}}=({\varvec{M}}^{\textrm{T}}\varvec{\varPsi } {\varvec{M}}+{\varvec{\varLambda }})^{-1}{\varvec{M}}^{\textrm{T}}\varvec{\varPsi } \end{aligned}$$
(26)

is a matrix of dimensionality \(2N_{\textrm{u}}\times 2N\).

GPC (Generalized Predictive Control) algorithm is the foundation of the second approach. In this control approach, the vector of predicted controlled variables is found using the following formula

$$\begin{aligned} \bar{\varvec{\varTheta }}(k)={\varvec{M}}\triangle \hat{\varvec{\varTheta }}(k)+\varvec{\varTheta }^0(k) \end{aligned}$$
(27)

where the matrix \({\varvec{M}}\) is the same as in the DMC algorithm (19). The free trajectory vector, denoted as \(\varvec{\varTheta }^0(k)\), is calculated using the dynamic model of the process

$$\begin{aligned} \varvec{\varTheta }^0(k)=\left[ \begin{array}{c} \varvec{\theta }^0(k+1|k)\\ \vdots \\ \varvec{\theta }^0(k+N|k) \end{array} \right] \end{aligned}$$
(28)

where the first entry \(\varvec{\theta }^0(k+1|k)\) can be expanded into \(\varvec{\theta }^0(k+1|k)=[\theta ^0_{\textrm{pan}}(k+1|k)\,\,\theta ^0_{\textrm{tilt}}(k+1|k)]^\textrm{T}\) and the last element into \(\varvec{\theta }^0(k+N|k)=[\theta ^0_{\textrm{pan}}(k+N|k)\,\,\theta ^0_{\textrm{tilt}}(k+N|k)]^\textrm{T}\). The considered model, in this algorithm, is as follows

$$\begin{aligned} {\varvec{A}}(q^{-1})\varvec{\varTheta }(k) ={\varvec{B}}(q^{-1})\hat{\varvec{\varTheta }}(k) \end{aligned}$$
(29)

where the entries of the matrices

$$\begin{aligned} {\varvec{A}}(q^{-1})&=\left[ \begin{array}{cc} A_{\textrm{pan}}(q^{-1}) &{} 0\\ 0 &{} A_{\textrm{tilt}}(q^{-1}) \end{array} \right] \end{aligned}$$
(30)
$$\begin{aligned} {\varvec{B}}(q^{-1})&=\left[ \begin{array}{cc} B_{\textrm{pan},\textrm{pan}}(q^{-1}) &{} B_{\textrm{pan},\textrm{tilt}}(q^{-1})\\ B_{\textrm{tilt},\textrm{pan}}(q^{-1}) &{} B_{\textrm{tilt},\textrm{tilt}}(q^{-1}) \end{array} \right] \end{aligned}$$
(31)

are the polynomials in the backward shift operator \(q^{-1}\)

$$\begin{aligned}&A_{\textrm{pan}}(q^{-1})=1+a_{1,\textrm{pan}} q^{-1}+\ldots +a_{n_{\textrm{A}},\textrm{pan}} q^{-n_{\textrm{A}}}\nonumber \\&A_{\textrm{tilt}}(q^{-1})=1+a_{1,\textrm{tilt}} q^{-1}+\ldots +a_{n_{\textrm{A}},\textrm{tilt}} q^{-n_{\textrm{A}}} \end{aligned}$$
(32)
$$\begin{aligned}&B_{\textrm{pan},\textrm{pan}}(q^{-1})=b_{1,\textrm{pan}}q^{-1}+\ldots +b_{n_{\textrm{B}},\textrm{pan}}q^{-n_{\textrm{B}}}\nonumber \\&B_{\textrm{pan},\textrm{tilt}}(q^{-1})=0\nonumber \\&B_{\textrm{tilt},\textrm{tilt}}(q^{-1})=b_{1,\textrm{tilt}}q^{-1}+\ldots +b_{n_{\textrm{B}},\textrm{tilt}}q^{-n_{\textrm{B}}}\nonumber \\&B_{\textrm{tilt},\textrm{pan}}(q^{-1})=0 \end{aligned}$$
(33)

In GPC, the cost function (23) can be rewritten as

$$\begin{aligned} J_{\textrm{GPC}}(k)=&\left\| \varvec{\varTheta }^{\textrm{SP}}(k) -{\varvec{M}}\triangle \hat{\varvec{\varTheta }}(k)-\varvec{\varTheta }^0(k)\right\| ^{2}_{\varvec{\varPsi }}\nonumber \\ {}&+\left\| \triangle \hat{\varvec{\varTheta }}(k)\right\| ^{2}_{\varvec{\varLambda }}. \end{aligned}$$
(34)

Similarly to DMC, as the cost function (34) is quadratic in terms of the decision variables, the following solution applies

$$\begin{aligned} \triangle \hat{\varvec{\varTheta }}(k)={\varvec{K}}(\varvec{\varTheta } ^{\textrm{SP}}(k)-\varvec{\varTheta }^0(k)). \end{aligned}$$
(35)

It is to be noted that the only difference between DMC and GPC is in the way the free trajectory is calculated. Respective stability analyzes can be found for DMC in [12] and for GPC in [13, 42]. Robust stability considerations are addressed in [8, 15].

2.3.3 Simplified pan–tilt camera model

A continuous-time model of the considered robot neck is proposed using second-order transfer functions for pan and tilt joints, respectively.

$$\begin{aligned} G_\textrm{pan}(s)&=\frac{K_\textrm{pan}(T_{1,\textrm{pan}}s+1)}{(T_{2,\textrm{pan}}s+1)(T_{3,\textrm{pan}}s+1)}, \end{aligned}$$
(36)
$$\begin{aligned} G_\textrm{tilt}(s)&=\frac{K_\textrm{tilt}(T_{1,\textrm{tilt}}s+1)}{(T_{2,\textrm{tilt}}s+1)(T_{3,\textrm{tilt}}s+1)}, \end{aligned}$$
(37)

where \(K_\textrm{pan}\) and \(K_\textrm{tilt}\) are static gains and \(T_{1,\textrm{pan}}\), \(T_{2,\textrm{pan}}\), \(T_{3,\textrm{pan}}\) and \(T_{1,\textrm{tilt}}\), \(T_{2,\textrm{tilt}}\), \(T_{3,\textrm{tilt}}\) are respective time constants. Pan and tilt joint positions, denoted as \({\hat{\theta }}\), measured at the current instant k of simulation form the model inputs. Target positions estimated for the pan and tilt, denoted as \(\theta ^{\textrm{SP}}\), are the model outputs. Transfer functions are discretized and rewritten in the form of the following difference equations applied to the MPC framework:

$$\begin{aligned} \theta _{\textrm{pan}}(k)=&-a_{1,\textrm{pan}}\cdot \theta _{\textrm{pan}}(k-1) -a_{2,\textrm{pan}}\cdot \theta _{\textrm{pan}}(k-2)\nonumber \\&+b_{1,\textrm{pan}}\cdot {\hat{\theta }}_{\textrm{pan}}(k-1),\end{aligned}$$
(38)
$$\begin{aligned} \theta _{\textrm{tilt}}(k)=&-a_{1,\textrm{tilt}}\cdot \theta _{\textrm{tilt}}(k-1) -a_{2,\textrm{tilt}}\cdot \theta _{\textrm{tilt}}(k-2)\nonumber \\&+b_{1,\textrm{tilt}}\cdot {\hat{\theta }}_{\textrm{tilt}}(k-1), \end{aligned}$$
(39)

where k is the time instant and \(a_{1}\), \(a_{2}\), \(b_{1}\) are constant coefficients for pan and tilt joints, respectively. The following parameters of the model have been chosen through numerous simulations: \(a_{1,\textrm{pan}}=-1.91\times 10^{0}\), \(a_{2,\textrm{pan}}=9.14\times 10^{-1}\), \(b_{1,\textrm{pan}}=3.38\times 10^{-3}\), \(a_{1,\textrm{tilt}}=-1.83\times 10^{0}\), \(a_{2,\textrm{tilt}}=8.39\times 10^{-1}\) and \(b_{1,\textrm{tilt}}=6.09\times 10^{-3}\). These models approximate very well real robot neck joints. Their step responses are shown in Fig. 4.

Fig. 4
figure 4

Step responses of the neck simplified dynamics model; output of the pan joint – left plot, output of the tilt joint – right plot

Full size image

Two performance indices are used to compare control strategies: integral absolute error (IAE), denoted as \(E_\textrm{IAE}\) (40), and mean squared error (MSE), denoted as \(E_\textrm{MSE}\) (41). The IAE index assigns equal weights to errors regardless of their absolute values. This index is robust with respect to outliers [32]. It is invaluable in checking the overall excessive changes of joint positions. If these changes are too large harmful vibrations of the neck of the real robot occur. This phenomenon can lead to potential damage of the mechanical parts of the robot. The excessive changes are minimal, so the IAE delivers more proper measure than a MSE index calculating a quadratic function of the control error.

$$\begin{aligned} E_\textrm{IAE}&=\frac{1}{N}\sum ^{k^{\textrm{max}}}_{l=1}|\triangle {\hat{\theta }}(l)|, \end{aligned}$$
(40)

where N is the total number of samples and \(k^{\textrm{max}}\) is the last sampling instant. MSE is used to take into account large control errors. They are magnified by the quadratic function.

$$\begin{aligned} E_\textrm{MSE}&=\frac{1}{N}\sum ^{k^{\textrm{max}}}_{l=1}(\triangle {\hat{\theta }}(l))^2. \end{aligned}$$
(41)

3 Results

The proposed control strategy is validated in two ways. First, the control scheme is implemented in the Gazebo simulator and tested with the Gazebo model of the robot. Secondly, the MPC controller is tested with a real Velma robot. The components of the system are implemented using the FABRIC framework (Framework for Agent-Based Robot Control Systems) [35].

3.1 Validation using the Gazebo simulator

The proposed control strategy is validated using the Gazebo simulator. All the simulations are performed in a discrete-time form, as the predictive control formulation is in a discrete-time form and the ultimate controller implementation is done on a real robot in the discrete-time form. The camera tracks sample recorded object trajectory representing a circular motion in two axes. Setpoint trajectory: \(\theta _{\textrm{pan}}^{\textrm{SP}}\) for the pan joint and \(\theta _{\textrm{tilt}}^{\textrm{SP}}\) for the tilt joint is provided by the visual estimation system. Three controllers are compared: PID, DMC and GPC. The PID uses the following parameters: \(K_{\textrm{G}}=0.1\), \(T_{\textrm{I}}=0.01\)s \(T_{\textrm{D}}=0.01\)s. The DMC is set up with: \(N=50\), \(N_\textrm{u}=25\), \(\lambda _\textrm{pan}=100\), \(\lambda _\textrm{tilt}=100\), \(\psi _\textrm{pan}=1\) and \(\psi _\textrm{tilt}=1\), while the GPC uses the same prediction and control horizons as well as the penalty imposed on the control error as the DMC, but differs in the penalty imposed on the control increments \(\lambda _\textrm{pan}=100\), \(\lambda _\textrm{tilt}=10000\).

The setpoint trajectory is circular. As the pan–tilt predictive control is implemented in the KUKA LWR 4+ robot, which is envisaged to be used as a companion robot, it is not expected that the system will have to track fast moving objects that rapidly change the direction of their motion. Thus, the parameters of the trajectory comply with the assumed robot operating conditions.

Fig. 5
figure 5

Simulation results: trajectories of the control system with the PID controller vs. trajectories of the same system, but with the DMC controller; controlled variables of both joints – first two top plots, manipulated variables of both joints – two bottom plots

Full size image
Fig. 6
figure 6

Simulation results: trajectories of the control system with the DMC controller vs. trajectories of the same system, but with the GPC controller; controlled variables of both joints – first two top plots, manipulated variables of both joints – two bottom plots

Full size image
Table 2 Simulation results in terms of performance indices
Full size table

Figure 5 shows the comparison of the performance of PID and DMC algorithms, while Fig. 6 depicts the operation of the two MPC algorithms: DMC and GPC. Table 2 presents respective performance indices for both joints. The IAE obtained for PID control has low enough values for both joints to prevent harmful effects on the robot. DMC control provides similar results in this regard. However, comparing the MSE, we notice that the DMC control gives much better results. The MSE values are much lower for both joints than in the PID control scenario. The DMC algorithm uses setpoint trajectory prediction over the prediction horizon, unavailable for PID. GPC algorithm, similarly to DMC, can utilize future changes in the setpoint trajectory. Still, the inclusion of past values of measured signals in the calculations tends to generate high control increments to quickly compensate for measurement noise, disturbances, or modeling inaccuracies. Therefore, a higher value of the penalty imposed on the control signal increments is often required to achieve stability of the controller, as it is in this case. Unfortunately, this causes a tradeoff with the control quality. As it is visible in Fig. 6 and the Table 2, even though the smoothness of the control signal is comparable to the PID and DMC algorithms, the quality is lower.

The MSE and IAE performance measures suggest the following interpretation. IAE index shows quite similar values, but MSE shows the superiority of the DMC control. It is due to the fact that each of these indices considers different performance aspects. IAE compares normal operation, while MSE focuses on the operation in the case of rapid changes and transient periods, due to the squared error. Comparison of both measures justified the interpretation of the results. Hence the DMC algorithm was implemented and used in the real Velma robot.

3.2 Validation on the Velma robot

The DMC algorithm with the same parameters has been implemented on the Velma robot. A similar target movement has been used for validation purposes. Figure 7 presents the results for this case. The following values of the performance indices have been obtained: \(E^{\textrm{pan}}_{\textrm{IAE}}=2.05\times 10^{-4}\), \(E^{\textrm{tilt}}_{\textrm{IAE}}= 2.86\times 10^{-4}\), \(E^{\textrm{pan}}_{\textrm{MSE}}=1.70\times 10^{-4}\) and \(E^{\textrm{tilt}}_{\textrm{MSE}}=6.19\times 10^{-3}\). The IAE values for both joints are much smaller than expected from the simulation results, what is even better. MSE index values are also not large. The values of these indices are close to the values obtained using the Gazebo simulator. The setpoint tracking is very accurate, also in the scenario with the use of the Velma robot. These results confirm that the assumed predictive control is appropriate.

Fig. 7
figure 7

Verification of the proposed control scheme with the Velma robot: trajectories of the control system using the DMC controller; controlled variables of both joints – first two top plots, manipulated variables of both joints – two bottom plots

Full size image

We see that the obtained responses look very similar and the level of the accuracy improvement is quite low. However, we have to keep in mind the fact that pan–tilt control is not used just to move the robot neck. We use it to control the camera that tracks the object and the object should be kept in the middle of the image. When the tracked object is in the middle of the image pattern recognition is improved, especially as the time needed for object recognition becomes lower. Moreover, the tracked object can be located quite far away from the head and therefore even small inaccuracies in the angles start to matter. Thus, the pan–tilt head operation performance has to be kept in the visual object tracking context, not just as simply movement.

4 Conclusions and further research

This paper describes the results of research focused on evaluating predictive control of a pan–tilt camera system. The issue considered in this paper is part of a larger project related to the launch of a whole visual servoing system, comprising recognition and tracking of objects with an RGB-D camera and predictive control of a KUKA LWR 4+ manipulator.

The algorithm evaluation and validation consist of three steps. Matlab simulations are followed with the simulation using the Velma robot Gazebo simulator. Finally, the selected predictive controller is implemented in the real Velma robot. Initially, three control strategies were taken into account: PID algorithm and two predictive methods: DMC and GPC. Detailed simulations point out the superiority of the DMC predictive controller, which is eventually implemented on the Velma robot. It is interesting that it matters which predictive strategy is used. They are not equivalent. One has to remember that both predictive algorithms are not equivalent. There is a single difference in the type of the model used. One cannot use one or the other with complete freedom. In addition to the different forms of the model, the control law is implemented differently, what translates into their specificity and the resulting control performance. In our case, the properties of the DMC control are more predisposed to this application.

Real-time pan–tilt object tracking confirms initial assumptions about the DMC predictive strategy and enables further investigation of the entire visual servoing Velma robot project. It is shown that even the simple predictive strategy, such as the DMC, improves tracking control. It is mostly due the fact that the MPC approach allows to incorporate into the control rule definition the knowledge about the future behavior of the setpoint. It is the main advantage of any predictive control over PID-type controllers. It is shown that we do not have to use complex predictive approaches. Simple DMC algorithm, with a simple dynamical model, improves performance. Moreover, we have to be aware that practice prefers simple approaches. Simplicity probably slightly lowers the performance, but it increases the robustness, control system sustainability, facilitates human supervision (simpler knowledge) and lowers maintenance requirements. DMC introduces improvement while being simple.

One issue that requires further analysis is the possible impact of disturbances. Actually, they virtually have not not been considered here. Although a possible movement of the robot column can be treated as a disturbance. During the realization of this project, we consciously decided not to analyze this issue. We are aware that such disturbance can be easily included into the MPC control. But then, its comparison with the PID control would be unfair from the PID perspective. Thus, the PID control layout should be modified with the feedforward disturbance decoupling. Nonetheless, this subject is really worth to be considered and as such is planned in the future research. Uncertainties are harder to be considered, especially in the simulation, as we cannot evaluate the risk they introduce. We are fully aware that they exist and impact the project, as for instance the unknown and unmodeled dynamics, which certainly occurs.

Unmodeled dynamics and the impact of unknown disturbances constitute two separate issues. The effect of unmodeled dynamics has been already taken into account. The proposed linear model, which is used as the internal model of the predictive strategy is considered as a simplification of the real pan–tilt system. It is shown that the controller that uses such a simplified model achieves good control performance. Therefore, the unmodeled dynamics is considered in the research. The other problem is the inaccurate knowledge of kinematic parameters of the considered system, e.g., Denavit–Hartenberg parameters of the kinematic model. However, our system has been calibrated, and thus those inaccuracies do not influence the system performance to a noticeable extent.

The aspect of unknown disturbances constitutes a different issue. Those disturbances can be produced by quantization effects introduced by the camera. The position of the tracked object is determined on the basis of its position in the image; hence the resolution is of significance. However, this problem is solved either by applying a camera with sufficient resolution or by using a second low-resolution camera mounted on the end effector, as described in [19]. In that case the resolution of the eye-in-hand camera does not influence significantly, the precision of locating the object, because as the gripper gets nearer to the object the image of the object occupies more space in the image, thus can be located more precisely.

Further operational validation with the robot should be performed to observe them and to investigate their impact. As the algorithm robustness is a significant feature of an embedded control strategy, the research on pan–tilt robust control offers an interesting alternative for future research.