Abstract
Moving-object tracking using a pan–tilt camera setup is quite a well-known task in robotics. However, the presented research addresses specific properties of the tracked object and introduces novel features to the pan–tilt camera control strategy. Pan–tilt camera control does not operate in an isolated environment. It is a part of the visual servoing system with specific goals. The system has to fulfill certain purposes, which affect its configuration and functionality. The pan–tilt system aims at keeping the visually tracked object within the middle of the image. At the same time, the overall visual servoing efficiently recognizes and tracks the object enabling its grasping by the robot arm. It uses a predictive strategy utilizing specific second-order linear models for pan and tilt joints. Model predictive control (MPC) introduces into the system the ability to predict camera operation over the specific horizon according to the predefined tracking goals. As the system anticipates future positions over the horizon of operation, the setpoint prediction of the future tracked system positions is required. Visual object recognition and tracking system use particular strategies for preparing online tracked object extrapolation over MPC horizon. Therefore, the pan–tilt camera system is intrinsically coupled to camera-based recognition and tracking. Predictive pan–tilt positioning keeps the tracked system in the middle of the image, while the visual system extrapolation improves the tracking performance. The proposed approach is thoroughly tested in the dedicated Gazebo-based robot simulator. Finally, the system is implemented and validated on the Velma robot. The results and their comparison with other control strategies confirm the initial assumptions, allowing further visual servoing system development.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Pan–tilt camera systems are frequently utilized in 3D visual tracking. For instance, they are used by autonomous vehicles or robots that have a vision head [2, 5, 20, 27]. Such vision systems have to solve two problems: real-time processing of visual data and inducing the correct tracking motion of the camera. Solving the problem of efficient visual information processing is a significant challenge and constitutes the majority of the research. Data acquired by an RGB-D camera need to be processed in real time [3], and thus the image must be segmented, target features must be identified and extracted, useful information must be structured into control data, and finally, these data must be delivered to the system component that requires them. Those calculations and data handling actions are time-consuming. The majority of the research focuses on their speed up using time-efficient algorithms, usually through parallelization of calculations [24] or faster hardware. The task of servopositioning of the camera supporting neck is considered separately. Control issues and very high control accuracy do not lie in the mainstream of research. The main goal is to keep the object within the image.
Practical experience shows that tracking and recognition are coupled. Image recognition performance depends on proper camera tracking and vice versa. Once we can keep the tracked object within the same sector of the image frame, we may limit the search and analysis space of the whole frame, what speeds up calculations. On the contrary, if we fail to achieve image processing within the system sampling period, the information about the target will be lost for one or more samples, what may cause the servomechanism tracking degradation or even induce losing the tracked object out of sight. Therefore, camera image processing and servo tracking tasks are closely coupled, and the correlation works in both directions. Better camera tracking operation speeds up visual data processing and prevents losing the object from sight. Fast visual data evaluation prevents sampling losses and tracking problems. As the tracked object can be located quite far away from the head and thus the cameras, even a small improvement in tracking accuracy may help.
One may find different control strategies applied in pan–tilt servoing. Classical PID-based feedback control is the primary choice. When the servomotor is overrated and fast, just simple P (proportional) version might be satisfactory, especially when the steady-state error is allowed [44]. However, most frequently full PID version is used [22, 41], sometimes modified by factors similar to those used in process control, such as derivative filtering [44] or lead-lag compensation [40]. Feedback control is improved by feedforward decoupling [27] in situations when disturbance measurement is available. A similar task can be solved alternately by active disturbance rejection control [5]. If accurate control is required, feedback control is replaced by more advanced control strategies [4]. Once system nonlinearities affect control performance, the multi-regional PID strategies can be applied, such as fuzzy PID [28].
Interesting, although quite different, idea uses synergetic control [18] and the analytical design of aggregated regulators (ADAR) methodology [38]. The use of linear quadratic regulation (LQR) [14, 33], improved augmented LQR (IALQR) [29] or linear quadratic Gaussian (LQG) control [31] forms the first steps towards the use of the model predictive control (MPC) techniques. Although MPC strategy is quite frequently, especially recently, used in robotic applications [10, 11, 16, 17, 21, 26], its application to a pan–tilt tracking system is rare [6, 34, 43].
Visual trackers can use various algorithms to evaluate and extrapolate tracked object trajectories. Kalman filters are the most obvious choices [30, 39]. However, the subject literature is much richer [36]. Interestingly, deep coupling between both tasks is not popular, although quite obvious. MPC strategy needs to have a setpoint trajectory over the prediction horizon. And yet the tracking algorithm allows us to evaluate it. The relationship works in the opposite direction as well. Accurate tracking and keeping the tracked target in the same position of the camera image enables improving image recognition algorithm minimizing tracking evaluation time, and minimizing the possible effect of losing a sample due to the prolonged calculations. It is important to stress that if the tracked object is at all times in the center of the image frame, the whole procedure gives the best results.
Following the above observations, the proposed research fills the observed gap. It focuses on efficient, predictive pan–tilt camera servomechanism (neck-MPC). The aim of the neck-MPC is high target-object tracking accuracy within a camera image frame. The application of predictive control algorithms, such as DMC and GPC in the pan–tilt tracking system enables practical realization of coupling between the servomechanism and image recognition. The DMC and GPC control rules represent a relatively simple predictive approach. However, their superiority lies not only in an embedded model or optimization, but in the repetitive operation over a horizon. It is especially important how the setpoint is formulated. The PID-type controller uses only the current setpoint and current control error in its rule, even if the future setpoint trajectory is known. Predictive control enables the incorporation of the knowledge about the future setpoint trajectory in the evaluation of the current manipulated variable. We acknowledge the fact that the more sophisticated is the control algorithm, the better is its performance. However, its robustness, sustainability and the number of engineers that can maintain it generally diminishes. Therefore, a simple approach, such as DMC or GPC, is taken into account in the present research. High tracking performance improves visual data processing, generating a beneficial synergy effect.
The paper is organized as follows. Section 2 presents the theoretical aspects of this research, i.e., the system kinematics (Sect. 2.1), reference PID control (Sect. 2.2), and the servomotor predictive control strategy (Sect. 2.3). The developed neck-MPC is validated with Matlab, the Gazebo simulator and the real Velma robot, what is described in Sect. 3. Section 4 summarizes the results of the work, draws conclusions, and presents the identified open issues for further research.
2 Pan–tilt control system
This section describes two main elements of the proposed contribution, i.e., the controlled pan–tilt system and the applied control strategy.
2.1 Neck kinematics
The relevant part of the kinematic structure of the Velma robot [35] equipped with a pan-tilt (neck) mechanism is presented in Fig. 1. Table 1 contains the respective D-H parameters [7].
The following frames are relevant to the considerations: 0 – base coordinate frame, 1 – robot torso frame, 2 – pan–tilt first link frame, 3 – pan-tilt second link frame, C – camera frame, G – goal frame (the tracked object). Homogeneous matrices represent those frames, e.g., \(^{0}_{C}\varvec{\mathcal {T}}\) – camera with respect to the base. A homogeneous matrix contains the orientation submatrix, e.g., \(^{0}_{C}\mathcal {R}\), and the position vector, e.g., \(^{0}_{C}\varvec{\mathcal {P}}\).
The task of the visual servo is to track the object of interest G in such a way that the location of its image will be approximately at the center of the camera imaging plane. For that to happen, the versor of the camera optical axis \(^{C}_{C}z\) and the vector from the camera to the center of the object \(^{C}_{G}\varvec{\mathcal {P}}\) should coincide. In such a situation, the vector cross-product of those two vectors should be 0. If this is not the case, the vector cross-product defines the axis of rotation about which the camera should be rotated so that the image of the object will become centered in the camera image plane. If \(^{C}_{G}\varvec{\mathcal {P}}\) is normalized the length of the resulting vector is equal to \(\left\| ^{C}_{C}z \times \frac{^{C}_{G}\varvec{\mathcal {P}}}{\left\| ^{C}_{G}\varvec{\mathcal {P}} \right\| } \right\| = \sin \phi \), where \(\phi \) is the required angle of rotation. If this value is nonzero, the versor of the axis of rotation \(\kappa \) has to be derived
If \(\sin \phi =0\), no camera motion is required, thus versor \(\kappa \) is irrelevant. Hence, if the motion is required, the orientation of the camera has to change by rotating about \(\kappa \) by \(\phi \), i.e., the desired rotation is [7]:
where here and further \(\text{ s }\) stands for \(\sin \) and \(\text{ c }\) for \(\cos \). The desired orientation of the camera with respect to the torso is: \(^{1}_{C}\varvec{\mathcal {R}}(\varphi _{c}, \psi _{c}) \varvec{\mathcal {R}}_{\kappa ,\phi }\). This has 3-DOF, while the neck has only 2-DOF; thus an artificial angle \(\gamma \) is introduced:
where \(\theta _{2}\) and \(\theta _{3}\) are the current value of the neck joint angles, while superscript \(\textrm{SP}\) represents the desired value (set-point) of the angles.
The value of \(^{1}_{C}\varvec{\mathcal {R}}(\theta _{2},\theta _{3})\) is produced by solving the direct kinematics problem:
using the current encoder readings. The value of \(\varvec{\mathcal {R}}_{\kappa ,\phi }\) is computed based on (2). Hence
is a known matrix. Thus the desired orientation is:
Equating (6) and (4) and using their last columns, as they do not depend on \(\gamma \), the inverse kinematics problem for the neck is solved:
It is important to note that equations (7) assume that \(sin(\theta _{3})\ne 0\). Fortunately, the angle \(\theta _{3}\) never equals to \(0+k\pi \), \(k=0,1,2,\ldots \), due to the kinematics limitations. In other words – it is physically impossible for the camera to be pointed vertically upwards.
2.2 PID – benchmark control strategy
A PID algorithm has also been formulated to check whether an MPC algorithm such as DMC would give better results than this standard controller. In continuous time this algorithm generates new values of the control signal according to the following formula:
where \(\varvec{{\hat{\theta }}}(0)\) is the initial value of control for the integral part, \({\varvec{e}}\) is the control error, \(K_{\textrm{G}}\) is the gain, \(T_\textrm{I}\) is the time constant of the the integral part and \(T_\textrm{D}\) is the time constant of the derivative part. Vectors in the formulation (8) have the following form: \(\varvec{{\hat{\theta }}}(t)=[{\hat{\theta }}_{\textrm{pan}}(t)\,{\hat{\theta }}_{\textrm{tilt}}(t)]^{\textrm{T}}\), \(\varvec{{\hat{\theta }}}(0)=[{\hat{\theta }}_{\textrm{pan}}(0)\,{\hat{\theta }}_{\textrm{tilt}}(0)]^{\textrm{T}}\), \({\varvec{e}}=[{\varvec{e}}_{\textrm{pan}}\,{\varvec{e}}_{\textrm{tilt}}]^{\textrm{T}}\). The control error is calculated as follows:
Formula (8) can be rewritten into discrete time (k) form, with the sampling period \(T_\textrm{s}\):
where
Vectors in the discrete-time notation presented in (10) have analogous form to the respective vectors in continuous time notation.
2.3 Predictive pan–tilt servoing
Model predictive control (MPC) constitutes the focal element of the system. The considered predictive pan–tilt camera system control is realized in a hierarchical control structure with the MPC delivering joint angle setpoints for the low level servomotors controlling the joint motors. Figure 2 presents the pan–tilt control system neck-MPC(\(\varvec{\theta }\)) tracking the target in the pan \(\theta _\textrm{pan}\left( k \right) \) and tilt \(\theta _\textrm{tilt}\left( k \right) \) joint angle spaces.
The neck-MPC controller operates in a direct mode with a high sampling frequency of 500 Hz. It directly controls the pan and tilt joint angles. Its setpoint is generated by the vision system, which acquires the image of the target. Visual object pose estimation operates with a longer sampling rate of 20 Hz. This setpoint value over the 50-ms interval is interpolated into the 2 ms sampling of the pan–tilt servomechanism. Therefore, the dynamic pan–tilt controller has enough time to realize each setpoint signal. Moreover, some time is left for setpoint signal shaping and smoothing during interpolation.
MPC controllers work with the sampling frequency of 500 Hz (denoted by time instant index k). They obtain their setpoint with 20 Hz (denoted by time instant index \(\tau \)), which is further interpolated from 20 Hz to 500 Hz. The employed MPC algorithm and the proposed modifications, i.e., the internal model, cost function and constraints, are described below. The schematic diagram of the implementation procedures of the proposed approach is sketched in Fig. 3.
2.3.1 Model predictive control—general formulation
The process input, i.e., manipulated variable (MV), is denoted by u and the output, controlled variable (CV), is denoted by y. The vector of decision variables calculated at each discrete sampling moment (\(k=0,1,2,\ldots \)) by the MPC [37] is
where \(N_{\textrm{u}}\) is the control horizon, i.e., the number of calculated future control increments defined as backward differences, i.e., \(\triangle u(k|k)=u(k|k)-u(k-1)\) and \(\triangle u(k+p|k)=u(k+p|k)-u(k+p-1|k)\) for \(p=1,\ldots ,N_{\textrm{u}}-1\). For \(p\ge N_{\textrm{u}}\) it is assumed that the manipulated variable is constant, i.e., \(u(k+p|k)=u(k+N_{\textrm{u}}-1|k)\). The decision variables of MPC (14) are calculated by solving the optimization problem:
subject to
The role of the first part of the MPC cost function is to minimize the predicted control errors over the prediction horizon N. The setpoint (CV demand) and predicted values of the CV for the future sampling instant \(k+p\) known or calculated at the current moment k are denoted by \(y^{\textrm{SP}}(k+p|k)\) and \({\bar{y}}(k+p|k)\). The predictions \({\bar{y}}(k+p|k)\) are calculated on-line using the model of the process, described by the general function \(f_{\textrm{model}}(\cdot )\).
The role of the second part of the cost function is to eliminate excessive MV changes. In general, the constraints may be imposed on
-
future excessive values of the manipulated variable (over the control horizon), limited by the minimal and maximal allowed values \(u^{\min }\) and \(u^{\max }\),
-
future MV changes with its maximal value denoted as \(\triangle u^{\max }\) and
-
predicted values of the controlled variable (over the prediction horizon), limited by the minimal and maximal allowed values \(y^{\min }\) and \(y^{\max }\).
Although the whole sequence of decision variables (14) is calculated at each sampling, only its first component is applied to the process. Measurement of the process output is updated during the next sampling period \(k+1\), and the procedure is repeated. Appropriate stability proofs exist and can be found in the literature [1, 23]. The considered MPC optimization task (15) is common in industrial practice [9].
2.3.2 Applied MPC formulation
The optimization problem (15) can be rewritten for the neck into the following form:
where
The optimization task (16) minimizes the control error for the estimated target pan and tilt positions as well as excessive changes of their next positions. It is subject to a single constraint. The joints can be positioned between \(\pm 90^{\circ }\), i.e., \(\pm 1.5708\) radians.
The first element of the predicted target estimated positions for both joints \(\bar{\varvec{\varTheta }}(k)\) can be expanded into \(\bar{\varvec{\theta }}(k+1|k)=[{\bar{\theta }}_{\textrm{pan}}(k+1|k)\;\; {\bar{\theta }}_{\textrm{tilt}}(k+1|k)]^\textrm{T}\) and the last element into \(\bar{\varvec{\theta }}(k+N|k)=[{\bar{\theta }}_{\textrm{pan}} (k+N|k)\;\;{\bar{\theta }}_{\textrm{tilt}}(k+N|k)]^\textrm{T}\). Similarly, all elements of \(\varvec{\theta }^{\textrm{SP}}(k)\), \(\hat{\varvec{\theta }}(k)\), \(\triangle \hat{\varvec{\theta }}(k)\), \(\hat{\varvec{\theta }}^{\max }\) and \(\triangle \hat{\varvec{\theta }}^{\max }\) are described, where \(\hat{\varvec{\theta }}^{\max }=[1.5708\,\,1.5708]^\textrm{T}\) and \(\triangle \hat{\varvec{\theta }}^{\max }=[0.002\,\,0.002]^\textrm{T}\). The matrix weighting the errors \(\varvec{\varPsi }=\textrm{diag}(\varvec{\psi }_1,\ldots ,\varvec{\psi }_N)\), where \(\varvec{\psi }_i=\left[ \psi _\textrm{pan}\,\,\psi _\textrm{tilt} \right] ^\textrm{T}\), while the matrix weighting the excessive changes of control \(\varvec{\varLambda }=\textrm{diag}(\varvec{\lambda }_0,\ldots ,\varvec{\lambda }_{N_\textrm{u}-1})\), where \(\varvec{\lambda }_i=\left[ \lambda _\textrm{pan}\,\,\lambda _\textrm{tilt}\right] ^\textrm{T}\). By manipulating the values of parameters inside the matrix \(\varvec{\varPsi }\) an improvement of control performance can be obtained [25]. The result may be significantly better, when the parameters of matrix \(\varvec{\varLambda }\) are changed. However, this approach is less effective, if the setpoint trajectory constantly changes, as in our case.
For the considered robot neck two MPC algorithms are used: DMC and GPC.
DMC (Dynamic Matrix Control) algorithm uses a step response model to calculate the prediction of the controlled variables. First, the step response coefficients are gathered from a dynamic model of the process. Next, the step responses are used to create the following matrix
for \(p=1,\ldots ,D\), where D denotes the horizon of sampling intervals needed for the output to stabilize. The step response matrix \(\varvec{m}_p\), in (18), is then used to formulate the following dynamic matrix
and also the past dynamic matrix
which are of dimensionality \(2N \times 2N_{\textrm{u}}\) and \(2N \times 2(D-1)\), respectively. Finally, the vector of the predicted controlled variables is calculated as follows
where the vectors
are of length 2N and \(2(D-1)\), respectively. The minimized MPC cost function (16), with the use of the prediction equation (21), can be rewritten as
The cost function in (23) is quadratic with respect to \(\triangle \hat{\varvec{\varTheta }}\). After differentiating it, we obtain
The optimal solution vector is found by making the gradient vector calculated by equation (24) equal to zero and as a result the following formula is obtained
where
is a matrix of dimensionality \(2N_{\textrm{u}}\times 2N\).
GPC (Generalized Predictive Control) algorithm is the foundation of the second approach. In this control approach, the vector of predicted controlled variables is found using the following formula
where the matrix \({\varvec{M}}\) is the same as in the DMC algorithm (19). The free trajectory vector, denoted as \(\varvec{\varTheta }^0(k)\), is calculated using the dynamic model of the process
where the first entry \(\varvec{\theta }^0(k+1|k)\) can be expanded into \(\varvec{\theta }^0(k+1|k)=[\theta ^0_{\textrm{pan}}(k+1|k)\,\,\theta ^0_{\textrm{tilt}}(k+1|k)]^\textrm{T}\) and the last element into \(\varvec{\theta }^0(k+N|k)=[\theta ^0_{\textrm{pan}}(k+N|k)\,\,\theta ^0_{\textrm{tilt}}(k+N|k)]^\textrm{T}\). The considered model, in this algorithm, is as follows
where the entries of the matrices
are the polynomials in the backward shift operator \(q^{-1}\)
In GPC, the cost function (23) can be rewritten as
Similarly to DMC, as the cost function (34) is quadratic in terms of the decision variables, the following solution applies
It is to be noted that the only difference between DMC and GPC is in the way the free trajectory is calculated. Respective stability analyzes can be found for DMC in [12] and for GPC in [13, 42]. Robust stability considerations are addressed in [8, 15].
2.3.3 Simplified pan–tilt camera model
A continuous-time model of the considered robot neck is proposed using second-order transfer functions for pan and tilt joints, respectively.
where \(K_\textrm{pan}\) and \(K_\textrm{tilt}\) are static gains and \(T_{1,\textrm{pan}}\), \(T_{2,\textrm{pan}}\), \(T_{3,\textrm{pan}}\) and \(T_{1,\textrm{tilt}}\), \(T_{2,\textrm{tilt}}\), \(T_{3,\textrm{tilt}}\) are respective time constants. Pan and tilt joint positions, denoted as \({\hat{\theta }}\), measured at the current instant k of simulation form the model inputs. Target positions estimated for the pan and tilt, denoted as \(\theta ^{\textrm{SP}}\), are the model outputs. Transfer functions are discretized and rewritten in the form of the following difference equations applied to the MPC framework:
where k is the time instant and \(a_{1}\), \(a_{2}\), \(b_{1}\) are constant coefficients for pan and tilt joints, respectively. The following parameters of the model have been chosen through numerous simulations: \(a_{1,\textrm{pan}}=-1.91\times 10^{0}\), \(a_{2,\textrm{pan}}=9.14\times 10^{-1}\), \(b_{1,\textrm{pan}}=3.38\times 10^{-3}\), \(a_{1,\textrm{tilt}}=-1.83\times 10^{0}\), \(a_{2,\textrm{tilt}}=8.39\times 10^{-1}\) and \(b_{1,\textrm{tilt}}=6.09\times 10^{-3}\). These models approximate very well real robot neck joints. Their step responses are shown in Fig. 4.
Two performance indices are used to compare control strategies: integral absolute error (IAE), denoted as \(E_\textrm{IAE}\) (40), and mean squared error (MSE), denoted as \(E_\textrm{MSE}\) (41). The IAE index assigns equal weights to errors regardless of their absolute values. This index is robust with respect to outliers [32]. It is invaluable in checking the overall excessive changes of joint positions. If these changes are too large harmful vibrations of the neck of the real robot occur. This phenomenon can lead to potential damage of the mechanical parts of the robot. The excessive changes are minimal, so the IAE delivers more proper measure than a MSE index calculating a quadratic function of the control error.
where N is the total number of samples and \(k^{\textrm{max}}\) is the last sampling instant. MSE is used to take into account large control errors. They are magnified by the quadratic function.
3 Results
The proposed control strategy is validated in two ways. First, the control scheme is implemented in the Gazebo simulator and tested with the Gazebo model of the robot. Secondly, the MPC controller is tested with a real Velma robot. The components of the system are implemented using the FABRIC framework (Framework for Agent-Based Robot Control Systems) [35].
3.1 Validation using the Gazebo simulator
The proposed control strategy is validated using the Gazebo simulator. All the simulations are performed in a discrete-time form, as the predictive control formulation is in a discrete-time form and the ultimate controller implementation is done on a real robot in the discrete-time form. The camera tracks sample recorded object trajectory representing a circular motion in two axes. Setpoint trajectory: \(\theta _{\textrm{pan}}^{\textrm{SP}}\) for the pan joint and \(\theta _{\textrm{tilt}}^{\textrm{SP}}\) for the tilt joint is provided by the visual estimation system. Three controllers are compared: PID, DMC and GPC. The PID uses the following parameters: \(K_{\textrm{G}}=0.1\), \(T_{\textrm{I}}=0.01\)s \(T_{\textrm{D}}=0.01\)s. The DMC is set up with: \(N=50\), \(N_\textrm{u}=25\), \(\lambda _\textrm{pan}=100\), \(\lambda _\textrm{tilt}=100\), \(\psi _\textrm{pan}=1\) and \(\psi _\textrm{tilt}=1\), while the GPC uses the same prediction and control horizons as well as the penalty imposed on the control error as the DMC, but differs in the penalty imposed on the control increments \(\lambda _\textrm{pan}=100\), \(\lambda _\textrm{tilt}=10000\).
The setpoint trajectory is circular. As the pan–tilt predictive control is implemented in the KUKA LWR 4+ robot, which is envisaged to be used as a companion robot, it is not expected that the system will have to track fast moving objects that rapidly change the direction of their motion. Thus, the parameters of the trajectory comply with the assumed robot operating conditions.
Figure 5 shows the comparison of the performance of PID and DMC algorithms, while Fig. 6 depicts the operation of the two MPC algorithms: DMC and GPC. Table 2 presents respective performance indices for both joints. The IAE obtained for PID control has low enough values for both joints to prevent harmful effects on the robot. DMC control provides similar results in this regard. However, comparing the MSE, we notice that the DMC control gives much better results. The MSE values are much lower for both joints than in the PID control scenario. The DMC algorithm uses setpoint trajectory prediction over the prediction horizon, unavailable for PID. GPC algorithm, similarly to DMC, can utilize future changes in the setpoint trajectory. Still, the inclusion of past values of measured signals in the calculations tends to generate high control increments to quickly compensate for measurement noise, disturbances, or modeling inaccuracies. Therefore, a higher value of the penalty imposed on the control signal increments is often required to achieve stability of the controller, as it is in this case. Unfortunately, this causes a tradeoff with the control quality. As it is visible in Fig. 6 and the Table 2, even though the smoothness of the control signal is comparable to the PID and DMC algorithms, the quality is lower.
The MSE and IAE performance measures suggest the following interpretation. IAE index shows quite similar values, but MSE shows the superiority of the DMC control. It is due to the fact that each of these indices considers different performance aspects. IAE compares normal operation, while MSE focuses on the operation in the case of rapid changes and transient periods, due to the squared error. Comparison of both measures justified the interpretation of the results. Hence the DMC algorithm was implemented and used in the real Velma robot.
3.2 Validation on the Velma robot
The DMC algorithm with the same parameters has been implemented on the Velma robot. A similar target movement has been used for validation purposes. Figure 7 presents the results for this case. The following values of the performance indices have been obtained: \(E^{\textrm{pan}}_{\textrm{IAE}}=2.05\times 10^{-4}\), \(E^{\textrm{tilt}}_{\textrm{IAE}}= 2.86\times 10^{-4}\), \(E^{\textrm{pan}}_{\textrm{MSE}}=1.70\times 10^{-4}\) and \(E^{\textrm{tilt}}_{\textrm{MSE}}=6.19\times 10^{-3}\). The IAE values for both joints are much smaller than expected from the simulation results, what is even better. MSE index values are also not large. The values of these indices are close to the values obtained using the Gazebo simulator. The setpoint tracking is very accurate, also in the scenario with the use of the Velma robot. These results confirm that the assumed predictive control is appropriate.
We see that the obtained responses look very similar and the level of the accuracy improvement is quite low. However, we have to keep in mind the fact that pan–tilt control is not used just to move the robot neck. We use it to control the camera that tracks the object and the object should be kept in the middle of the image. When the tracked object is in the middle of the image pattern recognition is improved, especially as the time needed for object recognition becomes lower. Moreover, the tracked object can be located quite far away from the head and therefore even small inaccuracies in the angles start to matter. Thus, the pan–tilt head operation performance has to be kept in the visual object tracking context, not just as simply movement.
4 Conclusions and further research
This paper describes the results of research focused on evaluating predictive control of a pan–tilt camera system. The issue considered in this paper is part of a larger project related to the launch of a whole visual servoing system, comprising recognition and tracking of objects with an RGB-D camera and predictive control of a KUKA LWR 4+ manipulator.
The algorithm evaluation and validation consist of three steps. Matlab simulations are followed with the simulation using the Velma robot Gazebo simulator. Finally, the selected predictive controller is implemented in the real Velma robot. Initially, three control strategies were taken into account: PID algorithm and two predictive methods: DMC and GPC. Detailed simulations point out the superiority of the DMC predictive controller, which is eventually implemented on the Velma robot. It is interesting that it matters which predictive strategy is used. They are not equivalent. One has to remember that both predictive algorithms are not equivalent. There is a single difference in the type of the model used. One cannot use one or the other with complete freedom. In addition to the different forms of the model, the control law is implemented differently, what translates into their specificity and the resulting control performance. In our case, the properties of the DMC control are more predisposed to this application.
Real-time pan–tilt object tracking confirms initial assumptions about the DMC predictive strategy and enables further investigation of the entire visual servoing Velma robot project. It is shown that even the simple predictive strategy, such as the DMC, improves tracking control. It is mostly due the fact that the MPC approach allows to incorporate into the control rule definition the knowledge about the future behavior of the setpoint. It is the main advantage of any predictive control over PID-type controllers. It is shown that we do not have to use complex predictive approaches. Simple DMC algorithm, with a simple dynamical model, improves performance. Moreover, we have to be aware that practice prefers simple approaches. Simplicity probably slightly lowers the performance, but it increases the robustness, control system sustainability, facilitates human supervision (simpler knowledge) and lowers maintenance requirements. DMC introduces improvement while being simple.
One issue that requires further analysis is the possible impact of disturbances. Actually, they virtually have not not been considered here. Although a possible movement of the robot column can be treated as a disturbance. During the realization of this project, we consciously decided not to analyze this issue. We are aware that such disturbance can be easily included into the MPC control. But then, its comparison with the PID control would be unfair from the PID perspective. Thus, the PID control layout should be modified with the feedforward disturbance decoupling. Nonetheless, this subject is really worth to be considered and as such is planned in the future research. Uncertainties are harder to be considered, especially in the simulation, as we cannot evaluate the risk they introduce. We are fully aware that they exist and impact the project, as for instance the unknown and unmodeled dynamics, which certainly occurs.
Unmodeled dynamics and the impact of unknown disturbances constitute two separate issues. The effect of unmodeled dynamics has been already taken into account. The proposed linear model, which is used as the internal model of the predictive strategy is considered as a simplification of the real pan–tilt system. It is shown that the controller that uses such a simplified model achieves good control performance. Therefore, the unmodeled dynamics is considered in the research. The other problem is the inaccurate knowledge of kinematic parameters of the considered system, e.g., Denavit–Hartenberg parameters of the kinematic model. However, our system has been calibrated, and thus those inaccuracies do not influence the system performance to a noticeable extent.
The aspect of unknown disturbances constitutes a different issue. Those disturbances can be produced by quantization effects introduced by the camera. The position of the tracked object is determined on the basis of its position in the image; hence the resolution is of significance. However, this problem is solved either by applying a camera with sufficient resolution or by using a second low-resolution camera mounted on the end effector, as described in [19]. In that case the resolution of the eye-in-hand camera does not influence significantly, the precision of locating the object, because as the gripper gets nearer to the object the image of the object occupies more space in the image, thus can be located more precisely.
Further operational validation with the robot should be performed to observe them and to investigate their impact. As the algorithm robustness is a significant feature of an embedded control strategy, the research on pan–tilt robust control offers an interesting alternative for future research.
Data availability
The datasets generated during and/or analyzed during the current study are not publicly available due to complexity of the experiments, but are available from the corresponding author on reasonable request.
References
Bemporad, A., Morari, M.: Robust model predictive control: a survey. In: Garulli, A., Tesi, A. (eds.) Robustness in Identification and Control, pp. 207–226. Springer, London, London (1999)
Chaumette, F.: Visual servoing. In: Ang, M.H., Khatib, O., Siciliano, B. (eds.) Encyclopedia of Robotics, pp. 1–9. Springer, Berlin, Heidelberg (2020)
Chaumette, F., Hutchinson, S.: Visual servo control. i. basic approaches. IEEE Robot. Autom. Mag. 13(4), 82–90 (2006)
Chaumette, F., Hutchinson, S.: Visual servo control. ii. advanced approaches [tutorial]. IEEE Robot. Autom. Mag. 14(1), 109–118 (2007)
Chen, H., Zhao, X., Tan, M.: A novel pan-tilt camera control approach for visual tracking. In: Proceeding of the 11th World Congress on Intelligent Control and Automation, pp. 2860–2865 (2014)
Chroust, S., Vincze, M.: Comparison of prediction methods for vision-based control of motion. In: IFAC Proceedings vol. 33(27), pp. 207–212 (2000). 6th IFAC Symposium on Robot Control (SYROCO 2000), Vienna, Austria, 21–3 Sep 2000
Craig, J.J.: Introduction to Robotics, Mechanics & Control. Addison-Wesley (1986)
Dai, L., Cheng, Z.: Robust stability conditions for siso dynamic matrix control. Dev. Chem. Eng. Miner. Process. 9(1–2), 49–56 (2001)
Domański, P.D.: Performance assessment of predictive control - a survey. Algorithms 13(4), 97 (2020)
Faulwasser, T., Weber, T., Zometa, P., Findeisen, R.: Implementation of nonlinear model predictive path-following control for an industrial robot. IEEE Trans. Control Syst. Technol. 25(4), 1505–1511 (2017)
Fehr, J., Schmid, P., Schneider, G., Eberhard, P.: Modeling, simulation, and vision-/mpc-based control of a powercube serial robot. Appl. Sci. 10(20) (2020)
Garcia, C.E., Morshedi, A.: Quadratic programming solution of dynamic matrix control (QDMC). Chem. Eng. Commun. 46(1–3), 73–87 (1986)
Gomma, H., Owens, D.: Stability analysis for generalized predictive control (gpc) and time varying weighting generalized predictive control (tgpc). In: Proceedings of the 2004 IEEE International Conference on Control Applications, 2004., vol. 2, pp. 1662–1667 Vol.2 (2004)
Han, S.E., Unel, M.: Stabilization of pan-tilt systems using acceleration based lmi-lqr controller. In: 2017 11th Asian Control Conference (ASCC), pp. 2552–2557 (2017)
Hongbin, D., Pingjing, Y.: Closed loop robust stability of mimo dynamic matrix control with input saturation. In: Chen, B., Westerberg A.W. (eds.) Process Systems Engineering 2003, 8th International Symposium on Process Systems Engineering, Computer Aided Chemical Engineering, vol. 15, pp. 790–795. Elsevier (2003)
Killpack, M.D., Kemp, C.C.: Fast reaching in clutter while regulating forces using model predictive control. In: 2013 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp. 146–153 (2013)
Kleff, S., Meduri, A., Budhiraja, R., Nicolas Mansard, L.R.: High frequency nonlinear model predictive control of a manipulator. In: IEEE International Conference on Robotics and Automation (ICRA 2021). Xi’an, China (2021)
Kolesnikov, A.A.: Introduction of synergetic control. In: 2014 American Control Conference, pp. 3013–3016 (2014)
Kornuta, T., Zieliński, C.: Robot control system design exemplified by multi-camera visual servoing. J. Intell. Robot. Syst. 77, 499–523 (2015)
Kumar, P., Dick, A., Sheng, T.S.: Real time target tracking with pan tilt zoom camera. In: 2009 Digital Image Computing: Techniques and Applications, pp. 492–497 (2009)
Li, J., Sun, J., Liu, L., Xu, J.: Model predictive control for the tracking of autonomous mobile robot combined with a local path planning. Meas. Control 0(0), 00202940211043,070 (2021)
Maaß, F.L.: An analysis on object detection and tracking with a tilt-pan camera on an embedded device. Hamburg University of Applied Sciences, Tech. rep. (2020)
Mayne, D., Rawlings, J., Rao, C., Scokaert, P.: Constrained model predictive control: stability and optimality. Automatica 36(6), 789–814 (2000)
Meribout, M., Nakanishi, M., Ogura, T.: A parallel algorithm for real-time object recognition. Pattern Recognit. 35, 1917–1931 (2002)
Nebeluk, R., Marusak, P.: Efficient mpc algorithms with variable trajectories of parameters weighting predicted control errors. Arch. Control Sci. 30(2), 325–363 (2020)
Nubert, J., Köhler, J., Berenz, V., Allgöwer, F., Trimpe, S.: Safe and fast tracking on a robot manipulator: robust mpc and neural network control. IEEE Robot. Autom. Lett. 5, 3050–3057 (2020)
Park, J., Hwang, W., Bahn, W., hun Lee, C., il Kim, T., Shaikh, M.M., soo Kim, K., il “Dan” Cho, D.: Pan/tilt camera control for vision tracking system based on the robot motion and vision information. In: IFAC Proceedings Vol. 44(1), pp. 3165–3170 (2011). 18th IFAC World Congress
Pebrianti, D., Peh, O., Samad, R., Mustafa, M., Abdullah, N.R.H., Bayuaji, L.: Intelligent control for visual servoing system. Indones. J. Electr. Eng. Comput. Sci. 6, 72 (2017)
Qu, Y., Yang, B., Yu, Z., Song, Y., Sun, Y.: Active visual servo pan/tilt control design based on improved augmented lqr. In: 2020 Chinese Control And Decision Conference (CCDC), pp. 3702–3707 (2020)
Raimondo, D.M., Gasparella, S., Sturzenegger, D., Lygeros, J., Morari, M.: A tracking algorithm for ptz cameras. In: IFAC Proceedings vol. 43(19), pp. 61–66 (2010). 2nd IFAC Workshop on Distributed Estimation and Control in Networked Systems
Ramelan, A., Saputro, J.S., Apribowo, C.H.B., Ibrahim, M.H., Pramono, S.: Design and simulation linear quadratic gaussian (lqg) for pan-tilt face tracking camera servos. AIP Conf. Proce. 2217(1), 030,073 (2020)
Rousseeuw, P.J., Leroy, A.M.: Robust Regression and Outlier Detection. John Wiley & Sons Inc, New York, NY, USA (1987)
Ruangpayoongsak, N.: Mobile robot positioning by using low-cost visual tracking system. MATEC Web Conf. 95, 08,006 (2017)
Saragih, C.F.D., Kinasih, F.M.T.R., Machbub, C., Rusmin, P.H., Rohman, A.S.: Visual servo application using model predictive control (mpc) method on pan-tilt camera platform. In: 2019 6th International Conference on Instrumentation, Control, and Automation. ICA, pp. 1–7 (2019)
Seredyński, D., Winiarski, T., Zieliński, C.: FABRIC: framework for agent-based robot control systems. In: Kozłowski K. (ed.) 12th International Workshop on Robot Motion and Control (RoMoCo), pp. 215–222 (2019). https://doi.org/10.1109/RoMoCo.2019.8787370
Tang, Y., Bilodeau, G.A.: Evaluation of trackers for pan-tilt-zoom scenarios. ArXiv:1711.04260, Cornell Univesity Library (2017)
Tatjewski, P.: Advanced Control of Industrial Processes, Structures and Algorithms. Springer, London (2007)
Tran, P., Nguyen, H., Nguyen, A., Le, T., Phan, H., Nguyen1, C.: Design control system for pan-tilt camera for visual tracking based on adar method taking into account energy output. In: E3S Web Conferences - III International Conference “Energy Efficiency and Energy Saving in Technical Systems, vol. 279, p. 02007 (2021)
Tsai, C.Y., Song, K.T., Dutoit, X., Brussel, H., Nuttin, M.: Robust mobile robot visual tracking control system using self-tuning kalman filter. In: Proceedings of the 2007 IEEE International Symposium on Computational Intelligence in Robotics and Automation, CIRA 2007, pp. 161–166 (2007)
Yosafat, S.R., Machbub, C., Hidayat, E.M.I.: Design and implementation of pan-tilt control for face tracking. In: 2017 7th IEEE International Conference on System Engineering and Technology (ICSET), pp. 217–222 (2017)
Zhang, B., Huang, J., Lin, J.: A novel control algorithm for object tracking by controlling pan/tilt automatically. In: 2010 2nd International Conference on Education Technology and Computer, vol. 1, pp. V1–596–V1–602 (2010)
Zhang, J., Xi, Y.: Some gpc stability results. Int. J. Control 70, 831–840 (1998)
Zhao, Y.H., Chai, L., Jin, L.Z.: The application of pid-dmc algorithm with feedforward compensation in the ptz platform. IOP Conf. Ser. Mater. Sci. Eng. (2018)
Żabiński, T., Grygiel, T., Kwolek, B.: Design and implementation of visual feedback for an active tracking. Mach. Graph. Vis. 15(3–4), 681–690 (2006)
Funding
The project was funded by POB Research Centre for Artificial Intelligence and Robotics of Warsaw University of Technology within the Excellence Initiative Program – Research University (ID-UB).
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by RN, KZ, DS, PC and MF. The first draft of the manuscript was written by PDD and CZ and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no relevant financial or nonfinancial interests to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Nebeluk, R., Zarzycki, K., Seredyński, D. et al. Predictive tracking of an object by a pan–tilt camera of a robot. Nonlinear Dyn 111, 8383–8395 (2023). https://doi.org/10.1007/s11071-023-08295-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11071-023-08295-z