Nonlinear Model Predictive Visual Path Following Control to Autonomous Mobile Robots

  • 241 Accesses

  • 1 Citations


This paper proposes a novel approach to the visual path following problem based on Nonlinear Model Predictive Control. Simplified visual features are extracted from the path to be followed. Then, aiming to calculate the control actions directly from the image plane, a regulatory model is obtained in the optimal control problem scope. For this purpose a Serret-Frenet system is placed in the center of camera’s field of view and the optimal control actions generate velocity references to an inner loop embedded in the robot. Stability issues are handled through a classical method and a new approach based on constraints relaxation is proposed in order to guarantee feasibility. Experimental results with a nonholonomic platform illustrate the performance of the proposed control scheme.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 199

This is the net price. Taxes to be calculated in checkout.


  1. 1.

    Efraim, H., Arogeti, S., Shapiro, A., Weiss, G.: Vision based output feedback control of micro aerial vehicles in indoor environments. J. Intell. Robot. Syst. 87(1), 169–186 (2017)

  2. 2.

    Kucukyildiz, G., Ocak, H., Karakaya, S., Sayli, O.: Design and implementation of a multi sensor based brain computer interface for a robotic wheelchair. J. Intell. Robot. Syst. 87(2), 247–263 (2017)

  3. 3.

    Ji, P., Song, A., Xiong, P., Yi, P., Xu, X., Li, H.: Egocentric-vision based hand posture control system for reconnaissance robots. J. Intell. Robot. Syst. 87(3), 583–599 (2017)

  4. 4.

    Chaumette, F., Hutchinson, S.: Visual servo control. i. Basic approaches. IEEE Robot. Autom. Mag. 13(4), 82–90 (2006)

  5. 5.

    Corke, P.: Robotics, Vision and Control: Fundamental Algorithms in MATLAB. Springer Tracts in Advanced Robotics. Springer, Berlin (2011)

  6. 6.

    Zhao, Y.-M., Xie, W.-F., Liu, S., Wang, T.: Neural network-based image moments for robotic visual servoing. J. Intell. Robot. Syst. 78(2), 239–256 (2015)

  7. 7.

    Araar, O., Aouf, N., Vitanov, I.: Vision based autonomous landing of multirotor uav on moving platform. J. Intell. Robot. Syst. 85(2), 369–384 (2017)

  8. 8.

    Kanellakis, C., Nikolakopoulos, G.: Survey on computer vision for uavs: current developments and trends. J. Intell. Robot. Syst. 87(1), 141–168 (2017)

  9. 9.

    Frezza, R., Soatto, S., Picci, G.: Visual path following by recursive spline updating. In: Proceedings of the 36th IEEE Conference on Decision and Control, 1997, vol. 2, pp. 1130–1134 (1997)

  10. 10.

    Diosi, A., Remazeilles, A., Segvic, S., Chaumette, F.: Outdoor visual path following experiments. In: 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4265–4270 (2007)

  11. 11.

    Delfín, J., Becerra, H.M., Arechavaleta, G.: Visual path following using a sequence of target images and smooth robot velocities for humanoid navigation. In: 2014 14th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp. 354–359 (2014)

  12. 12.

    Bertozzi, M., Broggi, A., Fascioli, A.: Vision-based intelligent vehicles: state of the art and perspectives. Robot. Auton. Syst. 32(1), 1–16 (2000)

  13. 13.

    Cherubini, A., Chaumette, F., Oriolo, G.: An Image-Based visual servoing scheme for following paths with nonholonomic mobile robots. In: International Conference on Control, Automation, Robotics and Vision, ICARCV 2008, pp. 108–113, Hanoi, Vietnam, France (2008)

  14. 14.

    de Lima, D.A., Victorino, A.C.: A visual servoing approach for road lane following with obstacle avoidance. In: 2014 IEEE 17th International Conference on Intelligent Transportation Systems (ITSC), pp. 412–417 (2014)

  15. 15.

    Sabatta, D.: A vision-based error metric for path following control. 2014 PRASA, RobMech and AfLaT International Joint Symposium (PRASA/RobMech/AfLaT 2014) (2014)

  16. 16.

    Mehrez, M.W., Mann, G.K.I., Gosine, R.G.: An optimization based approach for relative localization and relative tracking control in multi-robot systems. J. Intell. Robot. Syst. 85(2), 385–408 (2017)

  17. 17.

    Rybus, T., Seweryn, K., Sasiadek, J.Z.: Control system for free-floating space manipulator based on nonlinear model predictive control (nmpc). J. Intell. Robot. Syst. 85(3), 491–509 (2017)

  18. 18.

    Cao, G., Lai, E.M.-K., Alam, F.: Gaussian process model predictive control of an unmanned quadrotor. J. Intell. Robot. Syst. 88, 147–162 (2017)

  19. 19.

    Masri, M.A., Dbeis, S., Saba, M.A.: Autolanding a power-off uav using on-line optimization and slip maneuvers. J. Intell. Robot. Syst. 86(2), 255–276 (2017)

  20. 20.

    Li, Z., Yang, C., Su, C.Y., Deng, J., Zhang, W.: Vision-based model predictive control for steering of a nonholonomic mobile robot. IEEE Trans. Control Syst. Technol. 24(2), 553–564 (2016)

  21. 21.

    Faulwasser, T., Findeisen, R.: Nonlinear model predictive control for constrained output path following. IEEE Trans. Autom. Control 61(4), 1026–1039 (2016)

  22. 22.

    Płaskonka, J.: Different kinematic path following controllers for a wheeled mobile robot of (2,0) type. J. Intell. Robot. Syst. 77(3), 481–498 (2015)

  23. 23.

    Coulaud, J.B., Campion, G., Bastin, G., De Wan, M.: Stability analysis of a vision-based control design for an autonomous mobile robot. IEEE Trans. Robot. 22(5), 1062–1069 (2006)

  24. 24.

    Mayne, D.Q., Rawlings, J.B., Rao, C.V., Scokaert, P.O.M.: Constrained model predictive control: stability and optimality. Automatica 36(6), 789–814 (2000)

  25. 25.

    Grüne, L., Pannek, J.: Nonlinear Model Predictive Control: Theory and Algorithms. Communications and Control Engineering. 1st edn. Springer, Berlin (2011)

  26. 26.

    Scokaert, P.O.M., Mayne, D.Q., Rawlings, J.B.: Suboptimal model predictive control (feasibility implies stability). IEEE Trans. Autom. Control 44(3), 648–654 (1999)

  27. 27.

    de Oliveira, N., Biegler, L.T.: Constraint handing and stability properties of model-predictive control. AIChE J. 40(7), 1138–1155 (1994)

  28. 28.

    Microsoft. MIcrosoft LifeCam HD-3000 Product Guide. Accessed 18 Aug 2017

  29. 29.

    Park, E.J.: Exploring LEGO Mindstorms EV3: Tools and Techniques for Building and Programming Robots. 1st edn. Wiley, New York (2014)

  30. 30.

    Spellucci, P.: An sqp method for general nonlinear programs using only equality constrained subproblems. In: Mathematical Programming, vol. 82, pp. 413–448 (1998)

Download references

Author information

Correspondence to Tiago T. Ribeiro.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

(MP4 32.3 MB)

(MP4 31.6 MB)

(MP4 31.4 MB)

(MP4 15.9 MB)

(MP4 15.8 MB)

(MP4 15.8 MB)

(MP4 4.14 MB)

(MP4 6.26 MB)

(MP4 9.20 MB)

(MP4 32.3 MB)

(MP4 31.6 MB)

(MP4 31.4 MB)

(MP4 15.9 MB)

(MP4 15.8 MB)

(MP4 15.8 MB)

(MP4 4.14 MB)

(MP4 6.26 MB)

(MP4 9.20 MB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ribeiro, T.T., Conceição, A.G.S. Nonlinear Model Predictive Visual Path Following Control to Autonomous Mobile Robots. J Intell Robot Syst 95, 731–743 (2019) doi:10.1007/s10846-018-0896-3

Download citation


  • Path-following
  • Visual control
  • Nonlinear model predictive control
  • Autonomous robots