Journal of Intelligent & Robotic Systems

, Volume 74, Issue 3–4, pp 1029–1047 | Cite as

Vision Based Neuro-Fuzzy Controller for a Two Axes Gimbal System with Small UAV

  • Ashraf QadirEmail author
  • William Semke
  • Jeremiah Neubert


This paper presents the development of a vision-based neuro-fuzzy controller for a two axes gimbal system mounted on a small Unmanned Aerial Vehicle (UAV). The controller uses vision-based object detection as input and generates pan and tilt motion and velocity commands for the gimbal in order to keep the interest object at the center of the image frame. A readial basis function based neuro-fuzzy system and a learning algorithm is developed for the controller to address the dynamic and non-linear characteristics of the gimbal movement. The controller uses two separate, but identical radial basis function networks, one for pan and one for tilt motion of the gimbal. Each system is initialized with a fixed number of neurons that act as rules basis for the fuzzy inference system. The membership functions and rule strengths are then adjusted with the feedback from the visual tracking system. The controller is trained off-line until a desired error level is achieved. Training is then continued on-line to allow the system to accommodate air speed changes. The algorithm learns from the error computed from the detected position of the object in image frame and generates position and velocity commands for the gimbal movement. Several tests including lab tests and actual flight tests of the UAV have been carried out to demonstrate the effectiveness of the controller. Test results show that the controller is able to converge effectively and generate accurate position and velocity commands to keep the object at the center of the image frame.


Unmanned Aerial Vehicle (UAV) Zero Mean Normalized Cross Correlation (ZMNCC) Neuro-Fuzzy system Radial Basis Function Network (RBFN) 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Kumar, R., Samarasekera, S., Hsu, S., Hanna, K.: Registration of highly-oblique and zoomed in aerial video to reference imagery. In: Proceedings of IEEE Computer Society Computer Vision and Pattern Recognition Conference. Barcelona, Spain (2000)Google Scholar
  2. 2.
    Kumar, R., Sawhney, H.S., Asmuth, J.C., Pope, A., Hsu, S.: Registration of video to geo-referenced imagery. In: Proceedings of the 14th International Conference on Pattern Recognition, vol. 2, pp. 1393–1400. Brisbane, Australia (1998)Google Scholar
  3. 3.
    Stolle, S., Rysdyk, R.: Flight path following guidance for unmanned air vehicles with pan-tilt camera for target observation. In: 22nd Digital Avionics Systems Conference, Indianapolis (2003)Google Scholar
  4. 4.
    Dobrokhodov, V.N., Kaminer, I.I., Jones, K.D., Ghabcheloo, R.: Vision-based tracking and motion estimation for moving target using small UAVs. In: Proceedings of 2006 American Control Conference, Minneapolis, 14–16 June 2006Google Scholar
  5. 5.
    Barber, D.B., Redding, J.D., McLain, T.W., Beard, R.W., Taylor, C.N.: Vision-based target geo-location using a fixed-wing miniature air vehicle. J. Intell. Robot. Syst. 47, 361–382 (2006)CrossRefGoogle Scholar
  6. 6.
    Redding, J.D.: Vision based target localization from a small fixed-wing unmanned air vehicle. Master’s thesis, Brigham Young University, Provo, Utah 84602 (2005)Google Scholar
  7. 7.
    Qadir, A., Neubert, J., Semke, W.: On-board visual tracking with small unmanned aircraft systems. In: AIAA Infotech@Aerospace conference, St. Louis, MO, 28–31 March 2011Google Scholar
  8. 8.
    Jang, J.S.R.: ANFIS: adaptive-network-based fuzzy inference system. IEEE Trans. Syst. Man Cybern. 23, 665–684 (1993)CrossRefGoogle Scholar
  9. 9.
    Takagi, T., Sugeno, M.: Fuzzy identification of systems and its applications to modeling and control. IEEE Trans. Syst. Man Cybern. 15, 116–132 (1985)CrossRefzbMATHGoogle Scholar
  10. 10.
    Leng, G., McGinnity, T.M., Prasad, G.: An approach for on-line extraction of fuzzy rules using a self-organizing fuzzy neural network. Fuzzy Sets and Systems 150, 211–243 (2005)CrossRefzbMATHMathSciNetGoogle Scholar
  11. 11.
    Li, W., Hori, Y.: An algorithm for extracting fuzzy rules based on RBF neural network. IEEE Trans. Ind. Electron. 53(4), 1269–1276 (2006)CrossRefGoogle Scholar
  12. 12.
    Jang, J.S.R., Sun, C.T.: Functional equivalence between radial basis function networks and fuzzy inference systems. IEEE Trans. Neural Netw. 4, 156–159 (1993)CrossRefGoogle Scholar
  13. 13.
    Cho, K.B., Wang, B.H.: Radial basis function based adaptive fuzzy systems and their applications to system identification and prediction. Fuzzy Sets and Systems 83, 325–339 (1996)CrossRefMathSciNetGoogle Scholar
  14. 14.
    Diao, Y., Passino, K.M.: Adaptive neural/fuzzy control for interpolated nonlinear systems. IEEE Trans. Fuzzy Syst. 10, 583–595 (2002)CrossRefGoogle Scholar
  15. 15.
    Chen, B.S., Lee, C.H., Chang, Y.C.: Tracking design of uncertain nonlinear siso systems: adaptive fuzzy approach. IEEE Trans. Fuzzy Syst. 4, 32–43 (1996)CrossRefGoogle Scholar
  16. 16.
    Spooner, J.T., Passino, K.M.: Stable adaptive control using fuzzy systems and neural networks. IEEE Trans. Fuzzy Syst. 4, 339–359 (1996)CrossRefGoogle Scholar
  17. 17.
    Lee, C., Teng, C.: Identification and control of dynamic systems using recurrent fuzzy neural networks. IEEE Trans. Fuzzy Syst. 8, 349–366 (2000)CrossRefGoogle Scholar
  18. 18.
    Narendra, K.S., Parthasarathy, K.: Identification and control of dynamical systems using neural networks. IEEE Trans. Neural Netw. 1, 4–27 (1990)CrossRefGoogle Scholar
  19. 19.
    Polycarpou, M.M., Mears, M.J.: Stable adaptive tracking of uncertain systems using nonlinearly parameterized online approximators. Int. J. Control 70(3), 363–384 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  20. 20.
    Hsiao, F.-H., Xu, S.-D., Lin, C.-Y., Tsai, Z.-R.: Robustness design of fuzzy control for nonlinear multiple time-delay large-scale systems via neuralnetwork-based approach. IEEE Trans. Syst. Man Cybern. B Cybern. 38(1), 244–251 (2008)CrossRefGoogle Scholar
  21. 21.
    Lewis, F.L., Jagannathan, S., Yeildirek, A.: Neural Network Control of Robot Manipulators and Nonlinear Systems. Taylor and Francis, London (1999)Google Scholar
  22. 22.
    Golea, N., Golea, A., Benmahammed, K.: Stable indirect fuzzy adaptive control. Fuzzy Sets and Systems 13(7), 353–366 (2003)CrossRefMathSciNetGoogle Scholar
  23. 23.
    Wang, T., Lina, C., Liub, H.: Observer-based indirect adaptive fuzzy-neural tracking control for nonlinear SISO systems using VSS and H approaches. Fuzzy Sets and Systems 14(3), 211–232 (2004)Google Scholar
  24. 24.
    Labiod, S., Boucherit, M.S., Guerra, T.M.: Adaptive fuzzy control of a class of MIMO nonlinear systems. Fuzzy Sets and Systems 15(1), 59–77 (2005)CrossRefMathSciNetGoogle Scholar
  25. 25.
    Rovithakis, G.A., Christodoulou, M.A.: Adaptive control of unknown plants using dynamical neural networks. IEEE Trans. Syst. Man Cybern. 24, 400–412 (1994)CrossRefMathSciNetGoogle Scholar
  26. 26.
    Chen, F.C., Liu, C.C.: Adaptively controlling nonlinear continuous-time systems using multilayer neural netoworks. IEEE Trans. Automat. Control 39, 1306–1310 (1994)CrossRefzbMATHMathSciNetGoogle Scholar
  27. 27.
    Rovithakis, G.A., Christodoulou, M.A.: Direct adaptive regulation of unknown nonlinear dynamical systems via dynamic neural networks. IEEE Trans. Syst. Man Cybern. 25, 1578–1594 (1995)CrossRefGoogle Scholar
  28. 28.
    Wang, C., Liu, H., Lin, T.: Direct adaptive fuzzyneural control with state observer and supervisory controller for unknown nonlinear dynamical systems. IEEE Trans. Fuzzy Syst. 10, 39–49 (2002)CrossRefGoogle Scholar
  29. 29.
    Leu, Y., Wang, W., Lee, T.: Observer-based direct adaptive fuzzy-neural control for nonaffine nonlinear systems. IEEE Trans. Neural Netw. 16, 853–861 (2005)CrossRefGoogle Scholar
  30. 30.
    Phan, P., Gale, T.J.: Direct adaptive fuzzy control with a self-structuring algorithm. Fuzzy Sets and Systems 159, 871–899 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  31. 31.
    Christodoulou, M.A., Theodoridis, D.C., Boutalis, Y.S.: Building optimal fuzzy dynamical systems description based on recurrent neural network approximation. In: Proc. Int. Conf. of Networked Distributed Systems for Intelligent Sensing and Control. Kalamata, Greece (2007)Google Scholar
  32. 32.
    Boutalis, Y.S., Theodoridis, D.C., Christodoulou, M.A.: A new neuro FDS definition for indirect adaptive control of unknown nonlinear systems using a method of parameter hopping. IEEE Trans. Neural Netw. 20(4), 609–625 (2009)CrossRefGoogle Scholar
  33. 33.
    Theodoridis, D.C, Boutalis, Y.S., Christodoulou, M.A.: Indirect adaptive control of unknown multi variable nonlinear systems with parametric and dynamic uncertainties using a new neuro-fuzzy system description. Int. J. Neural. Syst. 20(2), 129–148 (2010)CrossRefGoogle Scholar
  34. 34.
    Cong, S., Liang, Y.: PID-like neural network nonlinear adaptive control for uncertain multivariable motion control system. IEEE Trans. Ind. Electron. 56(10), 3872–3879 (2009)CrossRefGoogle Scholar
  35. 35.
    Mbede, J.B., Huang, X., Wang, M.: Robust neural-fuzzy sensor based motion control among dynamic obstacles for robot manipulators. IEEE Trans. Fuzzy Syst. 11, 249–260 (2003)CrossRefGoogle Scholar
  36. 36.
    Patino, H.D., Carelli, R., Kuchen, B.R.: Neural networks for advanced control of robot manipulators. IEEE Trans. Neural Netw. 13, 343–354 (2002)CrossRefGoogle Scholar
  37. 37.
    Kelly, W., et al.: Neuro-fuzzy control of a robotic arm. In: Proceedings of the Artificial Neural Networks in Engineering Conference, pp. 837–842. St. Louis, MO, 10–13 Nov 1996Google Scholar
  38. 38.
    Beom, H.R., Cho, H.S.: A sensor-based navigation for a mobile robot using fuzzy-logic and reinforcement learning. IEEE Trans. Syst. Man Cybern. 25(3), 464–477 (1995)CrossRefGoogle Scholar
  39. 39.
    Lee, C.-H., Chiu, M.-H.: Recurret neuro fuzzy control design for tracking of mobile robots via hybrid algorithm. Expert. Syst. Appl. 36, 8993–8999 (2009)CrossRefGoogle Scholar
  40. 40.
    Rusu, P., Petriu, E.M., Whalen, T.E., Crnell, A., Spoelder, H.J.W.: Behavior-based neuro-fuzzy controller for mobile robot navigation. IEEE Trans. Instrum. Meas. 52(4), 1335–1340 (2003)CrossRefGoogle Scholar
  41. 41.
    Martins, F.N., Celeste, W.C., Carelli, R., Sarcinelli-Finho, M., Bastos-Filho, T.F.: An adaptive dynamic controller for autonomous mobile robot trajectory tracking. Control Eng. Pract. 16, 1354–1363 (2008)CrossRefGoogle Scholar
  42. 42.
    Wang, J.S., Lee, C.S.G.: Self-adaptive recurrent neuro-fuzzy control of an autonomous underwater vehicle. IEEE Trans. Rob. Autom. 19(2), 283–295 (2003)CrossRefzbMATHGoogle Scholar
  43. 43.
    Xu, B., Pandian, S.R., Sakagami, N., Petry, F.: Neuro-fuzzy control of underwater vehicle-manipulator systems. J. Franklin Inst. 349, 1125–1138 (2012)CrossRefzbMATHMathSciNetGoogle Scholar
  44. 44.
    Suresh, S., Kannan, N., Sundararajan, N., Saratchandran, P.: Neural adaptive control for vibration suppression in composite fin-tip of aircraft. Int. J. Neural. Syst. 18(3), 219–231 (2008)CrossRefGoogle Scholar
  45. 45.
    Collani, Y.V., Zhang, J., Knoll, A.: A neuo-fuzzy solution for fine-motion control based on vision and force sensors. Technical Report, Department of Technology, University of Bielefeld (1997)Google Scholar
  46. 46.
    Lin, F.-J., Chou, P.-H., Shieh, P.-H., Chen, S.-Y.: Robust control of an LUSM-based X – Y – θ motion control stage using an adaptive interval type-2 fuzzy neural network. IEEE Trans. Fuzzy Syst. 17(1), 24–38 (2009)CrossRefGoogle Scholar
  47. 47.
    Wu, C.S., Gao, J.Q.: Vision-based neuro-fuzzy control of weld penetration in gas tungsten arc welding of thin sheets. Int. J. Model. Ident. Control 1(2) (2006)Google Scholar
  48. 48.
    Lewis, J.P.: Fast normalized cross-correlation. In: Proceedings of Vision Interface, pp. 120–123 (1995)Google Scholar
  49. 49.
    Welch, G., Bishop, G.: An introduction to the Kalman Filter. In: Dept. Comp. Sci., Univ. North Carolina, Chapel Hill, TR95-041 (2000)Google Scholar
  50. 50.
    Ranganathan, J., Semke, W.: Three-axis gimbal surveillance algorithms for use in small UAS. In: Proceedings of the ASME International Mechanical Engineering Conference and Exposition, IMECE2008-67667 (2008)Google Scholar
  51. 51.
    Semke W., Ranganathan, J., Buisker, M.: Active gimbal control for surveillance using small unmanned aircraft systems. In: Proceedings of the International Model analysis Conference (IMAC) XXVI: A Conference and Exposition on Structural Dynamics (2008)Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  1. 1.Mechanical Engineering DepartmentUniversity of North DakotaGrand ForksUSA

Personalised recommendations