Human machine collaborative support scheduling system of intelligence information from multiple unmanned aerial vehicles based on eye tracker

  • Lixuan Jian (简立轩)
  • Dong Yin (尹 栋)
  • Lincheng Shen (沈林成)
  • Yifeng Niu (牛轶峰)


Many human-machine collaborative support scheduling systems are used to aid human decision making by providing several optimal scheduling algorithms that do not take operator’s attention into consideration. However, the current systems should take advantage of the operator’s attention to obtain the optimal solution. In this paper, we innovatively propose a human-machine collaborative support scheduling system of intelligence information from multi-UAVs based on eye-tracker. Firstly, the target recognition algorithm is applied to the images from the multiple unmanned aerial vehicles (multi-UAVs) to recognize the targets in the images. Then, the support system utilizes the eye tracker to gain the eye-gaze points which are intended to obtain the focused targets in the images. Finally, the heuristic scheduling algorithms take both the attributes of targets and the operator’s attention into consideration to obtain the sequence of the images. As the processing time of the images collected by the multi-UAVs is uncertain, however the upper bounds and lower bounds of the processing time are known before. So the processing time of the images is modeled by the interval processing time. The objective of the scheduling problem is to minimize mean weighted completion time. This paper proposes some new polynomial time heuristic scheduling algorithms which firstly schedule the images including the focused targets. We conduct the scheduling experiments under six different distributions. The results indicate that the proposed algorithm is not sensitive to the different distributions of the processing time and has a negligible computational time. The absolute error of the best performing heuristic solution is only about 1%. Then, we incorporate the best performing heuristic algorithm into the human-machine collaborative support systems to verify the performance of the system.


eye tracker polynomial time heuristics human machine interaction collaborative support scheduling system receding horizon optimization policy 

CLC number

TP 391.4 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    CUMMINGS M L, BRUNI S, MERCIER S, et al. Automation architecture for single operator multiple UAV command and control [J]. International Command and Control Journal, 2007, 1(2): 1–24.Google Scholar
  2. [2]
    JIAN L X, YIN D, SHEN LC, et al. Optimal attention allocation to visual search tasks of multi-UAVs based on operator model [C]//Proceedings of 2015 IEEE International Conference on Mechatronics and Automation. Beijing: IEEE, 2015: 1767–1771.CrossRefGoogle Scholar
  3. [3]
    SRIVASTAVA V, SURANA A, BULLO F. Adaptive attention allocation in human-robot systems [C]//merican Control Conference. Montréal: AACC, 2012: 2267–2774.Google Scholar
  4. [4]
    VERMA A, RAI R. Modeling multi operator-multi-UAV (MOMU) operator attention allocation problem [C]//Proceedings of 2013 International Mechanical Engineering Congress and Exposition. San Diego, California: ASME, 2013: 1–6.Google Scholar
  5. [5]
    BERTUCCELLI L F, BECKERS N W M, CUMMINGS M L. Developing operator models for UAV search scheduling [C]//In AIAA Conference on Guidance, Navigation and Control. Toronto: AIAA, 2010: 1–15.Google Scholar
  6. [6]
    BERTUCCELLI L F, CUMMINGS M L. Scenariobased robust scheduling for collaborative human-UAV visual search tasks [C]//Proceedings of 2011 50th IEEE Conference on Decision and Control and European Control Conference. Orlando, Florida: IEEE, 2011: 5702–5707.Google Scholar
  7. [7]
    BAKER K, TRIETSCH D. Principles of sequencing and scheduling [M]. New York: Wiley-Blackwell, 2009.CrossRefMATHGoogle Scholar
  8. [8]
    IRANPOOR M, FATEMI GHOMI S M T, ZANDIEH M. Due-date assignment and machine scheduling in a low machine-rate situation with stochastic processing times [J]. Computers & Operations Research, 2013, 40: 1100–1108.MathSciNetCrossRefMATHGoogle Scholar
  9. [9]
    SOTSKOV Y N, EGOROVA N G, LAI T C. Minimizing total weighted flow time of a set of jobs with interval processing times [J]. Mathematical and Computer Modelling, 2009, 50: 556–573.MathSciNetCrossRefMATHGoogle Scholar
  10. [10]
    PINEDO M L. Scheduling: Theory. Algorithms, and systems [M]. New York: Springer, 2012.CrossRefMATHGoogle Scholar
  11. [11]
    ALLAHVERDI A, AYDILEK H, AYDILEK A. Single machine scheduling problem with interval processing times to minimize mean weighted completion time [J]. Computers & Operations Research, 2014, 51: 200–207.MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Shanghai Jiaotong University and Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  • Lixuan Jian (简立轩)
    • 1
  • Dong Yin (尹 栋)
    • 1
  • Lincheng Shen (沈林成)
    • 1
  • Yifeng Niu (牛轶峰)
    • 1
  1. 1.College of Mechatronic Engineering and AutomationNational University of Defense TechnologyChangshaChina

Personalised recommendations