Mission evaluation: expert evaluation system for large-scale combat tasks of the weapon system of systems

  • Jianfei Ding
  • Guangya SiEmail author
  • Jun Ma
  • Yanzheng Wang
  • Zhe Wang
Research Paper


Mission evaluation is a new requirement for capability evaluation of the weapon system of systems (WSOS) in the era of big data, and is based on evaluating large-scale tasks with similar attributes. The use of traditional methods by military experts to evaluate large scale tasks incurs significant time cost and results in low accuracy, and is caused by a variety of factors that cause confusion. Therefore, we developed a system to assist military personnel in improving the efficiency of mission evaluation; the main innovations of our work include the qualitative and quantitative visualization of complex information is realized in a three-pane interface. We also realize the iterative and interactive evaluation modes of large-scale tasks by using the active learning method; moreover, the overall display of large-scale task evaluation results is realized using statistical graphics. In practical application, the system not only improves the users’ efficiency and accuracy scores, but also helps to achieve the recognition evaluation for the overall scoring results.


mission evaluation large-scale tasks weapon system of systems (WSOS) visual interactive active learning 



This work was supported by Major Program of the National Natural Science Foundation of China (Grant No. U1435218) and National Natural Science Foundation of China (Grant No. 61403401).


  1. 1.
    Qian X S, Yu J Y, Dai R W. A new discipline of science—the study of open complex giant system and its methodology. J Syst Eng Electron, 1993, 4: 2–12Google Scholar
  2. 2.
    Dai R W, Li Y D. Researches on hall for workshop of meta synthetic engineering and system complexity. Complex Syst Complex Sci, 2004, 1: 1–24Google Scholar
  3. 3.
    Liu C M, Dai R W. Visualization of experts evaluation opinions in the hall for workshop of meta synthetic engineering. Pattern Recog Artif Intell, 2005, 18: 6–11Google Scholar
  4. 4.
    Dai R W. The proposal and recent development of metasynthetic method(M) from qualitative to quantitative. Chinese J Nature, 2009, 31: 311–314Google Scholar
  5. 5.
    Hu X F, Si G Y. SDE98: an environment of metasynthetic workshop for military strategic decisionmaking. Mini-Micro Syst, 1999, 20: 2–7Google Scholar
  6. 6.
    Si G Y. Strategic decision integration discussion and research and implemenation of the simulation circumstance. Syst Eng, 2000, 18: 79–80Google Scholar
  7. 7.
    Zhang X X, Zhang P Z. Research on visualization of group decision argument opinion’s distributing—design and development of electronic common brain audiovisual room. J Manage Sci China, 2005, 8: 15–27Google Scholar
  8. 8.
    Li J, Zhang P Z, Jiang Y Z. Research on automatic topic visual clustering in the group argument support systems. J Syst Manage, 2009, 18: 325–331Google Scholar
  9. 9.
    Xiong C Q, Li D H, Zhang Y. Clustering analysis of experts’ opinion and its visualization in hall for workshop of meta-synthetic engineering. Pattern Recogn Artif Intell, 2009, 22: 017Google Scholar
  10. 10.
    Franks A, Miller A, Bornn L, et al. Counterpoints: advanced defensive metrics for NBA basketball. In: Proceedings of the 9th Annual MIT Sloan Sports Analytics Conference, Boston, 2015Google Scholar
  11. 11.
    Maheswaran R, Chang Y H, Henehan A, et al. Deconstructing the rebound with optical tracking data. In: Proceedings of MIT Sloan Sports Analytics Conference, Boston, 2012Google Scholar
  12. 12.
    Lucey P, Bialkowski A, Monfort M, et al. “Quality vs Quantity”: improved shot prediction in soccer using strategic features from spatiotemporal data. In: Proceedings of the 8th Annual MIT Sloan Sports Analytics Conference, Boston, 2014. 1–9Google Scholar
  13. 13.
    Rusu A, Stoica D, Burns E. Analyzing soccer goalkeeper performance using a metaphor-based visualization. In: Proceedings of the 15th International Conference on Information Visualization. Washington: IEEE Computer Society, 2011. 194–199Google Scholar
  14. 14.
    Rusu A, Stoica D, Burns E, et al. Dynamic visualizations for soccer statistical analysis. In: Proceedings of the 14th International Conference on Information Visualization. Washington: IEEE Computer Society, 2010. 207–212Google Scholar
  15. 15.
    Pileggi H, Stolper C D, Boyle J M, et al. Snapshot: visualization to propel ice hockey analytics. IEEE Trans Visual Comput Graph, 2012, 18: 2819–2828CrossRefGoogle Scholar
  16. 16.
    Perin C, Vuillemot R, Fekete J D. Soccer Stories: a kick-off for visual soccer analysis. IEEE Trans Visual Comput Graph, 2013, 19: 2506–2515CrossRefGoogle Scholar
  17. 17.
    Janetzko H, Sacha D, Stein M, et al. Feature-driven visual analytics of soccer data. In: Proceedings of IEEE Conference on Visual Analytics Science and Technology. Washington: IEEE Computer Society, 2014. 13–22Google Scholar
  18. 18.
    Angluin D. Queries and concept learning. Mach Learn, 1988, 2: 319–342MathSciNetGoogle Scholar
  19. 19.
    Tomanek K, Olsson F. A web survey on the use of active learning to support annotation of text data. In: Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, Boulder, 2009. 45–48Google Scholar
  20. 20.
    Settles B. Active Learning Literature Survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison, 2009Google Scholar
  21. 21.
    Dasgupta S, Langford J. A tutorial on active learning. In: Proceedings of International Conference of Machine Learning, Quebec, 2009Google Scholar
  22. 22.
    Macskassy S A. Using graph-based metrics with empirical risk minimization to speedup active learning on networked data. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, 2009. 597–606Google Scholar
  23. 23.
    Bilgic M, Mihalkova L, Getoor L. Active learning for networked data. In: Proceedings of the 27th International Conference on Machine Learning, Haifa, 2010. 79–86Google Scholar
  24. 24.
    Shi L, Zhao Y, Tang J. Batch mode active learning for networked data. ACM Trans Intell Syst Tech, 2012, 3: 33CrossRefGoogle Scholar
  25. 25.
    Hu X, Tang J, Gao H, et al. ActNet: active learning for networked teats inmicro-blogging. In: Proceedings of the 2013 SIAM International Conference on Data Mining, Austin, 2013. 306–314Google Scholar
  26. 26.
    Cesa-Bianchi N, Gentile C, Vitale F, et al. Active learning on trees and graphs. arXiv:1301.5112, 2013zbMATHGoogle Scholar
  27. 27.
    Chen Y, Li Z, Nie L, et al. A semi-supervised network model for micro blog topic classification. In: Proceedings of the 24th International Conference on Computational Linguistics, Mumbai, 2012. 561–576Google Scholar
  28. 28.
    Liu F C. Research on entity relation extraction technology based on text. National Defense Science and Technology University, 2013Google Scholar
  29. 29.
    Tong S, Chang E. Support vector machine active learning for image retrieval. In: Proceedings of the 9th ACM International Conference on Multimedia, Ottawa, 2001. 107–118Google Scholar
  30. 30.
    Hu X, Tang L, Tang J, et al. Exploiting social relations for sentiment analysis in micro blogging. In: Proceedings of the 6th ACM International Conference on Web Search and Data Mining, Rome, 2013. 537–546Google Scholar
  31. 31.
    Yan R, Yang J, Hauptmann A. Automatically labeling video data using multi-class active learning. In: Proceedings of 9th IEEE International Conference on Computer Vision, Nice, 2003. 516–523Google Scholar
  32. 32.
    Zhu X, Zhang P, Lin X, et al. Active learning from stream data using optimal weight classifier ensemble. IEEE Trans Syst Man Cybernet, 2010, 40: 1607–1621CrossRefGoogle Scholar
  33. 33.
    Zhang C, Chen T. An active learning framework for content-based information. IEEE Trans Multimed, 2002, 4: 260–268CrossRefGoogle Scholar
  34. 34.
    PraBni J S, Ropinski T, Hinrichs K. Uncertainty—aware guided volume segmentation. IEEE Trans Vis Comput Graph, 2010, 16: 1358–1365CrossRefGoogle Scholar
  35. 35.
    Top A, Hamarneh G, Abugharbieh R. Active Learning for Interactive 3D Image Segmentation Berlin: Springer, 2011. 603–610Google Scholar
  36. 36.
    Gao L, Cao Y P, Lai Y K, et al. Active exploration of large 3d model repositories. IEEE Trans Visual Comput Graph, 2015, 21: 1390–1402CrossRefGoogle Scholar
  37. 37.
    Cohen-Or D, Zhang H. From inspired modeling to creative modeling. Visual Comput, 2016, 1: 7–14CrossRefGoogle Scholar
  38. 38.
    Hofmann T, Puzicha J. Latent class models for collaborative filtering. In: Proceedings of the 16th International Joint Conference on Artificial Intelligence. San Francisco: Morgan Kaufmann Publishers Inc, 1999. 99: 688–693Google Scholar
  39. 39.
    Jin R, Si L. A Bayesian approach toward active learning for collaborative filtering. In: Proceedings of the 20th Conference on Uncertainty in Aartificial Intelligence, Banff, 2004. 278–285Google Scholar
  40. 40.
    Harple A S, Yang Y. Personalized active learning for collaborative filtering. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Singapore, 2008. 91–98Google Scholar
  41. 41.
    Boyd J R. A discourse on winning and losing. Maxwell Air Force Base: Air University Library, Document No. M-U 43947 (Briefing slides)Google Scholar
  42. 42.
    Nadler B, Lafon S, Coifman R R, et al. Diffusion maps, spectral clustering and eigen functions of Fokker-Planck operators. In: Proceedings of Neural Information Processing Systems, British Columbia, 2005. 955–962Google Scholar
  43. 43.
    Zhu X J, Lafferty J, Ghahramani Z. Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions. In: Proceedings of International Conference on Machine Learning Workshop, Washington, 2003. 58–65Google Scholar
  44. 44.
    Joshi A J, Porikli F, Papanikolopoulos N. Multi-class active learning for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, 2009. 2372–2379Google Scholar

Copyright information

© Science China Press and Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  • Jianfei Ding
    • 1
    • 2
  • Guangya Si
    • 1
    Email author
  • Jun Ma
    • 1
  • Yanzheng Wang
    • 1
  • Zhe Wang
    • 3
  1. 1.Department of Information Operation and Command TrainingNational Defence University of PLABeijingChina
  2. 2.Graduate SchoolNational Defence University of PLABeijingChina
  3. 3.Department of PhysiologyHebei Medical UniversityShijiazhuangChina

Personalised recommendations