Multimedia Tools and Applications

, Volume 75, Issue 24, pp 17393–17419 | Cite as

Dynamic node selection in camera networks based on approximate reinforcement learning

  • Qian Li
  • Zhengxing Sun
  • Songle Chen
  • Shiming Xia


In camera networks, dynamic node selection is an effective technique that enables video stream transmission with constrained network bandwidth, more economical node cooperation for nodes with constrained power supplies, and optimal use of a limited number of display terminals, particularly for applications that need to obtain high-quality video of specific targets. However, the nearest camera in a network cannot be identified by directional measurements alone. Furthermore, errors are introduced into computer vision algorithms by complex background, illumination, and other factors, causing unstable and jittery processing results. Consequently, in selecting camera network nodes, two issues must be addressed: First, a dynamic selection mechanism that can choose the most appropriate node is needed. Second, metrics to evaluate the visual information in a video stream must be modeled and adapted to various camera parameters, backgrounds, and scenes. This paper proposes a node selection method based on approximate reinforcement learning in which nodes are selected to obtain the maximum expected reward using approximate Q-learning. The Q-function is approximated by a Gaussian Mixture Model with parameters that are sequentially updated by a mini-batch stepwise Expectation–Maximization algorithm. To determine the most informative camera node dynamically, the immediate reward in Q-learning integrates the visibility, orientation, and image clarity of the object in view. Experimental results show that the proposed visual evaluation metrics can effectively capture the motion state of objects, and that the selection method reduces camera switching and related errors compared with state-of-the art methods.


Camera selection Approximate reinforcement learning Gaussian mixture model (GMM) Video analysis Camera networks 



This work is supported by the National Natural Science Foundation of China (61272219, 61100110, 41305138, 61473310 and 61321491), Program for New Century Excellent Talents of Ministry of Education of China (NCET-04-0460), Science and Technology Plan of Jiangsu Province (BE2010072, BE2011058, BY2012190, and BY2013072-04), and Innovation Foundation of State Key Lab for Novel Software Technology of China (ZZKT2013A12).


  1. 1.
    Agarwal A, Triggs B (2006) Recovering 3D human pose from monocular images. IEEE Trans Patt Anal Mach Intell 28(1):44–58. doi: 10.1109/TPAMI.2006.21 CrossRefGoogle Scholar
  2. 2.
    Agostini A, Celaya E (2010) Reinforcement learning with a Gaussian mixture model. In Neural Networks (IJCNN), The 2010 Int Joint Conf : 1–8. doi:  10.1109/IJCNN.2010.5596306
  3. 3.
    Botvinick M, Niv Y, Barto A (2009) Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 113(3):262–280. doi: 10.1016/j.cognition.2008.08.011 CrossRefGoogle Scholar
  4. 4.
    Busoniu L, Babuska R, DeSchutter B, Ernst D (2010) Reinforcement learning and dynamic programming using function approximators. CRC press. doi:  10.1201/9781439821091
  5. 5.
    Charfi Y, Wakamiya N, Murata M (2009) Challenging issues in visual sensor networks. IEEE Wirel Commun 16(2):44. doi: 10.1109/MWC.2009.4907559 CrossRefGoogle Scholar
  6. 6.
    Cohn D, Ghahramani Z, Jordan M (1996) Active learning with statistical models. J Artif Intell Res 4(1):129–145Google Scholar
  7. 7.
    Daniyal F, Taj M, Cavallaro A (2010) Content and task-based view selection from multiple video streams. Multimed Tools Applic 46:235–258. doi: 10.1007/s11042-009-0355-z CrossRefGoogle Scholar
  8. 8.
    Fu Y, Guo Y, Zhu Y (2010) Multi-view video summarization. IEEE Trans Multimed 12(7):717–729. doi: 10.1109/TMM.2010.2052025 CrossRefGoogle Scholar
  9. 9.
    Gupta A, Mittal A, Davis L (2007) Cost: An approach for camera selection and multi-object inference ordering in dynamic scenes. Proceedings of IEEE International Conference on Computer Vision, Rio de Janeiro: 1–8. doi: 10.1109/ICCV.2007.4408842
  10. 10.
    Han B, Joo S, Davis L (2011) Multi-camera tracking with adaptive resource allocation. Int J Comput Vis 91(1):45–58. doi: 10.1007/s11263-010-0373-3
  11. 11.
    Huber M (2012) Optimal pruning for multi-step sensor scheduling. IEEE Trans Autom Control 57(5):1338–1343. doi: 10.1109/TAC.2011.2175070 MathSciNetCrossRefGoogle Scholar
  12. 12.
    Javed O, Khan S, Rasheed Z (2000) Camera handoff: tracking in multiple uncalibrated stationary cameras. Proc Workshop Human Motion, IEEE: 113–118. doi:  10.1109/HUMO.2000.897380
  13. 13.
    Jiang H, Fels S, Little J (2008) Optimizing multiple object tracking and best view video synthesis. IEEE Trans Multimed 10(6):997–1012. doi: 10.1109/TMM.2008.2001379 CrossRefGoogle Scholar
  14. 14.
    Kveton B, Theocharous G (2012) Kernel-based reinforcement learning on representative states. Association for the Advancement of Artificial Intelligence, pp 977–983Google Scholar
  15. 15.
    Kwon J, Lee K (2010) Visual tracking decomposition. IEEE Conf Comput Vision Patt Recognit: 1269–1276. doi:  10.1109/CVPR.2010.5539821
  16. 16.
    Li Y, Bhanu B (2011) Utility-based camera assignment in a video network: a game theoretic framework. IEEE Sensors J 11(3):676–687. doi: 10.1109/JSEN.2010.2051148 CrossRefGoogle Scholar
  17. 17.
    Li Q, Sun Z, Chen S, Liu Y (2013) A method of camera selection based on partially observable Markov decision process model in camera networks. Ame Contrl Conf: 3833–3839. doi: 10.1109/ACC.2013.6580424
  18. 18.
    Li W, Zhang W (2012) Sensor selection for improving accuracy of target localization in wireless visual sensor networks. IET Wireless Sens Syst 2(4):293–301. doi: 10.1049/iet-wss.2012.0033 CrossRefGoogle Scholar
  19. 19.
    Liang P, Klein D (2009) Online EM for unsupervised models. In Proceedings of human language technologies: The 2009 annual conference of the North American chapter of the association for computational linguistics. Assoc Comput Linguist: 611–619. doi:  10.3115/1620754.1620843
  20. 20.
    McLachlan G, Peel D (2004) Finite mixture models. New York: J. Wiley. doi: 10.1002/0471721182
  21. 21.
    Mo Y, Ambrosino R, Sinopoli B (2011) Sensor selection strategies for state estimation in energy constrained wireless sensor networks. Automatica 47(7):1330–1338. doi: 10.1016/j.automatica.2011.02.001 MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Monari E, Kroschel K (2009) A knowledge-based camera selection approach for object tracking in large sensor networks. Third ACM/IEEE Int Conf Distrib Smart Cam. doi: 10.1109/ICDSC.2009.5289400 Google Scholar
  23. 23.
    Park J, Bhat C, Kak A (2006) A look-up table based approach for solving the camera selection problem in large camera networks. In IEEE Workshop on Distributed Smart Cameras, Boulder, CO, USA, Oct. 31, 2006Google Scholar
  24. 24.
    Rudoy D, Zelnik-Manor L (2012) Viewpoint selection for human actions. Int J Comput Vis 97(3):243–254. doi: 10.1007/s11263-011-0484-5 CrossRefGoogle Scholar
  25. 25.
    Rudoy D, Zelnik-Manor L (2012) Viewpoint selection for human actions. Int J Comput Vis 97(3):243–254. doi: 10.1007/s11263-011-0484-5 CrossRefGoogle Scholar
  26. 26.
    Sato M, Ishii S (2000) On-line EM algorithm for the normalized Gaussian network. Neural Comput 12(2):407–432. doi: 10.1162/089976600300015853 CrossRefGoogle Scholar
  27. 27.
    Shen C, Zhang C, Fels S (2007) A multi-camera surveillance system that estimates quality-of-view measurement. Proc IEEE Int Conf Image Process, San Antonio: III 193-III 196. doi: 10.1109/ICIP.2007.4379279
  28. 28.
    Singh S, Jaakkola T, Jordan M (1995) Reinforcement learning with soft state aggregation. Advances in neural information processing systems: 361–368. doi:  10.1162/089976600300015961
  29. 29.
    Soro S, Heinzelman W (2007) Camera selection in visual sensor networks. Proc IEEE Conf Adv Video Signal Based Surveill: 81–86. doi:  10.1109/AVSS.2007.4425290
  30. 30.
    Soro S, Heinzelman W (2009) A survey of visual sensor networks. Adv Multimed: 1–22. doi:  10.1155/2009/640386
  31. 31.
    Spaan M, Lima P (2009) A decision-theoretic approach to dynamic sensor selection in camera networks. In ICAPS: 1–8. doi: ocs/index.php/ICAPS/ICAPS09/paper/viewFile/758/1125Google Scholar
  32. 32.
    Sutton R, Barto A (1998) Reinforcement learning: an introduction. MIT Press. Cambridge, Massachusetts London, England. doi: 10.1109/TNN.1998.712192
  33. 33.
    Tessens L, Morbee M, Lee H, Philips W, Aghajan H (2008) Principal view determination for camera selection in distributed smart camera networks. Sec ACM/IEEE Int Conf Distrib Smart Cam: 1–10. doi: 10.1109/ICDSC.2008.4635699
  34. 34.
    Watkins C, Dayan P (1992) Q-learning. Mach Learn 8(3):279–292. doi: 10.1007/BF00992698 zbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Qian Li
    • 1
    • 2
  • Zhengxing Sun
    • 1
  • Songle Chen
    • 1
  • Shiming Xia
    • 2
  1. 1.State Key Laboratory for Novel Software TechnologyNanjing UniversityNanjingChina
  2. 2.College of Meteorology and OceanographyPLA University of Science and TechnologyNanjingChina

Personalised recommendations