Actionable Information in Vision

  • Stefano Soatto
Part of the Studies in Computational Intelligence book series (SCI, volume 411)


A notion of visual information is introduced as the complexity not of the raw images, but of the images after the effects of nuisance factors such as viewpoint and illumination are discounted. It is rooted in ideas of J. J. Gibson, and stands in contrast to traditional information as entropy or coding length of the data regardless of its use, and regardless of the nuisance factors affecting it. The non-invertibility of nuisances such as occlusion and quantization induces an “information gap” that can only be bridged by controlling the data acquisition process. Measuring visual information entails early vision operations, tailored to the structure of the nuisances so as to be “lossless” with respect to visual decision and control tasks (as opposed to data transmission and storage tasks implicit in communications theory). The definition of visual information suggests desirable properties that a visual representation should possess to best accomplish vision-based decision and control tasks.


Mobile Robot Control Task Actionable Information Cast Shadow Data Acquisition Process 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aloimonos, J., Weiss, I., Bandyopadhyay, A.: Active vision. International Journal of Computer Vision 1(4), 333–356 (1988)CrossRefGoogle Scholar
  2. 2.
    Alvarez, L., Guichard, F., Lions, P.L., Morel, J.M.: Axioms and fundamental equations of image processing. Arch. Rational Mechanics 123 (1993)Google Scholar
  3. 3.
    Arbel, T., Ferrie, F.P.: Informative views and sequential recognition. In: Conference on Computer Vision and Pattern Recognition (1995)Google Scholar
  4. 4.
    Arrow, K.J.: Information and economic behavior. In: Federation of Swedish Industries Stockholm, Sweden (1973)Google Scholar
  5. 5.
    Ayvaci, A., Raptis, M., Soatto, S.: Optical flow and occlusion detection with convex optimization. In: Proc. of Neuro Information Processing Systems (NIPS) (December 2010)Google Scholar
  6. 6.
    Ayvaci, A., Soatto, S.: Efficient model selection for detachable object detection. UCLA Technical Report, March 23 (2011), (submitted)Google Scholar
  7. 7.
    Ayvaci, A., Soatto, S.: Detachable object detection. Technical Report UCLA-CSD-100036 (November 19 February 22, 2011) (2010) (submitted)Google Scholar
  8. 8.
    Bajcsy, R.: Active perception. Proceedings of the IEEE 76(8), 966–1005 (1988)CrossRefGoogle Scholar
  9. 9.
    Bajcsy, R., Maver, J.: Occlusions as a guide for planning the next view. IEEE Trans. Pattern Anal. Mach. Intell. 15(5) (May 1993)Google Scholar
  10. 10.
    Batalin, M.A., Sukhatme, G.S.: Efficient exploration without localization. In: Proceedings of IEEE International Conference on Robotics and Automation, 2003. ICRA 2003, vol. 2 (2003)Google Scholar
  11. 11.
    Bernardo, J.M.: Expected information as expected utility. Annals of Stat. 7(3), 686–690 (1979)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Blake, A., Yuille, A.: Active vision. MIT Press Cambridge (1993)Google Scholar
  13. 13.
    Bourgault, F., Makarenko, A.A., Williams, S., Grocholsky, B., Durrant-Whyte, H.: Information based adaptive robotic exploration. In: Proc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), vol. 1 (2002)Google Scholar
  14. 14.
    Brooks, R.: Visual map making for a mobile robot. In: Proceedings of 1985 IEEE International Conference on Robotics and Automation, vol. 2 (1985)Google Scholar
  15. 15.
    Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2005, vol. 2 (2005)Google Scholar
  16. 16.
    Burns, J.B., Weiss, R.S., Riseman, E.M.: The non-existence of general-case view-invariants. In: Geometric Invariance in Computer Vision (1992)Google Scholar
  17. 17.
    Butler, R.B.: The effect of deprivation of visual incentives on visual exploration motivation in monkeys. Journal of Comparative and Physiological Psychology 50(2), 177 (1957)CrossRefGoogle Scholar
  18. 18.
    Candès, E.J., Donoho, D.L.: Curvelets, multiresolution representation, and scaling laws. In: Wavelet Applications in Signal and Image Processing (2000)Google Scholar
  19. 19.
    Caselles, V., Coll, B., Morel, J.-M.: Topographic maps and local contrast changes in natural images. Int. J. Comput. Vision 33(1), 5–27 (1999)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Castro, R., Kalish, C., Nowak, R., Qian, R., Rogers, T., Zhu, X.: Human active learning. In: Proc. of NIPS (2008)Google Scholar
  21. 21.
    Chen, H.F., Belhumeur, P.N., Jacobs, D.W.: In search of illumination invariants. In: Proc. IEEE Conf. on Comp. Vision and Pattern Recogn. (2000)Google Scholar
  22. 22.
    Claxton, K., Neumann, P.J., Araki, S., Weinstein, M.C.: Bayesian value-of-information analysis. International Journal of Technology Assessment in Health Care 17(01), 38–55 (2001)CrossRefGoogle Scholar
  23. 23.
    Cremers, D., Soatto, S.: Motion competition: a variational approach to piecewise parametric motion segmentation. Intl. J. of Comp. Vision, 249–265 (May 2005)Google Scholar
  24. 24.
    Da Cunha, A.L., Do, M.N., Vetterli, M.: On the information rates of the plenoptic function. In: ICIP, Atlanta, GA (2006)Google Scholar
  25. 25.
    Davison, A.J., Murray, D.W.: Simultaneous localization and map-building using active vision. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7), 865–880 (2002)CrossRefGoogle Scholar
  26. 26.
    Denzler, J., Brown, C.M.: Information theoretic sensor data selection for active object recognition and state estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(2), 145–157 (2002)CrossRefGoogle Scholar
  27. 27.
    Fogel, E., Huang, Y.F.: Value of Information in System Identification-Bounded Noise Case. Automatica 18(2), 229–238 (1982)MathSciNetzbMATHCrossRefGoogle Scholar
  28. 28.
    Franz, M.O., Schölkopf, B., Mallot, H.A., Bülthoff, H.H.: Learning view graphs for robot navigation. Autonomous Robots 5(1), 111–125 (1998)CrossRefGoogle Scholar
  29. 29.
    Gibson, J.J.: The theory of information pickup. In: Contemp. Theory and Research in Visual Perception, p. 662 (1968)Google Scholar
  30. 30.
    Gibson, J.J.: The myths of passive perception. Philosophy and Phenomenological Research 37(2), 234–238 (1976)CrossRefGoogle Scholar
  31. 31.
    Gibson, J.J.: The ecological approach to visual perception. LEA (1984)Google Scholar
  32. 32.
    Golubitsky, M., Guillemin, V.: Stable mappings and their singularities. Graduate texts in mathematics, vol. 14 (1974)Google Scholar
  33. 33.
    Good, I.J., Osteyee, D.B.: Information, weight of evidence. The singularity between probability measures and signal detection. Springer, Heidelberg (1974)zbMATHGoogle Scholar
  34. 34.
    Gould, J.P.: Risk, stochastic preference, and the value of information. Journal of Economic Theory 8(1), 64–84 (1974)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Guo, C., Zhu, S., Wu, Y.N.: Toward a mathematical theory of primal sketch and sketchability. In: Proc. 9th Int. Conf. on Computer Vision (2003)Google Scholar
  36. 36.
    Huang, J., Mumford, D.: Statistics of natural images and models. In: Proc. CVPR, pp. 541–547 (1999)Google Scholar
  37. 37.
    Hughes, S.B., Lewis, M.: Task-driven camera operations for robotic exploration. IEEE Transactions on Systems, Man and Cybernetics, Part A 35(4), 513–522 (2005)CrossRefGoogle Scholar
  38. 38.
    Itti, L., Koch, C.: Computational modelling of visual attention. Nature Rev. Neuroscience 2(3), 194–203 (2001)CrossRefGoogle Scholar
  39. 39.
    James, K.: On some possible characteristics of information in J. J. Gibson’s ecological approach to visual perception. Leonardo 13(2) (1980)Google Scholar
  40. 40.
    Jin, H., Soatto, S., Yezzi, A.: Multi-view stereo reconstruction of dense shape and complex appearance. Intl. J. of Comp. Vis. 63(3), 175–189 (2005)CrossRefGoogle Scholar
  41. 41.
    Jojic, N., Frey, B., Kannan, A.: Epitomic analysis of apperance and shape. In: Proc. ICCV (2003)Google Scholar
  42. 42.
    Jones, E., Soatto, S.: Visual-inertial navigation, localization and mapping: A scalable real-time large-scale approach. Intl. J. of Robotics Research (January 17, 2011)Google Scholar
  43. 43.
    Jones, S.D., Andersen, C., Crowley, J.L.: Appearance based processes for visual navigation. In: Processings of the 5th International Symposium on Intelligent Robotic Systems (SIRS 1997), pp. 551–557 (1997)Google Scholar
  44. 44.
    Karatzas, I., Shreve, S.E.: Brownian Motion and Stochastic Calculus. Springer, Heidelberg (1988)zbMATHCrossRefGoogle Scholar
  45. 45.
    Keeler, K.C.: Map representation and optimal encoding for image segmentation. PhD dissertation. Harvard University (October 1990)Google Scholar
  46. 46.
    Kunita, H.: Stochastic differential equations on manifolds. Cambridge University Press (1991)Google Scholar
  47. 47.
    Kutulakos, K.N., Dyer, C.R.: Global surface reconstruction by purposive control of observer motion. Artificial Intelligence 78(1-2), 147–177 (1995)CrossRefGoogle Scholar
  48. 48.
    Kutulakos, K.N., Jagersand, M.: Exploring objects by invariant-based tangential viewpoint control. In: Proceedings of International Symposium on Computer Vision, pp. 503–508 (1995)Google Scholar
  49. 49.
    Lee, T., Soatto, S.: Learning and matching multiscale template descriptors for real-time detection, localization and tracking. In: Proc. IEEE Conf. on Comp. Vision and Pattern Recogn., pp. 1457–1464 (2011)Google Scholar
  50. 50.
    Lindeberg, T.: Edge Detection and Ridge Detection with Automatic Scale Selection, vol. 30. Cambridge University Press (1998)Google Scholar
  51. 51.
    Lindeberg, T.: Principles for automatic scale selection. Technical report, KTH, Stockholm, CVAP (1998)Google Scholar
  52. 52.
    Lindley, D.V.: On a measure of the information provided by an experiment. Annals of Math. Stat. 27(4), 986–1005 (1956)MathSciNetzbMATHCrossRefGoogle Scholar
  53. 53.
    Ma, Y., Soatto, S., Kosecka, J., Sastry, S.: An invitation to 3D vision, from images to geometric models. Springer, Heidelberg (2003)Google Scholar
  54. 54.
    Marr, D.: Vision. W.H.Freeman & Co. (1982)Google Scholar
  55. 55.
    Marschak, J.: Remarks on the economics of information. Contributions to Scientific Research in Management (1960)Google Scholar
  56. 56.
    Mikolajczyk, K., Schmid, C.: An Affine Invariant Interest Point Detector. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, pp. 128–142. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  57. 57.
    Mumford, D., Gidas, B.: Stochastic models for generic images. Quarterly of Applied Mathematics 54(1), 85–111 (2001)MathSciNetCrossRefGoogle Scholar
  58. 58.
    Newman, P., Ho, K.: SLAM-loop closing with visually salient features. In: Proceedings of the 2005 IEEE International Conference on Robotics and Automation. ICRA 2005, pp. 635–642 (2005)Google Scholar
  59. 59.
    Olshausen, B.A., Field, D.J.: Sparse coding with an overcomplete basis set: A strategy employed by V1? In: Vision Research (1998)Google Scholar
  60. 60.
    Peruch, P., Vercher, J.-L., Gauthier, G.M.: Acquisition of spatial knowledge through visual exploration of simulated environments. Ecological Psychology 7(1), 1–20 (1995)CrossRefGoogle Scholar
  61. 61.
    Pito, R., Co, I.T., Boston, M.A.: A solution to the next best view problem for automated surface acquisition. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(10), 1016–1030 (1999)CrossRefGoogle Scholar
  62. 62.
    Pretto, A., Chiuso, A., Soatto, S.: Sufficient exploration for navigation and recognition. UCLA Technical Report (January 17, 2011) (submitted)Google Scholar
  63. 63.
    Robert, C.P.: The Bayesian Choice. Springer, New York (2001)zbMATHGoogle Scholar
  64. 64.
    Se, S., Lowe, D., Little, J.: Vision-based mobile robot localization and mapping using scale-invariant features. In: Proceedings of the IEEE International Conference on Robotics and Automation, ICRA (2001)Google Scholar
  65. 65.
    Shao, J.: Mathematical Statistics. Springer, Heidelberg (1998)Google Scholar
  66. 66.
    Sim, R., Dudek, G.: Effective exploration strategies for the construction of visual maps. In: Proceedings of 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003), vol. 4 (2003)Google Scholar
  67. 67.
    Soatto, S.: Actionable information in vision. In: Proc. of the Intl. Conf. on Comp. Vision (October 2009)Google Scholar
  68. 68.
    Soatto, S.: Steps Toward a Theory of Visual Information. Technical Report UCLA-CSD100028 (September 13, 2010)Google Scholar
  69. 69.
    Soatto, S.: Texture, structure and visual matching. Technical Report UCLA-CSD-100033 (October 11, 2010)Google Scholar
  70. 70.
    Soatto, S., Chiuso, A.: Controlled recognition bounds for scaling and occlusion channels. In: Proc. of the Data Compression Conference (March 2011)Google Scholar
  71. 71.
    Stachniss, C., Grisetti, G., Burgard, W.: Information gain-based exploration using rao-blackwellized particle filters. In: Proc. of RSS (2005)Google Scholar
  72. 72.
    Sundaramoorthi, G., Petersen, P., Soatto, S.: On the set of images modulo viewpoint and contrast changes. Technical Report UCLA-CSD090005 (2009) (submitted)Google Scholar
  73. 73.
    Taylor, C.J., Kriegman, D.J.: Vision-based motion planning and exploration algorithms for mobile robots. IEEE Trans. on Robotics and Automation 14(3), 417–426 (1998)CrossRefGoogle Scholar
  74. 74.
    Thrun, S.: Learning metric-topological maps for indoor mobile robot navigation. Artificial Intelligence 99(1), 21–71 (1998)zbMATHCrossRefGoogle Scholar
  75. 75.
    Tishby, N., Pereira, F.C., Bialek, W.: The information bottleneck method. In: Proc. of the Allerton Conf. (2000)Google Scholar
  76. 76.
    Todorovic, S., Ahuja, N.: Extracting subimages of an unknown category from a set of images. In: Proc. IEEE Conf. on Comp. Vis. and Patt. Recog. (2006)Google Scholar
  77. 77.
    Turing, A.M.: The chemical basis of morphogenesis. Phil. Trans. of the Royal Society of London, ser. B (1952)Google Scholar
  78. 78.
    Vedaldi, A., Soatto, S.: Features for recognition: viewpoint invariance for non-planar scenes. In: Proc. of the Intl. Conf. of Comp. Vision, pp. 1474–1481 (October 2005)Google Scholar
  79. 79.
    Vedaldi, A., Soatto, S.: Quick Shift and Kernel Methods for Mode Seeking. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part IV. LNCS, vol. 5305, pp. 705–718. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  80. 80.
    Wertheimer, M.: Laws of organization in perceptual forms. In: Ellis, W.D. (ed.) A Sourcebook of Gestalt Psychology, pp. 331–363. Harcourt, Brace and Company (1939)Google Scholar
  81. 81.
    Whaite, P., Ferrie, F.P.: From uncertainty to visual exploration. IEEE Transactions on Pattern Analysis and Machine Intelligence 13(10), 1038–1049 (1991)CrossRefGoogle Scholar
  82. 82.
    Wiener, N.: Cybernetics, or Control and Communication in Men and Machines. MIT Press (1949)Google Scholar
  83. 83.
    Wu, Y.N., Guo, C., Zhu, S.C.: From information scaling of natural images to regimes of statistical models. Quarterly of Applied Mathematics 66, 81–122 (2008)MathSciNetzbMATHGoogle Scholar
  84. 84.
    Zhang, H., Ostrowski, J.P.: Visual motion planning for mobile robots. IEEE Transactions on Robotics and Automation 18(2), 199–208 (2002)CrossRefGoogle Scholar

Copyright information

© Springer Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.University of CaliforniaLos AngelesUSA

Personalised recommendations