Skip to main content
SpringerLink
Log in

Learning invariant features using inertial priors

  • Published:
Annals of Mathematics and Artificial Intelligence Aims and scope Submit manuscript

Abstract

We address the technical challenges involved in combining key features from several theories of the visual cortex in a single coherent model. The resulting model is a hierarchical Bayesian network factored into modular component networks embedding variable-order Markov models. Each component network has an associated receptive field corresponding to components residing in the level directly below it in the hierarchy. The variable-order Markov models account for features that are invariant to naturally occurring transformations in their inputs. These invariant features give rise to increasingly stable, persistent representations as we ascend the hierarchy. The receptive fields of proximate components on the same level overlap to restore selectivity that might otherwise be lost to invariance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Adelson, E., Movshon, J.: Phenomenal coherence of moving visual patterns. Nature 300, 523–525 (1982)

    Article  Google Scholar 

  2. Barto, A.G., Sutton, R.S., Anderson, C.W.: Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybernet. 13(5), 835–846 (1983)

    Google Scholar 

  3. Basye, K., Dean, T., Kirman, J., Lejter, M.: A decision-theoretic approach to planning, perception, and control. IEEE Expert 7(4), 58–65 (1992)

    Article  Google Scholar 

  4. Bengio, Y., Frasconi, P.: An input output HMM architecture. In: e. a. Cowan, J.D. (ed.) Advances in Neural Information Processing, vol. 6. Morgan Kaufmann, San Francisco, CA (1994)

    Google Scholar 

  5. Berkes, P., Wiskott, L.: Slow feature analysis yields a rich repertoire of complex cell properties. J. Vis. 5(6), 579–602 (2005)

    Article  Google Scholar 

  6. Bishop, C.M., Svensén, M., Williams, C.K.I.: GTM: A principled alternative to the self-organizing map. In: Mozer, M.C., Jordan, M.I., Petsche, T. (eds.) Advances in Neural Information Processing Systems, vol. 9, pp. 354–360. MIT, Cambridge, MA (1997)

    Google Scholar 

  7. Braitenberg, V., Schuz, A.: Cortex: Statistics and Geometry of Neuronal Connections. Springer, Berlin Heidelberg New York (1991)

    Google Scholar 

  8. Bray, A., Martinez, D.: Kernel-based extraction of slow features: complex cells learn disparity and translation invariance from natural images. In: Advances in Neural Information Processing Systems, vol. 14. MIT, Cambridge, MA (2002)

    Google Scholar 

  9. Burns, G., Daoud, R., Vaigl, J.: LAM: An open cluster environment for MPI. In: Proceedings of Supercomputing Symposium, pp. 379–386 (1994)

  10. Cowell, R.G., Dawid, A.P., Lauritzen, S.L., Spiegelhalter, D.J.: Probabilistic networks and expert systems. Springer, Berlin Heidelberg New York (1999)

    MATH  Google Scholar 

  11. Dean, T.: A computational model of the cerebral cortex. In: Proceedings of AAAI-05, pp. 938–943. MIT, Cambridge, MA (2005)

    Google Scholar 

  12. Dean, T.: Scalable inference in hierarchical generative models. In: Proceedings of the Ninth International Symposium on Artificial Intelligence and Mathematics, (2006)

  13. Dean, T., Kanazawa, K.: A model for reasoning about persistence and causation. Comput. Intel. J. 5(3), 142–150 (1989)

    Google Scholar 

  14. Elidan, G., McGraw, I., Koller, D.: Residual belief propagation: informed scheduling for asynchronous message passing. In: Proceedings of the Twenty-second Conference on Uncertainty in AI (UAI), Boston, MA, July 2006

  15. Eskin, E., Grundy, W.N., Singer, Y.: Protein family classification using sparse markov transducers. In: Proceedings of the Eighth International Conference on Intelligent Systems for Molecular Biology, pp. 134–145 (2000)

  16. Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. In: Proceedings of CVPR-05 Workshop on Generative-Model Based Vision (2004)

  17. Fei-Fei, L., Perona, P.: A Bayesian hierarchical model for learning natural scene categories. In: Proceedings of CVPR (2005)

  18. Fine, S., Singer, Y., Tishby, N.: The hierarchical hidden Markov model: analysis and applications. Mach. Learn. 32(1), 41–62 (1998)

    Article  MATH  Google Scholar 

  19. Földiák, P.: Learning invariance from transformation sequences. Neural Comput. 3, 194–200 (1991)

    Google Scholar 

  20. Friedman, N.: The Bayesian structural EM algorithm. In: Proceedings of the Conference on Uncertainty in Artificial Intelligence, pp. 129–138 (1998)

  21. Friedman, N., Geiger, D., Goldszmidt, M.: Bayesian network classifiers. Mach. Learn. 29(2-3), 131–163 (1997)

    Article  MATH  Google Scholar 

  22. Fukushima, K.: Neocognitron: a self organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 36(4), 93–202 (1980)

    Article  MathSciNet  Google Scholar 

  23. George, D., Hawkins, J.: A hierarchical Bayesian model of invariant pattern recognition in the visual cortex. In: Proceedings of the International Joint Conference on Neural Networks, vol. 3, pp. 1812–1817. IEEE (2005)

  24. Hawkins, J., Blakeslee, S.: On Intelligence. Henry Holt and Company, New York, (2004)

    Google Scholar 

  25. Hawkins, J., George, D.: Hierarchical temporal memory: concepts, theory and terminology, (2006)

  26. Heckerman, D.: A tutorial on learning Bayesian networks. Tech. Rep. MSR-95-06, Microsoft Research (1995)

  27. Hinton, G.: Product of experts. In: Proceedings of International Conference on Neural Networks. vol. 1, pp. 1–6 (1999)

  28. Hinton, G., Sejnowski, T.: Learning and relearning in Boltzmann machines. In: Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume I: Foundations, pp. 282–317. MIT, Cambridge MA (1986)

    Google Scholar 

  29. Hinton, G.E., Osindero, S., Teh, Y.-W.: A fast learning algorithm for deep belief nets. Neural Comput. (2006)

  30. Huang, C., Darwiche, A.: Inference in belief networks: a procedural guide. Int. J. Approx. Reason. 15(3), 225–263 (1996)

    Article  MATH  MathSciNet  Google Scholar 

  31. Hubel, D.H.: Eye, Brain and Vision (Scientific American Library, Number 22). Freeman, San Francisco, CA (1995)

  32. Hubel, D.H., Wiesel, T.N.: Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol. 160, 106–154 (1962)

    Google Scholar 

  33. Jackendoff, R.: Foundations of Language: Brain, Meaning, Grammar, Evolution. Oxford University Press, Oxford, UK (2002)

    Google Scholar 

  34. Jensen, F.: Bayesian graphical models. Wiley, New York, NY (2000)

    Google Scholar 

  35. Jordan, M. (ed.): Learning in Graphical Models. MIT, Cambridge, MA (1998)

    MATH  Google Scholar 

  36. Kabán, A., Girolami, M.: A dynamic probabilistic model to visualise topic evolution in text streams. Journal of Intelligent Information System 18(2-3), 107–125 (2003)

    Google Scholar 

  37. Kearns, M., Valiant, L.G.: Cryptographic limitations on learning boolean functions and finite automata. In: Proceedings of the Twenty First Annual ACM Symposium on Theoretical Computing, pp. 433–444 (1989)

  38. Kubovy, M., Gepshtein, S.: Perceptual grouping in space and in space-time: an exercise in phenomenological psychophysics. In: Behrmann, M., Kimchi, R., Olson, C.R. (eds.) Perceptual Organization in Vision: Behavioral and Neural Perspectives, pp. 45–85. Lawrence Erlbaum, Mahwah, NJ, (2003)

    Google Scholar 

  39. Kulikowski, J.J., Bishop, P.O.: Fourier analysis and spatial representation in the visual cortex. Experientia 37, 160–163 (1981)

    Article  Google Scholar 

  40. LeCun, Y., Huang, F.-J., Bottou, L.: Learning methods for generic object recognition with invariance to pose and lighting. In: Proceedings of IEEE Computer Vision and Pattern Recognition, IEEE Press (2004)

  41. LeCun, Y., Matan, O., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D., Baird, H.S.: Handwritten zip code recognition with multilayer networks. In: Proceedings of the International Conference on Pattern Recognition. IAPR, Ed., vol. II, pp. 35–40. IEEE Press (1990)

  42. Lee, T.S., Mumford, D.: Hierarchical Bayesian inference in the visual cortex. J. Opt. Soc. Am. 2(7), 1434–1448 (July 2003)

    Google Scholar 

  43. Mayraz, G., Hinton, G.: Recognizing handwritten digits using hierarchical products of experts. IEEE Trans. Pattern Anal. Mach. Intel. 24, 189–197 (2002)

    Article  Google Scholar 

  44. McClelland, J., Rumelhart, D.: An interactive activation model of context effects in letter perception: I. An account of basic findings. Psychol. Rev. 88, 375–407 (1981)

    Google Scholar 

  45. Murphy, G.L.: The Big Book of Concepts. MIT, Cambridge, MA (2002)

    Google Scholar 

  46. Murphy, K.: The Bayes Net Toolbox for Matlab. Comput. Sci. Stat. 33 (2001)

  47. Murphy, K., Weiss, Y., Jordan, M.: Loopy-belief propagation for approximate inference: an empirical study. In: Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, pp. 467–475. Morgan Kaufmann, San Mateo, CA (2000)

    Google Scholar 

  48. Murphy, K.P.: Dynamic Bayesian Networks: Representation, Inference and Learning. Ph.D. thesis, University of California, Berkeley, Computer Science Division (2002)

  49. Murray, S.O., Olshausen, B.A., Woods, D.L.: Processing shape, motion and three-dimensional shape-from-motion in the human cortex. Cerebral Cortex 13(5), 508–516 (2003)

    Article  Google Scholar 

  50. Ng, A., Jordan, M., Weiss, Y.: On spectral clustering: analysis and an algorithm. In: Advances in Neural Information Processing Systems, vol. 14. MIT, Cambridge, MA (2002)

    Google Scholar 

  51. Olshausen, B.A., Anderson, A., Van Essen, D.C.: A neurobiological model of visual attention and pattern recognition based on dynamic routing of information. J. Neurosci. 13(11), 4700–4719 (1993)

    Google Scholar 

  52. Olshausen, B.A., Field, D.J.: How close are we to understanding V1? Neural Comput. 17, 1665–1699 (2005)

    Article  MATH  Google Scholar 

  53. Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Francisco, CA (1988)

    Google Scholar 

  54. Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. In: Proceedings of IEEE, vol. 77(2), pp. 257–286 (1989)

  55. Riesenhuber, M., Poggio, T.: Hierarchical models of object recognition in cortex. Nat. Neurosci. 2(11), 1019–1025 (November 1999)

    Article  Google Scholar 

  56. Rivest, R.L., Sloan, R.: A formal model of hierarchical concept learning. Inf. Comput. 114(1), 88–114 (1994)

    Article  MATH  MathSciNet  Google Scholar 

  57. Ron, D., Singer, Y., Tishby, N.: The power of amnesia: learning probabilistic automata with variable memory length. Mach. Learn. 25(2,3), 117–149 (1996)

    Article  MATH  Google Scholar 

  58. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In: Rumelhart, D.E., McClelland, J.L. (eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations. MIT, Cambridge, MA (1986)

    Google Scholar 

  59. Sage, K.H., Buxton, H.: Joint spatial and temporal structure learning for task based control. In: International Conference on Pattern Recognition (ICPR), vol. 2, pp. 48–51 (2004)

  60. Ullman, S.: The Interpretation of Visual Motion. MIT, Cambridge, MA, USA, (1979)

    Google Scholar 

  61. Ullman, S., Soloviev, S.: Computation of pattern invariance in brain-like structures. Neural Netw. 12, 1021–1036 (1999)

    Article  Google Scholar 

  62. Valiant, L.G.: A theory of the learnable. Commun. ACM 27, 1134–1142 (1984)

    Article  MATH  Google Scholar 

  63. Verbeek, J., Vlassis, N., Kröse, B.: Efficient greedy learning of Gaussian mixtures. Neural Comput. 15(2), 469–485 (2003)

    Article  MATH  Google Scholar 

  64. Wertheimer, M.: Untersuchungen zur lehre von der gestalt, II (Investigations of the principles of gestalt, II). Psychol. Forsch. 4, 301–350 (1923)

    Article  Google Scholar 

  65. Wiskott, L., Sejnowski, T.: Slow feature analysis: unsupervised learning of invariances. Neural Comput. 14(4), 715–770 (2002)

    Article  MATH  Google Scholar 

  66. Yedidia, J.S., Freeman, W.T., Weiss, Y.: Understanding belief propagation and its generalizations. In: Lakemeyer, G., Nebel, B. (eds.) Exploring Artificial Intelligence in the New Millennium, pp. 239–269. Morgan Kaufmann, San Francisco, CA (2003)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Brown University, Providence, RI, 02912, USA

    Thomas Dean

Authors
  1. Thomas Dean
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas Dean.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Dean, T. Learning invariant features using inertial priors. Ann Math Artif Intell 47, 223–250 (2006). https://doi.org/10.1007/s10472-006-9039-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10472-006-9039-9

Keywords

Mathematics Subject Classifications (2000)

Search

Navigation