Science China Information Sciences

, Volume 57, Issue 3, pp 1–15 | Cite as

A computational cognition model of perception, memory, and judgment

  • XiaoLan FuEmail author
  • LianHong Cai
  • Ye Liu
  • Jia Jia
  • WenFeng Chen
  • Zhang Yi
  • GuoZhen Zhao
  • YongJin Liu
  • ChangXu Wu
Research Paper


The mechanism of human cognition and its computability provide an important theoretical foundation to intelligent computation of visual media. This paper focuses on the intelligent processing of massive data of visual media and its corresponding processes of perception, memory, and judgment in cognition. In particular, both the human cognitive mechanism and cognitive computability of visual media are investigated in this paper at the following three levels: neurophysiology, cognitive psychology, and computational modeling. A computational cognition model of Perception, Memory, and Judgment (PMJ model for short) is proposed, which consists of three stages and three pathways by integrating the cognitive mechanism and computability aspects in a unified framework. Finally, this paper illustrates the applications of the proposed PMJ model in five visual media research areas. As demonstrated by these applications, the PMJ model sheds some light on the intelligent processing of visual media, and it would be innovative for researchers to apply human cognitive mechanism to computer science.


perception memory judgment computational cognition model 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Gallistel C R, King A. Memory and the Computational Brain: Why Cognitive Science Will Transform Neuroscience. New York: Blackwell/Wiley, 2009. iiv–xviGoogle Scholar
  2. 2.
    Hu S M, Chen T, Xu K, et al. Internet visual media processing: a survey with graphics and vision applications. Vis Comput, 2013, 29: 393–405Google Scholar
  3. 3.
    Hulusic V, Debattista K, Aggarwal V, et al. Maintaining frame rate perception in interactive environments by exploiting audio-visual cross-modal interaction. Vis Comput, 2011, 27: 57–66Google Scholar
  4. 4.
    Vazquez P-P, Marco J. Using normalized compression distance for image similarity measurement: an experimental study. Vis Comput, 2012, 28: 1063–1084Google Scholar
  5. 5.
    Eysenck M W, Keane M T. Cognitive Psychology: a Student’s Handbook. 6th ed. New York: Psychology Press, 2010. 1–50Google Scholar
  6. 6.
    National Institute on Drug Abuse. Computational neuroscience at the NIH. Nat Neurosci, 2000, 3: 1161–1164Google Scholar
  7. 7.
    Buschman T J, Miller E K. Top-down versus bottom-up control of attention in the prefrontal and posterior parietal cortices. Science, 2007, 315: 1860–1862Google Scholar
  8. 8.
    Navalpakkam V, Itti L. Search goal tunes visual features optimally. Neuron, 2007, 53: 605–617Google Scholar
  9. 9.
    Katsuki F, Constantinidis C. Early involvement of prefrontal cortex in visual bottom-up attention. Nat Neurosci, 2012, 15: 1160–1166Google Scholar
  10. 10.
    Corbetta M, Shulman G L. Control of goal-directed and stimulus-driven attention in the brain. Nat Rev Neurosci, 2002, 3: 201–215Google Scholar
  11. 11.
    Zanto T P, Rubens M T, Thangavel A, et al. Causal role of the prefrontal cortex in top-down modulation of visual processing and working memory. Nat Neurosci, 2011, 14: 656–661Google Scholar
  12. 12.
    Tomita H M, Ohbayashi K, Nakahara I, et al. Top-down signal from prefrontal cortex in executive control of memory retrieval. Nature, 1999, 401: 699–703Google Scholar
  13. 13.
    Itti L, Koch C. Computational modelling of visual attention. Nat Rev Neurosci, 2001, 2: 194–203Google Scholar
  14. 14.
    Cox D, Meyers E, Sinha P. Contextually evoked object-specific responses in human visual cortex. Science, 2004, 303: 115–117Google Scholar
  15. 15.
    Kouh M, Poggio T. A canonical neural circuit for cortical nonlinear operations. Neural Comput, 2008, 20: 1427–1451zbMATHMathSciNetGoogle Scholar
  16. 16.
    Poggio T, Bizzi E. Generalization in vision and motor control. Nature, 2004, 431: 768–774Google Scholar
  17. 17.
    Hung C P, Kreiman G, Poggio T, et al. Fast readout of object identity from macaque inferior temporal cortex. Science, 2005, 310: 863–866Google Scholar
  18. 18.
    Pylyshyn Z W. Computation and Cognition: toward a Foundation for Cognitive Science. Cambridge: The MIT Press, 1984. 1–16Google Scholar
  19. 19.
    Hasselmo M E, Sarter M. Modes and models of forebrain cholinergic neuromodulation of cognition. Neuropsychopharmacology, 2010, 36: 52–73Google Scholar
  20. 20.
    Tamietto M, de Gelder B. Neural bases of the non-conscious perception of emotional signals. Nat Rev Neurosci, 2010, 11: 697–709Google Scholar
  21. 21.
    Fries P, Reynolds J H, Rorie A E, et al. Modulation of oscillatory neuronal synchronization by selective visual attention. Science, 2001, 291: 1560–1563Google Scholar
  22. 22.
    Roberts M, Delicato L S, Herrero J, et al. Attention alters spatial integration in macaque V1 in an eccentricitydependent manner. Nat Neurosci, 2007, 10: 1483–1491Google Scholar
  23. 23.
    Qiu F T, Sugihara T, von der Heydt R. Figure-ground mechanisms provide structure for selective attention. Nat Neurosci, 2007, 10: 1492–1499Google Scholar
  24. 24.
    Hübner R, Steinhauser M, Lehle C. A dual-stage two-phase model of selective attention. Psychol Rev, 2010, 117: 759–784Google Scholar
  25. 25.
    Gondan M, Blurton S P, Hughes F, et al. Effects of spatial and selective attention on basic multisensory integration. J Exp Phychol-Hum Percep Perf, 2011, 37: 1887–1897Google Scholar
  26. 26.
    Schafer R J, Moore T. Selective attention from voluntary control of neurons in prefrontal cortex. Science, 2011, 332: 1568–1571Google Scholar
  27. 27.
    Couperus J W. Perceptual load influences selective attention across development. Develop Psychol, 2011, 47: 1431–1439Google Scholar
  28. 28.
    Cosman J D, Vecera S P. Object-based attention overrides perceptual load to modulate visual distraction. J Exp Phychol-Hum Percep Perf, 2012, 38: 576–579Google Scholar
  29. 29.
    Chen C C, Wu J K, Lin H W, et al. Visualizing long-term memory formation in two neurons of the drosophila brain. Science, 2012, 335: 678–685Google Scholar
  30. 30.
    Fell J, Axmacher N. The role of phase synchronization in memory processes. Nat Rev Neurosci, 2011, 12: 105–118Google Scholar
  31. 31.
    Fusi S, Abbott L F. Limits on the memory storage capacity of bounded synapses. Nat Neurosci, 2007, 10: 485–493Google Scholar
  32. 32.
    Eichenbaum H. A cortical-hippocampal system for declarative memory. Nat Rev Neurosci, 2000, 1: 41–50Google Scholar
  33. 33.
    McGaugh J L. Memory-a century of consolidation. Science, 2000, 287: 248–251Google Scholar
  34. 34.
    Tronson N C, Taylor J R. Molecular mechanisms of memory reconsolidation. Nat Rev Neurosci, 2007, 8: 262–275Google Scholar
  35. 35.
    Edelson M, Sharot T, Dolan R J, et al. Following the crowd: brain substrates of long-term memory conformity. Science, 2011, 333: 108–111Google Scholar
  36. 36.
    Frankland P W, Bontempi B. The organization of recent and remote memories. Nat Rev Neurosci, 2005, 6: 119–130Google Scholar
  37. 37.
    Nadel L, Hardt O. Update on memory systems and processes. Neuropsychopharmacology, 2011, 36: 251–273Google Scholar
  38. 38.
    Nader K, Hardt O. A single standard for memory: the case for reconsolidation. Nat Rev Neurosci, 2009, 10: 224–234Google Scholar
  39. 39.
    Gonzalez C, Dutt V. Instance-based learning: Integrating sampling and repeated decisions from experience. Psychol Rev, 2011, 118: 523–551Google Scholar
  40. 40.
    Homa D, Hout M C, Milliken L, et al. Bogus concerns about the false prototype enhancement effect. J Exp Psychol-Learn Mem Cogn, 2011, 37: 368–377Google Scholar
  41. 41.
    Smith J D, Minda J P. Distinguishing prototype-based and exemplar-based processes in dot-pattern category learning. J Exp Psychol-Learn Mem Cogn, 2002, 28: 800–811Google Scholar
  42. 42.
    Smith J D, Redford J S, Haas S M. Prototype abstraction by monkeys (Macaca mulatta). J Exp Psychol-Gen, 2008, 137: 390–401Google Scholar
  43. 43.
    Lewandowsky S, Palmeri T J, Waldmann M R. Introduction to the special section on theory and data in categorization: integrating computational, behavioral, and cognitive neuroscience approaches. J Exp Psychol-Learn Mem Cogn, 2012, 38: 803–806Google Scholar
  44. 44.
    Freedman D J, Assad J A. A proposed common neural mechanism for categorization and perceptual decisions. Nat Neurosci, 2011, 14: 143–146Google Scholar
  45. 45.
    Gold J I, Shadlen M N. The influence of behavioral context on the representation of a perceptual decision in developing oculomotor commands. J Neurosci, 2003, 23: 632–651Google Scholar
  46. 46.
    Kable J W, Glimcher P W. The neurobiology of decision: consensus and controversy. Neuron, 2009, 63: 733–745Google Scholar
  47. 47.
    Freedman D J, Assad J A. Experience-dependent representation of visual categories in parietal cortex. Nature, 2006, 443: 85–88Google Scholar
  48. 48.
    Freedman D J, Riesenhuber M, Poggio T, et al. Categorical representation of visual stimuli in the primate prefrontal cortex. Science, 2001, 291: 312–316Google Scholar
  49. 49.
    Freedman D J, Riesenhuber M, Poggio T, et al. A comparison of primate prefrontal and inferior temporal cortices during visual categorization. J Neurosci, 2003, 23: 5235–5246Google Scholar
  50. 50.
    Williams Z M, Elfar J C, Eskandar E N, et al. Parietal activity and the perceived direction of ambiguous apparent motion. Nat Neurosci, 2003, 6: 616–623Google Scholar
  51. 51.
    Toth L J, Assad J A. Dynamic coding of behaviourally relevant stimuli in parietal cortex. Nature, 2002, 415: 165–168Google Scholar
  52. 52.
    Stoet G, Snyder L H. Single neurons in posterior parietal cortex of monkeys encode cognitive set. Neuron, 2004, 42: 1003–1012Google Scholar
  53. 53.
    Gottlieb J. From thought to action: the parietal cortex as a bridge between perception, action, and cognition. Neuron, 2007, 53: 9–16Google Scholar
  54. 54.
    Chen Y, Martinez-Conde S, Macknik S L, et al. Task difficulty modulates the activity of specific neuronal populations in primary visual cortex. Nat Neurosci, 2008, 11: 974–982Google Scholar
  55. 55.
    Asplund C L, Todd J J, Snyder A P, et al. A central role for the lateral prefrontal cortex in goal-directed and stimulus-driven attention. Nat Neurosci, 2010, 13: 507–512Google Scholar
  56. 56.
    Solway A, Botvinick M M. Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates. Psychol Rev, 2012, 119: 120–154Google Scholar
  57. 57.
    Purcell B A, Heitz R P, Cohen J Y, et al. Neurally constrained modeling of perceptual decision making. Psychol Rev, 2010, 117: 1113–1143Google Scholar
  58. 58.
    Palmeri T J, Gauthier I. Visual object understanding. Nat Rev Neurosci, 2004, 5: 291–304Google Scholar
  59. 59.
    Peyrin C, Michel C M, Schwartz S, et al. The neural substrates and timing of top-down processes during coarse-to-fine categorization of visual scenes: a combined fMRI and ERP study. J Cognitive Neurosci, 2010, 22: 2768–2780Google Scholar
  60. 60.
    Gao Z F, Bentin S. Coarse-to-fine encoding of spatial frequency information into visual short-term memory for faces but impartial decay. J Exp Phychol-Hum Percep Perf, 2011, 37: 1051–1064Google Scholar
  61. 61.
    Goffaux V, Peters J, Haubrechts J, et al. From coarse to fine? Spatial and temporal dynamics of cortical face processing. Cereb Cortex, 2011, 21: 467–476Google Scholar
  62. 62.
    Griffiths O, Mitchell C J. Selective attention in human associative learning and recognition memory. J Exp Psychol-Gen, 2008, 137: 626–648Google Scholar
  63. 63.
    Deng W, Aimone J B, Gage F H. New neurons and new memories: how does adult hippocampal neurogenesis affect learning and memory? Nat Rev Neurosci, 2010, 11: 339–350Google Scholar
  64. 64.
    De Fockert J W, Rees G, Frith C D, et al. The role of working memory in visual selective attention. Science, 2001, 291: 1803–1806Google Scholar
  65. 65.
    Saalmann Y B, Pigarev I N, Vidyasagar T R. Neural mechanisms of visual attention: how top-down feedback highlights relevant locations. Science, 2007, 316: 1612–1615Google Scholar
  66. 66.
    Sigala N, Logothetis N K. Visual categorization shapes feature selectivity in the primate temporal cortex. Nature, 2002, 415: 318–320Google Scholar
  67. 67.
    Kundel H L, Nodine C F. Interpreting chest radiographs without visual search. Radiology, 1975, 116: 527–532Google Scholar
  68. 68.
    Treisman A M, Gelade G. A feature-integration theory of attention. Cog Psychol, 1980, 12: 97–136Google Scholar
  69. 69.
    Liu Y J, Fu Q F, Liu Y, et al. 2D-line-drawing-based 3D object recognition. In: Computational Visual Media, Beijing, 2012. 146–153Google Scholar
  70. 70.
    Liu Y J, Luo X, Joneja A, et al. User-adaptive sketch-based 3D CAD model retrieval. IEE Trans Autom Sci Eng, 2013, 99: 1–13Google Scholar
  71. 71.
    Wolfe J M. Guided Search 4.0: current progress with a model of visual search. In: Integrated Models of Cognitive Systems. New York: Oxford, 2007. 99–119Google Scholar
  72. 72.
    Wolfe J M, Cave K R, Franzel S L. Guided search: an alternative to feature integration model for visual search. J Exp Phychol-Hum Percep Perf, 1989, 15: 419–433Google Scholar
  73. 73.
    Williams C C, Henderson J M, Zacks R T. Incidental visual memory for targets and distractors in visual search. Percept Psychophys, 2005, 67: 816–827Google Scholar
  74. 74.
    Wolfe J M. Guided search 2.0: a revised model of visual search. Psychonomic Bull Rev, 1994, 1: 202–238Google Scholar
  75. 75.
    Hao F, Zhang H, Fu X L. Modulation of attention by faces expressing emotion: evidence from visual marking. In: Tao J H, Tan T N, Picard R W, eds. Affective Computing and Intelligent Interaction. Berlin/Heidelberg: Springer-Verlag, 2005. 127–134Google Scholar
  76. 76.
    Hao F, Fu X L. Visual marking: a mechanism of prioritizing selection. Adv Psychol Sci, 2006, 14: 7–11Google Scholar
  77. 77.
    Sternberg S. High-speed scanning in human memory. Science, 1966, 153: 652–654Google Scholar
  78. 78.
    Hawkins J, Blakeslee S. On Intelligence. New York: Times Books, 2004Google Scholar
  79. 79.
    Bear M F, Connors B W, Paradiso M A. Neuroscience: exploring the brain. 3rd ed. Philadelphia: Lippincott Williams & Wilkins, 2006Google Scholar
  80. 80.
    Rinkus G J. A cortical sparse distributed coding model linking mini’and macrocolumn-scale functionality. Front Neuroanat, 2010, 4: 17Google Scholar
  81. 81.
    Mountcastle V. An Organizing Principle for Cerebral Function: the Unit Model and the Distributed System. Cambridge: MIT Press, 1978Google Scholar
  82. 82.
    Elad M. Sparse and Redundant Representations: from Theory to Applications in Signal and Image Processing. New York: Springer, 2010Google Scholar
  83. 83.
    Bruckstein A M, Donoho D L, Elad M. From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev, 2009, 51: 34–81zbMATHMathSciNetGoogle Scholar
  84. 84.
    Yi Z, Tan K K. Convergence Analysis of Recurrent Neural Networks. Dordrecht: Kluwer Academic Publishers, 2004zbMATHGoogle Scholar
  85. 85.
    Tang H J, Tan K C, Yi Z. Neural Networks: Computational Models and Applications. Heidelberg: Springer-Verlag, 2007Google Scholar
  86. 86.
    Seung H S. How the brain keeps the eyes still. Proc Nat Acad Sci USA, 1996, 93: 13339–13344Google Scholar
  87. 87.
    Wu S, Amari S, Nakahara H. Population coding and decoding in a neural field: a computational study. Neural Comput, 2002, 14: 999–1026zbMATHGoogle Scholar
  88. 88.
    Zhang K. Representation of spatial orientation by the intrinsic dynamics of head-direction cell ensembles: a theory. J Neurosci, 1996, 16: 2112–2126Google Scholar
  89. 89.
    Yu J, Yi Z, Zhang L. Representations of continuous attractors of recurrent neural networks. IEEE Trans Neural Netw, 2009, 20: 368–372Google Scholar
  90. 90.
    Wu C, Liu Y. Queuing network modeling of the psychological refractory period (PRP). Psychol Rev, 2008, 115: 913–954Google Scholar
  91. 91.
    Johnson J G, Busemeyer J R. Rule-based decision field theory: a dynamic computational model of transitions among decision-making strategies. In: Betsch T, Haberstroh S, eds. The Routines of Decision Making. Mahwah: Lawrence Erlbaum, 2005. 3–19Google Scholar
  92. 92.
    Zhao G, Wu C, Qiao C. A mathematical model for the prediction of speeding with its validation. IEEE Trans Intell Transp Syst, 2013, 14: 828–836Google Scholar
  93. 93.
    Wang X H, Jia J, Hu P Y, et al. Understanding the emotional impact of image. In: ACM Multimedia, Nara, 2012. 1369–1370Google Scholar
  94. 94.
    Jia J, Wu S, Wang X H, et al. Can we understand van Gogh’s mood? Learning to infer affects from images in social networks. In: ACM Multimedia, Nara, 2012. 857–860Google Scholar
  95. 95.
    Wang X H, Jia J, Cai L H. Affective image adjustment with a single word. Vis Comput, 2013, 29: 1121–1133Google Scholar
  96. 96.
    Wang X H, Jia J, Liao H Y, et al. Affective image colorization. J Comput Sci Technol, 2012, 27: 1119–1128Google Scholar
  97. 97.
    Kobayashi S. Art of Color Combinations. Tokyo: Kodansha International, 1995Google Scholar
  98. 98.
    Zhang Y F, Hu S M, Martin R R. Shrinkability maps for content-aware video resizing. Comput Graph Forum, 2008, 27: 1797–1804Google Scholar
  99. 99.
    Zhang G X, Cheng M M, Hu S M, et al. A shape-preserving approach to image resizing. Comput Graph Forum, 2009, 28: 1897–1906Google Scholar
  100. 100.
    Dahan M J, Chen N, Shamir A, et al. Combining color and depth for enhanced image segmentation and retargeting. Vis Comput, 2012, 28: 1181–1193Google Scholar
  101. 101.
    Liu Y J, Luo X, Xuan Y M, et al. Image retargeting quality assessment. Comput Graph Forum, 2011, 30: 583–592Google Scholar
  102. 102.
    Chen L. Topological structure in visual perception. Science, 1982, 218: 699–700Google Scholar
  103. 103.
    Anderson J R, Bothell D, Byrne M D, et al. An integrated theory of the mind. Psychol Rev, 2004, 111: 1036–1060Google Scholar

Copyright information

© Science China Press and Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • XiaoLan Fu
    • 1
    Email author
  • LianHong Cai
    • 2
  • Ye Liu
    • 1
  • Jia Jia
    • 2
  • WenFeng Chen
    • 1
  • Zhang Yi
    • 3
  • GuoZhen Zhao
    • 4
  • YongJin Liu
    • 2
  • ChangXu Wu
    • 4
  1. 1.State Key Laboratory of Brain and Cognitive Science, Institute of PsychologyChinese Academy of SciencesBeijingChina
  2. 2.TNLIST, Department of Computer Science and TechnologyTsinghua UniversityBeijingChina
  3. 3.College of Computer ScienceSichuan UniversityChengduChina
  4. 4.Key Laboratory of Behavioral Science, Institute of PsychologyChinese Academy of SciencesBeijingChina

Personalised recommendations