Frontiers of Computer Science

, Volume 10, Issue 2, pp 216–232 | Cite as

Cognitive mechanism related to line drawings and its applications in intelligent process of visual media: a survey

  • Yongjin Liu
  • Minjing Yu
  • Qiufang Fu
  • Wenfeng Chen
  • Ye Liu
  • Lexing Xie
Review Article

Abstract

Line drawings, as a concise form, can be recognized by infants and even chimpanzees. Recently, how the visual system processes line-drawings attracts more and more attention from psychology, cognitive science and computer science. The neuroscientific studies revealed that line drawings generate similar neural actions as color photographs, which give insights on how to efficiently process big media data. In this paper, we present a comprehensive survey on line drawing studies, including cognitive mechanism of visual perception, computational models in computer vision and intelligent process in diverse media applications. Major debates, challenges and solutions that have been addressed over the years are discussed. Finally some of the ensuing challenges in line drawing studies are outlined.

Keywords

line drawings cognitive computation visual media intelligent process 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Supplementary material

11704_2015_4450_MOESM1_ESM.ppt (2 mb)
Supplementary material, approximately 2.04 MB.

References

  1. 1.
    Fischetti M. Computers vs. brains. Scientific American, 2011, 305(5): 104–104CrossRefGoogle Scholar
  2. 2.
    Fu X L, Cai L H, Liu Y, Jia J, Chen WF, Zhang Y, Zhao GZ, Liu YJ, Wu C X. A computational cognition model of perception, memory and judgment. Science China Information Sciences, 2014, 57(3): 1–15CrossRefGoogle Scholar
  3. 3.
    Ullman S. High-level vision: Object recognition and visual cognition. Cambridge: MIT press, 2000MATHGoogle Scholar
  4. 4.
    Liu Y J, Fu Q F, Liu Y, Fu X L. A distributed computational cognitive model for object recognition. Science China Information Sciences, 2013, 56(9): 1–13CrossRefGoogle Scholar
  5. 5.
    Riddoch M, Humphreys G. Object Recognition. In Rapp B, ed. Handbook of Cognitive Neuropsychology. Hove: Psychology PressGoogle Scholar
  6. 6.
    Canny J. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1986 (6): 679–698Google Scholar
  7. 7.
    Cole F, Golovinskiy A, Limpaecher A, Barros H S, Finkelstein A, Funkhouser T, Rusinkiewicz S. Where do people draw lines? ACM Transactions on Graphics, 2008, 27(3): 88CrossRefGoogle Scholar
  8. 8.
    Sayim B, Cavanagh P. What Line Drawings Reveal About the Visual Brain. Frontiers in Human Neuroscience, 2011, 5Google Scholar
  9. 9.
    Yonas A, Arterberry M E. Infants perceive spatial structure specified by line junctions. Perception. 1994, 23(12): 1427–1435CrossRefGoogle Scholar
  10. 10.
    Itakura S. Recognition of line-drawing representations by a chimpanzee (Pan troglodytes). The Journal of general psychology, 1994, 121(3): 189–197CrossRefGoogle Scholar
  11. 11.
    Cole F, Sanik K, DeCarlo D, Finkelstein A, Funkhouser T, Rusinkiewicz S, Singh M. How well do line drawings depict shape? ACM Transactions on Graphics, 2009, 28(3): 28CrossRefGoogle Scholar
  12. 12.
    Koenderink J J, Van Doorn A J, Kappers A M L. Surface perception in pictures. Perception & Psychophysics, 1992, 52(5): 487–496CrossRefGoogle Scholar
  13. 13.
    Koenderink J J, van Doorn A J, Kappers A M L, Todd J T. Ambiguity and the “mental eye” in pictorial relief. Perception, 2001, 30(4): 431–448CrossRefGoogle Scholar
  14. 14.
    Marr D, Vision A. A Computational Investigation into the Human Representation and Processing of Visual Information. Cambridge: MIT Press, 2010CrossRefGoogle Scholar
  15. 15.
    Watt R J, Rogers B J. Human vision and cognitive science. Research Directions in Cognitive Science: A European Perspective, 1989, 1: 9–22Google Scholar
  16. 16.
    Sonka M, Hlavac V, Boyle R. Image processing, analysis, and machine vision. Cengage Learning, 2014Google Scholar
  17. 17.
    Biederman I. Recognition-by-components: a theory of human image understanding. Psychological review, 1987, 94(2): 115CrossRefGoogle Scholar
  18. 18.
    Biederman I, Ju G. Surface versus edge-based determinants of visual recognition. Cognitive Psychology, 1988, 20(1): 38–64CrossRefGoogle Scholar
  19. 19.
    Fu Q, Liu Y J, Chen W, Fu X. The time course of natural scene categorization in human brain: simple line-drawings vs. color photographs. Journal of Vision, 2013, 13(9): 1060CrossRefGoogle Scholar
  20. 20.
    Del Viva M M, Punzi G, Benedetti D. Information and Perception of Meaningful Patterns. PLoS One, 2013, 8(7): e69154CrossRefGoogle Scholar
  21. 21.
    Morgan M J. Features and the ‘primal sketch’. Vision Research, 2011, 51(7): 738–753CrossRefGoogle Scholar
  22. 22.
    Delorme A, Richard G, Fabre-Thorpe M. Key visual features for rapid categorization of animals in natural scenes. Frontiers in Psychology, 2010, 1: 21Google Scholar
  23. 23.
    Naber M, Hilger M, Einhäuser W. Animal detection and identification in natural scenes: image statistics and emotional valence. Journal of Vision, 2012, 12(1): 25CrossRefGoogle Scholar
  24. 24.
    Derrington A M, Krauskopf J, Lennie P. Chromatic mechanisms in lateral geniculate nucleus of macaque. The Journal of Physiology, 1984, 357(1): 241–265CrossRefGoogle Scholar
  25. 25.
    Campbell F W, Robson J G. Application of Fourier analysis to the visibility of gratings. The Journal of physiology, 1968, 197(3): 551CrossRefGoogle Scholar
  26. 26.
    Joubert O R, Rousselet G A, Fabre-Thorpe M, Fize D. Rapid visual categorization of natural scene contexts with equalized amplitude spectrum and increasing phase noise. Journal of Vision, 2009, 9(1): 2CrossRefGoogle Scholar
  27. 27.
    Jolicoeur P. The time to name disoriented natural objects. Memory & Cognition, 1985, 13(4): 289–303CrossRefGoogle Scholar
  28. 28.
    Farah M J, Hammond K M. Mental rotation and orientation-invariant object recognition: dissociable processes. Cognition, 1988, 29(1): 29–46CrossRefGoogle Scholar
  29. 29.
    Westheimer G. The Fourier theory of vision. Perception, 2001, 30(5): 531–542CrossRefGoogle Scholar
  30. 30.
    Oppenheim A V, Lim J S. The importance of phase in signals. In: Proceedings of the IEEE. 1981, 69(5): 529–541CrossRefGoogle Scholar
  31. 31.
    Piotrowski L N, Campbell F W. A demonstration of the visual importance and flexibility of spatial-frequency amplitude and phase. Perception, 1982, 11(3): 337–346CrossRefGoogle Scholar
  32. 32.
    Guyader N, Chauvin A, Peyrin C, Hérault J, Marendaz C. Image phase or amplitude? Rapid scene categorization is an amplitude-based process. Comptes Rendus Biologies, 2004, 327(4): 313–318CrossRefGoogle Scholar
  33. 33.
    Chen W F, Liang J, Liu Y J, Fu Q F, Fu X L. Rapid natural scene categorization of line drawings is less influenced by amplitude spectra: evidence from a subliminal perception study. ASSC 8: Poster SessionGoogle Scholar
  34. 34.
    Morrone M C, Burr D C. Feature detection in human vision: a phasedependent energy model. In: Proceedings of the Royal Society of London. 1988, 235(1280): 221–245CrossRefGoogle Scholar
  35. 35.
    Devlin H, Tracey I, Johansen-Berg H, Clare S. What is Functional Magnetic Resonance Imaging (fMRI)? FMRIB Centre, Department of Clinical Neurology, University of OxfordGoogle Scholar
  36. 36.
    Walther D B, Caddigan E, L F F, Beck D M. Natural scene categories revealed in distributed patterns of activity in the human brain. The Journal of Neuroscience, 2009, 29(34): 10573–10581CrossRefGoogle Scholar
  37. 37.
    Walther D B, Chai B, Caddigan E, Beck DM, Li F F. Simple line drawings suffice for functional MRI decoding of natural scene categories. In: Proceedings of the National Academy of Sciences, 2011, 108(23): 9661–9666CrossRefGoogle Scholar
  38. 38.
    Kim S G, Richter W, Uǧurbil K. Limitations of temporal resolution in functional MRI. Magnetic Resonance in Medicine, 1997, 37(4): 631–636CrossRefGoogle Scholar
  39. 39.
    Vanrullen R, Thorpe S J. The time course of visual processing: from early perception to decision-making. Journal of Cognitive Neuroscience, 2001, 13(4): 454–461CrossRefGoogle Scholar
  40. 40.
    Johnson J S, Olshausen B A. Timecourse of neural signatures of object recognition. Journal of Vision, 2003, 3(7): 4CrossRefGoogle Scholar
  41. 41.
    Fu Q F, Liu Y J, Dienes Z, Wu J H, Chen W F, Fu X L. Different time courses of natural scene categorization of color photographs and line-drawings: evidence from event-related potentials. Submitted for publication, 2014Google Scholar
  42. 42.
    Liu Y J, Ma C X, Fu Q, Fu X L, Qin S F, Xie L X. A Sketch-Based Approach for Interactive Organization of Video Clips. ACM Transactions on Multimedia Computing, Communications, and Applications, 2014, 11(1)Google Scholar
  43. 43.
    Snodgrass J G, Vanderwart M. A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. Journal of Experimental Psychology: Human Learning and Memory, 1980, 6(2): 174Google Scholar
  44. 44.
    Magnié M N, Besson M, Poncet M, Dolisi C. The Snodgrass and Vanderwart set revisited: norms for object manipulability and for pictorial ambiguity of objects, chimeric objects, and nonobjects. Journal of Clinical and Experimental Neuropsychology, 2003, 25(4): 521–560CrossRefGoogle Scholar
  45. 45.
    Rossion B, Pourtois G. Revisiting Snodgrass and Vanderwart’s object pictorial set: the role of surface detail in basic-level object recognition. Perception, 2004, 33(2): 217–236CrossRefGoogle Scholar
  46. 46.
    Viggiano M P, Vannucci M, Righi S. A new standardized set of ecological pictures for experimental and clinical research on visual object processing. Cortex, 2004, 40(3): 491–509CrossRefGoogle Scholar
  47. 47.
    Abel S, Weiller C, Huber W, Willmes K. Neural underpinnings for model-oriented therapy of aphasic word production. Neuropsychologia, 2014, 57: 154–165CrossRefGoogle Scholar
  48. 48.
    Janssen N, Carreiras M, Barber H A. Electrophysiological effects of semantic context in picture and word naming. Neuroimage, 2011, 57(3): 1243–1250CrossRefGoogle Scholar
  49. 49.
    Schnur T T. The persistence of cumulative semantic interference during naming. Journal of Memory and Language, 2014, 75: 27–44CrossRefGoogle Scholar
  50. 50.
    Strijkers K, Holcomb P J, Costa A. Conscious intention to speak proactively facilitates lexical access during overt object naming. Journal of Memory and Language, 2011, 65(4): 345–362CrossRefGoogle Scholar
  51. 51.
    Kang H, Lee S, Chui C. Coherent line drawing. In: Proceedings of 5th International Symposium on Non-photorealistic Animation and Rendering. 2007, 43–50CrossRefGoogle Scholar
  52. 52.
    Arbel T, Ferrie F P. Viewpoint selection by navigation through entropy maps. In: Proceedings of the 7th IEEE International Conference on Computer Vision. 1999, 1: 248–254CrossRefGoogle Scholar
  53. 53.
    Laporte C, Arbel T. Efficient discriminant viewpoint selection for active bayesian recognition. International Journal of Computer Vision, 2006, 68(3): 267–287CrossRefGoogle Scholar
  54. 54.
    Secord A, Lu J, Finkelstein A, Sing M, Nealen A. Perceptual models of viewpoint preference. ACM Transactions on Graphics, 2011, 30(5): 109CrossRefGoogle Scholar
  55. 55.
    Chen D Y, Tian X P, Shen Y T, Ouhyoung M. On visual similarity based 3D model retrieval. Computer graphics forum. Publishing, 2003, 22(3): 223–232Google Scholar
  56. 56.
    Cyr C M, Kimia B B. 3D object recognition using shape similiarity based aspect graph. In: Proceedings the 8th IEEE International Conference on Computer Vision. 2001, 1: 254–261Google Scholar
  57. 57.
    Liu Y J, Luo X, Joneja A, Ma C X, Fu X L, Song D W. User-adaptive sketch-based 3-D CAD model retrieval. IEEE Transactions on Automation Science and Engineering, 2013, 10(3): 783–795CrossRefGoogle Scholar
  58. 58.
    Duda R O, Hart P E, Stork D G. Pattern Classification. John Wiley & Sons, 1999.MATHGoogle Scholar
  59. 59.
    Chalechale A, Naghdy G, Mertins A. Sketch-based image matching using angular partitioning. IEEE Transactions on Systems, Man and Cybernetics, 2005, 35(1): 28–41CrossRefGoogle Scholar
  60. 60.
    Eitz M, Richter R, Boubekeur T, Hildebrand K, Alexa M. Sketch-based shape retrieval. ACM Transactions on Graphics, 2012, 31(4): 31Google Scholar
  61. 61.
    Hung C C, Carlson E T, Connor C E. Medial axis shape coding in macaque inferotemporal cortex. Neuron, 2012, 74(6): 1099–1113CrossRefGoogle Scholar
  62. 62.
    Lescroart M D, Biederman I. Cortical representation of medial axis structure. Cerebral Cortex, 2013, 23(3): 629–637CrossRefGoogle Scholar
  63. 63.
    Li Z, Qin S, Jin X. Skeleton-enhanced line drawings for 3D models. Graphical Models, 2014, 76(6): 620–632CrossRefGoogle Scholar
  64. 64.
    Lake B M, Salakhutdinov R, Tenenbaum J. One-shot learning by inverting a compositional causal process. Advances in neural information processing systems. 2013: 2526–2534Google Scholar
  65. 65.
    Ma C X, Liu Y J, Yang H Y, Teng D X, Wang H A, Dai G Z. KnitSketch: a sketch pad for conceptual design of 2D garment patterns. IEEE Transactions on Automation Science and Engineering, 2011, 8(2): 431–437CrossRefGoogle Scholar
  66. 66.
    Sivic J, Zisserman A. Video Google: A text retrieval approach to object matching in videosGoogle Scholar
  67. 67.
    Baeza-Yates R, Ribeiro-Neto B. Modern Information Retrieval. New York: ACM Press, 1999Google Scholar
  68. 68.
    Sugihara K. Machine Interpretation of Line Drawings. Cambridge: MIT Press, 1986Google Scholar
  69. 69.
    Hoffman D D. Visual Intelligence: How We Create What We See. New York: W.W. Norton & Company, 2000Google Scholar
  70. 70.
    Chen T, Cheng M M, Tan P, Ariel S, Hu S M. Sketch2Photo: internet image montage.ACM Transactions on Graphics, 2009, 28(5): 124Google Scholar
  71. 71.
    Cao Y, Wang C H, Zhang L Q, Zhang L. Edgel index for large-scale sketch-based image search. IEEE Conference on Computer Vision and Pattern Recognition, 2011: 761–768Google Scholar
  72. 72.
    Truong B T, Venkatesh S. Video abstraction: a systematic review and classification. ACM Transactions on Multimedia Computing, Communications, and Applications, 2007, 3(1): 3CrossRefGoogle Scholar
  73. 73.
    Ma C X, Liu Y J, Wang H A, Teng D X, Dai G Z. Sketch-based annotation and visualization in video authoring. IEEE Transactions on Multimedia, 2012, 14(4): 1153–1165CrossRefGoogle Scholar
  74. 74.
    Collomosse J P, McNeill G, Qian Y. Storyboard sketches for content based video retrieval. In: Proceedings of the 12th IEEE International Conference on Computer Vision, 2009, 245–252Google Scholar
  75. 75.
    Igarashi T, Matsuoka S, Tanaka H. Teddy: a sketching interface for 3D freeform design. In: Proceedings of ACM SIGGRAPH Courses, 2007: 21Google Scholar
  76. 76.
    Nealen A, Igarashi T, Sorkine O, Alexa M. FiberMesh: designing freeform surfaces with 3D curves. ACM Transactions on Graphics, 2007, 26(3): 41CrossRefGoogle Scholar
  77. 77.
    Liu Y J, Ma C X, Zhang D L. EasyToy: plush toy design using editable sketching curves. IEEE Computer Graphics and Applications, 2011, 31(2): 49–57MathSciNetCrossRefGoogle Scholar
  78. 78.
    Zhu X, Jin X, Liu S, Zhoo H. Analytical solutions for sketch-based convolution surface modeling on the GPU. The Visual Computer, 2012, 28(11): 1115–1125CrossRefGoogle Scholar
  79. 79.
    Yu C C, Liu Y J, Wu T F, Li K Y, Fu X L. A global energy optimization framework for 2.1D sketch extraction from monocular images. Graphical Models, 2014, 76(5): 507–521CrossRefGoogle Scholar
  80. 80.
    Liu Y J, Zhang J B, Hou J C, Ren J C. Cylinder detection in large-scale point cloud of pipeline plant. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(10): 1700–1707CrossRefGoogle Scholar

Copyright information

© Higher Education Press and Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Yongjin Liu
    • 1
  • Minjing Yu
    • 1
  • Qiufang Fu
    • 2
  • Wenfeng Chen
    • 2
  • Ye Liu
    • 2
  • Lexing Xie
    • 3
  1. 1.Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and TechnologyTsinghua UniversityBeijingChina
  2. 2.State Key Laboratory of Brain and Cognitive Science, Institute of PsychologyChinese Academy of SciencesBeijingChina
  3. 3.Research School of Computer ScienceThe Australian National UniversityCanberraAustralia

Personalised recommendations