Skip to main content

A Computational Model for Amodal Completion

Journal of Mathematical Imaging and Vision volume 56, pages 511–534 (2016)Cite this article

Abstract

This paper presents a computational model to recover the most likely interpretation of the 3D scene structure from a planar image, where some objects may occlude others. The estimated scene interpretation is obtained by integrating some global and local cues and provides both the complete disoccluded objects that form the scene and their ordering according to depth. Our method first computes several distal scenes which are compatible with the proximal planar image. To compute these different hypothesized scenes, we propose a perceptually inspired object disocclusion method, which works by minimizing the Euler’s elastica as well as by incorporating the relatability of partially occluded contours and the convexity of the disoccluded objects. Then, to estimate the preferred scene, we rely on a Bayesian model and define probabilities taking into account the global complexity of the objects in the hypothesized scenes as well as the effort of bringing these objects in their relative position in the planar image, which is also measured by an Euler’s elastica-based quantity. The model is illustrated with numerical experiments on, both, synthetic and real images showing the ability of our model to reconstruct the occluded objects and the preferred perceptual order among them. We also present results on images of the Berkeley dataset with provided figure-ground ground-truth labeling.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25

References

  1. Arias, P., Facciolo, G., Caselles, V., Sapiro, G.: A variational framework for exemplar-based image inpainting. Int. J. Comput. Vis. 93, 319–347 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  2. Aujol, J.F., Ladjal, S., Masnou, S.: Exemplar-based inpainting from a variational point of view. SIAM J. Math. Anal. 42(3), 1246–1285 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  3. Ballester, C., Bertalmío, M., Caselles, V., Sapiro, G., Verdera, J.: Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans. IP 10(8), 1200–1211 (2001)

    MathSciNet  MATH  Google Scholar 

  4. Bellettini, G., Dal Maso, G., Paolini, M.: Semicontinuity and relaxation properties of a curvature depending functional in 2d. Ann. Sc. Norm. Super. Pisa, Cl. Sci. 20(2), 247–297 (1993)

    MathSciNet  MATH  Google Scholar 

  5. Burge, J., Fowlkes, C., Banks, M.: Natural-scene statistics predict how the figure-ground cue of convexity affects human depth perception. J. Neurosci. 30(21), 7269–7280 (2010)

    Article  Google Scholar 

  6. Cao, F., Gousseau, Y., Masnou, S., Prez, P.: Geometrically guided exemplar-based inpainting. SIAM J. Imaging Sci. 4(4), 1143–1179 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  7. Carrigan, S., Palmer, E., Kellman, P.: Differentiating local and global processes in amodal completion through dot localization. J. Vis. 15(12), 1123–1123 (2015)

    Article  Google Scholar 

  8. Chan, T., Shen, J.H.: Mathematical models for local nontexture inpaintings. SIAM J. Appl. Math. 62(3), 1019–1043 (2001)

    MathSciNet  MATH  Google Scholar 

  9. Chan, T., Shen, J.H.: Nontexture inpainting by curvature-driven diffusions. J. Vis. Commun. Image Represent. 12(4), 436–449 (2001)

    Article  Google Scholar 

  10. Chen, Y., Sundaram, H.: Estimating complexity of 2d shapes. In: IEEE 7th Workshop on Multimedia Signal Processing, pp. 1–4 (2005)

  11. Citti, G., Sarti, A.: A cortical based model of perceptual completion in the roto-translation space. J. Math. Imaging Vis. 24(3), 307–326 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  12. Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based inpainting. IEEE Trans. IP 13(9), 1200–1212 (2004)

    Google Scholar 

  13. Demanet, L., Song, B., Chan, T.: Image inpainting by correspondence maps: a deterministic approach. Appl. Comput. Math. 1100, 217–250 (2003)

    Google Scholar 

  14. Esedoglu, S., Ruuth, S.J., Tsai, R.: Threshold dynamics for high order geometric motions. Interfaces Free Bound. 10(3), 263–282 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  15. Fowlkes, C., Martin, D., Malik, J.: Local figure/ground cues are valid for natural images. J. Vis. 7(8), 1–9 (2007)

    Article  Google Scholar 

  16. Froyen, V., Kogo, N., Singh, M., Feldman, J.: Modal and amodal shape completion. J. Vis. 15(12), 321–321 (2015)

    Article  Google Scholar 

  17. Gerbino, W., Salmaso, D.: The effect of amodal completion on visual matching. Acta Psychol. 65(1), 25–46 (1987)

    Article  Google Scholar 

  18. Grompone von Gioi, R.: Toward a computational theory of perception. apcap 2009 p. 49 (2009)

  19. von Gioi, R.G., Jakubowicz, J., Morel, J.M., Randall, G.: Lsd: a line segment detector. Image Process. Line 2, 35–55 (2012)

    Article  Google Scholar 

  20. Grzibovskis, R., Heintz, A.: A convolution-thresholding scheme for the willmore flow. Tech. Rep. 34

  21. Li, H., Cai, J., Nguyen, T.N.A., Zheng, J.: A benchmark for semantic image segmentation. In: ICME (2013)

  22. Hayashi, T., Sasaki, M.: Contour completion of partly occluded skew-symmetry objects. In: 2014 IEEE International Symposium on Multimedia (ISM), pp. 90–93 (2014)

  23. van der Helm, P.: Bayesian confusions surrounding simplicity and likelihood in perceptual organization. Acta Psychol. 138(3), 337–346 (2011)

    Article  Google Scholar 

  24. Kang, S.H., Zhu, W., Shen, J.: Illusory shapes via corner fusion. SIAM J. Imaging Sci. 7(4), 1907–1936 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  25. Kanizsa, G.: Amodal completion and shrinking of visual fields. Studia Psychol. 14, 208–210 (1972)

    Google Scholar 

  26. Kanizsa, G.: Organization in Vision: Essays on Gestalt Perception. Praeger, New York (1979)

    Google Scholar 

  27. Kanizsa, G.: Seeing and thinking. Acta Psychol. 59(1), 23–33 (1985)

    Article  Google Scholar 

  28. Kanizsa, G.: Vedere et pensare. Il Mulino, Bologna (1991)

    Google Scholar 

  29. Kawai, N., Sato, T., Yokoya, N.: Image inpainting considering brightness change and spatial locality of textures and its evaluation. In: Advances in Image and Video Technology, pp. 271–282 (2009)

  30. Kellman, P., Shipley, T.: A theory of visual interpolation in object perception. Cogn. Psychol. 23(2), 141–221 (1991)

    Article  Google Scholar 

  31. Knill, D., Kersten, D., Yuille, A.: Introduction: a bayesian formulation of visual perception. In: Knill, D., Richards, W. (eds.) Perception as Bayesian Inference. Cambridge University Press, Cambridge (1996)

    Chapter  Google Scholar 

  32. Koffka, K.: Principles of Gestalt Psychology. Routledge and Kegan Paul, London (1935)

    Google Scholar 

  33. Leeuwenberg, E., van der Helm, P.A.: Structural Information Theory. The Simplicity of Visual Form. Cambridge University Press, Cambridge (2012)

    Book  Google Scholar 

  34. Leung, T., Malik, J.: Contour continuity in region based image segmentation. In: Computer Vision ECCV, pp. 544–559. Springer (1998)

  35. van Lier, R., van der Helm, P., Leeuwenberg, E.: Integrating global and local aspects of visual occlusion. Perception 23, 883–903 (1994)

    Article  Google Scholar 

  36. Mansfield, A., Prasad, M., Rother, C., Sharp, T., Kohli, P., van Gool, L.: Transforming image completion. In: Proceedings of BMVC, pp. 121.1–121.11 (2011)

  37. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proc. 8th ICCV, vol. 2, pp. 416–423 (2001)

  38. Masnou, S.: Disocclusion: a variational approach using level lines. IEEE Trans. IP 11(2), 68–76 (2002)

    MathSciNet  Google Scholar 

  39. Masnou, S., Morel, J.M.: Level lines based disocclusion. In: Proc. of IEEE ICIP (1998)

  40. Meijster, A., Roerdink, J.B., Hesselink, W.H.: A general algorithm for computing distance transforms in linear time. Mathematical Morphology and Its Applications to Image and Signal Processing, pp. 331–340. Springer, New York (2002)

    Chapter  Google Scholar 

  41. Merriman, B., Bence, J., Osher, S.: Diffusion generated motion by mean curvature. In: J.E. Taylor, editor, Computational Crystal Growers Workshop, pp. 73–83. American Mathematical Society, Providence, Rhode Island (1992). Also available as UCLA CAM Report 92-18, April1992

  42. Michotte, A., Thines, G., Crabbé, G.: Amodal completion of perceptual structures. Michottes Experimental Phenomenology of Perception, pp. 140–167. Erlbaum, Hillsdale (1991)

    Google Scholar 

  43. Moravec, L., Beck, J.: Amodal completion: simplicity is not the explanation. Bull. Psychon. Soc. 24(4), 269–272 (1986)

    Article  Google Scholar 

  44. Mumford, D.: Elastica and computer vision. Algebraic Geometry and Its Applications, pp. 491–506. Springer, New York (1994)

    Chapter  Google Scholar 

  45. Murray, M., Foxe, D., Javitt, D., Foxe, J.: Setting boundaries: brain dynamics of modal and amodal illusory shape completion in humans. J. Neurosci. 24(31), 6898–6903 (2004)

    Article  Google Scholar 

  46. Nitzberg, M., Mumford, D., Shiota, T.: Filtering, Segmentation, and Depth. Lecture Notes in Computer Science, vol. 662. Springer, Berlin (1993)

    Book  MATH  Google Scholar 

  47. Otero, I.R., Delbracio, M.: Computing an exact Gaussian scale-space. Image Process. Line 6, 8–26 (2016)

    Article  Google Scholar 

  48. Ringach, D., Shapley, R.: Spatial and temporal properties of illusory contours and amodal boundary completion. Vis. Res. 36(19), 3037–3050 (1996)

    Article  Google Scholar 

  49. Rubin, N.: The role of junctions in surface completion and contour matching. Perception 30(3), 339–366 (2001)

    Article  Google Scholar 

  50. Esedoglu, S., S.R., Tsai, R.: Threshold dynamics for shape reconstruction and disocclusion. In: Proc of IEEE ICIP, vol 2, pp. 502–505 (2005)

  51. Sekuler, A.B.: Local and global minima in visual completion: effects of symmetry and orientation. Perception 23, 529–529 (1994)

    Article  Google Scholar 

  52. Sethian, J.A.: Curvature and the evolution of fronts. Commun. Math. Phys. 101(4), 487–499 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  53. Singh, M., Hoffman, D.: Completing visual contours: the relationship between relatability and minimizing inflections. Percept. Psychophys. 61(5), 943–951 (1999)

    Article  Google Scholar 

  54. Thornber, K.K., Williams, L.R.: Characterizing the distribution of completion shapes with corners using a mixture of random processes. Pattern Recogn. 33(4), 543–553 (2000)

    Article  Google Scholar 

  55. Van Lier, R., Van der Helm, P., Leeuwenberg, E.: Competing global and local completions in visual occlusion. J. Exp. Psychol.: Hum. Percept. Perform. 21(3), 571 (1995)

    Google Scholar 

  56. Van Lier, R., Leeuwenberg, E., Van der Helm, P.: Multiple completions primed by occlusion patterns. Perception 24, 727–727 (1995)

    Article  Google Scholar 

  57. Wertheimer, M.: Untersuchungen zur Lehre von der Gestalt, II [Investigations in Gestalt Theory, II: Laws of organization in perceptual forms], pp. 301–350. Psychologische Forschung (1923)

  58. Wexler, Y., Shechtman, E., Irani, M.: Space-time completion of video. IEEE Trans. PAMI 29(3), 463–476 (2007)

    Article  Google Scholar 

  59. Williams, L.R., Jacobs, D.W.: Stochastic completion fields: a neural model of illusory contour shape and salience. Neural Comput. 9(4), 837–858 (1997)

    Article  Google Scholar 

Download references

Acknowledgments

The first, second and fifth authors acknowledge partial support by MICINN Project, Reference MTM2012-30772, and by GRC Reference 2014 SGR 1301, Generalitat de Catalunya. The third author is supported by a Beatriu de Pinós fellowship (Marie-Curie COFUND action).

Author information

Authors and Affiliations

  1. Universitat Pompeu Fabra, Barcelona, Spain

    Maria Oliver, Gloria Haro, Baptiste Mazin & Coloma Ballester

  2. Centre de Visió per Computador, Barcelona, Spain

    Mariella Dimiccoli

Authors
  1. Maria Oliver
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gloria Haro
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mariella Dimiccoli
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Baptiste Mazin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Coloma Ballester
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Oliver.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Oliver, M., Haro, G., Dimiccoli, M. et al. A Computational Model for Amodal Completion. J Math Imaging Vis 56, 511–534 (2016). https://doi.org/10.1007/s10851-016-0652-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-016-0652-x

Keywords

  • Perception
  • Visual completion
  • Disocclusion
  • Bayesian model
  • Relatability
  • Euler elastica