Advertisement

Object-Centered Image Stitching

  • Charles Herrmann
  • Chen Wang
  • Richard Strong Bowen
  • Emil Keyder
  • Ramin ZabihEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11207)

Abstract

Image stitching is typically decomposed into three phases: registration, which aligns the source images with a common target image; seam finding, which determines for each target pixel the source image it should come from; and blending, which smooths transitions over the seams. As described in [1], the seam finding phase attempts to place seams between pixels where the transition between source images is not noticeable. Here, we observe that the most problematic failures of this approach occur when objects are cropped, omitted, or duplicated. We therefore take an object-centered approach to the problem, leveraging recent advances in object detection [2, 3, 4]. We penalize candidate solutions with this class of error by modifying the energy function used in the seam finding stage. This produces substantially more realistic stitching results on challenging imagery. In addition, these methods can be used to determine when there is non-recoverable occlusion in the input data, and also suggest a simple evaluation metric that can be used to evaluate the output of stitching algorithms.

Notes

Acknowledgements

This research has been supported by NSF grants IIS-1161860 and IIS-1447473 and by a Google Faculty Research Award. We also thank Connie Choi for help collecting images.

References

  1. 1.
    Szeliski, R.: Image alignment and stitching: a tutorial. Found. Trends Comput. Graph. Vis. 2(1), 1–104 (2007)CrossRefGoogle Scholar
  2. 2.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. The MIT Press, Cambridge (2016)zbMATHGoogle Scholar
  3. 3.
    Liu, W., et al.: SSD: single shot MultiBox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46448-0_2CrossRefGoogle Scholar
  4. 4.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. TPAMI 39(6), 1137–1149 (2017)CrossRefGoogle Scholar
  5. 5.
    Szeliski, R.: Computer Vision: Algorithms and Applications. Springer, Berlin (2010).  https://doi.org/10.1007/978-3-642-12848-6CrossRefzbMATHGoogle Scholar
  6. 6.
    Kwatra, V., Schödl, A., Essa, I., Turk, G., Bobick, A.: Graphcut textures: image and video synthesis using graph cuts. SIGGRAPH 22(3), 277–286 (2003)CrossRefGoogle Scholar
  7. 7.
    Li, N., Liao, T., Wang, C.: Perception-based seam cutting for image stitching. Signal Image Video Process. 12, 967–974 (2018)CrossRefGoogle Scholar
  8. 8.
    Zaragoza, J., Chin, T.J., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. In: CVPR, pp. 2339–2346 (2013)Google Scholar
  9. 9.
    Chen, Y.-S., Chuang, Y.-Y.: Natural image stitching with the global similarity prior. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 186–201. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46454-1_12CrossRefGoogle Scholar
  10. 10.
    Herrmann, C., et al.: Robust image stitching using multiple registrations. In: ECCV (2018)Google Scholar
  11. 11.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. TPAMI 23(11), 1222–1239 (2001)CrossRefGoogle Scholar
  12. 12.
    Szeliski, R., et al.: A comparative study of energy minimization methods for Markov random fields. TPAMI 30(6), 1068–1080 (2008)CrossRefGoogle Scholar
  13. 13.
    Lin, C.C., Pankanti, S.U., Natesan Ramamurthy, K., Aravkin, A.Y.: Adaptive as-natural-as-possible image stitching. In: CVPR, pp. 1155–1163 (2015)Google Scholar
  14. 14.
    Lin, K., Jiang, N., Cheong, L.-F., Do, M., Lu, J.: SEAGULL: seam-guided local alignment for parallax-tolerant image stitching. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 370–385. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46487-9_23CrossRefGoogle Scholar
  15. 15.
    Agarwala, A., et al.: Interactive digital photomontage. SIGGRAPH 23(3), 294–302 (2004)CrossRefGoogle Scholar
  16. 16.
    Ozawa, T., Kitani, K.M., Koike, H.: Human-centric panoramic imaging stitching. In: Augmented Human International Conference, pp. 20:1–20:6 (2012)Google Scholar
  17. 17.
    Flores, A., Belongie, S.: Removing pedestrians from google street view images. In: IEEE International Workshop on Mobile Vision, pp. 53–58 (2010)Google Scholar
  18. 18.
    Perez, P., Gangnet, M., Blake, A.: Poisson image editing. In: SIGGRAPH, pp. 313–318 (2003)Google Scholar
  19. 19.
    Liu, F., Gleicher, M., Jin, H., Agarwala, A.: Content-preserving warps for 3D video stabilization. SIGGRAPH 28(3), 44 (2009)Google Scholar
  20. 20.
    Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: CVPR, pp. 3262–3269 (2014)Google Scholar
  21. 21.
    Kolmogorov, V., Rother, C.: Minimizing nonsubmodular functions with graph cuts-a review. TPAMI 29(7), 1274–1279 (2007)CrossRefGoogle Scholar
  22. 22.
    Lowe, D.: Object recognition from local scale-invariant features. In: ICCV, pp. 1150–1157 (1999)Google Scholar
  23. 23.
    Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: large displacement optical flow with deep matching. In: ICCV, pp. 1385–1392 (2013)Google Scholar
  24. 24.
    Wang, Z., Simoncelli, E., Bovik, A.: Multiscale structural similarity for image quality assessment. In: Asilomar Conference on Signals, Systems and Computers, pp. 1398–1402 (2004)Google Scholar
  25. 25.
    Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. IJCV 74(1), 59–73 (2007)CrossRefGoogle Scholar
  26. 26.
    Agarwal, S., Mierle, K., et al.: Ceres solver. http://ceres-solver.org. Accessed 25 Jul 2018

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Charles Herrmann
    • 1
  • Chen Wang
    • 1
    • 2
  • Richard Strong Bowen
    • 1
  • Emil Keyder
    • 2
  • Ramin Zabih
    • 1
    • 2
    Email author
  1. 1.Cornell TechNew YorkUSA
  2. 2.Google ResearchNew YorkUSA

Personalised recommendations