Skip to main content
SpringerLink
Log in

Stitching Videos from a Fisheye Lens Camera and a Wide-Angle Lens Camera for Telepresence Robots

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

Many telepresence robots are equipped with a forward-facing camera for video communication and a downward-facing camera for navigation. In this paper, we propose to stitch videos from the FF-camera with a wide-angle lens and the DF-camera with a fisheye lens for telepresence robots. We aim at providing a compact and efficient user interface for telepresence robots with user-friendly interactive experiences. To this end, we present a multi-homography-based video stitching method which stitches videos from a wide-angle camera and a fisheye camera. The method consists of video image alignment, seam cutting, and image blending. We directly align the wide-angle video image and the fisheye video image based on the multi-homography alignment without calibration, distortion correction, and unwarping procedures. Thus, we can obtain a stitched video with shape preservation in the non-overlapping regions and alignment in the overlapping area for telepresence. To alleviate ghosting effects caused by moving objects and/or moving cameras during telepresence robot driving, an optimal seam is found for aligned video composition, and the optimal seam will be updated in subsequent frames, considering spatial and temporal coherence. The final stitched video is created by image blending based on the optimal seam. We conducted a user study to demonstrate the effectiveness of our method and the superiority of user interfaces with a stitched video.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. https://www.orah.co/software/videostitch-studio/.

  2. http://www.kolor.com/autopano/.

References

  1. Bae I (2018) Public acceptance of fundamental rights via a telepresence robot and a video call stand in south korea. Int J Social Robot 10(4):503–517

    Article  Google Scholar 

  2. Bay H, Tuytelaars T, Gool LV (2006) Surf: speeded up robust features. In: Proceedings of the European conference on computer vision, pp 404–417

  3. Brown M, Lowe DG (2007) Automatic panoramic image stitching using invariant features. Int J Comput Vis 74(1):59–73

    Article  Google Scholar 

  4. Burt PJ, Adelson EH (1983) A multiresolution spline with application to image mosaics. ACM Trans Graph 2(4):217–236

    Article  Google Scholar 

  5. Byröd M, Brown MA, Åström K (2009) Minimal solutions for panoramic stitching with radial distortion. In: Proceedings of British machine vision conference, pp 1–11

  6. Cesta A, Cortellessa G, Orlandini A, Tiberio L (2016) Long-term evaluation of a telepresence robot for the elderly: methodology and ecological case study. Int J Soc Robot 8(3):421–441

    Article  Google Scholar 

  7. Chang CH, Sato Y, Chuang YY (2014) Shape-preserving half-projective warps for image stitching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3254–3261

  8. Chang CH, Chou CN, Chang EY (2017) Clkn: cascaded Lucas–Kanade networks for image alignment. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2213–2221

  9. Chin T, Yu J, Suter D (2012) Accelerated hypothesis generation for multistructure data via preference analysis. IEEE Trans Pattern Anal Mach Intell 34(4):625–638

    Article  Google Scholar 

  10. DeTone D, Malisiewicz T, Rabinovich A (2016) Deep image homography estimation. arXiv preprint arXiv:1606.03798

  11. Double (2017) Double robotics. http://www.doublerobotics.com

  12. El-Saban MA, Refaat M, Kaheel A, Abdul-Hamid A (2009) Stitching videos streamed by mobile phones in real-time. In: Proceedings of the 17th ACM international conference on multimedia, pp 1009–1010

  13. Erlik Nowruzi F, Laganiere R, Japkowicz N (2017) Homography estimation from image pairs with hierarchical convolutional networks. In: Proceedings of the IEEE international conference on computer vision workshops, pp 913–920

  14. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  15. Gao J, Kim SJ, Brown MS (2011) Constructing image panoramas using dual-homography warping. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 49–56

  16. Gu H, Yu Y, Sun W (2009) A new optimal seam selection method for airborne image stitching. In: Proceedings of the IEEE international workshop on imaging systems and techniques, pp 159–163

  17. Guo H, Liu S, He T, Zhu S, Zeng B, Gabbouj M (2016) Joint video stitching and stabilization from moving cameras. IEEE Trans Image Process 25(11):5491–5503

    Article  MathSciNet  Google Scholar 

  18. Hartley R, Zisserman A (2003) Multiple view geometry in computer vision. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  19. He B, Yu S (2015) Parallax-robust surveillance video stitching. Sensors 16(1):7

    Article  Google Scholar 

  20. Ho T, Budagavi M (2017) Dual-fisheye lens stitching for 360-degree imaging. In: Proceedings of the IEEE international conference on acoustics, speech and signal processing, pp 2172–2176

  21. Jia Y, Xu B, Shen J, Pei M, Dong Z, Hou J, Yang M (2015) Telepresence interaction by touching live video images. arXiv preprint arXiv:1512.04334

  22. Jiang W, Gu J (2015) Video stitching with spatial-temporal content-preserving warping. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 42–48

  23. Jin H (2008) A three-point minimal solution for panoramic stitching with lens distortion. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–8

  24. Ju MH, Kang HB (2013) Panoramic image generation with lens distortions. In: Proceedings of the 20th IEEE international conference on image processing, pp 1296–1300

  25. Ju MH, Kang HB (2014) Stitching images with arbitrary lens distortions. Int J Adv Rob Syst 11(1):2

    Article  Google Scholar 

  26. Kim BS, Choi KA, Park WJ, Kim SW, Ko SJ (2017) Content-preserving video stitching method for multi-camera systems. IEEE Trans Consum Electron 63(2):109–116

    Article  Google Scholar 

  27. Kristoffersson A, Coradeschi S, Loutfi A (2013) A review of mobile robotic telepresence. Adv Hum Comput Interact 2013:1–17

    Article  Google Scholar 

  28. Kukelova Z, Heller J, Bujnak M, Pajdla T (2015) Radial distortion homography. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 639–647

  29. Lee KY, Sim JY (2015) Robust video stitching using adaptive pixel transfer. In: Proceedings of the IEEE international conference on image processing, pp 813–817

  30. Lee WT, Chen HI, Chen MS, Shen I, Chen BY et al (2017) High-resolution 360 video foveated stitching for real-time vr. Comput Graph Forum 36(7):115–123

    Article  Google Scholar 

  31. Li J, Xu W, Zhang J, Zhang M, Wang Z, Li X (2015) Efficient video stitching based on fast structure deformation. IEEE Trans Cybern 45(12):2707–2719

    Article  Google Scholar 

  32. Li Y, Monga V (2016) Siasm: sparsity-based image alignment and stitching method for robust image mosaicking. In: Proceedings of the IEEE conference on image proceesing, pp 1828–1832

  33. Lin K, Liu S, Cheong LF, Zeng B (2016) Seamless video stitching from hand-held camera inputs. Comput Graph Forum 35(2):479–487

    Article  Google Scholar 

  34. Lin WY, Liu S, Matsushita Y, Ng TT, Cheong LF (2011) Smoothly varying affine stitching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 345–352

  35. Lou Z, Gevers T (2014) Image alignment by piecewise planar region matching. IEEE Trans Multimed 16(7):2052–2061

    Article  Google Scholar 

  36. Mayer S, Tschofen A, Dey AK, Mattern F (2014) User interfaces for smart things-a generative approach with semantic interaction descriptions. ACM Trans Comput Hum Interact 21(2):12

    Article  Google Scholar 

  37. Meng X, Wang W, Leong B (2015) Skystitch: a cooperative multi-uav-based real-time video surveillance system with stitching. In: Proceedings of the 23rd ACM international conference on multimedia, pp 261–270

  38. Meyer B (2016) Mobile devices and spatial enactments of learning: ipads in lower secondary schools. In: Proceedings of the 12th international conference on mobile learning, pp 3–10

  39. Neustaedter C, Venolia G, Procyk J, Hawkins D (2016) To beam or not to beam: a study of remote telepresence attendance at an academic conference. In: Proceedings of the 19th ACM conference on computer-supported cooperative work and social computing, pp 418–431

  40. Nguyen T, Chen SW, Skandan S, Taylor CJ, Kumar V (2018) Unsupervised deep homography: a fast and robust homography estimation model. IEEE Robot Autom Lett 3(3):2346–2353

    Article  Google Scholar 

  41. Nie Y, Su T, Zhang Z, Sun H, Li G (2018) Dynamic video stitching via shakiness removing. IEEE Trans Image Process 27(1):164–178

    Article  MathSciNet  Google Scholar 

  42. Niemelä M, van Aerschot L, Tammela A, Aaltonen I, Lammi H (2019) Towards ethical guidelines of using telepresence robots in residential care. Int J Soc Robot 1–9

  43. Okumura Ki, Raut S, Gu Q, Aoyama T, Takaki T, Ishii I (2013) Real-time feature-based video mosaicing at 500 fps. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, pp 2665–2670

  44. Peng Y, Xu W, Li J, Zhang M (2016) A fast and stable seam selection algorithm for video stitching based on seam temporal propagation constraint. In: Proceedings of the 3rd international conference on systems and informatics, pp 572–577

  45. Perazzi F, Sorkine-Hornung A, Zimmer H, Kaufmann P, Wang O, Watson S, Gross M (2015) Panoramic video from unstructured camera arrays. Comput Graph Forum 34(2):57–68

    Article  Google Scholar 

  46. Silva R, Feijó B, Gomes PB, Frensh T, Monteiro D (2016) Real time \(360^o\) video stitching and streaming. In: ACM SIGGRAPH 2016 posters, p 70

  47. Su T, Nie Y, Zhang Z, Sun H, Li G (2016) Video stitching for handheld inputs via combined video stabilization. In: Proceedings of SIGGRAPH ASIA technical briefs, p 25

  48. Szeliski R (2006) Image alignment and stitching: a tutorial. Found Trends® Comput Graph Vis 2(1):1–104

    MathSciNet  MATH  Google Scholar 

  49. Szeliski R, Shum HY (1997) Creating full view panoramic image mosaics and environment maps. In: Proceedings of the 24th international conference on computer graphics and interactive techniques, pp 251–258

  50. Technologies S (2017) Beam. http://suitabletech.com

  51. Tsui KM, Norton A, Brooks DJ, McCann E, Medvedev MS, Allspaw J, Suksawat S, Dalphond JM, Lunderville M, Yanco HA (2014) Iterative design of a semi-autonomous social telepresence robot research platform: a chronology. Intel Serv Robot 7(2):103–119

    Article  Google Scholar 

  52. Xu B, Jia Y (2017) Wide-angle image stitching using multi-homography warping. In: Proceedings of the IEEE international conference on image processing, pp 1467–1471

  53. Zaragoza J, Chin TJ, Brown MS, Suter D (2013) As-projective-as-possible image stitching with moving dlt. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2339–2346

  54. Zhu Z, Lu J, Wang M, Zhang S, Martin R, Liu H, Hu S (2016) A comparative study of algorithms for realtime panoramic video blending. arXiv preprint arXiv:1606.00103

Download references

Author information

Authors and Affiliations

  1. Tus College of Digit, Guangxi University of Science and Technology, Liuzhou, 545006, People’s Republic of China

    Yanmei Dong

  2. Beijing Laboratory of Intelligent Information Technology, School of Computer Science, Beijing Institute of Technology, Beijing, 100081, People’s Republic of China

    Yanmei Dong, Mingtao Pei, Lijia Zhang, Bin Xu, Yuwei Wu & Yunde Jia

Authors
  1. Yanmei Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingtao Pei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lijia Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bin Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuwei Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yunde Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuwei Wu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Funding

This work was supported by the Natural Science Foundation of China (NSFC) under Grants No. 61773062 and No. 61702037.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dong, Y., Pei, M., Zhang, L. et al. Stitching Videos from a Fisheye Lens Camera and a Wide-Angle Lens Camera for Telepresence Robots. Int J of Soc Robotics 14, 733–745 (2022). https://doi.org/10.1007/s12369-020-00744-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-020-00744-8

Keywords

Search

Navigation