SocialSync: Sub-Frame Synchronization in a Smartphone Camera Network

  • Richard Latimer
  • Jason Holloway
  • Ashok Veeraraghavan
  • Ashutosh Sabharwal
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8926)

Abstract

SocialSync is a sub-frame synchronization protocol for capturing images simultaneously using a smartphone camera network. By synchronizing image captures to within a frame period, multiple smartphone cameras, which are often in use in social settings, can be used for a variety of applications including light field capture, depth estimation, and free viewpoint television. Currently, smartphone camera networks are limited to capturing static scenes due to motion artifacts caused by frame misalignment. Because frame misalignment in smartphones camera networks is caused by variability in the camera system, we characterize frame capture on mobile devices by analyzing the statistics of camera setup latency and frame delivery within an Android app. Next, we develop the SocialSync protocol to achieve sub-frame synchronization between devices by estimating frame capture timestamps to within millisecond accuracy. Finally, we demonstrate the effectiveness of SocialSync on mobile devices by reducing motion-induced artifacts when recovering the light field.

Keywords

Multiple viewpoints Camera array Camera network Synchronization Smartphone Mobile device 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Supplementary material

336126_1_En_43_MOESM1_ESM.zip (379 kb)
Supplementary material (ZIP 380 KB)

References

  1. 1.
    Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S.M., Szeliski, R.: Building rome in a day. Communications of the ACM 54(10), 105–112 (2011)CrossRefGoogle Scholar
  2. 2.
    Bouguet, J.Y.: Camera calibration toolbox for matlab (2008)Google Scholar
  3. 3.
    Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(9), 1124–1137 (2004)Google Scholar
  4. 4.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(11), 1222–1239 (2001)CrossRefGoogle Scholar
  5. 5.
    Buehler, C., Bosse, M., McMillan, L., Gortler, S., Cohen, M.: Unstructured lumigraph rendering. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 425–432. ACM (2001)Google Scholar
  6. 6.
    Debevec, P.E., Taylor, C.J., Malik, J.: Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 11–20. ACM (1996)Google Scholar
  7. 7.
    Facebook, Ericsson, Qualcomm: A focus on efficiency. Tech. rep. (September 2013) http://internet.org white paper
  8. 8.
    Georgiev, T., Yu, Z., Lumsdaine, A., Goma, S.: Lytro camera technology: theory, algorithms, performance analysis. In: IS&T/SPIE Electronic Imaging, pp. 86671J–86671J. International Society for Optics and Photonics (2013)Google Scholar
  9. 9.
    Gupta, A., Cozza, R., Lu, C.: Market share analysis: Mobile phones, worldwide, 4q13 and 2013. Tech. rep., Gartner, Inc. (February 2014) (white paper)Google Scholar
  10. 10.
    Heptagon Advanced Micro Optics. http://www.hptg.com/products/imaging (2014), (Online accessed March 31, 2014)
  11. 11.
    Kolmogorov, V., Zabin, R.: What energy functions can be minimized via graph cuts? IEEE Transactions on Pattern Analysis and Machine Intelligence 26(2), 147–159 (2004)CrossRefGoogle Scholar
  12. 12.
    Latimer, R., Holloway, J., Veeraraghavan, A., Sabharwal, A.: Supplementary material for SocialSync: Sub-frame synchronization in a smartphone camera network (2014), Computer Vision-ECCV 2014. LF4CV submission. Supplied as additional materialGoogle Scholar
  13. 13.
    Lourakis, M.A., Argyros, A.: SBA: A software package for generic sparse bundle adjustment. ACM Trans. Math. Software 36(1), 1–30 (2009)CrossRefMathSciNetGoogle Scholar
  14. 14.
    Marwah, K., Wetzstein, G., Bando, Y., Raskar, R.: Compressive light field photography using overcomplete dictionaries and optimized projections. ACM Transactions on Graphics (TOG) 32(4), 46 (2013)CrossRefGoogle Scholar
  15. 15.
    Mills, D.L.: Network time protocol (ntp). Network (1985)Google Scholar
  16. 16.
    Mills, D.L.: Computer Time Synchronization: The Network Time Protocol on Earth and in Space, 2 edn. CRC Press (2010)Google Scholar
  17. 17.
    Naemura, T., Tago, J., Harashima, H.: Real-time video-based modeling and rendering of 3d scenes. IEEE Computer Graphics and Applications 22(2), 66–73 (2002)Google Scholar
  18. 18.
    Nayar, S., Ben-Ezra, M.: Motion-based motion deblurring. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(6), 689–698 (2004)CrossRefGoogle Scholar
  19. 19.
    Shechtman, E., Caspi, Y., Irani, M.: Space-time super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(4), 531–545 (2005)CrossRefGoogle Scholar
  20. 20.
    Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: Exploring photo collections in 3d. In: SIGGRAPH Conference Proceedings, pp. 835–846. ACM Press, New York (2006)Google Scholar
  21. 21.
    Tanimoto, M.: Overview of free viewpoint television. Signal Processing: Image Communication 21(6), 454–461 (2006)Google Scholar
  22. 22.
    Tsai, R.Y.: A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses. IEEE Journal of Robotics and Automation 3(4), 323–344 (1987)CrossRefGoogle Scholar
  23. 23.
    Venkataraman, K., Lelescu, D., Duparré, J., McMahon, A., Molina, G., Chatterjee, P., Mullis, R., Nayar, S.: Picam: an ultra-thin high performance monolithic camera array. ACM Transactions on Graphics (TOG) 32(6), 166 (2013)CrossRefGoogle Scholar
  24. 24.
    Wilburn, B., Joshi, N., Vaish, V., Levoy, M., Horowitz, M.: High-speed videography using a dense camera array. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2004, vol. 2, pp. II-294. IEEE (2004)Google Scholar
  25. 25.
    Wilburn, B., Joshi, N., Vaish, V., Talvala, E.V., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M.: High performance imaging using large camera arrays. ACM Transactions on Graphics (TOG) 24(3), 765–776 (2005)CrossRefGoogle Scholar
  26. 26.
    Zhang, C., Chen, T.: A self-reconfigurable camera array. In: ACM SIGGRAPH 2004 Sketches. p. 151. ACM (2004)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Richard Latimer
    • 1
  • Jason Holloway
    • 1
  • Ashok Veeraraghavan
    • 1
  • Ashutosh Sabharwal
    • 1
  1. 1.Rice UniversityHoustonUSA

Personalised recommendations