High-accuracy, high-speed 3D structured light imaging techniques and potential applications to intelligent robotics

Abstract

This paper presents some of the high-accuracy and high-speed structured light 3D imaging methods developed in the optical metrology community. These advanced 3D optical imaging technologies could substantially benefit the intelligent robotics community as another sensing tool. This paper mainly focuses on one special 3D imaging technique: the digital fringe projection (DFP) method because of its numerous advantageous features compared to other 3D optical imaging methods in terms of accuracy, resolution, speed, and flexibility. We will discuss technologies that enabled 3D data acquisition, reconstruction, and display at 30 Hz or higher with over 300,000 measurement points per frame. This paper intends to introduce the DFP technologies to the intelligent robotics community, and casts our perspectives on potential applications for which such sensing methods could be of value.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

References

  1. Abbott, J., Nagy, Z., Beyeler, F., Nelson, B.: Robotics in the small. IEEE Robot. Autom. Mag. 14, 92–103 (2007)

    Article  Google Scholar 

  2. Bell, T., Zhang, S.: Towards superfast three-dimensional optical metrology with digital micromirrror device (dmd) platforms. Opt. Eng. 53(11), 112206 (2014)

  3. Cappelleri, D., Efthymiou, D., Goswami, A., Vitoroulis, N., Zavlanos, M.: Towards mobile microrobot swarms for additive micromanufacturing. Int. J. Adv. Robot. Syst. 11(9), 150 (2014).

    Article  Google Scholar 

  4. Cheang, U.K., Lee, K., Julius, A.A., Kim, M.J.: Multiple-robot drug delivery strategy through coordinated teams of microswimmers. Appl. Phys. Lett. 105(8), 083705 (2014)

  5. Cheng, Y.Y., Wyant, J.C.: Two-wavelength phase shifting interferometry. Appl. Opt. 23, 4539–4543 (1984)

    Article  Google Scholar 

  6. Cheng, Y.Y., Wyant, J.C.: Multiple-wavelength phase shifting interferometry. Appl. Opt. 24, 804–807 (1985)

    Article  Google Scholar 

  7. Chowdhury, S., Jing, W., Cappelleri, D.J.: Controlling multiple microrobots: recent progress and future challenges. J. Micro Bio Robot. 10(1), 1–11 (2015a)

  8. Chowdhury, S., Jing, W., Cappelleri, D.J.: Towards independent control of multiple magnetic mobile microrobots. Micromachines 7(1), 3 (2015b)

  9. Chowdhury, S., Jing, W., Jaron, P., Cappelleri, D.: Path planning and control for autonomous navigation of single and multiple magnetic mobile microrobots. In: Proceedings of the ASME 2015 International Design Engineering Technical Conferences & Computers and Informatio in Engineering Conference. Boston, Massachusetts (2015c)

  10. DeVon, D., Bretl, T.: Control of many robots moving in the same direction with different speeds: a decoupling approach. In: American Control Conference. ACC’09., pp. 1794–1799. IEEE (2009)

  11. Dhond, U.R., Aggarwal, J.K.: Structure from stereo-a review. IEEE Trans. Syst. Man. Cybern. 19(6), 1489–1510 (1989)

  12. Diller, E., Floyd, S., Pawashe, C., Sitti, M.: Control of multiple heterogeneous magnetic microrobots in two dimensions on nonspecialized surfaces. IEEE Trans. Robot. 28(1), 172–182 (2012)

    Article  Google Scholar 

  13. Diller, E., Sitti, M., et al.: Micro-scale mobile robotics. Found. Trends Robot. 2(3), 143–259 (2013)

    Article  Google Scholar 

  14. Floyd, S., Pawashe, C., Sitti, M.: An untethered magnetically actuated micro-robot capable of motion on arbitrary surfaces. In: Robotics and Automation, 2008. ICRA 2008. IEEE International Conference on, pp. 419–424. IEEE (2008)

  15. Frutiger, D.R., Vollmers, K., Kratochvil, B.E., Nelson, B.J.: Small, fast, and under control: wireless resonant magnetic micro-agents. Int. J. Robot. Res. 29(5), 613–636 (2010)

    Article  Google Scholar 

  16. Fusco, S., Huang, H.W., Peyer, K.E., Peters, C., Hberli, M., Ulbers, A., Spyrogianni, A., Pellicer, E., Sort, J., Pratsinis, S.E., Nelson, B.J., Sakar, M.S., Pan, S.: Shape-switching microrobots for medical applications: The influence of shape in drug delivery and locomotion. ACS Appl. Mater. Interfaces 7(12), 6803–6811 (2015)

    Article  Google Scholar 

  17. Ghiglia, D.C., Pritt, M.D. (eds.): Two-Dimensional Phase Unwrapping: Theory, Algorithms, and Software. Wiley, New York (1998)

    Google Scholar 

  18. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000). ISBN:0521623049

  19. Huang, H.W., Sakar, M.S., Riederer, K., Shamsudhin, N., Petruska, A., Pan, S., Nelson, B.J.: Magnetic microrobots with addressable shape control. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 1719–1724 (2016)

  20. Huang, P.S., Zhang, S.: Fast three-step phase-shifting algorithm. Appl. Opt. 45(21), 5086–5091 (2006)

    Article  Google Scholar 

  21. Huang, P.S., Zhang, S., Chiang, F.P.: Trapezoidal phase-shifting method for three-dimensional shape measurement. Opt. Eng. 44(12), 123601 (2005)

  22. Huang, Y., Shang, Y., Liu, Y., Bao, H.: Handbook of 3D Machine Vision: Optical Metrology and Imaging, 1st edn., chap. 3D shapes from Speckle, pp. 33–56. CRC, USA (2013)

  23. Jia, P., Kofman, J., English, C.: Two-step triangular-pattern phase-shifting method for three-dimensional object-shape measurement. Opt. Eng. 46(8), 083201 (2007)

  24. Jing, W., Cappelleri, D.: Incorporating in-situ force sensing capabilities in a magnetic microrobot. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014), pp. 4704–4709 (2014a)

  25. Jing, W., Cappelleri, D.: A magnetic microrobot with in situ force sensing capabilities. Robotics 3(2), 106–119 (2014b)

  26. Jing, W., Cappelleri, D.J.: Towards functional mobile magnetic microrobots. In: Small-Scale Robotics. From Nano-to-Millimeter-Sized Robotic Systems and Applications, pp. 81–100. Springer, New York (2014c)

  27. Jing, W., Chen, X., Lyttle, S., Fu, Z., Shi, Y., Cappelleri, D.: A magnetic thin film microrobot with two operating modes. In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp. 96–101 (2011)

  28. Jing, W., Pagano, N., Cappelleri, D.J.: A micro-scale magnetic tumbling microrobot. In: ASME 2012 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, pp. 187–196 (2012)

  29. Jing, W., Pagano, N., Cappelleri, D.J.: A novel micro-scale magnetic tumbling microrobot. J. Micro Bio Robot. 8(1), 1–12 (2013a)

  30. Jing, W., Pagano, N., Cappelleri, D.J.: A tumbling magnetic microrobot with flexible operating modes. In: 2013 IEEE International Conference on Robotics and Automation (ICRA), pp. 5514–5519 (2013b)

  31. Karpinsky, N., Hoke, M., Chen, V., Zhang, S.: High-resolution, real-time three-dimensional shape measurement on graphics processing unit. Opt. Eng. 53(2), 024105 (2014)

  32. Keferstein, C.P., Marxer, M.: Testing bench for laser triangulation sensors. Sensor Rev. 18(3), 183–187 (1998)

    Article  Google Scholar 

  33. Kummer, M.P., Abbott, J.J., Kratochvil, B.E., Borer, R., Sengul, A., Nelson, B.J.: Octomag: An electromagnetic system for 5-dof wireless micromanipulation. Robot. IEEE Trans. 26(6), 1006–1017 (2010)

    Article  Google Scholar 

  34. Lei, S., Zhang, S.: Flexible 3-d shape measurement using projector defocusing. Opt. Lett. 34(20), 3080–3082 (2009)

    Article  Google Scholar 

  35. Li, B., Wang, Y., Dai, J., Lohry, W., Zhang, S.: Some recent advances on superfast 3d shape measurement with digital binary defocusing techniques. Opt. Laser Eng. 54, 236–246 (2014)

    Article  Google Scholar 

  36. Li, B., Zhang, S.: Microscopic structured light 3d profilometry: Binary defocusing technique vs. sinusoidal fringe projection. Opt. Laser Eng. (2016) (in press)

  37. Liu, K., Wang, Y., Lau, D.L., Hao, Q., Hassebrook, L.G.: Dual-frequency pattern scheme for high-speed 3-d shape measurement. Opt. Express 18, 5229–5244 (2010)

    Article  Google Scholar 

  38. Malacara, D. (ed.): Optical Shop Testing, 3rd edn. Wiley, New York (2007)

    Google Scholar 

  39. Manneberg, G., Hertegard, S., Liljencrantz, J.: Measurement of human vocal fold vibrations with laser triangulation. Opt. Eng. 40(9), 2041–2044 (2001)

    Article  Google Scholar 

  40. Morano, R.A., Ozturk, C., Conn, R., Dubin, S., Zietz, S., Nissanov, J.: Structured light using pseudorandom codes. IEEE Trans. Pattern Anal. Mach. Intell. 20, 322–327 (1998)

  41. Palagi, S., Mark, A.G., Reigh, S.Y., Melde, K., Qiu, T., Zeng, H., Parmeggiani, C., Martella, D., Sanchez-Castillo, A., Kapernaum, N., Giesselmann, F., Wiersma, D.S., Lauga, E., Fischer, P.: Structured light enables biomimetic swimming and versatile locomotion of photoresponsive soft microrobots. Nat. Mater. 15(6), 647–653 (2016)

  42. Pan, J., Huang, P., Zhang, S., Chiang, F.P.: Color n-ary gray code for 3-d shape measurement. In: 12th International Conference on Experimental Mechanics. Politecnico di Bari, Italy (2004)

  43. Pawashe, C., Floyd, S., Sitti, M.: Modeling and experimental characterization of an untethered magnetic micro-robot. Int. J. Robot. Res. 28(8), 1077–1094 (2009a)

  44. Pawashe, C., Floyd, S., Sitti, M.: Multiple magnetic microrobot control using electrostatic anchoring. Appl. Phys. Lett. 94(16), 164108 (2009b)

  45. Salvi, J., Fernandez, S., Pribanic, T., Llado, X.: A state of the art in structured light patterns for surface profilometry. Patt. Recogn. 43(8), 2666–2680 (2010)

    Article  MATH  Google Scholar 

  46. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comp. Vis. 47(1–3), 7–42 (2002)

    Article  MATH  Google Scholar 

  47. Sirikasemlert, A., Tao, X.: Objective evaluation of textural changes in knitted fabrics by laser triangulation. Text. Res. J. 70(12), 1076–1087 (2000)

    Article  Google Scholar 

  48. Steager, E.B., Sakar, M.S., Magee, C., Kennedy, M., Cowley, A., Kumar, V.: Automated biomanipulation of single cells using magnetic microrobots. Int. J. Robot. Res. 32(3), 346–359 (2013)

    Article  Google Scholar 

  49. Towers, D.P., Jones, J.D.C., Towers, C.E.: Optimum frequency selection in multi-frequency interferometry. Opt. Lett. 28, 1–3 (2003)

    Article  Google Scholar 

  50. Wang, Y., Laughner, J.I., Efimov, I.R., Zhang, S.: 3d absolute shape measurement of live rabbit hearts with a superfast two-frequency phase-shifting technique. Opt. Express 21(5), 5822–5632 (2013)

    Article  Google Scholar 

  51. Wang, Y., Zhang, S.: Superfast multifrequency phase-shifting technique with optimal pulse width modulation. Opt. Express 19(6), 5143–5148 (2011)

    Article  Google Scholar 

  52. Xu, J., Zhao, J., Luo, R.: A 3d imaging sensor for mobile manipulation. IEEE International Conference on Cyber Technology in Automation. Control and Intelligent Systems (CYBER), pp. 315–320. Bangkok, Thailand (2012)

  53. Zhang, C., Xu, J., Xi, N., Jia, Y., Li, W.: Development of an omni-directional 3d camera for robot navigation. In: IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pp. 262–267. Kaohsiung, Taiwan (2012)

  54. Zhang, S.: Recent progresses on real-time 3-d shape measurement using digital fringe projection techniques. Opt. Laser Eng. 48(2), 149–158 (2010)

    Article  Google Scholar 

  55. Zhang, S.: High-speed 3D imaging with digital fringe projection technique, 1st edn. CRC, New York (2016)

    Google Scholar 

  56. Zhang, S., Huang, P.S.: High-resolution real-time three-dimensional shape measurement. Opt. Eng. 45(12), 123601 (2006)

  57. Zhang, S., Huang, P.S.: Novel method for structured light system calibration. Opt. Eng. 45(8), 083601 (2006)

  58. Zhang, S., Li, X., Yau, S.T.: Multilevel quality-guided phase unwrapping algorithm for real-time three-dimensional shape reconstruction. Appl. Opt. 46(1), 50–57 (2007)

    Article  Google Scholar 

  59. Zhang, S., Royer, D., Yau, S.T.: Gpu-assisted high-resolution, real-time 3-d shape measurement. Opt. Express 14(20), 9120–9129 (2006)

    Article  Google Scholar 

  60. Zhang, S., van der Weide, D., Oliver, J.: Superfast phase-shifting method for 3-d shape measurement. Opt. Express 18(9), 9684–9689 (2010)

    Article  Google Scholar 

  61. Zhang, S., Yau, S.T.: High-resolution, real-time 3-d absolute coordinate measurement based on a phase-shifting method. Opt. Express 14(7), 2644–2649 (2006)

    Article  Google Scholar 

  62. Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)

    Article  Google Scholar 

  63. Zhang, Z.: Microsoft kinect sensor and its effect. IEEE Multimed. 19(2), 4–10 (2012)

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank many current and former students working in our laboratory for their tremendous work in advancing 3D imaging technologies to their current state. In particular, we would like to thank Dr. Yajun Wang for his development of binary dithering and optimal pulse width modulation methods; Dr. Nik Karpinsky for implementation of graphics processing unit (GPU) programming; Jae-Sang Hyun and Zexuan Zhu for their hardware system design and developments; and Maggie Hao, Chufan Jiang and Ziping Liu for serving as models to test our systems.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Song Zhang.

Additional information

This research was supported by National Science Foundation research Grant Numbers CMMI-1523048 and CMMI-1531048.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Li, B., An, Y., Cappelleri, D. et al. High-accuracy, high-speed 3D structured light imaging techniques and potential applications to intelligent robotics. Int J Intell Robot Appl 1, 86–103 (2017). https://doi.org/10.1007/s41315-016-0001-7

Download citation

Keywords

  • 3D optical sensing
  • 3D optical imaging
  • Micro robotics
  • Human robotic interaction
  • Perception and vision