High-accuracy, high-speed 3D structured light imaging techniques and potential applications to intelligent robotics

  • Beiwen Li
  • Yatong An
  • David Cappelleri
  • Jing Xu
  • Song ZhangEmail author
Regular Paper


This paper presents some of the high-accuracy and high-speed structured light 3D imaging methods developed in the optical metrology community. These advanced 3D optical imaging technologies could substantially benefit the intelligent robotics community as another sensing tool. This paper mainly focuses on one special 3D imaging technique: the digital fringe projection (DFP) method because of its numerous advantageous features compared to other 3D optical imaging methods in terms of accuracy, resolution, speed, and flexibility. We will discuss technologies that enabled 3D data acquisition, reconstruction, and display at 30 Hz or higher with over 300,000 measurement points per frame. This paper intends to introduce the DFP technologies to the intelligent robotics community, and casts our perspectives on potential applications for which such sensing methods could be of value.


3D optical sensing 3D optical imaging Micro robotics Human robotic interaction Perception and vision 



We would like to thank many current and former students working in our laboratory for their tremendous work in advancing 3D imaging technologies to their current state. In particular, we would like to thank Dr. Yajun Wang for his development of binary dithering and optimal pulse width modulation methods; Dr. Nik Karpinsky for implementation of graphics processing unit (GPU) programming; Jae-Sang Hyun and Zexuan Zhu for their hardware system design and developments; and Maggie Hao, Chufan Jiang and Ziping Liu for serving as models to test our systems.


  1. Abbott, J., Nagy, Z., Beyeler, F., Nelson, B.: Robotics in the small. IEEE Robot. Autom. Mag. 14, 92–103 (2007)CrossRefGoogle Scholar
  2. Bell, T., Zhang, S.: Towards superfast three-dimensional optical metrology with digital micromirrror device (dmd) platforms. Opt. Eng. 53(11), 112206 (2014)Google Scholar
  3. Cappelleri, D., Efthymiou, D., Goswami, A., Vitoroulis, N., Zavlanos, M.: Towards mobile microrobot swarms for additive micromanufacturing. Int. J. Adv. Robot. Syst. 11(9), 150 (2014).CrossRefGoogle Scholar
  4. Cheang, U.K., Lee, K., Julius, A.A., Kim, M.J.: Multiple-robot drug delivery strategy through coordinated teams of microswimmers. Appl. Phys. Lett. 105(8), 083705 (2014)Google Scholar
  5. Cheng, Y.Y., Wyant, J.C.: Two-wavelength phase shifting interferometry. Appl. Opt. 23, 4539–4543 (1984)CrossRefGoogle Scholar
  6. Cheng, Y.Y., Wyant, J.C.: Multiple-wavelength phase shifting interferometry. Appl. Opt. 24, 804–807 (1985)CrossRefGoogle Scholar
  7. Chowdhury, S., Jing, W., Cappelleri, D.J.: Controlling multiple microrobots: recent progress and future challenges. J. Micro Bio Robot. 10(1), 1–11 (2015a)Google Scholar
  8. Chowdhury, S., Jing, W., Cappelleri, D.J.: Towards independent control of multiple magnetic mobile microrobots. Micromachines 7(1), 3 (2015b)Google Scholar
  9. Chowdhury, S., Jing, W., Jaron, P., Cappelleri, D.: Path planning and control for autonomous navigation of single and multiple magnetic mobile microrobots. In: Proceedings of the ASME 2015 International Design Engineering Technical Conferences & Computers and Informatio in Engineering Conference. Boston, Massachusetts (2015c)Google Scholar
  10. DeVon, D., Bretl, T.: Control of many robots moving in the same direction with different speeds: a decoupling approach. In: American Control Conference. ACC’09., pp. 1794–1799. IEEE (2009)Google Scholar
  11. Dhond, U.R., Aggarwal, J.K.: Structure from stereo-a review. IEEE Trans. Syst. Man. Cybern. 19(6), 1489–1510 (1989)Google Scholar
  12. Diller, E., Floyd, S., Pawashe, C., Sitti, M.: Control of multiple heterogeneous magnetic microrobots in two dimensions on nonspecialized surfaces. IEEE Trans. Robot. 28(1), 172–182 (2012)CrossRefGoogle Scholar
  13. Diller, E., Sitti, M., et al.: Micro-scale mobile robotics. Found. Trends Robot. 2(3), 143–259 (2013)CrossRefGoogle Scholar
  14. Floyd, S., Pawashe, C., Sitti, M.: An untethered magnetically actuated micro-robot capable of motion on arbitrary surfaces. In: Robotics and Automation, 2008. ICRA 2008. IEEE International Conference on, pp. 419–424. IEEE (2008)Google Scholar
  15. Frutiger, D.R., Vollmers, K., Kratochvil, B.E., Nelson, B.J.: Small, fast, and under control: wireless resonant magnetic micro-agents. Int. J. Robot. Res. 29(5), 613–636 (2010)CrossRefGoogle Scholar
  16. Fusco, S., Huang, H.W., Peyer, K.E., Peters, C., Hberli, M., Ulbers, A., Spyrogianni, A., Pellicer, E., Sort, J., Pratsinis, S.E., Nelson, B.J., Sakar, M.S., Pan, S.: Shape-switching microrobots for medical applications: The influence of shape in drug delivery and locomotion. ACS Appl. Mater. Interfaces 7(12), 6803–6811 (2015)CrossRefGoogle Scholar
  17. Ghiglia, D.C., Pritt, M.D. (eds.): Two-Dimensional Phase Unwrapping: Theory, Algorithms, and Software. Wiley, New York (1998)zbMATHGoogle Scholar
  18. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000). ISBN:0521623049Google Scholar
  19. Huang, H.W., Sakar, M.S., Riederer, K., Shamsudhin, N., Petruska, A., Pan, S., Nelson, B.J.: Magnetic microrobots with addressable shape control. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 1719–1724 (2016)Google Scholar
  20. Huang, P.S., Zhang, S.: Fast three-step phase-shifting algorithm. Appl. Opt. 45(21), 5086–5091 (2006)CrossRefGoogle Scholar
  21. Huang, P.S., Zhang, S., Chiang, F.P.: Trapezoidal phase-shifting method for three-dimensional shape measurement. Opt. Eng. 44(12), 123601 (2005)Google Scholar
  22. Huang, Y., Shang, Y., Liu, Y., Bao, H.: Handbook of 3D Machine Vision: Optical Metrology and Imaging, 1st edn., chap. 3D shapes from Speckle, pp. 33–56. CRC, USA (2013)Google Scholar
  23. Jia, P., Kofman, J., English, C.: Two-step triangular-pattern phase-shifting method for three-dimensional object-shape measurement. Opt. Eng. 46(8), 083201 (2007)Google Scholar
  24. Jing, W., Cappelleri, D.: Incorporating in-situ force sensing capabilities in a magnetic microrobot. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014), pp. 4704–4709 (2014a)Google Scholar
  25. Jing, W., Cappelleri, D.: A magnetic microrobot with in situ force sensing capabilities. Robotics 3(2), 106–119 (2014b)Google Scholar
  26. Jing, W., Cappelleri, D.J.: Towards functional mobile magnetic microrobots. In: Small-Scale Robotics. From Nano-to-Millimeter-Sized Robotic Systems and Applications, pp. 81–100. Springer, New York (2014c)Google Scholar
  27. Jing, W., Chen, X., Lyttle, S., Fu, Z., Shi, Y., Cappelleri, D.: A magnetic thin film microrobot with two operating modes. In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp. 96–101 (2011)Google Scholar
  28. Jing, W., Pagano, N., Cappelleri, D.J.: A micro-scale magnetic tumbling microrobot. In: ASME 2012 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, pp. 187–196 (2012)Google Scholar
  29. Jing, W., Pagano, N., Cappelleri, D.J.: A novel micro-scale magnetic tumbling microrobot. J. Micro Bio Robot. 8(1), 1–12 (2013a)Google Scholar
  30. Jing, W., Pagano, N., Cappelleri, D.J.: A tumbling magnetic microrobot with flexible operating modes. In: 2013 IEEE International Conference on Robotics and Automation (ICRA), pp. 5514–5519 (2013b)Google Scholar
  31. Karpinsky, N., Hoke, M., Chen, V., Zhang, S.: High-resolution, real-time three-dimensional shape measurement on graphics processing unit. Opt. Eng. 53(2), 024105 (2014)Google Scholar
  32. Keferstein, C.P., Marxer, M.: Testing bench for laser triangulation sensors. Sensor Rev. 18(3), 183–187 (1998)CrossRefGoogle Scholar
  33. Kummer, M.P., Abbott, J.J., Kratochvil, B.E., Borer, R., Sengul, A., Nelson, B.J.: Octomag: An electromagnetic system for 5-dof wireless micromanipulation. Robot. IEEE Trans. 26(6), 1006–1017 (2010)CrossRefGoogle Scholar
  34. Lei, S., Zhang, S.: Flexible 3-d shape measurement using projector defocusing. Opt. Lett. 34(20), 3080–3082 (2009)CrossRefGoogle Scholar
  35. Li, B., Wang, Y., Dai, J., Lohry, W., Zhang, S.: Some recent advances on superfast 3d shape measurement with digital binary defocusing techniques. Opt. Laser Eng. 54, 236–246 (2014)CrossRefGoogle Scholar
  36. Li, B., Zhang, S.: Microscopic structured light 3d profilometry: Binary defocusing technique vs. sinusoidal fringe projection. Opt. Laser Eng. (2016) (in press)Google Scholar
  37. Liu, K., Wang, Y., Lau, D.L., Hao, Q., Hassebrook, L.G.: Dual-frequency pattern scheme for high-speed 3-d shape measurement. Opt. Express 18, 5229–5244 (2010)CrossRefGoogle Scholar
  38. Malacara, D. (ed.): Optical Shop Testing, 3rd edn. Wiley, New York (2007)Google Scholar
  39. Manneberg, G., Hertegard, S., Liljencrantz, J.: Measurement of human vocal fold vibrations with laser triangulation. Opt. Eng. 40(9), 2041–2044 (2001)CrossRefGoogle Scholar
  40. Morano, R.A., Ozturk, C., Conn, R., Dubin, S., Zietz, S., Nissanov, J.: Structured light using pseudorandom codes. IEEE Trans. Pattern Anal. Mach. Intell. 20, 322–327 (1998)Google Scholar
  41. Palagi, S., Mark, A.G., Reigh, S.Y., Melde, K., Qiu, T., Zeng, H., Parmeggiani, C., Martella, D., Sanchez-Castillo, A., Kapernaum, N., Giesselmann, F., Wiersma, D.S., Lauga, E., Fischer, P.: Structured light enables biomimetic swimming and versatile locomotion of photoresponsive soft microrobots. Nat. Mater. 15(6), 647–653 (2016)Google Scholar
  42. Pan, J., Huang, P., Zhang, S., Chiang, F.P.: Color n-ary gray code for 3-d shape measurement. In: 12th International Conference on Experimental Mechanics. Politecnico di Bari, Italy (2004)Google Scholar
  43. Pawashe, C., Floyd, S., Sitti, M.: Modeling and experimental characterization of an untethered magnetic micro-robot. Int. J. Robot. Res. 28(8), 1077–1094 (2009a)Google Scholar
  44. Pawashe, C., Floyd, S., Sitti, M.: Multiple magnetic microrobot control using electrostatic anchoring. Appl. Phys. Lett. 94(16), 164108 (2009b)Google Scholar
  45. Salvi, J., Fernandez, S., Pribanic, T., Llado, X.: A state of the art in structured light patterns for surface profilometry. Patt. Recogn. 43(8), 2666–2680 (2010)CrossRefzbMATHGoogle Scholar
  46. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comp. Vis. 47(1–3), 7–42 (2002)CrossRefzbMATHGoogle Scholar
  47. Sirikasemlert, A., Tao, X.: Objective evaluation of textural changes in knitted fabrics by laser triangulation. Text. Res. J. 70(12), 1076–1087 (2000)CrossRefGoogle Scholar
  48. Steager, E.B., Sakar, M.S., Magee, C., Kennedy, M., Cowley, A., Kumar, V.: Automated biomanipulation of single cells using magnetic microrobots. Int. J. Robot. Res. 32(3), 346–359 (2013)CrossRefGoogle Scholar
  49. Towers, D.P., Jones, J.D.C., Towers, C.E.: Optimum frequency selection in multi-frequency interferometry. Opt. Lett. 28, 1–3 (2003)CrossRefGoogle Scholar
  50. Wang, Y., Laughner, J.I., Efimov, I.R., Zhang, S.: 3d absolute shape measurement of live rabbit hearts with a superfast two-frequency phase-shifting technique. Opt. Express 21(5), 5822–5632 (2013)CrossRefGoogle Scholar
  51. Wang, Y., Zhang, S.: Superfast multifrequency phase-shifting technique with optimal pulse width modulation. Opt. Express 19(6), 5143–5148 (2011)CrossRefGoogle Scholar
  52. Xu, J., Zhao, J., Luo, R.: A 3d imaging sensor for mobile manipulation. IEEE International Conference on Cyber Technology in Automation. Control and Intelligent Systems (CYBER), pp. 315–320. Bangkok, Thailand (2012)Google Scholar
  53. Zhang, C., Xu, J., Xi, N., Jia, Y., Li, W.: Development of an omni-directional 3d camera for robot navigation. In: IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pp. 262–267. Kaohsiung, Taiwan (2012)Google Scholar
  54. Zhang, S.: Recent progresses on real-time 3-d shape measurement using digital fringe projection techniques. Opt. Laser Eng. 48(2), 149–158 (2010)CrossRefGoogle Scholar
  55. Zhang, S.: High-speed 3D imaging with digital fringe projection technique, 1st edn. CRC, New York (2016)Google Scholar
  56. Zhang, S., Huang, P.S.: High-resolution real-time three-dimensional shape measurement. Opt. Eng. 45(12), 123601 (2006)Google Scholar
  57. Zhang, S., Huang, P.S.: Novel method for structured light system calibration. Opt. Eng. 45(8), 083601 (2006)Google Scholar
  58. Zhang, S., Li, X., Yau, S.T.: Multilevel quality-guided phase unwrapping algorithm for real-time three-dimensional shape reconstruction. Appl. Opt. 46(1), 50–57 (2007)CrossRefGoogle Scholar
  59. Zhang, S., Royer, D., Yau, S.T.: Gpu-assisted high-resolution, real-time 3-d shape measurement. Opt. Express 14(20), 9120–9129 (2006)CrossRefGoogle Scholar
  60. Zhang, S., van der Weide, D., Oliver, J.: Superfast phase-shifting method for 3-d shape measurement. Opt. Express 18(9), 9684–9689 (2010)CrossRefGoogle Scholar
  61. Zhang, S., Yau, S.T.: High-resolution, real-time 3-d absolute coordinate measurement based on a phase-shifting method. Opt. Express 14(7), 2644–2649 (2006)CrossRefGoogle Scholar
  62. Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)CrossRefGoogle Scholar
  63. Zhang, Z.: Microsoft kinect sensor and its effect. IEEE Multimed. 19(2), 4–10 (2012)CrossRefGoogle Scholar

Copyright information

© Springer Singapore 2017

Authors and Affiliations

  • Beiwen Li
    • 1
  • Yatong An
    • 1
  • David Cappelleri
    • 1
  • Jing Xu
    • 2
  • Song Zhang
    • 1
    Email author
  1. 1.School of Mechanical EngineeringPurdue UniversityWest LafayetteUSA
  2. 2.Department of Mechanical EngineeringTsinghua UniversityBeijingChina

Personalised recommendations