Skip to main content

Continuous Optimization Framework for Depth Sensor Viewpoint Selection

  • Conference paper
  • First Online:
Algorithmic Foundations of Robotics XIII (WAFR 2018)

Part of the book series: Springer Proceedings in Advanced Robotics ((SPAR,volume 14))

Included in the following conference series:

  • 968 Accesses


Distinguishing differences between areas represented with point cloud data is generally approached by choosing a optimal viewpoint. The most informative view of a scene ultimately enables to have the optimal coverage over distinct points both locally and globally while accounting for the distance to the foci of attention. Measures of surface saliency, related to curvature inconsistency, extenuate differences in shape and are coupled with viewpoint selection approaches. As there is no analytical solution for optimal viewpoint selection, candidate viewpoints are generally discretely sampled and evaluated for information and require (near) exhaustive combinatorial searches. We present a consolidated optimization framework for optimal viewpoint selection with a continuous cost function and analytically derived Jacobian that incorporates view angle, vertex normals and measures of task related surface information relative to viewpoint. We provide a mechanism in the cost function to incorporate sensor attributes such as operating range, field of view and angular resolution. The framework is evaluated as competing favourably with the state-of-the-art approaches to viewpoint selection while significantly reducing the number of viewpoints to be evaluated in the process.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others


  1. Anders, G.J., Mies, W., Sweetser, J.N., Woodfill, J.: Best-known-methods for tuning intel realsense D400 depth cameras for best performance. Intel (2018)

    Google Scholar 

  2. Andersen, M.R., Jensen, T., Lisouski, P., Mortensen, A.K., Hansen, M.K., Gregersen, T., Ahrendt, P.: Kinect depth sensor evaluation for computer vision applications. Tech. Rep. Electron. Comput. Eng. 1(6) (2012)

    Google Scholar 

  3. Botsch, M., Kobbelt, L., Pauly, M., Alliez, P., Lévy, B.: Polygon Mesh Processing. CRC Press, Boca Raton (2010)

    Book  Google Scholar 

  4. Boumal, N., Mishra, B., Absil, P.A., Sepulchre, R.: Manopt, a matlab toolbox for optimization on manifolds. J. Mach. Learn. Res. 15(1), 1455–1459 (2014)

    MATH  Google Scholar 

  5. Cate, A.D., Behrmann, M.: Perceiving parts and shapes from concave surfaces. Atten. Percept. Psychophys. 72(1), 153–167 (2010)

    Article  Google Scholar 

  6. Chen, X., Saparov, A., Pang, B., Funkhouser, T.: Schelling points on 3D surface meshes. ACM Trans. Graph. (TOG) 31(4), 29 (2012)

    Article  Google Scholar 

  7. Foissotte, T., Stasse, O., Wieber, P.B., Escande, A., Kheddar, A.: Autonomous 3D object modeling by a humanoid using an optimization-driven next-best-view formulation. Int. J. Hum. Rob. 7(03), 407–428 (2010)

    Article  Google Scholar 

  8. Gal, R., Cohen-Or, D.: Salient geometric features for partial shape matching and similarity. ACM Trans. Graph. (TOG) 25(1), 130–150 (2006)

    Article  Google Scholar 

  9. Gonzalez-Barbosa, J.J., García-Ramírez, T., Salas, J., Hurtado-Ramos, J.B., et al.: Optimal camera placement for total coverage. In: IEEE International Conference on Robotics and Automation 2009. ICRA 2009, pp. 844–848. IEEE (2009)

    Google Scholar 

  10. Johnson, A.E., Hebert, M.: Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Trans. Pattern Anal. Mach. Intell. 21(5), 433–449 (1999)

    Article  Google Scholar 

  11. Lee, C.H., Varshney, A., Jacobs, D.W.: Mesh saliency. ACM Trans. Graph. (TOG) 24, 659–666 (2005)

    Article  Google Scholar 

  12. Lee, J., Moghaddam, B., Pfister, H., Machiraju, R.: Finding optimal views for 3D face shape modeling. In: Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition 2004, pp. 31–36. IEEE (2004)

    Google Scholar 

  13. Leifman, G., Shtrom, E., Tal, A.: Surface regions of interest for viewpoint selection. IEEE Trans. Pattern Anal. Mach. Intell. 38(12), 2544–2556 (2016)

    Article  Google Scholar 

  14. May, S., Mies, W., Edwards, J., Williams, F., Wise, J., Morgan, J., Savell, J., Cross, H.: Beef carcass composition of slaughter cattle differing in frame size, muscle score, and external fatness. J. Anim. Sci. 70(8), 2431–2445 (1992)

    Article  Google Scholar 

  15. Newcombe, R.A., Fox, D., Seitz, S.M.: Dynamicfusion: reconstruction and tracking of non-rigid scenes in real-time. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 343–352 (2015)

    Google Scholar 

  16. Norman, J.F., Norman, H.F., Lee, Y.L., Stockton, D., Lappin, J.S.: The visual perception of length along intrinsically curved surfaces. Percept. Psychophys. 66(1), 77–88 (2004)

    Article  Google Scholar 

  17. Ohbuchi, R., Osada, K., Furuya, T., Banno, T.: Salient local visual features for shape-based 3D model retrieval. In: IEEE International Conference on Shape Modeling and Applications 2008, SMI 2008, pp. 93–102. IEEE (2008)

    Google Scholar 

  18. Pece, F., Kautz, J., Weyrich, T.: Three depth-camera technologies compared. In: First BEAMING Workshop, Barcelona, vol. 2011, p. 9 (2011)

    Google Scholar 

  19. Quin, P., Paul, G., Alempijevic, A., Liu, D., Dissanayake, G.: Efficient neighbourhood-based information gain approach for exploration of complex 3D environments. In: 2013 IEEE International Conference on Robotics and Automation (ICRA), pp. 1343–1348. IEEE (2013)

    Google Scholar 

  20. Shilane, P., Funkhouser, T.: Distinctive regions of 3D surfaces. ACM Trans. Graph. (TOG) 26(2), 7 (2007)

    Article  Google Scholar 

  21. Skinner, B., Vidal Calleja, T., Valls Miro, J., De Bruijn, F., Falque, R.: 3D point cloud upsampling for accurate reconstruction of dense 2.5 D thickness maps. In: Australasian Conference on Robotics and Automation (2014)

    Google Scholar 

  22. Vieira, T., Bordignon, A., Peixoto, A., Tavares, G., Lopes, H., Velho, L., Lewiner, T.: Learning good views through intelligent galleries. In: Computer Graphics Forum, vol. 28, pp. 717–726. Wiley Online Library (2009)

    Google Scholar 

  23. Wu, K., Ranasinghe, R., Dissanayake, G.: Active recognition and pose estimation of household objects in clutter. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 4230–4237. IEEE (2015)

    Google Scholar 

Download references


This work was possible due to the financial and in-kind support, and efforts of many individuals from NSW Department of Primary Industries, University of Technology Sydney, and Meat and Livestock Australia.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Behnam Maleki .

Editor information

Editors and Affiliations

A Appendix

A Appendix


Let \(\mathbf {t}\), \(\mathbf {p}\) and \(\mathbf {z}\) be three column vectors in \(\mathrm{I\!R}^n\) and \(\mathbf {R}\) is a \(n\times n\) matrix. Then for \(\mathbf {u}(\mathbf {R},\mathbf {t})=\frac{\left\Vert \mathbf {R}\mathbf {z}\,\varvec{\times }\,(\mathbf {p}\,-\,\mathbf {t})\right\Vert }{\mathbf {R}\mathbf {z}\,\bullet \, (\mathbf {p}\,-\,\mathbf {t})}\), we have:

$$\begin{aligned} \frac{\partial \mathbf {u}}{\partial \mathbf {t}} = \frac{(\mathbf {R}\mathbf {z}\cdot \mathbf {R}\mathbf {z})}{((\mathbf {p}-\mathbf {t})\cdot \mathbf {R}\mathbf {z})^3\mathbf {u}}\Big ((\mathbf {p}-\mathbf {t})\cdot (\mathbf {p}-\mathbf {t})\mathbf {R}\mathbf {z}-((\mathbf {p}-\mathbf {t})\cdot \mathbf {R}\mathbf {z})(\mathbf {p}-\mathbf {t})\Big ) \end{aligned}$$
$$\begin{aligned} \frac{\partial \mathbf {u}}{\partial \mathbf {R}} = \frac{((\mathbf {p}-\mathbf {t})\cdot (\mathbf {p}-\mathbf {t}))}{(\mathbf {R}\mathbf {z}\cdot (\mathbf {p}-\mathbf {t}))^3\mathbf {u}}\Big ((\mathbf {R}\mathbf {z}\cdot (\mathbf {p}-\mathbf {t}))\mathbf {R}\mathbf {z}-(\mathbf {R}\mathbf {z}\cdot \mathbf {R}\mathbf {z})(\mathbf {p}-\mathbf {t})\Big )\mathbf {z}^\intercal \end{aligned}$$

Note: To avoid notational confusion, \(\mathbf {u}\) denotes the value of function \(\mathbf {u}(\mathbf {R},\mathbf {t})\).


Assume \(\mathbf {a}\) and \(\mathbf {b}\) are two column vectors in \(\mathrm{I\!R}^n\). The auxiliary variable \(\pmb {\kappa }\) is defined as a scalar function of \(\mathbf {a},\mathbf {b}\) which first vector, \(\mathbf {a}\), is a constant,

$$\begin{aligned} \mathbf {\pmb {\kappa }} = \frac{\Vert \mathbf {a}\times \mathbf {b}\Vert ^2}{(\mathbf {a}\cdot \mathbf {b})^2}= \frac{(\mathbf {b}\cdot \mathbf {b})(\mathbf {a}\cdot \mathbf {a})-(\mathbf {b}\cdot \mathbf {a})^2}{(\mathbf {b}\cdot \mathbf {a})^2}= \frac{(\mathbf {b}\cdot \mathbf {b})(\mathbf {a}\cdot \mathbf {a})}{(\mathbf {b}\cdot \mathbf {a})^2} - 1 \end{aligned}$$

The differential of \(\pmb {\kappa }\) is:

$$\begin{aligned} \begin{aligned} \mathrm {d}\pmb {\kappa }&= \frac{2(\mathbf {b}\cdot \mathrm {d}\,\mathbf {b})(\mathbf {a}\cdot \mathbf {a})}{(\mathbf {b}\cdot \mathbf {a})^2} - \frac{2(\mathbf {b}\cdot \mathbf {b})(\mathbf {a}\cdot \mathbf {a})(\mathbf {a}\cdot \mathrm {d}\,\mathbf {b})}{(\mathbf {b}\cdot \mathbf {a})^3}\\&= \frac{2(\mathbf {a}\cdot \mathbf {a})}{(\mathbf {b}\cdot \mathbf {a})^3}\,\Big ((\mathbf {b}\cdot \mathbf {a})\mathbf {b}-(\mathbf {b}\cdot \mathbf {b})\mathbf {a}\Big )\cdot \mathrm {d}\,\mathbf {b}. \end{aligned} \end{aligned}$$
$$\begin{aligned} \pmb {\kappa } = \mathbf {u}^2 \implies \mathrm {d}\pmb {\kappa }=2\mathbf {u}\,\mathrm {d}\mathbf {u} \end{aligned}$$

by substituting:

$$\begin{aligned} \mathbf {a} = \mathbf {R}\mathbf {z} \end{aligned}$$
$$\begin{aligned} \mathbf {b} = (\mathbf {p}-\mathbf {t}) \end{aligned}$$
$$\begin{aligned} \mathrm {d}\mathbf {b} = -\mathrm {d}\mathbf {t} \end{aligned}$$
$$\begin{aligned} \mathrm {d}\mathbf {u} = \frac{\mathrm {d}\pmb {\kappa }}{2\mathbf {u}} \,\,= \frac{(\mathbf {a}\cdot \mathbf {a})}{(\mathbf {b}\cdot \mathbf {a})^3\mathbf {u}}\,\Big ((\mathbf {b}\cdot \mathbf {a})\mathbf {b}-(\mathbf {b}\cdot \mathbf {b})\mathbf {a}\Big )\cdot (-\mathrm {d}\mathbf {t}) \end{aligned}$$
$$\begin{aligned} \frac{\partial \mathbf {u}}{\partial \mathbf {t}} = \frac{(\mathbf {a}\cdot \mathbf {a})}{(\mathbf {b}\cdot \mathbf {a})^3\mathbf {u}}\,\Big ((\mathbf {b}\cdot \mathbf {b})\mathbf {a}-(\mathbf {b}\cdot \mathbf {a})\mathbf {b}\Big ) \end{aligned}$$

And also if:

$$\begin{aligned} \mathbf {a} = (\mathbf {p}-\mathbf {t}) \end{aligned}$$
$$\begin{aligned} \mathbf {b} = \mathbf {R}\mathbf {z} \end{aligned}$$
$$\begin{aligned} \mathrm {d}\mathbf {b}=\mathrm {d}\mathbf {R}\,\mathbf {z} \end{aligned}$$
$$\begin{aligned} \begin{aligned} \mathrm {d}\mathbf {u}&= \frac{(\mathbf {a}\cdot \mathbf {a})}{(\mathbf {b}\cdot \mathbf {a})^3\mathbf {u}}\,\Big ((\mathbf {b}\cdot \mathbf {a})\mathbf {b}-(\mathbf {b}\cdot \mathbf {b})\mathbf {a}\Big )\cdot \mathrm {d}\mathbf {R}\,\mathbf {z} \\&= \frac{(\mathbf {a}\cdot \mathbf {a})}{(\mathbf {b}\cdot \mathbf {a})^3\mathbf {u}}\,\Big ((\mathbf {b}\cdot \mathbf {a})\mathbf {b}-(\mathbf {b}\cdot \mathbf {b})\mathbf {a}\Big )\mathbf {z}^\intercal \cdot \mathrm {d}\mathbf {R} \end{aligned} \end{aligned}$$
$$\begin{aligned} \frac{\partial \mathbf {u}}{\partial \mathbf {R}} = \frac{(\mathbf {a}\cdot \mathbf {a})}{(\mathbf {b}\cdot \mathbf {a})^3\mathbf {u}}\,\Big ((\mathbf {b}\cdot \mathbf {a})\mathbf {b}-(\mathbf {b}\cdot \mathbf {b})\mathbf {a}\Big )\mathbf {z}^\intercal \ \end{aligned}$$

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Maleki, B., Alempijevic, A., Vidal-Calleja, T. (2020). Continuous Optimization Framework for Depth Sensor Viewpoint Selection. In: Morales, M., Tapia, L., Sánchez-Ante, G., Hutchinson, S. (eds) Algorithmic Foundations of Robotics XIII. WAFR 2018. Springer Proceedings in Advanced Robotics, vol 14. Springer, Cham.

Download citation

Publish with us

Policies and ethics