Semantic-aware label placement for augmented reality in street view

Abstract

In an augmented reality (AR) application, placing labels in a manner that is clear and readable without occluding the critical information from the real world can be a challenging problem. This paper introduces a label placement technique for AR used in street view scenarios. We propose a semantic-aware task-specific label placement method by identifying potentially important image regions through a novel feature map, which we refer to as guidance map. Given an input image, its saliency information, semantic information and the task-specific importance prior are integrated in the guidance map for our labeling task. To learn the task prior, we created a label placement dataset with the users’ labeling preferences, as well as use it for evaluation. Our solution encodes the constraints for placing labels in an optimization problem to obtain the final label layout, and the labels will be placed in appropriate positions to reduce the chances of overlaying important real-world objects in street view AR scenarios. The experimental validation shows clearly the benefits of our method over previous solutions in the AR street view navigation and similar applications.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

References

  1. 1.

    Azuma, R.T.: A survey of augmented reality. Teleoper. Virtual Environ. 6(4), 355–385 (1997)

    Article  Google Scholar 

  2. 2.

    Carmigniani, J., Furht, B., Anisetti, M., Ceravolo, P., Damiani, E., Ivkovic, M.: Augmented reality technologies, systems and applications. Multimed. Tools Appl. 51(1), 341–377 (2011)

    Article  Google Scholar 

  3. 3.

    Chang, G., Morreale, P., Medicherla, P.: Applications of augmented reality systems in education. In: Society for Information Technology Teacher Education International Conference Association for the Advancement of Computing in Education (AACE), pp. 1380–1385 (2010)

  4. 4.

    Hartmann, K., Götzelmann, T., Ali, K., Strothotte, T.: Metrics for functional and aesthetic label layouts. In: International Symposium on Smart Graphics, pp. 115–126 (2005)

  5. 5.

    Achanta, R., Hemami, S., Estrada, F., Susstrunk, S.: Frequency-tuned salient region detection. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2009)

  6. 6.

    Wang, W., Lai, Q., Fu, H., Shen, J., Ling, H.: Salient object detection in the deep learning era: an in-depth survey. arXiv:1904.09146 (2019)

  7. 7.

    Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation, arXiv preprint. arXiv:1706.05587 (2017)

  8. 8.

    Rakholia, N., Hegde, S., Hebbalaguppe, R.: Where to place: a real-time visual saliency based label placement for augmented reality applications. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 604–608 (2018)

  9. 9.

    Bell, B., Feiner, S., Höllerer, T.: View management for virtual and augmented reality. In: Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology, pp. 101–110 (2001)

  10. 10.

    Cmolík, L., Bittner, J.: Layout-aware optimization for interactive labeling of 3D models. Comput. Graph. 34(4), 378–387 (2010)

    Article  Google Scholar 

  11. 11.

    Tenmoku, R., Kanbara, M., Yokoya, N.: Annotating user-viewed objects for wearable AR systems. In: Proceedings of the 4th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 192–193 (2005)

  12. 12.

    Makita, K., Kanbara, M., Yokoya, N.: View management of annotations for wearable augmented reality. In: 2009 IEEE International Conference on Multimedia and Expo, pp. 982–985 (2009)

  13. 13.

    Zhang, B., Li, Q., Chao, H., Chen, B., Ofek, E., Xu, Y.Q.: Annotating and navigating tourist videos. In: Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, pp. 260–269 (2010)

  14. 14.

    Iwai, D., Yabiki, T., Sato, K.: View management of projected labels on nonplanar and textured surfaces. IEEE Trans. Vis. Comput. Graph. 19(8), 1415–1424 (2013)

    Article  Google Scholar 

  15. 15.

    Tatzgern, M., Kalkofen, D., Schmalstieg, D.: Dynamic compact visualizations for augmented reality. In: 2013 IEEE Virtual Reality (VR), pp. 3–6 (2013)

  16. 16.

    Tatzgern, M., Kalkofen, D., Grasset, R., Schmalstieg, D.: Hedgehog labeling: View management techniques for external labels in 3D space. In: 2014 IEEE Virtual Reality (VR), pp. 27–32 (2014)

  17. 17.

    Lauber, F., Butz, A.: View management for driver assistance in an HMD. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 1–6 (2013)

  18. 18.

    Orlosky, J., Kiyokawa, K., Takemura, H.: Towards intelligent view management: a study of manual text placement tendencies in mobile environments using video see-through displays. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 281–282 (2013)

  19. 19.

    Azuma, R., Furmanski, C.: Evaluating label placement for augmented reality view management. In: Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality, p. 66 (2003)

  20. 20.

    Leykin, A., Tuceryan, M.: Automatic determination of text readability over textured backgrounds for augmented reality systems. In: Third IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 224–230 (2004)

  21. 21.

    Rosten, E., Reitmayr, G., Drummond, T.: Real-time video annotations for augmented reality. In: International Symposium on Visual Computing, pp. 294–302 (2005)

  22. 22.

    Tanaka, K., Kishino, Y., Miyamae, M., Terada, T., Nishio, S.: An information layout method for an optical see-through head mounted display focusing on the viewability. In: 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 139–142 (2008)

  23. 23.

    Grasset, R., Langlotz, T., Kalkofen, D., Tatzgern, M., Schmalstieg, D.: Image-driven view management for augmented reality browsers. In: 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 177–186 (2012)

  24. 24.

    Ichihashi, K., Fujinami, K.: Estimating visibility of annotations for view management in spatial augmented reality based on machine-learning techniques. Sensors 19(4), 939–966 (2019)

    Article  Google Scholar 

  25. 25.

    Sato, M., Fujinami, K.: Nonoverlapped view management for augmented reality by tabletop projection. J. Vis. Lang. Comput. 25(6), 891–902 (2014)

    Article  Google Scholar 

  26. 26.

    Li, G., Liu, Y., Wang, Y.: An empirical evaluation of labelling method in augmented reality. In: Proceedings of the 16th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry. https://doi.org/10.1145/3284398.3284422 (2018)

  27. 27.

    Li, G., Liu, Y., Wang, Y., Xu, Z.: Evaluation of labelling layout method for image-driven view management in augmented reality. In: Proceedings of the 29th Australian Conference on Computer-Human Interaction, pp. 266–274 (2017)

  28. 28.

    Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213–3223 (2016)

  29. 29.

    Jiang H., Wang J., Yuan Z., Wu Y., Zheng N. and Li S., Salient object detection: a discriminative regional feature integration approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2083–2090 (2013)

  30. 30.

    Hou, Q., Cheng, M.M., Hu, X., Borji, A., Tu, Z., Torr, P.H.: Deeply supervised salient object detection with short connections. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3203–3212 (2017)

  31. 31.

    Peterson, S.D., Axholt, M., Cooper, M., Ellis, S.R.: Evaluation of alternative label placement techniques in dynamic virtual environments. In: International Symposium on Smart Graphics, pp. 43–55 (2009)

  32. 32.

    Stein, T., Décoret, X.: Dynamic label placement for improved interactive exploration. In: Proceedings of the 6th International Symposium on Non-photorealistic Animation and Rendering, pp. 15–21 (2008)

  33. 33.

    Vollick, I., Vogel, D., Agrawala, M., Hertzmann, A.: Specifying label layout style by example. In: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, pp. 221–230 (2007)

  34. 34.

    Augmented reality head-up display, from http://continental-head-up-display.com/ar-hud/, Retrieved 30 April 2019 (2015)

  35. 35.

    Jia, J., Zhang, Y., Wu, X., Guo, W.: Image-based label placement for augmented reality browsers. In: 2018 IEEE 4th International Conference on Computer and Communications (ICCC), pp. 1654–1659 (2018)

  36. 36.

    Lhuillier, A., Garderen, M., Weiskopf, D.: Density-based label placement. Vis. Comput. 35(6–8), 1041–1052 (2019)

    Article  Google Scholar 

Download references

Funding

This study was funded by the National Natural Science Foundation of China under Grants 61802109 and 61902109.

Author information

Affiliations

Authors

Corresponding authors

Correspondence to Wei Guo or Haibin Ling.

Ethics declarations

Conflict of interest

Jianqing Jia has visited Haibin Ling’s Group in Temple University during October 12, 2018 to May 20, 2019. Semir Elezovikj declares he has no conflict of interest. Heng Fan declares he has no conflict of interest. Shuojin Yang declares he has no conflict of interest. Jing Liu declares he has no conflict of interest. Wei Guo declares she has no conflict of interest. Chiu C. Tan declares he has no conflict of interest. Haibin Ling serves as associate editors for IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), Pattern Recognition (PR) and Computer Vision and Image Understanding (CVIU).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (wmv 6791 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Jia, J., Elezovikj, S., Fan, H. et al. Semantic-aware label placement for augmented reality in street view. Vis Comput (2020). https://doi.org/10.1007/s00371-020-01939-w

Download citation

Keywords

  • Label placement
  • Augmented reality
  • Guidance map
  • Street view
  • Image-based layout