Abstract
In an augmented reality (AR) application, placing labels in a manner that is clear and readable without occluding the critical information from the real world can be a challenging problem. This paper introduces a label placement technique for AR used in street view scenarios. We propose a semantic-aware task-specific label placement method by identifying potentially important image regions through a novel feature map, which we refer to as guidance map. Given an input image, its saliency information, semantic information and the task-specific importance prior are integrated in the guidance map for our labeling task. To learn the task prior, we created a label placement dataset with the users’ labeling preferences, as well as use it for evaluation. Our solution encodes the constraints for placing labels in an optimization problem to obtain the final label layout, and the labels will be placed in appropriate positions to reduce the chances of overlaying important real-world objects in street view AR scenarios. The experimental validation shows clearly the benefits of our method over previous solutions in the AR street view navigation and similar applications.
Similar content being viewed by others
References
Azuma, R.T.: A survey of augmented reality. Teleoper. Virtual Environ. 6(4), 355–385 (1997)
Carmigniani, J., Furht, B., Anisetti, M., Ceravolo, P., Damiani, E., Ivkovic, M.: Augmented reality technologies, systems and applications. Multimed. Tools Appl. 51(1), 341–377 (2011)
Chang, G., Morreale, P., Medicherla, P.: Applications of augmented reality systems in education. In: Society for Information Technology Teacher Education International Conference Association for the Advancement of Computing in Education (AACE), pp. 1380–1385 (2010)
Hartmann, K., Götzelmann, T., Ali, K., Strothotte, T.: Metrics for functional and aesthetic label layouts. In: International Symposium on Smart Graphics, pp. 115–126 (2005)
Achanta, R., Hemami, S., Estrada, F., Susstrunk, S.: Frequency-tuned salient region detection. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2009)
Wang, W., Lai, Q., Fu, H., Shen, J., Ling, H.: Salient object detection in the deep learning era: an in-depth survey. arXiv:1904.09146 (2019)
Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation, arXiv preprint. arXiv:1706.05587 (2017)
Rakholia, N., Hegde, S., Hebbalaguppe, R.: Where to place: a real-time visual saliency based label placement for augmented reality applications. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 604–608 (2018)
Bell, B., Feiner, S., Höllerer, T.: View management for virtual and augmented reality. In: Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology, pp. 101–110 (2001)
Cmolík, L., Bittner, J.: Layout-aware optimization for interactive labeling of 3D models. Comput. Graph. 34(4), 378–387 (2010)
Tenmoku, R., Kanbara, M., Yokoya, N.: Annotating user-viewed objects for wearable AR systems. In: Proceedings of the 4th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 192–193 (2005)
Makita, K., Kanbara, M., Yokoya, N.: View management of annotations for wearable augmented reality. In: 2009 IEEE International Conference on Multimedia and Expo, pp. 982–985 (2009)
Zhang, B., Li, Q., Chao, H., Chen, B., Ofek, E., Xu, Y.Q.: Annotating and navigating tourist videos. In: Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, pp. 260–269 (2010)
Iwai, D., Yabiki, T., Sato, K.: View management of projected labels on nonplanar and textured surfaces. IEEE Trans. Vis. Comput. Graph. 19(8), 1415–1424 (2013)
Tatzgern, M., Kalkofen, D., Schmalstieg, D.: Dynamic compact visualizations for augmented reality. In: 2013 IEEE Virtual Reality (VR), pp. 3–6 (2013)
Tatzgern, M., Kalkofen, D., Grasset, R., Schmalstieg, D.: Hedgehog labeling: View management techniques for external labels in 3D space. In: 2014 IEEE Virtual Reality (VR), pp. 27–32 (2014)
Lauber, F., Butz, A.: View management for driver assistance in an HMD. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 1–6 (2013)
Orlosky, J., Kiyokawa, K., Takemura, H.: Towards intelligent view management: a study of manual text placement tendencies in mobile environments using video see-through displays. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 281–282 (2013)
Azuma, R., Furmanski, C.: Evaluating label placement for augmented reality view management. In: Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality, p. 66 (2003)
Leykin, A., Tuceryan, M.: Automatic determination of text readability over textured backgrounds for augmented reality systems. In: Third IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 224–230 (2004)
Rosten, E., Reitmayr, G., Drummond, T.: Real-time video annotations for augmented reality. In: International Symposium on Visual Computing, pp. 294–302 (2005)
Tanaka, K., Kishino, Y., Miyamae, M., Terada, T., Nishio, S.: An information layout method for an optical see-through head mounted display focusing on the viewability. In: 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 139–142 (2008)
Grasset, R., Langlotz, T., Kalkofen, D., Tatzgern, M., Schmalstieg, D.: Image-driven view management for augmented reality browsers. In: 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 177–186 (2012)
Ichihashi, K., Fujinami, K.: Estimating visibility of annotations for view management in spatial augmented reality based on machine-learning techniques. Sensors 19(4), 939–966 (2019)
Sato, M., Fujinami, K.: Nonoverlapped view management for augmented reality by tabletop projection. J. Vis. Lang. Comput. 25(6), 891–902 (2014)
Li, G., Liu, Y., Wang, Y.: An empirical evaluation of labelling method in augmented reality. In: Proceedings of the 16th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry. https://doi.org/10.1145/3284398.3284422 (2018)
Li, G., Liu, Y., Wang, Y., Xu, Z.: Evaluation of labelling layout method for image-driven view management in augmented reality. In: Proceedings of the 29th Australian Conference on Computer-Human Interaction, pp. 266–274 (2017)
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213–3223 (2016)
Jiang H., Wang J., Yuan Z., Wu Y., Zheng N. and Li S., Salient object detection: a discriminative regional feature integration approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2083–2090 (2013)
Hou, Q., Cheng, M.M., Hu, X., Borji, A., Tu, Z., Torr, P.H.: Deeply supervised salient object detection with short connections. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3203–3212 (2017)
Peterson, S.D., Axholt, M., Cooper, M., Ellis, S.R.: Evaluation of alternative label placement techniques in dynamic virtual environments. In: International Symposium on Smart Graphics, pp. 43–55 (2009)
Stein, T., Décoret, X.: Dynamic label placement for improved interactive exploration. In: Proceedings of the 6th International Symposium on Non-photorealistic Animation and Rendering, pp. 15–21 (2008)
Vollick, I., Vogel, D., Agrawala, M., Hertzmann, A.: Specifying label layout style by example. In: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, pp. 221–230 (2007)
Augmented reality head-up display, from http://continental-head-up-display.com/ar-hud/, Retrieved 30 April 2019 (2015)
Jia, J., Zhang, Y., Wu, X., Guo, W.: Image-based label placement for augmented reality browsers. In: 2018 IEEE 4th International Conference on Computer and Communications (ICCC), pp. 1654–1659 (2018)
Lhuillier, A., Garderen, M., Weiskopf, D.: Density-based label placement. Vis. Comput. 35(6–8), 1041–1052 (2019)
Funding
This study was funded by the National Natural Science Foundation of China under Grants 61802109 and 61902109.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of interest
Jianqing Jia has visited Haibin Ling’s Group in Temple University during October 12, 2018 to May 20, 2019. Semir Elezovikj declares he has no conflict of interest. Heng Fan declares he has no conflict of interest. Shuojin Yang declares he has no conflict of interest. Jing Liu declares he has no conflict of interest. Wei Guo declares she has no conflict of interest. Chiu C. Tan declares he has no conflict of interest. Haibin Ling serves as associate editors for IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), Pattern Recognition (PR) and Computer Vision and Image Understanding (CVIU).
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (wmv 6791 KB)
Rights and permissions
About this article
Cite this article
Jia, J., Elezovikj, S., Fan, H. et al. Semantic-aware label placement for augmented reality in street view. Vis Comput 37, 1805–1819 (2021). https://doi.org/10.1007/s00371-020-01939-w
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-020-01939-w