Skip to main content
SpringerLink
Log in

StairNetV3: depth-aware stair modeling using deep learning

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Vision-based stair modeling can help autonomous mobile robots deal with the challenge of climbing stairs, especially in unfamiliar environments. To address the problem that current monocular methods are difficult to model stairs accurately without depth information in scenes with fuzzy visual cues, this paper proposes a depth-aware stair modeling method for monocular vision. Specifically, we take the prediction of depth images and the extraction of stair geometric features as joint tasks in a convolutional neural network, with the designed information propagation architecture, we can achieve effective supervision for stair geometric feature learning by depth features. In addition, to complete the stair modeling, we take the convex lines, concave lines, tread surfaces and riser surfaces as stair geometric features and apply Gaussian kernels to enable StairNetV3 to predict contextual information within the stair lines. Combined with the depth information obtained by depth sensors, we propose a point cloud reconstruction method that can quickly segment point clouds of stair step surfaces. The experiments show that the proposed method has a significant improvement over the previous best monocular vision method, with an intersection over union increase of 3.4\(\%\), and the lightweight version has a fast detection speed and can meet the requirements of most real-time applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availibility

Our dataset is available at https://data.mendeley.com/datasets/6kffmjt7g2/1.

References

  1. Krausz, N.E., Hargrove, L.J.: Recognition of ascending stairs from 2d images for control of powered lower limb prostheses. In: 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 615–618 (2015). https://doi.org/10.1109/NER.2015.7146698

  2. Murakami, S., Shimakawa, M., Kivota, K., Kato, T.: Study on stairs detection using RGB-depth images. In: 2014 Joint 7th International Conference on Soft Computing and Intelligent Systems (SCIS) and 15th International Symposium on Advanced Intelligent Systems (ISIS), pp. 1186–1191 (2014). https://doi.org/10.1109/SCIS-ISIS.2014.7044705

  3. Shahrabadi, S., Rodrigues, J.M., Buf, J.: Detection of indoor and outdoor stairs. In: Iberian Conference on Pattern Recognition & Image Analysis, pp. 847–854 (2013). https://doi.org/10.1007/978-3-642-38628-2_100

  4. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 679–698 (1986)

    Article  CAS  PubMed  Google Scholar 

  5. Hough, P.V.C.: Method and Means for Recognizing Complex Patterns (1962)

  6. Westfechtel, T., Ohno, K., Mertsching, B., Nickchen, D., Kojima, S., Tadokoro, S.: 3d graph based stairway detection and localization for mobile robots. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 473–479 (2016). https://doi.org/10.1109/IROS.2016.7759096

  7. Perez-Yus, A., Gutierrez-Gomez, D., Lopez-Nicolas, G., Guerrero, J.J.: Stairs detection with odometry-aided traversal from a wearable RGB-D camera. Comput. Vis. Image Underst. 154, 192–205 (2017). https://doi.org/10.1016/j.cviu.2016.04.007

    Article  Google Scholar 

  8. Zhao, X., Chen, W., Yan, X., Wang, J., Wu, X.: Real-time stairs geometric parameters estimation for lower limb rehabilitation exoskeleton. In: 2018 Chinese Control And Decision Conference (CCDC), pp. 5018–5023 (2018). https://doi.org/10.1109/CCDC.2018.8408001

  9. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981). https://doi.org/10.1145/358669.358692

    Article  MathSciNet  Google Scholar 

  10. Oh, K.W., Choi, K.S.: Supervoxel-based staircase detection from range data. IEIE Trans. Smart Process. Comput. 4(6), 403–406 (2015). https://doi.org/10.5573/IEIESPC.2015.4.6.403

    Article  Google Scholar 

  11. Lee, J.-T., Kim, H.-U., Lee, C., Kim, C.-S.: Semantic line detection and its applications. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 3249–3257 (2017). https://doi.org/10.1109/ICCV.2017.350

  12. Zhao, K., Han, Q., Zhang, C.-B., Xu, J., Cheng, M.-M.: Deep hough transform for semantic line detection. IEEE Trans. Pattern Anal. Mach. Intell. 44(9), 4793–4806 (2022). https://doi.org/10.1109/TPAMI.2021.3077129

    Article  PubMed  Google Scholar 

  13. Zhou, Y., Qi, H., Ma, Y.: End-to-end wireframe parsing. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 962–971 (2019). https://doi.org/10.1109/ICCV.2019.00105

  14. Xue, N., Wu, T., Bai, S., Wang, F., Xia, G.-S., Zhang, L., Torr, P.H.S.: Holistically-attracted wireframe parsing. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2785–2794 (2020). https://doi.org/10.1109/CVPR42600.2020.00286

  15. Zhang, H., Luo, Y., Qin, F., He, Y., Liu, X.: Elsd: efficient line segment detector and descriptor. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2949–2958 (2021). https://doi.org/10.1109/ICCV48922.2021.00296

  16. Dai, X., Gong, H., Wu, S., Yuan, X., Yi, M.: Fully convolutional line parsing. Neurocomputing 506, 1–11 (2022). https://doi.org/10.1016/j.neucom.2022.07.026

    Article  Google Scholar 

  17. Qin, Z., Wang, H., Li, X.: Ultra fast structure-aware deep lane detection. In: Computer Vision—ECCV 2020, pp. 276–291. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58586-0_17

  18. Haris, M., Hou, J., Wang, X.: Lane line detection and departure estimation in a complex environment by using an asymmetric kernel convolution algorithm. Vis. Comput. 39, 519–538 (2023). https://doi.org/10.1007/s00371-021-02353-6

    Article  Google Scholar 

  19. Platt, J.C.: Sequential minimal optimization: a fast algorithm for training support vector machines. In: Advances in Kernel Methods-Support Vector Learning, vol. 208 (1998)

  20. Khaliluzzaman, M., Yakub, M., Chakraborty, N.: Comparative analysis of stairways detection based on RGB and RGB-D image. In: 2018 International Conference on Innovations in Science, Engineering and Technology (ICISET), pp. 519–524 (2018). https://doi.org/10.1109/ICISET.2018.8745624

  21. Khaliluzzaman, M., Deb, K., Jo, K.-H.: Geometrical feature based stairways detection and recognition using depth sensor. In: IECON 2018—44th Annual Conference of the IEEE Industrial Electronics Society, pp. 3250–3255 (2018). https://doi.org/10.1109/IECON.2018.8591340

  22. Redmon, J., Farhadi, A.: YOLOv3: An Incremental Improvement (2018)

  23. Patil, U., Gujarathi, A., Kulkarni, A., Jain, A., Malke, L., Tekade, R., Paigwar, K., Chaturvedi, P.: Deep learning based stair detection and statistical image filtering for autonomous stair climbing. In: 2019 Third IEEE International Conference on Robotic Computing (IRC), pp. 159–166 (2019). https://doi.org/10.1109/IRC.2019.00031

  24. Wang, C., Pei, Z., Shuang, Q., Tang, Z.: Deep leaning-based ultra-fast stair detection. Sci. Rep. 12, 16124 (2022). https://doi.org/10.1038/s41598-022-20667-w

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Wang, C., Pei, Z., Qiu, S., Tang, Z.: RGB-D-based stair detection and estimation using deep learning. Sensors (2023). https://doi.org/10.3390/s23042175

    Article  PubMed  PubMed Central  Google Scholar 

  26. Huang, X., Tang, Z.: Staircase detection algorithm based on projection-histogram. In: 2018 2nd IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), pp. 1130–1133 (2018). Doi: https://doi.org/10.1109/IMCEC.2018.8469186

  27. Carbonara, S., Guaragnella, C.: Efficient stairs detection algorithm assisted navigation for vision impaired people. In: 2014 IEEE International Symposium on Innovations in Intelligent Systems and Applications (INISTA) Proceedings, pp. 313–318 (2014). https://doi.org/10.1109/INISTA.2014.6873637

  28. Khaliluzzaman, M., Deb, K., Jo, K.-H.: Stairways detection and distance estimation approach based on three connected point and triangular similarity. In: 2016 9th International Conference on Human System Interactions (HSI), pp. 330–336 (2016). https://doi.org/10.1109/HSI.2016.7529653

  29. Diamantis, D.E., Koutsiou, D.C.C., Iakovidis, D.K.: Staircase detection using a lightweight look-behind fully convolutional neural network. In: Macintyre, J., Iliadis, L., Maglogiannis, I., Jayne, C. (eds.) Engineering Applications of Neural Networks, pp. 522–532. Springer, Cham (2019)

  30. Lee, Y.H., Leung, T.-S., Medioni, G.: Real-time staircase detection from a wearable stereo system. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), pp. 3770–3773 (2012)

  31. Rekhawar, N., Govindani, Y., Rao, N.: Deep learning based detection, segmentation and vision based pose estimation of staircase. In: 2022 1st International Conference on the Paradigm Shifts in Communication, Embedded Systems, Machine Learning and Signal Processing (PCEMS), pp. 78–83 (2022). https://doi.org/10.1109/PCEMS55161.2022.9807915

  32. Glenn, J.: yolov5 (2019). https://github.com/ultralytics/yolov5

  33. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

  34. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  35. Wang, S., Pan, H., Zhang, C., Tian, Y.: RGB-D image-based detection of stairs, pedestrian crosswalks and traffic signs. J. Vis. Commun. Image Represent. 25(2), 263–272 (2014). https://doi.org/10.1016/j.jvcir.2013.11.005

    Article  CAS  Google Scholar 

  36. Wang, S., Tian, Y.: Detecting stairs and pedestrian crosswalks for the blind by RGBD camera. In: 2012 IEEE International Conference on Bioinformatics and Biomedicine Workshops, pp. 732–739 (2012). https://doi.org/10.1109/BIBMW.2012.6470227

  37. Munoz, R., Rong, X., Tian, Y.: Depth-aware indoor staircase detection and recognition for the visually impaired. In: 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 1–6 (2016). https://doi.org/10.1109/ICMEW.2016.7574706

  38. Pérez-Yus, A., López-Nicolás, G., Guerrero, J.J.: Detection and modelling of staircases using a wearable depth sensor. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) Computer Vision—ECCV 2014 Workshops, pp. 449–463. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16199-0_32

  39. Yifei, Y., Jianzhong, W.: Stair area recognition in complex environment based on point cloud. J. Electron. Meas. Instrum. 34(4), 124–133 (2020). https://doi.org/10.13382/j.jemi.B1902746

    Article  Google Scholar 

  40. Matsumura, H., Premachandra, C.: Deep-learning-based stair detection using 3d point cloud data for preventing walking accidents of the visually impaired. IEEE Access 10, 56249–56255 (2022). https://doi.org/10.1109/ACCESS.2022.3178154

    Article  Google Scholar 

  41. Charles, R.Q., Su, H., Kaichun, M., Guibas, L.J.: Pointnet: deep learning on point sets for 3d classification and segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77–85 (2017). https://doi.org/10.1109/CVPR.2017.16

  42. Zatout, C., Larabi, S.: Semantic scene synthesis: application to assistive systems. Vis. Comput. 38, 2691–2705 (2022). https://doi.org/10.1007/s00371-021-02147-w

    Article  Google Scholar 

  43. Zatout, C., Larabi, S., Mendili, I., Barnabé, S.A.E.: Ego-semantic labeling of scene from depth image for visually impaired and blind people. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 4376–4384 (2019). https://doi.org/10.1109/ICCVW.2019.00538

  44. Zhou, X., Wang, D., Krähenbühl, P.: Objects as Points (2019)

  45. Hu, J., Shen, L., Albanie, S., Sun, G., Wu, E.: Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell. 42(8), 2011–2023 (2020). https://doi.org/10.1109/TPAMI.2019.2913372

    Article  PubMed  Google Scholar 

  46. Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5987–5995 (2017). https://doi.org/10.1109/CVPR.2017.634

  47. Wang, C., Pei, Z., Qiu, S., Tang, Z.: Stair dataset with depth maps. Mendeley Data (2023). https://doi.org/10.17632/p28ncjnvgk.2

  48. Intel: Depth Camera D435i. https://www.intelrealsense.com/depth-camera-d435i/

  49. Wang, C., Pei, Z., Qiu, S., Wang, Y., Tang, Z.: RGB-D stair dataset. Mendeley Data (2023). https://doi.org/10.17632/6kffmjt7g2.1

    Article  Google Scholar 

  50. Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Garcia-Rodriguez, J.: A review on deep learning techniques. Appl. Semant. Segm. 9, 1–7 (2017)

    CAS  Google Scholar 

  51. Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization (2017)

  52. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (2017)

  53. Zhou, Q.-Y., Park, J., Koltun, V.: Open3D: a modern library for 3D data processing. arXiv:1801.09847 (2018)

  54. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  Google Scholar 

Download references

Funding

No funding was received to assist with the preparation of this manuscript.

Author information

Author notes

  1. Zhongcai Pei, Shuang Qiu and Yachun Wang have contributed equally to this work.

Authors and Affiliations

  1. School of Automation Science and Electrical Engineering, Beihang University, Xueyuan Road, Beijing, 100191, Beijing, China

    Chen Wang, Zhongcai Pei, Shuang Qiu, Yachun Wang & Zhiyong Tang

Authors
  1. Chen Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhongcai Pei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuang Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yachun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhiyong Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

CW, ZP, SQ and YW made the dataset. CW and SQ made all the figures used in the paper. CW designed the software architecture and wrote the paper. CW, YW and ZT conceived the experiments and conducted the experiments. YW, SQ and ZP analysed the results. All authors reviewed the manuscript.

Corresponding author

Correspondence to Zhiyong Tang.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, C., Pei, Z., Qiu, S. et al. StairNetV3: depth-aware stair modeling using deep learning. Vis Comput (2024). https://doi.org/10.1007/s00371-024-03268-8

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00371-024-03268-8

Keywords

Search

Navigation