Skip to main content

Real-time road surface and semantic lane estimation using deep features

Signal, Image and Video Processing volume 12, pages 1133–1140 (2018)Cite this article

Abstract

In this article, we present a robust real-time road surface and semantic lane marker estimation algorithm using the deconvolution neural network and extra trees-based decision forest. Our proposed algorithm simultaneously performs three environment perception tasks on colour and depth images, even under challenging conditions, namely road surface estimation, lane marker localization, and lane marker semantic information estimation. The lane marker semantic information implies the lane marker type such as dotted lane marker or continuous lane marker. The task of road surface estimation is performed with a trained deconvolution neural network. For the lane marker localization task, a scene-based extra trees regression framework is used to localize the lane markers in the given road. To account for the variations in the number and characteristics of the lane markers in the road scene, multiple regression models indexed with scene labels are used. The pre-defined scene labels correspond to the lane marker variations in a given scene, and an extra trees-based classification model is trained to estimate them from the road features. The road features, given as an input to the extra trees frameworks, are extracted from the road image using the trained filters of the deconvolution network. The proposed algorithm is validated using multiple acquired datasets. A comparative analysis is also conducted with baseline algorithms, and an improved accuracy is reported. Moreover, a detailed parameter evaluation is also performed. We report a computational time of 90 ms per frame.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. Adachi E, Inayoshi H, Kurita, T.: Estimation of lane state from car-mounted camera using multiple-model particle filter based on voting result for one-dimensional parameter space. In: MVA (2007)

  2. Alvarez, JM., Gevers, T., Lopez, AM.: 3d scene priors for road detection. In: CVPR (2010)

  3. Aly, M.: Real time detection of lane markers in urban streets. In: IVS (2008)

  4. Andrew, H., Lai, S., Nelson, H., Yung, C.: Lane detection by orientation and length discrimination. SMC 30(4), 539–548 (2000)

    Google Scholar 

  5. Arshad, N., Moon, K., Park, S., Kim, J.: Lane detection with moving vehicle using colour information. In: World Congress on Engineering and Computer Science (2011)

  6. Bertozzi, M., Broggi, A.: Gold: a parallel real-time stereo vision system for generic obstacle and lane detection. TIP 7(1), 62–81 (1998)

    Google Scholar 

  7. Cheng, H.Y., Jeng, B.S., Tseng, P.T., Fan, K.C.: Lane detection with moving vehicles in the traffic scenes. IEEE Trans ITS 7(4), 571–582 (2006)

    Google Scholar 

  8. Choi, H., Park, J., Choi, W., Oh, S.: Vision-based fusion of robust lane tracking and forward vehicle detection in a real driving environment. Int. J. Automot. Technol. 13(4), 653–669 (2012)

    Article  Google Scholar 

  9. Collado, JM., Hilario, C., de la Escalera, A., Armingol, JM.: Detection and classification of road lanes with a frequency analysis. In: IVS (2005)

  10. El Jaafari, I., El Ansari, M., Koutti, L.: Fast edge-based stereo matching approach for road applications. Signal Image Video Process. 11(2), 267–274 (2017)

    Article  Google Scholar 

  11. Friedman, N., Geiger, D., Goldszmidt, M.: Bayesian network classifiers. Mach. Learn. 29(2–3), 131–163 (1997)

    Article  MATH  Google Scholar 

  12. Gao, Y., Song, Y., Yang, Z.: A real-time drivable road detection algorithm in urban traffic environment. In: ICCVG (2012)

  13. He, Y., Wang, H., Zhang, B.: Color-based road detection in urban traffic scenes. IEEE Trans. ITS 5(4), 309–318 (2004)

    Google Scholar 

  14. Huang, A.S., Teller, S.: Probabilistic lane estimation for autonomous driving using basis curves. Auton. Robot. 31(2), 269–283 (2011)

    Article  Google Scholar 

  15. Jia, B., Feng, W., Zhu, M.: Obstacle detection in single images with deep neural networks. Signal Image Video Process. 10(6), 1033–1040 (2016)

    Article  Google Scholar 

  16. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guaddarrame, S., Darrel, T.: Caffe: Convolutional architecture for fast feature embedding. In: arXiv preprint arXiv:1408.5093 (2014)

  17. John, V., Liu, Z., Guo, C., Mita, S., Kidono, K.: Real-time lane estimation using deep features and extra trees regression. In: PSIVT (2015)

  18. John, V., Guo, C., Mita, S., Kidono, K., Guo, C., Ishimaru, K.: Fast road scene segmentation using deep learning and scene-based models. In: ICPR (2016)

  19. Kim, J., Lee, M.: Robust lane detection based on convolutional neural network and random sample consensus. In: NIPS (2014)

  20. Kowsari, T., Beauchemin, SS., Bauer, MA.: Map-based lane and obstacle-free area detection. In: VISAPP (2014)

  21. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015)

  22. Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. CoRR abs/1505.04366 (2015)

  23. Ozgunalp, U., Ai, X., Dahnoun, N.: Stereo vision-based road estimation assisted by efficient planar patch calculation. Signal Image Video Process. 10(6), 1127–1134 (2016)

    Article  Google Scholar 

  24. Prochazka, Z.: Road region segmentation based on sequential monte-carlo estimation. In: ICARCV (2008)

  25. Protasov, S., Khan, A.M., Sozykin, K., Ahmad, M.: Using deep features for video scene detection and annotation. Signal Image Video Process. (2018). https://doi.org/10.1007/s11760-018-1244-6.

  26. Samadzadegan, F., Sarafraz, A., Tabibi, M.: Automatic lane detection in image sequences for vision-based navigation purpose. In: IEVM (2006)

  27. Sehestedt, S., Kodagoda, S., Alempijevic, A., Dissanayake, G.: Efficient lane detection and tracking in urban environments. In: ECMR (2007)

  28. Son, TT., Mita, S., Takeuchi, A.: Road detection using segmentation by weighted aggregation based on visual information and a posteriori probability of road regions. In: SMC (2008)

  29. Sotelo, M.A., Rodriguez, F.J., Magdalena, L., Bergasa, L.M., Boquete, L.: A color vision-based lane tracking system for autonomous driving on unmarked roads. Auton. Robot. 16(1), 95–116 (2004)

    Article  Google Scholar 

  30. Southall, B., Taylor, CJ.: Stochastic road shape estimation. In: ICCV (2001)

  31. Wang, Y., Shen, D., Teoh, E.K.: Lane detection using spline model. Pattern Recognit. Lett. 21(9), 677–689 (2000)

    Article  Google Scholar 

  32. Wu, M., Lam, S.K., Srikanthan, T.: Nonparametric technique based high-speed road surface detection. IEEE Trans. ITS 16(2), 874–884 (2015)

    Google Scholar 

  33. Yenikaya, S., Yenikaya, G., Düven, E.: Keeping the vehicle on the road: a survey on on-road lane detection systems. ACM Comput. Surv. 46(1), 2:1–2:43 (2013)

    Article  Google Scholar 

  34. Yun, S., Guo-ying, Z., Yong, Y.: A road detection algorithm by boosting using feature combination. In: IVS (2007)

Download references

Author information

Authors and Affiliations

  1. Toyota Technological Institute, Nagoya, Japan

    V. John & S. Mita

  2. Toyota Central R & D Labs, Nagakute, Japan

    C. Guo & K. Kidono

  3. University of British Columbia, Vancouver, Canada

    Z. Liu

Authors
  1. V. John
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Z. Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. S. Mita
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. C. Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. K. Kidono
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to V. John.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

John, V., Liu, Z., Mita, S. et al. Real-time road surface and semantic lane estimation using deep features. SIViP 12, 1133–1140 (2018). https://doi.org/10.1007/s11760-018-1264-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-018-1264-2

Keywords

  • Deep learning
  • Intelligent vehicles
  • Lane and road surface detection