Enabling Pedestrian Safety Using Computer Vision Techniques: A Case Study of the 2018 Uber Inc. Self-driving Car Crash

  • Puneet KohliEmail author
  • Anjali Chadha
Conference paper
Part of the Lecture Notes in Networks and Systems book series (LNNS, volume 69)


Human lives are important. The decision to allow self-driving vehicles operate on our roads carries great weight. This has been a hot topic of debate between policy-makers, technologists and public safety institutions. The recent Uber Inc. self-driving car crash, resulting in the death of a pedestrian, has strengthened the argument that autonomous vehicle technology is still not ready for deployment on public roads. In this work, we analyze the Uber car crash and shed light on the question, “Could the Uber Car Crash have been avoided?”. We apply state-of-the-art Computer Vision models to this highly practical scenario. More generally, our experimental results are an evaluation of various image enhancement and object recognition techniques for enabling pedestrian safety in low-lighting conditions using the Uber crash as a case study.


Computer vision Pedestrian detection Image enhancement Autonomous vehicles 


  1. 1.
    Association for Safe International Road Travel: Annual global road crash statistics (2013).
  2. 2.
    Blincoe, L.J., Miller, T.R., Zaloshnja, E., Lawrence, B.A.: The economic and societal impact of motor vehicle crashes, 2010. (Revised) (report no. dot hs 812 013). National Highway Traffic Safety Administration, Washington, DC (2015)Google Scholar
  3. 3.
    Anderson, J.M., Nidhi, K., Stanley, K.D., Sorensen, P., Samaras, C., Oluwatola, O.A.: Autonomous Vehicle Technology: A Guide for Policymakers. Rand Corporation (2014)Google Scholar
  4. 4.
    Fagnant, D.J., Kockelman, K.: Preparing a nation for autonomous vehicles: opportunities, barriers and policy recommendations. Transp. Res. Part A: Policy Pract. 77, 167–181 (2015)Google Scholar
  5. 5.
    Kalra, N., Paddock, S.M.: Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability? Transp. Res. Part A: Policy Pract. 94, 182–193 (2016)Google Scholar
  6. 6.
    Shashua, A.: Experience counts, particularly in safety-critical areas (2018).
  7. 7.
    Benenson, R., Mathias, M., Timofte, R., Van Gool, L.: Pedestrian detection at 100 frames per second. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2903–2910. IEEE (2012)Google Scholar
  8. 8.
    Dollár, P., Appel, R., Belongie, S., Perona, P.: Fast feature pyramids for object detection. IEEE Trans. Pattern Anal. Mach. Intell. 36(8), 1532–1545 (2014)CrossRefGoogle Scholar
  9. 9.
    Dollr, P., Tu, Z., Perona, P., Belongie, S.J.: Integral channel features. In: Cavallaro, A., Prince, S., Alexander, D.C. (eds.) BMVC, pp. 1–11. British Machine Vision Association (2009)Google Scholar
  10. 10.
    Sermanet, P., Kavukcuoglu, K., Chintala, S., LeCun, Y.: Pedestrian detection with unsupervised multi-stage feature learning. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3626–3633. IEEE (2013)Google Scholar
  11. 11.
    Zhang, S., Bauckhage, C., Cremers, A.B.: Informed Haar-like features improve pedestrian detection. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 947–954. IEEE (2014)Google Scholar
  12. 12.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 1, pp. 886–893, June 2005Google Scholar
  13. 13.
    Burges, C.J.C.: A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Discov. 2(2), 121–167 (1998)CrossRefGoogle Scholar
  14. 14.
    Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55(1), 119–139 (1997)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Benenson, R., Omran, M., Hosang, J.H., Schiele, B.: Ten years of pedestrian detection, what have we learned? CoRR, abs/1411.4304 (2014)Google Scholar
  16. 16.
    Paisitkriangkrai, S., Shen, C., Van Den Hengel, A.: Strengthening the effectiveness of pedestrian detection with spatially pooled features. In: European Conference on Computer Vision, pp. 546–561. Springer, Heidelberg (2014)Google Scholar
  17. 17.
    Liu, F., Picard, R.W.: Finding periodicity in space and time. In: Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271), pp. 376–383, January 1998Google Scholar
  18. 18.
    Polana, R., Nelson, R.: Detecting activities. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2–7, June 1993Google Scholar
  19. 19.
    Cutler, R., Davis, L.S.: Robust real-time periodic motion detection, analysis, and applications. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 781–796 (2000). AugCrossRefGoogle Scholar
  20. 20.
    Viola, P., Jones, M.J., Snow, D.: Detecting pedestrians using patterns of motion and appearance. Int. J. Comput. Vis. 63(2), 153–161 (2005). JulyCrossRefGoogle Scholar
  21. 21.
    Barinova, O., Lempitsky, V., Kholi, P.: On detection of multiple object instances using hough transforms. IEEE Trans. Pattern Anal. Mach. Intell. 34(9), 1773–1784 (2012)CrossRefGoogle Scholar
  22. 22.
    Sharma, V., Davis, J.W.: Integrating appearance and motion cues for simultaneous detection and segmentation of pedestrians. In: IEEE 11th International Conference on Computer Vision, ICCV 2007, pp.1–8. IEEE (2007)Google Scholar
  23. 23.
    Spinello, L., Triebel, R., Siegwart, R.: Multimodal detection and tracking of pedestrians in urban environments with explicit ground plane extraction. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2008, pp. 1823–1829. IEEE (2008)Google Scholar
  24. 24.
    Chidiac, H., Ziou, D.: Classification of image edges. Vis. Interface 99, 17–24 (1999)Google Scholar
  25. 25.
    Clavier, E., Clavier, S., Labiche, J.: Image sorting and image classification: a global approach. In: Proceedings of the Fifth International Conference on Document Analysis and Recognition, ICDAR 1999, pp. 123–126. IEEE (1999)Google Scholar
  26. 26.
    Haralick, R.M., Shapiro, L.G.: Computer and Robot Vision. Addison-Wesley, Boston (1992)Google Scholar
  27. 27.
    Bovik, A.C.: Handbook of Image and Video Processing. Academic Press, Cambridge (2010)zbMATHGoogle Scholar
  28. 28.
    Garcia, P., Pla, F., Gracia, I.: Detecting edges in colour images using dichromatic differences. In: Seventh International Conference on Image Processing And Its Applications (Conf. Publ. No. 465), vol. 1, pp. 363–367, July 1999Google Scholar
  29. 29.
    Canny, J.: A computational approach to edge detection. In: Readings in Computer Vision, pp. 184–203. Elsevier, Amsterdam (1987)Google Scholar
  30. 30.
    Dollar, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: an evaluation of the state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 34(4), 743–761 (2012). AprilCrossRefGoogle Scholar
  31. 31.
    Wojek, C., Walk, S., Roth, S., Schiele, B.: Monocular 3D scene understanding with explicit occlusion reasoning. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1993–2000. IEEE (2011)Google Scholar
  32. 32.
    Mathias, M., Benenson, R., Timofte, R., Van Gool, L., Handling occlusions with Franken-classifiers. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 1505–1512. IEEE (2013)Google Scholar
  33. 33.
    Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part-based models. IEEE Trans. Pattern Anal. Mach. Intell. 32(9), 1627–1645 (2010)CrossRefGoogle Scholar
  34. 34.
    Ouyang, W., Wang, X.: A discriminative deep model for pedestrian detection with occlusion handling. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3258–3265. IEEE (2012)Google Scholar
  35. 35.
    Ouyang, W., Zeng, X., Wang, X.: Modeling mutual visibility relationship in pedestrian detection. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3222–3229. IEEE (2013)Google Scholar
  36. 36.
    Ouyang, W., Wang, X.: Joint deep learning for pedestrian detection. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 2056–2063. IEEE (2013)Google Scholar
  37. 37.
    Wu, B., Nevatia, R.: Detection of multiple, partially occluded humans in a single image by Bayesian combination of edgelet part detectors. In: Tenth IEEE International Conference on Computer Vision, ICCV 2005, vol. 1, pp. 90–97. IEEE (2005)Google Scholar
  38. 38.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)Google Scholar
  39. 39.
    Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. CoRR, abs/1612.08242 (2016)Google Scholar
  40. 40.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779–788, June 2016Google Scholar
  41. 41.
    Girshick, R.: Fast R-CNN. arXiv preprint arXiv:1504.08083 (2015)
  42. 42.
    Saini, S., Nikhil, S., Konda, K.R., Bharadwaj, H.S., Ganeshan, N.: An efficient vision-based traffic light detection and state recognition for autonomous vehicles. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 606–611, June 2017Google Scholar
  43. 43.
    Girshick, R.B., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. CoRR, abs/1311.2524 (2013)Google Scholar
  44. 44.
    Zhang, L., Lin, L., Liang, X., He, K.: Is faster R-CNN doing well for pedestrian detection? In: European Conference on Computer Vision, pp. 443–457. Springer, Heidelberf (2016)CrossRefGoogle Scholar
  45. 45.
    Li, J., Liang, X., Shen, S.M., Tingfa, X., Feng, J., Yan, S.: Scale-aware fast R-CNN for pedestrian detection. IEEE Trans. Multimed. 20(4), 985–996 (2018)Google Scholar
  46. 46.
    Tian, Y., Luo, P., Wang, X., Tang, X.: Deep learning strong parts for pedestrian detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1904–1912 (2015)Google Scholar
  47. 47.
    Tian, Y., Luo, P., Wang, X., Tang, X.: Pedestrian detection aided by deep learning semantic tasks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5079–5087 (2015)Google Scholar
  48. 48.
    Appel, R., Fuchs, T., Dollár, P., Perona, P.: Quickly boosting decision trees–pruning underachieving features early. In: International Conference on Machine Learning, pp. 594–602 (2013)Google Scholar
  49. 49.
    Zhang, N., Donahue, J., Girshick, R., Darrell, T.: Part-based R-CNNs for fine-grained category detection. In: European Conference on Computer Vision, pp. 834–849. Springer, Heidelberg (2014)Google Scholar
  50. 50.
    Ess, A., Leibe, B., Schindler, K., Van Gool, L.: A mobile vision system for robust multi-person tracking. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008, pp. 1–8. IEEE (2008)Google Scholar
  51. 51.
    Dollar, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: a benchmark. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 304–311, June 2009Google Scholar
  52. 52.
    Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the KITTI dataset. Int. J. Rob. Res. 32(11), 1231–1237 (2013). SeptemberCrossRefGoogle Scholar
  53. 53.
    Wojek, C., Walk, S., Schiele, B.: Multi-cue onboard pedestrian detection. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. pages 794–801. IEEE (2009)Google Scholar
  54. 54.
    Pizer, S.M., Amburn, E.P., Austin, J.D., Cromartie, R., Geselowitz, A., Greer, T., ter Haar Romeny, B., Zimmerman, J.B., Zuiderveld, K.: Adaptive histogram equalization and its variations. Comput. Vis. Graph. Image Process. 39(3), 355–368 (1987)CrossRefGoogle Scholar
  55. 55.
    Reza, A.M.: Realization of the contrast limited adaptive histogram equalization (CLAHE) for real-time image enhancement. J. VLSI Sig. Process. Syst. Sig. Image Video Technol. 38(1), 35–44 (2004)CrossRefGoogle Scholar
  56. 56.
    Land, E.H.: The retinex theory of color vision. Sci. Am. 237(6), 108–129 (1977)CrossRefGoogle Scholar
  57. 57.
    Jobson, D.J., Rahman, Z., Woodell, G.A.: A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Trans. Image Process. 6(7), 965–976 (1997)CrossRefGoogle Scholar
  58. 58.
    Lore, K.G., Akintayo, A., Sarkar, S.: LLNet: a deep autoencoder approach to natural low-light image enhancement. CoRR, abs/1511.03995 (2015)Google Scholar
  59. 59.
    Gharbi, M., Chen, J., Barron, J.T., Hasinoff, S.W., Durand, F.: Deep bilateral learning for real-time image enhancement. ACM Trans. Graph. 36(4), 1–12 (2017)CrossRefGoogle Scholar
  60. 60.
    Shen, L., Yue, Z., Feng, F., Chen, Q., Liu, S., Ma, J.: MSR-Net: low-light image enhancement using deep convolutional network. CoRR, abs/1711.02488 (2017)Google Scholar
  61. 61.
    Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  62. 62.
    Cai, B., Xiangmin, X., Jia, K., Qing, C., Tao, D.: DehazeNet: an end-to-end system for single image haze removal. IEEE Trans. Image Process. 25(11), 5187–5198 (2016)MathSciNetCrossRefGoogle Scholar
  63. 63.
    Xu, Y., Xu, L., Li, D., Wu, Y.: Pedestrian detection using background subtraction assisted support vector machine. In: 2011 11th International Conference on Intelligent Systems Design and Applications, pp. 837–842, November 2011Google Scholar
  64. 64.
    KaewTraKulPong, P., Bowden, R.: An Improved Adaptive Background Mixture Model for Real-time Tracking with Shadow Detection, pp. 135–144. Springer, Boston (2002)CrossRefGoogle Scholar
  65. 65.
    Zivkovic, Z.: Improved adaptive Gaussian mixture model for background subtraction. In: Proceedings of the 17th International Conference on Pattern Recognition, ICPR 2004, vol. 2, pp. 28–31, August 2004Google Scholar
  66. 66.
    Zivkovic, Z., van der Heijden, F.: Efficient adaptive density estimation per image pixel for the task of background subtraction. Pattern Recogn. Lett. 27(7), 773–780 (2006). MayCrossRefGoogle Scholar
  67. 67.
    Shadeed, W.G., Abu-Al-Nadi, D.I., Mismar, M.J.: Road traffic sign detection in color images. In: Proceedings of the 2003 10th IEEE International Conference on Electronics, Circuits and Systems, ICECS 2003, vol. 2, pp. 890–893. IEEE (2003)Google Scholar
  68. 68.
    Broggi, A., Cerri, P., Medici, P., Porta, P.P., Ghisio, G.: Real time road signs recognition. In: 2007 IEEE Intelligent Vehicles Symposium, pp. 981–986. IEEE (2007)Google Scholar
  69. 69.
    Tarel, J.-P., Hautiere, N., Caraffa, L., Cord, A., Halmaoui, H., Gruyer, D.: Vision enhancement in homogeneous and heterogeneous fog. IEEE Intell. Transp. Syst. Mag. 4(2), 6–20 (2012)CrossRefGoogle Scholar
  70. 70.
    Singh, S., Godara, A., Gaurav, G.: Detection of partial invisible objects in images using histogram equalization. Int. J. Comput. Appl. 85(9), 40–44 (2014). JanuaryGoogle Scholar
  71. 71.
    Pisano, E.D., Zong, S., Hemminger, B.M., DeLuca, M., Johnston, R.E., Muller, K., Braeuning, M.P., Pizer, S.M.: Contrast limited adaptive histogram equalization image processing to improve the detection of simulated spiculations in dense mammograms. J. Digit. Imaging 11(4), 193 (1998)CrossRefGoogle Scholar
  72. 72.
    Tian, Q.M., Luo, Y.P., Hu, D.C.: Nighttime pedestrian detection with a normal camera using SVM classifier. In: Third International Conference on Image and Graphics (ICIG 2004), pp. 116–119. IEEE (2004)Google Scholar
  73. 73.
    Chang, W.-Y., Chiu, C.-C., Yang, J.-H.: Block-based connected-component labeling algorithm using binary decision trees. Sensors 15(9), 23763–23787 (2015)CrossRefGoogle Scholar
  74. 74.
    Ying, Z., Li, G., Ren, Y., Wang, R., Wang, W.: A new image contrast enhancement algorithm using exposure fusion framework. In: International Conference on Computer Analysis of Images and Patterns, pp. 36–46. Springer, Heidelberg (2017)CrossRefGoogle Scholar
  75. 75.
    Ying, Z., Li, G., Ren, Y., Wang, R., Wang, W.: A new low-light image enhancement algorithm using camera response model. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 3015–3022, October 2017Google Scholar
  76. 76.
    Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018)
  77. 77.
    Wu, B., Iandola, F., Jin, P.H., Keutzer, K.: SqueezeDet: unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving. arXiv preprint arXiv:1612.01051 (2016)
  78. 78.
    Teichmann, M., Weber, M., Zoellner, M., Cipolla, R., Urtasun, R.: MultiNet: real-time joint semantic reasoning for autonomous driving. arXiv preprint arXiv:1612.07695 (2016)
  79. 79.
    Alvar, S.R., Bajić, I.V.: MV-YOLO: motion vector-aided tracking by semantic object detection. arXiv preprint arXiv:1805.00107 (2018)
  80. 80.
    Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S.E., Fu, C.-Y., Berg, A.C.: SSD: single shot multibox detector. CoRR, abs/1512.02325 (2015)Google Scholar
  81. 81.
    Du, X., El-Khamy, M., Lee, J., Davis, L.: Fused DNN: a deep neural network fusion approach to fast and robust pedestrian detection. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 953–961. IEEE (2017)Google Scholar
  82. 82.
    Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. arXiv preprint arXiv:1708.02002 (2017)

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Texas A&M UniversityCollege StationUSA

Personalised recommendations