Skip to main content
SpringerLink
Log in

A GPU-accelerated real-time contextual awareness application for the visually impaired on Google’s project Tango device

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

An application for the recently introduced Google Project Tango Tablet Development Kit to assist visually impaired (VI) users in understanding their environmental context by identifying and locating multiple faces and objects in their vicinity in real-time is presented. CUDA-based GPU-accelerated algorithms would be utilized to detect and recognize faces and objects from the visual data, while the locations of these entities relative to the user would be estimated from the depth data acquired via the tablet. The interaction would be speech based with the user being offered several options for requesting information about the identities and/or relative locations of face and objects. The aim is to create a portable, affordable, power-efficient, standalone assistive application to increase the autonomy of VI users which can run in real time on the device itself.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Krishna S, Colbry D, Black J, Balasubramanian V, Panchanathan S (2008) A systematic requirements analysis and development of an assistive device to enhance the social interaction of people who are blind or visually impaired. In: Workshop on computer vision applications for the visually impaired. Marseille, France

  2. Manduchi R, Kurniawan S (2011) Mobility-related accidents experienced by people with visual impairment. AER Journal: Research and Practice in Visual Impairment and Blindness 4(2):44–54

  3. Legood R, Scuffham P, Cryer C (2002) Are we blind to injuries in the visually impaired? A review of the literature. Inj Prev 8:155–160

    Article  Google Scholar 

  4. Jafri R, Ali S (2014) A GPS-based personalized pedestrian route recording smartphone application for the blind. In: Stephanidis C (ed) HCI international 2014—Posters’ extended abstracts. Communications in Computer and Information Science (CCIS), vol. 435, Springer International Publishing, New York, pp 232–237, June 22–27 2014

  5. Jafri R, Ali S, Arabnia H, Fatima S (2014) Computer vision-based object recognition for the visually impaired in an indoors environment: a survey. Vis Comput 30: 1197–1222

  6. (2012). Visual impairment and blindness: fact sheet number 282. http://www.who.int/mediacentre/factsheets/fs282/en/ ed: WHO media center

  7. Cheng KT, Wang YC (2011) Using mobile GPU for general-purpose computing—a case study of face recognition on smartphones. In: VLSI Design, Automation and Test (VLSI-DAT), 2011 International Symposium on, pp 1–4

  8. Nvidia, “Nvidia compute unified device architecture”, http://www.nvidia.com/object/cuda_home_new.html

  9. Google Project Tango. https://www.google.com/atap/project-tango/

  10. Nvidia Tegra K1. http://www.nvidia.com/object/tegra-k1-processor.html

  11. Jafri R, Ali SA, Arabnia HR (2013) Face recognition for the visually impaired. In: Proceedings of the 2013 International Conference on Information and Knowledge Engineering (IKE ’13), Las Vegas, Nevada, pp 153–159

  12. Jafri R, Ali SA, Arabnia HR (2013) Computer vision-based object recognition for the visually impaired using visual tags. In: Proceedings of the 2013 International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV ’13), Las Vegas, Nevada, pp 400–406

  13. Jafri R, Ali SA (2014) A multimodal tablet–based application for the visually impaired for detecting and recognizing objects in a home environment. In: Proceedings of the 14th International Conference on Computers Helping People with Special Needs (ICCHP 2014), Lecture Notes in Computer Science (LNCS), Springer International Publishing, Vol. 8547, Paris, France, pp 356–359, July 9-11 2014

  14. Jafri R, Ali SA (2014) Exploring the potential of eyewear-based wearable display devices for use by the visually impaired. In: Proceedings of the 3rd International Conference on User Science and Engineering (i-USEr 2014), Shah Alam, Malaysia, pp 119–124

  15. Jafri R, Khan MM (2016) Obstacle detection and avoidance for the visually impaired in indoors environments using Google’s Project Tango Device. In: Miesenberger K, Bühler C, Penaz P (eds.) Computers Helping People with Special Needs: 15th International Conference, ICCHP 2016, Linz, Austria, July 13–15, 2016, Proceedings, Part II. Lecture Notes in Computer Science (LNCS),vol. 9759, SpringerInternational Publishing, pp 179–185

  16. Jafri R, Campos RL, Ali SA, Arabnia HR (2016) Utilizing the google project tango tablet development kit and the unity engine for image and infrared data-based obstacle detection for the visually impaired. In: Proceedings of the 2016 International Conference on Health Informatics and Medical Systems (HIMS’15), Las Vegas, Nevada

  17. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: IEEE Computer Society Conference On Computer Vision and Pattern Recognition, pp 511–518

  18. Hefenbrock D, Oberg J, Thanh NTN, Kastner R, Baden SB (2010) Accelerating Viola-Jones face detection to FPGA-level using GPUs. Annual IEEE symposium on field-programmable custom computing machines, pp 11–18

  19. Sharma B, Thota R, Vydyanathan N, Kale A (2009) Towards a robust, real-time face processing system using CUDA-enabled GPUs. In: High performance computing (HiPC). Int Conference on 2009, pp 368–377

  20. Oro D, Fernández C, Saeta JR, Martorell X, Hernando J (2011) Real-time GPU-based face detection in HD video sequences. In: Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, pp 530–537

  21. Oro D, Fern’ndez C, Segura C, Martorell X, Hernando J (2012) Accelerating boosting-based face detection on GPUs. In: Parallel Processing (ICPP), 2012 41st International Conference on, pp 309–318

  22. Krpec J, Němec M (2012) Face detection CUDA accelerating. In: The Fifth International Conference on Advances in Computer–human Interactions (ACHI 2012), pp 155–160

  23. Ren M, Zhang S, Lei Y, Zhang M (2014) CUDA-based real-time face recognition system. In: Digital information and communication technology and it’s applications (DICTAP). Fourth International Conference on, pp 237–241

  24. Jain V, Patel D (2016) A GPU based implementation of robust face detection system. Proc Comput Sci 87:156–163

    Article  Google Scholar 

  25. Martínez-Zarzuela M, Díaz-Pernas FJ, Antón-Rodríguez M, Perozo-Rondón F, González-Ortega D (2011) AdaBoost face detection on the GPU using haar-like features. In: Ferrández JM, Álvarez JR, Sánchez, de la Paz, Toledo FJ (eds) New Challenges on Bioinspired Applications: 4th International Work-conference on the Interplay Between Natural and Artificial Computation, IWINAC 2011, La Palma, Canary Islands, Spain, May 30–June 3, 2011. Proceedings, Part II. Springer, Berlin, pp 333–342

  26. Li E, Wang B, Yang L, Peng Yt, Du Y, Zhang Y, Chiu YJ (2012) GPU and CPU cooperative accelaration for face detection on modern processors. In: Multimedia and expo (ICME), 2012 IEEE International Conference on, pp 769–775

  27. Kong J, Deng Y (2010) GPU accelerated face detection. In: Intelligent control and information processing (ICICIP). International Conference on 2010, pp 584–588

  28. Jia H, Zhang Y, Wang W, Xu J (2012) Accelerating Viola-Jones facce detection algorithm on GPUs. In: High Performance Computing and Communication & 2012 IEEE 9th International Conference on Embedded Software and Systems (HPCC-ICESS), 2012 IEEE 14th International Conference on, pp 396–403

  29. Cheng S, Asthana A, Zafeiriou S, Shen J, Pantic M (2014) Real-time generic face tracking in the wild with CUDA. In: Proceedings of the 5th ACM Multimedia Systems Conference, Singapore, Singapore, pp 148–151

  30. Nasse F, Thurau C, Fink GA (2009) Face detection using gpu-based convolutional neural networks. In: Jiang X, Petkov N (eds) Computer Analysis of Images and Patterns: 13th International Conference, CAIP 2009, Münster, Germany, September 2–4, 2009. Proceedings, Springer, Berlin, pp 83–90

  31. Poli G, Saito JH, Mari JF, Zorzan MR (2008) Processing neocognitron of face recognition on high performance environment based on GPU with CUDA architecture. In: Computer architecture and high performance computing, 2008. SBAC-PAD ’08. 20th international symposium on, pp 81–88

  32. Huqqani AA, Schikuta E, Ye S, Chen P (2013) Multicore and GPU parallelization of neural networks for face recognition. Proc Comput Sci 18:349–358

    Article  Google Scholar 

  33. Goodall T, Gibson S, Smith MC (2012) Parallelizing principal component analysis for robust facial recognition using CUDA. In: Application accelerators in high performance computing (SAAHPC), 2012 symposium on, pp 121–124

  34. Woo Y, Yi C, Yi Y (2013) Fast PCA-based face recognition on GPUs. In: Acoustics, speech and signal processing (ICASSP). IEEE International Conference on 2013, pp 2659–2663

  35. Refaie MN, Salman AA, Ahmad I (2012) Hybrid parallel approach based on wavelet transformation and principle component analysis for solving face recognition problem. In: ICT and Knowledge Engineering (ICT & Knowledge Engineering), 2011 9th International Conference on, pp 72–79

  36. Dixon W, Powers N, Song Y, Soyata T (2015) Theoretical foundation and GPU implementation of face recognition In: Soyata T(ed) Enabling real-time mobile cloud computing through emerging technologies. IGI Global, Hershey, pp 322–341

  37. Yi S, Yoon I, Oh C, Yi Y (2014) Real-time integrated face detection and recognition on embedded GPGPUs. In: Embedded systems for real-time multimedia (ESTIMedia), 2014 IEEE 12th symposium on, pp 98–107

  38. Dwith CYN, Rathna GN (2012) Parallel implementation of LBP based face recognition on GPU using OpenCL. In: Parallel and Distributed Computing, Applications and Technologies (PDCAT), 2012 13th International Conference on 2012, pp 755–760

  39. Rahimzadeh Arashloo S, Kittler J (2014) Fast pose invariant face recognition using super coupled multiresolution Markov Random fields on a GPU. Pattern Recognit Lett 48:49–59

    Article  Google Scholar 

  40. Herout A, Jošth R, Juránek R, Havel J, Hradiš M, Zemčík P (2011) Real-time object detection on CUDA. J Real-Time Image Process 6:159–170

    Article  Google Scholar 

  41. Obukhov A (2011) Haar classifiers for object detection with cuda. In: GPU computing gems emerald edition (Applications of GPU Computing Series), vol 1. Elsevier, pp 517–544

  42. Tek SC, Gökmen M (2012) GPU accelerated real-time object detection on high resolution videos using modified census transform. In: VISAPP (1), pp 685–688

  43. Harvey JP (2009) GPU acceleration of object classification algorithms using NVIDIA CUDA

  44. Li Z, Nevatia R (2008) Efficient scan-window based object detection using GPGPU. In: Computer vision and pattern recognition workshops, CVPRW ’08. IEEE Computer Society Conference on 2008, pp 1–7

  45. Bordallo López M, Nykänen H, Hannuksela J, Silvén O, Vehviläinen M (2011) Accelerating image recognition on mobile devices using GPGPU, pp 78720R-78720R-10

  46. Uetz R, Behnke S (2009) Large-scale object recognition with CUDA-accelerated hierarchical neural networks. In: Intelligent computing and intelligent systems, 2009. ICIS 2009. IEEE International Conference on 2009, pp 536–541

  47. Ciresan DC, Meier U, Masci J, Maria Gambardella L, Schmidhuber J (2011) Flexible, high performance convolutional neural networks for image classification. In: IJCAI Proceedings-international Joint Conference on Artificial Intelligence, pp 1237

  48. Kim J, Park E, Cui X, Kim H, Gruver WA (2009) A fast feature extraction in object recognition using parallel processing on CPU and GPU. In: Systems, man and cybernetics, SMC 2009. IEEE International Conference on 2009, pp 3842–3847

  49. Cornelis N, Gool LV (2008) Fast scale invariant feature detection and matching on programmable graphics hardware. In: Computer vision and pattern recognition workshops, CVPRW ’08. IEEE Computer Society Conference on 2008, pp 1–8

  50. Leuh K (2011) Cm: high-level programming environment for efficient utilization of intel processor-GPU. Internal Report, Intel

Download references

Acknowledgments

This research project was supported by a grant from the “Research Center of the Female Scientific and Medical Colleges”, Deanship of Scientific Research, King Saud University.

Author information

Authors and Affiliations

  1. Department of Information Technology, King Saud University, Riyadh, Saudi Arabia

    Rabia Jafri

Authors
  1. Rabia Jafri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rabia Jafri.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jafri, R. A GPU-accelerated real-time contextual awareness application for the visually impaired on Google’s project Tango device. J Supercomput 73, 887–899 (2017). https://doi.org/10.1007/s11227-016-1891-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-016-1891-8

Keywords

Search

Navigation