Abstract
In this paper, we present an approach for navigating a robotic wheelchair that provides users with multiple levels of autonomy and navigation capabilities to fit their individual needs and preferences. We focus on three main aspects: (i) egocentric computer vision based motion control to provide a natural human-robot interface to wheelchair users with impaired hand usage; (ii) techniques that enable user to initiate autonomous navigation to a location, object or person without use of the hands; and (iii) a framework that learns to navigate the wheelchair according to its user’s, often subjective, criteria and preferences. These contributions are evaluated qualitatively and quantitatively in user studies with several subjects demonstrating their effectiveness. These studies have been conducted with healthy subjects, but they still indicate that clinical tests of the proposed technology can be initiated.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Data Availability
Data are not publicly available to preserve the subjects’ privacy.
Code Availability
Our code is not available because it requires a custom modification of a particular wheelchair, and thus is not of broad interest. It can be made available upon request.
References
Organization, W.H.: Assistive technology, fact sheet (2016)
Shields, M.: Use of wheelchairs and other mobility support devices. Health Rep. 15, 37—41 (2004)
Ceres, R., Pons, J.L., Calderon, L., Jimenez, A.R., Azevedo, L.: A robotic vehicle for disabled children. IEEE Engineering in Medicine and Biology magazine 24(6), 55–63 (2005)
Fehr, L., Langbein, W.E., Skaar, S.B.: Adequacy of power wheelchair control interfaces for persons with severe disabilities: a clinical survey. J. Rehabil. Res. Dev. 37(3), 353 (2000)
Simpson, R.C., LoPresti, E.F., Cooper, R.A.: How many people would benefit from a smart wheelchair?. J. Rehabil. Res. Dev. 45(1), 53 (2008)
Kutbi, M., Du, X., Chang, Y., Sun, B., Agadakos, N., Li, H., Hua, G., Mordohai, P.: Usability studies of an egocentric vision-based robotic wheelchair. ACM Transactions on Human-Robot Interaction (THRI) 10(1), 1–23 (2020)
Li, H., Kutbi, M., Li, X., Cai, C., Mordohai, P., Hua, G.: An egocentric computer vision based co-robot wheelchair. In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (2016)
Wang, M., Deng, W.: Deep face recognition: a survey. Neurocomputing 429, 215–244 (2021)
Chen, S. -H., Chen, Y. -L., Chiou, Y. -H., Tsai, J. -C., Kuo, T. -S.: Head-controlled device with M3S-based for people with disabilities. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), vol. 2, pp. 1587–1589 (2003)
Simpson, R.C.: Smart wheelchairs: A literature review. J. Rehabil. Res. Dev. 42(4), 423 (2005)
Carlson, T., Leeb, R., Chavarriaga, R., Millán, J.D.R.: The Birth of the Brain-Controlled Wheelchair. In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5444–5445 (2012)
Sisbot, E.A., Marin-Urias, L.F., Alami, R., Simeon, T.: A human aware mobile robot motion planner. IEEE Trans. Robot. 23(5), 874–883 (2007)
Chang, Y., Kutbi, M., Agadakos, N., Sun, B., Mordohai, P.: A shared autonomy approach for wheelchair navigation based on learned user preferences. In: Fifth International Workshop on Assistive Computer Vision and Robotics, pp. 1490–1499 (2017)
Cook, A.M., Polgar, J.M.: Assistive technologies: Principles and practice. Elsevier Health Sciences (2014)
Lipskin, R.: An evaluation program for powered wheelchair control systems. Bull. Prosth. Res 6, 121–219 (1970)
Nguyen, H., King, L., Knight, G.: Real-time head movement system and embedded linux implementation for the control of power wheelchairs. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), vol. 2 (2004)
Lund, M.E., Christiensen, H.V., Caltenco, H., Lontis, E.R., Bentsen, B., Struijk, L.N.A.: Inductive tongue control of powered wheelchairs. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (2010)
Mace, M., Abdullah-Al-Mamun, K., Vaidyanathan, R., Wang, S., Gupta, L.: Real-time implementation of a non-invasive tongue-based human-robot interface. In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (2010)
Lopes, A.C., Pires, G., Nunes, U.: Robchair: Experiments evaluating brain-computer interface to steer a semi-autonomous wheelchair. In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (2012)
Carlson, T., Millan, J.D.R.: Brain-controlled wheelchairs: a robotic architecture. IEEE Robot. Autom. Mag. 20, 65–73 (2013)
Boucher, P., Atrash, A., Kelouwani, S., Honoré, W., Nguyen, H., Villemure, J., Routhier, F., Cohen, P., Demers, L., Forget, R., Pineau, J.: Design and validation of an intelligent wheelchair towards a clinically-functional outcome. J. Neuroeng. Rehabil. 10(1), 1 (2013)
Fezari, M., Bousbia-Salah, M.: Speech and sensor in guiding an electric wheelchair. Autom. Control. Comput. Sci. 41(1), 39–43 (2007)
Nishimori, M., Saitoh, T., Konishi, R.: Voice Controlled Intelligent Wheelchair. In: SICE (2007)
Gandhi, V., Prasad, G., Coyle, D., Behera, L., McGinnity, T.M.: EEG-Based mobile robot control through an adaptive brain–robot interface. IEEE Transactions on Systems Man, and Cybernetics: Systems, 44(9), 1278–1285 (2014)
Liu, Y., Li, Z., Zhang, T., Zhao, S.: Brain-Robot interface-based navigation control of a mobile robot in corridor environments. IEEE Transactions on Systems, Man, and Cybernetics Systems (2018)
Quintero, C.P., Ramirez, O., Jagersand, M.: Vibi: Assistive vision-based interface for robot manipulation. In: Proc. of the IEEE International Conference on Robotics and Automation (2015)
Pasteau, F., Krupa, A., Babel, M.: Vision-based assistance for wheelchair navigation along corridors. In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (2014)
Purwanto, D., Mardiyanto, R., Arai, K.: Electric wheelchair control with gaze direction and eye blinking. Artificial Life and Robotics 14(3), 397–400 (2009)
Gray, J., Jia, P., Hu, H.H., Lu, T., Yuan, K.: Head gesture recognition for hands-free control of an intelligent wheelchair. Industrial Robot: An International Journal 34(1), 60–68 (2007)
Xu, R., Hartshorn, R., Huard, R., Irwin, J., Johnson, K., Nelson, G., Campbell, J., Ay, S.A., Taylor, M.E.: Towards a semi-autonomous wheelchair for users with ALS. In: Workshop on Autonomous Mobile Service Robots (At IJCAI) (2016)
Halawani, A., Ur Réhman, S., Li, H., Anani, A.: Active vision for controlling an electric wheelchair. Intell. Serv. Robot. 5(2), 89–98 (2012)
Kim, B.K., Tanaka, H., Sumi, Y.: Robotic wheelchair using a high accuracy visual marker lentibar and its application to door crossing navigation. In: Proc. of the IEEE International Conference on Robotics and Automation (2015)
Zolotas, M., Elsdon, J., Demiris, Y.: Head-mounted augmented reality for explainable robotic wheelchair assistance. In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1823–1829 (2018)
Chacón-Quesada, R., Demiris, Y.: Augmented reality control of smart wheelchair using eye-gaze–enabled selection of affordances. In: Robots for Assisted Living Workshop (2018)
Gulati, S., Jhurani, C., Kuipers, B., Longoria, R.: A framework for planning comfortable and customizable motion of an assistive mobile robot. In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4253–4260 (2009)
Shiomi, M., Iio, T., Kamei, K., Sharma, C., Hagita, N.: Effectiveness of social behaviors for autonomous wheelchair robot to support elderly people in Japan. PloS one, vol. 5, 10 (2015)
Parikh, S.P., Grassi, V., Kumar, V., Okamoto, J.: Incorporating user inputs in motion planning for a smart wheelchair. In: Proc. of the IEEE International Conference on Robotics and Automation, vol. 2, pp. 2043–2048 (2004)
Parikh, S.P., Grassi, V., Kumar, V., Okamoto, J.J.: Usability study of a control framework for an intelligent wheelchair. In: Proc. of the IEEE International Conference on Robotics and Automation, pp. 4745–4750 (2005)
Zeng, Q., Teo, C.L., Rebsamen, B., Burdet, E.: A collaborative wheelchair system. IEEE Trans. Neural Syst. Rehabil. Eng. 16(2), 161–170 (2008)
Galindo, C., Gonzalez, J., Fernandez-Madrigal, J.-A.: Control architecture for human–robot integration: application to a robotic wheelchair. IEEE Transactions on Systems, Man, and Cybernetics Part B (Cybernetics) 36(5), 1053–1067 (2006)
Urdiales, C., Peula, J.M., Fdez-Carmona, M., Barrué, C., Pérez, E. J., Sánchez-Tato, I., del Toro, J.C., Galluppi, F., Cortés, U., Annichiaricco, R., Caltagirone, C., Sandoval, F.: A new multi-criteria optimization strategy for shared control in wheelchair assisted navigation. Auton. Robot. 30(2), 179–197 (2011)
Li, Q., Chen, W., Wang, J.: Dynamic shared control for human-wheelchair cooperation. In: Proc. of the IEEE International Conference on Robotics and Automation, pp. 4278–4283 (2011)
Carlson, T., Demiris, Y.: Collaborative control for a robotic wheelchair: evaluation of performance, attention, and workload. IEEE Transactions on Systems, Man, and Cybernetics Part B (Cybernetics) 42(3), 876–888 (2012)
Goil, A., Derry, M., Argall, B.D.: Using machine learning to blend human and robot controls for assisted wheelchair navigation. In: IEEE International Conference on Rehabilitation Robotics (ICORR) (2013)
Kruse, T., Pandey, A.K., Alami, R., Kirsch, A.: Human-aware robot navigation: a survey. Robot. Auton. Syst. 61(12), 1726–1743 (2013)
Kirby, R., Simmons, R., Forlizzi, J.: Companion: a Constraint-Optimizing Method for Person-Acceptable Navigation. In: RO-MAN, pp. 607–612 (2009)
Cosgun, A., Sisbot, A., Christensen, H.I.: Anticipatory robot path planning in human environments. In: The 25Th IEEE International Symposium on Robot and Human Interactive Communication, pp. 562–569 (2016)
Helbing, D., Molnar, P.: Social force model for pedestrian dynamics. Phys. Rev. E 51(5), 4282 (1995)
Morales, Y., Miyashita, T., Hagita, N.: Social robotic wheelchair centered on passenger and pedestrian comfort. Robot. Auton. Syst. 87, 355–362 (2017)
Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: Ros: An open-source robot operating system. In: ICRA Workshop on Open Source Software (2009)
Labbe, M., Michaud, F.: Online global loop closure detection for large-scale multi-session graph-based SLAM. In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2661–2666 (2014)
Marder-Eppstein, E., Berger, E., Foote, T., Gerkey, B., Konolige, K.: The office marathon: robust navigation in an indoor office environment. In: International Conference on Robotics and Automation (2010)
Marchand, E., Spindler, F., Chaumette, F.: Visp for visual servoing: a generic software platform with a wide class of robot control skills. IEEE Robotics & Automation Magazine 12(4), 40–52 (2005)
Nebehay, G., Pflugfelder, R.: Clustering of static-adaptive correspondences for deformable object tracking. In: CVPR (2015)
Walker, W., Lamere, P., Kwok, P., Raj, B., Singh, R., Gouvea, E., Wolf, P., Woelfel, J.: Sphinx-4: A flexible open source framework for speech recognition (2004)
Suzuki, S., Abe, K.: Topological structural analysis of digitized binary images by border following, Computer Vision. Graphics, and Image Processing 30(1), 32–46 (1985)
Fischler, M.A., Bolles, R.C.: Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography, Commun ACM (1981)
Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: ECCV, pp. 430–443, Springer (2006)
Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: Orb: an efficient alternative to sift or surf. In: ICCV (2011)
Li, H., Lin, Z., Shen, X., Brandt, J., Hua, G.: A convolutional neural network cascade for face detection. In: CVPR (2015)
Li, H., Hua, G., Lin, Z., Brandt, J., Yang, J.: Probabilistic elastic matching for pose variant face verification. In: CVPR (2013)
Nebehay, G., Pflugfelder, R.: Consensus-based matching and tracking of keypoints for object tracking. In: IEEE Winter Conference on Applications of Computer Vision (WACV) (2014)
Linder, T., Arras, K.O.: People detection, tracking and visualization using ros on a mobile service robot. In: Robot Operating System (ROS), pp. 187–213, Springer (2016)
Linder, T., Breuers, S., Leibe, B., Arras, K.O.: On multi-modal people tracking from mobile platforms in very crowded and dynamic environments. In: Proc. of the IEEE International Conference on Robotics and Automation, pp. 5512–5519 (2016)
Jafari, O.H., Mitzel, D., Leibe, B.: Real-Time Rgb-D based people detection and tracking for mobile robots and head-worn cameras. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 5636–5643, IEEE (2014)
Ess, A., Schindler, K., Leibe, B., Van Gool, L.: Object detection and tracking for autonomous navigation in dynamic environments. The International Journal of Robotics Research 29(14), 1707–1725 (2010)
LaValle, S.M.: Planning Algorithms. Cambridge University Press, Cambridge (2006)
Kuderer, M., Sprunk, C., Kretzschmar, H., Burgard, W.: Online generation of homotopically distinct navigation paths. In: Proc. of the IEEE International Conference on Robotics and Automation, pp. 6462–6467 (2014)
Bhattacharya, S., Likhachev, M., Kumar, V.: Topological constraints in search-based robot path planning. Auton. Robot. 33(3), 273–290 (2012)
Mandel, C., Frese, U.: Comparison of wheelchair user interfaces for the paralysed: head-joystick vs. verbal path selection from an offered route-set. In: EMCR (2007)
DIJKSTRA, E.: A note on two problems in connexion with graphs. Numer. Math. 1, 269–271 (1959)
Choset, H., Burdick, J.: Sensor-based exploration: The hierarchical generalized voronoi graph. The International Journal of Robotics Research 19(2), 96–125 (2000)
Lau, B., Sprunk, C., Burgard, W.: Efficient grid-based spatial representations for robot navigation in dynamic environments. Robot. Auton. Syst. 61(10), 1116–1130 (2013)
Herbrich, R., Graepel, T., Obermayer, K.: Support vector learning for ordinal regression. International Conference on Neural Networks (1999)
Park, J.J., Kuipers, B.: A smooth control law for graceful motion of differential wheeled mobile robots in 2D environment. In: Proc. of the IEEE International Conference on Robotics and Automation, pp. 4896–4902 (2011)
Jain, A., Das, D., Gupta, J.K., Saxena, A.: Planit: A crowdsourcing approach for learning to plan paths from large scale preference feedback. In: Proc. of the IEEE International Conference on Robotics and Automation, pp. 877–884 (2015)
Kutbi, M., Chang, Y., Sun, B., Mordohai, P.: Learning to navigate robotic wheelchairs from demonstration: is training in simulation viable?. In: Seventh International Workshop on Assistive Computer Vision and Robotics (2019)
Acknowledgements
Research reported in this publication was primarily supported by the National Institute of Nursing Research of the National Institutes of Health under Award Number R01NR015371 and also partially supported by the National Science Foundation under Awards IIS-1527294 and IIS-1637761. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the National Science Foundation. The main contributions of all authors took place while they were with Stevens Institute of Technology. Mohammed Kutbi prepared this manuscript while at the Saudi Electronic University.
Author information
Authors and Affiliations
Contributions
Mohammed Kutbi prepared the manuscript and contributed to conception, design and implementation, Haoxiang Li contributed to conception, design and implementation, Yizhe Chang contributed to conception, design and implementation, Bo sun contributed to design and implementation, Xin Li contributed to implementation, Changjiang Cai contributed to implementation, Nikolaos Agadagos contributed to implementation, Gang Hua contributed to conception, and Philippos Mordohai contributed to conception and manuscript preparation.
Corresponding author
Ethics declarations
Ethics approval
All studies involving non-researchers as participants received IRB approval at Stevens Institute of Technology.
Consent to participate
Informed consent was obtained from all individual participants included in the study.
Consent for Publication
No images of subjects are shown in the paper. All individuals depicted are authors of the manuscript.
Conflict of Interests
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Kutbi, M., Li, H., Chang, Y. et al. Egocentric Computer Vision for Hands-Free Robotic Wheelchair Navigation. J Intell Robot Syst 107, 10 (2023). https://doi.org/10.1007/s10846-023-01807-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-023-01807-4