This paper considers the control of a mobile manipulator with the end-effector commanded by a human user. The motivation is to empower mobility impaired individuals to tackle certain activities of daily living. The prototype robotic system consists of a redundant articulated arm mounted on a mobile base. The human user specifies the desired motion of certain degrees of freedom, such as the translation or rotation of the arm end-effector. An optimization-based control algorithm determines the full arm and base motion to minimize deviation from the human intent while satisfying certain equality and inequality constraints. These constraints may involve physical requirements of the system, e.g., collision avoidance and joint limits, and human user consideration, such as intuitiveness of the combined arm and base motion. This constraint optimization problem is strictly convex and may be efficiently solved as a quadratic problem. We demonstrate the effectiveness of the algorithm in simulation and experimentation, using a dual-arm Baxter robot mounted on a wheelchair as a prototype testbed. Human user commands the end-effector of one of the arms using a 3-degree-of-freedom interface, either a sip-puff or a joystick type of device. We demonstrate effective object retrieval and placement in an environment with obstacles.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Baxter: Redefining robotics and manufacturing. http://www.rethinkrobotics.com/products/baxter (2014). Accessed 14 March 2014
Cai, H., Mostofi, Y,: To ask or not to ask: a foundation for the optimization of human-robot collaborations. In: Proceedings of 2015 American Control Conference, Chicago, IL, Jul 1–3 (2015)
Chen, T.L., Ciocarlie, M., Cousins, S., Grice, P., Hawkins, K., Hsiao, K., Kemp, C.C., King, C.H., Lazewatsky, D.A., Leeper, A., Nguyen, H., Paepcke, A., Pantofaru, C., Smart, W.D., Takayama, L.: Robots for humanity: a case study in assistive mobile manipulation. In: IEEE Robotics and Automation Magazine, Special issue on Assistive Robotics, vol 20(1) (2013)
Cipriani, C., Zaccone, F., Micera, S., Chiara Carrozza, M.: On the shared control of an EMG-controlled prosthetic hand: analysis of user prosthesis. IEEE Trans. Robot. 24(1), 170–184 (2008)
Cunningham, A., Keddy-Hector, W., Sinha, U., Whalen, D., Kruse, D., Braasch, J., Wen, J.T., Jamster.: A mobile dual-arm assistive robot with Jamboxx control. In: Conference on Automation Science and Engineering (CASE), pp. 509–514, Aug (2014)
Dietrich, A., Wimboeck, T., Albu-Schäffer, A., Hirzinger, G.: Integration of reactive, torque-based self-collision avoidance into a task hierarchy. IEEE Trans. Robot. 28(6), 1278–1293 (2012)
Dietrich, A., Wimboeck, T., Albu-Schäffer, A., Hirzinger, G.: Reactive whole-body control: dynamic mobile manipulation using a large number of actuated degrees of freedom. IEEE Robot. Autom. Mag. Special Issue Mobile Manipul. 19(2), 20–33 (2012)
Dragan, A.D., Srinivasa, S.S.: A policy-blending formalism for shared control. Int. J. Robot. Res. 32(7), 790–805 (2013)
Flacco, F., Luca, A.D.: Discrete-time redundancy resolution at the velocity level with acceleration/torque optimization properties. Robot. Autonom. Syst. (2015) (accepted and in press)
Franchi, A., Secchi, C., Ryll, M., Bulthoff, H.H., Giordano, P.R.: Shared control: balancing autonomy and human assistance with a group of quadrotor UAVs. IEEE Robot. Autom. Mag. 19(3), 57–68 (2012)
Jiang, J., Astolfi, A.: Shared-control for the kinematic model of a rear-wheel drive car. In: Proceedings of 2015 American Control Conference, Chicago, IL, Jul 1–3 (2015)
Kim, H., Biggs, S.J., Schloerb, D.W., Carmena, J.M., Lebedev, M.A., Nicolelis, M.A.L., Srinivasan, M.A.: Continuous shared control for stabilizing reaching and grasping with brain-machine interfaces. IEEE Trans. Biomed. Eng. 53(6), 1164–1173 (2006)
Kruse, D., Radke, R., Wen, J.: Collaborative human-robot manipulation of highly deformable materials. In: Proceedings of the IEEE Conference on Robotics and Automation (ICRA), Seattle, WA, May 26–30 (2015)
Lacey, G., MacNamara, S.: Context-aware shared control of a robot mobility aid for the elderly blind. Int. J. Robot. Res. 19(11), 1054–1065 (2000)
Leeper, A., Hsiao, K., Ciocarlie, M., Sucan, I., Salisbury, K.: Methods for collision-free arm teleoperation in clutter using constraints from 3d sensor data. In: IEEE-RAS International Conference on Humanoid Robots, Oct 15–17 (2013)
Luca, A.D., Oriolo, G., Giordano, P.R.: Image-based visual servoing schemes for nonholonomic mobile manipulators. Robotica 25, 131–145 (2007)
Madigan, E.A., Newman, W.S.: What do users want from “smart wheelchairs”. In: NI 2012: Proceedings of the 11th International Congress on Nursing Informatics, vol. 2012. American Medical Informatics Association (2012)
Nudehi, S.S., Mukherjee, R., Ghodoussi, M.: A shared-control approach to haptic interface design for minimally invasive telesurgical training. IEEE Trans. Control Syst. Technol. 13(4), 588–592 (2005)
Ogren, P., Smith, C., Karayiannidis, Y., Kragic, D.: A multi objective control approach to online dual arm manipulation. In IFAC Symposium on Robot Control, SyRoCo, Dubrovnik, Croatia (2012)
Pineau, J., Atrash, A.: SmartWheeler: a robotic wheelchair test-bed for investigating new models of human-robot interaction. In: AAAI Spring Symposium: Multidisciplinary Collaboration for Socially Assistive Robotics, pp. 59–64 (2007)
Storms, J.G., Vozar, S., Tilbury, D.M.: Predicting human performance during teleoperation. In: ACM/IEEE International Conference on Human-Robot Interaction, Bielefeld University, Germany, Mar 3–6 (2014)
Storms, J.G., Tilbury, D.M.: Blending of human and obstacle avoidance control for a high speed mobile robot. In: American Control Conference (ACC), Portland, Oregon, USA, Jun 4–6 (2014)
Stückler, J., Behnke, S.: Following human guidance to cooperatively carry a large object. In: 11th IEEE-RAS International Conference on Humanoid Robots, Bled, Slovenia, October (2011)
Tan, J., Xi, N., Wang, Y.: Integrated task planning and control for mobile manipulators. Int. J. Robot. Res. 22(5), 337–354 (2003)
Tanner, H.G., Loizou, S.G., Kyriakopoulos, K.J.: Nonholonomic navigation and control of cooperating mobile manipulators. IEEE Trans. Robot. Autom. 19(1), 53–64 (2003)
Venator, E., Lee, G.S., Newman, W.: Hardware and software architecture of ABBY: An industrial mobile manipulator. In: 2013 IEEE International Conference on Automation Science and Engineering (CASE), pp. 324–329. IEEE (2013)
Wang, H., Liu, X.P.: Adaptive shared control for a novel mobile assistive robot. IEEE/ASME Trans. Mechatron. 19(6), 1725–1736 (2014)
Wang, Z., Mülling, K., Deisenroth, M.P., Amor, H.B., Vogt, D., Schölkopf, B., Peters, J.: Probabilistic movement modeling for intention inference in human–robot interaction. Int. J. Robot. Res. 32(7), 841–858 (2013)
Wason Technology. Robot Raconteur. https://robotraconteur.com (2014). Accessed 28 July 2015
White, G.D., Bhatt, R.M., Krovi, V.N.: Dynamic redundancy resolution in a nonholonomic wheeled mobile manipulator. Robotica 25(3), 147–156 (2007)
White, G.D., Bhatt, R.M., Tang, C.P., Krovi, V.N.: Experimental evaluation of dynamic redundancy resolution in a nonholonomic wheeled mobile manipulator. IEEE/ASME Trans. Mechatron. 14(3), 349–357 (2009)
The authors would like to thank Andrew Cunningham and Jingyu Su for the wheelchair control system implementation, Dan Kruse for the Robot Raconteur Baxter Bridge, and John Wason for the development and sharing of Robot Raconteur. The enthusiasm, encouragement and feedback from David Whalen is a continuing source of inspiration. This work is supported primarily by the Center for Automation Technologies and Systems (CATS) under a block grant from the New York State Empire State Development Division of Science, Technology and Innovation (NYSTAR), and in part by the National Science Foundation Smart Lighting ERC under NSF Cooperative Agreement No. EEC-0812056.
About this article
Cite this article
Lu, L., Wen, J.T. Human-directed coordinated control of an assistive mobile manipulator. Int J Intell Robot Appl 1, 104–120 (2017). https://doi.org/10.1007/s41315-016-0005-3