Skip to main content

Human-directed coordinated control of an assistive mobile manipulator


This paper considers the control of a mobile manipulator with the end-effector commanded by a human user. The motivation is to empower mobility impaired individuals to tackle certain activities of daily living. The prototype robotic system consists of a redundant articulated arm mounted on a mobile base. The human user specifies the desired motion of certain degrees of freedom, such as the translation or rotation of the arm end-effector. An optimization-based control algorithm determines the full arm and base motion to minimize deviation from the human intent while satisfying certain equality and inequality constraints. These constraints may involve physical requirements of the system, e.g., collision avoidance and joint limits, and human user consideration, such as intuitiveness of the combined arm and base motion. This constraint optimization problem is strictly convex and may be efficiently solved as a quadratic problem. We demonstrate the effectiveness of the algorithm in simulation and experimentation, using a dual-arm Baxter robot mounted on a wheelchair as a prototype testbed. Human user commands the end-effector of one of the arms using a 3-degree-of-freedom interface, either a sip-puff or a joystick type of device. We demonstrate effective object retrieval and placement in an environment with obstacles.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15


  • Baxter: Redefining robotics and manufacturing. (2014). Accessed 14 March 2014

  • Cai, H., Mostofi, Y,: To ask or not to ask: a foundation for the optimization of human-robot collaborations. In: Proceedings of 2015 American Control Conference, Chicago, IL, Jul 1–3 (2015)

  • Chen, T.L., Ciocarlie, M., Cousins, S., Grice, P., Hawkins, K., Hsiao, K., Kemp, C.C., King, C.H., Lazewatsky, D.A., Leeper, A., Nguyen, H., Paepcke, A., Pantofaru, C., Smart, W.D., Takayama, L.: Robots for humanity: a case study in assistive mobile manipulation. In: IEEE Robotics and Automation Magazine, Special issue on Assistive Robotics, vol 20(1) (2013)

  • Cipriani, C., Zaccone, F., Micera, S., Chiara Carrozza, M.: On the shared control of an EMG-controlled prosthetic hand: analysis of user prosthesis. IEEE Trans. Robot. 24(1), 170–184 (2008)

    Article  Google Scholar 

  • Cunningham, A., Keddy-Hector, W., Sinha, U., Whalen, D., Kruse, D., Braasch, J., Wen, J.T., Jamster.: A mobile dual-arm assistive robot with Jamboxx control. In: Conference on Automation Science and Engineering (CASE), pp. 509–514, Aug (2014)

  • Dietrich, A., Wimboeck, T., Albu-Schäffer, A., Hirzinger, G.: Integration of reactive, torque-based self-collision avoidance into a task hierarchy. IEEE Trans. Robot. 28(6), 1278–1293 (2012)

    Article  Google Scholar 

  • Dietrich, A., Wimboeck, T., Albu-Schäffer, A., Hirzinger, G.: Reactive whole-body control: dynamic mobile manipulation using a large number of actuated degrees of freedom. IEEE Robot. Autom. Mag. Special Issue Mobile Manipul. 19(2), 20–33 (2012)

    Article  Google Scholar 

  • Dragan, A.D., Srinivasa, S.S.: A policy-blending formalism for shared control. Int. J. Robot. Res. 32(7), 790–805 (2013)

    Article  Google Scholar 

  • Flacco, F., Luca, A.D.: Discrete-time redundancy resolution at the velocity level with acceleration/torque optimization properties. Robot. Autonom. Syst. (2015) (accepted and in press)

  • Franchi, A., Secchi, C., Ryll, M., Bulthoff, H.H., Giordano, P.R.: Shared control: balancing autonomy and human assistance with a group of quadrotor UAVs. IEEE Robot. Autom. Mag. 19(3), 57–68 (2012)

    Article  Google Scholar 

  • Jiang, J., Astolfi, A.: Shared-control for the kinematic model of a rear-wheel drive car. In: Proceedings of 2015 American Control Conference, Chicago, IL, Jul 1–3 (2015)

  • Kim, H., Biggs, S.J., Schloerb, D.W., Carmena, J.M., Lebedev, M.A., Nicolelis, M.A.L., Srinivasan, M.A.: Continuous shared control for stabilizing reaching and grasping with brain-machine interfaces. IEEE Trans. Biomed. Eng. 53(6), 1164–1173 (2006)

    Article  Google Scholar 

  • Kruse, D., Radke, R., Wen, J.: Collaborative human-robot manipulation of highly deformable materials. In: Proceedings of the IEEE Conference on Robotics and Automation (ICRA), Seattle, WA, May 26–30 (2015)

  • Lacey, G., MacNamara, S.: Context-aware shared control of a robot mobility aid for the elderly blind. Int. J. Robot. Res. 19(11), 1054–1065 (2000)

    Article  Google Scholar 

  • Leeper, A., Hsiao, K., Ciocarlie, M., Sucan, I., Salisbury, K.: Methods for collision-free arm teleoperation in clutter using constraints from 3d sensor data. In: IEEE-RAS International Conference on Humanoid Robots, Oct 15–17 (2013)

  • Luca, A.D., Oriolo, G., Giordano, P.R.: Image-based visual servoing schemes for nonholonomic mobile manipulators. Robotica 25, 131–145 (2007)

    Google Scholar 

  • Madigan, E.A., Newman, W.S.: What do users want from “smart wheelchairs”. In: NI 2012: Proceedings of the 11th International Congress on Nursing Informatics, vol. 2012. American Medical Informatics Association (2012)

  • Nudehi, S.S., Mukherjee, R., Ghodoussi, M.: A shared-control approach to haptic interface design for minimally invasive telesurgical training. IEEE Trans. Control Syst. Technol. 13(4), 588–592 (2005)

    Article  Google Scholar 

  • Ogren, P., Smith, C., Karayiannidis, Y., Kragic, D.: A multi objective control approach to online dual arm manipulation. In IFAC Symposium on Robot Control, SyRoCo, Dubrovnik, Croatia (2012)

    Google Scholar 

  • Pineau, J., Atrash, A.: SmartWheeler: a robotic wheelchair test-bed for investigating new models of human-robot interaction. In: AAAI Spring Symposium: Multidisciplinary Collaboration for Socially Assistive Robotics, pp. 59–64 (2007)

  • Storms, J.G., Vozar, S., Tilbury, D.M.: Predicting human performance during teleoperation. In: ACM/IEEE International Conference on Human-Robot Interaction, Bielefeld University, Germany, Mar 3–6 (2014)

  • Storms, J.G., Tilbury, D.M.: Blending of human and obstacle avoidance control for a high speed mobile robot. In: American Control Conference (ACC), Portland, Oregon, USA, Jun 4–6 (2014)

  • Stückler, J., Behnke, S.: Following human guidance to cooperatively carry a large object. In: 11th IEEE-RAS International Conference on Humanoid Robots, Bled, Slovenia, October (2011)

  • Tan, J., Xi, N., Wang, Y.: Integrated task planning and control for mobile manipulators. Int. J. Robot. Res. 22(5), 337–354 (2003)

    Article  Google Scholar 

  • Tanner, H.G., Loizou, S.G., Kyriakopoulos, K.J.: Nonholonomic navigation and control of cooperating mobile manipulators. IEEE Trans. Robot. Autom. 19(1), 53–64 (2003)

    Article  Google Scholar 

  • Venator, E., Lee, G.S., Newman, W.: Hardware and software architecture of ABBY: An industrial mobile manipulator. In: 2013 IEEE International Conference on Automation Science and Engineering (CASE), pp. 324–329. IEEE (2013)

  • Wang, H., Liu, X.P.: Adaptive shared control for a novel mobile assistive robot. IEEE/ASME Trans. Mechatron. 19(6), 1725–1736 (2014)

    Article  Google Scholar 

  • Wang, Z., Mülling, K., Deisenroth, M.P., Amor, H.B., Vogt, D., Schölkopf, B., Peters, J.: Probabilistic movement modeling for intention inference in human–robot interaction. Int. J. Robot. Res. 32(7), 841–858 (2013)

    Article  Google Scholar 

  • Wason Technology. Robot Raconteur. (2014). Accessed 28 July 2015

  • White, G.D., Bhatt, R.M., Krovi, V.N.: Dynamic redundancy resolution in a nonholonomic wheeled mobile manipulator. Robotica 25(3), 147–156 (2007)

    Google Scholar 

  • White, G.D., Bhatt, R.M., Tang, C.P., Krovi, V.N.: Experimental evaluation of dynamic redundancy resolution in a nonholonomic wheeled mobile manipulator. IEEE/ASME Trans. Mechatron. 14(3), 349–357 (2009)

    Article  Google Scholar 

Download references


The authors would like to thank Andrew Cunningham and Jingyu Su for the wheelchair control system implementation, Dan Kruse for the Robot Raconteur Baxter Bridge, and John Wason for the development and sharing of Robot Raconteur. The enthusiasm, encouragement and feedback from David Whalen is a continuing source of inspiration. This work is supported primarily by the Center for Automation Technologies and Systems (CATS) under a block grant from the New York State Empire State Development Division of Science, Technology and Innovation (NYSTAR), and in part by the National Science Foundation Smart Lighting ERC under NSF Cooperative Agreement No. EEC-0812056.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Lu Lu.

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, L., Wen, J.T. Human-directed coordinated control of an assistive mobile manipulator. Int J Intell Robot Appl 1, 104–120 (2017).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: