Abstract
Robotic manipulation systems suffer from two main problems in unstructured human environments: uncertainty and clutter. We introduce a planning framework addressing these two issues. The framework plans rearrangement of clutter using nonprehensile actions, such as pushing. Pushing actions are also used to manipulate object pose uncertainty. The framework uses an action library that is derived analytically from the mechanics of pushing and is provably conservative. The framework reduces the problem to one of combinatorial search, and demonstrates planning times on the order of seconds. With the extra functionality, our planner succeeds where traditional grasp planners fail, and works under high uncertainty by utilizing the funneling effect of pushing. We demonstrate our results with experiments in simulation and on HERB, a robotic platform developed at the Personal Robotics Lab at Carnegie Mellon University.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
Note that the NGR has a 3D volume in space. In Fig. 5 it is shown as a 2D region for clarity of visualization.
References
Agarwal, P., Latombe, C., Motwani, R., & Raghavan, P. (1997). Nonholonomic path planning for pushing a disk among obstacles. In IEEE ICRA.
Akella, S., & Mason, M. T. (1998). Posing polygonal objects in the plane by pushing. The International Journal of Robotics Research, 17(1), 70–88.
BenShahar, O., & Rivlin, E. (1998a). Practical pushing planning for rearrangement tasks. IEEE Transactions on Robotics and Automation, 14, 549–565.
BenShahar, O., & Rivlin, E. (1998b). To push or not to push: on the rearrangement of movable objects by a mobile robot. IEEE Transactions on Systems, Man and Cybernetics. Part B. Cybernetics, 28(5), 667–679.
Berenson, D., Srinivasa, S. S., Ferguson, D., & Kuffner, J. J. (2009a). Manipulation planning on constraint manifolds. In IEEE ICRA.
Berenson, D., Srinivasa, S. S., & Kuffner, J. J. (2009b). Addressing pose uncertainty in manipulation planning using task space regions. In IEEE/RSJ international conference on intelligent robots and systems.
Berretty, R. P., Goldberg, K. Y., Overmars, M. H., & van der Stappen, A. F. (2001). Orienting parts by insideout pulling. In IEEE international conference on robotics and automation.
Brost, R. C. (1988). Automatic grasp planning in the presence of uncertainty. The International Journal of Robotics Research, 7(1).
Chang, L. Y., Srinivasa, S., & Pollard, N. (2010). Planning pregrasp manipulation for transport tasks. In IEEE ICRA.
Chartrand, G. (1985). Introductory graph theory (pp. 135–139). New York: Dover.
Chen, P., & Hwang, Y. (1991). Practical path planning among movable obstacles. In IEEE international conference on robotics and automation.
Diankov, R., & Kuffner, J. (2008). OpenRAVE: a planning architecture for autonomous robotics (Tech. Rep. CMURITR0834). Robotics Institute.
Diankov, R., Srinivasa, S., Ferguson, D., & Kuffner, J. (2008). Manipulation planning with caging grasps. In Humanoids.
Fikes, R. E., & Nilsson, N. J. (1971). Strips: a new approach to the application of theorem proving to problem solving. Artificial Intelligence, 2(3–4), 189–208.
Goyal, S., Ruina, A., & Papadopoulos, J. (1991). Planar sliding with dry friction, part 1: limit surface and moment function. Wear, 143, 307–330.
Hauser, K., & NgThowHing, V. (2011). Randomized multimodal motion planning for a humanoid robot manipulation task. The International Journal of Robotics Research, 30(6), 678–698.
Howe, R. D., & Cutkosky, M. R. (1996). Practical forcemotion models for sliding manipulation. The International Journal of Robotics Research, 15(6), 557–572.
Kaelbling, L. P., & LozanoPerez, T. (2011). Hierarchical planning in the now. In IEEE international conference on robotics and automation.
Kappler, D., Chang, L. Y., Pollard, N. S., Asfour, T., & Dillmann, R. (2012). Templates for pregrasp sliding interactions. Robotics and Autonomous Systems, 60, 411–423.
Lavalle, S. M., & Kuffner, J. J. (2000). Rapidlyexploring random trees: progress and prospects. In Algorithmic and computational robotics: new directions.
Lynch, K. M. (1999a). Locally controllable manipulation by stable pushing. IEEE Transactions on Robotics and Automation, 15(2), 318–327.
Lynch, K. M. (1999b). Toppling manipulation. In IEEE/RSJ international conference on intelligent robots and systems (pp. 152–159).
Lynch, K. M., & Mason, M. T. (1996). Stable pushing: mechanics, controllability, and planning. The International Journal of Robotics Research, 15(6), 533–556.
Martinez, M., Collet, A., & Srinivasa, S. (2010). MOPED: a scalable and low latency object recognition and pose estimation system. In IEEE ICRA.
Mason, M. T. (1986). Mechanics and planning of manipulator pushing operations. The International Journal of Robotics Research, 5(3), 53–71.
Omrcen, D., Boge, C., Asfour, T., Ude, A., & Dillmann, R. (2009). Autonomous acquisition of pushing actions to support object grasping with a humanoid robot. In IEEERAS humanoids (pp. 277–283).
Overmars, M. H., Nieuwenhuisen, D., Nieuwenhuisen, D., Frank, A., & Overmars, H. (2006). An effective framework for path planning amidst movable obstacles. In International workshop on the algorithmic foundations of robotics.
Peshkin, M., & Sanderson, A. (1988). The motion of a pushed, sliding workpiece. IEEE Journal of Robotics and Automation, 4(1), 569–598.
Srinivasa, S. S., Ferguson, D., Helfrich, J. C., Berenson, D., Collet, A., Diankov, R., Gallagher, G., Hollinger, G., Kuffner, J., & Weghe, M. V. (2009). HERB: a home exploring robotic butler. Autonomous Robots.
Stilman, M., & Kuffner, J. J. (2006). Planning among movable obstacles with artificial constraints. In International workshop on the algorithmic foundations of robotics (pp. 1–20).
Stilman, M., Schamburek, J. U., Kuffner, J., & Asfour, T. (2007). Manipulation planning among movable obstacles. In IEEE international conference on robotics and automation,
van den Berg, J. P., Stilman, M., Kuffner, J., Lin, M. C., & Manocha, D. (2008). Path planning among movable obstacles: a probabilistically complete approach. In International workshop on the algorithmic foundations of robotics (pp. 599–614).
Wilfong, G. (1988). Motion planning in the presence of movable obstacles. In Proceedings of the fourth annual symposium on computational geometry (pp. 279–288).
Winograd, T. (1971). Procedures as a representation for data in a computer program for understanding natural language (Tech. Rep. MACTR84). MIT.
Acknowledgements
Special thanks to Matt Mason, Chris Atkeson, Tomas LozanoPerez, Charlie Kemp, Jim Rehg, Alberto Rodriguez and members of the Personal Robotics Lab at Carnegie Mellon University for insightful comments and discussions. This material is based upon work supported by DARPABAA1028, NSFIIS0916557, and NSFEEC0540865. Mehmet R. Dogar was partially supported by the Fulbright Science and Technology Fellowship.
Author information
Authors and Affiliations
Corresponding author
Appendix: Computing Conservative Capture Regions
Appendix: Computing Conservative Capture Regions
We compute the capture region for a pushgrasp using a pushing simulation. This simulation assumes that an object’s limit surface can be approximated with an ellipsoid in the forcemoment space as in Howe and Cutkosky (1996).
As described in Sect. 2.2.2 two parameters of this simulation affect the boundaries of the computed capture region: the pressure distribution between the object and the support surface, ρ; and the coefficient of friction between the robot finger and the object, μ _{ c }. These values are difficult to know and we assume that our robot does not know the exact values for any object.
When we run our simulation we generate capture regions that are conservative with respect to these parameters; i.e. capture regions that work for a range of reasonable values of μ _{ c } and ρ. For μ _{ c } such a reasonable region is given by the values between a very small value (a very slippery contact between the robot finger and the object), and a very high value (a high friction contact). The pressure distribution ρ can take any rotationally symmetric shape, but it is limited by the boundaries of the object making contact with the support surface.
One way to achieve a conservative capture region is by discretizing the values a parameter can take, running the simulation for each of the values, and then intersecting the resulting capture regions to find a capture region that works for all the values. But for certain object shapes we can do better.
For a cylindrical object the conservative capture region can be found simply by running the simulation for specific values of μ _{ c } and ρ. For μ _{ c } if we choose a very high value the computed capture region will also be valid for any lower value. For ρ if we choose a pressure distribution that is completely at the periphery of the cylinder (like a rim), the capture region will be valid for any other rotationally symmetric pressure distribution that is closer to the center of the cylinder. This is equivalent to saying that, as μ _{ c } gets smaller or as ρ gets concentrated around the center, the required pushing distance for the pushgrasp will decrease. In this section we prove this claim.
In Fig. 17 we present some of the force and velocity vectors that determine how a cylinder moves under quasistatic pushing. The following notation will be used:

\(\hat{\mathbf{n}}\) is the unit normal at the contact between the finger and the object.

f _{ L } and f _{ R } are the left and right edges of the friction cone. The friction cone is found by drawing the vectors that make the angle α=arctanμ _{ c } with \(\hat{\mathbf{n}}\).

f is the force applied to the object by the finger. The direction of f is bounded by the friction cone.

v and ω are the linear and angular velocities of the object at its center. v and ω can be found by computing the force and moment at the center of the object due to f, finding the corresponding point on the limit surface, and taking the normal to the limit surface. Our ellipsoid approximation of the limit surface dictates that v∥f (Howe and Cutkosky 1996).

m _{ L } and m _{ R } are the left and right edges of the motion cone. The edges of the motion cone are found by:

taking one of the edges of the friction cone, say the left edge;

computing the force and moment it creates at the object center;

using the limit surface to find the corresponding linear and angular velocity of the object, in response to this force and moment;

using the linear and angular velocity at the center of the object to find the velocity at the contact point. This gives the left edge of the motion cone; the right one is found by starting with the right edge of the friction cone.


\(\hat{\mathbf{v}}_{\mathbf{d}}\) is the unit vector pointing in the opposite direction of ω×r where r is the vector from the center of the object to the contact; \(\hat{\mathbf{v}}_{\mathbf{d}} = \frac{(\boldsymbol {\omega} \times\mathbf{r})}{\boldsymbol{\omega} \times\mathbf{r}}\).

v _{ p } is the velocity of the pusher/finger at the contact point. The voting theorem (Mason 1986) states that v _{ p } and the edges of the friction cone votes on the direction the object will rotate. For a cylinder the friction cone edges always fall on different sides of the center of the object, and v _{ p } alone dictates the rotation direction; v _{ p }.(ω×r)>0. In terms of \(\hat{\mathbf{v}}_{\mathbf{d}}\) this means \(\mathbf {v}_{\mathbf{p}}.\hat{\mathbf{v}}_{\mathbf{d}} < 0\).

v _{ c } is the velocity of the object at the contact point; v _{ c }=v+ω×r.

The graspline is the line at the fingertip orthogonal to the pushing direction. The pushgrasp continues until the object center passes the graspline.
During quasistatic pushing, the contact between the finger and the object can display three different modes: separation, sticking, or sliding. The case of separation is trivial, in which the finger moves away from the object resulting in no motion for the object. In the case of sticking contact the contact point on the finger and the contact point on the object moves together, i.e. v _{ p }=v _{ c }. This happens when f falls inside the friction cone, and correspondingly when v _{ c } falls inside the motion cone. In sliding contact the object slides on the finger as it is being pushed. In this case f aligns with the friction cone edge opposing the direction the object is sliding on the finger. Similarly v _{ c } aligns with the motion cone edge opposing the direction the object is sliding on the finger. v _{ p } is outside of the motion cone.
What follows is a series of lemmas and their proofs; which we then use to prove our main theorem.
Lemma 1
During sticking and sliding contact \(\mathbf{v}.\hat{\mathbf{n}} = \mathbf{v}_{\mathbf{p}}.\hat{\mathbf{n}}\).
Proof
During sticking contact v _{ p }=v _{ c }, which implies
We know that v _{ c }=v+ω×r. Then,
Since \((\boldsymbol{ \omega} \times\mathbf{r}).\hat{\mathbf{n}} = 0\),
During sliding contact, the relation between v _{ c } and v _{ p } is given by
where v _{ slide } is the velocity the object slides on the finger. Taking the projection of both sides along the contact normal gives
Sliding can only happen along the contact tangent. For a cylindrical object, this means \(\mathbf{v}_{\mathbf{slide}}.\hat{\mathbf{n}} = 0\). Then,
This is also illustrated in Fig. 18. The rest of the proof proceeds the same as the sticking contact case.
□
Lemma 2
During sticking contact, as we change ρ such that it is concentrated closer to the center of the cylinder, the magnitude of the angular velocity, ω, will get larger.
Proof
As ρ concentrates at the center, the limit surface ellipsoid gets a more flattened shape at the top and bottom. This implies that for the same force and moment applied to the object, the ratio ω/v will get larger (Howe and Cutkosky 1996).
We can express v as,
and using Lemma 1,
During sticking contact v _{ p }=v _{ c }, hence
Since \((\boldsymbol{ \omega} \times\mathbf{r}) = \boldsymbol{ \omega}\mathbf{r}\hat{\mathbf{v}}_{\mathbf{d}}\) we have
Rearranging and projecting both sides onto \(\hat{\mathbf{v}}_{\mathbf{d}}\) gives:
Inserting this into Eq. (2),
Except ω, the terms on the right hand side are independent of ρ. Since \(\mathbf{v}_{\mathbf{p}}.\hat{\mathbf{v}}_{\mathbf{d}} < 0\), as ω increases v decreases, and vice versa. Then the only way ω/v can increase is when ω increases. □
Lemma 3
For a given configuration of the object and the finger, if we change the pressure distribution ρ such that it is concentrated closer to the center of the object, the change in v will have a nonnegative projection on \(\hat{\mathbf{v}}_{\mathbf{d}}\).
Proof
We will look at different contact modes separately. The separation mode is trivial. The object will not move for both values of ρ. The change in v will have a null projection on \(\hat{\mathbf{v}}_{\mathbf{d}}\).
Assume that the contact mode is sliding. Then f will be aligned with one of the friction cone edges; let’s assume f _{ R } without loss of generality. Since v∥f, then v is also a vector with direction f _{ R }
where \(\hat{\mathbf{f}}_{\mathbf{R}}\) is the unit direction along f _{ R }. Inserting this into the result from Lemma 1 we have
Then
Multiplying both sides with \({\hat{f}_{R}}\) we have
None of the terms get affected by a change in ρ, i.e. the change in v will have a null projection on \(\hat{\mathbf{v}}_{\mathbf{d}}\).
As ρ concentrates at the center, ω/v will get larger. The motion cone edges will then get more and more aligned with the direction of ω×r, making the motion cone wider. At the point when the motion cone edge reaches v _{ p } the contact is no more a sliding contact but a sticking one.
When the contact is sticking we have
Then
If we rewrite (ω×r) using the direction \(\hat{\mathbf{v}}_{\mathbf{d}}\), we get
Except ω, the terms on the right hand side are independent of ρ. By Lemma 2, we know that as ρ concentrates around the center of the object, ω increases; i.e. the change in v has a positive projection on \(\hat{\mathbf{v}}_{\mathbf{d}}\). □
Now we look at the effect of μ _{ c }, the friction coefficient between the object and the finger.
Lemma 4
For a given configuration of the object and the finger, if we decrease the value of μ _{ c }, the change in v will have a nonnegative projection on \(\hat{\mathbf{v}}_{\mathbf{d}}\).
Proof
Again we look at the two contact modes separately.
The sticking contact case is trivial. f is inside the friction cone. If we decrease μ _{ c }, the friction cone will get narrower, but as long as it does not get narrow enough to leave f outside, the contact is still sticking. There is no effect to the velocities of the motion, including v. The change in v has a null projection on \(\hat{\mathbf{v}}_{\mathbf{d}}\).
If we continue to decrease μ _{ c } at one point the contact will become sliding. f will be at the edge of the friction cone and the friction cone will get narrower as we decrease μ _{ c }. Without loss of generality, let’s assume f is along f _{ R }. Since v∥f, v will also move with f _{ R }. Pictorially v will change as in Fig. 19, resulting in a change along \(\hat{\mathbf{v}}_{\mathbf{d}}\). Formally, in the proof of Lemma 3 we showed that during sliding contact
By the definition of the friction cone we have
Replacing this into the equation above and noting that \(\hat{\mathbf {v}}_{\mathbf{d}}.\hat{\mathbf{n}}=0\) we have
Then we have
Except μ _{ c } itself, the terms on the right hand side are independent of μ _{ c }. The contact mode requires that \(\mathbf{v}_{\mathbf {p}}.\hat{\mathbf{n}} > 0\). Hence, as μ _{ c } decreases the change in v will be positive in the direction of \(\hat {\mathbf{v}}_{\mathbf{d}}\).
□
Now we are ready to state and prove our main theorem.
Theorem 1
For a cylindrical object under quasistatic pushing, where the quasistatic motion is approximated by the ellipsoid limit surface (Howe and Cutkosky 1996), as μ _{ c } gets smaller or as ρ gets concentrated around the center, the required pushing distance for a pushgrasp will decrease or stay the same (but not increase).
Proof
The pushgrasp starts at a certain configuration between the finger and the object, and continues until the object’s center passes the graspline at the fingertip and orthogonal to the pushing direction (Fig. 20). Since we assume that μ _{ c } is uniform all around the object, we can ignore the rotation of the cylinder and simply consider its position relative to the finger. Then, independent of ρ or μ _{ c }, the finger and the object will go through all the configurations between r _{ start } to r _{ final } during the pushgrasp. We will show below that the velocity the object center moves towards the graspline never decreases as μ _{ c } gets smaller or as ρ gets concentrated around the center.
For a given configuration of the object and the finger, the object center’s velocity is given by v (ω does not have an effect). We can express v using its components
Lemma 1 tells us that the component of v along \(\hat{\mathbf{n}}\) is fixed for different ρ or μ _{ c }:
Hence, the only change in the object center’s motion happens along \(\hat{\mathbf{v}}_{\mathbf{d}}\). Lemmas 3 and 4 states that the change in v will be nonnegative along \(\hat{\mathbf{v}}_{\mathbf{d}}\). □
Rights and permissions
About this article
Cite this article
Dogar, M.R., Srinivasa, S.S. A Planning Framework for NonPrehensile Manipulation under Clutter and Uncertainty. Auton Robot 33, 217–236 (2012). https://doi.org/10.1007/s105140129306z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s105140129306z