Text2Motion: from natural language instructions to feasible plans

Abstract

We propose Text2Motion, a language-based planning framework enabling robots to solve sequential manipulation tasks that require long-horizon reasoning. Given a natural language instruction, our framework constructs both a task- and motion-level plan that is verified to reach inferred symbolic goals. Text2Motion uses feasibility heuristics encoded in Q-functions of a library of skills to guide task planning with Large Language Models. Whereas previous language-based planners only consider the feasibility of individual skills, Text2Motion actively resolves geometric dependencies spanning skill sequences by performing geometric feasibility planning during its search. We evaluate our method on a suite of problems that require long-horizon reasoning, interpretation of abstract goals, and handling of partial affordance perception. Our experiments show that Text2Motion can solve these challenging problems with a success rate of 82%, while prior state-of-the-art language-based planning methods only achieve 13%. Text2Motion thus provides promising generalization characteristics to semantically diverse sequential manipulation tasks with geometric dependencies between skills. Qualitative results are made available at https://sites.google.com/stanford.edu/text2motion.

Appendices

Appendix A Implementation details

The Text2Motion planner integrates both shooting and greedy-search to construct skill sequences that are feasible for the robot to execute in the environment. The planning procedure relies on four core components: (1) a library of learned robot skills, (2) a method for detecting when a skill is out-of-distribution (OOD), (3) a large language model (LLM) to perform task-level planning, and (4) a geometric feasibility planner that is compatible with the learned robot skills. All evaluated language-based planners use the above components, while saycan-gs and innermono-gs are myopic agents that do not perform geometric feasibility planning. We provide implementation details of these components in the following subsections.

1.1 A.1 Learning robot skills and dynamics

Skill library overview: All evaluated language planners interface an LLM with a library of robot skills $\mathcal {L}=\{\psi ^1, \ldots , \psi ^N\}$. Each skill $\psi$ has a language description (e.g. Pick(a)) and is associated with a parameterized manipulation primitive (Felip et al., 2013) $\phi (a)$. A primitive $\phi (a)$ is controllable via its parameter a which determines the motion (Khatib, 1987) of the robot’s end-effector through a series of waypoints. For each skill $\psi$, we train a policy $\pi (a|s)$ to output parameters $a\in \mathcal {A}$ that maximize primitive’s $\phi (a)$ probability of success in a contextual bandit setting (Eq. 1) with a skill-specific binary reward function $R(s, a, s')$. We also train an ensemble of Q-functions $Q_{1:B}^\pi (s, a)$ and a dynamics model $T^\pi (s' | s, a)$ for each skill, both of which are required for geometric feasibility planning. We discuss the calibration of Q-function ensembles for OOD detection of skills in “Appendix A.2”.

We learn four manipulation skills to solve tasks in simulation and in the real-world: $\psi ^{\text {Pick}}$, $\psi ^{\text {Place}}$, $\psi ^{\text {Pull}}$, $\psi ^{\text {Push}}$. Only a single policy per skill is trained, and thus, the policy must learn to engage the primitive over objects with differing geometries (e.g. $\pi ^{\text {Pick}}$ is used for both Pick(box) and Pick(hook)). The state space $\mathcal {S}$ for each policy is defined as the concatenation of geometric state features (e.g. pose, size) of all objects in the scene, where the first n object states correspond to the n skill arguments and the rest are randomized. For example, the state for the skill Pick(hook) would have be a vector of all objects’ geometric state features with the first component of the state corresponding to the hook.

Parameterized manipulation primitives: We describe the parameters a and reward function $R(s, a, s')$ of each parameterized manipulation primitive $\phi (a)$ below. A collision with a non-argument object constitutes an execution failure for all skills, and as a result, the policy receives a reward of 0. For example, $\pi ^{\text {Pick}}$ would receive a reward of 0 if the robot collided with a box during the execution of Pick(hook).

• Pick(obj): $a\sim \pi ^{\text {Pick}}(a \vert s)$ denotes the grasp pose of obj w.r.t the coordinate frame of obj. A reward of 1 is received if the robot successfully grasps obj.

• Place(obj, rec): $a\sim \pi ^{\text {Place}}(a \vert s)$ denotes the placement pose of obj w.r.t the coordinate frame of rec. A reward of 1 is received if obj is stably placed on rec.

• Pull(obj, tool): $a\sim \pi ^{\text {Pull}}(a \vert s)$ denotes the initial position, direction, and distance of a pull on obj with tool w.r.t the coordinate frame of obj. A reward of 1 is received if obj moves toward the robot by a minimum of $d_{\text {Pull}}=0.05m$.

• Push(obj, tool, rec): $a\sim \pi ^{\text {Push}}(a \vert s)$ denotes the initial position, direction, and distance of a push on obj with tool w.r.t the coordinate frame of obj. A reward of 1 is received if obj moves away from the robot by a minimum of $d_{\text {Push}}=0.05m$ and if obj ends up underneath rec.

Dataset generation: All planners considered in this work rely on accurate Q-functions $Q^\pi (s, a)$ to estimate the feasibility of skills proposed by the LLM. This places a higher fidelity requirement on the Q-functions than typically needed to learn a reliable policy, as the Q-functions must characterize both skill success (feasibility) and failure (infeasibility) at a given state. Because the primitives $\phi (a)$ reduce the horizon of policies $\pi (a|s)$ to a single timestep, and the reward functions are $R(s, a, s')\in \{0, 1\}$, the Q-functions can be interpreted as binary classifiers of state-action pairs. Thus, we take a staged approach to learning the Q-functions $Q^\pi$, followed by the policies $\pi$, and lastly the dynamics models $T^\pi$.

Scenes in our simulated environment are instantiated from a symbolic specification of objects and spatial relations, which together form a symbolic state. The goal is to learn a Q-function that sufficiently covers the state-action space of each skill. We generate a dataset that meets this requirement in four steps: a) enumerate all valid symbolic states; b) sample geometric scene instances s per symbolic state; c) uniformly sample actions over the action space $a\sim \mathcal {U}^{[0,1]^d}$; (d) simulate the states and actions to acquire next states $s'$ and compute rewards $R(s, a, s')$. We slightly modify this sampling strategy to maintain a minimum success-failure ratio of 40%, as uniform sampling for more challenging skills such as Pull and Push seldom emits a success ($\sim$3%). We collect 1 M $(s, a, s', r)$ tuples per skill of which 800K of them are used for training ($\mathcal {D}_t$), while the remaining 200K are used for validation ($\mathcal {D}_v$). We use the same datasets to learn the Q-functions $Q^\pi$, policies $\pi$, and dynamics models $T^\pi$ for each skill.

Model training: We train an ensemble of Q-functions with mini-batch gradient descent and logistic regression loss. Once the Q-functions have converged, we distill their returns into stochastic policies $\pi$ through the maximum-entropy update (Haarnoja et al., 2018):

$$\begin{aligned} \begin{aligned} \pi ^* \leftarrow \arg \max _{\pi } \, {\text {E}}_{(s,a)\sim \mathcal {D}_t} [\min (Q_{1:B}^\pi (s, a)) \\ - \alpha \log \pi (a|s) ]. \end{aligned} \end{aligned}$$

Instead of evaluating the policies on $\mathcal {D}_v$, which contains states for which no feasible action exists, the policies are synchronously evaluated in an environment that exhibits only feasible states. This simplifies model selection and standardizes skill capabilities across primitives. All Q-functions achieve precision and recall rates of over 95%. The average success rates of the converged policies over 100 evaluation episodes are: $\pi _{\text {Pick}}$ with 99%, $\pi _{\text {Place}}$ with 90%, $\pi _{\text {Pull}}$ with 86%, $\pi _{\text {Push}}$ with 97%.

We train a deterministic dynamics model per skill using the forward prediction loss:

$$\begin{aligned} L_{\text {dynamics}}\left( T^\pi ; \mathcal {D}_t \right) = {\text {E}}_{(s,a,s')\sim \mathcal {D}_t}||T^\pi (s, a) - s'||_2^2. \end{aligned}$$

The dynamics models converge to within millimeter accuracy on the validation split.

Hyperparameters: The Q-functions, policies, and dynamics models are MLPs with hidden dimensions of size [256, 256] and ReLU activations. We train an ensemble of $B=8$ Q-functions with a batch size of 128 and a learning rate of 1$e^{-4}$ with a cosine annealing decay (Loshchilov and Hutter, 2017). The Q-functions for Pick, Pull, and Push converged on $\mathcal {D}_v$ in 3 M iterations, while the Q-function for Place required 5 M iterations. We hypothesize that this is because classifying successful placements demands carefully attending to the poses and shapes of all objects in the scene so as to avoid collisions. The policies are trained for 250K iterations with a batch size of 128 and a learning rate of 1$e^{-4}$, leaving all other parameters the same as Haarnoja et al. (2018). The dynamics models are trained for 750K iterations with a batch size of 512 and a learning rate of 5$e^{-4}$; only on successful transitions to avoid the noise associated with collisions and truncated episodes. The parallelized training of all models takes approximately 12 hours on an Nvidia Quadro P5000 GPU and 2 CPUs per job.

1.2 A.2 Out-of-distribution detection

The training dataset described in Sect. 1 contain both successes and failures for symbolically valid skills like Pick(box). However, when using LLMs for robot task planning, it is often the case that the LLM will propose symbolically invalid skills, such as Pick(table), that neither the skill’s policy, Q-functions, or dynamics model have observed in training. We found that a percentage of out-of-distribution (OOD) queries would result in erroneously high Q-values, causing the invalid skill to be selected. Attempting to execute such a skill leads to control exceptions or other failures.

Whilst there are many existing techniques for OOD detection of deep neural networks, we opt to detect OOD queries on the learned Q-functions via deep ensembles due to their ease of calibration (Lakshminarayanan et al., 2017). A state-action pair is classified as OOD if the empirical variance of the predicted Q-values is above a determined threshold:

$$\begin{aligned} \mathop {}F_{\text {OOD}}\left( \psi \right) = \mathbb {1} \left( \mathop {}{\text {Var}}_{i\sim 1:B}\left[ Q^{\pi }_i(s, a)\right] \ge \epsilon ^{\psi } \right) , \end{aligned}$$

where each threshold $\epsilon ^\psi$ is unique to skill $\psi$.

To determine the threshold values, we generate a calibration dataset of 100K symbolically invalid states and actions for each skill. The process takes less than an hour on a single CPU as the actions are infeasible and need not be simulated in the environment (i.e. rewards are known to be 0). We compute the mean and variance of the Q-ensemble for each (s, a) sample in both the training dataset (in-distribution inputs) and the calibration dataset (out-of-distribution inputs), and produce two histograms by binning the computed ensemble variances by the ensemble means. We observe that the histogram of variances corresponding to OOD inputs is uniform across all Q-value bins and is an order of magnitude large than the ensemble variances computed over in-distribution inputs. This allows us to select thresholds $\epsilon ^\psi$ which are low enough to reliably detect OOD inputs, yet will not be triggered for in-distribution inputs: $\epsilon ^{\text {Pick}} = 0.10$, $\epsilon ^{\text {Place}} = 0.12$, $\epsilon ^{\text {Pull}} = 0.10$, and $\epsilon ^{\text {Push}} = 0.06$.

1.3 A.3 Task planning with LLMs

Text2Motion, greedy-search , and the myopic planning baselines saycan-gs and innermono-gs usecode-davinci-002 (Chen et al., 2021) to generate and score skills, while shooting queries text-davinci-003 (Ouyang et la., 2022) to directly output full skill sequences. In our experiments, we used a temperature setting of 0 for all LLM queries.

To maintain consistency in the evaluation of various planners, we allow Text2Motion, saycan-gs , and innermono-gs to generate $K=5$ skills $\{\psi ^1_t, \ldots , \psi ^K_t\}$ at each timestep t. Thus, every search iteration of greedy-search considers five possible extensions to the current running sequence of skills $\psi _{1:t-1}$. Similarly, shooting generates $K=5$ skill sequences.

As described in Sect. 4.3, skills are selected at each timestep t via a combined usefulness and geometric feasibility score:

$$\begin{aligned} S_{\text {skill}}(\psi _t)&= S_{\text {llm}}(\psi _t) \cdot S_{\text {geo}}(\psi _t) \\&\approx p(\psi _t \mid i, s_{1:t}, \psi _{1:t-1}) \cdot Q^{\pi _t}(s_t, a^*_t), \end{aligned}$$

where Text2Motion, greedy-search , and shooting use geometric feasilibity planning (details below in“Appendix A.4”) to compute $S_{\text {geo}}(\psi _t)$, while saycan-gs and innermono-gs use the current value function estimate $V^{\pi _t}(s_t) = {\text {E}}_{a_t\sim \pi _t}[Q^{\pi _t}(s_t, a_t)]$. We find that in both cases, taking $S_{\text {llm}}(\psi _t)$ to be the SoftMax log-probability score produces a winner-takes-all effect, causing the planner to omit highly feasible skills simply because their associated log-probability was marginally lower than the LLM-likelihood of another skill. Thus, we dampen the SoftMax operation with a $\beta$-coefficient to balance the ranking of skills based on both feasibility and usefulness. We found $\beta =0.3$ to work well.

1.4 A.4 Geometric feasibility planning

Given a sequence of skills $\psi _{1:H}$, geometric feasibility planning computes parameters $a_{1:H}$ that maximizes the success probability of the underlying sequence of primitives $\phi _{1:H}$. For example, given a skill sequence Pick(hook), Pull(box, hook), geometric feasibility planning would compute a 3D grasp position on the hook that enables a successful pull on the box thereafter.

Text2Motion is agnostic to the method that fulfils the role of geometric feasibility planning. In our experiments we leverage Sequencing Task-Agnostic Policies (STAP) (Agia et al., 2022). Specifically, we consider the PolicyCEM variant of STAP, where optimization of the skill sequence’s success probability (Eq. 4) is warm started with parameters sampled from the policies $a_{1:H}\sim \pi _{1:H}$. We perform ten iterations of the Cross-Entropy Method (Rubinstein, 1999), sampling 10K trajectories at each iteration and selecting 10 elites to update the mean of the sampling distribution for the following iteration. The standard deviation of the sampling distribution is held constant at 0.3 for all iterations.

Algorithm 1

Shooting-based LLM planner

Algorithm 2

Search-based LLM planner

1.5 A. 5 Algorithms

In this section, we provide algorithm pseudocode for all language-based planners proposed in this work: shooting in Algorithm 1, greedy-search in Algorithm 2, and Text2Motion in Algorithm 3.

Algorithm 3

Text2Motion hybrid planner

Appendix B Experiment details

Table 2 TableEnv manipulation task suite

We refer to Table 2 for an overview of the tasks in the TableEnv Manipulation suite.

1.1 B.1 Scene descriptions as symbolic states

For the remainder of this section, we use the following definitions of terms:

• Predicate: a binary-valued function over objects that evaluates to true or false (e.g. on(a, b))

• Spatial Relation: a predicate grounded over objects that evaluates to true (e.g. on(rack, table))

• Predicate Classifier: a function that implements whether a predicate is true or false in the scene. In this work, we use hand-crafted predicate classifiers for each spatial relation we model

• Symbolic State: the set of all predicates that hold true in the scene

• Satisfaction Function: a binary-valued function that takes as input a geometric state, uses the predicate classifiers to detect what predicates hold true in the geometric state, and collects those predicates into a set to form a symbolic state. The satisfaction function evaluates to true if the predicted goals (predicates) hold in the symbolic state

To provide scene context to Text2Motion and the baselines, we take a heuristic approach to converting a geometric state s into a basic symbolic state. Symbolic states consist of combinations of one or more of the following predicates that have been grounded: on(a, b), under(a, b), and inhand(a). inhand(a) = True when the height of object a is above a predefined threshold. on(a, b) = True when (i) object a is above b (determined by checking if the centroid of a’s axis-aligned bounding box is greater than b’s axis-aligned bounding box), (ii) a’s bounding box intersects b’s bounding box, and (iii) inhand(a) = False. under(a, b) = True when on(a, b) = False and a’s bounding box intersects b’s bounding box.

The proposed goal proposition prediction technique outputs goal propositions consisting of combinations of the three spatial relations above. As an example, for the natural language instruction “Put two of the boxes under the rack" and a symbolic state [on(red box, table), on(green box, rack), on(hook, rack), on(blue box, rack)], the LLM might predict the set of three goals {[under(red box, rack), under(blue box, rack)], [under(red box, rack),    under(green box,     rack)],     [under(green box, rack), under(blue box, rack)]}. We highlight that objects are neither specified as within or beyond the robot workspace, as we leave it to the skill’s Q-functions to determine feasibility (Sect. 1).

Since planning in high-dimensional observation spaces is not the focus of this work, we assume knowledge of objects in the scene and use hand-crafted heuristics to detect spatial relations between objects. There exists several techniques to convert high-dimensional observations into scene descriptions, such as the one used in Zeng et al. (2022). We leave exploration of these options to future work.

1.2 B.2 In-context examples

For all experiments and methods, we use the following in-context examples to construct the prompt passed to the LLMs.

Appendix C Derivations

We provide two derivations to support our approximation of the skill score $S_{\text {skill}}$ (used to select skills while planning with greedy-search and Text2Motion) defined in Eq. 8. The skill score is expressed as a product of two terms:

$$\begin{aligned} \begin{aligned} S_{\text {skill}}(\psi _t)&= p(\psi _t \mid i, s_1, \psi _{1:t-1}, r_{1:t-1}) \\&\quad \quad \quad \quad p(r_t \mid i, s_1, \psi _{1:t}, r_{1:t-1}). \end{aligned} \end{aligned}$$

(C1)

1.1 C. 1 Skill usefulness derivation

Equation 9 defines the first term in the skill score product to be the skill usefulness score $S_{\text {llm}}$. We derive the approximation of $S_{\text {llm}}$ given in Eq. 10, which corresponds to quantity we use in our experiments.

$$\begin{aligned} S_{\text {llm}}(\psi _t)&= p(\psi _t \mid i, s_1, \psi _{1:t-1}, r_{1:t-1}) \nonumber \\&= \int p(\psi _t \mid i, s_{1:t}, \psi _{1:t-1}, r_{1:t-1}) \nonumber \\&\quad \quad \quad \quad p(s_{2:t} \mid i, s_1, \psi _{1:t-1}, r_{1:t-1}) \,d s_{2:t} \nonumber \\&= \mathop {}{\text {E}}_{s_{2:t}}\left[ p(\psi _t \mid i, s_{1:t}, \psi _{1:t-1}, r_{1:t-1})\right] \end{aligned}$$

(C2)

$$\begin{aligned}&\approx \mathop {}{\text {E}}_{s_{2:t}}\left[ p(\psi _t \mid i, s_{1:t}, \psi _{1:t-1})\right] \end{aligned}$$

(C3)

$$\begin{aligned}&\approx p(\psi _t \mid i, s_{1:t}, \psi _{1:t-1}) \end{aligned}$$

(C4)

The final expression is given in Eq. C4. Here, we compute a single sample Monte-Carlo estimate of Eq. C3 under the future state trajectory $s_{2} \sim T^{\pi _{1}}(\cdot | s_{1}, a^*_{1}), \ldots , s_{t} \sim T^{\pi _{t-1}}(\cdot | s_{t-1}, a^*_{t-1})$, where $a^*_{1:t-1}$ is computed by STAP (Agia et al., 2022). The key insight is that future state trajectories $s_{2:t}$ are only ever sampled after STAP has performed geometric feasibility planning to maximize the success probability (Eq. 3) of the running plan $\psi _{1:t-1}$. By doing so, we ensure that the future states $s_{2:t}$ correspond to a successful execution of the running plan $\psi _{1:t-1}$, i.e. achieving positive rewards $r_{1:t-1}$. This supports the independence assumption on rewards $r_{1:t-1}$ used to derive Eq. C3 from Eq. C2.

1.2 C. 2 Skill feasibility derivation

Equation 11 defines the second term in the skill score product (Eq. C1) as the skill feasibility score $S_{\text {geo}}$. We derive the approximation provided in Eq. 12, which is the quantity we use in our experiments.

$$\begin{aligned} S_{\text {geo}}(\psi _t)&= p(r_t \mid i, s_1, \psi _{1:t}, r_{1:t-1}) \end{aligned}$$

(C5)

$$\begin{aligned}&= p(r_t \mid s_1, \psi _{1:t}, r_{1:t-1}) \end{aligned}$$

(C6)

$$\begin{aligned}&= \int p(r_t \mid s_{1:t}, \psi _{1:t}, r_{1:t-1}) \nonumber \\&\quad \quad \quad \quad p(s_{2:t} \mid s_1, \psi _{1:t}, r_{1:t-1}) \,d s_{2:t} \nonumber \\&= \mathop {}{\text {E}}_{s_{2:t}}\left[ p(r_t \mid s_{1:t}, \psi _{1:t}, r_{1:t-1})\right] \end{aligned}$$

(C7)

$$\begin{aligned}&\approx \mathop {}{\text {E}}_{s_{2:t}}\left[ p(r_t \mid s_{1:t}, \psi _{1:t})\right] \end{aligned}$$

(C8)

$$\begin{aligned}&= \mathop {}{\text {E}}_{s_{2:t}}\left[ p(r_t \mid s_{1:t}, a^*_{1:t})\right] \end{aligned}$$

(C9)

$$\begin{aligned}&= \mathop {}{\text {E}}_{s_{2:t}}\left[ p(r_t \mid s_t, a^*_t)\right] \end{aligned}$$

(C10)

$$\begin{aligned}&= \mathop {}{\text {E}}_{s_{2:t}}\left[ Q^{\pi _t}(s_t, a^*_t)\right] \end{aligned}$$

(C11)

$$\begin{aligned}&\approx Q^{\pi _t}(s_t, a^*_t) \end{aligned}$$

(C12)

From Eqs. C5 to C6, the reward $r_t$ is conditionally independent of the instruction i given the initial state $s_1$, running plan $\psi _{1:t}$, and previous rewards $r_{1:t-1}$. As described in “Appendix C.1”, we can use STAP to make an independence assumption on the previous rewards $r_{1:t-1}$ between Eq. C7 and Eq. C8. The reward probability in Eq. C8 depends on the parameters $a^*_{1:t}$ computed by STAP and fed to the underlying primitive sequence $\phi _{1:t}$, which gives Eq. C9. Equation C10 comes from the Markov assumption, and can be reduced to Eq. C11 by observing that the reward probability $p(r_t \mid s_t, a^*_t)$ is equal to the Q-value $Q^{\pi _t}(s_t, a^*_t)$ in the contextual bandit setting we consider. The final expression given in Eq. C12, which represents a single sample Monte-Carlo estimate of Eq. C11 under a sampled future state trajectory $s_{2} \sim T^{\pi _{1}}(\cdot | s_{1}, a^*_{1}), \ldots , s_{t} \sim T^{\pi _{t-1}}(\cdot | s_{t-1}, a^*_{t-1})$.

Appendix D Real world demonstration

1.1 D. 1 Hardware setup

We use a Kinect V2 camera for RGB-D image capture and manually adjust the color thresholds to segment objects in the scene. Given the segmentation masks and the depth image, we can estimate object poses to construct the geometric state of the environment. For the skill library, we use the same set of policies, Q-functions, and dynamics models trained in simulation. We run robot experiments on a Franka Panda robot manipulator.

1.2 D.2 Robot demonstration

Please see our project page for demonstrations of Text2Motion operating on a real robot.