Online learning and control of attraction basins for the development of sensorimotor control strategies

Abstract

Imitation and learning from humans require an adequate sensorimotor controller to learn and encode behaviors. We present the Dynamic Muscle Perception–Action(DM-PerAc) model to control a multiple degrees-of-freedom (DOF) robot arm. In the original PerAc model, path-following or place-reaching behaviors correspond to the sensorimotor attractors resulting from the dynamics of learned sensorimotor associations. The DM-PerAc model, inspired by human muscles, permits one to combine impedance-like control with the capability of learning sensorimotor attraction basins. We detail a solution to learn incrementally online the DM-PerAc visuomotor controller. Postural attractors are learned by adapting the muscle activations in the model depending on movement errors. Visuomotor categories merging visual and proprioceptive signals are associated with these muscle activations. Thus, the visual and proprioceptive signals activate the motor action generating an attractor which satisfies both visual and proprioceptive constraints. This visuomotor controller can serve as a basis for imitative behaviors. In addition, the muscle activation patterns can define directions of movement instead of postural attractors. Such patterns can be used in state-action couples to generate trajectories like in the PerAc model. We discuss a possible extension of the DM-PerAc controller by adapting the Fukuyori’s controller based on the Langevin’s equation. This controller can serve not only to reach attractors which were not explicitly learned, but also to learn the state/action couples to define trajectories.

Appendix: summary of the parameters and equations used in the Dynamic Muscle PerAc model

The different parameters and equations presented in this article are respectively summarized in Tables 1 and 2.

The proprioceptive (visual) categorization depends on the vigilance parameter $\lambda ^P$ ($\lambda ^V$) and the parameter $\beta ^P$ ($\beta ^V$) of the Gaussian similarity measure. High vigilance values induce that recruited categories overlap. On the contrary, we use $\lambda ^P=\lambda ^V=0.005$ to avoid interferences between categories. The values of the Gaussian parameters are very low so the categories are selective enough. During the learning step, different values are used to increase progressively the number of learned categories ($\beta ^P = 0.002$ then $\beta ^P = 0.001$, and $\beta ^V = 2\cdot 10^{-4}$ then $\beta ^V = 5 \cdot 10^{-5}$). During the tests, vision must drive the movements, thus the proprioceptive categories must be less selective than the visual categories ($\beta ^P = 0.1$ and $\beta ^V = 5\cdot 10^{-5}$).

In the experiments, muscle activation learning depends on the learning factor $\varepsilon ^A=10^{-3}$ and the decay factor $\alpha ^A=10^{-4}$. As the learning factor is small, the stiffness $K_j$ of each joint changes slowly. Still, the equilibrium position is rapidly adapted because it depends on the ratio of the muscle activations. Also, the decay must be slow enough to allow the learning. With an error threshold $th_D=0.01$, the muscle activations around a joint are adapted if the position error is over a hundredth of the rotational range.

The parameters $th_L$ and $\gamma ^L$ define the dynamics of the “learning enable” signal $L$, i.e. determine the amount of time to learn each postural attractor. The used values are $th_L=10^{-5}$ and $\gamma ^L=0.95$, thus the motor exploration resumes after a time period of about 10 s without correction of the movements.