Hierarchical and multiple hand action representation using temporal postural synergies

Abstract

The notion of synergy enables one to provide simplified descriptions of hand actions. It has been used in a number of different meanings ranging from kinematic and dynamic synergies to postural and temporal postural synergies. However, relatively little is known about how representing an action by synergies might take into account the possibility to have a hierarchical and multiple action representation. This is a key aspect for action representation as it has been characterized by action theorists and cognitive neuroscientists. Thus, the aim of the present paper is to investigate whether and to what extent a hierarchical and multiple action representation can be obtained by a synergy approach. To this purpose, we took advantage of representing hand action as a linear combination of temporal postural synergies (TPSs), but on the assumption that TPSs have a tree-structured organization. In a tree-structured organization, a hand action representation can involve a TPS only if the ancestors of the synergy in the tree are themselves involved in the action representation. The results showed that this organization is enough to force a multiple representation of hand actions in terms of synergies which are hierarchically organized.

Appendix: Tree-structured synergy method (TSSM)

We will use the following notations. Bold uppercase letters refer to matrices, for example, X, V, and bold lowercase letters designate vectors, for example, x, v. We denote by X _i and X ^j the ith row and the jth column of a matrix X, respectively. We use the notation x _i and v _ij to refer to the ith element of the vector x and the element in the ith row and the jth column of the matrix V, respectively. Given ${{\bf x}\in\mathbb{R}^{p}}$ we use the notation $\|x\|$ to refer to $l_{\infty}$ norm. Given two vectors x and y in ${\mathbb{R}^{p}}$, we denote by ${\bf x}\circ{\bf y}=(x_{1}y_{1},x_{2}y_{2},\ldots,x_{p}y_{p})\in R^{p}$ the element-wise product of x and y.

Tree-structured stage

The update of the U values is performed in this stage, and, more importantly, following the approach suggested by Jenatton et al. (2010), a tree-structured representation of the rows in X is found. The main difficulty is that the optimization of the ${\bf U}_{i}, i\in{1,2, \ldots,n}$, for a fixed V involves the nonsmooth regularization term $\Upomega({\bf U}_{i})=\sum\nolimits_{j=1}^{r}w_{j}\|{\bf D}_{j}\circ{\bf U}_{i}\|$, where w _j are positive weights.² In this case the update of the vectors U _i can be performed using a proximal method (Combettes and Wajs 2006). In general, proximal approaches are used when one has to minimize a convex nonsmooth objective function which assumes the following general form:

$$ f({\bf u})+\lambda\Upomega({\bf u}) $$

where f(u) is the usual data-fitting term $\frac{1}{2}\Vert{\bf x}-{\bf u}{\bf V}^{T}\Vert_{2}^{2}$ and $\Upomega({\bf u})$ is a nondifferentiable regularization term. In a nutshell, the proximal approach consists of two consecutive updating steps: first, the vector u is updated using the standard gradient update rule w.r.t the first term of the objective function as follows:

$$ \bar{{\bf u}}\leftarrow{\bf u}-\frac{1}{\sigma_{{\bf U}}}\nabla f({\bf u})={\bf u}+\frac{1}{\sigma_{{\bf U}}}({\bf x}-{\bf u}{\bf V}^{T}){\bf V} $$

(5)

then, starting from the value $\bar{{\bf u}}$ the new value for u is computed by applying a proximal operator $\Uppi_{{\bf U}}$ defined by the following minimization problem:

$$ \Uppi_{{\bf U}}({\bf u})=argmin_{{\bf v}}\frac{1}{2}\Vert{\bf u}-{\bf v}\Vert_{2}^{2}+\lambda\Upomega({\bf v}) $$

(6)

Thus, we obtain ${\bf u}^{new}\leftarrow\Uppi_{{\bf U}}(\bar{{\bf u}})$. For a number of regularization terms, the minimization problem expressed in (6) can lead to closed-form solutions. For example, when $\Upomega({\bf u})$ is the ℓ₁ norm of u the corresponding proximal operator $\Uppi_{{\bf U}}$ is the well-known soft-thresholding operator. In the case of the regularization term used here, this minimization problem can be solved by a primal-dual approach which enables us to implement the proximal operator defined in (6) by the procedure presented in Algorithm (1).

Summarizing, the optimization of the U _i values is performed using the gradient descent rule expressed in (5) and, then, applying the proximal operator as defined previously.

Synergy dictionary stage

This stage consists in updating the V’s values while keeping fixed the values of U. Note that the objective function in (1) is composed of two terms to be minimized, and the second term does not depend on V. Therefore, the optimization problem posed in (1) can be, in this stage, reformulated as follows:

$$ \min\limits_{{\bf V}}\frac{1}{2np}\|{\bf X}-{\bf U}{\bf V}^{T}\|_{F}^{2} \; s.t. \; \forall i \|{\bf V}^{i}\|_{2}\leq1 $$

(7)

Due the fact that the columns of V are constrained to lie inside the unit ball, the update of V is performed in two consecutive steps. First, we apply a standard gradient updating rule as follows

$$ {{\bar{\bf{V}}}}\leftarrow{\bf V}+\frac{1}{\sigma_{{\bf V}}np}({\bf X}-{\bf U}{\bf V}^{T}){\bf U}^{T} $$

(8)

where η is a parameter. Then, we use the projection operator $\Uppi({\bf v})=\frac{{\bf v}}{max\{1,\Vert{\bf v}\Vert_{2}\}}$ in order to project the columns of ${{\bar{\bf{V}}}}$ on the unit ball in ${\mathbb{R}^{p}}$. Consequently, the update of V is computed as follows:

$$ {\bf V}\leftarrow\Uppi \left({\bf V}+\frac{1}{\sigma_{{\bf V}}np}({\bf X}-{\bf U}{\bf V}^{T}){\bf U}^{T} \right) $$

(9)

The overall algorithm of TSSM is reported in algorithm 2. Note that a fixed step gradient descent procedure was adopted with the two learning rate σ _U and σ _V chosen equal to the Lipschitz constant of ∇ f(u) and ∇ f(v), respectively.