Temporal Parameter-Free Deep Skinning of Animated Meshes

Abstract

In computer graphics, animation compression is essential for efficient storage, streaming and reproduction of animated meshes. Previous work has presented efficient techniques for compression by deriving skinning transformations and weights using clustering of vertices based on geometric features of vertices over time. In this work we present a novel approach that assigns vertices to bone-influenced clusters and derives weights using deep learning through a training set that consists of pairs of vertex trajectories (temporal vertex sequences) and the corresponding weights drawn from fully rigged animated characters. The approximation error of the resulting linear blend skinning scheme is significantly lower than the error of competent previous methods by producing at the same time a minimal number of bones. Furthermore, the optimal set of transformation and vertices is derived in fewer iterations due to the better initial positioning in the multidimensional variable space. Our method requires no parameters to be determined or tuned by the user during the entire process of compressing a mesh animation sequence.

Appendices

A Appendix A

We build an appropriate neural network model that classifies each vertex by capturing mesh geometry and vertex kinematics. Then we use a set of human and animal animations to train the neural network model. We achieve this by using as input features the trajectories of all vertices and as output the weights that represent how each vertex is influenced by a bone. The output weight is conceived by the network as the probability of a bone to influence the corresponding vertex. Subsequently, we provide as input to our network arbitrary mesh animation sequences and predict their weights. From the per vertex classifier we determine the number of bones and the weights for each vertex.

Fig. 1.

Temporal Deep Skinning workflow.

Fig. 2.

Deep Skinning optimization workflow for weights and transformations.

The first error measure is the percentage of deformation known as distortion percentage (DisPer).

$$\begin{aligned} DisPer = 100 \cdot \frac{\Vert A_{orig} - A_{Approx}\Vert _F}{\Vert A_{orig} - A_{avg}\Vert _F}. \end{aligned}$$

(3)

where $\Vert \cdot \Vert _F$ is the Frobenius matrix metric. In Eq. 3 $A_{orig}$ is a 3NP matrix which consists of the real vertex coordinates in all frames of the model. Similarly, $A_{Approx}$ has all the approximated vertex coordinates and matrix $A_{avg}$ contains in each column, the average of the original coordinates in all frames. [14] replaces 100 by 1000 and divides by the surrounding sphere diameter. Sometimes this measure tends to be sensitive to the translation of the entire character, therefore we use a different measure that is invariant to translation. The root mean square (ERMS) error measure in Eq. 4 is an alternative way to express distortion with the difference that we use $\sqrt{3NP}$ in the denominator so as to obtain the average deformation per vertex and frame during the sequence. $3NP$ is the total number of elements in the $A_{orig}$ matrix. [21] uses as denominator the diameter of the bounding box multiplied by $\sqrt{NP}$.

$$\begin{aligned} ERMS= 100 \cdot \frac{\Vert A_{orig} - A_{Approx}\Vert _F}{\sqrt{3NP}} \end{aligned}$$

(4)

Max distance denotes the largest vertex error in every frame. So this measure represents the average of max distances over all frames.

$$\begin{aligned} MaxAvgDist = \frac{1}{P}\sum _{f=1}^{P}\max _{i=1,...,N}{\Vert v_{orig}^{f,i} - v_{Approx}^{f,i}\Vert } \end{aligned}$$

(5)

Finally, we introduce an additional measure that characterizes the normal distortion - (NormDistort) and is used to measure the different behavior of two animation sequences during rendering. We compute the average difference between the original and the approximated face normals by the norm of their cross product that equals to the sine of the angle between the two normal vectors. Therefore for a model with F faces and P frames, where $NV^{i,j}$ is the normal vector of face j at frame i, Eq. 6 computes the normal distortion measure.

$$\begin{aligned} NormDistort = sin^{-1}(\frac{1}{FP} \sum _{i=1}^{P}\sum _{j=1}^{F}{||NV^{i,j}_{orig} \times NV^{i,j}_{Approx}||}) \end{aligned}$$

(6)

B Appendix B

Fig. 3.

Training time for LSTM & CNN networks (Sect. 3.4).

Fig. 4.

LSTM-network.

The first network that we propose as the first step and mean of animation compression is a Recurrent Neural Network (RNN).

The type of RNN network used is a Long Short-term Memory network firstly introduced by [9] (LSTM), which consists of units made up of a cell remembering time inconstant data values, a corresponding forget cell, an input and an output gate being responsible of controlling the flow of data in and out of the remembering component of it Fig. 4. Thus, utilization of many network units for LSTM construction (120 units used) produces a network that is able yo predict weights even for models with a large number of bones. Regarding the activation functions we used (i) an alternative for the activation function (cell and hidden state) by using sigmoid instead of tanh and (ii) the default for the recurrent activation function (for input, forget and output gate) which is sigmoid. The main reason of using the sigmoid function instead of the hyperbolic tangent is that our training procedure involves the network deciding per vertex whether it belongs or not to the influence range of a bone. This results in higher efficacy and additionally makes our model learn more effectively.

Fig. 5.

CNN-network.

The second network that we have used successfully is a feed-forward network called Convolutional Neural Network (CNN) [15] that uses convolutional operations to capture patterns in order to determine classes mainly in image classification problems. CNNs are additionally able to be used in classification of sequence data with quite impressive results. On top of the two convolutional layers utilized, we have also introduced a global max-pooling layer (down-sampling layer) and a simple dense layer so that we have the desirable number of weights for each proxy bone, as it is illustrated in Fig. 5. In the two convolutional layers (Conv1D) used we utilize 8 filters of kernel size 2. The number of filters and kernel size have been determined experimentally. However, CNN with small kernel size is working efficiently and it is a reasonable network option on capturing animation sequences due to its capturing capabilities of almost minor transitions from one frame to the next one which is a consequence of small vertex movements within a two consecutive frames interval Fig. 6.

Fig. 6.

Convolutional kernel & strides representation given an animation sequence input. Blue is used to highlight the previous step of computations (convolutions of input data with filter) and red the next step. In this manner, a Conv1D layer is capable of capturing vertex trajectories in an animation sequence. (Color figure online)

Fig. 7.

Hybrid-network.

The last network that we have considered for completeness is a hybrid neural network (Fig. 7) that is a combination of the two aforementioned networks with some modifications. Unfortunately, the hybrid network does not perform equally well as its counterparts but it still derives comparable results.

$$\begin{aligned} L(y, y_{pred}) = -\frac{1}{N}\sum _{i=0}^{N} ( (1-y) \cdot log( 1-y_{pred} ) + y\cdot log(y_{pred}) ) \end{aligned}$$

(7)

where y are the real values (1: belongs to a bone or 0: does not) and $y_{pred}$ are the predicted values. Binary cross-entropy measures how far in average a prediction is from the real value for every class. To this end, we also used binary accuracy which calculates the percentage of matched prediction-label pairs the 0/1 threshold value set to 0.5. What we have inferred by these plots is that for CNN there is no reason to increase the batch-size higher than 4096 owing to the fact that accuracy and loss values tend to be almost identical after increasing batch-size from 2048 samples to 4096. Likewise, for the LSTM case (see Fig. 9) we observe that batch-size 2048 is the best option. From Figs. 8 and 9 we infer that we should use at least 20 epochs for training. After that the improvement of loss and accuracy is negligible but as we observed occasional overfitting is alleviated by increasing further the number of epochs.

Fig. 8.

Average per epoch Accuracy & Loss for CNN.

Fig. 9.

Average per epoch Accuracy & Loss for LSTM.

Fig. 10.

Error Metrics for batch-size tuning in CNN.

Fig. 11.

Error Metrics for batch-size tuning in LSTM.

C Appendix C

The entire method was developed¹ using Python and Tensorflow under the Blender 2.79b scripting API. The training part runs on a system with an NVIDIA GeForce RTX 2080Ti GPU with 11 GB GDDR6 RAM. We trained our network models with Adam Optimizer [16], $learning Rate=0.001$ for 20–100 epochs with $batchSize=4096$ over a training data-set that incorporates 60 animated character models of different size in terms of number of vertices, animations and frames per animation. We have inferred that 20 epochs are usually enough to have our method converging in terms of the error metrics and most importantly towards an acceptable visual outcome. However to obtain better RMS and distortion errors without over-fitting 100 epochs is a safe choice independently of the training set size. Furthermore, with this choice of batch-size we overcome the over-fitting problem that was apparent by observing the Max Average Distance metric and was manifested by locally distorted meshes.

The rest of our algorithm (prediction and optimization) was developed and ran on a commodity computer equipped with an Intel Core i7-4930K 3.4 GHz processor with 48 Gb under Windows 10 64-bit operating System. In addition, the FESAM algorithm was developed and ran on the same system.

Images for the experiments section.

Table 1. Comparative evaluation of our method versus Method I [11], Method II [13], Method III [14], Method IV [26], Method V [1].

Fig. 12.

Quantitative error results for animal characters.

Fig. 13.

Quantitative error results for human characters.

More specifically, presents a comparison of our method on four benchmark animation sequences, that were not produced by fully animated rigs, with all previous combinations of LBS, quaternion-based and SVD methods. N is the number of Vertices, F is the number of frames and the number in round brackets is the result of the method combined with SVD. Our method derives better results in terms of both error and compression rate as compared to methods I–IV. Method V is only cited for reference since it only obtains compression and is not compatible with any of the standard animation pipelines.

Table 2. Comparison between temporal deep skinning and four methods. Specifically method A [21], Method B [27], Method C [6], Method D [8].

Fig. 14.

Speed up of fitting time by using the conjugate gradient optimization method.

Fig. 15.

Qualitative error metric MaxAvgDistr results for humans and animals.

Fig. 16.

Visual comparison of Deep Skinning, FESAM-WT and the original frames for two models.Six frames have been selected in which structural flaws are marked by small circles.

Fig. 17.

Qualitative normal distortion metric results for humans and animals.

Fig. 18.

Distance error comparison in a particular frame between Deep Skinning and FESAM-WT.

Fig. 19.

Original and approximate representations.

In this case of Table 2 we cite the results from the papers since such methods are difficult to reproduce and this goes beyond the scope of this paper. For two models (horse gallop and samba) we have measured the ERMS error and the compression rate percentage (CRP). Note that the results of [21] were converted to our ERMS metric by multiplying by $\frac{D}{\sqrt{3}}$, where D is the diagonal of the bounding box of the rest pose.