An Error-Based Addressing Architecture for Dynamic Model Learning

Abstract

We present a distributed supervised learning architecture, which can generate trajectory data conditioned by control commands and learned from demonstrations. The architecture consists of an ensemble of neural networks (NNs) which learns the dynamic model and a separate addressing NN that decides from which NN to draw a prediction. We introduce an error-based method for automatic assignment of data subsets to the ensemble NNs for training using the loss profile of the ensemble. Our code is publicly available (Code: https://github.com/NicoBach/distributed-dynamics-model).

N. Bach and A. Melnik—Equal contribution.

A Appendix - Performance of the Architecture in the Acrobot Environment

The prediction generated by the addressing NN and the losses computed between the output of the ensemble models and the target outputs correlated as designed, resulting in a $77.5\%$ overlap between addressing NN prediction and minimal loss. Figure 8 shows this correlation. The distribution of about 200,000 data points is approximately uniform, excluding $model\ 3$ and $model\ 7$. In theory, each model should perform on 25,000 data points, which is mostly the case. The addressing network performs slightly worse in terms of uniformity, having also $model\ 1$ as an additional outlier. The reasons for that are hypothesized in the discussion.

Fig. 8.

data points per model in terms of distribution by minimal loss and by prediction.

Fig. 9.

Mean losses of all data points per ensemble model (right column) and of the distributed data points per ensemble model (left column). Additionally, mean losses for three fully connected neural networks. (right: focus on the distributed losses)

Fig. 10.

Losses of the distributed data points.

In Fig. 11, we can observe the prediction of the addressing NN when distributing data points over models. On the x-axis the predicted values are divided into intervals of [0, ..., 0.1], [0.1, ..., 0.2] and so forth. The shown x-value represents the upper bound of the interval. The accumulated number of points in the corresponding interval is shown on the y-axis. Most of the data points get distributed between a value of 0.9 to 1.0 in all models.

Additionally, Fig. 10 shows the losses of the distributed data points. After the data points get distributed in clusters by the addressing NN, the chosen models perform a forward step on these data points and the L1-loss is computed. They again get subdivided into ten equally sized intervals, ranging from lowest to highest loss per model. As we can see, most data points fall into the first interval with the lowest upper bound.

Fig. 11.

Prediction values of the addressing neural network for data points for distributing them over the 8 models.

Next, we calculate the mean loss over the training data, as well as the mean loss over the clusters of training data provided by the sorting process for each ensemble network. For comparing the performance of the architecture to traditionally used feed-forward NNs, we also trained three fully connected models with three hidden layers of different sizes per model on the same training data. The training duration and optimizer settings were the same as for the architecture, but the activation function was changed to ReLu. Then the mean loss was computed for each of the three models over the whole training data. In Fig. 9, this can be observed. The performance of each ensemble model over the whole training data is worse than the fully connected models, although they exhibit a lower mean loss on each of the sorted training clusters. Furthermore, a low loss on the specific training cluster corresponds to a high loss on all training data.

In Fig. 9 we show, that the architecture is able to outperform basic NNs, which are trained on all training data points. Specialized models had a bigger mean loss over the complete training set. However, the mean loss over the clusters of training data provided by the sorting process for each subnetwork could be shown to be lower (see Fig. 9). The training process of the NN architecture could, therefore, be interpreted as more sample efficient as the commonly used training procedure because the subnetworks were trained on a lower number of data points due to the sorting mechanism. The here developed technique could also dynamically subdivide training data into coherent clusters, such that the subnetworks learned specific parts of the dynamics, which an environment can provide.