KIcker: An Industrial Drive and Control Foosball System automated with Deep Reinforcement Learning

Abstract

The majority of efforts in the field of sim-to-real Deep Reinforcement Learning focus on robot manipulators, which is justified by their importance for modern production plants. However, there are only a few studies for a more extensive use in manufacturing processes. In this paper, we contribute to this by automating a complex manufacturing-like process using simulation-based Deep Reinforcement Learning. The setup and workflow presented here are designed to mimic the characteristics of real manufacturing processes and proves that Deep Reinforcement Learning can be applied to physical systems built from industrial drive and control components by transferring policies learned in simulation to the real machine. Aided by domain randomization, training in a virtual environment is crucial due to the benefit of accelerated training speed and the desire for safe Reinforcement Learning. Our key contribution is to demonstrate the applicability of simulation-based Deep Reinforcement Learning in industrial automation technology. We introduce an industrial drive and control system, based on the classic pub game Foosball, from both an engineering and a simulation perspective, describing the strategies applied to increase transfer robustness. Our approach allowed us to train a self-learning agent to independently learn successful control policies for demanding Foosball tasks based on sparse reward signals. The promising results prove that state-of-the-art Deep Reinforcement Learning algorithms are able to produce models trained in simulation, which can successfully control industrial use cases without using the actual system for training beforehand.

Appendix: Derivation of a Simple Policy Gradient Loss

Starting from Eq. 2, we introduce a short notation for the cumulative discounted reward obtained for a given trajectory: $R(\tau ) = {\sum }_{k=0}^{\infty } \gamma ^{k} r(s_{k},a_{k})$. This allows us to easily derive a basic policy gradient algorithm using the following loss function which we want to optimize:

$$ L(\theta) = \underset{\tau\sim\pi_{\theta}}{\mathbb{E}} R(\tau). $$

(13)

To expand the expectation value, one integrates the cumulative returns over all trajectories weighted by the probability for that trajectory:

$$ L(\theta) = {\int}_{\tau} p_{\theta}(\tau) R(\tau) d\tau. $$

(14)

Since only the trajectory-probability depends on 𝜃, we can take the derivative of J(𝜃) with respect to 𝜃. By using a simple log-derivative identity, this can be reformulated:

$$ \begin{array}{@{}rcl@{}} \nabla_{\theta} L(\theta) & =& {\int}_{\tau} \nabla_{\theta} p_{\theta}(\tau) R(\tau) d\tau \\ & =& {\int}_{\tau} p_{\theta}(\tau) \nabla_{\theta} \log \ p_{\theta}(\tau) R(\tau) d\tau \\ & =& \underset{\tau\sim\pi_{\theta}}{\mathbb{E}} \nabla_{\theta} \log \ p_{\theta}(\tau) R(\tau) \end{array} $$

(15)

The term $\nabla _{\theta } \log \ p_{\theta }(\tau )$ is evaluated by decomposing p_𝜃(τ):

$$ \begin{array}{@{}rcl@{}} \nabla_{\theta} \log \ p_{\theta}(\tau) &=& \nabla_{\theta} \log \left( s_{0} \sum\limits_{k=0}^{\infty} T(s_{k+1}|s_{k},a_{k}) \pi(a_{k}|s_{k})\right) \\ &=& \nabla_{\theta} s_{0} + \sum\limits_{k=0}^{\infty} \nabla_{\theta} T(s_{k+1}|s_{k},a_{k}) + \\ && {} \sum\limits_{k=0}^{\infty} \nabla_{\theta} \pi(a_{k}|s_{k}) \\ &=& \sum\limits_{k=0}^{\infty} \nabla_{\theta} \pi(a_{k}|s_{k}) \end{array} $$

(16)

So, overall the gradient of the loss function can be formulated as:

$$ \nabla_{\theta} L(\theta) = \underset{\tau\sim\pi_{\theta}}{\mathbb{E}} \left( \sum\limits_{k=0}^{\infty} \nabla_{\theta} \pi(a_{k}|s_{k})\right) \left( \sum\limits_{k=0}^{\infty} \gamma^{k} r(s_{k},a_{k})\right) $$

(17)

This expression is very similar to a standard maximum likelihood gradient used in supervised learning. The only difference is the last term coming from the cumulative rewards. It acts as a weight term for the gradient likelihood terms. With this, policy updates

$$ \theta^{\prime} \leftarrow \theta + \alpha \nabla_{\theta} L(\theta) $$

(18)

can be performed iteratively.