Learning to steer nonlinear interior-point methods

Abstract

Interior-point or barrier methods handle nonlinear programs by sequentially solving barrier subprograms with a decreasing sequence of barrier parameters. The specific barrier update rule strongly influences the theoretical convergence properties as well as the practical efficiency. While many global and local convergence analyses consider a monotone update that decreases the barrier parameter for every approximately solved subprogram, computational studies show a superior performance of more adaptive strategies. In this paper we interpret the adaptive barrier update as a reinforcement learning task. A deep Q-learning agent is trained by both imitation and random action selection. Numerical results based on an implementation within the nonlinear programming solver WORHP show that the agent successfully learns to steer the barrier parameter and additionally improves WORHP’s performance on the CUTEst test set.

Appendix A: Additional numerical results

The appendix provides additional numerical results for the setups of Sect. 4.

Table 5 Overall training iterations and CPU time together with mean $\overline{{\hat{\varphi }}}$ and standard deviation $s({\hat{\varphi }})$ over the last 1000 episodes of five different training runs with parallel testing on the training and testing set with general nonlinear programming instances

Table 5 and Fig. 9 compare the performance of the deep Q-learning agent using the most successful configurations in Sect. 4.2, i.e., $\mathcal {O}_3$ and $\mathcal {O}_3$ without regret update, to the configurations $\mathcal {O}_1$ and $\mathcal {O}_1$ without regret update as the latter observation space also yields good performance in Sect. 4.1. The results, however, show that the observation space $\mathcal {O}_3$, i.e., the addition of the extrapolation approach of Sect. 2.2.3, provides better results in general.

Fig. 9

Performance measure ${\hat{\varphi }}$ for training (training set) and parallel testing (testing set) with general nonlinear programming instances. Considered configurations are $\mathcal {O}_1$, $\mathcal {O}_1$ without regret update, all trained by $\varepsilon$-greedy action selection. For further explanations, see Fig. 1

Table 6 Overall training iterations and CPU time together with mean $\overline{{\hat{\varphi }}}$ and standard deviation $s({\hat{\varphi }})$ over the last 1000 episodes of five different training runs with parallel testing on the training and testing set with general nonlinear programming instances

Fig. 10

Performance measure ${\hat{\varphi }}$ for training (training set) and parallel testing (testing set) with general nonlinear programming instances. Considered configurations are $\mathcal {O}_3$ with 10/20-, 25/25- and 50/25 training/testing sets, trained by $\varepsilon$-greedy action selection. For further explanations, see Fig. 1

If the number of nonlinear programming instances is increased in the training set, the learning task becomes more complex and may require more episodes. This effect is illustrated in Table 6 and Fig. 10 which consider in addition to the training and testing set of Sect. 4.2 one with 25 training and testing instances⁹ and one with 50 training and 25 testing instances¹⁰. The mean performance drops significantly for the larger training sets and in two replications even yield worse mean performance over the last 1000 episodes compared to the monotone Fiacco–McCormick update of Sect. 2.2.1. Note that in the current implementation running the testing set is just invoked if the training performance is above the baseline, which is why we get a steady testing performance for the first run of the 50/25 sets (cf. Fig. 10, right). The comparison of the learning performance between the different training and testing sets, however, is actually difficult, because the addition of a new nonlinear program may introduce higher complexity by its own, i.e., the new nonlinear program may be hard to solve and/or very sensitive to barrier parameter updates. Therefore, these results should be understood more like an outlook on what future research can address as they indicate that large training sets with very different nonlinear programs are currently still complicated and may need many more episodes to learn.

Table 7 Comparison of the barrier update strategies monotone Fiacco–McCormick (mono), LOQO rule (loqo) and extrapolation approach (quality) for IPOPT and WORHP with the reinforcement learning-based strategy (WORHP IP learned) on the training and testing set of Sect. 4.1

Table 8 Comparison of the barrier update strategies monotone Fiacco–McCormick (mono), LOQO rule (loqo) and extrapolation approach (quality) for IPOPT and WORHP with the reinforcement learning-based strategy (WORHP IP learned) on the training and testing set of Sect. 4.2

Finally, Tables 7 and 8 list the detailed numerical results of Sects. 4.1 and 4.2, respectively, for all considered barrier update strategies of IPOPT and WORHP. This includes the solver termination status, number of iterations, CPU time, objective function values and number of function evaluations for all nonlinear programming instances of the considered training and testing sets. For WORHP IP learned CPU time does not include the Python code, e.g., starting the Python interpreter within the CUTEst-C-interface of WORHP. The results given for WORHP IP learned have the global convergence framework enabled and list the number of switches to the monotone strategy.