INNES: An intelligent network penetration testing model based on deep reinforcement learning

Abstract

Penetration testing (PT) is a crucial way to ensure the security of computer systems. However, it requires a high threshold and can only be implemented by trained experts. Automated tools can reduce the pressure of talent shortages, and reinforcement learning (RL) is a promising approach for achieving automated PT. Due to the unreasonable characterization of the PT process and the low efficiency of RL data, the applicability of the model is limited, and it is difficult to reuse, which hinders its practical application. In this paper, we propose an INNES (INtelligent peNEtration teSting) model based on deep reinforcement learning (DRL). First, the model characterizes the key elements of PT more reasonably based on the Markov decision process (MDP), fully considering the commonality of the PT process in different scenarios to improve its applicability. Second, the DQN_valid algorithm is designed to constrain the agent’s action space, to improve the agent’s decision-making accuracy, and avoid invalid exploration, according to the feature that enables the effective action space to gradually increase during the PT process. The experimental results show that our model is not only effective for automated PT in the network environment but also has portability, which provides a possible future direction for practical application of intelligent PT based on RL.

Appendix A Network scenario details

To verify the validity of our model, we conducted experiments on two parts of the scenarios. One part is the scenarios included in the currently open-source CyberBattleSim platform developed by Microsoft. Sample1 and Sample2 are randomly generated by the scenario automatic generation module contained by CyberBattleSim. The other part is the scenarios that we abstractly construct from the real network environment. In RL, the input and output of the agent, namely the scale of state space and action space, need to be defined before the neural network structure design. However, when applying RL to solve PT problems, it is nearly impossible to fully describe all the elements in the network scenarios. Therefore, we limit the size of the problem so that agents can be applied to all scenarios within that range. In this paper, the network scenarios limit of the PT tasks that the agents we build can perform is shown in Table 8. We design the network scenarios according to the scope given in the table.

Table 8 The elements included in our network scenarios

Fig. 8

Details about our network scenarios

Figure 8 shows the structure of each scenario built in this paper. In the figure, the start node is the starting point of the agent. It is not included in the network scenario, so each scenario contains 9 nodes. The lines between nodes represent the reachability between them. The black line A$\rightarrow$B indicates that the agent can discover node B after vulnerability exploitation at node A, and the red line A$\rightarrow$B indicates that the agent can discover the credential of node B after vulnerability exploitation at node A. The red nodes indicate the starting point of the penetration testing or the agent already owned nodes before testing began, and the green nodes indicate the penetration testing target. In node information, None means that the element does not exist on the node. None on the Start node means that this information does not participate in the agent’s decision. We set up a search operation on the Start node to discover other nodes.