State Constraints and Safety Consideration

Abstract

Controlling a real-world system with state constraints has drawn increasing attention due to practical needs, such as operating limits and safety guarantees. Equipping RL/ADP with the ability to handle constrained behaviors is of practical significance in both training process and controller implementation. Basically, there are three constrained RL/ADP methods, including penalty function method, Lagrange multiplier method, and feasible descent direction method. The phenomenon of infeasibility occurs when the constrained OCP has no solution due to overly tight state confinement, i.e., there is no available policy that can satisfy the strict constraint. Hence, handling constrained OCP is a systematic task, in which an optimal policy and its feasible working region must be simultaneously learned to ensure the recursive feasibility. A new three-element learning architecture called actor-critic-scenery (ACS) is proposed to address the issue, whose elements include policy improvement (PIM), policy evaluation (PEV), and a newly added region identification (RID) step. By equipping an OCP with hard state constraint, the safety guarantee is equivalent to solving this constrained control task to output its safe policy. Two basic training modes are proposed for safe policy search, and their corresponding safety-critical ACS algorithms can be designed in both model-free and model-based settings.