Abstract
Optimization problems are widely used in many real-world applications. These problems are rarely unconstrained and are usually considered constrained optimization problems. Regarding the number of objectives, the optimization problems can be categorized into single- (for one), multi- (usually for two and three), and many- (more than three) objective optimization problems. In this paper, an Any-Objective Optimization (AOO) framework is introduced based on Deep Reinforcement Learning (DRL) models. The term any-objective optimization is coined to indicate the generalized structure of the proposed algorithm that regardless of the number of objectives, can solve the constrained optimization problems with any number of objectives. To trade off the multiple conflicting objectives, RL algorithms can be extended to a framework called Multi-Objective Reinforcement Learning (MORL). By converting a constrained optimization problem into an environment that can be explored by the MORL and deep learning algorithms, any constrained optimization problem can be tackled. In this research, to solve a constrained optimization problem with any number of objective functions, a novel reward function is introduced, and the algorithm begins a heuristic search in the environment to find the optimal solution(s) and generates an archive of the optimal Pareto front solution. The corresponding environment is constructed modular, such that any RL algorithm with arbitrary reward function types (scalar or vector) can be utilized. To evaluate the proposed algorithm, some popular test function-defined constrained optimization problems with continuous variable and objective spaces as illustrative examples are considered, and five of the widely used DRL algorithms are implemented to test the case studies. To demonstrate the capabilities of the proposed algorithm, the obtained results are compared with structurally similar GA-based well-known existing single-, multi-, and many-objective optimization algorithms, respectively. The results show that the proposed framework can be a well-performing baseline for a new type of DRL-based optimization algorithm.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
Due to privacy and ethical concerns, neither the data nor the source of the data can be made available.
References
Alarcon-Rodriguez A, Ault G, Galloway S (2010) Multi-objective planning of distributed energy resources: a review of the state-of-the-art. Renew Sustain Energy Rev 14:1353–1366
Blank J, Deb K (2020) Pymoo: multi-objective optimization in python. IEEE Access 8:89497–89509. https://doi.org/10.1109/ACCESS.2020.2990567
Cai D, Yuping W (2015) A new uniform evolutionary algorithm based on decomposition and CDAS for many-objective optimization. Knowl Based Syst 85:131–142. https://doi.org/10.1016/J.KNOSYS.2015.04.025
Campos Ciro G, Dugardin F, Yalaoui F, Kelly R (2016) A NSGA-II and NSGA-III comparison for solving an open shop scheduling problem with resource constraints. IFAC-PapersOnLine 49:1272–1277. https://doi.org/10.1016/J.IFACOL.2016.07.690
Chen X, Ghadirzadeh A, Bjorkman M, Jensfelt P (2019) Meta-Learning for Multi-objective Reinforcement Learning. IEEE International Conference on Intelligent Robots and Systems 977–983. https://doi.org/10.1109/IROS40897.2019.8968092
Cheng R, Jin Y, Olhofer M, Sendhoff B (2016) A reference vector guided evolutionary algorithm for many-objective optimization. IEEE Trans Evol Comput 20:773–791. https://doi.org/10.1109/TEVC.2016.2519378
Cheng T, Chen M, Fleming PJ et al (2017) A novel hybrid teaching learning based multi-objective particle swarm optimization. Neurocomputing 222:11–25. https://doi.org/10.1016/J.NEUCOM.2016.10.001
Coello Coello CA, Lechuga MS (2002) MOPSO: a proposal for multiple objective particle swarm optimization. Proceedings of the 2002 Congress on Evolutionary Computation, CEC 2002 2:1051–1056. https://doi.org/10.1109/CEC.2002.1004388
Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput 6:182–197. https://doi.org/10.1109/4235.996017
Fadaee M, Radzi MAM (2012) Multi-objective optimization of a stand-alone hybrid renewable energy system by using evolutionary algorithms: a review. Renew Sustain Energy Rev 16:3364–3369. https://doi.org/10.1016/J.RSER.2012.02.071
Ferreira JC, Steiner MTA, Canciglieri Junior O (2020) Multi-objective optimization for the green vehicle routing problem: A systematic literature review and future directions. Cogent Eng 7:1807082. https://doi.org/10.1080/23311916.2020.1807082
Fujimoto S, van Hoof H, Meger D (2018) Addressing Function Approximation Error in Actor-Critic Methods. 35th International Conference on Machine Learning, ICML 2018 4:2587–2601. https://doi.org/10.48550/arxiv.1802.09477
Gong YJ, Chen WN, Zhan ZH et al (2015) Distributed evolutionary algorithms and their models: a survey of the state-of-the-art. Appl Soft Comput 34:286–300. https://doi.org/10.1016/J.ASOC.2015.04.061
Gong D, Sun F, Sun J, Sun X (2017) Set-based many-objective optimization guided by a preferred region. Neurocomputing 228:241–255. https://doi.org/10.1016/J.NEUCOM.2016.09.081
Gunantara N (2018) A review of multi-objective optimization: methods and its applications. Cogent Eng 5:1502242. https://doi.org/10.1080/23311916.2018.1502242
Haarnoja T, Zhou A, Abbeel P, Levine S (2018a) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. 35th International Conference on Machine Learning, ICML. 5: 2976–2989. https://doi.org/10.48550/arxiv.1801.01290
Haarnoja T, Zhou A, Hartikainen K, et al (2018b) Soft actor-critic algorithms and applications. https://doi.org/10.48550/arxiv.1812.05905
Hiroyasu T, Nakayama S, Miki M (2005) Comparison study of SPEA2+, SPEA2, and NSGA-II in diesel engine emissions and fuel economy problem. 2005 IEEE Congress on Evolutionary Computation, IEEE CEC 2005 Proceedings 1: 236–242. https://doi.org/10.1109/CEC.2005.1554690
Hojjati A, Monadi M, Faridhosseini A, Mohammadi M (2018) Application and comparison of NSGA-II and MOPSO in multi-objective optimization of water resources systems. J Hydrol Hydromech 66:323–329. https://doi.org/10.2478/JOHH-2018-0006
Ishibuchi H, Imada R, Setoguchi Y, Nojima Y (2016) Performance comparison of NSGA-II and NSGA-III on various many-objective test problems. 2016 IEEE congress on evolutionary computation. CEC 2016:3045–3052. https://doi.org/10.1109/CEC.2016.7744174
Ishibuchi H, Masuda H, Tanigaki Y, Nojima Y (2015) Modified distance calculation in generational distance and inverted generational distance. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 9019:110–125. https://doi.org/10.1007/978-3-319-15892-1_8/COVER
Jones DF, Mirrazavi SK, Tamiz M (2002) Multi-objective meta-heuristics: an overview of the current state-of-the-art. Eur J Oper Res 137:1–9. https://doi.org/10.1016/S0377-2217(01)00123-0
Joshi M, Ghadai RK, Madhu S et al (2021) Comparison of NSGA-II MOALO and MODA for multi-objective optimization of micro-machining processes. Materials (basel). https://doi.org/10.3390/MA14175109
King RTFA, Deb K, Rughooputh HCS (2016) Comparison of NSGA-II and SPEA2 on the multiobjective environmental/economic dispatch problem. Univ Mauritius Res J 16:485–511. https://doi.org/10.4314/umrj.v16i1
Li K, Zhang T, Wang R et al (2021) Deep reinforcement learning for combinatorial optimization: covering salesman problems. IEEE Trans Cybern. https://doi.org/10.1109/TCYB.2021.3103811
Lillicrap TP, Hunt JJ, Pritzel A et al (2015) Continuous control with deep reinforcement learning. 4th International Conference on learning representations, ICLR 2016-Conference Track Proceedings. https://doi.org/10.48550/arxiv.1509.02971
Liu C, Du Y (2019) A membrane algorithm based on chemical reaction optimization for many-objective optimization problems. Knowl Based Syst 165:306–320. https://doi.org/10.1016/J.KNOSYS.2018.12.001
Liu C, Xu X, Hu D (2015) Multiobjective reinforcement learning: a comprehensive overview. IEEE Trans Syst Man Cybern Syst 45:385–398. https://doi.org/10.1109/TSMC.2014.2358639
Liu H, Li Y, Duan Z, Chen C (2020a) A review on multi-objective optimization framework in wind energy forecasting techniques and applications. Energy Convers Manag. https://doi.org/10.1016/J.ENCONMAN.2020.113324
Liu Q, Li X, Liu H, Guo Z (2020b) Multi-objective metaheuristics for discrete optimization problems: a review of the state-of-the-art. Appl Soft Comput 93:106382. https://doi.org/10.1016/J.ASOC.2020.106382
Liu S, Yu Q, Lin Q, Tan KC (2020c) An adaptive clustering-based evolutionary algorithm for many-objective optimization problems. Inf Sci 537:261–283. https://doi.org/10.1016/J.INS.2020.03.104
Mnih V, Kavukcuoglu K, Silver D et al (2015) Human-level control through deep reinforcement learning. Nature 518:7540. https://doi.org/10.1038/nature14236
Mnih V, Kavukcuoglu K, Silver D, et al (2013) Playing Atari with Deep Reinforcement Learning. https://doi.org/10.48550/arxiv.1312.5602
Mohammadi M, Khodaygan S (2020) An algorithm for numerical nonlinear optimization: fertile field algorithm (FFA). J Ambient Intell Humaniz Comput 11:865–878. https://doi.org/10.1007/S12652-019-01598-3/TABLES/12
Nguyen TT, Nguyen ND, Vamplew P et al (2020) A multi-objective deep reinforcement learning framework. Eng Appl Artif Intell 96:103915. https://doi.org/10.1016/J.ENGAPPAI.2020.103915
Niu B, Wang H, Wang J, Tan L (2013) Multi-objective bacterial foraging optimization. Neurocomputing 116:336–345. https://doi.org/10.1016/J.NEUCOM.2012.01.044
Pang LM, Ishibuchi H, Shang K (2020) NSGA-II with simple modification works well on a wide variety of many-objective problems. IEEE Access 8:190240–190250. https://doi.org/10.1109/ACCESS.2020.3032240
Parisi S, Pirotta M, Restelli M (2016) Multi-objective reinforcement learning through continuous pareto manifold approximation. J Artif Intell Res 57(187):227. https://doi.org/10.1613/JAIR.4961
Sallam KM, Elsayed SM, Chakrabortty RK, Ryan MJ (2020) Improved Multi-operator Differential Evolution Algorithm for Solving Unconstrained Problems. 2020 IEEE Congress on Evolutionary Computation, CEC 2020 - Conference Proceedings. https://doi.org/10.1109/CEC48606.2020.9185577
Shao Y, Lin JCW, Srivastava G et al (2021) Multi-Objective Neural Evolutionary Algorithm for Combinatorial Optimization Problems. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2021.3105937
Shinde SS, Thangavelu S, Jeyakumar G (2019a) Evolutionary computing approaches for solving multi-objective and many-objective optimization problems: A review. Proceedings - 2019a 5th International Conference on Computing, Communication Control and Automation, ICCUBEA 2019a. https://doi.org/10.1109/ICCUBEA47591.2019.9129081
Shinde SS, Thangavelu S, Jeyakumar G (2019b) Evolutionary computing approaches for solving multi-objective and many-objective optimization problems: A review. Proceedings - 2019b 5th International Conference on Computing, Communication Control and Automation, ICCUBEA 2019b. https://doi.org/10.1109/ICCUBEA47591.2019.9129081
Sombolestan SM, Rasooli A, Khodaygan S (2019) Optimal path-planning for mobile robots to find a hidden target in an unknown environment based on machine learning. J Ambient Intell Humaniz Comput 10:1841–1850. https://doi.org/10.1007/S12652-018-0777-4/TABLES/4
Srinivasan S, Ramakrishnan S (2011) Evolutionary multi objective optimization for rule mining: a review. Artif Intell Rev 2011 36:3 36:205–248. https://doi.org/10.1007/S10462-011-9212-3
Sun Y, Zhang C, Gao L, Wang X (2010) Multi-objective optimization algorithms for flow shop scheduling problem: a review and prospects. Int J Adv Manuf Technol 55:723–739. https://doi.org/10.1007/S00170-010-3094-4
Sutton RS, Barto AG (2018) Reinforcement Learning: An Introduction, Second Edition. The MIT Press
Talaat FM, Saraya MS, Saleh AI et al (2020) A load balancing and optimization strategy (LBOS) using reinforcement learning in fog computing environment. J Ambient Intell Humaniz Comput 11:4951–4966. https://doi.org/10.1007/S12652-020-01768-8/FIGURES/7
Tian Y, Cheng R, Zhang X, Jin Y (2017) PlatEMO: a MATLAB platform for evolutionary multi-objective optimization [Educational Forum]. IEEE Comput Intell Mag 12:73–87. https://doi.org/10.1109/MCI.2017.2742868
Vamplew P, Yearwood J, Dazeley R, Berry A (2008) On the limitations of scalarisation for multi-objective reinforcement learning of pareto fronts. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5360 LNAI: 372–378. https://doi.org/10.1007/978-3-540-89378-3_37/COVER
Vamplew P, Dazeley R, Berry A, et al (2010) Empirical evaluation methods for multiobjective reinforcement learning algorithms. Mach Learn 84: 1 84:51–80. https://doi.org/10.1007/S10994-010-5232-5
van Moffaert K, Nowé A (2014) Multi-objective reinforcement learning using sets of pareto dominating policies. J Mach Learn Res 15:3483–3512. https://doi.org/10.5555/2627435
van Veldhuizen DA (1999) Multiobjective Evolutionary Algorithms: classifications, analyses, and new innovations. Storming Media
Vesikar Y, Deb K, Blank J (2019) Reference point based NSGA-III for preferred solutions. Proceedings of the 2018 IEEE Symposium Series on Computational Intelligence, SSCI 2018 1587–1594. https://doi.org/10.1109/SSCI.2018.8628819
Wang Q, Tang C (2021) Deep reinforcement learning for transportation network combinatorial optimization: a survey. Knowl Based Syst 233:107526. https://doi.org/10.1016/J.KNOSYS.2021.107526
Zhang Q, Li H (2007) MOEA/D: a multiobjective evolutionary algorithm based on decomposition. IEEE Trans Evol Comput 11:712–731. https://doi.org/10.1109/TEVC.2007.892759
Zhang Y, Bai R, Qu R et al (2022) A deep reinforcement learning based hyper-heuristic for combinatorial optimisation with uncertainties. Eur J Oper Res 300:418–427. https://doi.org/10.1016/J.EJOR.2021.10.032
Zhao C, Zhou Y, Chen Z (2021) Decomposition-based evolutionary algorithm with automatic estimation to handle many-objective optimization problem. Inf Sci 546:1030–1046. https://doi.org/10.1016/J.INS.2020.08.084
Zhou J, Zou J, Yang S et al (2021) Niche-based and angle-based selection strategies for many-objective evolutionary optimization. Inf Sci 571:133–153. https://doi.org/10.1016/J.INS.2021.04.050
Zou F, Yen GG, Tang L, Wang C (2021) A reinforcement learning approach for dynamic multi-objective optimization. Inf Sci 546:815–834. https://doi.org/10.1016/J.INS.2020.08.101
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Honari, H., Khodaygan, S. Deep reinforcement learning-based framework for constrained any-objective optimization. J Ambient Intell Human Comput 14, 9575–9591 (2023). https://doi.org/10.1007/s12652-023-04630-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-023-04630-9