Skip to main content

Advertisement

SpringerLink
Log in

DDPG-based continuous thickness and tension coupling control for the unsteady cold rolling process

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Cold rolling is an important part of the iron and steel industry, and the unsteady rolling process of cold rolling usually brings significant influences on the stability of product quality. In the unsteady rolling process, various disturbances and uncertainties such as variable lubrication state, variable equipment working conditions lead to difficulties in the establishment of state space model of thickness and tension, which has become a thorny problem in thickness and tension control. In this paper, we present a model-free controller based on Deep Deterministic Policy Gradient (DDPG), which can continuously control the thickness tension of the unsteady rolling process without the mathematical model. We first formulate the thickness and tension control problem to Markov Decision Process (MDP). We apply strategies such as dividing state space variables with the mechanism model, defining reward function and state normalization, the random disturbance and complex uncertainties of the unsteady cold rolling process are coped with by utilizing the DDPG controller. In addition, these strategies also ensure the learning performance and stability of DDPG controller under random disturbance. Simulations and experiments show that the proposed the DDPG controller does not require any prior knowledge of uncertain parameters and can operate without knowing unsteady rolling mathematical models, which has better accuracy, stability, and rapidity for thickness and tension in the unsteady rolling process than proportional integral (PI) controller. The artificial intelligence–based controller brings both product quality improvement and intelligence to cold rolling.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

The data are available on reasonable demand.

Code availability

The code is available on reasonable demand.

References

  1. Zhang X, Zhang Q, Sun C (2009) Gauge and tension control in unsteady state of cold rolling using mixed H2/H∞ control. In IEEE International Conference on Control and Automation Christchurch, New Zealand, pp 9–11

  2. Seung-Ho S, Seung-Ki S (2000) A new tension controller for continuous strip processing line. IEEE Trans Ind Appl 36:2. https://doi.org/10.1109/28.833782

  3. Wang Y, Xia J, Wang Z, Shen H (2020) Design of a fault-tolerant output-feedback controller for thickness control in cold rolling mills. Appl Math Comput. https://doi.org/10.1016/j.amc.2019.124841

    Article  MathSciNet  MATH  Google Scholar 

  4. Friebel T, Zabet K, Haber R, Jelali M (2015) Predictive functional control of tandem cold metal rolling. IEEE Conf Control Appl 324–329

  5. Li B, Fan X, Jiang C, Jiang G (2014) Decoupling control of thickness and tension based on DRNN-PID in cold-rolling. In Proceeding of the 11th World Congress on Intelligent Control and Automation, pp 1180–1184

  6. Tan S, Wang L, Liu J (2014) Research on decoupling method of thickness and tension control in rolling process. In Proceeding of the 11th World Congress on Intelligent Control and Automation Shenyang, pp. 4715–4717

  7. An S (2016) The decoupling control of tandem cold rolling tension and gauge. In 2016 3rd International Conference on Information Science and Control Engineering (ICISCE), pp. 1154–1158

  8. Hu Y-J, Sun J, Wang Q-L, Yin F-C, Zhang D-H (2018) Characteristic analysis and optimal control of the thickness and tension system on tandem cold rolling. Int J Adv Manuf Technol. https://doi.org/10.1007/s00170-018-3088-1

    Article  Google Scholar 

  9. Hu Y, Sun J, Chen SZ, Zhang X, Peng W, Zhang D (2019) Optimal control of tension and thickness for tandem cold rolling process based on receding horizon control. Ironmak Steelmak 1–11. https://doi.org/10.1080/03019233.2019.1615813

    Article  Google Scholar 

  10. Koofigar HR, Sheikholeslam F, Hosseinnia S (2011) Unified gauge-tension control in cold rolling mills: a robust regulation technique. Int J Precis Eng Manuf 12(3):393–403. https://doi.org/10.1007/s12541-011-0051-6

    Article  Google Scholar 

  11. Ogasahara T, Hovd M, Asano K (2016) Explicit model predictive controller design for thickness and tension control in a cold rolling mill. IFAC-PapersOnLine 49(20):126–131

    Article  Google Scholar 

  12. Hu Y, Sun J, Peng W, Zhang D (2021) Nash equilibrium-based distributed predictive control strategy for thickness and tension control on tandem cold rolling system. J Process Control 97:92–102. https://doi.org/10.1016/j.jprocont.2020.11.014

    Article  Google Scholar 

  13. Ozaki K, Ohtsuka T, Fujimoto K, Kitamura A, Nakayama M (2010) Nonlinear receding horizon control of thickness and tension in a tandem cold mill with a variable rolling speed. Tetsu-to-Hagane 96(7):459–467. https://doi.org/10.2355/tetsutohagane.96.459

    Article  Google Scholar 

  14. Cao L, Li X, Wang Q, Zhang D (2021) Vibration analysis and numerical simulation of rolling interface during cold rolling with unsteady lubrication. Tribol Int. https://doi.org/10.1016/j.triboint.2020.106604

    Article  Google Scholar 

  15. Sun B, He M, Wang Y, Gui W, Yang C, Zhu Q (2018) A data-driven optimal control approach for solution purification process. J Process Control 68:171–185. https://doi.org/10.1016/j.jprocont.2018.06.005

    Article  Google Scholar 

  16. Frikha MS, Gammar SM, Lahmadi A, Andrey L (2021) Reinforcement and deep reinforcement learning for wireless Internet of Things: a survey. Comput Commun 178:98–113. https://doi.org/10.1016/j.comcom.2021.07.014

    Article  Google Scholar 

  17. Viharos ZJ, Jakab R (2021) Reinforcement learning for statistical process control in manufacturing. Measurement. https://doi.org/10.1016/j.measurement.2021.109616

    Article  Google Scholar 

  18. Nian R, Liu J, Huang B (2020) A review on reinforcement learning: introduction and applications in industrial process control. Comput Chem Eng. https://doi.org/10.1016/j.compchemeng.2020.106886

    Article  Google Scholar 

  19. Du Y, Zandi H, Kotevska O, Kurte K, Munk J, Amasyali K, Makee E, Li F (2021) Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning. Appl Energy. https://doi.org/10.1016/j.apenergy.2020.11611

    Article  Google Scholar 

  20. Gu S, Ethan H, Timothy L, Sergey L (2017) Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), pp 3389–3396

  21. Liu L, Chen E, Gao Z, Wang Y (2019) Research on motion planning of seven degree of freedom manipulator based on DDPG. In Wang K., Wang Y., Strandhagen J., Yu T. (eds) Advanced Manufacturing and Automation VIII. IWAMA 2018. Lecture Notes in Electrical Engineering, vol 484. Springer, Singapore. https://doi.org/10.1007/978-981-13-2375-1_44

  22. Qiu C, Hu Y, Chen Y, Zeng B (2019) Deep deterministic policy gradient (DDPG)-based energy harvesting wireless communications. IEEE Internet Things J 6(5):8577–8588. https://doi.org/10.1109/jiot.2019.2921159

    Article  Google Scholar 

  23. Wang Y, Shen H, Duan D (2017) On stabilization of quantized sampled-data neural-network-based control systems. IEEE Transactions on Cybernetics 47(10):3124–3135. https://doi.org/10.1109/tcyb.2016.2581220

    Article  Google Scholar 

  24. Qi Z, Peng S, Honghai L, Shengyuan X (2012) Neural-network-based decentralized adaptive output-feedback control for large-scale stochastic nonlinear systems. IEEE Trans Syst Man Cybern Part B (Cybern) 42(6):1608–1619. https://doi.org/10.1109/tsmcb.2012.2196432

  25. Buşoniu L, de Bruin T, Tolić D, Kober J, Palunko I (2018) Reinforcement learning for control: performance, stability, and deep approximators. Annu Rev Control 46:8–28. https://doi.org/10.1016/j.arcontrol.2018.09.005

    Article  MathSciNet  Google Scholar 

  26. Gao G, Li J, Wen Y (2020) DeepComfort: energy-efficient thermal comfort control in buildings via reinforcement learning. IEEE Internet Things J 7(9):8472–8484. https://doi.org/10.1109/jiot.2020.2992117

    Article  Google Scholar 

  27. Ma Y, Zhu W, Benton MG, Romagnoli J (2019) Continuous control of a polymerization system with deep reinforcement learning. J Process Control 75:40–47. https://doi.org/10.1016/j.jprocont.2018.11.004

    Article  Google Scholar 

  28. Siraskar R (2021) Reinforcement learning for control of valves. Mach Learn Appl. https://doi.org/10.1016/j.mlwa.2021.100030

    Article  Google Scholar 

  29. Spielberg S, Gopaluni RB, Loewen PD (2017) Deep reinforcement learning approaches for process control. In 2017 6th International Symposium on Advanced Control of Industrial Processes (AdCONIP), pp: 28–31

  30. Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou L, Wierstra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. arXiv:1312.5602

  31. Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D (2015) Continuous control with deep reinforcement learning. arXiv:1509.02971

  32. Sutton RS, McAllester D, Singh S, Mansour Y (2000) Policy gradient methods for reinforcement learning with function approximation. Adv Neural Inf Process Syst 1057–1063

  33. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiler M, Fidjeland AK, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533. https://doi.org/10.1038/nature14236

    Article  Google Scholar 

  34. Fang X, Han Y, Wang J, Zhao Q (2019) A cognitive control approach for microgrid performance optimization in unstable wireless communication. Neurocomputing 355:168–182. https://doi.org/10.1016/j.neucom.2019.04.048

    Article  Google Scholar 

  35. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, Levenberg J, Monga R, Morre S, GMurray, D, G., Steiner, B., Tucker, P., (2016) Tensorflow: a system for large-scale machine learning. OSDI 16:265–283

    Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China (U21A20475,U1908213), Colleges and Universities in Hebei Province Science Research Program (QN2020504), The Fundamental Research Funds for the Central Universities (N2223001).

Author information

Authors and Affiliations

  1. College of Information Science and Engineering, Northeastern University, Shenyang, 110819, People’s Republic of China

    Wenying Zeng & Jinkuan Wang

  2. School of Control Engineering, Northeastern University at Qinhuangdao, Qinhuangdao, 066004, People’s Republic of China

    Yan Zhang, Yinghua Han & Qiang Zhao

Authors
  1. Wenying Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinkuan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yinghua Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qiang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Wenying Zeng: conceptualization, methodology, investigation, data curation, software, formal analysis, experiment, and writing of the manuscript. Jinkuan Wang: conceptualization, resources, funding acquisition, supervision, project administration, and review. Yan Zhang: data collection and curation, writing review and editing. Yinghua Han: methodology, supervision, and writing including review and editing. Qiang Zhao: methodology, supervision, and writing including review and editing.

Corresponding author

Correspondence to Jinkuan Wang.

Ethics declarations

Ethics approval and consent to participate

All authors understand and approve the ethical responsibilities of the authors. The authors consent to participate.

Consent for publication

The authors consent to transfer the copyright of the article to publish.

Conflict of interest

All authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zeng, W., Wang, J., Zhang, Y. et al. DDPG-based continuous thickness and tension coupling control for the unsteady cold rolling process. Int J Adv Manuf Technol 120, 7277–7292 (2022). https://doi.org/10.1007/s00170-022-09239-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-022-09239-4

Keywords

Search

Navigation