Modares Mechanical Engineering

Modares Mechanical Engineering

Reinforcement learning control of four degree of freedom inverted pendulum

Authors
Morteza Khoshroo
Mojtaba Eftekhari
Mahdi Eftekhari
Department of Mechanical Engineering, Shahid Bahonar University of Kerman, Kerman, Iran
Abstract
In this paper, a robust linear quadratic regulator (LQR) based Reinforcement learning method is designed for a four degree of freedom inverted pendulum. The considered system contains a four degree of freedom inverted pendulum with a concentrated mass at the tip of it. The bottom of inverted pendulum is moved in x-y plane in x and y directions. For tracking control of two angles of inverted pendulum, two plane forces are applied in x and y directions at the bottom of pendulum. The governing equations of the system are derived using the Lagrange method and then a robust linear quadratic regulator (LQR) based Reinforcement learning controller is designed. The inverted pendulum is learned for a range of different angles, different lengths and different masses. The parametric uncertainties are defined as various lengths and masses of inverted pendulum and the disturbances are defined as impact and continuous forces which are applied on the inverted pendulum. After learning, the controller can learn online the system for any arbitrary angle, length, mass or disturbance which are not learned in the defined range. Numerical results show that the good performance of the reinforcement learning controller for the inverted pendulum in the presence of structural and parametric uncertainties, impact and continuous disturbances and sensor noises.
Keywords
Reinforcement learning
LQR controller
Four degree of freedom inverted pendulum
[1] A. Kokabi, M. Mahmoodghaznavi, Welding Technology, pp. 108-160, Tehran: Publishing institute of Sharif university, 2005. (in Persian فارسی(
[2] P. T. Houldcroft, ed, Submerged-arc Welding, pp. 9-18, Cambridge: Woodhead publishing, 1989.
[3] W. De Long, H. Heverly, E. Szumachowski, Bonded Arc Welding Flux and Liquid Binding Agent Therefor, U. S. Patent No. 3,769,099, 1973.
[4] M. Tokuhisa, Y. Hirai, Basic Bonded Fluxes for Submerged Arc Welding Having an Excellent Removability of Slag at a Narrow Groove, U. S. Patent No. 4,436, 562, 1984.
[5] G. Thewlis, Classification and quantification of microstructures in steels, Materials Science and Technology, Vol. 20, No. 2, pp. 143-160, 2004.
[6] M. Fattahi, N. Nabhani, M. Hosseini, N. Arabian, et al, Effect of Ticontaining inclusions on the nucleation of acicular ferrite and mechanical properties of multipass weld metals, Micron, Vol. 45, No. 1, pp. 107-114, 2013.
[7] B. HKDH, Mechanism of the transition from bainite to acicular ferrite, Materials Transactions, JIM, Vol. 32, No. 8, pp. 679-688, 1991.
[8] S. S. Babu, The mechanism of acicular ferrite in weld deposits, Current Opinion in Solid state and Materials Science, Vol. 8, No. 3, pp. 267-278, 2004.
[9] H. Mabuchi, R. Uemori, M. Fujioka, The role of Mn depletion in intragranular ferrite transformation in the heat affected zone of welded joints with large heat input in structural steels, ISIJ International, Vol. 36, No. 11, pp. 1406-1412, 1996.
[10] T. K. Lee, K. HJ, Effect of inclusion size on the nucleation of acicular ferrite in welds, ISIJ International, Vol. 40, No. 12, pp. 1260-1268, 2000.
[11] R. J. Davis, ed, Alloying: Understanding the Basics, pp. 123-132, Ohio: ASM international, 2001.
[12] M. Khazaee, A. Kokabi, effect of alloying addition on mechanical properties and microstructure of weld metal of St12 steel welded with self-shielded flux-cored arc welding, The 13th Scientific Student Conference On Metallurgical and Materials Engineering, Amirkabir university of technology, Tehran, Iran, 2016. (in Persian فارسی(
[13] S. D. Bhole, J. B. Nemade, L. Collins, C. Lui, Effect of nickel and molybdenum additions on weld metal toughness in a submerged arc welded HSLA line-pipe steel, Journal of Materials Processing Technology, Vol. 173, No. 1, pp. 92-100, 2006.
[14] K. Junhua, et al, Influence of Mo content on microstructure and mechanical properties of high strength pipeline steel, Materials & Design, Vol. 25, No. 8, pp. 723-728, 2004.
[15] J. C. F. Jorge, L. F. G. Souza, J. M. A. Rebello, The effect of chromium on the microstructure/toughness relationship of C–Mn weld metal deposits, Materials Characterization, Vol. 47, No. 3, pp. 195-205, 2001.
[16] W. W. Bose-Filho, A. L. M. Carvalho, M. Strangwood, Effects of alloying elements on the microstructure and inclusion formation in HSLA multipass welds, Materials Characterization, Vol. 58, No. 1, pp. 29-39, 2007.
[17] X. W. Chen, et al, Effects of Mo, Cr and Nb on microstructure and mechanical properties of heat affected zone for Nb-bearing X80 pipeline steels, Materials & Design, Vol. 53, pp. 888-901, 2014.
[18] I. Datta, M. Parekh, Filler metal flux basicity determination using the optical basicity index, Welding Journal, Vol. 68, No. 2, pp. 68-74, 1989.
[19] P. Krauklis, F. J. Barbaro, K. E. Easterling, Pro International Conference Martensitic Transformations, Monterey Institute for Advanced Studies, Monterey, pp. 439, 1992.
[20] B. Singh, Z. A. Khan, A. N. Siddiquee, Review on effect of flux composition on its behavior and bead geometry in submerged arc welding (SAW), Journal of Mechanical Engineering Research, Vol. 5, No. 7, pp. 123-127, 2013.
[21] L. F. G de Souza, et al, Microstructural analysis of a single pass 2.25% Cr– 1.0% Mo steel weld metal with different manganese contents, Materials Characterization, Vol. 55, No. 1, pp. 19-27, 2005.
[22] D. L. Olson, et al, Nature and Behavior of fluxes used for Welding, ASM International, ASM Handbook, Vol. 6, pp. 55-63, 1993.
Modares Mechanical Engineering
Volume 18, Issue 1
March 2018
Pages 388-396