A feasibility study of elementary reinforcement learning-based process control

Simple item page
dc.contributor.advisorLouw, Tobias Mulleren_ZA
dc.contributor.advisorBradshaw, Steven Martinen_ZA
dc.contributor.authorBras, Edward Hendriken_ZA
dc.contributor.otherStellenbosch University. Faculty of Engineering. Dept. of Process Engineering.en_ZA
dc.date.accessioned2022-03-09T08:31:17Zen_ZA
dc.date.accessioned2022-04-29T09:35:55Zen_ZA
dc.date.available2022-03-09T08:31:17Zen_ZA
dc.date.available2022-04-29T09:35:55Zen_ZA
dc.date.issued2022-04en_ZA
dc.descriptionThesis (MEng)--Stellenbosch University, 2022.en_ZA
dc.description.abstractENGLISH SUMMARY: The classical control paradigm is widely used in industry, has well-understood theoretical guarantees, and forms part of the foundational knowledge of chemical engineers. Challenging non-linear dynamics prevent its successful application in certain cases, while classical controllers cannot automatically accommodate changing closed-loop dynamics. Advances in computational capabilities have led to a significant research interest in the application of Reinforcement Learning (RL) to control processes. In RL, a computational agent interacts with an environment to maximise the cumulative scalar rewards received. It may be viewed as an alternative paradigm for control, as is done in this thesis, or as an approach to potentially enhancing the performance of classical controllers. This simulation-based study’s purpose is to investigate the feasibility of elementary RL techniques to automatically determine the final element adjustments in a single-loop RL-based control scheme. It places into context what the strengths and limitations are of using elementary RL to control processes and highlights nuances of RL-based control without trying to outperform classical control. The control of a self-regulatory water tank model and the Van de Vusse reaction scheme model (used for benchmarking and requires advanced control solutions) were studied by applying two algorithms – Q-learning and SARSA – in a control scheme synthesized purely for theoretical study. Subsequently, these algorithms and the One-Step Actor-Critic algorithm were applied to the control of particle size in a qualitatively accurate grinding circuit model. All simulations leveraged the simplest possible RL design to allow interpretable and clear accounts of how these systems behave. The results show that the use of elementary RL techniques to obtain interpretable RL-based controllers for simulation-based study worked well for the water tank and Van de Vusse reaction scheme models. This was not the case for the grinding circuit case study. Replacing the classical control paradigm is not likely using elementary RL. Significant safety concerns arise since large amounts of operational data may be required and insufficient training in certain regions of the state-action space leads to unpredictable control behaviour. The strengths and weaknesses of the algorithms studied were investigated. It is unlikely that a reduction of control loop specific tuning parameters in comparison to classical control will be realised in practical control problems by applying RL-based control. Where applicable, classical control outperformed the elementary RL-based controllers which stresses that algorithmic adjustments are required, as is recognised in state-of-the-art RL-based control approaches. To conclude, the most practically feasible RL-based control solutions are likely to lie in the enhancement of existing control solutions by incorporating RL principles. The studied elementary RL-based control methods are not feasible for practical robust control. The control engineer must not be removed completely from the loop, and existing domain knowledge must be reconciled with computational thinking instead.en_ZA
dc.description.abstractAFRIKAANS OPSOMMING: Die klassieke beheerparadigma word algemeen gebruik in industrie, het welbegrepe teoretiese waarborge, en vorm deel van die fundamentele kennis van chemiese ingenieurswese. Uitdagende nie-liniêre dinamika voorkom sy suksesvolle toepassing in sekere gevalle, terwyl klassieke beheerders nie outomaties veranderende geslote-lus dinamika kan akkommodeer nie. Vooruitgang in rekenvermoë het gelei tot ’n beduidende navorsingbelangstelling in die toepassing van versterkingsleer (RL) om prosesse te beheer. In RL reageer ʼn rekenagent met ’n omgewing om die kumulatiewe skalaarbelongings ontvang, te maksimeer. Dit mag gesien word as ’n alternatiewe paradigma vir beheer, soos dit gedoen word in hierdie tesis, of as ’n benadering om die doeltreffendheid van klassieke beheerders potensieel te vergroot. Hierdie simulasiegebaseerde studie se doel is om die uitvoerbaarheid van elementêre RL-tegnieke te ondersoek om outomaties die finale elementwysigings in ’n enkellus RL-gebaseerde beheerskema te bepaal. Dit plaas wat die sterkpunte en beperkinge is van die gebruik van elementêre RL om prosesse te beheer, in konteks, beklemtoon nuanse van RL-gebaseerde beheer sonder om die klassieke beheer te probeer oortref. Die beheer van ’n self-regulerende watertenkmodel en die Van de Vusse-reaksieskemamodel (gebruik vir normstelling en vereis gevorderde beheeroplossings) is bestudeer deur twee algoritmes toe te pas – Q-leer en SARSA – in ’n beheerskema gesintetiseer uitsluitlik vir teoretiese studie. Vervolgens, is hierdie algoritmes en die “One-Step Actor-Critic”-algoritme toegepas op die beheer van partikelgrootte in ’n kwalitatiewe akkurate slypkringmodel. Alle simulasies het die eenvoudigste moontlike RL-ontwerp gebruik om interpreteerbare en duidelike rekenskap te gee van hoe hierdie sisteme hul gedra. Die resultate het getoon dat die gebruik van elementêre RL-tegnieke om interpreteerbare RL-gebaseerde beheerders te verkry vir simulasiegebaseerde studie goed gewerk het vir die watertenk- en Van der Vusse-reaksieskemamodelle. Dit was nie die geval vir die slypkringgevallestudie nie. Om die klassieke beheerparadigma te vervang deur elementêre RL te gebruik, is nie waarskynlik nie. Beduidende veiligheidskommer kom voor aangesien groot hoeveelhede bedryfsdata vereis mag word en onvoldoende opleiding in sekere streke van die staataksiespasie na onvoorspelbare beheergedrag lei. Die sterkpunte en swakpunte van die algoritmes bestudeer is ondersoek. Dit is onwaarskynlik dat ’n reduksie van beheerlus spesifieke instemmingparameters, in vergelyking met klassieke beheer, gerealiseer sal word in praktiese beheerprobleme deur RL-gebaseerde beheer toe te pas. Waar toepaslik, het die klassieke beheer die elementêre RL-gebaseerde beheerders oortref, wat beklemtoon dat die algoritmiese wysigings vereis word, soos dit erken word in hipermoderne RL-gebaseerde beheerbenaderings. Ter slotte, die mees praktiese uitvoerbare RL-gebaseerde beheeroplossings lê waarskynlik in die vergroting van bestaande beheeroplossings deur RL-beginsels te inkorporeer. Die bestudeerde RL-gebaseerde beheermetodes is nie uitvoerbaar vir praktiese robuste beheer nie. Die beheeringenieur moet nie heeltemal verwyder word uit die lus nie, en bestaande domeinkennis moet eerder met rekendenke versoen word.af_ZA
dc.description.versionMastersen_ZA
dc.format.extent189 pages : illustrationsen_ZA
dc.identifier.urihttp://hdl.handle.net/10019.1/124833en_ZA
dc.language.isoen_ZAen_ZA
dc.publisherStellenbosch : Stellenbosch Universityen_ZA
dc.rights.holderStellenbosch Universityen_ZA
dc.subjectNon-linear adaptive controlen_ZA
dc.subjectReinforcement learningen_ZA
dc.subjectComputational chemistryen_ZA
dc.subjectUCTDen_ZA
dc.titleA feasibility study of elementary reinforcement learning-based process controlen_ZA
dc.typeThesisen_ZA
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
bras_feasibility_2022.pdf
Size:
4.23 MB
Format:
Adobe Portable Document Format
Description:
Download
Collections
Masters Degrees (Chemical Engineering)