Reinforcement Learning for Automated Flowsheet Synthesis of Steady-State Processes (2021-2024)

Flowsheet synthesis is a central element in the conceptual design of chemical processes. By nature this process is highly creative, and hard to formalize for automation. Prevalent methods of computer-aided flowsheet synthesis however are formalized algorithms using knowledge-bases rules for choosing between alternative planning steps, often stemming from superstructure optimization.

The aim of this research project is to use modern reinforcement learning (RL) methods for automated but creative flowsheet synthesis of steady-state chemical processes. Here, a learning agent sequentially generates process flowsheets by interacting with a process simulator, which provides feedback on the quality of the generated flowsheets based on a cost function. The agent has no prior knowledge of chemical engineering and learns exclusively through this automated interaction with the simulator. Only the process simulator contains the a priori available physical knowledge, i.e. physical-chemical material properties and a series of general apparatus models.

The central working hypothesis is that this approach is capable of independently generating feasible process flowsheets which are nearly optimal with respect to a given cost function.

The project is thus divided into two closely intertwined parts: On the one hand, robust and fast simulation environments for specific problems of interest must be developed and implemented. For this purpose, simplified shortcut and surrogate models for the apparatuses are first used (Burger Group, Chemical and Thermal Process Engineering, Technical University of Munich). On the other hand (Grimm group, Bioinformatics and Machine Learning, University of Applied Sciences Weihenstephan-Triesdorf and Technical University of Munich, Campus Straubing), new RL methods must be developed to overcome the combinatorial challenges of flowsheet synthesis: the planning horizon as well as the number of possible planning decisions (discrete and continuous) are extremely large. In addition, it is unclear a priori how to evaluate unfinished flowsheets amid such a sequential synthesis, which makes it difficult for the agent to learn.

In order to better address these challenges, the problem of flowsheet synthesis is transformed into a competitive two-player game. This leads to a generalized formulation and analysis as a combinatorial planning problem and allows the application of algorithms developed for complex highly combinatorial game applications (Chess, Go).

Project Information:

Project title: Reinforcement Learning for Automated Flowsheet Synthesis of Steady-State Processes

Project Partners

• Technical University of Munich, Chemical Process Engineering
  Project Coordinator: Prof. Dr. Jakob Burger
  Project Advisor: Quirin Göttl

• Weihenstephan-Triesdorf University of Applied Sciences & TUM Campus Straubing for Biotechnology and Sustainability
  Project Coordinator: Prof. Dr. Dominik Grimm
  Project Advisor: Jonathan Pirnay

Funding: The project is supported by funds of the German Research Foundation (DFG) .

Convex Envelope Method for determining liquid multi-phase equilibria in systems with arbitrary number of components.
Q Göttl, J Pirnay, DG Grimm, J Burger
Computers and Chemical Engineering, 2023
in press [Code]

Policy-Based Self-Competition for Planning Problems.
J Pirnay, Q Göttl, J Burger, DG Grimm
International Conference on Learning Representations (ICLR), 2023
(https://openreview.net/forum?id=SmufNDN90G) [Code]

Talk on Policy-Based Self-Competition for Planning Problems at the International Conference on Learning Representations (ICLR) 2023

Link to SlideShare Video at ICLR Website