Recently we showed that reinforcement learning can be used to automatically generate process flowsheets without heuristics or prior knowledge. For this purpose, SynGameZero, a novel two-player game has been developed. In this work we extend SynGameZero by structuring the agent’s actions in several hierarchy levels, which improves the approach in terms of scalability and allows the consideration of more sophisticated flowsheet problems. We successfully demonstrate the usability of our novel framework for the fully automated synthesis of an ethyl tert-butyl ether process.