Generative Reward Machines for reinforcement learning in problems with poor observability
Loading...
Files
Full Text E-thesis
Date
2026-06-01
Authors
Rezaei, Saeid
Journal Title
Journal ISSN
Volume Title
Publisher
University College Cork
Published Version
Abstract
Reinforcement learning provides methodologies for sequential decision-making in autonomous systems. While reinforcement learning has achieved success in fully observable environments, real-world applications frequently suffer from partial observability, system delays, and non-stationary dynamics. For example, autonomous robots performing specific tasks and systems managing complex logistics networks often face these constraints. The dynamic and noisy nature of these practical applications causes standard reinforcement learning algorithms to fail.
The first question addressed in this research explores how specific environment conditions can render reinforcement learning problems fundamentally not solvable. While other research classifies problem complexity into easy, medium, and hard categories based merely on adjusting environment settings, this work takes a theoretical approach. It reviews specific conditions, demonstrating through mathematical proofs that optimal and epsilon-optimal policies cannot exist when environments feature changing reward functions and aleatoric uncertainty. By identifying the underlying issues that make a problem not solvable, this research explains why certain environments become truly hard for standard algorithms. This theoretical perspective acts as a guide to determine whether a decision-making problem can be solved or whether it requires specialized approaches.
To address these unsolvable problems and environments with obstructed state visibility, this research proposes the Generative Reward Machine framework. Unlike Reward Machines that rely on fully observable environments, the Generative Reward Machine utilizes an estimator function to transform noisy, partial observations into a stable state representation. This generative mechanism allows the system to establish stationary reward signals despite environmental uncertainty. By creating this intermediate state space, the system can develop policies even when the true state of the environment remains hidden.
The efficacy of the Generative Reward Machine framework is empirically validated across diverse domains. The first evaluations involve Reconnaissance Blind Chess, a game characterized by severe information constraints, where the framework adapted strategic hyperparameters to classify opponent skill levels. Further validations applied the framework to hyperparameter optimization for biometric footstep recognition and the Omnigenomic Benchmark. In these applications, the Generative Reward Machine provided a stable learning signal, enabling the framework to achieve top-ranking performance.
The final question addressed in this research evaluates whether the Generative Reward Machine can improve capacity management within Physical Internet distribution centers. To test this, the research introduces Gym-DC, a simulation environment that models physical packet routing under varying levels of stochasticity, including random delays and unreliable service providers. While standard deep reinforcement learning agents experience severe performance degradation in the hardest Gym-DC settings, the proposed Generative Reward Machine-driven agent maintains robustness. The framework outperforms standard reinforcement learning approaches and heuristic baselines. Ultimately, this research demonstrates that integrating generative state abstractions can improve performance in stochastic routing problems in Physical Internet.
Description
Keywords
Uncertainty quantification , Generative Reward Machine , Reconnaissance Blind Chess , Biometric footstep recognition , Reinforcement learning
Citation
Rezaei, S. 2026. Generative Reward Machines for reinforcement learning in problems with poor observability. PhD Thesis, University College Cork.