In-Context Reinforcement Learning: Survey on Non-Stationarity and Adaptation

In the evolving field of in-context reinforcement learning (ICRL), researchers A Run and Ziluo Ding have published a comprehensive survey focusing on its applications under non-stationary conditions. Their paper, titled "In-Context Reinforcement Learning under Non-Stationarity: A Survey," was submitted on July 1, 2026, and is accessible via arXiv.

Understanding In-Context Reinforcement Learning

ICRL has gained traction due to advancements in decision-pretrained transformers and retrieval-augmented agents. This approach enables models to infer latent task rules and enhance behaviors based on interaction context without requiring test-time parameter updates. The survey emphasizes the importance of trial-and-error evidence and how it influences learning within a contextual framework.

However, existing research primarily centers around pretraining objectives and theoretical mechanisms, often neglecting the challenges posed by non-stationary environments. The authors argue that in such contexts, the accumulated evidence may not align with the current task requirements, leading to potentially stale or misleading information.

Challenges in Non-Stationary Environments

One of the key issues in non-stationary ICRL is the necessity for policies to adapt while keeping deployed parameters fixed. As environments change, the reward specifications and observation channels may become misaligned. This misalignment can result in previously useful context becoming irrelevant or misleading, necessitating a dynamic understanding of which parts of the accumulated evidence still support current decision rules.

The authors categorize the literature on non-stationary ICRL by addressing three critical questions: what changes occur, how these changes unfold, and how observable these changes are to the agent. This structured approach sheds light on the intricacies of adapting to shifting environments.

Future Directions and Implications

Looking ahead, the survey suggests that further exploration into non-stationary ICRL could lead to more robust decision-making models capable of handling real-world complexities. The integration of methodologies from meta-reinforcement learning and retrieval-augmented frameworks may enhance the adaptability of ICRL systems.

As ICRL continues to evolve, understanding its implications for practical applications in various domains will be crucial. The survey encourages researchers to delve deeper into the realm of non-stationarity to unlock the full potential of reinforcement learning in complex, changing environments.