1 Introduction↩︎

In-context reinforcement learning (ICRL) asks whether a pretrained or fine-tuned sequential decision model can infer task rules and improve future actions from the interaction history available in its context, without test-time parameter updates [1]. The context may contain states, actions, rewards, transition outcomes, demonstrations, feedback, retrieved memories, or summaries, but the defining property is learning-like adaptation through that context rather than through an online optimizer [2], [3]. That idea runs through black-box meta-reinforcement learning [1], [4], gradient-based meta-learning [5], offline trajectory modeling [6], decision transformers [7], algorithm distillation [2], supervised ICRL [8], and long-context meta-RL [9], [10]. It also inherits a broader question from in-context learning: when does a sequence model merely condition on examples [11], [12], and when does its forward pass implement something close to inference [13]–[15], optimization [16], [17], or learning [18], [19]? This capability is especially appealing in reinforcement learning. If useful adaptation can happen through the input stream, then a system can respond quickly without running an optimizer, changing deployed weights, or hiding the adaptation process inside an online update.

That promise is much easier to state in stationary settings than in changing ones. Recent survey work already organizes ICRL broadly around pretraining, context construction, architectures, evaluation, and theory [20]. Here we focus on the case where the environment changes during the lifetime over which context is accumulated. By non-stationarity, we mean temporal changes in the decision process that alter which accumulated evidence remains valid for action. In a stationary few-shot task, more context is usually more evidence about one latent task. Under non-stationarity, the same extra trajectory can be useful evidence, obsolete evidence, a misleading demonstration, or a memory that becomes useful again only after a previous regime returns. The question is no longer simply whether the agent uses context. It is whether the agent can tell which parts of its context still describe the world it is acting in.

For ICRL, non-stationarity is best viewed as a change in the validity conditions of context: past trajectories, rewards, feedback, demonstrations, or retrieved memories may no longer license the same policy choice. The underlying source can be a changed reward specification [21], [22], transition kernel [23], [24], observation channel [25], [26], action interface [27], [28], preference or safety model [22], [29], task distribution [30], [31], or behavior policy behind the data [32], [33]. These sources impose different adaptation burdens. Reward switches require reinterpretation of feedback, transition drift changes action–outcome evidence, action remapping changes the meaning of behavior, and behavior-policy shift changes the trustworthiness of demonstrations or retrieved trajectories. An abrupt reward switch creates a different adaptation problem [34], [35] from gradual transition drift [23], [36]; a recurring latent mode creates a different memory problem [37], [38] from one-way degradation [39], [40]; an observed task descriptor creates a different inference problem [37], [41] from a hidden change point [34], [42]. For ICRL, the relevant consequence is that context has a lifetime. It can be useful when it identifies the current mode, harmful when it anchors the agent to a past mode, and useful again when an old mode recurs. This framing draws on older Markov decision process [43], partially observable decision process [44], contextual MDP [37], hidden-parameter MDP [38], [45], and epistemic-POMDP [46] formalisms, but shifts the emphasis from solving a known model class to managing evidence inside a fixed context window.

We use Non-Stationary ICRL for fixed-parameter, context-conditioned sequential decision making in which the decision process changes during the policy’s context-bearing lifetime. The deployed policy must use recent and stored interaction history to detect regime evidence, revise its latent task belief, and change its action distribution without test-time parameter updates. The central difficulty is not only selecting the policy appropriate for the current regime, but also deciding which previous rewards, transitions, demonstrations, feedback, or memories still provide valid evidence for that choice. It covers algorithm distillation [2] and decision-pretrained transformers [47]–[49] when interaction history drives adaptation; supervised ICRL [8], [50] and offline-pretrained agents [51], [52] when task examples, rewards, or trajectories identify the current regime; retrieval-augmented ICRL [33], [53] and retrieval-based RL [54], [55] when memory selection decides what evidence the model sees; and value-aware agents [56], [57] or model-aware agents [58], [59] when values or dynamics are inferred from context. It leaves out standard offline RL policies that do not adapt to new test-time evidence, deployment-time fine-tuning, and domain-specific robotics or multi-agent surveys unless their mechanisms clarify the fixed-parameter context problem. We treat LLM agents and broader foundation-agent systems as bridge cases when they close the loop between context, action, feedback, and future behavior. Pure conversational memory, retrieval-augmented question answering, or one-shot prompting is therefore outside the core claim unless the retained context changes decisions in a repeated environment. The closest bodies of work are therefore not one lineage but several intersecting ones: lifelong and continual ICRL [60], [61]; non-stationary MDP analyses [23], [24], [62], dynamic-regret analyses [63]–[65], and near-optimal non-stationary control [66]; value- and uncertainty-aware ICRL [51], [56], [57], compositional or suboptimal-data ICRL [32], [67], and model-aware ICRL [58], [59]; retrieval-augmented ICRL [33], [53], history curation [68], and retrieval-based RL [54], [55]; large heterogeneous sequence-agent datasets [31], [69], generalist sequence agents [49], [70], and mixture-of-experts routing [71]; and reward-feedback agents [22], [72], safety-feedback agents [29], [73], and barrier-style safe ICRL [74].

The non-stationary lens changes how the field should be evaluated. Many current ICRL evaluations test held-out task generalization or average improvement with longer prompts. Those tests are useful, but they do not by themselves show whether an agent can recover after a shift, reject stale demonstrations, reuse old context when a mode recurs, or operate under a finite context budget. For non-stationary ICRL, evaluation should report lifecycle curves, post-shift recovery, dynamic regret, stale-context sensitivity, retrieval utility, and adaptation under controlled changes in shift frequency and observability. Recent work begins to address these issues directly through lifelong ICRL [60], continual ICRL [61], classical non-stationary RL [62], [75], [76], near-optimal non-stationary control [66], bandit tracking [35], [77], and dynamic-regret analyses [65], but the area still lacks a shared taxonomy and reporting standard.

Figure 1 gives the organizing picture. The frozen decision model is only one part of the system. The context pipeline decides which observations, demonstrations, rewards, retrieved memories, and summaries reach that model at each decision point. Once the environment changes, this pipeline is both the adaptation mechanism and a common source of failure. ICRL systems should therefore be evaluated by how their policy model, interaction history, memory, and feedback loop behave together under shifts. Table 1 makes this distinction explicit by contrasting the scope of this paper with a general ICRL survey. Table 1 then positions the paper against adjacent survey literatures, and Figure 2 shows the resulting domain boundary.

4pt

The survey builds its argument in four steps. Figure 2 is a domain-boundary view: it identifies which literatures overlap in the problem studied here. The survey first casts non-stationary ICRL as a context-validity problem. It then separates non-stationarity into three questions that are often conflated: what changes, how the change unfolds, and how directly the change can be observed by a frozen context-conditioned agent. On that basis, it reviews methods by the mechanism that manages context–raw histories, curated histories, retrieval, compression, value-aware inference, model-based planning, reflection, and forgetting–rather than by publication chronology. Finally, it argues for evaluation protocols that expose adaptation rather than memorization or domain-randomization-style generalization, and it highlights problems that stationary benchmarks tend to miss: action-semantics shift, context poisoning, memory isolation, uncertainty-calibrated retrieval, and theory for finite, stale, or conflicting context. The same source/pattern/observability vocabulary is used throughout the survey so that method claims, benchmark claims, and open problems can be stated at the level of shift actually being tested.

The rest of the survey is organized as follows. Section 2 defines ICRL and its boundaries. Section 3 formalizes non-stationary ICRL and introduces context validity. Section 4 develops the taxonomy. Section 5 explains why non-stationarity changes the role of context. Section 6 reviews method families through the lens of context management. Section 7 discusses training data and objectives. Section 8 proposes evaluation protocols and metrics. Section 9 summarizes theoretical links and gaps. Section 10 closes with design principles and open problems.

2 Foundations of In-Context Reinforcement Learning↩︎

2.1 Fixed-Parameter Test-Time Adaptation↩︎

We use an operational definition, defining the concept through observable, actionable, and testable criteria. An agent performs ICRL when a fixed-parameter decision model uses the current episode or a broader interaction history to infer latent task structure and improve future action selection at test time. For an observation \(o_t\), action \(a_t\), and context \(C_t\), the policy can be written as \[\pi_\theta(a_t \mid o_t, C_t), \qquad \theta \;\text{fixed at test time}. \label{eq:icrl-policy}\tag{1}\] The context is usually a finite serialization of interaction evidence, \[C_t = \operatorname{pack}_B \left( \tau_{1:k}, d_{1:\ell}, q_{1:m}, z_t \right), \qquad \tau_i=(o_1^i,a_1^i,r_1^i,\ldots,o_{H_i}^i), \label{eq:context-serialization}\tag{2}\] where \(\operatorname{pack}_B\) denotes a budget-limited formatting, retrieval, or compression operator; \(\tau_{1:k}\) are trajectories, \(d_{1:\ell}\) are demonstrations or descriptions, \(q_{1:m}\) are retrieved memories or summaries, and \(z_t\) denotes other task hints or feedback. The context \(C_t\) may contain recent trajectories, older episodes, demonstrations, rewards, return prompts, task descriptions, retrieved memories, compressed summaries, value estimates, natural-language feedback, or other tokens that influence the model’s action. The defining constraint is not the token type but the adaptation channel: the agent adapts through \(C_t\), not through gradient updates to \(\theta\). Merely conditioning on a prompt is therefore not enough; the context must carry evidence from which the model can change later decisions in a task-relevant direction. Equivalently, deployment-time adaptation is a forward-pass map from evidence to behavior, \[C_t \mapsto \pi_\theta(\cdot \mid o_t,C_t), \qquad \theta_{t+1}=\theta_t=\theta, \label{eq:fixed-parameter-adaptation}\tag{3}\] whereas online RL or gradient-based meta-learning would allow \(\theta_{t+1}\neq\theta_t\) after observing new experience.

This definition covers several lines of work. Algorithm distillation trains a model on histories generated by a learning algorithm, so that the trained model can imitate the algorithm’s within-lifetime improvement through its context [2]. Decision-pretrained transformers and supervised-pretraining approaches show that trajectory models can learn useful in-context decision rules from offline data [6], [7], [92], prompting and generalized decision-transformer objectives [47], [99], supervised ICRL theory and pretraining [8], [50], agentic or variable-action formulations [28], [48], and large multi-task decision datasets [49], [69], [70]. Long-context meta-RL agents connect the same idea to recurrent or transformer policies trained by reinforcement learning over multiple tasks or episodes, including transformer meta-RL [9], AMAGO-style agents [10], [100], and RELIC-style long-context systems [101]. Retrieval-augmented agents make the context partly external by selecting relevant experience from a memory database, including retrieval-augmented decision transformers [33], REGENT-style retrieval [53], and earlier retrieval-based RL systems [54], [55]. History curation and value-aware variants extend this design space by filtering learning histories [68], routing heterogeneous decision tokens [71], or replacing pure imitation with value-sensitive objectives [51], [56], [57]. Efficient long-sequence alternatives such as S4-style state-space models [102], [103], recurrent-memory and S5-style models [104], [105], gated or xLSTM-style recurrent attention [106], and Mamba-style sequence models [107], [108] are also relevant because non-stationary ICRL often asks the agent to retain long histories while still reacting quickly to recent evidence.

ICRL is therefore a behavioral property of the deployment loop, not a single architecture. A transformer with a trajectory prompt may fail to perform ICRL if the prompt is ignored. A recurrent policy may perform ICRL if its hidden state or explicit context implements fast adaptation. A large language model belongs in scope only when it is embedded in a sequential decision process with action, feedback, and reward-like signals, rather than merely recalling conversational state. This includes LLM bandit agents [72], reward-feedback agents [22], ReAct-style agents that act in an environment [95], reflection or experience-library systems whose stored lessons change later trials [96], [109], [110], and tool-use or computer-control agents whose future actions depend on previous tool outcomes [111]. It excludes static prompting [11], [12], one-shot retrieval-augmented question answering [112], generic chatbot memory [113]–[115], and summaries that do not affect a repeated decision loop. In latent-task notation, many ICRL systems can be read as amortizing a Bayes-adaptive control rule: \[b_t(\xi) = p(\xi \mid C_t) \propto p(\xi) \prod_{(o,a,r,o') \in C_t} p_\xi(o',r \mid o,a), \qquad \pi^\star(a \mid o,C_t) = \int \pi_\xi^\star(a \mid o)\, b_t(\xi)\, d\xi, \label{eq:latent-task-belief}\tag{4}\] where \(\xi\) is a hidden task, dynamics, reward, or goal parameter. The learned policy need not explicitly represent \(b_t\), but successful ICRL behavior requires some computation with the same effect: evidence in context changes the posterior over which decision rule is currently appropriate. Recent theory strengthens this operational reading by showing that transformers can represent policy-improvement-like computations from in-context trajectories [116], complementing chain-of-thought and temporal-difference analyses of ICRL [52], [117]. A fixed-parameter model can still implement an update-like decision rule inside the forward pass, so ICRL should not be dismissed as static behavior cloning. The guarantees usually rely on coherent context drawn from a stable task family; non-stationary ICRL asks what happens when the same context window also contains obsolete, conflicting, or regime-mismatched evidence. This behavioral definition is consistent with broader ICL work showing that in-context behavior can arise from distributional structure [13], meta-training and function learning [14], [15], [18], implicit linear-model fitting or gradient-descent-like computations [16], [19], and demonstration formatting [12], [17]. For this survey, those results are useful background but not sufficient: reinforcement learning adds actions, rewards, exploration, credit assignment, and the possibility that old examples become actively wrong.

This behavioral definition also avoids a common ambiguity in the literature. A method can be transformer-based, pretrained on trajectories, and evaluated on held-out tasks without demonstrating ICRL. The decisive test is counterfactual: if the same observation is paired with different valid contexts, does the policy change in the direction implied by those contexts? One compact diagnostic is a context-contrast score, \[\begin{align} \Delta_{\mathrm{ctx}}(o;C,C') &= D\!\left( \pi_\theta(\cdot \mid o,C) \,\middle\|\, \pi_\theta(\cdot \mid o,C') \right), \\ J(\pi_\theta;C) &> J(\pi_\theta;C') \quad \text{when C contains better current evidence}. \end{align} \label{eq:context-contrast}\tag{5}\] where \(D\) may be total variation, KL divergence, or another distance between action distributions. Large distributional change alone is not enough; the changed action distribution must improve the downstream decision objective under the task implied by the context. For non-stationary ICRL, the counterfactual is stronger. If the prompt contains both old and new evidence, can the policy privilege the evidence that is valid under the current regime? This is the reason the survey treats context construction, retrieval, and forgetting as first-class parts of the agent.

2.2 Nearby Paradigms↩︎

ICRL overlaps with several mature paradigms but is not identical to them. Black-box meta-RL trains an agent over a distribution of tasks such that within-episode or within-lifetime computation implements a learning procedure [1], [4]. ICRL can be seen as a modern sequence-modeling version of this idea, with explicit context windows, retrieval, and trajectory tokens.

Classical RL and planning texts frame the stationary baseline through MDPs and value functions [43], [118], while POMDP work shows how hidden state can be converted into belief-state inference [44]. Contextual MDPs [37] and hidden-parameter MDPs [38], [45], [119] are especially close to meta-RL because a latent or hidden parameter controls rewards and dynamics across related tasks. Epistemic-POMDP [46] and task-inference views [41], [120] make the same point from a different angle: what looks like poor generalization can be implicit partial observability over an unobserved task, regime, or data-collection process. For non-stationary ICRL this perspective is useful but incomplete. The latent task belief is not only uncertain; it can expire, conflict with new evidence, or become useful again when a previous regime returns. This is why recurrent memory baselines [121], [122], trajectory-contrastive context encoders [123], and non-stationary meta-RL methods [87], [88] are important bridge literatures for the survey.

The older recurrent meta-RL line, including RL\(^2\) [1] and learning-to-reinforcement-learn systems [4], already made the key move of representing fast adaptation in hidden state rather than in test-time weight updates. Attentive and latent-belief meta-RL methods such as SNAIL [124], PEARL [85], and VariBAD [86] sharpened the same idea by treating recent experience as evidence for a latent task or Bayes-adaptive belief. Gradient-based meta-RL, represented by MAML-style training [5] and later meta-RL variants [84], is adjacent but distinct because its adaptation channel is usually an explicit inner-loop parameter update rather than a fixed-parameter forward pass. The operational difference can be summarized as \[\underbrace{\pi_{\theta'}(a_t\mid o_t),\quad \theta'=\theta-\alpha \nabla_\theta \mathcal{L}(\theta;\tau_{1:k})}_{\text{gradient-based adaptation}} \qquad\text{versus}\qquad \underbrace{\pi_\theta(a_t\mid o_t,C_t),\quad \theta \;\text{unchanged}}_{\text{in-context adaptation}} . \label{eq:adaptation-channel-comparison}\tag{6}\] Decision Transformer [7], Trajectory Transformer [6], RvS [92], and Multi-Game Decision Transformer [97] then made trajectory sequence modeling a scalable substrate for offline decision models. Offline RL also learns from previously collected data [89], [90], but standard offline RL usually produces a policy that acts without using new test-time examples as context; conservative Q-learning [125] and implicit Q-learning [126] are therefore background for value objectives, not ICRL by themselves.

Continual RL studies agents that persist across changing experiences and often update parameters, replay buffers, or other long-lived state [39], [40]. Non-stationary RL studies algorithms for changing MDPs, often with change-point tests [34], [127], sliding windows or discounting [35], bandit and MDP variation-budget guarantees [21], [75], [76], dynamic or near-optimal regret bounds [36], [62], [66], transformer-based regret analysis [65], or restart and policy-search schedules [23], [83], [128]. Survey work [80] and policy-search variants [24], [63], [64] further broaden this algorithmic view. ICRL imposes the additional constraint that the main policy parameters are fixed at test time.

The distinction among continual RL, lifelong RL, and ICRL is therefore operational rather than merely terminological. Continual RL asks how an agent keeps learning across a stream of tasks or changing environments [39], [40]; the usual toolbox may include online gradient updates in persistent agents [39], [40], replay [129], [130], regularization [131]–[133], architectural expansion [134], or explicit task-free segmentation [79]. Lifelong RL is often used more broadly for persistent agents that accumulate and reuse experience over long horizons [27], [78], [135]; it emphasizes retention [131]–[133], forward transfer [136], [137], and recovery or reuse when older skills or modes become relevant again [33], [53], [138]. ICRL, in contrast, asks whether the deployed decision model can adapt with its parameters fixed, by changing only the context, hidden state, retrieval set, or summary supplied to the forward pass. Non-stationary ICRL is the intersection of these ideas: it inherits continual and lifelong RL’s changing-world concern, but it restricts the adaptation mechanism to context management. For a fixed context budget \(B\), the ICRL boundary can be written as the constrained deployment problem \[\max_{g} \;\mathbb{E}\!\left[ \sum_{t=1}^{T} \gamma^{t-1} r_t \right] \quad \text{s.t.}\quad C_t=g(h_{1:t-1}),\;|C_t|\le B,\;\theta\;\text{fixed}, \label{eq:context-budget-objective}\tag{7}\] which places the burden of adaptation on what is selected, compressed, retrieved, or forgotten before the policy’s forward pass. Recent direct work on lifelong ICRL [60] and continual ICRL [61], together with adjacent non-stationary in-context-learning theory [139], makes this intersection increasingly explicit.

Figure 3 gives a compact historical timeline for this lineage. The important transition is from hidden-state adaptation in early meta-RL, to trajectory sequence modeling, to explicit ICRL, and finally to the current non-stationary turn where retrieval, value awareness, and lifecycle evaluation become central. Table 3 uses this distinction to define the core, bridge, and excluded cases for the rest of the survey.

None

4pt

2.3 Boundary Choices Under Non-Stationarity↩︎

This survey draws the boundary around the adaptation channel rather than around the mere presence of non-stationarity: a method is central only when a fixed-parameter sequential decision agent must adapt to within-lifetime changes by selecting, compressing, retrieving, or forgetting context. Boundary choices matter more under non-stationarity than under stationary task generalization. If online parameter updates are allowed, adaptation can be attributed to continual learning rather than context use. If the environment is not sequential, the problem becomes supervised in-context learning or contextual inference rather than reinforcement learning. If the setting is purely offline and the agent receives no test-time context, then stale-context behavior cannot be studied. This survey therefore uses a strict core definition and a conservative extended scope. The aim is not to claim every context-conditioned model as ICRL, but to isolate the cases where changing context is the mechanism by which the agent adapts to changing decision processes.

The conservative boundary is also useful for negative results. If a fixed model fails after a reward switch, the failure should not immediately be attributed to lack of model capacity. It may be caused by a context policy that supplies the wrong evidence, a training distribution that never invalidates old context, or an evaluation protocol that rewards memorization. Conversely, if a system succeeds only because it fine-tunes online, the result may be valuable for continual RL but does not answer the ICRL question. Maintaining this boundary keeps the survey focused on the mechanisms that make adaptation through context possible or impossible while deployed policy parameters remain fixed.

3 Problem Setting for Non-Stationary ICRL↩︎

3.1 From Stationary MDPs to Changing Decision Processes↩︎

Let the agent interact with a sequence of decision processes \[\mathcal{M}_t = (\mathcal{S}_t, \mathcal{A}_t, P_t, R_t, \Omega_t, O_t, \gamma_t), \label{eq:changing-mdp}\tag{8}\] where \(\mathcal{S}_t\) is the state space, \(\mathcal{A}_t\) is the action space, \(P_t\) is the transition kernel, \(R_t\) is the reward function, \(\Omega_t\) is the observation space, \(O_t\) is the observation function, and \(\gamma_t\) is the discount factor or horizon convention. This notation extends the standard MDP formalism [43] and allows partial observability [44], hidden contexts [37], or latent task parameters [38], [45], [46] to enter through \(O_t\) and through the unobserved variables that govern \(P_t\) and \(R_t\). Non-stationarity means that at least one decision-relevant component changes over time or across phases of the same context-bearing lifetime, for example \(P_t \neq P_{t+1}\), \(R_t \neq R_{t+1}\), or a change in \(O_t\), \(\mathcal{A}_t\), goals, constraints, or action semantics. Not every component must change. The survey focuses on changes that alter which actions are good, which observations are informative, which memories are relevant, or which previous trajectories should be trusted.

At time \(t\), the agent acts with a fixed policy class \(\pi_\theta(a_t \mid o_t, C_t)\). The context \(C_t\) is produced by a context-management process: \[C_t = g_\phi(h_{1:t-1}, m_{1:t-1}, b_t), \label{eq:context-manager}\tag{9}\] where \(h_{1:t-1}\) denotes interaction history, \(m_{1:t-1}\) denotes external memories or retrieved items, and \(b_t\) denotes budget constraints such as context length, retrieval count, or latency. In the strictest ICRL setting, both \(\theta\) and \(\phi\) are fixed after training. Some systems use hand-designed retrieval or summarization rules [33], [53]; others learn the context manager [68], [85], [86]. The key distinction from continual RL remains that the adaptation channel at deployment is the constructed context, not online weight modification. The policy must therefore infer task rules, reward meanings, transition structure, or action effects from history and express adaptation through future actions.

This formulation separates three objects that are often conflated. The decision model maps observations and context to actions. The context manager chooses what evidence is visible to the decision model. The environment sequence determines whether old evidence remains compatible with the present decision problem. In stationary ICRL, the second object can often be treated as a formatting detail: include demonstrations, rewards, or previous episodes, and ask whether the model uses them. In non-stationary ICRL, the context manager is a policy in its own right. It must trade off recency, reliability, diversity, budget, and the possibility that a previously encountered regime has returned. Classical non-stationary bandit and RL algorithms often encode this tradeoff through sliding windows and discount factors [35], [77], optimism bonuses [21], [62], [75], restart schedules [83], or explicit changepoint posteriors [34], [127]. The ICRL version differs because those operations may be implicit in the prompt, retrieval index, attention pattern, or memory state rather than in a hand-coded online estimator.

None

Figure 4 frames the lifecycle perspective behind the adaptation metrics below, including dynamic regret, recovery time, context efficiency, and stale-context sensitivity. A non-stationary benchmark should therefore reveal pre-shift competence, immediate post-shift degradation, recovery speed, and the reuse of context when a previous regime returns.

3.2 Context Validity↩︎

We define context validity as a decision-time property. A context item \(c_i \in C_t\) is valid at time \(t\) if conditioning on it improves, calibrates, or preserves decision quality under the current environment \(\mathcal{M}_t\) compared with ignoring it. It is invalid if it systematically worsens decisions, biases value estimates, hides a change point, or consumes scarce context budget without useful information. One way to express this decision-relative notion is the marginal context value \[\nu_t(c_i) = \mathbb{E}\!\left[ Q_t^{\pi_\theta}(s_t,a_t \mid C_t) - Q_t^{\pi_\theta}(s_t,a_t \mid C_t \setminus \{c_i\}) \right], \label{eq:context-value}\tag{10}\] where the expectation is over the current state, action-selection randomness, and any retrieval randomness. Positive \(\nu_t(c_i)\) indicates helpful evidence, negative \(\nu_t(c_i)\) indicates harmful or misleading evidence, and values near zero indicate context that may be syntactically relevant but decision-irrelevant. This definition is intentionally performance-relative: old context is not invalid merely because it is old, and recent context is not valid merely because it is recent. For recurring modes, an old trajectory may become valid again [39], [40], [60]. For abrupt reward relabeling, even a high-return recent trajectory may be misleading [21], [22], [87].

This definition deliberately differs from syntactic relevance. A trajectory can be syntactically similar to the current observation stream while being decision-invalid because the reward changed [33], [57]. A demonstration can be high quality under its original regime while being invalid under a new action interface [27], [28]. A recent item can be invalid because it was produced by an exploratory or corrupted behavior policy [89], [144], [145]. The survey therefore uses validity as a causal and decision-theoretic notion: context is valid when it improves the action distribution for the present regime, not when it looks similar, recent, or successful in isolation.

The validity lens gives a compact statement of the non-stationary ICRL problem:

This is a stronger requirement than stationary few-shot generalization. It requires the model or its context manager to solve a joint problem of inference, memory selection, and control.

3.3 Objectives↩︎

The objective can be expressed in several complementary ways. The standard cumulative-reward objective is \[J(\pi_\theta, g) = \mathbb{E}\left[\sum_{t=1}^{T} \gamma_t r_t \right],\] where actions are produced by \(\pi_\theta(a_t \mid o_t, g(h_{1:t-1},m_{1:t-1},b_t))\). For non-stationary environments, average return hides important lifecycle behavior. Two agents can have similar average return while one recovers immediately after shifts and the other fails catastrophically but performs well between shifts. Ergodicity-focused work gives a second reason to distrust a single average-return number: when reward processes are non-ergodic, ensemble expectations can diverge from the time-average experience of an individual deployed agent [146], [147]. In such settings, optimizing or reporting expected return can favor a policy that looks good across many hypothetical runs while producing poor long-run outcomes on the actual trajectory that carries the agent’s context. This concern is especially acute for ICRL because the context window is itself trajectory-dependent; early failures, stale memories, or unlucky regime exposure can change the evidence available for later decisions. Dynamic regret is often more diagnostic: \[\mathrm{Reg}_T^{\mathrm{dyn}} = \sum_{t=1}^{T} V_t^{\pi_t^\star}(s_t) - \sum_{t=1}^{T} V_t^{\pi_\theta}(s_t), \label{eq:dynamic-regret}\tag{11}\] where \(\pi_t^\star\) is an oracle policy for the current environment or current latent mode. This objective follows the same motivation as adversarial and non-stationary bandit work [21], [82], [148], non-stationary MDP regret analyses [62], [75], [76], near-optimal or kernelized variants [66], [149], and non-ergodic RL critiques of expectation-based objectives [146], [147]: a static comparator or ensemble average can make an agent look good simply because the environment changes or because failures are averaged away, whereas a dynamic comparator exposes the price paid for delayed detection, adaptation, and context repair along the realized lifetime. The amount of environmental movement can be summarized by a variation budget, for example \[B_T = \sum_{t=2}^{T} \left( \|P_t-P_{t-1}\|_{\infty} + \|R_t-R_{t-1}\|_{\infty} \right), \label{eq:variation-budget}\tag{12}\] with analogous terms added when observation functions, action semantics, constraints, or task priors change. This budget is not the only possible formalization, but it makes explicit that a method’s adaptation claim depends on how often and how much the environment changes. Post-shift recovery time, post-shift area under the learning curve, context efficiency, and stale-context sensitivity provide additional views that are more directly connected to ICRL. Given a shift at \(\tau\), one useful recovery statistic is \[T_{\mathrm{rec}}(\tau,\alpha) = \min\left\{k \geq 0: \frac{1}{w}\sum_{i=0}^{w-1} J_{\tau+k+i}^{\pi_\theta} \geq \alpha \frac{1}{w}\sum_{i=0}^{w-1} J_{\tau+k+i}^{\pi^\star} \right\}, \label{eq:recovery-time}\tag{13}\] where \(\alpha\) is a target fraction of oracle performance and \(w\) is a smoothing window. These objectives are summarized in Table 4, which shows why stationary ICRL metrics are insufficient once context validity changes over time.

3.4 Non-Stationarity Is Not Domain Randomization↩︎

Held-out task evaluation and domain randomization both test generalization across tasks or environment parameters, but they do not by themselves test non-stationary ICRL. In non-stationary ICRL, the task may change during the agent’s lifetime, so the agent must adjust its behavior from context rather than from a reset, retraining, or a clean task descriptor. A model can therefore generalize well to a new stationary task while still failing when the task changes halfway through an episode; conversely, it may handle recurring change points in a narrow environment family without achieving broad held-out generalization.

This distinction is a useful diagnostic for benchmarks. If changes occur only between episodes and the agent receives a clean task descriptor at the start of each episode, the benchmark mainly tests task-conditioned generalization. If the task changes within a lifetime but the prompt or history is reset at the change point, it tests fast reconditioning rather than robustness to stale context [61], [68]. The central non-stationary ICRL regime arises when old and new regimes coexist in the same context window, forcing the agent to decide which evidence remains valid [33], [57]. Papers reporting non-stationary ICRL results should therefore specify which regime they evaluate.

4 A Taxonomy of Non-Stationarity for ICRL↩︎

Non-stationarity is too broad a label to support useful comparisons by itself. The term can refer to qualitatively different stressors, including announced or inferable reward changes in non-stationary bandits and MDPs [21], [22], [35], [76], gradual variation in transition dynamics [23], [24], [36], recurrent latent regimes in contextual or hidden-parameter MDPs [37], [38], [45], and changes in the action interface or action-to-effect mapping [27], [28]. These regimes do not exercise the same adaptation mechanism. An append-all history may be adequate for gradual reward estimation and harmful after abrupt action remapping. A retrieval-augmented model may be strong on recurring modes and still fail under subtle observation drift if its retrieval key is calibrated to the wrong surface features [151], [152]. A value-aware model may react to reward shifts while remaining brittle when transition dynamics move outside its support [153]–[155].

We therefore describe a non-stationary ICRL setting along three axes: the component that changes, the temporal structure of the change, and the evidence by which the agent can observe it. This is narrower than general RL generalization taxonomies, which mainly ask whether a trained policy transfers to held-out environments; here the issue is whether a fixed model can keep using, rejecting, or reactivating context during a lifetime [93]. It is also more operational than POMDP and contextual-MDP formalisms [37], [44], epistemic-POMDP views [46], and hidden-parameter MDP formalisms [38], [45]. Those formalisms justify treating task identity or regime as latent. They do not by themselves say how a finite prompt, memory index, or retrieval policy should be managed under drift. They also do not reduce non-stationary ICRL to domain randomization, because the relevant change happens inside the interaction trajectory rather than only between sampled training or test instances.

Figure 5 shows a compact map of the axes, with representative labels for each dimension, and Figure 6 gives a tree view for placing papers and benchmarks. The full category definitions appear in Tables [tab:what-changes]–6. A concrete non-stationary setting can be described along all three independent dimensions simultaneously. For example, a setting labeled as dynamics/recurring/latent denotes a task in which the transition function changes across recurring regimes, while the current regime is not directly observed and must be inferred from the interaction context. The main practical point is that these axes should be reported independently. Explicit reward switches and latent recurring dynamics are both non-stationary, but they are not interchangeable evidence for the same adaptation claim.

None

4.1 Changing Components↩︎

The first axis characterizes where change enters the decision process. Table [tab:what-changes] summarizes the main cases. The rows are not mutually exclusive, since deployed systems often face multiple forms of change at the same time. The central question is which part of the accumulated context has become unreliable.

These sources of change stress different failure modes of context use. Reward and preference shifts are the cleanest validity test: a trajectory that was optimal before a reward switch can become actively misleading afterward, especially for models trained to imitate high-return histories. Dynamics and observation shifts test whether the agent infers latent environment structure from context rather than replaying familiar action sequences or overfitting to superficial tokens. Action-semantics shifts are particularly sharp for sequence models, because the same action token or embedding may denote a different intervention after the interface changes. Task-distribution, behavior-policy, and resource shifts instead alter the credibility of accumulated evidence: the agent must decide whether demonstrations, retrieved trajectories, and old priors still describe the current decision problem.

4.2 Temporal Patterns of Change↩︎

The second axis is temporal structure. An abrupt change asks for change-point detection, gradual drift for tracking, recurring modes for memory indexing, and adversarial or strategically timed changes for robustness. Table 5 links these patterns to the context-management implication and to metrics that reveal the corresponding failure.

These temporal patterns imply different forms of memory. Abrupt non-stationarity favors short-term evidence and explicit forgetting. Gradual drift favors smoothing and recency-weighted estimation. Recurring modes favor long-term memory with mode-specific retrieval. Expanding support favors exploration and uncertainty calibration. A single context-length ablation cannot distinguish these cases.

4.3 Shift Observability↩︎

The third axis asks how directly the agent can observe that a shift has occurred. At one extreme, the change is announced by a task descriptor, instruction, goal token, or other explicit signal. At the other, it must be inferred from prediction errors, transition mismatches, or delayed outcomes that may appear only after consequential actions have already been taken. This distinction is central for ICRL because the same context must often serve two roles at once: diagnosing the current regime and choosing actions under that diagnosis. Table 6 separates the resulting observability regimes and the abilities they stress.

Observability also changes what a benchmark can legitimately claim to measure. When every shift is announced by a clean descriptor, success mostly demonstrates task conditioning, instruction use, or prompt parsing. When shifts are hidden behind sparse delayed feedback, the same label instead covers exploration, belief updating, and long-horizon credit assignment. Both regimes are valuable, but they should not be treated as interchangeable evidence for a single adaptation capability.

4.4 A Compact Naming Scheme↩︎

For reporting experiments and organizing the literature, it is useful to name a non-stationary setting with a compact three-part notation: \[{ \setlength{\fboxsep}{2.2pt} \text{\fcolorbox{nsblue}{nslight}{\textcolor{nsblue}{\normalfont\scshape source}}} \;\textcolor{nsgray}{\boldsymbol{/}}\; \text{\fcolorbox{nsgreen}{nslight}{\textcolor{nsgreen}{\normalfont\scshape temporal pattern}}} \;\textcolor{nsgray}{\boldsymbol{/}}\; \text{\fcolorbox{nsorange}{nslight}{\textcolor{nsorange}{\normalfont\scshape observability}}} } .\] For example, an unannounced reward switch is ‘reward/abrupt/partial’, a recurring hidden dynamics mode is ‘dynamics/recurring/latent’, and an action remapping announced by an interface token is ‘action/abrupt/explicit’. The notation is not a substitute for a full benchmark description. Its purpose is more modest: it makes the adaptation claim legible before the details of the environment, dataset, or model architecture are introduced.

4.5 Using the Taxonomy Analytically↩︎

The taxonomy is intended to discipline comparisons, not merely to label benchmarks. Success on ‘reward/abrupt/explicit’ tasks may reflect effective use of a goal token, whereas success on ‘dynamics/gradual/partial’ tasks requires tracking transition evidence over time. Performance on ‘task/recurring/latent’ tasks may come from memory indexing and mode reuse; performance on ‘behavior-policy/abrupt/noisy’ tasks instead depends on judging whether demonstrations and retrieved trajectories are trustworthy. These are distinct capabilities, so they should support distinct claims.

The same taxonomy suggests a minimal stress-test rule. For each claimed source of non-stationarity, evaluation should include at least one temporal pattern where stale context is harmful and at least one observability setting where trivial task-ID conditioning is unavailable. For each claimed memory mechanism, evaluation should include one case where old context should be discarded and one where old context should be recovered. This two-sided requirement prevents degenerate solutions from looking general: a recent-only policy can excel on one-way drift, and an append-all policy can excel on stationary or recurring tasks, but neither by itself solves context validity.

5 The Role of Context Under Non-Stationarity↩︎

5.1 Context Is Evidence, Memory, and Intervention↩︎

In stationary ICRL, context is usually treated as evidence about a fixed latent task. Relevant examples improve inference; irrelevant ones are expected to be ignored. Non-stationarity breaks that tidy picture. The context is now a time-stamped record of decisions, rewards, and observations that may have been generated under different regimes. It is also an intervention: the retrieved or summarized context changes what the frozen model attends to, and therefore changes the policy it implements. This is why external-memory work in language agents is relevant as a systems analogy even when it is not itself reinforcement learning: MemoryBank [113], LongMem [114], and MemGPT [115] make persistent memory explicit. Retrieval-augmented language models [112], [160], [161], nearest-neighbor language models [162], and in-context retrieval [163] make retrieval and evidence selection explicit. The same evidence-management problem appears in older sequential decision theory under different names: POMDP belief states [44], contextual MDP latent contexts [37], hidden-parameter MDP embeddings [38], [45], task-inference variables [41], [120], and Bayesian changepoint posteriors [34] all encode the idea that recent observations should update beliefs about the current regime. Non-stationary ICRL differs in its engineering constraint: the belief update may be only an implicit computation inside a fixed sequence model, retrieval rule, or prompt summary.

Context therefore plays three roles at once. It is evidence about rewards, dynamics, goals, preferences, and uncertainty. It is memory, preserving information that may become useful again if a mode recurs. It is also a control input to a frozen policy, because changing the context can change behavior without changing weights. Non-stationary ICRL is hard because these roles do not always agree. The best evidence for the current mode may require discarding long-term memory; the best long-term memory may exceed the context budget; the most salient trajectory may be precisely the one that should no longer be trusted. Thus the core question is not whether the agent remembers history, but whether it can infer when history has stopped supporting the current policy. In a finite-budget system, this tradeoff can be written as a context-selection problem: \[C_t^\star = \arg\max_{C\subseteq \mathcal{M}_{t-1},\;|C|\leq b_t} \mathbb{E}\!\left[V_t^{\pi_\theta}(s_t \mid C)\right] - \lambda\,\mathrm{cost}(C), \label{eq:context-selection}\tag{14}\] where \(\mathcal{M}_{t-1}\) is the memory available before acting, \(b_t\) is the context budget, and the cost term can represent tokens, retrieval calls, latency, or risk of stale evidence. Most existing systems approximate this objective implicitly; non-stationary evaluation should make the approximation visible.

5.2 Failure Modes↩︎

Table 7 summarizes common failure modes that are easy to miss in stationary evaluation.

Several of these failures come from context management rather than from the policy class itself. A sufficiently expressive model can still fail when its prompt is dominated by invalid evidence, when retrieval keys optimize surface similarity, or when compression removes a rare but decisive reward event. Conversely, a modest model can look robust if it receives clean, recent, mode-specific evidence. The policy and the context pipeline therefore have to be evaluated together.

The failure modes also explain why longer context is an ambiguous intervention. Increasing the context window can improve adaptation by making more evidence available, but it can also increase the fraction of invalid evidence, make attention more diffuse, and amplify high-salience pre-shift trajectories. The right comparison is therefore not “short context versus long context” in isolation. It is short context, full history, recent-only history, oracle-truncated history, learned retrieval, and adversarially contaminated retrieval under the same shift schedule. Only this comparison reveals whether scale is solving adaptation or merely hiding the cost of invalid context. This point is consistent with the broader ICL literature: training distributions [13], transformer function learning [14], gradient-free or probabilistic ICL mechanisms [17], [18], demonstration ordering [12], and contextual learning dynamics [19] can encourage in-context inference, memorization, or failure depending on how examples are clustered, repeated, ordered, or distributed across tasks. For ICRL, the analogous distributional feature is not only task diversity but whether context windows contain the right conflicts: pre-shift versus post-shift evidence, high-return stale trajectories versus low-return current evidence, and recurring modes separated by misleading intervening data.

Retrieval and long context make the problem sharper rather than eliminating it. Retrieval-augmented language systems [165], retrieval-based RL systems [54], [55], and retrieval-augmented ICRL systems [33], [53] show that external memories can improve sample efficiency and reuse, but they also introduce a new decision variable: which memory should be treated as evidence for the current regime. Long-context studies similarly warn that simply appending more examples can dilute or distort in-context learning when the relevant evidence is far from the query or mixed with distractors [166], [167]. For non-stationary ICRL, this means retrieval precision and attention allocation should be reported as adaptation mechanisms, not only as engineering choices.

5.3 The Context Lifecycle↩︎

A useful system-level view is to give context a lifecycle:

1. Write: which interactions, demonstrations, and feedback are stored?

2. Index: how are stored items keyed for future retrieval?

3. Retrieve: which items are selected for the current decision?

4. Compress: how are long histories summarized within a finite context budget?

5. Trust or weight: after evidence is available to the policy, how much influence should each item have on action selection?

6. Forget or isolate: when should context be removed, downweighted, or stored in a mode-specific partition?

These entries should not be read as a mandatory serial architecture. Write, index, retrieve, compress, and forget or isolate are often discrete context-manager operations: a system can log an item, create a key, retrieve a subset, summarize a history, or move an item out of the active memory. Trust is different. It denotes the influence assignment given to available evidence once it reaches the decision computation. In an explicit system, trust may be implemented by a reliability score, uncertainty gate, value-fusion weight, safety filter, or regime posterior. In a plain transformer or recurrent policy, it may be an emergent property of attention, hidden-state dynamics, value prediction, and decoding rather than a separately exposed module. The lifecycle is therefore an analytic decomposition of responsibilities, not a claim that every ICRL agent contains six separate processing blocks.

This distinction clarifies the relation between retrieve and trust. Retrieval answers a candidate-selection question: which memories or summaries enter the context window at all? Trust answers an influence question: conditional on evidence being present, how strongly does it shape the action distribution? A retrieval-augmented agent can retrieve the right item and still fail by over-weighting a stale high-return trajectory; conversely, a long-context agent with no explicit retriever can still succeed if attention or value inference suppresses invalid evidence. For non-stationary ICRL, both questions should be reported when possible: selection quality through retrieval or context-ablation metrics, and influence quality through attention, gating, counterfactual prompting, value sensitivity, or stale-evidence stress tests. Under non-stationarity, every operation or influence facet in this lifecycle can become either an adaptation mechanism or a failure point. Figure 7 visualizes this lifecycle, and Figure 8 states the design question and non-stationary risk at each operation or facet.

None

5.4 Implication for Survey Organization↩︎

The rest of the survey therefore organizes methods by how they manage context. Algorithm distillation learns from histories that contain improvement [2]. Supervised ICRL learns mappings from contextual examples to decisions [8], [50], [168]. Retrieval-augmented methods select external memories [33], [53], [54]. Value-aware and Bayesian methods combine context with priors [51], [56], [57]. Model-based methods infer dynamics or planning computations in context [58], [59]. World-model and communication-based methods compress experience [113]–[115], [164]. Reflection-style agents use textual or symbolic summaries of feedback [22], [96], [109], [110]. For non-stationarity, the key question for each family is the same: what makes the context valid, and what happens when it is not?

This lifecycle view suggests a decomposition of responsibility. The write operation controls what evidence can ever be used. The index and retrieve operations control which evidence is considered similar or relevant enough to expose. The compress operation controls which rare events survive a finite budget. The trust facet controls how strongly available evidence affects the frozen policy, even when that weighting is implicit in the model rather than implemented as a separate module. The forget or isolate operation controls whether old evidence disappears, is downweighted, or remains available for future recurrence. A non-stationary ICRL paper need not solve all parts of the lifecycle, but it should state which responsibility its contribution improves, which parts are held fixed, and whether trust is explicitly modeled or only inferred from behavior.

6 Method Families Through the Lens of Context Management↩︎

Methods for ICRL differ less by their names than by where they place the burden of adaptation. Some rely on the sequence model to infer the right update from raw history. Others move part of the problem into retrieval, summarization, memory partitioning, value estimation, or model inference. For non-stationary settings, the useful organizing question is simple: when the world changes, what mechanism decides which context still matters?

Figure 9 summarizes the resulting landscape. The horizontal axis separates implicit history use from explicit context selection through retrieval, summaries, and memory policies. The vertical axis separates raw behavioral evidence from value- or model-aware evidence. The empty region is as informative as the occupied one: many systems scale context length or memory access without estimating validity, while value- and model-aware systems still depend on the evidence selected for them.

None

6.1 Algorithm Distillation and Learning Histories↩︎

Algorithm distillation trains a sequence model on histories generated by a learning algorithm, with the aim that the trained model internalizes the algorithm’s within-lifetime improvement [2]. For non-stationary ICRL, this family is important because learning histories can include exploration, failure, reward feedback, and policy improvement. If the pretraining histories include changes in task or reward, the model may learn a pattern of adaptation through context. If they are stationary, the model may instead learn that older high-return behavior should remain trusted. This distinction mirrors broader ICL evidence that models can shift between memorization, system identification, and genuine in-context learning depending on distributional burstiness [13], probabilistic or gradient-free ICL structure [17], [18], example ordering [12], and contextual learning dynamics [19].

The critical design variable is the distribution of histories. Histories generated by a strong learner on stationary tasks provide clean examples of improvement, but they may not teach change detection. Histories generated under weak data or noisy policies may better reflect deployment, but they can also create ambiguous supervision. Cross-episodic curricula [169], noise distillation [3], contrastive context encoders [123], and filtered learning histories [68] all modify the history distribution to make within-context improvement easier to learn. From the non-stationary perspective, filtering should not merely remove low-return episodes; it should preserve transitions that reveal when old context became invalid. ICPRL adds a physically grounded variant of this idea by training a vision-language policy on multi-episode trial-and-error histories, so that later actions can condition on previous physical failures and successes without test-time weight updates [59]. The adjacent meta-exploration literature is useful here because it explicitly treats early actions as information-gathering, not only as return maximization, through exploration objectives [144], [170], [171], decoupled exploration [172], hyperstate inference [173], or first-explore strategies [174].

6.2 Supervised ICRL and Decision-Pretrained Transformers↩︎

Supervised pretraining can induce ICRL when the model is trained on datasets where context examples identify a latent decision problem [8], [50]. Decision Transformer [7], Trajectory Transformer [6], RvS [92], prompting variants [99], generalized decision transformers [47], chain-of-hindsight or agentic variants [48], and ICDT-style models [168] provide the broader sequence-modeling substrate. These methods are attractive because they can use offline data and scale with standard supervised objectives. Recent scaling and routing variants, including multi-game decision transformers [97], online decision transformers [175], hierarchical prompt transformers [176], Vintix-style action-model formulations [70], and mixture-of-experts formulations [71], further test whether the same substrate can support broader task mixtures. Theoretical results on in-context policy improvement clarify why this family is more than sequence imitation: under structured conditions, a transformer can use trajectory context to implement a policy-improvement procedure rather than simply replaying the behavior policy [116]. This gives supervised and decision-pretrained ICRL a principled bridge to classical policy iteration, but it also exposes the non-stationary missing assumption. Policy improvement is well grounded when the trajectories describe the same decision process; after reward, dynamics, or action-semantics shifts, the model must first infer which trajectories are still valid before any improvement step is meaningful. Their weakness under non-stationarity is objective mismatch. Next-action prediction can reward copying behavior from context even when that behavior is no longer optimal. Return conditioning can encode a desired outcome, but it does not by itself solve which past examples are relevant after a shift.

The non-stationary version of supervised ICRL should therefore include contrastive context designs. For example, prompts can contain pre-shift and post-shift trajectories with identical observations but different rewards. Training can require the model to attend to the most recent valid evidence, infer a latent mode, or predict change-point indicators. Without such designs, strong held-out-task performance may reflect static generalization rather than adaptation to changing environments. Demonstration-conditioned sequential decision studies show that task diversity, dataset size, and trajectory structure matter for in-context generalization, but those results still need explicit stale-context and shift tests before they establish non-stationary adaptation [150]. The weak-label and suboptimal-data variants are particularly important for this survey. Offline ICRL rarely has perfect learning histories or optimal labels at scale; offline RL coverage issues [89], conservative and implicit value learning [125], [126], return-conditioned value learning [177], Q-learning-style ICRL [51], suboptimal-data ICRL [32], random-policy state-action distillation [178], and TD-style ICRL [52] can turn imperfect historical trajectories into useful adaptation signals, but only if the context window exposes which behavior was suboptimal under the current regime.

6.3 Efficient Sequence Backbones↩︎

The default ICRL backbone is a causal transformer, but long non-stationary lifetimes make the quadratic cost of attention a practical bottleneck. Structured state-space models, recurrent-memory transformers, gated recurrent attention, and Mamba-style sequence models therefore deserve explicit treatment. S4-style ICRL [102] and Decision S4 [103] show that state-space sequence models can support long-horizon memory and sequence-based RL, while Decision Mamba [107], [179] and Mamba-based scaling of algorithm distillation [180] test selective state-space decision models. Related selective-state, recurrent-memory, and xLSTM-style models [104], [105], [108] and transformer-RL memory studies [106], [181] test whether recurrent state can replace or complement transformer attention in decision modeling. For non-stationarity, the architectural question is not only speed. A compressed recurrent state can make old information cheap to carry, but it can also make invalid information hard to remove. Thus efficient backbones should be evaluated with the same stale-context, oracle-reset, and recurring-mode tests as transformer systems. Recent non-stationary ICL theory [139] makes this architectural point explicit: adaptive filtering [182], gating [183], [184], retention [185], and recurrent attention [186] can be interpreted as learnable forgetting factors when the target function changes over time.

6.4 Long-Context Meta-RL Agents↩︎

Long-context agents such as AdA-style systems [187], AMAGO-style systems [10], [100], RELIC-style systems [101], transformer meta-RL systems [9], and hierarchical meta-RL agents [188] use recurrent, transformer, or external-memory mechanisms to solve adaptive RL problems across long horizons and open-ended task spaces. They are natural candidates for non-stationary ICRL because they can integrate long histories and delayed feedback. However, long context is not automatically better. When the environment changes, a longer history increases both evidence and contamination. The relevant question becomes whether the model has learned attention patterns that separate current-mode evidence from stale-mode evidence. Large context, open-ended learning [189], dataset efforts such as XLand and XLand-100B [69], broader generalist-agent sequence models [49], [70], [98], and episodic or long-term recall systems [159], [190] make this question sharper because they scale learning histories far beyond toy settings while still requiring tests of whether scaled histories teach invalidation rather than only broad task recognition.

Long-context meta-RL also raises a reporting issue. If a model is trained online with RL across non-stationary episodes, then test-time adaptation may be partly due to learned recurrent dynamics rather than explicit context retrieval or prompting. This is still within our ICRL boundary when parameters are fixed at deployment, but papers should report the lifetime state, context window, reset rules, and whether memory persists across episodes.

6.5 Retrieval-Augmented ICRL↩︎

Retrieval-augmented methods externalize part of the context problem. Instead of fitting the entire history into the model’s input, they retrieve trajectories, transitions, or summaries from memory in RA-DT [33], REGENT [53], in-context Q-learning with retrieval [67], and earlier retrieval-based RL systems [54], [55]. This is one of the most promising families for non-stationarity because retrieval can, in principle, select mode-relevant memories and ignore stale ones. It is also one of the most fragile. If retrieval is based on surface similarity, it may select trajectories from the wrong reward regime. If retrieval is based on return, it may prefer high-return old behavior that is now invalid. If retrieval is based on recentness, it may fail when an old mode recurs.

For non-stationary ICRL, retrieval should be evaluated with validity metrics, not only downstream return. Precision of retrieved context, counterfactual performance without retrieval, and adversarial stale-memory tests are needed. The best retrieval key may combine observations, actions, rewards, inferred latent modes, uncertainty, and temporal metadata. This creates a bridge between ICRL and memory systems: retrieval is not just a scaling trick, but a learned or designed adaptation policy. The language-model retrieval literature is a useful warning because nearest-neighbor language models [162], retrieval-augmented generation and pretraining [112], [160], [161], in-context retrieval [163], and Atlas-style retrieval [165] can improve prediction while leaving relevance and trust as separate design problems.

6.6 Value-Aware and Bayesian Context Use↩︎

Value-aware methods attempt to make ICRL less dependent on imitation of raw trajectories. Q-learning objectives can improve offline ICRL [51] by teaching the model value-sensitive behavior rather than pure behavior cloning, building on the broader offline RL lesson that conservative and implicit value regularization matter under support mismatch [125], [126]. Scalable in-context Q-learning [56] and compositional Q-learning [67] extend this idea by conditioning value estimates on compact context or retrieved transitions. Bayesian-fusion approaches combine context with value priors, aiming to decide how much to trust contextual evidence under uncertainty [57].

This family is especially relevant under reward and dynamics shifts. If the model estimates uncertainty over values, it can downweight context that conflicts with the current reward signal. If it only predicts next actions, it may reproduce old behavior even after feedback changes. The limitation is that value-aware objectives still need the right evidence. A Q-function conditioned on invalid retrieved context can still produce invalid actions. Thus value awareness and context validity are complementary: values help interpret context, while context management determines which evidence values are conditioned on.

6.7 Action-Space and Interface Adaptation↩︎

Action-space variation deserves separate treatment because it is both a scaling issue and a non-stationarity issue. Variable-action ICRL shows that models can be designed to generalize across discrete action spaces with different size, order, and semantic content [28]. For this survey, the non-stationary version is stricter: the action interface may change during a lifetime while old context still contains action tokens from the previous interface. This makes action embeddings, action descriptions, and interface metadata part of context validity rather than a mere implementation detail. Benchmarks that only switch rewards or tasks miss this failure mode.

6.8 Model-Based In-Context Planning↩︎

Model-based ICRL distills dynamics inference [58] or planning computation [59] into context-conditioned computation. Under dynamics shifts, this is a direct fit: the agent must infer how actions change the world from recent evidence and plan accordingly. Model-based context also gives a clearer diagnostic than action prediction. If a model predicts transitions correctly after a shift but still acts poorly, the failure may lie in planning. If it predicts old transitions after observing new ones, the failure is context adaptation. ICPRL is useful here because it separates an adaptive policy from a world model used for lookahead search in visual physics tasks [59]. For a non-stationary benchmark, the same architecture would need to show that the world-model-guided search reacts to changed physical parameters or action outcomes, rather than merely exploiting a static simulator prior. World-model work [191], simulator-learning work [192], Genie-style interactive generation [193], and instructable simulators [194] provide adjacent mechanisms for this direction, but the survey treats them as relevant only when the learned model is used in a repeated decision loop with context-conditioned adaptation.

The challenge is compounding error. A model-based in-context planner may infer the wrong latent dynamics from stale history and then plan confidently under that wrong model. Non-stationary evaluation should therefore report both predictive accuracy and control performance across pre-shift, immediately post-shift, and recovered phases.

6.9 Context Compression and Communicative World Models↩︎

Finite context makes compression unavoidable. Communication- or world-model-based approaches can summarize interaction histories into compact language-agent memory [113]–[115], predictive video representations [195], or communicative world-model representations [164]. Compression is useful under long horizons and recurring tasks, but it can erase the rare evidence that marks a shift. The key question is not compression ratio alone, but preservation of decision-relevant change evidence. For example, a summary that preserves average reward may miss a sudden preference reversal; a summary that preserves the most recent transition may lose an older mode that will recur.

6.10 LLM and Foundation Agents as Extended ICRL Cases↩︎

LLM-based and foundation-agent systems are in scope only when the model participates in a sequential decision loop where context, actions, feedback, and later behavior are coupled. Recent work frames LLMs as in-context reinforcement learners under bandit feedback [72], reward feedback [22], or online regret criteria [196]. In these settings, the frozen model weights are not the adaptation channel. The adaptation channel is the prompt, transcript, retrieved memory, reflection, executable skill, or summary that changes future actions after previous outcomes. ReAct-style agents [95] connect reasoning traces to environment actions, while Reflexion [96], ExpeL [109], Voyager [110], and general computer control agents [111] store verbal reinforcement, experience summaries, executable skills, or interaction memories for later trials.

Non-stationarity makes this bridge case more than a systems detail. User preferences can drift, hidden instructions can change, reward feedback can be reinterpreted, tool schemas and website layouts can move, and old reflections can become stale policy hints. A memory may be semantically similar to the current situation but invalid because it was produced under a different reward function, tool interface, or user constraint. Natural-language summaries are especially risky because they compress many observations into declarative advice; after a shift, the advice can remain fluent and authoritative while becoming wrong. This is the LLM-agent analogue of stale trajectories in decision-transformer systems and stale retrieved transitions in retrieval-augmented ICRL.

The survey still treats these systems as extended cases rather than the core of ICRL. Many LLM-agent papers do not report scalar rewards, controlled action spaces, lifecycle curves, no-memory baselines, or context ablations. Success in an interactive language environment may reflect instruction following, prompt parsing, static task recognition, or retrieval of a near duplicate rather than reinforcement learning through context. Bandit studies provide an important caution: frozen LLMs often need explicit summaries [140], [141] or algorithmic scaffolding [142] to explore reliably in context. Multi-turn agent training methods such as PAPRIKA [197], ICAL [198], ArCHer [199], VAGEN [200], SCoRe-style self-correction [201], and cultural-accumulation systems [202] are therefore adjacent unless the experiment shows that repeated feedback changes future decisions through the context channel.

For a large-model result to support a non-stationary ICRL claim, the evidence should show post-shift behavioral change without parameter updates, compare memory policies, and expose stale-context failure. Useful baselines include no memory, last-\(k\) transcript, full transcript, summary memory, retrieval memory, oracle-current-rule context, and stale-memory adversaries. Small-scale reinforcement-tuned transformer studies that report adaptation to changing gridworld conditions are useful proof-of-concept evidence [203], but a convincing large-model benchmark must also test whether verbal memories, tool histories, and retrieved examples remain valid under preference, instruction, reward, or interface shifts. Without environment interaction, scalar or preference feedback, and repeated decisions, LLM memory is better viewed as supervised or conversational in-context learning.

6.11 Method Matrix↩︎

Figure 10 condenses the preceding method discussion by mapping each family to its context mechanism, non-stationary strength, and main risk.

None

6.12 Cross-Family Lessons↩︎

Across these families, architecture is not the cleanest dividing line. Algorithm distillation and supervised ICRL ask the pretrained model to infer the meaning of context inside its forward pass. Retrieval and compression make the context pipeline more explicit: the model can adapt only to evidence that was selected and preserved. Value-aware and model-based methods change what the evidence means, treating context as information about values, dynamics, or uncertainty rather than as behavior to imitate. These choices are complementary under non-stationarity, but each needs different evidence. Table 9 turns this point into a standard for evaluating claims about each method family.

The common pattern is that useful systems need three things at once. Their training histories must contain invalidation events, otherwise old evidence is always presented as trustworthy. Their context managers need validity cues–time, shift likelihood, reward compatibility, uncertainty, or latent-mode metadata–instead of relying only on similarity. Their evaluations must test both forgetting and reuse. Aggressive forgetting can look good after an abrupt shift and fail when a mode recurs; preserving everything can look good on recurrence and fail after an irreversible preference change. The target is not more memory in the abstract, but controlled influence of memory.

7 Training Data and Objectives↩︎

7.1 Training Data as an Implicit Non-Stationarity Assumption↩︎

Every ICRL training pipeline encodes an assumption about how context should be interpreted. Stationary trajectories teach that old evidence remains relevant. Learning histories teach that behavior can improve across time. Mixed-task datasets teach that context identifies a latent task. Retrieval databases teach that similar past experience can help the present. Non-stationary training data must teach something more specific: the validity of context can change. This point is easy to miss if ICRL is treated only as offline RL at scale: offline RL datasets [145], [204], [205], offline RL surveys [89], [90], and offline sequence-modeling substrates [6], [7], [92] provide the data substrate, but they do not automatically create within-context invalidation events.

This suggests a simple audit question for training data:

If the answer is no, then robust non-stationary adaptation is unlikely to emerge reliably from scale alone. The model may still generalize, but the benchmark should not interpret this as evidence of context-validity reasoning.

This audit question should be applied at the level of individual context windows, not only at the dataset level. A corpus may contain many tasks and still present each window as internally coherent. Such a corpus teaches task inference but not conflict resolution. To train non-stationary ICRL, windows should sometimes contain pre-shift and post-shift evidence together, with supervision or rewards that make the distinction consequential. The model must experience cases where the correct action is not the action most often seen, not the action seen in the highest-return old trajectory, and not necessarily the action associated with the nearest retrieved neighbor. General ICL studies make the same lesson visible in a simpler setting: example arrangement [12], task diversity and burstiness [13], transformer function learning [14], [15], gradient-free or probabilistic ICL mechanisms [17], [18], and contextual learning dynamics [19] can determine whether a transformer learns from context or memorizes a distributional shortcut. For ICRL, dataset construction must additionally expose rewards, interventions, and control consequences.

7.2 Stationary Pretraining, Non-Stationary Testing↩︎

A common pattern is to pretrain on stationary tasks and test on related but changing tasks. This creates a useful stress test, but it should be interpreted carefully. Failure may indicate that the model never learned change detection. Success may indicate that the non-stationary benchmark is solvable through recency heuristics or static generalization. To separate these cases, evaluations should include context-order permutations, stale-context injections, repeated-mode tests, and ablations that remove post-shift reward evidence.

7.3 Non-Stationary Pretraining↩︎

Direct non-stationary pretraining can vary reward functions, dynamics, observations, action mappings, and task distributions across the lifetime. Work on lifelong ICRL [60] and continual ICRL [61] moves in this direction. The main design choice is whether changes are random, curriculum-based, or adversarial. Randomized worlds [31], contextual MDP families [37], hidden-parameter MDPs [38], [45], open-ended or human-timescale task generators [187], [189], unsupervised environment design [206]–[208], PLR-style or evolving generators [209], and non-stationary MDP benchmark suites [42] can provide scale, but the generator must expose the shift types the survey cares about. Otherwise, a large randomized dataset can still underrepresent action-semantics shifts, hidden preference changes, or recurring modes. Recent curriculum-design work makes this point concrete: generators can prioritize levels near the competence frontier [210], improve regret approximations for task discovery [211], regularize environment design [212], or evolve the mechanics of the game itself [213].

7.4 Objective Alignment↩︎

The training objective should align with the desired adaptation behavior. Behavior cloning rewards reproducing the data policy. Return-conditioned sequence modeling rewards action sequences associated with high returns in Decision Transformer [7], Trajectory Transformer [6], and RvS [92]. Q-learning-style objectives reward value-sensitive decisions in ICRL [32], [51], [56] and TD-style variants [52]. They can inherit useful lessons from conservative [125], optimistic [205], implicit [126], or return-conditioned offline value learning [177]. Model-based objectives reward predictive structure in DICP [58] and ICPRL [59]. Contrastive or auxiliary objectives can reward change-point identification, latent-mode inference, or retrieval validity. Table 10 summarizes how common training choices shape expected behavior under non-stationarity.

7.5 Reporting Requirements↩︎

Papers should report the stationarity assumptions of both training and testing. At minimum, a non-stationary ICRL paper should specify which components can change, whether changes appear in training, whether changes are signaled, whether context persists across changes, and whether the agent has a finite context or retrieval budget. Without these details, it is difficult to know whether a method has learned non-stationary adaptation or has simply exploited a benchmark convention.

7.6 Data Construction Patterns↩︎

Several practical data-construction patterns follow from the survey’s taxonomy. Contrastive windows pair two contexts that share observations but differ in rewards, dynamics, or action semantics. Shift-labeled windows include explicit change-point metadata, useful for supervised auxiliary losses and for measuring whether a model can use clean signals. Shift-unlabeled windows remove metadata and require inference from feedback. Recurring-mode windows force the model to store old evidence without letting it dominate every later decision. Contaminated windows insert invalid demonstrations or retrieved trajectories, making robustness to stale evidence part of the training objective.

These patterns can be mixed with existing ICRL training objectives. For behavior cloning, the key is to avoid rewarding stale imitation after a shift. For return conditioning, the key is to ensure that high desired returns cannot be achieved by repeating pre-shift actions. For Q-learning objectives, the key is to provide enough post-shift evidence for the value function to disambiguate regimes. For model-based objectives, the key is to evaluate predictions separately before and after changes. The training objective need not explicitly name “context validity”, but it must create a learning signal where validity affects action quality.

8 Evaluation of Adaptation Beyond Memorization↩︎

8.1 Limits of Average Return↩︎

Average return is necessary, but it is a poor summary of adaptation. It collapses the lifetime into one number and can hide the exact interval where non-stationarity matters. One agent can score well by performing strongly before a shift and failing afterward. Another can recover after each shift but have lower steady-state return. For non-stationary ICRL, the curve itself is the evidence: behavior before the shift, damage immediately afterward, recovery, and performance when an old mode returns. This aligns with non-stationary bandits’ move from static to dynamic comparators [21], [35], [148] and with analogous non-stationary RL analyses [23], [75], [76]. Related MDP regret work [62] and deep RL generalization studies [93], [216] make the same warning concrete: held-out performance alone can hide shortcut learning. Figure 11 illustrates the lifecycle quantities that should be read from this curve.

8.2 Core Metrics↩︎

Several metrics have to be read together because each exposes a different failure mode. For a shift at \(\tau\), post-shift area under the curve can be reported as \[\mathrm{AUC}_{\tau:K} = \sum_{k=0}^{K-1} J_{\tau+k}^{\pi_\theta}, \label{eq:post-shift-auc}\tag{15}\] while stale-context sensitivity can be measured by the drop between a valid context and a deliberately contaminated one: \[S_{\mathrm{stale}} = J\!\left(\pi_\theta,g_{\mathrm{valid}}\right) - J\!\left(\pi_\theta,g_{\mathrm{valid}\cup\mathrm{stale}}\right). \label{eq:stale-sensitivity}\tag{16}\] To separate memory scale from adaptation quality, context efficiency can be normalized as \[\eta_{\mathrm{ctx}} = \frac{J(\pi_\theta,g)-J(\pi_\theta,g_{\emptyset})}{\mathrm{tokens}(C)+\rho\,\mathrm{retrievals}(C)}, \label{eq:context-efficiency}\tag{17}\] where \(\rho\) converts retrieval calls into a token-equivalent cost. Table 11 lists these metrics and explains what each one measures.

8.3 Benchmark Dimensions↩︎

A useful benchmark suite has to vary four dimensions at once: source of change, temporal pattern, observability, and context budget. The first includes reward, dynamics, observation, action semantics, task distribution, and behavior policy. The second includes abrupt, gradual, recurring, expanding, and adversarial changes. The third distinguishes explicit, partial, latent, delayed, and noisy evidence. The fourth covers short context, long context, external retrieval, compressed memory, and no-memory baselines. Without this structure, a benchmark can easily support claims that are technically correct but much narrower than the label “non-stationary ICRL” suggests. Table 12 translates the factorial design into concrete diagnostic benchmark blocks.

Procedural environment families are especially useful for this purpose. XLand-MiniGrid [214] provides scalable JAX meta-RL environments with many task instances, and XLand-100B [69] turns this family into a large-scale ICRL learning-history dataset. Kinetix [215] provides an open-ended 2D physics-based control space with procedurally generated tasks. NS-Gym [42] is a more direct non-stationary MDP substrate, while PHYRE [217], I-PHYRE [218], and DeepPHY [219] provide interactive physical reasoning tasks where repeated attempts and visual dynamics can be repackaged into context-adaptation protocols [59]. These environments are not automatically non-stationary merely because they are procedural; they become non-stationary ICRL benchmarks when the evaluator constructs within-lifetime schedules that switch, drift, recur, or corrupt rewards, dynamics, observations, action interfaces, or task distributions. Offline datasets such as D4RL [145], RL Unplugged [204], and Atari replay [205] can also be useful when they are repackaged into contextual windows with controlled policy quality, support mismatch, and post-shift evidence, but they should not be confused with non-stationary benchmarks unless the test protocol actually changes the task within the context lifetime.

8.4 Candidate Environment Families↩︎

Several existing environment families are useful substrates for non-stationary ICRL, even if they were not originally introduced under that label. Their value is not that they automatically solve the benchmark-design problem, but that they are programmable, procedurally diverse, memory-demanding, or already tied to continual-learning metrics. Table [tab:candidate-environments] groups these candidates by the kind of non-stationary ICRL evidence they can support.

The critical distinction is between an environment substrate and a non-stationary ICRL protocol. Procgen [25], MiniGrid [220], Craftax [221], MiniHack [222], and Jumanji [223] are not evidence of non-stationary ICRL merely because they generate many tasks. They become relevant when the evaluator specifies a within-lifetime shift process, keeps old context visible after the shift, prevents online parameter updates, and measures recovery through context. Continual RL benchmarks such as CRLMaze [224] and CORA [137] provide useful shift schedules and metrics, but they need the same care because many continual RL baselines assume online learning. An ICRL protocol should freeze the main policy parameters and compare raw-history, recent-only, retrieval, compression, oracle-change-point, and stale-context adversary variants.

This table also clarifies what should be excluded from the core claim. Robotics-specific suites and controllers such as Meta-World [229], Continual World [136], LIBERO [230], RoboMimic [231], Open X-Embodiment [232], RoboMME [233], and humanoid whole-body-control systems [234] are informative for continual learning and generalist action modeling, but they are outside the main scope of this survey. They can be mentioned as adjacent benchmarks when discussing evaluation traditions, but the paper should not rely on them for its central non-stationary ICRL argument. Similarly, multi-agent suites should be avoided because they introduce strategic non-stationarity, for example when concurrently updating agents make the environment non-stationary from each individual agent’s perspective [143], which is explicitly outside scope here.

8.5 Constructing Stale-Context Stress Tests↩︎

A non-stationary ICRL benchmark should be specified as a protocol over lifetimes, not only as a set of environment IDs. The core intervention is simple: create a period in which old context is genuinely useful, change the decision process, and then keep some of the old evidence visible so the agent must decide whether to trust it. This makes the failure modes in Table 7 operational. For example, to test stale imitation, the evaluator can collect high-return trajectories under regime \(m_0\), switch to a regime \(m_1\) with the same observations but different rewards or dynamics, and inject the \(m_0\) trajectories into the post-shift context while also providing a small amount of valid \(m_1\) evidence. If performance drops relative to the same post-shift context without the injected trajectories, the agent is not merely slow to learn; it is being actively pulled by stale evidence. This protocol connects continual ICRL evaluation [61], history filtering and weak-data ICRL [32], [68], retrieval-augmented ICRL [33], [53], and non-stationary MDP or bandit stress tests [21], [75], [76].

A minimal protocol can be written as follows:

1. Choose a base environment and shift family. Specify which component changes: reward, transition, observation, action semantics, constraint, task distribution, or behavior policy [23], [27], [29].

2. Generate a lifetime. Run a pre-shift phase under \(m_0\), a post-shift phase under \(m_1\), and optionally a recurrence phase where \(m_0\) returns [61], [138].

3. Build paired contexts. For the same post-shift query state, construct at least four contexts: no-context, valid-current context, valid-current plus stale context, and oracle-truncated context that removes pre-shift evidence.

4. Freeze adaptation channels. Keep policy parameters fixed and vary only the context policy: append-all, last-\(k\), retrieval, compression, learned filter, oracle filter, and stale adversary [20], [33], [68].

5. Measure lifecycle outcomes. Report pre-shift return, immediate post-shift drop, recovery time, post-shift AUC, dynamic regret, stale-context sensitivity, and recurring-mode re-adaptation [61], [65].

6. Attribute the failure. Use ablations to determine whether the bottleneck is change detection, retrieval, compression, influence weighting, forgetting, or policy capacity.

The most important design rule is to use paired evaluations. The valid and contaminated contexts should be identical except for the intervention being tested. Otherwise, a method can look robust simply because the contaminated run also has more useful post-shift evidence, or it can look fragile because the contaminated context changes budget, ordering, or formatting. When possible, the injected stale item should be strong under its original regime: a high-return demonstration, a confident value estimate, a retrieved near-neighbor, or a concise verbal rule. Weak stale evidence is too easy to ignore and underestimates the risk faced by long-context and retrieval-augmented agents. This paired-design principle is the context analogue of oracle-change and controlled-ablation baselines in changepoint, non-stationary regret, and ICRL reporting [20], [34], [65].

These interventions also define a practical “non-stationary ICRL-ready” benchmark checklist. A benchmark is ready for this survey’s claims only if it can freeze the policy, expose the agent to within-lifetime shifts, keep old context available after the shift, inject or remove specific context items, control context budget, and report paired lifecycle metrics. It should also support oracle variants: oracle-current-mode context, oracle-change-point truncation, oracle retrieval, and oracle summary. Without these controls, a benchmark may still be useful for generalization or continual learning, but it cannot isolate whether a frozen policy adapts through context under stale evidence.

8.6 Baselines That Matter↩︎

The most informative baselines are often simple. A recent-only context baseline tests whether long-term memory is helping or hurting. A full-history baseline tests whether selective retrieval matters. A no-context baseline tests whether performance is static generalization. An oracle-change-point baseline estimates the value of knowing when to forget. A stale-context adversary estimates how badly the model can be misled by old evidence. For retrieval methods, random retrieval and similarity-only retrieval should be compared with any learned validity-aware retrieval method. The minimum reporting items in Table 14 are intended to make these baselines and ablations comparable across papers.

8.7 Distinguishing Adaptation from Memorization↩︎

Non-stationary evaluation should rule out three shortcuts. The first is memorization of task IDs or environment templates. The second is recency-only adaptation, where the model ignores long context and simply follows the last few rewards. The third is retrieval shortcutting, where a memory database contains near-duplicates of test tasks. Countermeasures include held-out generators, changed task order, unseen shift schedules, adversarial stale-memory insertion, and explicit tests of recurring modes where recency alone is insufficient.

8.8 A Practical Evaluation Stack↩︎

A practical stack should include at least three layers. The first layer is a controlled diagnostic layer. AnyMDP [31], MiniGrid [220], BabyAI [225], and Alchemy-style tasks [226] are appropriate here because the evaluator can exactly control which part of the decision process changes. This layer should be used for mechanism claims: reward switches, transition switches, action remapping, hidden rule changes, and context contamination. The experiments should be small enough that oracle baselines are meaningful, including oracle-change-point, oracle-retrieval, and oracle-current-mode policies. Generalization benchmarks should be used with the same caution: a held-out level or task family tests extrapolation, whereas non-stationary ICRL requires within-lifetime invalidation and recovery [46], [93], [216].

The second layer is a scalable procedural layer. XLand-MiniGrid and XLand-100B [69], [214], Kinetix [215], PHYRE/I-PHYRE/DeepPHY [217]–[219], Procgen [25], Craftax [221], MiniHack/NLE [222], [227], and Jumanji [223] are natural candidates because they support many task instances while remaining programmable enough to impose explicit non-stationary schedules. This layer should test whether a method that works on clean diagnostics still works when observations are richer, horizons are longer, rewards are sparser, and the task distribution has many modes. It should also test context efficiency: if a method only succeeds by appending all experience, it may not be viable once the environment family becomes large.

The third layer is a memory and continual-protocol layer. POPGym [228], POPGym Arcade [26], NS-Gym [42], CRLMaze [224], and CORA [137] are useful here because they foreground memory, partial observability, non-stationary streams, and continual-learning metrics. This layer should measure whether the context policy can handle delayed evidence, old-observation poisoning, forward transfer, forgetting, and re-adaptation when old modes return. Unlike the first two layers, this layer is less about task diversity and more about diagnosing the lifetime of information.

An optional extended layer can include interactive language environments such as ScienceWorld [156], WebShop [157], and ALFWorld [158] when the paper explicitly studies LLM reward-feedback agents. These environments are useful for testing preference drift, natural-language feedback, tool-use histories, and changing user instructions. However, they should remain extended cases unless the experiment contains repeated decisions, external feedback, and a clear mechanism by which future behavior improves through context. Otherwise, the result is better described as conversational memory or prompting rather than ICRL. For LLM-agent bridge cases, a non-stationary protocol should vary user preferences, reward feedback, hidden instructions, tool schemas, website distributions, or feedback reliability across repeated trials. Baselines should include no-memory, last-\(k\) context, full transcript, summary memory, retrieval memory, oracle-current-rule context, and stale-memory adversary variants. Metrics should include post-shift success rate, correction latency, invalid tool-call rate, stale-memory sensitivity, and preference compliance after drift.

Across all layers, the stack should be reported as a lifecycle evaluation rather than as a single leaderboard. The paper should state the shift source, temporal pattern, observability, context budget, and reset rule for every environment. It should also state whether the environment is intrinsically non-stationary or merely a programmable substrate used to construct a non-stationary protocol. This distinction is important for review: success on a broad procedural benchmark is not automatically success on non-stationary ICRL unless stale context is present and recovery must occur through context.

8.9 Interpreting Evaluation Outcomes↩︎

Evaluation should be diagnostic enough to explain why a method succeeds. High return with high stale-context sensitivity suggests that the method is powerful but brittle. Low stale-context sensitivity with poor recurring-mode performance suggests over-forgetting. Good recovery with poor dynamic regret may indicate that the method eventually adapts but pays a large cost after every change. Strong oracle-change-point performance and weak learned-change-point performance indicates that the main bottleneck is detection or context management rather than policy capacity.

The conclusion is that non-stationary ICRL evaluation should not end with a single return number. A convincing result must identify which failure mode was solved: recognizing that the regime changed, selecting evidence from the current regime, retaining useful evidence from older recurring regimes, or acting well after the relevant context has been selected. The strongest evidence is therefore a consistent pattern across return, dynamic regret, recovery time, stale-context sensitivity, and oracle gaps. If a method improves return but remains sensitive to stale context, the claim should be limited to better exploitation under favorable histories. If it closes the learned-to-oracle change-point gap, the claim can be about change detection or context filtering. If it improves recurring-mode performance without hurting post-shift recovery, the claim can be about useful long-term memory. This interpretation discipline keeps benchmark results tied to the central question of the survey: whether a frozen policy can use context to adapt under changing decision processes, not merely whether it performs well on a diverse task suite.

9 Theory and Conceptual Links↩︎

9.1 Existing Theoretical Coverage↩︎

Existing theory has begun to make ICRL more than an analogy between prompting and reinforcement learning. Supervised pretraining analyses show that decision transformers can implement in-context policies under structured task distributions [8], [50]. General ICL theory supplies the broader mechanisms: transformers can learn simple function classes in context [14], [15], encode implicit models or gradient-descent-like computations inside the forward pass [16], [19], and depend strongly on demonstration structure [12], [18]. Recent work studies chain-of-thought or explicit iterative computation as a path toward in-context reinforcement learning [117]. Other work shows that transformers can implement policy-improvement-like procedures in context [116]. These results are important because they show that ICRL is not merely an empirical analogy: under certain assumptions, the architecture can represent update-like computations. Adjacent non-stationary ICL theory [139], while not reinforcement learning, also supports the relevance of recency, adaptive filtering [182], gating [183], [184], retention-style sequence models [185], and recurrent attention mechanisms [186] when the target function changes over time. Long-context ICL studies provide a complementary caution: increasing the number of examples can help only when the model can keep the relevant evidence accessible and distinguish it from distractors [166], [167].

9.2 Added Challenges Under Non-Stationarity↩︎

Non-stationarity adds two theoretical problems that are easy to blur together. The comparator problem asks what the agent should be compared against when the optimal policy changes over time. Dynamic regret is the natural answer in many settings, including adversarial or bandit settings [21], [35], [82], adaptive-window or weighted bandits [77], [83], [148], non-stationary MDPs [36], [75], [76], refined MDP bounds [62], [66], [149], and transformer-based analyses [65]. The validity problem asks which context items should influence the current policy. This part is less developed theoretically. Most in-context theory assumes that examples are sampled from a coherent latent task or distribution. Non-stationary ICRL requires theory for contexts that are finite, ordered, partially invalid, and sometimes contradictory. POMDP [44], CMDP [37], epistemic-POMDP [46], task-inference [41], and HiP-MDP theory [38], [45] suggest one route: treat the current regime as latent and context as evidence about that regime. The missing piece is that an ICRL agent observes this evidence through a bounded and possibly learned context channel, so inference error and context-selection error become part of the control problem. One useful decomposition is to separate environmental movement from context-management error: \[\begin{align} \mathrm{Reg}^{\mathrm{dyn}}_T(\pi_\theta,g) \leq{}& \underbrace{\mathcal{A}(T,B_T)}_{\mathrm{move}} + \underbrace{\sum_{t=1}^{T}\epsilon_{\mathrm{ctx}}(t;g)}_{\mathrm{ctx}} + \underbrace{\sum_{t=1}^{T}\epsilon_{\mathrm{model}}(t;\theta)}_{\mathrm{model}} . \end{align} \label{eq:regret-decomposition}\tag{18}\] Here \(B_T\) is a variation budget: the movement term captures adaptation to environmental drift, the context term captures retrieval, compression, or stale-context mistakes, and the model term captures the approximation limits of the frozen decision model. This is not a theorem as stated; it is a target form for future theory that would make context-management assumptions explicit. The main gap is therefore an interface between ICRL theory, non-stationary RL regret analysis, and retrieval or memory theory: the target object is not only a changing MDP, but a changing MDP observed through a bounded context channel.

9.3 Open Theoretical Questions↩︎

Several questions would make the theory more precise:

• Which variation budgets or change-point assumptions make fixed-parameter context adaptation competitive with online updating?

• How should regret bounds scale with context length, memory size, retrieval error, and shift observability?

• When is stale context harmless because attention can suppress it, and when does it induce unavoidable lower bounds?

• How do value-aware and Bayesian context mechanisms alter sample complexity under reward or dynamics shift?

• Can recurring-mode memory be given a formal advantage over recency-only adaptation?

Table 15 maps existing theory topics to the non-stationary pieces that remain missing.

A useful first step is to make the underlying assumptions explicit:

• Under bounded variation in rewards or dynamics, theory could characterize how regret scales with context length.

• Under a finite set of recurring latent regimes, theory could identify when retrieval reduces re-adaptation latency.

• Under adversarial stale context, theory could characterize robustness guarantees for attention or retrieval policies.

These stylized settings would not capture the full complexity of deployment, but they would turn vague claims about “using context” into testable statements about when context helps and when it provably hurts.

10 Design Principles and Open Problems↩︎

10.1 Design Principles↩︎

The preceding sections point to several design principles for non-stationary ICRL systems.

1. Treat context as a managed resource. Longer context is useful only when the system can identify what remains valid, a lesson shared by long-memory LLM systems [113]–[115], long-context ICL studies [166], [167], efficient sequence models [102], and long-context ICRL agents [101].

2. Train on invalidation, not only adaptation. Training histories should include moments where old evidence becomes wrong, not only stationary improvement traces from algorithm distillation [2], noise or history filtering [3], [68], continual or suboptimal ICRL [32], [61], or environment design [206], [207].

3. Separate recency from relevance. Recency is a useful heuristic in sliding-window and discounted methods [35], [77], [81], but it fails under changepoints and recurring modes [34].

4. Expose retrieval to stale-memory tests. Retrieval should be evaluated by validity, not only similarity or return, in retrieval-based RL [54], [55] and retrieval-augmented ICRL [33], [53].

5. Report lifecycle curves. A single average return hides post-shift failure and recovery in non-stationary MDPs [62], [75], [76] and continual ICRL [61].

6. Include oracle context baselines. Oracle-change-point baselines [34] and general ICRL reporting baselines [20] estimate the headroom available from better context management.

7. State the adaptation channel. Papers should make clear whether adaptation occurs through prompt tokens, hidden state, external memory, retrieved trajectories, summaries, or online parameter updates.

10.2 Open Problems↩︎

Not every item in the research agenda is open in the same sense. Some topics already have direct starting points, such as continual ICRL benchmarks and metrics [61], dynamic-regret analyses for non-stationary transformers [65], history filtering [68], retrieval-augmented ICRL [33], [53], and safe ICRL mechanisms [29], [74]. Other topics are seeded only by adjacent literatures, such as changepoint detection, continual learning, retrieval, or safe RL, and still need to be reformulated under the fixed-parameter context-adaptation constraint. A third group is more genuinely open: the field lacks clear benchmarks, formal objectives, or accepted measurement protocols. Table [tab:open-problems] therefore distinguishes seeded directions from partly seeded and truly open ones.

Safety is a related but distinct pressure on this agenda. Safe ICRL extends the adaptation problem to constrained decision processes [237]–[239], where context must help the agent improve reward while respecting cost budgets [29]. Recent shielding work makes the same point more explicit under OOD deployment shifts: Q-Barrier [74] suggests that a frozen safe-ICRL policy may need an action-level latent budget and cost filter, not only a longer history, to preserve a useful reward-safety tradeoff. The non-stationary version is harder because a context item may be reward-valid but safety-invalid after a constraint or cost shift. This reinforces the need to report validity with respect to rewards, dynamics, and constraints separately. Classical safe exploration [240], model-based safety guarantees [241], safe policy improvement [242], and modern safe-RL benchmarks [243] provide useful components, but they still need to be recast under the fixed-parameter, context-only adaptation constraint.

10.3 A Research Agenda↩︎

The next phase of the field should move beyond the broad question of whether context helps. For non-stationary ICRL, the sharper questions are which context helps, when it helps, when it hurts, and how a frozen agent can know the difference. The roadmap should proceed in stages. The first stage is consolidation: seeded directions such as continual ICRL benchmarks [61], safe ICRL [29], [74], and retrieval-augmented ICRL [33], [53] should be evaluated under a common lifecycle protocol rather than treated as isolated demonstrations. The second stage is translation: adjacent tools from changepoint detection, non-stationary regret, continual learning, retrieval, and safe RL should be rewritten in context-channel terms, specifying what is stored, retrieved, suppressed, or exposed to the frozen model. The third stage is genuinely open theory and stress testing: finite-context bounds, adversarial stale-context threat models, and validity-aware memory metrics should become standard targets.

Claims about non-stationary adaptation should specify the shift source, temporal pattern, observability regime, context policy, and training exposure. They should report post-shift curves and stale-context ablations, compare against simple context policies before attributing gains to architecture scale, and state whether a result is solving a new open problem or extending an already seeded line.

The agenda also requires a cleaner separation between the pretrained decision model, the context manager, and the memory store. Current papers often blur these components together. In stationary settings, that may be tolerable because most useful context points in the same direction. In non-stationary settings, the distinction is central. A strong policy with a weak context manager can fail, while a modest policy with a good validity filter can recover quickly.

10.4 Reporting Standards for a Mature Literature↩︎

A mature non-stationary ICRL literature would make claims at the right level of specificity. Instead of saying that a model handles non-stationarity, a paper would say that it handles reward/abrupt/partial shifts with a finite context budget, or dynamics/recurring/latent shifts with an external memory. Instead of reporting only average return, it would show lifecycle curves, recovery time, stale-context sensitivity, and oracle-context headroom. Instead of treating retrieval as a black-box improvement, it would report whether retrieved items are valid under the current regime. Instead of presenting long context as inherently beneficial, it would show when additional context helps, when it hurts, and what mechanism controls its influence.

This reporting standard is demanding, but it is aligned with the central promise of ICRL. If adaptation is supposed to happen without weight updates, then the evidence passed to the frozen model is the adaptation interface. The field should therefore scrutinize that interface with the same care that reinforcement-learning papers apply to rewards, dynamics, and exploration.

11 Conclusion↩︎

This survey has argued that non-stationary In-context reinforcement learning is not just ICRL evaluated on a larger or more diverse task distribution. It is the problem of adapting through context while deployed policy parameters remain fixed, even when accumulated evidence becomes obsolete, misleading, or useful again after a regime returns. The key difficulty is context validity: a fixed-parameter policy must infer which accumulated evidence still describes the current decision process and which evidence should be downweighted, retired, or reactivated. This view connects black-box meta-RL [1], [4], decision sequence modeling [7], [8], algorithm distillation [2], retrieval-augmented agents [33], [53], and long-context adaptive systems [101], but it shifts the organizing question from “does context help?” to “when should context be trusted?”.

From this perspective, the survey made five contributions to the non-stationary ICRL problem. First, it defined the setting and separated it from adjacent notions such as stationary few-shot adaptation, domain randomization, online continual learning, standard offline RL, robotics-specific lifelong learning, and multi-agent strategic non-stationarity. The defining constraint is that adaptation occurs through context while the deployed policy parameters remain fixed. Second, it introduced a taxonomy that describes a non-stationary ICRL setting along three axes: what changes, how the change unfolds over time, and how observable the change is to the agent. This source–pattern–observability vocabulary is intended to prevent overbroad claims: reward specifications, transition kernels, action interfaces, observation channels, constraint models, and data sources create different problems, and abrupt, gradual, recurring, expanding, or adversarial changes require different evidence-management behavior. Third, the survey recast methods as context-management mechanisms rather than only as model families. Raw history conditioning, context filtering, retrieval, compression, recurrent memory, value-aware inference, model-based prediction, reflection, and forgetting are all ways of deciding which past evidence reaches the frozen decision policy and how strongly it should influence action. Under non-stationarity, these mechanisms are not engineering details; they are the adaptation channel. Fourth, the survey translated that view into training and evaluation requirements. Training data should contain invalidation, mixed pre-shift and post-shift histories, stale demonstrations, suboptimal or mismatched trajectories, recurring modes, and finite-context pressure [32], [68]. Evaluation should then report lifecycle curves, post-shift recovery, dynamic regret, stale-context sensitivity, retrieval utility, oracle-context gaps, recurring-mode reuse, and finite-context degradation rather than only average return. Benchmarks such as procedural environments, continual RL suites, and memory-focused tasks become evidence for non-stationary ICRL only when they impose within-lifetime shifts and require recovery through context. Fifth, the survey identified the theoretical gap created by stale and conflicting context. Existing change-point methods [34], non-stationary bandit regret [21], non-stationary MDP regret [75], [76], and continual ICRL [61] provide useful pieces, but they do not yet explain how regret, sample efficiency, or robustness should scale with context length, retrieval error, memory size, shift observability, and context poisoning. A mature theory of non-stationary ICRL should treat the context manager as part of the control system, because the agent acts through a bounded and imperfect evidence channel.

The practical conclusion is that future work should make its claim at the level of the shift it actually solves. A method that improves return under clean histories is not yet a non-stationary ICRL solution if it fails under stale context. A method that recovers after abrupt reward changes has not necessarily solved gradual dynamics drift or hidden action remapping. A method that forgets quickly may adapt after one-way change but fail when old regimes recur. Progress in non-stationary ICRL will come from systems that can detect change, select currently valid evidence, preserve reusable memories, reject obsolete context, and act well under a finite context budget. The central benchmark for the field is therefore not larger context alone, but disciplined evidence use across the lifetime of a changing decision process.

References↩︎

---

1. Corresponding author: ziluo@pku.edu.cn↩︎