1 Introduction↩︎

Designing effective reward functions remains a major challenge in reinforcement learning (RL) [1]–[4]. This challenge is especially pronounced in open-ended or visually complex environments, where task goals are often abstract and difficult to specify precisely. For example, in open-world environments such as Minecraft, agents are often assigned high-level goals (e.g., “build a shelter”) that require long-horizon planning and semantic understanding, while intermediate states provide little explicit reward signal. As a result, hand-crafting reward functions in such settings is typically impractical, due to ambiguous objectives, sparse feedback, and the absence of privileged state information. To address this bottleneck, recent approaches have explored leveraging large pre-trained models as sources of reward, enabling agents to learn from high-level task descriptions or multimodal alignment signals.

Large language models (LLMs) have been shown to provide structured code or textual feedback for RL agents in text-based or programmatic settings. However, these methods often rely on privileged state access or handcrafted programmatic interfaces, limiting their applicability in real-world visual domains. When working with visual observations, contrastive vision-language models (cVLMs) such as CLIP are widely used to compute similarity scores between observations and goals for reward shaping. While simple and scalable, these methods typically require task-specific fine-tuning or retraining to be effective across different domains.

To move beyond direct similarity-based alignment, preference-based reward modeling has emerged as a promising alternative. These methods learn reward functions by comparing observation pairs and inferring preferences that are typically obtained from human feedback or large pre-trained models. This formulation enables finer-grained feedback and greater robustness in visually diverse or ambiguous environments. For instance, RL-VLM-F [5] leverages generative vision-language models (VLMs) to generate pairwise preferences, which are then used to train a reward model that assigns scalar values to individual states. We call this approach absolute reward. However, we observe that absolute reward signals can be inconsistent across training steps (Fig. 1), due to the continual exposure to newly encountered states as training progresses. These inconsistencies, in turn, make long-horizon tasks difficult to optimize. In particular, absolute reward formulations fundamentally collapse in cyclic task structures or under ambiguous state orderings. In these scenarios, progress cannot be uniquely defined by a global state ordering, a limitation illustrated in Fig. 2.

To address these limitations, we propose VLM-AR3L (Fig. 3), a framework that leverages VLMs to provide both absolute and relative rewards in reinforcement learning, without requiring fine-tuning or access to privileged internal environment information. Our contributions are as follows:

• Dual-reward Design: We introduce a dual-reward framework that jointly learns absolute and relative reward from VLM-generated preferences, with the relative reward providing robustness to reward inconsistency during training (Fig. 1) and resolving tasks with ill-defined global state ordering (Fig. 2).

• Relative reward modeling: We train a Siamese network with VLM-generated preference supervision to model relative reward, avoiding repeated VLM inference during policy optimization, which substantially reduces computational overhead and improves performance.

• Comprehensive evaluation: We evaluate VLM-AR3L across diverse domains, including structured robotic control and open-ended tasks. VLM-AR3L consistently outperforms prior VLM-based methods and even surpasses carefully human-designed reward functions. We further conduct detailed ablation studies and VLM accuracy analyses to assess the reliability and scalability of our approach.

2 Related Work↩︎

2.0.0.1 LLM-based Reward Learning.

Recent research has explored the use of large pre-trained models, particularly large language models (LLMs), as reward functions for reinforcement learning agents. LLM-based approaches have demonstrated the ability to generate structured code for downstream training [6]–[10], or to provide scalar reward signals from textual task descriptions in language-conditioned or text-based environments [11]–[13]. However, many of these methods assume structured environments or require privileged state access, making them difficult to apply in high-dimensional, real-world visual domains. Additionally, for many open-ended or complex tasks, it is inherently challenging to specify reward logic in code, particularly when task-specific abstractions are involved or when full observability is lacking.

2.0.0.2 CLIP-based Reward Learning.

When working with visual observations, a common strategy is to use CLIP-based vision-language models (cVLMs) to compute similarity-based scores between agent observations and language goals, typically via cosine similarity in the shared embedding space [14]–[21]. While conceptually simple, these scores are often noisy and fragile, exhibiting high sensitivity to the phrasing of goals. To mitigate this, methods such as MineDojo [22] and CLIP4MC [23] fine-tune CLIP on domain-specific datasets (e.g., Minecraft videos) to generate denser and more accurate reward signals. Other approaches like LIV [24] and FuRL [25] propose tailored fine-tuning procedures to enhance reward quality. While these methods reduce dependence on explicit ground-truth reward annotations, they typically require task-specific fine-tuning to capture subtle variations in visual input. Without fine-tuning, CLIP-based methods often fail to provide nuanced or reliable reward signals, limiting their generalization to new tasks or environments.

2.0.0.3 Preference-based Reward Learning.

Beyond similarity-based rewards, recent works explore preference-based reward learning, which uses generative VLMs with multimodal reasoning capabilities, such as Gemini or GPT. A notable direction involves generating preference-based supervision without human labels. For instance, Constitutional AI [26] and Motif [27] use LLMs to produce preference labels or intrinsic rewards by comparing model outputs under high-level alignment principles. Extending this paradigm to the visual domain, RL-VLM-F [5] proposes learning reward functions by querying vision-language models to compare agent observation pairs and generate preference labels, which are then used to supervise reward learning. Several follow-up works further explore the use of VLMs for visual preference modeling and feedback generation in various embodied environments [28]–[30]. However, these methods largely adopt absolute reward formulations, which are fundamentally limited in cyclic task structures or under ambiguous state orderings, where a consistent global ordering of states is difficult to define. Motivated by this limitation, we explore a dual-reward formulation that augments absolute rewards with relative rewards, aiming to improve robustness and scalability in complex, open-ended environments.

3 Background↩︎

We consider a standard reinforcement learning (RL) setting modeled as a partially observable Markov decision process (POMDP) [31], defined by the tuple \((\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{O}, \mathcal{R}, \gamma)\). Here, \(\mathcal{S}\) denotes the set of environment states, \(\mathcal{A}\) denotes the set of possible actions, \(\mathcal{T}\) denotes the (unknown) state transition dynamics, \(\mathcal{O}\) denotes the observation space, \(\mathcal{R}\) denotes the reward function, and \(\gamma \in [0, 1)\) denotes the discount factor that prioritizes immediate rewards over long-term ones. At each time step \(t\), the agent receives an observation \(s_t \in \mathcal{O}\) (e.g., an RGB image) and selects an action \(a_t \in \mathcal{A}\), which causes the environment state in \(\mathcal{S}\) to evolve according to \(\mathcal{T}\). The objective is to learn a policy \(\pi\) that maximizes the expected cumulative discounted reward over time.

3.0.0.1 Preference-Based Reinforcement Learning.

Our work builds on the framework of preference-based reinforcement learning (pbRL), where agents learn reward functions from pairwise comparisons over trajectory segments or behavior snippets [32]–[35]. In pbRL, a trajectory segment \(\sigma\) typically consists of a sequence of states \(\{s_1, \dots, s_H\}\), where \(H\) denotes the sequence length of the segment. In this paper, we consider the special case that \(H = 1\), as multi-image or video-based VLM reasoning remains less reliable in complex visual environments. Given a pair of segments \((\sigma_i, \sigma_j)\), an annotator provides a preference label \(y \in \{0, 1, -1\}\), where \(y = 0\) indicates that \(\sigma_i\) is preferred, \(y = 1\) that \(\sigma_j\) is preferred, and \(y = -1\) that the two are equally preferable or incomparable. Given a reward function \(r\) over the states, we follow the standard Bradley-Terry model [36] to compute the preference probability of a pair of segments: \[\label{eq:BTM} \mathbb{P}[\sigma_i \succ \sigma_j] = \frac{ \exp\!\left(r(s_i) \right) }{ \sum_{k \in \{i,j\}} \exp\!\left(r(s_k) \right) }.\tag{1}\]

The reward model is then trained by minimizing a cross-entropy loss over a dataset of annotated preferences: \[\begin{align} \label{CELoss} \mathcal{L}_\text{reward}(\theta) = - \mathbb{E}_{(\sigma_i, \sigma_j, y) \sim \mathcal{D}} & \Big[ \mathbb{1}_{\{y = 0\}} \log \mathbb{P}_\theta[\sigma_i \succ \sigma_j] \nonumber \\ & +\, \mathbb{1}_{\{y = 1\}} \log \mathbb{P}_\theta[\sigma_j \succ \sigma_i] \Big], \end{align}\tag{2}\] where \(\theta\) denotes the learnable parameters of the reward model.

In typical pbRL, the reward model and policy are trained iteratively: the reward model is updated using newly collected preference annotations, and the policy is optimized with standard RL algorithms using the learned reward model.

4 Method↩︎

Fig. 3 illustrates the overall architecture of the proposed VLM-AR3L framework. The training process consists of two intertwined pipelines: (1) reward model training and (2) policy learning. In the reward model training phase, observation pairs \((\sigma_i, \sigma_j)\) are sampled from the agent’s replay buffer, along with a high-level language goal \(g\). Each triplet \((\sigma_i, \sigma_j, g)\) is passed to a frozen VLM, which outputs a preference label \(y\) indicating which observation better reflects progress toward the goal. These preference labels are used to supervise the training of both an absolute reward model and a relative reward model. During policy learning, the absolute reward model assigns state-based evaluations to the current observation \(s_t\), while the relative reward model estimates progress by comparing the current observation \(s_t\) with a previous observation \(s_{t-k}\), jointly providing dense and informative reward signals at each timestep. These signals guide the agent’s policy updates using standard reinforcement learning algorithms, eliminating the need for hand-crafted rewards or privileged state information.

4.1 Absolute Reward Model↩︎

Absolute reward models assign scalar values to individual states, where higher scores reflect states that are closer to the goal. As illustrated in Fig. 1, this state-based formulation provides a stable evaluation signal when a global ordering over the state space is well-defined, as each state is mapped to a fixed reward value independent of the agent’s trajectory.

Given a parameterized absolute reward function \(F^{\,\mathrm{abs}}_\psi\) over the states, the model outputs a scalar reward value for each state following the formulation in  Eq. (1 ): \[\label{eq:absolute-reward-fn} F^{\,\mathrm{abs}}_\psi(s_t) = r_\psi(s_t),\tag{3}\] where \(r_\psi(s_t) \in \mathbb{R}\) denotes the absolute reward assigned to state \(s_t\). The absolute reward at time step \(t\) is then given by: \[\label{eq:absolute-reward} R^{\,\mathrm{abs}}_\psi(s_t) = F^{\,\mathrm{abs}}_{\psi}(s_t).\tag{4}\]

4.2 Relative Reward Model↩︎

In contrast to the absolute reward model, the relative reward model evaluates progress between pairs of states. This design reduces reliance on a global understanding of the task space and improves robustness in partially observable or long-horizon scenarios. When new states are introduced and the reward models are updated, the relative reward of previously seen transitions remains stable. In contrast, absolute reward often suffers from instability, where earlier states may receive drastically different scores due to shifts in global normalization or value scaling (Fig. 1). Such robustness is especially beneficial in long-horizon settings, where consistent reward signals are essential for learning.

To implement the relative model, we employ a Siamese neural network [37] to learn a relative function \(F^{\,\mathrm{rel}}_\phi\). Given a pair of states \((s_i, s_j)\), the network outputs the preference probability: \[F^{\,\mathrm{rel}}_{\phi}(s_i, s_j)=\mathbb{P}_\phi[\sigma_i \succ \sigma_j].\] The relative model is trained using the same loss defined in Eq. (2 ). Architectural and training details are provided in the supplementary material (Appendix 8.1).

During policy learning, the relative model aims to assess whether a transition indicates progress toward the goal \(g\). To improve the model’s sensitivity to meaningful changes, particularly when consecutive observations are highly similar, we introduce a temporal offset \(k \geq 1\) and compare the current observation \(s_t\) with a past observation \(s_{t-k}\). To ensure the reliability of the model’s predictions, we apply a symmetric confidence check. Specifically, a positive reward is assigned only if the model’s confidence in preferring \(s_t\) over \(s_{t-k}\) exceeds a threshold \(\tau\), and the reverse prediction also indicates that \(s_{t-k}\) is not preferred over \(s_t\) with confidence exceeding \(\tau\). This bidirectional check ensures both strong and consistent preference. Formally, the relative reward at time step \(t\) is computed as: \[\label{eq:relative-reward} R^{\,\mathrm{rel}}_\phi(s_t, s_{t-k}) = \begin{cases} \;\;1, & \text{if } p^+_t > \tau \;\text{and} \;p^-_t < 1-\tau, \\ -1, & \text{if } p^-_t > \tau \;\text{and} \;p^+_t < 1-\tau, \\ \;\;0, & \text{otherwise}, \end{cases}\tag{5}\] \[\text{where } \quad p^+_t = F^{\,\mathrm{rel}}_{\phi}(s_t, s_{t-k}), \quad p^-_t = F^{\,\mathrm{rel}}_{\phi}(s_{t-k}, s_t).\]

4.3 Dual-Reward Design↩︎

Absolute and relative reward models offer complementary perspectives, balancing their individual strengths and weaknesses. Absolute rewards provide stable, state-based evaluations when a global ordering of the state space is well-defined, whereas relative rewards offer robust, transition-level supervision that remains reliable in cyclic, long-horizon, or partially observable settings. Motivated by this complementarity, we combine both reward signals to guide policy learning. The dual-reward is computed as: \[\label{eq:combined-reward} R^{\,\mathrm{dual}}_{\psi,\phi}(s_t, s_{t-k}) = \alpha R^{\,\mathrm{abs}}_\psi(s_t) + (1-\alpha) R^{\,\mathrm{rel}}_\phi(s_t, s_{t-k}),\tag{6}\] where \(\alpha\) is the weighting coefficient for the absolute reward signals, and both absolute and relative rewards are normalized to the same bounded range. Although the framework learns two reward models, both are trained using a single shared VLM querying pipeline. The VLM prompting strategy is described in Appendix 7.1.

5 Experiments↩︎

5.1 Setup↩︎

5.1.0.1 Environments and Tasks.

To assess the proposed VLM-AR3L, we conduct evaluations on ten tasks across four distinct environments (Fig. 4): Cart Pole from Gym [47]; Straighten Rope and Pass Water from SoftGym [48]; Soccer, Sweep Into, and Drawer Open from MetaWorld [49]; and Combat Spider, Milk Cow, Shear Sheep, and Hunt Cow from MineDojo [22]. MineDojo, in particular, features long-horizon, open-ended tasks built on top of Minecraft. These tasks demand multi-step reasoning, complex object interaction, and semantic goal understanding, thereby posing significant challenges for visual reward learning. Collectively, these benchmarks encompass both structured robotic manipulation and expansive open-world scenarios, facilitating a comprehensive evaluation of VLM-AR3L’s generalization and adaptability. Detailed descriptions of the environments and tasks are provided in the Appendix 9.

5.1.0.2 Baselines.

We evaluate our approach against several baselines that, like our framework, generate reward functions using only text goals and visual observations, without relying on privileged internal environment state information. In addition, we include oracle baselines using ground-truth rewards to assess whether VLM-generated rewards can serve as practical alternatives to human-designed reward functions.

• RL-VLM-F [5]: A state-of-the-art preference-based method that queries a VLM for pairwise preferences between observations to train a scalar-valued reward function over individual states.

• CLIP [16]: A similarity-based method using cosine similarity between CLIP embeddings of the current observation and the language goal for reward shaping.

• MineCLIP [22]: A domain-adapted CLIP variant fine-tuned on Minecraft videos, as proposed in MineDojo. Results are not directly comparable to the original paper, as self-imitation learning is omitted to ensure a fair and consistent training protocol across baselines.

• GT Dense Reward (Oracle): Hand-crafted, incremental reward functions based on full access to environment ground truth.

• GT Sparse Reward (Oracle): Binary success/failure signals given only at episode completion.

We exclude FuRL [25], RoboCLIP [17] and CLIP4MC [23] from our comparisons for the following reasons: FuRL relies on privileged reward signals, which falls outside our problem setting; RoboCLIP has been reported to perform poorly under comparable assumptions (as reported in RL-VLM-F); and CLIP4MC does not release its fine-tuned CLIP model, precluding an identical reproduction. Implementation details for all baselines are provided in Appendix 10. For policy optimization, we employ Soft Actor-Critic (SAC) [50] for Gym, SoftGym, and MetaWorld tasks, and Proximal Policy Optimization (PPO) [51] for MineDojo tasks, selected based on environment characteristics. Comprehensive training configurations and hyperparameters are detailed in the Appendix 8.2.

5.2 VLM’s Label Accuracy↩︎

To assess the reliability of using VLMs as supervision, we evaluate several VLMs on multi-image reasoning across our tasks. For each task, we construct 100 image pairs with balanced pseudo preference labels (a 50/50 split) and prompt each VLM to identify the image that better aligns with the task goal.

Table 1 summarizes the accuracy of the tested VLMs across tasks. This evaluation serves to identify which models are most suitable as automated annotators for learning relative rewards. Gemini-2.0-Flash stands out for its consistently superior performance, achieving over 70% accuracy in all tasks, reflecting its robust multimodal reasoning capabilities. In contrast, GPT-4.1-nano underperforms relative to Gemini, showing noticeably lower accuracy across most tasks. Besides Gemini and GPT models, several open-source VLMs demonstrate competitive performance on specific tasks despite lacking general robustness. For example, Phi-3.5-Vision-Instruct achieves 90% accuracy on the Milk Cow and Shear Sheep tasks, while DeepSeek-VL2-Tiny excels in the Combat Spider task with an accuracy of 85%. MiniCPM-o-2.6 performs well in Pass Water, reaching 73%. On the other hand, CLIP and MineCLIP (evaluated only in MineDojo tasks) exhibit weak performance, underscoring their limitations in modeling multi-step interactions and temporal dependencies. Additional details and expanded evaluation results are provided in the Appendix 7.2.

5.3 Comparison with Baselines↩︎

Building on the evaluation in Section 5.2, we adopt Gemini-2.0-Flash as the primary VLM for both VLM-AR3L and RL-VLM-F, given its consistently superior annotation performance across tasks. Table 2 reports the best success rates of all evaluated methods, and Fig. 5 presents the corresponding learning curves. We first examine similarity-based baselines such as CLIP and MineCLIP. While CLIP performs well in the low-dimensional Cart Pole environment, it struggles significantly in more complex environments. Notably, MineCLIP fails to generate reliable reward signals despite being fine-tuned on Minecraft videos. These results align with prior findings [5], [23], [25], suggesting that CLIP-based models often lack the granularity necessary to distinguish subtle visual differences, resulting in noisy and unstable reward signals during training.

Next, we evaluate RL-VLM-F, a preference-based method that learns scalar rewards over individual states. While RL-VLM-F achieves performance comparable to our method on simpler tasks, its efficacy degrades substantially in long-horizon and visually complex domains such as MineDojo. We attribute this degradation to its reliance on absolute state-based rewards, which requires a global understanding of the task space and often lacks robustness to novel or unseen state variations. In contrast, our proposed VLM-AR3L leverages relative comparisons between transitions, enabling it to generate more stable and goal-consistent rewards even as the state distribution evolves during training.

Finally, when compared against ground-truth dense and sparse rewards, our VLM-AR3L shows strong potential as a practical alternative. In most tasks, VLM-AR3L matches the performance of GT Dense Reward. Moreover, in tasks such as Milk Cow, Shear Sheep, and Hunt Cow, where GT Sparse Reward fails to provide sufficient guidance for learning, VLM-AR3L significantly outperforms the baseline, demonstrating its ability to handle sparse feedback and challenging long-horizon scenarios without requiring handcrafted rewards. These results underscore the practical applicability of our method for real-world settings where reward design is often infeasible.

5.4 Ablation Studies↩︎

5.4.0.1 The Impact of VLM.

While Section 5.2 evaluates VLMs as annotators in terms of preference-label accuracy, we further study how the choice of VLM affects end-to-end RL performance when used to generate training supervision for VLM-AR3L. Specifically, we instantiate the same training pipeline and only vary the annotator VLM (Gemini-2.0, Phi-3.5, MiniCPM-o 2.6) across all tasks (Fig. 6 ; the full figures and table are provided in the Appendix 11.2). Overall, the performance trends are broadly consistent with the label accuracy results in Section 5.2. While Phi-3.5 and MiniCPM-o 2.6 do not excel across all tasks, both exhibit clear task-specific strengths. Given their relatively small model sizes and open-source availability, they offer promising potential for broader adoption in future applications.

5.4.0.2 The Impact of Relative Reward Architecture.

An alternative design choice for relative reward, instead of explicitly training a Siamese model (denoted as Siamese in Fig. 7), is to directly query a VLM at each timestep to evaluate progress (denoted as VLM-Direct). Another baseline derives relative rewards based on differences between absolute reward predictions (denoted as Abs-Diff). As shown in Fig. 7, explicitly training a Siamese model consistently outperforms direct VLM querying during rollouts (VLM-Direct). This improvement stems from preference-based training, which reduces noise in VLM judgments by learning a more stable comparison function, while also lowering computational overhead by avoiding repeated VLM queries at inference time (see Appendix 8.3). We observe that computing relative rewards from absolute reward estimates (Abs-Diff) leads to inferior performance. This is because when the absolute reward model fails to assign reliable scores to individual states, the resulting differences become noisy and unreliable, leading to inaccurate relative assessments. We also examine the role of the symmetric comparison rule in Eq. (5 ) by evaluating a variant that omits this consistency check (Siamese w/o Sym). Removing the rule leads to notable performance drops across tasks, as it no longer filters inconsistent VLM judgments or prevents biased rewards when observations are visually similar.

5.4.0.3 The Impact of Absolute and Relative rewards.

Fig. 8 (Full figures are provided in the Appendix 11.3) analyzes how absolute and relative rewards contribute to each task. Absolute rewards are most beneficial in tasks with a well-defined global progress measure. In contrast, relative rewards become critical in tasks with ambiguous state ordering, cyclic structure, or long-horizon exploration. Combining the two consistently leads to further improvements, indicating that the reward signals are complementary.

5.4.0.4 The Impact of Hyper-parameters.

All results above are obtained using a single shared set of hyperparameters, with \(\alpha = 0.5\), \(\tau = 0.52\), and \(k = 16\), without task-specific tuning. We further confirm this robustness through hyperparameter ablation studies (see Appendix 11.1), which show only minor performance variations.

6 Conclusion↩︎

Our results demonstrate that absolute and relative rewards provide complementary supervision for reinforcement learning with VLM-generated preferences. Absolute rewards are effective when a global notion of progress is well defined, while relative rewards are more robust in scenarios involving ambiguous, cyclic, or long-horizon state transitions. Integrating both reward types leads to more stable optimization and consistently outperforms existing VLM-based reward learning methods. Notably, in several tasks, our approach even surpasses manually engineered human reward functions.

Acknowledgements↩︎

This research was funded by the National Science and Technology Council (NSTC) under grant numbers 114-2218-E-007-019. The last author was partially supported by the Hon Hai Research Institute. We express our sincere appreciation to the NVIDIA AI Technology Center (NVAITC) for the technical support and access to the Taipei-1 supercomputer.

7 VLM↩︎

7.1 Prompts↩︎

To evaluate VLM prompting strategies, we crafted three prompt templates of increasing reasoning complexity:

• S1 — Single Stage

  1. The goal is {task_description}. Is the goal better achieved in Image 1 or Image 2? Relay a single line of 1 if the goal is better achieved in Image 1, or 2 if it is better achieved in Image 2. Reply 0 if the text is unsure or there is no difference.

• S2 — Two Stages

  1. The goal is {task_description}. Is the goal better achieved in Image 1 or Image 2?

  2. Based on the text above, Relay a single line of 1 if the goal is better achieved in Image 1, or 2 if it is better achieved in Image 2. Reply 0 if the text is unsure or there is no difference.

• S2 + CoT — Two Stages with Chain-of-Thought

  1. • What is shown in Image 1?

     • What is shown in Image 2?

     • The goal is {<task_description}. Is there any difference between Image 1 and Image 2 in terms of achieving the goal?

  2. Same as S2 Stage 2.

• S1₂ — Single Stages (Binary)

  1. The goal is {task_description}. Is Image 2 more likely to achieve the goal? Reply a single line of 1 if yes, otherwise 0.

• S2₂ — Two Stages (Binary)

  1. The goal is {task_description}. Is Image 2 more likely to achieve the goal?

  2. Based on the text above, Reply a single line of 1 if yes, otherwise 0.

7.1.0.1 Rationale.

• Single-Stage (S1). The VLM is asked to output only a categorical label. This setting measures the model’s raw decision quality without exposing its reasoning.

• Two-Stage (S2). We first solicit an open-ended answer that explains why the goal is better achieved in one image. The model then self-produces the final label in Stage 2. Recording Stage 1 lets us inspect the model’s internal rationale and assess whether it is aligned with the label it finally assigns.

• Two-Stage + CoT (S2 + CoT). Stage 1 is further broken into three fixed sub-questions (describe Image 1, describe Image 2, compare w.r.t.the goal). This structured Chain-of-Thought (CoT) explicitly guides the model to observe, compare, and reason step-by-step before emitting the label in Stage 2, reducing the chance of shortcut reasoning.

• Binary (S1₂ and S2₂). The VLM is required to output a binary label (0 or 1). Empirically, we find that some models perform better when the prompt is kept as simple as possible.

7.2 VLM accuracy↩︎

7.2.0.1 Evaluation Protocol.

For each task we curate 100 image pairs with pseudo ground-truth preference labels:

• 50 pairs in which Image 2 represents an improvement over Image 1 (label= 1), and

• 50 pairs in which it does not (label= 0).¹

The VLM’s raw prediction \(y\!\in\!\{0,1,2\}\) is post-processed by \[\hat{y}=\max(y-1,\,0),\] so that

• \(y=2\) (Image 2 better) \(\;\longrightarrow\; \hat{y}=1\), and

• \(y\in\{0,1\}\) (unsure/no difference or Image 1 better) \(\;\longrightarrow\; \hat{y}=0\).

We then compare \(\hat{y}\) with the pseudo ground-truth label for accuracy and other downstream metrics.

Table 3 summarizes the performance of different prompt designs evaluated across multiple models. We observed that for Gemini 2.0, even the simplest prompt design (S1) yielded strong performance. In contrast, GPT-4.1-nano performed better with the slightly more elaborate S2 prompt, and showed further improvements in accuracy with Chain-of-Thought prompting (S2 + CoT). Interestingly, Phi-3.5 exhibited the opposite trend in MineDojo-style tasks, favoring simpler prompts—achieving the best results with S1₂, a binary decision format with a single-stage comparison.

As a result, in all subsequent experiments, we adopt the following prompt configuration per model:

• Gemini 2.0 and MiniCPM use S1,

• Phi-3.5 uses S1₂.

Table 4 provides detailed statistics on computational resources and runtime used for evaluating each model.

8 Hyper-parameters and Network Architectures↩︎

8.1 Relative Reward Model (Siamese)↩︎

8.1.1 Architecture↩︎

The overall architecture of the Siamese model is summarized in Table 5. Each image is first processed by a shared CNN encoder that produces a \(128\)-dimensional embedding. The two embeddings are concatenated and passed through a fully connected layer with \(64\) hidden units, followed by a final linear layer that outputs a two-dimensional logits vector indicating the preference direction (e.g., regression vs.improvement). Because the Siamese architecture is inherently symmetric, we relabel \(\,y=-1\) (“equally preferable” or “incomparable”) to \(\,y=0\) during training, resulting in a binary target \(y\in\{0,1\}\).

8.1.1.1 Training Details.

• Loss : cross-entropy on the two-class logits

• Ensemble : three independently initialized Siamese models; their softmax outputs are averaged at inference for greater robustness

• Sampling strategy : uniform from the replay buffer

• Sampling frequency : 50 pairs per 1 000 environment steps

• Buffer capacity : 500 000 transition pairs

• Training frequency : 50 gradient updates per 1 000 environment steps

• Batch size : 32

8.2 Policy model↩︎

8.2.0.1 MineDojo tasks.

• Framework : stable-baselines3 — Proximal Policy Optimization (PPO)

• Network : frozen MineCLIP image encoder \(\rightarrow\) MLP(64, 64)

• Entropy coefficient : \(0.01\)

• Other parameters : default

• Vectorised environment : DummyVecEnv with 4 workers

• Observation wrapper : 4-frame stack

8.2.0.2 Other tasks.

• Framework : modified RL-VLM-F — Soft Actor–Critic (SAC)

• Network : 3 hidden layers × 256 units, tanh activations (actor & critic)

• Learning rates : \(3\!\times\!10^{-4}\) for both actor and critic

• Batch size : 512

• Replay buffer : 500 000 transitions

8.3 Compute Resources↩︎

All experiments were conducted on NVIDIA L40 GPUs with 48GB memory. We also verified that all experiments can be reproduced on NVIDIA RTX 3090 GPUs with 24GB memory. Each experiment run with a single seed can typically be completed within 24 hours. By training a Siamese relative reward model, our scheduling strategy reduces the number of VLM queries during policy learning by approximately \(20\times\) compared to direct VLM querying at rollout time.

9 Details on Tasks and Environments↩︎

We benchmark our approach on ten goal-conditioned control tasks drawn from four simulators: Gym, SoftGym, MetaWorld, and MineDojo. The tasks are as follows.

• CartPole: Balance an upright pole on a moving cart.

• Straighten Rope: Straighten a rope from a randomly tangled configuration.

• Pass Water: Move a glass of water to a target location without spilling its contents.

• Open Drawer: Use a robotic arm to pull open a drawer on a desk.

• Soccer: Push a soccer ball into a designated goal area using a robotic arm.

• Sweep Into: Sweep a green cube into a hole on the table using a robotic arm.

• Combat Spider: Eliminate a spider in combat. The agent is initialized with a diamond sword, a shield, full diamond armor, and a nearby spider.

• Milk Cow: Find and approach a cow, then collect milk using an empty bucket. The agent is initialized with a bucket, and a cow is spawned nearby.

• Shear Sheep: Find and approach a sheep, then collect its wool using a shear. The agent starts with a shear and a sheep nearby.

• Hunt Cow: Find and approach a cow, then hunt it using a sword. The cow will attempt to flee, requiring the agent to chase it down. The agent is initialized with a diamond sword and a cow nearby.

For each task we hand-craft a concise natural-language goal prompt that is fed into the VLM during preference querying (see Appendix 7.1). Table 6 summarizes the task IDs and the exact goal prompts. For image–language models based on frozen CLIP or MineCLIP encoders we supply a different set of goal descriptions phrased as achievement statements. Table 7 lists the exact prompts; The prompts adopt a declarative “goal achieved’’ style (e.g.“The drawer is opened.”) that is empirically more compatible with the image–text contrastive objectives. Table 8 summarises the observation modality, action space, and episode horizon of each task.

Additional notes:

• Straighten Rope : two end-point pickers are pre-grasped at reset; the 6-D action is the 3-D \(\Delta\) position of each picker.

• Pass Water : the glass container moves along a single horizontal axis, resulting in a 1-D action.

• MetaWorld tasks : we initialise the gripper close to the target object and re-align the camera such that the object is centred, producing cleaner images for the VLM.

• MineDojo tasks : observations are limited to first-person RGB frames only (no yaw, pitch, GPS, or voxel data) to better reflect real-world vision. We enable fast reset with random teleportation, fix the world seed to 123, and constrain the biome to SunflowerPlains.

10 Details on Baselines↩︎

10.1 VLM-AR3L (Ours)↩︎

In all experiments, we set the absolute reward weighting to \(\alpha = 0.5\), the confidence threshold to \(\tau = 0.52\), and the temporal offset to \(k = 16\).

10.2 RL-VLM-F↩︎

Released by [5], RL-VLM-F employs two image–reward backbones that differ by environment suite:

• Custom CNN : Table 9; used for Gym and MetaWorld.

• ResNet-18 : Table 10; used for SoftGym and MineDojo.

10.2.0.1 ResNet-18 Backbone (SoftGym & MineDojo).

A vanilla ResNet-18 (BasicBlock \(\times[2,2,2,2]\)) is adopted for high‐variance visual inputs. ImageNet initialisation is retained, but the final FC layer is replaced with a 1- or 2-unit head.

Training protocol. Except for the backbone choice above, all optimisation hyper-parameters and data-collection schedules follow the settings described in Section 8.1.

10.3 CLIP↩︎

For image–language matching we use the original OpenAI CLIP ViT-L/14 @ 336 px checkpoint. During interaction we compute a static similarity reward at each step:

1. Text embedding. The goal prompt \(g\) is tokenised and encoded once; its feature vector is \(\ell_{2}\)-normalised.

2. Image embedding. The current RGB observation is passed through the frozen CLIP image encoder, and \(\ell_{2}\)-normalised.

3. Reward. The cosine similarity \(r_{t} = \langle f_{\text{img}},f_{\text{text}}\rangle\) is used directly as an reward (higher implies closer to the goal).

10.4 MineCLIP↩︎

For long-horizon visual tasks we employ the MineCLIP-attn reward model of [22]. Its inference pipeline consists of three stages:

1. Frame encoder – each RGB frame is embedded by a frozen ViT-B/16 backbone.

2. Temporal aggregator – a 2-layer Transformer (\(d_{\text{model}}{=}512\)) compresses the \(16\)-frame sequence into a single video feature \(\mathbb{v}\!\in\!\mathbb{R}^{F}\).

3. Reward head – an MLP takes \(\mathbb{v}\) and the tokenised goal prompt \(\mathbb{g}\) and outputs a two-logit vector \(\mathbb{z}=[z_{\mathrm{\small pos}},z_{\mathrm{\small neg}}]\); the positive logit is taken as the scalar reward, \(r_{t} = z_{\mathrm{\small pos}}\).

10.4.0.1 Implementation notes.

• A sliding window stores the most recent 16 frames. Only when the window is full do we query MineCLIP; otherwise the reward is set to 0.

• Input frames are normalised with the official MineCLIP statistics \((\mu,\sigma)\).

10.5 GT Dense Reward (Oracle)↩︎

For ablation‐style oracle comparisons, we define shaped dense rewards (Table 11) along with their task-specific weighting coefficients (Table 12). The meaning of each variable used in these reward definitions is provided in Table 13.

10.6 GT Sparse Reward (Oracle)↩︎

Sparse rewards remove shaped terms and give a single bonus \(r_t = \lambda_{\text{success}}\mathbb{1}[\text{task success}]\) (Table 14).

11 Additional Experimental Results↩︎

11.1 Ablation Hyper-parameters↩︎

We conduct ablation studies on the temporal offset \(k\) and the weighting coefficient \(\alpha\). As shown in Fig. 9, varying \(k\) does not result in significant performance differences, indicating that our method is relatively insensitive to the choice of temporal offset. Fig. 10 investigates the weighting coefficient \(\alpha\) for the absolute reward signals. The method performs consistently well over a broad range of \(\alpha\), while performance deteriorates only when extremely small or large values are used.

11.2 Ablation VLM↩︎

See Figure 11 and Table 15. Our results show that Phi-3.5 exhibits strong performance in interpreting Minecraft-specific visual states. MiniCPM-o 2.6, on the other hand, performs competitively on more structured tasks such as Soccer and Drawer Open.

11.3 Ablation Absolute and Relative rewards↩︎

See Figure 12. We compare three variants: absolute-only (\(\alpha{=}1\)), relative-only (\(\alpha{=}0\)), and the dual reward (default \(\alpha{=}0.5\)). Absolute rewards tend to work well in tasks with a clear global progress measure, while relative rewards provide more reliable guidance in long-horizon or visually ambiguous tasks. Combining them yields the best or near-best performance in most tasks, supporting the complementarity discussed in Section 4.

12 Broader Impact↩︎

Our work leverages vision-language models (VLMs) to provide reward signals for reinforcement learning without requiring extensive manual reward engineering. This approach has several positive societal implications, including:

• Reducing Development Costs: By using pre-trained VLMs to generate rewards, we reduce the need for domain-specific reward design, potentially lowering the cost and effort required to train RL agents in complex or open-ended environments.

• Scalability: The method can be applied to various domains without extensive task-specific tuning, facilitating broader adoption of RL in areas like robotics, game environments, and simulation-based training.

However, there are also potential negative implications:

• Bias Propagation: If the VLMs used for reward labeling exhibit biases, these biases could be propagated to the RL agents, potentially leading to biased or harmful behaviors.

• Unintended Applications: The method could be exploited to train agents for undesirable purposes, such as manipulating environments to achieve harmful objectives or reinforcing negative behaviors.

To mitigate these risks, we recommend implementing bias detection mechanisms in VLM outputs and restricting the application of the trained RL agents to controlled and monitored environments. Additionally, transparency in VLM training procedures can further minimize potential harms.

References↩︎

---

1. Label 0 subsumes both “Image 1 is better than Image 2’’ and”the two images are indistinguishable w.r.t.the goal.’’↩︎