1 Introduction↩︎

Reinforcement learning (RL) has achieved notable success in gaming AI [1]–[3], autonomous driving [4]–[6], and robotic manipulation [7]–[9], etc. However, a central challenge of RL lies in designing reward functions which serve as the learning signals for policy optimization. Designing appropriate rewards is notoriously difficult: poorly specified rewards often lead to suboptimal or unsafe behaviors, while shaping rewards demands substantial domain expertise and engineering effort. To address this issue, recent research has explored alternative approaches that leverage human demonstrations [10]–[12] or large-scale pre-trained models [13]–[17] to automatically derive rewards, thereby reducing reliance on hand-crafted reward engineering.

A prominent direction is preference-based reinforcement learning (PbRL) [18]–[22], where reward models are derived from human comparisons of trajectory segments. It eliminates the need for hand-crafted rewards, and aligning with human preferences effectively avoids reward hacking [23], achieving policies that meet human expectations. However, this paradigm often requires a large number of preference labels, which is costly and limits scalability. To mitigate this burden, recent work has explored leveraging large language models (LLMs) and vision-language models (VLMs) to directly generate or synthesize preference labels based on textual or visual goals [15], [17], [24]–[28]. Another line of research exploits the code-generation and reasoning capabilities of LLMs to output executable reward function code, thereby avoiding intricate reward engineering [14], [16], [29]–[31]. In addition, VLMs such as CLIP [32] have also been employed to score states based on the similarity of visual or textual task objectives to obtain task rewards at each step [13], [33]–[37]. Collectively, these approaches reduce dependence on manual reward design and broaden the applicability of RL. Despite these advances, significant limitations remain. Learned reward models, such as those inferred from human preference, often suffer from low feedback efficiency [18], [20], [38], unstable training and poor generalization to unseen scenarios. Reward functions generated by LLMs or VLMs tend to be coarse or noisy [29], [31], [39], and may fail to capture the fine-grained structure required for complex manipulation tasks [14], [16], [40]. A key open problem, therefore, is how to obtain reward signals that are both feedback-efficient and reliably aligned with task-specific human intentions.

Cognitive science research [41]–[43] shows that human learning is influenced by prior knowledge, which supports encoding, comprehension, and schema construction processes in novel tasks. For aspects of tasks involving specific dynamics, constraints, or feedback, it often requires further iterative refinement through interaction with the environment. Motivated by this, we decompose rewards into two complementary components: (i) Formal Rewards (FR), which is explicitly designed and encoded based on task knowledge, representing the part that can be directly specified; and (ii) Residual Rewards (RR), which captures task aspects that are difficult to hand-design and must be learned from observations, enabling adaptation to nuanced preferences that cannot be easily encoded. This decomposition allows task rewards to combine structured and task-relevant features with flexible preference alignment, reducing the learning difficulty of preference-based RL. Then in this paper, we present CoRe, a hybrid framework that integrates FR and RR with vision-language models (VLMs) feedback to achieve preference-aligned policies in robotic manipulation tasks. Our contributions are twofold: (1) We develop a Formal Reward Module (FRM) that leverages VLM-based preference to automatically design and optimize formal rewards, guiding continuous policy improvement while maintaining alignment with human intentions. (2) We introduce a Residual Reward Module (RRM) that learns residual rewards from video-level preferences. By employing VLMs to generate preference labels and assess state importance, RRM captures fine-grained features of manipulation tasks and complements formal ones. Through the synergy between FRM and RRM, CoRe effectively leverages the common-sense knowledge from VLMs which is aligned with human preference and enables the automatic construction of reliable, preference-aligned rewards, eliminating the need for manual reward engineering. Extensive experiments on both simulated and real-world manipulation tasks demonstrate that CoRe outperforms existing methods in terms of effectiveness and efficiency.

2 Related Work↩︎

LLMs/VLMs for Reward Design in RL. Leveraging the strong code generation, in-context understanding, and instruction-following capabilities of coding-oriented LLMs, recent work has explored using LLMs to directly generate RL reward functions, thereby avoiding complex reward engineering [14], [16], [29], [31], [39], [44], [45]. [14] ([14]) introduces an automated reward design framework powered by GPT-4 [46]. Without requiring detailed prompt engineering, it can generate human-level reward functions based on the environment code through evolutionary search and reward reflection. [45] ([45]) presents a reward-policy co-evolution framework that avoids retraining policies from scratch for each reward candidate. [39] ([39]) proposes to parameterize rewards using LLMs and minimize the discrepancy between predicted rewards and the LLM’s ranking over trajectory segments, which stabilizes reward learning and improves policy consistency. To better align with human intent, some studies have moved beyond textual task descriptions and instead utilize visual demonstrations to guide reward design [30], [40], [47]–[49]. [30] ([30]) transforms video demonstrations into textual descriptions of key events, which are then interpreted by LLMs to generate evaluation benchmarks. This enables accurate reward modeling and policy learning directly from videos. [40] ([40]) leverages VLMs to extract visual feature functions from video demonstrations and task prompts, then applies inverse reinforcement learning to recover task-aligned rewards. This approach generalizes well to out-of-distribution tasks. However, these code-generation methods often require access to the entire environment code to design detailed reward functions [15], [17], [27]. In this paper, we employ LLMs to refine the reward function and utilize preference labels for reward search. This ensures the reward function remains continuously aligned with preference, enabling incremental performance improvements of the policy. We also do not require environment code to obtain fine-grained rewards due to the design of the learnable residual reward.

Reward Learning from Preference. PbRL provides a solution to avoid reward engineering [11], [18]–[22], [50]. However, for complex manipulation tasks, PbRL typically requires a large number of preference labels to learn accurate reward models, which can be costly when relying on human labeling. To address this challenge, several works have proposed informative query selection schemes that identify high-value samples to accelerate reward learning [11], [18], [51]. Another line of work aims to optimize the reward learning framework [19], [20], [52], [53]. [19] ([19]) proposed Meta-Reward-Net (MRN) which integrates Q-function performance into reward learning, enabling efficient policy optimization with limited labels. [53] ([53]) leverages a learned world model to estimate the importance of individual states within a trajectory segment, guiding rewards to be proportional to these importance scores, thus accelerating reward and policy learning. These focus on enhancing feedback efficiency, thereby reducing the number of required labels. Recently, LLMs/VLMs have shown impressive capabilities in common sense reasoning and visual understanding [46], [54]. Motivated by this, several studies have explored replacing human labeling with LLMs/VLMs to provide preference labels automatically [15], [17], [25], [27], [28], [55], [56]. [15] ([15]) directly queries VLMs for image-level preference labels using goal text, avoiding reward noise introduced by direct VLMs scoring [13], [33]–[37]. [28] ([28]) sample multiple LLM-based evaluation functions to generate score-level synthetic preference labels, achieving performance comparable to expert-scripted teachers. In this paper, we propose using VLMs to capture video-level preference and evaluate state importance, enabling efficient reward learning. Our method can significantly enhance feedback efficiency and the performance of reward models.

3 Method↩︎

In this section, we introduce our CoRe, a framework that combines the Formal Reward Module (FRM) and the Residual Reward Module (RRM) with VLMs feedback to achieve preference-aligned RL without human involvement.

3.1 Formal Reward Module (FRM)↩︎

Code-based formal rewards defined over explicit task features are easy to design and can efficiently express simple task goals. Recent work shows that LLMs can automatically generate and iteratively refine executable formal reward code [14]. At iteration \(n\), given the task description \(t_d\), the environment code (we only need some task knowledge \(t_s\), such as the definition of states, not code), the previous best reward \(\hat{r}_{best}^{n-1}\) and the policy feedback \(t_f^{n-1}\) collected under that reward, LLMs generate a set of reward candidates \(\hat{r}^{n}=[\hat{r}_{1}^{n}, \hat{r}_{2}^{n}, ..., \hat{r}_{K}^{n}]\). The task completion function \(F_{c}(\hat{r}_{k}^n)\) evaluates the policy trained with candidate \(\hat{r}_{k}^n\). The best reward is selected accordingly: \[\hat{r}_{best}^{n} = \max_{\substack{\hat{r}_{k}^{n} \in \hat{r}^{n}}} F_{c}(\hat{r}_{k}^{n}). \label{eq:reward95search}\tag{1}\]

However, this approach may not be sufficient for complicated or abstract task goals, such as aligning with human expectations. As shown in 2a, LLMs identify the Hammer task features of formal rewards as “Hammer” and “Nail,” representing the hammering action and the nail’s location, respectively. According to 1 , the reward candidate with high “Nail” weights will be selected as optimizing the “Nail” reward component directly yields the highest task completion. Nevertheless, excessive focus on task completion may lead to undesirable behavior, such as striking with the hammer handle. To overcome this issue, we replace task completion-based evaluation with preference-guided reward search, where the reward candidate most consistent with preference is selected (2b). We denote this process as Reward-Preference Alignment.

Specifically, there exists a preference dataset \(\mathcal{D}\) containing triples \((\sigma^0,\sigma^1,y)\). Each trajectory segment \(\sigma=\{({s}_k,{a}_k),\ldots,({s}_{k+H},{a}_{k+H})\}\) represents a short part of an episode in a standard Markov Decision Process [57], where \({s}_t\) and \({a}_t\) denote the state and action at time step \(t\), and \(H\) is the segment length. The label \(y\in\{(0,1),(1,0),(0.5,0.5)\}\) indicates preference for \(\sigma^1\), preference for \(\sigma^0\), or no preference, respectively. \(F_{p}(\hat{r}_{k}^{n}, \mathcal{D})\) represents the preference prediction accuracy rate of \(\hat{r}_{k}^{n}\) for \(\mathcal{D}\), and the best reward \(\hat{r}_{best}^{n} \in \hat{r}^{n}\) can be defined as: \[\hat{r}_{best}^{n} = \max_{\substack{\hat{r}_{k}^{n} \in \hat{r}^{n}}} F_{p}(\hat{r}_{k}^{n}, \mathcal{D}), \label{eq:32rpa}\tag{2}\]

\[\begin{align} F_{p}(\hat{r}_k^{n}, \mathcal{D})=&\mathop{\mathbb{E}}_{(\sigma^{0},\sigma^{1},y){\sim}\mathcal{D}} \Big[y(0)*\mathbb{I}(\hat{R}_{k}^{n}(\sigma^0)>\hat{R}_{k}^{n}(\sigma^1))\\ &\qquad\qquad + y(1)*\mathbb{I}(\hat{R}_{k}^{n}(\sigma^1)>\hat{R}_{k}^{n}(\sigma^0))\Big], \end{align} \label{eq:fp}\tag{3}\] where \(\hat{R}_{k}^{n}(\sigma^i)=\sum_{t} \hat{r}_{k}^{n}(s_t^{i}, a_t^{i})\). This method enables formal rewards to be gradually optimized and aligned with human preferences, ensuring continuous policy improvement. It can also quickly verify the reward candidate, avoid training and evaluation from scratch, and improve the efficiency of policy learning. Importantly, it dynamically aligns the FRM’s optimization objective with the preference to ensure stability throughout the training process.

3.2 Residual Reward Module (RRM)↩︎

Nonlinear reward models with a large number of parameters excel in expressing complex and abstract task goals, but often suffer from feedback inefficiency [15], [56], [58]. This inefficiency arises because the single state-based preference conveys limited information, while the segment-based preference (e.g., trajectory preference) makes accurate state-wise credit assignment difficult. To address these issues, we propose video-level preference image-based reward learning from VLMs’ feedback. Rather than relying on low-expression preference, such as natural language [24], [26], coordinate-based feedback [27], or single-frame image [15], which may not adequately express preference, videos can better capture task dynamics and convey richer information. For credit assignment, we introduce a two-stage multimodal prompting query based on the chain-of-thought [59] to assess the relative importance of individual image frame which serves as an approximate prior for rewards.

Specifically, the segment with image observations \({I}\) is formed as \(\sigma=\left\{{({s}_k, {a}_k, {I}_{k}), \ldots, ({s}_{k+H}, {a}_{k+H}, {I}_{k+H})}\right\}\). To enhance visual understanding, we input task description \(l_d\), start image \(I_s\), goal image \(I_d\) and videos \(\{I_t^i\}_{i \in \{0,1\}}\) into VLMs \(F_{\text{VLM}}\), and obtain the preference label: \[y = F_{\text{VLM}}\left(l_d, I_s, I_d, \{I_t^i\}_{i \in \{0,1\}}\right). \label{eq:32query95vlm}\tag{4}\] When trajectory segments are not meaningfully comparable, VLMs output “incomparable" labels, which are excluded from residual reward training. During preference labeling, VLMs also assess task completion across the frames, producing the per-frame importance weight over the full segment \(W^{i} = \{w_t^i\}_{i \in \{0,1\}}\), where \(w_t^i\) denotes the state importance of frame \(t\) in \(\sigma^i\). Each preference feedback is stored as a triple \((\sigma^0, \sigma^1, y, W^0, W^1)\) in the preference data \(\mathcal{D}\). Based on this, we use the estimated importance weight as a prior over predicted state rewards. In addition to optimizing the cross-entropy loss \(\mathcal{L}^{Reward}\) to match preference [18], we enforce consistency between the predicted state reward distribution and the prior importance weight via the Kullback–Leibler divergence [60]. The resulting objective can be defined as: \[\mathcal{L} = \mathcal{L}^{Reward} + \alpha \cdot \mathcal{L}^{KL}, \label{eq:32reward95learning}\tag{5}\]

\[\begin{align} \mathcal{L}^{KL} = \mathop{\mathbb{E}}_{(\sigma^{0},\sigma^{1}, W^0, W^1){\sim}\mathcal{D}} \big[D_{KL}(W^0||R_{\psi}^0) \;\;\;\\+D_{KL}(W^1||R_{\psi}^1)\big], \end{align}\] where \(R_{\psi}^i=\left[\hat{r}_{\psi}(I_k^i), ..., \hat{r}_{\psi}(I_{k+H}^i)\right]\). \(\alpha\) is a weighting coefficient. \(\hat{r}_{\psi}\) is the reward model. Video-level preference reward learning enables the residual reward to capture VLM-aligned preference effectively. This significantly accelerates reward learning and improves feedback efficiency.

4 Experiments↩︎

4.1 Experimental Setup↩︎

Tasks. We evaluate CoRe on a diverse set of robotic manipulation tasks involving rigid, articulated, and deformable objects. These tasks span a wide range of primitive manipulation skills, including pushing, pulling, pressing, twisting, and grasping. Specifically, we consider seven from MetaWorld [61] (Soccer, Sweep Into, Drawer Open, Button Press, Dial Turn, Hammer and Peg Insert) and three from SoftGym [62] (Fold Cloth, Straighten Rope and Pass Water) which cover both rigid-body and deformable-object manipulation. Detailed task descriptions and visualizations are provided in the Appendix 6.

Baselines. We compare CoRe against representative methods that leverage pretrained LLMs or VLMs for reward construction, including: (i) similarity-based reward scoring: CLIP Score [36] (ii) code-based reward generation: Eureka [14] and Text2Rward [16] and (iii) preference- or rating-based reward learning from visual feedback: RL-VLM-F [15], PrefVLM [56] and ERL-VLM [17]. In addition, we report results using Environment Sparse Reward (Env Sparse) and Environment Dense Reward (Env Dense) provided by the benchmark. Detailed description can be found in the Appendix 8.

2.0pt

Implementation Details. We adopt SAC [63] for policy learning with state observations. CLIP Score, RL-VLM-F, PrefVLM, and ERL-VLM follow PEBBLE’s unsupervised pre-training [18], while other methods train policies directly. To ensure fairness, all methods have the same hyperparameters for policy learning, with the only difference being the reward. Reward functions for FRM, Eureka and Text2Reward are generated by GPT-4.1 mini [46], with five reward searches performed, sampling four reward candidates each time. The reward feedback for Text2Reward is provided by the author. PrefVLM uses LIV [35] as the VLM. Preference labels for RRM, RL-VLM-F and ERL-VLM are provided by Gemini 2.0 Flash [54] with a uniform sampling scheme for queries. The number of labels and images is shown in ¿tbl:tab:label95num?. The reward model is trained on images rendered by the simulator. To reduce computational cost and improve query efficiency in RRM, each video is downsampled before being input into VLM. The importance weight evaluated by VLM is first linearly interpolated and then normalized using a softmax function to obtain the importance weight of the full segment. To standardize reward scales, FRM and RRM rewards are clipped to \([-0.5, 0.5]\). The total reward is the sum of FRM and RRM. The full procedure for CoRe in 9. Detailed hyperparameters and training implementation for all methods can be found in Appendix 7.

To ensure a fair comparison among all methods, we render only task-relevant objects during training of the image-based reward model by making the robot arm (MetaWorld) and pickers (SoftGym) transparent and adjusting the camera view to focus on the objects. We evaluate performance using the success rate for MetaWorld and the episode return for SoftGym. Metrics are recorded per episode and averaged over three runs.

4.2 Main Results↩︎

3 shows the learning curves across all tasks. CoRe demonstrated faster learning than humans in Sweep Into and Hammer, ultimately reaching or surpassing human-level performance. In Soccer and Drawer Open, it learned slightly slower but still matched human performance. Notably, in Straighten Rope, CoRe achieved the fastest learning and highest stability, ultimately exceeding human performance. Overall, CoRe enables faster and more stable policy learning. As summarized in ¿tbl:tab:main95exp?, CoRe achieves an average success rate of \(99.0\%\) in seven MetaWorld tasks, outperforming Sparse (\(69.6\%\)), CLIP Score (\(4.8\%\)), Eureka (\(75.5\%\)), Text2Reward (\(74.2\%\)), RL-VLM-F (\(56.8\%\)), PrefVLM (\(18.6\%\)) and ERL-VLM (\(37.0\%\)), and approaching Human (\(99.3\%\)). In SoftGym tasks, CoRe attains \(-0.10\) in Fold Cloth, \(-30.0\) in Pass Water, and \(20.6\) in Straighten Rope—the best among all methods. We observe that most methods achieve near-perfect performance on simple tasks, but struggle with complex ones. For example, Drawer Open rewards depend mainly on ball position and can be easily specified by Sparse, Human, and LLM/VLM-based methods, all of which achieve \(100\%\) except CLIP Score and PrefVLM, which suffer from the data shift in pre-trained VLMs. In contrast, tasks like Hammer and Peg Insert involve intricate actions (grasping, positioning, aligning), where Sparse rewards are ineffective. Hammer’s \(66.7\%\) reflects reward hacking (striking with the handle), while Peg Insert remains at \(0\%\). Eureka and Text2Reward struggle to design suitable reward components and weights (\(33.3\%\) and \(32.0\%\) in Hammer, respectively). Although Eureka learns fast in the early training because of its structured reward design from LLM in Peg Insert (rapidly achieve a 70% success rate), but its performance could not be further improved when not finding the implicit rewards. RL-VLM-F suffers from limited feedback and unstable reward learning (\(14.7\%\) in Peg Insert). The rating used by ERL-VLM struggles to accurately assign credit for these tasks (\(9.3\%\) in Hammer and Peg Insert). The learning rate of CoRe may not be as fast as structured-rewards-design method (as in Peg Insert), but accurate rewards estimation from FRM and RRM still make CoRe achieved the highest success rate (nearly 100 %). Collectively, most baselines exhibit large variance due to incomplete reward specification or reward instability. By contrast, CoRe excels in simple and complex tasks, owing to the complementary synergy of its two modules.

As shown in ¿tbl:tab:label95num?, CoRe demonstrates superior sample efficiency. Across tasks, it uses 0.15K–0.5K labels, compared to 0.5K–21K for other methods, corresponding to a 3-40\(\times\) reduction. For example, in Fold Cloth, RL-VLM-F uses 500 labels while CoRe uses 150 (\(\approx3\times\)); in MetaWorld, PrefVLM uses 21K labels while CoRe uses 0.5K (\(\approx40\times\)). Other methods require more labels due to limited information in image preferences, noisy labels or coarse ratings which can hinder fine-grained reward learning.

As shown in ¿tbl:tb:32train95cost?, a full CoRe training run requires approximately 2.00M tokens, $0.37 API cost, and 2.15 hours of wall-clock time on average across the above ten tasks. Compared with methods such as RL-VLM-F and ERL-VLM, CoRe achieves a more favorable trade-off between performance, computational cost, and runtime efficiency. This efficiency mainly stems from the structured reward with Reward-Preference Alignment in FRM and video-level residual reward learning in RRM.

4.3 Ablation Study↩︎

Effects of FRM and RRM. To disentangle the contributions of CoRe’s two components, we conducted an ablation study across 10 tasks. ¿tbl:tab:ablation? shows that FRM (FRM\(^4\)) generally performs better in MetaWorld tasks, while RRM performs better in Softgym tasks. This suggests that code-based FRM is effective for rigid-object manipulation but limited in expressiveness, making it insufficient for deformable objects. Conversely, RRM leverages vision-based reward models to handle deformable tasks but struggles with complex long-horizon ones due to limited feedback. By combining them, CoRe achieves consistent gains: RRM complements FRM’s limited expressiveness, while FRM facilitates RRM’s reward learning. The two modules are thus mutually reinforcing, enabling more efficient, robust policy learning.

Reward-Preference Alignment in FRM. We further ablate the number of Reward-Preference Alignment iterations in FRM. FRM\(^i\) denotes the \(i\)-th alignment, and policy success rates are reported in ¿tbl:tab:ablation?. We find that alignment consistently improves policy. For example, in Sweep Into and Dial Turn, FRM\(^0\) achieved only 30% success, but after one alignment, success rates rose to 93.3% and 79.3%, respectively. However, FRM is sensitive to task difficulty and label noise: in Hammer and Fold Cloth, the improvement remains limited. Overall, Reward-Preference Alignment enables fast validation of reward candidates, ensures consistency with human preference, and drives continuous improvement.

VLMs preference for images and videos. We evaluate the labeling accuracy of VLMs under three conditions: Correct, Incorrect, and No Preference. As shown in 4, we find that video preference consistently outperforms image preference. The average results of video-level preference labels across all tasks achieve 51.5% Correct, 15.0% Incorrect, and 33.5% No Preference, while image-level preference labels reach only 41.6%, 15.6%, and 42.8%, respectively. This improvement stems from our video-level preference prompts, which compare every frame and better capture task-relevant details, yielding higher-quality preference labels.

To evaluate whether VLM-derived importance weights reflect true state importance, we measure their Spearman correlation with task progress [64]. Across 10 tasks, the average \(\rho\) is \(0.48\), indicating moderate alignment. However, directly using these weights as rewards leads to high variance due to noise [15]. Instead, RRM treats preference prediction as the primary objective and uses importance weights only as an auxiliary constraint, improving reward accuracy, efficiency, and stability.

Alignment of rewards with task progress. We further assess the quality of the learned rewards from both temporal and spatial perspectives. Specifically, we examine their alignment with task progress along expert trajectories and distribution across object positions. We compare PrefVLM, RL-VLM-F, ERL-VLM, RRM without importance distribution (RRM w/o imp), RRM, and CoRe. Except for PrefVLM, all others are image-based rewards. For each method, normalized rewards (0–1) are computed and visualized for Peg Insert and Sweep Into in 5 and 6. PrefVLM, RL-VLM-F, and ERL-VLM suffer from inaccurate rewards and significant fluctuations, with spatial distributions that remain unsmooth and imprecise. RRM w/o imp leverages video-level preference to achieve more accurate rewards and yields smoother reward distributions; however, credit assignment under limited feedback still introduces some noise. RRM further improves accuracy and stability, generating rewards with near-linear correlation with task progress and consistent spatial smoothness. With the integration of FRM, CoRe yields the most accurate and stable rewards, showing strong consistency with task progress and the smoothest spatial distribution, thereby facilitating stable policy optimization. Quantitatively, Spearman correlations between learned rewards and task progress along expert trajectories averaged over ten tasks are \(0.09\) (PrefVLM), \(0.44\) (RL-VLM-F), \(0.48\) (ERL-VLM), \(0.56\) (RRM w/o imp), \(0.77\) (RRM), and \(0.88\) (CoRe), confirming that CoRe provides the most accurate and reliable rewards. These results highlight that incorporating importance weights and the FRM yields stable, precise reward models, with improvements also reflected in policy performance (¿tbl:tab:ablation?).

4.4 Deploy on the Real Robot↩︎

We deploy policies trained via CLIP Score, Eureka, Text2Reward, PrefVLM, RL-VLM-F, ERL-VLM and CoRe on UR5 robotic arms across five real-world tasks: Drawer Open, Dial Turn, Hammer, Fold Cloth, and Straighten Rope (8). For Drawer Open, Dial Turn and Hammer, we reconfigure simulation environments to match real setups and use Aruco markers for object pose estimation. Success rate (higher is better) is used as the performance metric. For Fold Cloth and Straighten Rope, key points are tracked via color detection (e.g., cloth corners in Fold Cloth, rope endpoints and midpoint in Straighten Rope). Task-specific geometric metrics are used for evaluation: in Straighten Rope, the task completion percentage (higher is better) is defined as the rope length at the end of an episode relative to its maximum length; in Fold Cloth, the performance metric is measured as the positional error (cm, lower is better) of the three corner points corresponding to the diagonal cloth at the end of an episode. To validate the policy’s robustness when transferring from sim to real, they were directly executed on the UR5 without fine-tuning. All methods were repeated twenty times. As summarized in ¿tbl:tab:real95experiment?, CoRe consistently outperforms other methods across all tasks, demonstrating not only superior performance in simulation but also robustness to real-world noise with high-quality reward signals, thereby enabling reliable robotic deployment.

5 Conclusion and Future Work↩︎

In this paper, we introduced CoRe, a framework that decomposes reinforcement learning rewards into formal rewards and residual rewards to achieve preference-aligned policy learning in robotic manipulation. The Formal Reward Module (FRM) leverages LLMs and preference to iteratively design and refine code-based formal rewards, while the Residual Reward Module (RRM) exploits video-level preference and importance weight from VLMs to capture fine-grained alignment with human intentions. Through their complementary synergy, CoRe enables efficient and robust rewards construction without manual reward engineering. Extensive experiments in both simulation and the real world demonstrate that CoRe achieves superior performance, stability, and human alignment compared with existing methods.

Due to the reliance on iterative LLM/VLMs querying, future work will explore uncertainty-aware reward modeling and more efficient querying strategies to further improve the robustness and scalability of reward learning. In addition, evaluating CoRe in more diverse and unstructured real-world environments is an important direction for future research, which may further improve its generalizability across complex manipulation scenarios.

Acknowledgements↩︎

This work was supported in part by New Generation Artificial Intelligence-National Science and Technology Major Project 2025ZD0122904, in part by National Natural Science Foundation of China(Grant Number:62473366) and in part by National Science and Technology Major Project 2026ZD1609003.

Impact Statement↩︎

This work presents CoRe, a reward learning framework for robotic manipulation that reduces the need for manually designed reward functions in RL and improves the alignment of policies with human preferences. As CoRe relies on LLMs/VLMs for reward generation and preference feedback, potential biases or inaccuracies in their outputs may propagate into the learned reward functions and policies. In addition, iterative querying of LLMs/VLMs introduces additional computational and financial costs during training. These factors may raise robustness, scalability, and safety considerations in real-world robotic applications.

Appendix

6 Details on Tasks and Environments↩︎

We evaluate our method on ten simulated and five real-world robotic manipulation tasks involving rigid, articulated, and deformable objects as shown in 7 and 8, which together cover a wide range of primitive actions such as pushing, pulling, pressing, twisting, and grabbing.

6.1 MetaWorld Tasks↩︎

Seven tasks from MetaWorld [61] involving the manipulation of rigid and articulated objects on the table, with a simulated Sawyer robot.

• Soccer: The goal is to kick the ball into the goal.

• Sweep Into: The goal is to sweep a green puck into a hole.

• Drawer Open: The goal is to pull the green drawer completely out.

• Button Press: The goal is to press the red button from top to bottom. The robot needs to keep pressing down because there’s a spring that will automatically push the button back up.

• Dial Turn: The goal is to rotate a dial 180 degrees counterclockwise.

• Hammer: The goal is to grasp a hammer and hammer a gray nail. The robot needs to hit the nail with the hammerhead, not the handle.

• Peg Insert: The goal is to grasp a green peg and insert it horizontally into the hole in the block.

Observation Space. We use the SAC algorithm and state-based observations for policy learning, and high-dimensional RGB images rendered by a simulator for image-based reward learning. Following the original MetaWorld settings [61], the state observation is a 39-dimensional vector, consisting of the 3D position of the end-effector, gripper status, the 3D position and quaternion of target objects, and the 3D position of the goal. Following RL-VLM-F [15], we set the resolution of the image observation to 300\(\times\)​300. We also render only task-relevant objects by setting the robot arm transparent to mitigate VLM hallucinations and promote visual understanding. The camera view is adjusted to focus on objects, and the initial state is modified to bring the end-effector closer to them. The example of image observation is shown in 11.

Action Space. The action in MetaWorld tasks is a 4-dimensional vector consisting of the 3D position displacement of the end-effector and the torque the gripper fingers should apply. All dimensions of the action are normalized to [-1, 1].

6.2 SoftGym Tasks↩︎

Three tasks from SoftGym [62] involving the manipulation of deformable objects on a table following [15].

• Fold Cloth: The goal is to fold a cloth diagonally from the upper left corner to the lower right corner. There is a transparent picker that is always fixed to the upper left corner of the cloth.

• Straighten Rope: The goal is to straighten a rope from a random configuration. The rope is controlled by two transparent pickers, which are fixed to its endpoints at the start of the task.

• Pass Water: The goal is to pass a glass of water to the designated location (the red dot) while ensuring that no water spills out. The glass can only move in a fixed direction.

Observation Space.

The settings for policy and reward learning are the same as in MetaWorld, but the state observation and the image observation resolutions are set differently. We all follow the RL-VLM-F settings.

• Fold Cloth: The state observation consists of the positions of a subset of particles from the cloth mesh. The cloth has an original resolution of 40\(\times\)​40 particles, which is uniformly subsampled to a \(6\times6\) grid. The final state includes the 3D position of the picker (3) and the 3D positions of all subsampled cloth particles (108), resulting in a 111-dimensional vector. The resolution of the image observation is 360\(\times\)​360.

• Straighten Rope: The state observation comprises the positions of all particles along the rope (10 particles, 30) and the two pickers (6), resulting in a 36-dimensional vector. The resolution of the image observation is 240\(\times\)​240.

• Pass Water: The state observation includes the container dimensions (width, length, and height), the target container position, the water height within the container, and the amounts of water inside and outside the container, yielding a 7-dimensional state space. The resolution of the image observation is 360\(\times\)​360.

Action Space.

All dimensions of the action are also normalized to [-1, 1]

• Fold Cloth: We employ a pick-and-place action primitive. The cloth is assumed to be grasped at the top left corner at initialization, and the action specifies a 2D target placement location, resulting in a 2-dimensional vector.

• Straighten Rope: Two pickers are used to manipulate the rope, one attached to each end. Accordingly, the action space consists of the 3D displacement vectors for both pickers, resulting in a 6-dimensional action space. The rope endpoints are assumed to be grasped at the start of each episode.

• Pass Water: The motion of the container is constrained to a single dimension; thus, the action is a 1-dimensional scalar representing the displacement of the container along that axis.

6.3 Real Robot Taks↩︎

To validate the policies robustness when transferring from sim to real, we designed five real-world operational tasks based on the simulation task, as shown in 8. We first simulated real-world scenarios in a simulation environment, then retrained the policy and directly deployed it to the UR5 robot to perform the manipulation tasks.

Observation Space. For Drawer Open, Dial Turn and Hammer, the state observation of policy inputs is consistent with the settings in the simulation, where the 3D position of the end-effector and the gripper status are obtained in real-time through the Python SDK of UR5. The 3D position and quaternion of the object are obtained using visual detection of Aruco markers. We use the Intel RealSense D435i depth camera, intrinsic and extrinsic parameter calibration for accurate pose estimation. For Fold Cloth, the state observation is a 15-dimensional vector, consisting of the 3D position of the four corner points of the cloth and the end-effector. For Rope Straighten, the state observation is a 15-dimensional vector, consisting of the 3D position of two endpoints and midpoints of the rope followed by the 3D position of two end-effectors. Since we marked the keypoints of the cloth and rope, it was easy to obtain the 3D positions of the keypoints using visual color detection and the depth camera.

Action Space.

The action settings are basically consistent with the simulation. However, due to the limitations of the Robotiq 2F-85 gripper, we used the absolute position of the gripper instead of torque in the simulation.

7 Training Details and Hyperparameters↩︎

7.1 Reward Optimization↩︎

Formal Reward Optimization.

We followed the iterative process of Eureka’s reward design [14], except for the reward search and reward design prompts. For reward search, Eureka uses task completion to select the best reward candidates defined in 1 , while our method uses Reward-Preference Alignment defined in 2 . For reward design prompts, Eureka needs access to all the Python code of the environment to obtain the complex reward function. Because of the learnable residual rewards, our method does not require the full Python code; it only needs some descriptions of the reward function, such as the definitions of the input and output variables of the reward function (state, action and target), which are easy to design, as shown in ¿tbl:tab:prompt32for32reward32info?. Our method performs five reward searches, sampling four reward candidates each time by using GPT-4.1 mini [46].

Image-based Residual Reward Learning.

For the image-based reward model, we employ a 4-layer convolutional neural network for MetaWorld tasks and a standard ResNet-18 [65] for SoftGym tasks following [15]. Following PEBBLE [18], we use an ensemble of three reward models, with a tanh activation applied to the output reward. All models are optimized using Adam [66] with an initial learning rate of \(3\times10^{-4}\). The ResNet-based reward model has a batch size of 8, while others have a batch size of 32. The query selection scheme is uniform sampling. The loss for reward model optimization consists of the cross-entropy loss of preference labels and the KL divergence loss of importance weights defined in 5 . For MetaWorld tasks, we use a segment size of 20 and downsample to 3 for VLM querying (Gemini 2.0 Flash [54]).

7.2 Policy Learning↩︎

Following PEBBLE [18] and RL-VLM-F [15], we adopt Soft Actor-Critic (SAC) [63] as the off-policy learning algorithm. We use the same actor–critic network architectures and hyperparameter settings as in the original work. The policy is learned with state observation for all methods. Due to the formalized reward function, our method directly optimizes the policy without requiring unsupervised pre-training.

7.3 Training Details↩︎

The full procedure of our method is summarized in 9. Policy learning and reward learning are performed simultaneously: the policy is optimized using experiences from the replay buffer and the learned reward model, while the reward model is updated based on policy performance and preference labels over trajectory segments provided by VLMs. The optimization of both the formal reward and the residual reward is also carried out in parallel. Specifically, prior to training, we perform a task-completion-based reward search to obtain an initial formal reward. During training, the policy collects experience and updates online at each time step. The formal reward is updated every \(M\) time steps, while VLM-based preference labeling and residual reward updates are conducted every \(N\) time steps. After each reward update, all experiences stored in the replay buffer are relabeled accordingly.

8 Baselines↩︎

To verify the performance, we compare CoRe with existing methods that also leverage pretrained LLM/VLM to directly generate reward score, reward code or preference feedback. The reward code method requires the environment code and the task description. The reward score and preference feedback require the task description and the agent’s image observations. The reward score is calculated directly through similarity, while preference feedback involves learning the reward model from the feedback.

• CLIP Score [36]: The reward score is calculated as the cosine similarity between the embeddings of the task description and the agent’s current image observation in the CLIP [32] latent space. We run this method through the implementation of RL-VLM-F ¹.

• Eureka [14]: This baseline is the LLM-driven code-based reward generation method. It needs to access the environment code, automatically design reward functions based on task description, and continuously optimize them through reward reflection. We extended this method to the MetaWorld and SoftGym tasks based on the original implementation of Eureka ².

• Text2Rward [16]: Similar to Eureka, LLMs generate reward functions based on the environment code and task description, but this baseline requires human natural language feedback to optimize the reward function. We performed five rounds of reward optimization based on the original implementation of Text2Reward ³ and our online human feedback. The final reward function was used to train the policy.

• RL-VLM-F [15]: Similar to the PbRL framework, but the preference feedback is obtained by querying VLMs based on the task description and two agents’ image observations. The image-based reward model is trained on these image-level preference labels. We will run this method directly through the original implementation of RL-VLM-F.

• PrefVLM [56]: Instead of querying VLMs ([35]) for the image, this baseline first uses the similarity reward score of VLM to obtain video-level preference. The noise labels from VLMs are selected through a noise filtering [22] and then provided to human annotators which then is used to fine-tune VLM. Since the original implementation is not publicly available, we reimplemented the method based on the paper and the PEBBLE codebase ⁴, keeping all architectural choices and hyperparameters consistent with those reported in the original work.

• ERL-VLM [17]: Query VLM with an image or a sequence of images to obtain the rating based on task description (discrete, e.g., bad, average, good), and then use the ratings to train an image-based reward model for rating-based RL [67]. We directly ran the method using the original implementation of ERL-VLM ⁵.

• Env Sparse: Environment Sparse Reward indicates that the reward for task success is 1, and all other rewards are 0.

• Env Dense: Environment Dense Reward is the dense reward provided by the benchmark.

To ensure fairness, all reward function generation methods (Eureka, Text2Reward and our method) are powered by GPT-4.1 mini [46], and all preference labels (RL-VLM-F, ERL-VLM and our method) are provided by Gemini 2.0 Flash [54]. The parameters related to reward construction are consistent with those in the original paper.

9 Additional Experiments↩︎

9.1 Analysis with Stronger Models↩︎

We further investigate whether simply scaling to stronger models can compensate for the limitations of single-form reward learning. As shown in ¿tbl:tab:strong95model?, stronger models improve the performance of FRM and RRM to some extent, but the gains are relatively limited compared with the increased API cost(e.g., FRM: 54.0% → 54.7% with  9× cost; RRM: 49.3% → 62.6% with  6× cost). In contrast, CoRe achieves substantially better performance while maintaining favorable efficiency, suggesting that reward decomposition is more effective than solely scaling models.

9.2 Experiments with Open-source VLMs↩︎

We additionally evaluate CoRe using open-source VLMs to verify that the framework is not restricted to proprietary models. As shown in ¿tbl:tab:open95source?, replacing Gemini 2.0 Flash with Qwen3-VL-8B-Instruct[68] still achieves strong performance across manipulation tasks (84.0% in Sweep Into, 88.7% in Hammer). In contrast, Video-LLaVA-7B [69] fails to produce sufficiently reliable preference labels due to limited instruction-following and video understanding capabilities. These results suggest that CoRe is model-agnostic and can benefit from future advances in open-source models.

9.3 Robustness under Visual Disturbances↩︎

To further evaluate robustness under visual disturbances, we introduce a randomly moving red cube into the environment as shown in 10. ¿tbl:tab:visual95disturbance? shows that CoRe maintains stable performance with only minor degradation under noisy settings, demonstrating robustness to perceptual disturbances and environmental noise.

References↩︎