1 Introduction↩︎

Large language models (LLMs) have achieved impressive progress in natural language understanding [2] and mathematical reasoning [3]. However, their ability for solving abstract symbolic tasks still have some limitations [4], [5] and recent work try improving LLMs to solve the typical Abstraction and Reasoning Corpus (ARC; [1]), where the model must infer a latent transformation rule from a small number of grid-based input-output examples and apply it to a new test input, as Figure 1 (a) shows.

Yet current results remain limited, and stronger frontier LLM-based systems often require substantial inference-time cost to achieve higher ARC performance, underscoring the difficulty of abstract rule induction in ARC [6], [7]. Recent progress on improving ARC reasoning ability of LLMs has increasingly relied on expanding supervision beyond the original benchmark tasks. [8] builds training pipeline around ARC-specific data augmentation using the RE-ARC dataset [9]. [10] propose program-based data synthesis for ARC tasks to expose the model to substantially more ARC-style supervision. NVARC [11] pushes this trend further by introducing a multi-stage synthetic data generation pipeline, where LLMs are used to produce puzzle summaries, compose more complex task descriptions, and generate corresponding input-output grid logic in code, yielding a large synthetic corpus for ARC-style training. Although these approaches differ in how additional supervision is constructed, the overall pipelines still rely only on input-output supervision over correct targets, as Figure 1 (b) shows.

From the perspective of learning and cognitive science, learning on difficult problems does not arise only from correct examples, but also from identifying and correcting informative errors [12], [13]. This idea is also useful for training LLMs. For example, learning from step-level preference signals and erroneous traces can improve long-chain mathematical reasoning performance [14], [15]. [16] also shows that human solvers in ARC can improve across attempts using feedback. Inspired by them, we propose DiARC, a preference alignment framework for improving ARC reasoning ability through encourage LLMs to distinguish the correct output from rejected alternatives, as an example shown in Figure 1 (c).

Specifically, DiARC constructs preference pairs in which the chosen response is the answer to the query input and the rejected response is a negative derived from that task. The negative samples are constructed from three levels: (1) visual modifications applied directly to the output grid to produce candidates that remain plausible while violating the underlying rule; (2) recurring domain-specific language patterns extracted from task-specific transformation programs, yielding reusable program-level rewrites that produce rule-level negatives across tasks; and (3) LLM-assisted edits to each task’s transformation rule, constructing task-specific negatives that are as close as possible to the semantic opposite of the original rule. Training on these preference pairs allows to optimize model not only for correctness, but also for discrimination between the correct rule and plausible near-miss alternatives.

We evaluate on six ARC benchmarks including: ARC-AGI-1 [1], ARC-AGI-2 [17], MiniARC [18], ConceptARC [19], 1D-ARC [20], and ARCcommunity. Experimental results using three open-source LLMs show that DiARC yields consistent gains across all ARC datasets. By using Qwen3 model, it achieves over 96% accuracy on ARC-AGI-1, MiniARC, and ConceptARC, outperforming various ARC-specialized models and strong closed-source LLMs.

2 Preliminaries and Motivations↩︎

2.0.0.1 ARC Task Formulation.

An ARC-like task \(\tau\) consists of a support set \(\mathcal{S}_{\tau}=\{(I_i, O_i)\}_{i=1}^{n_\tau}\), a query input grid \(q_\tau\), and a target output grid \(O_\tau^{*}\). Each grid is transferred to numerical entries in \(\{0,\dots,9\}\), where each integer denotes a grid with corresponding color. Each task is governed by a latent task-specific transformation \(T_\tau\) such that \(O_i = T_\tau(I_i)\) for \(i=1,\dots,n_\tau\), and \(O_\tau^{*} = T_\tau(q_\tau)\).

The support set \(\mathcal{S}_{\tau}\) and query \(q_\tau\) are serialized to the input prompt \(x_\tau\), and the target output \(O_\tau^{*}\) is serialized to the output \(y_\tau\) for LLMs. Thus, each ARC task is converted into a standard sequence-to-sequence example \((x_\tau, y_\tau)\) and can be treated as next grid prediction in a language modeling style.

2.0.0.2 Supervised Fine-Tuning.

Given a autoregressive LLMs \(\pi_{\theta}\), let \(\mathcal{D}_{\mathrm{SFT}}=\{(x_\tau, y_\tau)\}\) denote the training set constructed from serialized ARC tasks, supervised fine-tuning (SFT) optimizes the model by maximizing the conditional likelihood of each token in target sequence given the input prompt: \[\scalebox{0.84}{ \mathcal{L}_{\mathrm{SFT}}^\theta = -\mathbb{E}_{(x_\tau,y_\tau)\sim\mathcal{D}_{\mathrm{SFT}}} \left[ \sum_{t=1}^{|y_\tau|} \log \pi_{\theta}(y_{\tau_t} \mid x_\tau, y_{\tau_{<t}}) \right]. } \label{eq:sft95loss}\tag{1}\]

This method uses all positive samples, i.e., correct input-output pairs, and requires the model to accurately predict the integer corresponding to the next grid at each prediction step during training, as Figure 1 (b) shows.

2.0.0.3 Our Motivations.

Our motivation consists of three parts: 1) The output sequences of ARC tasks are often very long, making it difficult to accurately predict at each step from a generative perspective. 2) SFT-based methods focus on local information, whereas ARC tasks are designed with global grid characteristics in mind. 3) SFT-based methods only rely on a large number of positive samples and lack guidance from incorrect outputs. Compared with generation, holistically judging which output is more reasonable is a relatively easy skill to acquire, yet it is a crucial ability for ARC reasoning. Thereby, inspired by preference alignment post-training [21], we consider using preference optimization to further enhance the model’s ability to solve ARC tasks.

3 Method↩︎

3.1 Overview↩︎

Given the task with support set \(\mathcal{S}_{\tau}\), query \(q_\tau\), target \(O_\tau^{*}\), the serialized input and output \((x^*, y^*)\), we denote the chosen output by \(y^{+}=y^{*}\). We aim to optimize the model not only to prefer the correct output, but also to reject plausible yet rule-incorrect alternatives during post-training.

To this end, we use a negative samples generator (\(\mathcal{NG}\) in short) to construct a rejected output for the query while keeping the support demonstrations unchanged. We denote negative sample \(y^{-}=\mathcal{NG}(\mathcal{S}_{\tau}, q_\tau, y^{+})\), where the implementation of \(\mathcal{NG}\) depends on the negative-construction strategy. We then obtain a preference dataset \(\mathcal{D}_{\text{pref}}=\{(x,y^{+},y^{-})\}\), where \(y^{+}\) is the chosen output and \(y^{-}\) is the rejected output. To construct informative rejected alternatives, we generate negatives at three levels ranging from visual modifications of the output grid to program-level rule reversal, including output-level visual transformation (\(\S\)3.2), DSL-level rule inversion (\(\S\)3.3), task-specific rule editing (\(\S\)3.4), and we introduce them as follows.

3.2 Output-Level Visual Transformations↩︎

The first strategy \(\mathcal{NG}_{\mathrm{Vis}}\) constructs negatives directly in the output space through visual transformations of the output grid, motivated by the strongly visual nature of ARC tasks [22], as illustrated in Figure 2 (b). The key idea is to generate candidate outputs that remain visually and structurally plausible while deviating from the correct transformation rule. We use four families of atomic operations, yielding 28 transformations in total.

The first family, grid-block operators, contains three transformations: pooldownup, which applies \(2\times2\) mode pooling followed by upsampling; modecenter, which replaces only the center cell of each \(3\times3\) block with the block mode; and removesmall, which removes connected components smaller than a preset size threshold. These operations preserve coarse layout while damaging fine-grained structure.

The second family, rigid shifts, contains 16 transformations. We shift the full grid content in eight directions \(\{\mathcal{U},\mathcal{D},\mathcal{L},\mathcal{R},\mathcal{UL},\mathcal{UR},\mathcal{DL},\mathcal{DR}\}\) and use two magnitudes, 10% and 30%, with newly exposed cells filled by the background color. This produces spatially coherent but misaligned outputs.

The third family consists of seven morphological and geometry-related operators: erosion, dilation, skeleton, edge, boundary, convexhull, and fillholes. Intuitively, these operators thin, expand, outline, simplify, or complete objects while keeping the overall spatial layout largely unchanged.

The fourth family consists of two local perturbation operators: noise5, which injects 5% salt-and-pepper noise, and swap10, which swaps 10% of boundary pixels with neighboring pixels of different colors. These operations introduce localized appearance and boundary inconsistencies while largely preserving the global arrangement of objects in the grid.

3.3 DSL-Level Rule Inversion↩︎

ARC tasks are represented in a higher-level domain-specific language (DSL) as executable transformation programs built from predefined primitives. Prior works represent solutions as executable programs and derives additional supervision through program mutation or decomposition [23], [24]. Based on them, the second strategy \(\mathcal{NG}_{\mathrm{DSL}}\) construct negatives at the program level by inverting recurring local rule patterns in task-specific transformation programs, as shown in Figure 2 (c).

We first analyze a pool of task-specific programs that map input grids to output grids, identify recurring local DSL patterns across tasks, and retain those with relatively stable semantics. To obtain a reusable pattern set, we apply a greedy coverage criterion: at each step, we select the candidate pattern whose rewrite covers the largest number of previously uncovered tasks. For each retained pattern \(m\), we then define a unified program-level rewrite \(\phi_m\) that acts as a semantic reversal of the original local rule.

Given a task program \(p\) and query input \(q\), the original program produces the chosen output \(y^{+}=p(q)\), while the rewritten program produces the rejected output \(y^{-}=\phi_m(p)(q)\). The resulting negative is therefore rule-incorrect due to a localized semantic inversion, while remaining close to the original task logic.

3.4 Task-Specific Rule Editing↩︎

Motivated by program refinement and explicit rule-guided code editing based works [25], [26], the third strategy \(\mathcal{NG}_{\mathrm{Edit}}\) constructs negatives at the level of individual tasks by using LLM-assisted edits to each task’s transformation rule, as Figure 2 (d) shows.

Concretely, for a task-specific transformation program, we use GPT-5.4 [27] to assist rule editing and construct an alternative rule that is as close as possible to the semantic opposite of the original rule. When a clean reversal is not feasible, GPT-5.4 instead generates a task-specific alternative rule that stays close to the original transformation logic but leads to a different transformation behavior. The resulting negative therefore stays close to the original solution in overall structure and execution pattern, while deviating in the task-relevant rule itself.

3.5 Preference Optimization Pipeline↩︎

After obtaining the negative samples and the constructed preference data, we train the model with direct preference optimization (DPO; [28]). Let \(\pi_{\theta}\) denote the policy model and \(\pi_{\mathrm{ref}}\) the reference model. For each preference triple \((x,y^{+},y^{-})\), DPO encourages the model to assign higher relative likelihood to the chosen output than to the rejected output: \[\scalebox{0.8}{ \begin{align} \mathcal{L}_{\mathrm{DPO}}^\theta = -\mathbb{E}_{(x,y^{+},y^{-})\sim\mathcal{D}_{\mathrm{pref}}} \log \sigma \Bigg( \beta &\log \frac{\pi_{\theta}(y^{+}\mid x)}{\pi_{\mathrm{ref}}(y^{+}\mid x)} \\ - \beta &\log \frac{\pi_{\theta}(y^{-}\mid x)}{\pi_{\mathrm{ref}}(y^{-}\mid x)} \Bigg), \end{align} } \label{eq:dpo95pair}\tag{2}\] where \(\beta\) is a temperature parameter. In practice, we initialize \(\pi_{\theta}\) from \(\pi_{\mathrm{SFT}}\) and use a frozen copy of \(\pi_{\mathrm{SFT}}\) as the reference model \(\pi_{\mathrm{ref}}\). Unlike reinforcement learning methods that require an explicit reward model, DPO directly optimizes pairwise preferences between correct and incorrect outputs. This makes it particularly suitable for ARC, where supervision is sparse and many errors are better characterized as plausible but rule-incorrect alternatives rather than compressed into a single reward score. Figure 3 shows the overall pipeline of preference data construction and DPO training.

4 Experiments↩︎

4.1 Experimental Setup↩︎

Dataset and Evaluation. We evaluate on six benchmarks that cover the original ARC setting, its harder successor, and four related ARC-style transfer or additional benchmarks. ARC-AGI-1 [1] is the original ARC benchmark for few-shot abstract rule induction from grid-based input-output examples, and we use its 400-task public evaluation set. ARC-AGI-2 [17] is a harder successor that preserves the same input-output format while providing a more challenging test of abstract reasoning and problem solving, and we use its 120-task public evaluation set. MiniARC [18] is a compact 5\(\times\)​5 ARC-style benchmark designed to reduce variation in grid size and focus more directly on abductive reasoning. ConceptARC [19] organizes ARC-style tasks into concept groups in order to evaluate abstraction and generalization over recurring spatial and semantic concepts. 1D-ARC [20] is a one-dimensional ARC-style benchmark that simplifies the original two-dimensional grid setting into one-dimensional transformation tasks while preserving the core requirement of abstract rule induction. ARCcommunity¹ is a collection of community-created ARC-style tasks that extends evaluation beyond the original benchmark distribution and provides an additional test of generalization to diverse human-designed problems.

For the standard evaluation on ARC-AGI-1 and ARC-AGI-2, each test input allows two candidate outputs and is counted as correct if either exactly matches the ground-truth grid. If a task contains multiple test inputs, the task score is the average over its test inputs. For MiniARC, ConceptARC, 1D-ARC, and ARCcommunity, we report single-response exact-match accuracy.

Models. We use LLMs across three families including Llama-3.2-3B [29], Mistral-NeMo-Minitron-8B [30], and Qwen3-4B [31]. During training, we optimized their vocabulary and the effective fine-tuned parameter size is slightly reduced.

4.2pt

Preference Data Construction. For ARC-AGI-1, all three negative-construction strategies are built on top of RE-ARC [9], which provides task-specific generators for the 400 original ARC training tasks. For output-level visual transformation, we first use these generators to sample ARC tasks and then apply output-space transformations to construct rejected outputs. For DSL-level rule inversion and task-specific rule editing, we operate on the corresponding task-specific transformation programs to generate chosen/rejected pairs. For the remaining five benchmarks, RE-ARC-style per-task generators are unavailable. We therefore use only the output-level visual transformation strategy for preference construction. For ARC-AGI-2, preference data is constructed from the released NVARC-full [11] resources.

For MiniARC, ConceptARC, 1D-ARC, and ARCcommunity, we use their official training sets to construct preference data with the same ARC-oriented negative-construction pipeline. Across all six benchmarks, we keep the overall preference-learning setup as consistent as possible: the original support demonstrations and query input are always preserved, and only the rejected output is varied according to the available negative-construction strategy. This design helps isolate the contribution of DiARC from other sources of variation and makes cross-benchmark and cross-backbone comparisons easier to interpret.

Detailed statistics of the constructed preference datasets used in these experiments are provided in Appendix 8.

Training Details. We train all models with a shared setup including learning rate \(1\times10^{-6}\), effective batch size 8, cosine learning schedule, warmup ratio 0.1, maximum sequence length 4096, and maximum prompt length 3584. LoRA [48] fine-tuning is used and the model-specific settings are summarized in Appendix 9. All experiments are conducted on a single NVIDIA L40 GPU.

4.2 Main Results↩︎

Tables 1 and 2 show the results on all six ARC-style benchmarks, Table 3 breaks down the results with different output-level negative families, and Table 4 compares the DSL-level inversion and task-specific editing on ARC-AGI-1.

4.2.0.1 DiARC brings broad gains across backbones and benchmarks.

As shown in Tables 1 and 2, DiARC improves the performance of SFT baselines on all six benchmarks for Llama-3.2-3B, Qwen3-4B, and Minitron-8B, with an average absolute gain of 2.48 points across the 18 model–benchmark pairs. The gains are not limited to weaker SFT baselines but also strong ones like Qwen3-4B. The gains also do not scale monotonically with model size: all three backbones benefit from DiARC, with average improvements of 2.52, 2.21, and 2.72 points for Llama-3.2-3B, Qwen3-4B, and Minitron-8B, respectively. When using the Qwen3 backbone model, DiARC achieves an accuracy of over 96 on ARC-AGI-1, MiniARC, and ConceptARC, surpassing both closed-source models and existing open-source model methods, demonstrating its effectiveness.

4.2.0.2 Useful preference signals can be constructed at multiple levels.

Table 3 shows that the four output-level families improve over the initialization models on average, with morphology giving the strongest mean performance, although the best family varies across datasets and backbones. One possible reason is that ARC is built around a set of human-like priors, especially objectness as well as basic geometry and topology [1]. Compared with other perturbations, morphology-based negatives more directly alter object boundaries, connectivity, and other structural properties while preserving the object layout. This makes them more likely to stay within the kind of object-centric transformation space that ARC is intended to test, and therefore more likely to provide informative near-miss alternatives. At the same time, the most effective negative family differs across backbones, which may be related to differences in the models’ underlying capabilities, as they may develop different inductive biases and characteristic error patterns, and thus benefit from different forms of hard negative supervision.

Table 4 shows that both DSL-level inversion and task-specific editing provide gains on ARC-AGI-1. Under task-specific editing, however, we observe a temporary performance drop when around 200 tasks are edited, followed by a recovery as the number of edited tasks increases. This is because LLM-assisted editing tends to map negative samples into a few recurring task types, such as set/extraction operations, compressed summaries, and bounding-box or contour extraction. When the edited set is relatively small, these recurring patterns can dominate the training signal and make the model biased toward a narrow set of negative constructions. As the edited set becomes larger and more diverse, this bias is diluted, allowing the model to better capture the intended contrast between positive and negative rules. Taken together, results suggest that the benefit of DiARC does not depend on a single negative-construction recipe.

5 Analysis↩︎

5.1 Impacts of Generation and Discrimination↩︎

Our ARC system first generate candidate solutions and then use product-of-experts scoring to select the final answer. DiARC can therefore improve performance through two different channels: it may increase the number of correct candidate solutions produced by the model, or it may improve the model’s ability to assign higher preference to correct candidates that are already generated.

To distinguish these effects, we define two gain quantities. Let generation gain be the number of correct candidates generated after DiARC training but not generated by the base model, minus the number of correct candidates generated by the base model but no longer generated after DiARC training. Let discrimination gain be the number of correct candidates that are generated in both settings, not selected by the base model, but selected after DiARC training, minus the number of correct candidates that are generated in both settings, selected by the base model, but not selected after DiARC training. Positive generation gain therefore indicates a net gain in candidate generation, while positive discrimination gain indicates a net gain in candidate selection.

Table 5 summarizes these effects across six benchmarks. Overall, both quantities are positive on average: the mean generation gain is \(+2.19\) and the mean discrimination gain is \(+2.44\), while the mean improvement in solved items is \(+4.63\) under the benchmark-specific evaluation metric. These results suggest that the gains from DiARC arise from both stages of the pipeline: the model produces more correct candidates and also becomes better at assigning higher preference to correct candidates once they are generated.

5.2 Compatibility with Test-Time Scaling↩︎

Different test-time scaling (TTS) methods have been used to improve reasoning performance of LLMs. We further analyze whether DiARC is compatible with different techniques. Following [8], we combine base and DiARC-enhanced ones with test-time training (TTT), task augmentation (Aug), reranking (PoE, product-of-experts scoring), and searching (DFS, depth-first search) methods, comparing how improvements in the underlying models propagate to final accuracy.

Table 6 shows the results. Across different TTS techniques, DiARC-enhanced model remain compatible and preserve competitive performance relative to their corresponding base models. This suggests that DiARC is not limited to direct model evaluation, but can also be integrated into advanced test-time scaling pipeline for model improvement.

5.3 Case Study↩︎

Figure 4 shows the prediction results of Llama-3.2-3B model on ARC-AGI-1 task 31adaf00 and see Appendix 10 for more cases. In this task, the main challenge lies in identifying all valid blank pockets rather than generating a new structure. We suggests that the output-level perturbation based negative sample is helpful when dealing with similar problems, because it preserves the global layout while removing small valid filled regions, closely matching the model’s typical under-completion errors. Thereby, our model gives the precise prediction while the original model does not.

6 Related Work↩︎

6.0.0.1 Improving ARC Reasoning of Models.

Existing approaches to improving ARC reasoning ability can be broadly grouped into three types: (1) recursive models [36], [37], [49] view ARC as an iterative reasoning process in which intermediate states must be repeatedly refined, and therefore improve performance by revisiting and updating candidate solutions through repeated computation; (2) vision models treat ARC as a visual reasoning problem [22], [50], [51], formulating ARC problems as an image-to-image or grid-to-grid prediction task and exploit visual inductive biases to improve performance on ARC-style benchmarks; and (3) large language models solve ARC-style problems by redesigning task representations, prompting strategies, and ARC-oriented training data. [20] shows that object-based representations can substantially improve GPT-style reasoning on ARC. Subsequent works explore explicit hypothesis generation and program verification [52], [53], and self-improving program synthesis through hindsight replay [23]. More recent approaches improve ARC reasoning through data augmentation and synthetic program-based supervision [10], [54]. These efforts have made some progress, but have yet to tap into the potential improvement of negative samples for ARC reasoning ability.

6.0.0.2 Preference Alignment of LLMs.

Preference alignment has become a standard paradigm for adapting LLMs to desired behaviors. Early work studies learning from human preferences and reinforcement learning from human feedback, with the goal of making model outputs more helpful, honest, and harmless [21]. More recently, direct preference optimization methods simplify this pipeline by optimizing pairwise preferences without separately training an explicit reward model [28]. A growing body of work has further explored simpler and more efficient preference-based objectives, such as reference-free or monolithic optimization variants [55], [56]. Our work extends this line of research to ARC-style reasoning by constructing task-specific chosen-rejected output pairs, so that the model learns not only to fit correct outputs, but also to distinguish them from plausible yet rule-incorrect alternatives.

7 Conclusion↩︎

We introduced DiARC, a preference alignment framework for ARC-like tasks by training LLMs to distinguish correct outputs from task-specific rejected alternatives. We proposed three negative sample construction strategies operating at different levels: output-level visual transformations, DSL-level rule inversion, and task-specific rule editing. Across six ARC-style benchmarks, our method consistently improves the ARC reasoning ability of open-source LLMs, outperforms ARC-specialized and strong closed-source models. These results suggest that, with suitable constructed negative samples, preference alignment can also be a effective post-training techniques for improving ARC-style reasoning ability of LLMs.

Limitations↩︎

First, the full negative-construction pipeline of our method depends on auxiliary ARC resources, especially RE-ARC task generators and task-specific transformation programs. At present, such standardized per-task generation resources are available for ARC-AGI-1, but comparable RE-ARC-style generators are not yet available for the remaining benchmarks. As a result, all three negative-construction strategies can be applied on ARC-AGI-1, but only the output-level visual transformation strategy is used on other benchmarks. Second, our rejected outputs are constructed automatically rather than annotated by humans, so their informativeness and semantic quality may vary across tasks and construction methods, and the performance can be further improved when negatives are reasonably manually evaluated and assigned.

Acknowledgments↩︎

This work was supported in parts by Guangdong Basic and Applied Basic Research Foundation (2026A1515011358), Shenzhen Natural Science Foundation (JCYJ20250604181610014), and Intelligent Computing Center of Shenzhen University.

8 Data Statistics↩︎

This appendix provides summary statistics of the constructed preference data used in our experiments. Table 7 reports the number of samples for output-level negative construction on the six ARC-style benchmarks, together with the additional ARC-AGI-1 data used for the DSL-level inversion and task-specific editing settings.

4.5pt

9 Additional Training Details↩︎

8pt

10 Case Study↩︎

We present more representative cases on ARC-AGI-1 in Figure 5, and on ARC-AGI-2 ,MiniARC, ConceptARC, 1D-ARC, and ARCcommunity in Figure 6, respectively.

11 Additional Visualizations of Negative Construction Strategies↩︎

We provide additional visualizations of the negative-construction operations that are not individually shown in the main pages.

11.1 Output-Level Visual Transformations↩︎

Figures 7 and 8 show additional examples of output-level visual transformations.

11.2 DSL-Level Rule Inversion↩︎

Figures 9 and 10 show additional examples of DSL-level rule inversion.

11.3 Task-Specific Rule Editing↩︎

Figure 11 shows additional examples of task-specific rule editing. We used the prompt template in Figure 12 to guide GPT-5.4 in editing task-specific transformation rules and constructing negative samples.

12 AI Assistance Statement↩︎

We utilized GPT-5.4 to assist in polishing the text, refining grammar, and improving the clarity of the manuscript. All content was reviewed and verified by the authors. We also employed GPT-5.4 in the task-specific rule editing process described in Appendix 11.3. The generated content was reviewed and verified by the authors.

References↩︎

---

1. https://github.com/arc-community/arc↩︎