1 Introduction↩︎

Recently, enabling a large-scale model to improve itself has gained large traction in the AI community [1]–[3]. This venture actually can be found across the history of AI—from early symbolic self‑modification to the self‑play dynamics that powered breakthroughs in games like chess, and beyond [4], [5]. With the rapid advancement of large-scale pretrained models, realizing their effective self-improvement has emerged as a pivotal challenge on the path toward artificial general intelligence (AGI), promising to endow these models with an unprecedented degree of adaptability and generalizability. However, for this moment, self-improvement is a research area whose definition and boundary are not crystal clear.

To the best of our knowledge, self-improvement in contemporary research mainly concerns four schools: i) Continual learning provides the machinery for models to adapt to new tasks without catastrophic forgetting, employing elastic weight consolidation, dynamic architecture expansion, or episodic memory replay so that each time of learning builds upon previous knowledge rather than erasing it [6]; ii) Self-distillation leverages the model as its own oracle: the model is iteratively retrained on softened distributions produced by its own earlier checkpoints, which progressively sharpens predictions without external ground truth [7], [8]; iii) Test-time adaptation, in contrast, updates model statistics or a small subset of parameters on a per-sample or per-distribution basis using unsupervised objectives like entropy minimization, pseudo-labeling, or self-supervised rotation prediction, which allows the model to rapidly conforming to individual user patterns after deployment [9]; iv) Training-free finetuning explores token-level attention modulation for improving foundation model performance without any parameter modifications [10], [11].

It can be seen that a diverse spectrum of approaches are related to model self-improvement. Yet, no single methodology has yet crystallized into a dominant trajectory. Our intention here is not to reconcile or unify these existing routes. Rather, given the multi-faceted nature of self-improvement, we take a step back and design a new mechanism in a principled way, thereby offering a perspective that is inherently distinct from existing lines. We turn our attention to the roots of brain intelligence and seek inspiration from our brain. This aligns with the idea of “NeuroAI" [12]–[14] which argues that insights from neuroscience can greatly accelerate the development of next‑generation AI. This claim is well grounded: the brain remains the most intelligent system known, and artificial neural networks can be viewed as simplified abstractions of it. Thus, mechanisms of neuroscience can more or less shed light on the challenges in AI.

Motivated by the NeuroAI paradigm, it is noted that human’s self-improving is often sudden, manifesting as the so-called “aha moment" or enlightenment. In the human brain, enlightenment is not merely subjective flashes of clarity. Neuroimaging and electrophysiological studies suggest that such insights often involve the abrupt recruitment of the anterior superior temporal gyrus and the anterior cingulate cortex, accompanied by a transient burst of gamma‑band activity [15]. In addition, enlightenment often appears during resting time when active learning or training has ceased. This resting-time‑driven plasticity change can sometimes establish entirely new microcircuits, thereby enabling a qualitative leap in performance rather than incremental adjustment [16]. Given the compelling characteristics of enlightenment, can we introduce the related mechanism into large models and examine the associated computational benefits in self-improvement?

Translated to artificial systems, enlightenment suggests that a new operational phase in manipulating models beyond training and testing, referred to as retrospection, is important, which tunes models in the resting time and not necessarily relying on data. More importantly, the model may wire new circuits to improve itself during retrospection [17]. Furthermore, we posit that enlightenment in artificial networks should fulfill the following conditions: i) no gradient is computed, since we highlight the sudden change; ii) no data is involved, since we highlight the resting-time adjustment that is before testing and after training; and iii) no weight update is needed, since we highlight modifying circuits. Per these conditions, we propose Enlightenment, a novel and highly effective training-free retrospection method for large models. As Figure 1 shows, we focus on modifying the network structure, with an emphasis on shortcuts, and we even inject connections that do not exist previously into large-scale models. Table 1 shows that the proposed Enlightenment occupies a unique niche relative to other techniques. Particularly, the existing training-free finetuning methods focus on plasticity adjustments, while our method emphasizes internal structures.

Specifically, guided by the principle of modifying shortcuts, we implement Enlightenment for VLMs and LLMs, respectively. For VLMs, we apply a single shared scalar modulator to every residual connection within the Transformer decoder layers, covering both the attention and feed-forward network (FFN) sub-blocks. For LLMs, we take a bolder step: inserting new shortcuts to aggregate information across per-head attention outputs from the head to the rest. The efficacy of such intra-layer shortcuts is corroborated by prior work [18], [19]. Furthermore, to complement these architectural augmentations, shortcuts are assigned with dynamic scaling hyperparameters, providing a per-head level recalibration. Enlightenment expands the representational capacity of large models without altering pre-existing weights. Meanwhile, our method introduces only a small number of hyperparameters, and the effect of self-improvement is insensitive to most of them. In brief, we summarize our contributions as threefold:

• Inspired by the neuroscience of enlightenment, we introduce Enlightenment, a training‑free tuning paradigm that operates in a novel retrospection phase between training and inference. This paradigm focuses on shortcuts instead of weights.

• To instantiate this paradigm, we propose a lightweight architectural intervention strategy that modulates shortcut connections via scaling hyperparameters and even inserting cross‑head shortcuts, thereby expanding representational capacity without post‑training.

• Extensive experiments on diverse multimodal and reasoning benchmarks demonstrate that Enlightenment consistently delivers salient performance improvements across three widely-used VLM and three mainstream LLM variants, while requiring no additional data, no gradient computation, and no weight updates.

2 Related Work↩︎

2.1 Continual Learning↩︎

Continual learning (CL), also referred to as lifelong learning, addresses the challenge of enabling LLMs to acquire new knowledge without forgetting previously learned tasks [20], [21]. Continual learning can be regarded as a form of self-improvement. Recently, continual instruction tuning (CIT), which refines a large model’s instruction-following ability [22], [23], has been intensively investigated.

i) Task-incremental CIT fine-tunes LLMs on sequential task-specific instructions, with methods such as Progressive Prompts [24] freezing most parameters and learning per-task prompt tokens to reduce computation and forgetting, while the scalable language model [25] incorporates vector space retrieval for knowledge expansion. ii) Domain-incremental CIT adapts LLMs to evolving domains. Task-adaptive pre-training (TAPT) [26] employs a data selection strategy that retrieves in-domain unlabeled text for fine-tuning, and ConPET [27] combines parameter-efficient tuning with dynamic replay to prevent forgetting. iii) Tool-incremental CIT represents a particularly promising direction, extending LLMs beyond text generation to real-world tool use. ToolLLM [28] demonstrates that LLMs can master over 16,000 real-world APIs through CIT, enabling appropriate tool selection and invocation for diverse tasks. ToolkenGPT [29] takes a complementary approach by augmenting frozen language models with tool embeddings, updating only tool-specific parameters to eliminate catastrophic forgetting while continuously expanding tool capabilities.

2.2 Self-Distillation↩︎

Self-Distillation (SD) transfers knowledge from a model or its historical states, enabling self-evolution without an external teacher network [30]. In the instruction-following domain, Self-Instruct [31] establishes a closed-loop pipeline for autonomously generating, filtering, and refining high-quality fine-tuning data. For preference alignment, self-play fine-tuning (SPIN) [32] casts the process as an adversarial self-play game between pseudo-responses and gold-standard SFT trajectories under a direct preference optimization (DPO) objective, breaking reliance on third-party evaluators. Self-distillation fine-tuning (SDFT) [33] further addresses manifold contraction from multi-stage fine-tuning by using an undegraded historical model state as a distillation anchor, recovering broad semantic capabilities without sacrificing task-specific performance.

In reasoning domains, SD has enabled breakthroughs in long-chain logical reasoning and code generation. On-policy self-correction (OPSD) [34] trains a single LLM in dual roles—a blind student and a privileged teacher—applying KL or Jensen-Shannon divergence as a probabilistic penalty over on-policy solution trajectories. Simple self-distillation (SSD) [35] takes a simpler route, using the model’s own unverified temperature-sampled outputs as cross-entropy targets to reshape the token-level decoding state, suppress long-tail trajectory drift in chain-of-thought generation, and consolidate both diversity and generalization.

2.3 Test-Time Adaptation↩︎

Test-Time Adaptation (TTA) updates model parameters during inference to mitigate distributional shifts without costly offline retraining [9], fundamentally blurring the boundary between training and deployment [36]. Early TTA methods address the absence of ground-truth labels through self-supervised objectives, such as token-level entropy minimization [37] and next-token likelihood optimization [38], combined with selective parameter-efficient strategies including isolated layer updates, sparse weight adjustments, and low-rank parameter-efficient fine-tuning (PEFT) to scale gradient updates to billion-parameter models while preserving foundational linguistic capabilities [39].

The optimization landscape of test-time fine-tuning increases risks of catastrophic forgetting and localized overfitting on single instances or small batches [40]. To mitigate these instabilities, recent work increasingly frames TTA through the lens of meta-learning and online optimization, leveraging principles from model-agnostic meta-learning [41] and Reptile [42] to optimize initial pre-trained states for rapid downstream adaptation.

2.4 Training-Free Finetuning for Large Models↩︎

Training-free tuning methods have garnered considerable attention as effective strategies for enhancing the inference-time performance of LLMs and VLMs without incurring additional computational overhead from parameter optimization. This kind of methods mainly explores token-level attention modulation for improving foundation model performance without any parameter modifications. These methods are based on the well-documented “attention sink” phenomenon, wherein a disproportionately large share of attention weight is systematically concentrated on the initial Beginning-of-Sequence (BOS) token [10], [11], [43]–[45]. Attention sink motivates weight re-distribution techniques. Representative approaches such as Post-hoc Attention Steering (PASTA, [46]) and AutoPASTA [47] identify and selectively amplify the attention weights assigned to task-relevant tokens, while Attention Calibration (ACT) suppresses non-initial sink tokens to redistribute attention more effectively [48]. This principle has also been extended to vision-language models, where attention re-weighting toward visual tokens has been shown to effectively mitigate hallucinations [49], [50]. More recently, Han et al. introduced ZeroTuning that applies head-specific attention adjustments to the BOS token, demonstrating consistent and generalizable performance improvements across foundation LLMs [10].

As Table 1 shows, Enlightenment represents a new paradigm for training-free finetuning methods. Our opinion is that weights and architecture are the two cornerstones of a model. While existing training-free adaptations predominantly manipulate attention weights, our work reveals that we can also re-allocate weight distribution from an architectural perspective. This architectural view opens up an orthogonal yet equally promising direction, which is to treat model distribution as a global structural property rather than a parametric one. To the best of our knowledge, this view goes beyond weight-centric finetuning paradi- gms, which was not explored before.

3 Methodology↩︎

The overall framework of Enlightenment is illustrated in Figure 1. The motivation behind our approach draws a loose conceptual analogy to neuroscience, where sudden insights in human learning are sometimes associated with rapid reorganization of neural pathways. We adopt this only as a high-level intuition, without claiming a direct biological basis. To reflect this notion in machine learning, we modify structures by either adjusting existing or establishing new shortcuts within pre-trained models, enabling them to self-improve their performance during inference without requiring additional post-training.

The unique characteristics of Enlightenment are that it requires no training data, no gradient computation, and no weight updates. It is built upon a unified principle: modifying shortcuts for key modules/layers by exerting scaling factors over them, while establishing new shortcuts can be thought as turning the scaling factor from 0 to 1 for otherwise absent paths. The standard residual connection offers no mechanism to control the mixing between the residual stream and the sublayer output at inference time. Specifically, shortcuts are assigned with dynamic scaling parameters, ensuring the network can dynamically calibrate its attention to specific layers and more precisely govern the propagation strength of features. As a result, Enlightenment can enrich the representational capacity of large models without altering or tuning their pre-trained weights. What’s more favorable is that this enhanced expressivity is generalizable across diverse tasks and model families.

While the underlying principle is shared, the integration points and companion modules regarding shortcut modification could differ across different model families. From an information-flow perspective, multimodal networks require residual damping to suppress visual noise, while text-only models require attention modulation to redistribute linguistic focus [11], [51], [52]. Now, we elaborate on how we perform shortcut modification on VLMs and LLMs, respectively.

3.1 Enlightenment for LLM↩︎

LLMs frequently exhibit an “attention sink” phenomenon, wherein disproportional attention weight is systematically allocated to the initial BOS token [10], [11], [43]–[45]. Beyond attention weights modification, our proposed method recalibrates BOS attention behavior by mixing heads with scaled shortcuts. This provides a per-head level recalibration rather than globally suppressing BOS attention. In this section, we first examine the basic mechanics of the attention in LLMs before detailing the Enlightenment integration.

Attention mechanism. Modern LLMs rely on the multi-head attention (MHA) mechanism to capture complex dependencies across tokenized sequences [53]. We define an input sequence as a matrix \(\mathbf{X}\in\mathbb{R}^{L\times d_{\text{model}}}\), where \(L\) denotes the sequence length (total number of tokens) and \(d_{\text{model}}\) represents the hidden state dimension.

To process information across distinct representation subspaces, the hidden dimension \(d_{\text{model}}\) is split into \(H\) parallel attention heads, each operating on a lower-dimensional feature space \(d_h=d_{\text{model}}/H\). For a given attention head \(h\in\{1,\dots,H\}\), the input activation sequence \(\mathbf{X}\) is linearly projected into query (\(\mathbf{Q}_h\)), key (\(\mathbf{K}_h\)), and value (\(\mathbf{V}_h\)) matrices using learned projection weight matrix: \[\mathbf{Q}_h=\mathbf{X}\mathbf{W}_q^h,\quad\mathbf{K}_h=\mathbf{X}\mathbf{W}_k^h,\quad\mathbf{V}_h=\mathbf{X}\mathbf{W}_v^h.\] Here, \(\mathbf{W}_q^h,\mathbf{W}_k^h,\mathbf{W}_v^h\in\mathbb{R}^{d_{\text{model}}\times d_h}\) are head-specific projection matrices. The scaled dot-product attention for the \(h\)-th head maps the queries and keys to a valid probability distribution via the softmax operator, weighting the corresponding values accordingly: \[\begin{align}\mathbf{A}_h&=\mathrm{softmax}\left(\frac{\mathbf{Q}_h\mathbf{K}_h^\top}{\sqrt{d_h}}\right)\in\mathbb{R}^{L\times L}\\\mathbf{O}_h&=\mathrm{AttnOutput}(\mathbf{Q}_h,\mathbf{K}_h,\mathbf{V}_h)=\mathbf{A}_h\mathbf{V}_h\in\mathbb{R}^{L\times d_h}.\end{align} \label{eq:ah}\tag{1}\] In this formulation, \(\mathbf{A}_h\) represents the post-softmax attention weight matrix, where each element \(A_{h,q,k}\) defines the normalized alignment score between the \(q\)-th query token and the \(k\)-th key token. The factor \(\sqrt{d_h}\) acts as a regularizing scaling variance to alleviate vanishing gradients during backpropagation.

The individual head outputs \(\mathbf{O}_h\) are stacked across the head dimension to a tensor \(\mathbf{O}\in\mathbb{R}^{H\times L\times d_h}\) (or batched as \(\mathbb{R}^{B\times H\times L\times d_h}\)). In a vanilla transformer, these heads are aggregated via a learned linear combination using the output projection weights \(\mathbf{W}_o^h\in\mathbb{R}^{d_h\times d_{\text{model}}}\): \[\mathrm{MHA}(\mathbf{X})=\sum_{h=0}^H\mathbf{O}_h\mathbf{W}_o^h. \label{eq:mha}\tag{2}\]

Enlightenment modifies the attention head \(\mathbf{A}_h\) across a designated subset of target layers \(\mathcal{L}\) and heads \(\mathcal{H}\) through head-to-head information mixing. By coupling the recalibration with the scattering shortcuts, the model successfully redistributes its linguistic focus. The overall framework is depicted in Figure 2.

Attention head mixing by inserting new shortcuts. First, let \(A_{b,h,q,k}\) denote the attention weight that the \(q\)-th query position assigns to the \(k\)-th key position in the \(h\)-th head. For a target layer \(\ell \in \mathcal{L}\) and head \(h \in \mathcal{H}\), leveraging the attention sink, we assign a scaling parameter \(\gamma_{\ell,h} > 0\) to the initial attention matrix: \[\begin{cases} & \bar{A}_{b,h,q,0} = \gamma_{h} \cdot A_{b,h,q,0} \\ & \bar{A}_{b,h,q,\geq1} = A_{b,h,q,\geq1}, \end{cases} \label{eq:bos95scaling}\tag{3}\] and normalize the values of newly scaled attention matrices \(\bar{A}\) across all positions: \[\tilde{{A}}_{b,h,q,:} = \frac{{\bar{A}}_{b,h,q,:}}{\sum_{k}\bar{A}_{b,h,q,k}}. \label{eq:attnscale}\tag{4}\] According to Eq. 1 , the output of the \(h\)-th attention head is \[{\mathbf{O}}_h = \tilde{\mathbf{A}}_h\mathbf{V}_h= [\tilde{\mathbf{A}}_{h,0}\mathbf{V}_{h,0}, \tilde{\mathbf{A}}_{h,1}\mathbf{V}_{h,1},\cdots,\tilde{\mathbf{A}}_{h,n}\mathbf{V}_{h,n}].\]

After that, we establish new shortcuts that link the output of head 0 to all other target heads. For each edge from source head 0 to a target destination head \(h\), we set a mixing rate \(\alpha\) to control the additive injection. The matrix combining all heads’ output is \[\tilde{\mathbf{O}} = [\tilde{\mathbf{A}}_0 \mathbf{V}_0, \alpha \tilde{\mathbf{A}}_0 \mathbf{V}_0 + \tilde{\mathbf{A}}_1 \mathbf{V}_1, \cdots , \alpha \tilde{\mathbf{A}}_0 \mathbf{V}_0 + \tilde{\mathbf{A}}_H \mathbf{V}_H].\] When \(\alpha = 1\), the attention module remains unmodified. This process enriches the representation of the \(h\)-th head with contextual contributions from the initial head, which is a feature reuse.

Finally, based on Eq. 2 , the output of the MHA is \[\begin{align} \mathrm{MHA}(\mathbf{X}) &= \tilde{\mathbf{A}}_0 \mathbf{V}_0\mathbf{W}_{0}+ (\alpha \tilde{\mathbf{A}}_0 \mathbf{V}_0 + \tilde{\mathbf{A}}_1 \mathbf{V}_1)\mathbf{W}_{1} + \\ &\cdots + (\alpha \tilde{\mathbf{A}}_0 \mathbf{V}_0 + \tilde{\mathbf{A}}_H \mathbf{V}_H)\mathbf{W}_H. \end{align} \label{eq:emha}\tag{5}\] These new shortcuts make the LLM dynamically redistribute linguistic focus from the initial attention block to other heads without introducing additional weights or training overhead.

Adaptive hyperparameter modulation. Hyperparameters \(\Gamma=[\gamma_1, \gamma_2, \cdots, \gamma_H]\) and \(\alpha\) are introduced in Enlight- enment. Among them, \(\alpha\) is a global parameter whose value is easy to determine by grid search. In contrast, in previous works, to alleviate the attention sink, initial heads exhibiting high attention values should be attenuated, while those with low attention should be amplified [10], [11], [43]–[45]. Therefore, \(\Gamma\) should be carefully calibrated for each head across layers to conform to this phenomenon. To this end, we introduce an adaptive strategy:

\(\small \bullet\) First, we calculate the value of the initial attention matrix of each head: \[a_0^{(h)} = \frac{1}{B\cdot Q} \sum_{b=1}^B \sum_{q=1}^Q A_{b,h,q,0}, \label{eq:a0h}\tag{6}\] which serves as a runtime indicator of how strongly the \(h\)-th head attends to the initial position. We normalize \(a_0^{(h)}\) to \([-1, 1]\) via \[\hat{a}_0^{(h)} = 2 \cdot \frac{a_0^{(h)} - a_{\min}}{a_{\max} - a_{\min}} - 1, \label{eq:x0h}\tag{7}\] where \(a_{\max}\) and \(a_{\min}\) are the extreme values in \(a^{(h)}\).

\(\small \bullet\) Second, we aim at constructing a function \(\gamma=f(\hat{a}_0)\). The general principle is that this function should be smooth over mid-range values, large for small attention values, and small for large attention values, thereby producing a more balanced scaling for attention values. To achieve this, we define four anchor pairs to ensure a basic shape for \(f\). Each pair maps a high attention value to a lower \(\gamma\), while a low initial attention value is amplified: \[(\hat{a_0}, \gamma) \in \{(0.05, 1.5), (0.15, 1.1), (0.30, 1), (0.50, 0.7)\}. \label{eq:pair}\tag{8}\] We fit these four anchor points to the Chebyshev polynomial function: \[\gamma_h = T_d(x_h; \mathbf{c}) = \sum_{j=0}^{d} c_j \, T_j(x_h),\] with \(\mathbf{c} = [c_0, \dots, c_d]\) are coefficients, and \(T_j\) represents the \(j\)-th Chebyshev polynomial [54] satisfying \[T_0(x)=1, T_1(x)=x, T_{j}(x)=2x T_{j-1}(x) - T_{j-2}(x). \label{eq:chevb}\tag{9}\]

Figure 3 presents a 3rd-order Chebyshev polynomial function. \(\gamma_h\) of each head can be determined by this function.

Furthermore, we set \(\beta\) as a task-dependent factor, which scales the Chebyshev polynomial \(T_d\) horizontally. \[\gamma_h = \beta\cdot T_d(x_h; \mathbf{c}).\] Adjusting \(\beta\) can effectively handle domain/task shift.

Table 2 presents all the hyperparameters and their default values utilized in Enlightenment for LLMs.

3.2 Enlightenment for VLM↩︎

The core idea is to apply a single shared scalar to every residual connection within the Transformer decoder layers. Pre-trained sublayers, such as attention and Feed-Forward Networks, frequently become overly active during inference [55], [56]. This excessive activity leads to excessive information, introducing visual noise that degrades downstream predictive performance. By uniformly scaling down these sublayer outputs, the method effectively applies residual damping. This mechanism acts as a regularizer, suppressing superfluous sublayer activity while securely preserving the critical residual pathway that carries the core embedding information.

This method integrates scalar-modulated residual into the linear operations across layers, where a scaling hyperparameter can be adjusted to control contributions of these connections. The overall framework is illustrated in Figure 4.

Scalar-modulated residuals. The standard transformer decoder layer computes \(x_{\ell+1} = x_\ell + F_\ell(\text{LN}(x_\ell))\), where \(F_\ell\) is either self-attention or an FFN. We replace this with a single scalar-modulated variant:

\[\label{eq:epiphany} x_{\ell+1} = x_\ell + \eta \cdot F_\ell(\text{LN}(x_\ell)),\tag{10}\]

where \(\eta \in (0, 1]\) is a scalar shared across all \(L\) decoder layers and both sublayer types (attention and FFN). At \(\eta = 1\), the model operates in its unmodified state. At \(\eta = 0.5\), the sublayer outputs are uniformly halved, introducing a form of residual damping that mitigates noise while preserving the primary residual pathway carrying the original embedding information.

The computational overhead is minimal, requiring only one scalar hyperparameter and one additional multiplication operation per layer. Consequently, Enlightenment requires no training data, no gradient computation, no weight updates. Our ablation studies demonstrate that \(\eta\) can be easily tuned within the interval \([0.1, 1]\). Regarding the target layers, applying Enlightenment across all layers serves as a robust default strategy that consistently yields performance improvements.

In summary, Table ¿tbl:tab:HN95structure? compares the structural modifications applied to VLMs and LLMs by the proposed Enlight- enment framework.

4 Experiments on LLMs↩︎

4.1 Experimental Setup↩︎

Models. We evaluate Enlightenment on three models: Qwen3.5-9B-Base [57], DeepSeek-R1-Distill-Qwen-14B [58], and Mimo-7B-RL [59]. Specifically, Qwen3.5 uses hybrid attention with 8 full-attention layers. DeepSeek-R1-14B is based on Qwen2 with 40 layers. Mimo-7B is a Qwen2-based model with 36 layers, GQA (32 query heads, 8 KV heads), 4096 hidden size, and SiLU activation, trained with RL. All results are test on 3 seeds (11, 22, 33) and compute the average values, with all decoder layers applied scalar-modulated factor for Enlightenment.

In addition, model compression is increasingly useful in the era of LLMs. In the field of model compression, the performance degradation needs to be controlled carefully. Therefore, we also evaluate if Enlightenment works for the compressed LLM. Now, in industry, the mainstream model compression is based on quantization. We evaluate the W4A4C2 quantization on the DeepSeek-R1-Distill-Qwen-1.5B checkpoint. Specifically, the base model is a full-precision FP16 baseline, while the W4A4C2 quantization model uses W4 weight dequantization (4-bit restored), A4 activation fake-quantization via LobCQ (4-bit, group size 128, outlier ratio 0.01), and C2 KV-cache vector quantization (2-bit K/V, group size 32, residual length 32).

Datasets. Our evaluation spans several standard benchmarks. For classification, we include SST-2 (Stanford Sentiment Treebank; sentiment analysis; 2 classes) [60], CB (CommitmentBank; natural language inference; 3 classes) [61], and BoolQ (Boolean Questions; yes/no QA; 2 classes) [62]. For multiple-choice and multi-class tasks, we use ARCE (AI2 Reasoning Challenge Easy; multiple-choice science QA; 4 classes) [63], ARCC (AI2 Reasoning Challenge Challenge; multiple-choice science QA; 4 classes) [63], CQA (CommonsenseQA; multiple-choice commonsense QA; 5 classes) [64], and PIQA (Physical Interaction QA; multiple-choice physical commonsense QA; 2 classes) [65]. Across all datasets, we adopt perplexity-based scoring over label templates. For compressed models, all results use the vendored lm-evaluation-harness ¹. Tasks are evaluated in a zero-shot, log-likelihood-based multiple-choice setting with batch size 8, max length 2048, and no bootstrapping.

Baselines. We compare Enlightenment against ZeroTuning (ZT) [10]. This method is the most recent SOTA training-free tuning method for LLMs, serving as an updated version of earlier methods [46]–[48]. We therefore adopt it as the sole baseline for comparison. For Enlightenment, we apply attention head mixing with uniform per-head rates. All methods include no training and are evaluated at inference time. Specifically, for the W4A4C2 quantization model, we apply the Enlightenment attention head mixing to all layers and all heads at a base rate of \(\alpha=0.1, \beta=0.5\).

4.2 Main Results↩︎

Table 3 compares the accuracy of three LLMs across seven benchmarks. Overall, Enlightenment achieves the best average performance for all three models, improving Qwen3.5-9B from 80.4% to 81.8%, DeepSeek-R1-14B from 76.0% to 77.9%, and Mimo-7B-RL from 82.3% to 83.6%. Compared with the base setting, Enlightenment yields average gains of +1.5, +1.9, and +1.3 points, respectively, while ZT provides smaller or negligible improvements. At the dataset level, Enlightenment obtains the highest accuracy in most cases, particularly showing clear gains on SST-2, BoolQ, and ARCC. Although ZT slightly outperforms Enlightenment on some tasks for DeepSeek-R1-14B, such as CB, CQA, and PIQA, overall, Enlightenment provides a more consistent and stronger improvement across models and datasets.

Table 4 demonstrates that Enlightenment consistently improves the performance of the quantized LLM across all evaluated tasks. Under the aggressive W4A4C2 quantization setting, the average score drops substantially from 0.580 in FP16 to 0.396, indicating a clear degradation caused by low-bit quantization. Applying Enlightenment after quantization raises the average score to 0.441, yielding a +0.045 absolute improvement over W4A4C2. The gain is especially pronounced on BoolQ, where performance increases from 0.378 to 0.567, recovering a large portion of the accuracy lost during quantization. Improvements are also observed on RTE, WinoGrande, ARCE, PIQA, and MMLU, showing that the benefit is not limited to a single benchmark. These results suggest that Enlightenment can effectively mitigate quantization-induced degradation and enhance the robustness of low-bit LLM inference without restoring the model to full precision.

We also provide qualitative examples comparing baseline and Enlightenment on the SST-2 sentiment classification task using Qwen3.5-9B-Base. The model is prompted with

Sentence: ...

Answer: ...

While individual greedy-decoded outputs are often identical, the improvement from Enlightenment operates on the log-probability margin between the correct and incorrect label. Table 5 shows four examples from the SST-2 validation set where Enlightenment increases the log-probability margin for the correct label. The “Margin” column shows the difference \(\log P(\text{correct}) - \log P(\text{wrong})\); larger is better.

In each case, the Enlightenment suppresses the log-probab- ility of the wrong label (more negative) while marginally increasing or maintaining the correct label probability, widening the margin by \(+0.19\) to \(+0.31\) log-probability units.

Moreover, Table 6 shows a CommonsenseQA example. CommonsenseQA is a 5-way multiple-choice task. The evaluation computes \(\log P(\text{answer letter})\) for each of the five choices and selects the highest. Under the baseline, the model assigns nearly identical log-probabilities to choices A and B (both \(-5.97\)), and the tie is resolved in favor of the wrong answer. Enlightenment shifts the distribution: choice B’s score rises relative to choice A (\(A\) from \(-5.97\) to \(-6.06\), \(B\) from \(-5.97\) to \(-6.00\)), producing a clear margin of \(0.06\) nats in favor of the correct answer. The remaining choices (C, D, E) are unaffected because the BOS scaling targets only the top-ranking candidates.

4.3 Hyperparameter Sensitivity↩︎

We evaluate the sensitivity of hyperparameters in the Enligh- tenment by varying each hyperparameter in isolation while keeping all others at their defaults. The evaluation is based on the BoolQ task (100 samples, seed 11). Table 7 reports accuracy across two hyperparameters, polynomial degree (\(d\)) and mixing strength (\(\alpha\)). As the task-dependent factor \(\beta\) can be used to finetune the model in different tasks, we set it to 1 in this experiment.

The hyperparameter sensitivity results on BoolQ show that the method is generally robust to changes in both the polynomial degree \(d\) and the mixing coefficient \(\alpha\). Using the default setting \(d=3\) achieves the best accuracy of 94.0%, while lower degrees lead to slight drops, with \(d=1\) reaching 92.0% and \(d=2\) reaching 93.0%. This suggests that a moderate polynomial degree is sufficient to capture the desired effect, and increasing it further does not provide additional gains. For the mixing coefficient, the default value \(\alpha=0.1\) also achieves 94.0% accuracy, and performance remains unchanged across different settings. In contrast, disabling mixing by setting \(alpha=1\) reduces accuracy to 91.0%, indicating that the mixing mechanism is important for maintaining strong performance.

4.4 Ablation Study↩︎

Adaptive hyperparameter modulation. We use Qwen3.5-9B to conduct the ablation study. Table 8 evaluates the individual and combined impact of our proposed mechanisms by isolating the head mixing component (HeadMix only) and applying adaptive hyperparameter modulation (Adap.+ HeadMix). First, applying HeadMix yields highly inconsistent results across tasks, leading to a minor average regression to 80.3% compared to the Base model’s 80.4%. This instability indicates that while head mixing can capture specialized attention patterns, it introduces optimization variability. In contrast, the joint configuration (Adap.+HeadMix) successfully mitigates these vulnerabilities, securing the highest performance on six out of seven datasets and achieving a superior overall average accuracy of 81.8%. The adaptive hyperparameter modulation framework acts as a vital stabilizer, reversing the isolated regressions of HeadMix on SST-2 and CQA to produce net gains over Base (+1.2% and +0.8%, respectively). This strong synergy is particularly pronounced on reasoning-heavy datasets like CB (+2.8%) and PIQA (+2.1%).

Different layer and head ranges. We conduct experiments to analyze the impact of applying Enlightenment in different layer and head ranges. The results are presented in Table 9. For target layers \(\mathcal{L}\), it indicates that our method is highly robust to the choice of target layers, since all tested layer configurations, ranging from full-layer intervention to sparse subsets, consistently achieve \(94.0\%\) accuracy. This suggests that a small number of selected layers is already sufficient to capture the relevant effect. For target heads \(\mathcal{H}\), the trend is similarly stable: most partial head selections yield \(94.0\%\), while using all heads provides a slightly higher accuracy of \(95.0\%\). Overall, these findings show that the approach is not sensitive to precise layer/head selection.

5 Experiments on VLMs↩︎

5.1 Experimental Setup↩︎

Models: We evaluate 3 VLMs: MiMo-VL-7B [59], Qwen2-VL-7B [66], Qwen3-VL-8B [67] that were widely used in recent two years. All models are loaded at BF16 precision with eager attention. The experiments are conducted on NVIDIA RTX PRO 6000 GPUs (96 GiB VRAM each). For each evaluation task, a fresh copy of the model is loaded, and the compared method is applied before running the benchmark.

Benchmarks: We use 5 benchmarks that cover the mainstream multimodal tasks. MMBench [68] (4,876 natural-image MCQ), MME [69] (2,374 Yes/No perception), AI2D [70] (3,088 diagram MCQ), RealworldQA [71] (765 spatial reasoning MCQ), MMMU (541 Architecture & Engineering MCQ via HuggingFace datasets) [72].

Methods. Similar to LLM experiments, we compare our method with the ZeroTuning (ZT) [10] that uses the original uniform BOS scaling. For Enlightenment, we apply a single shared scalar modulator \(\gamma=0.5\) to all residual connections (\(x + \gamma \cdot F\)), patching both attention and MLP sublayers on every decoder layer.

Inference details. All benchmarks use greedy decoding. For standard models, we set Max New Tokens=10 for MCQ (MMBench, AI2D, MMMU) and Yes/No (MME), \(20\) for free-form RealworldQA. For MiMo-VL-7B—a reasoning model that generates chain-of-thought via <think> tags, we set Max New Tokens=256 for MCQ and \(64\) for free-form to accommodate deliberation. Answer extraction for MCQ checks the first option letter appearing in the response for Qwen models or the last option letter after </think> for the MiMo model. MME matches response substrings against “yes”/“no” ground truth.

5.2 Main Results↩︎

Table 10 compares the accuracy of three settings, Base, ZT, and Enlightenment, across five multimodal benchmarks on MiMo-VL-7B, Qwen2-VL-7B, and Qwen3-VL-8B. Overall, Enlightenment consistently achieves the strongest performance, yielding the highest average accuracy across all three models. Compared with the Base, Enlightenment improves average accuracy by +2.24% for MiMo-VL-7B, +1.92% for Qwen2-VL-7B, and +3.06% for Qwen3-VL-8B. These gains are particularly pronounced on challenging benchmarks, such as MME for MiMo-VL-7B, where accuracy increases over 8%, and AI2D for Qwen3-VL-8B, where performance rises substantially from 33.55% to 45.53%. In contrast, ZT provides only marginal improvement. Although Enlightenment is not the top performer in every individual case, it achieves the best or tied-best results on most datasets and consistently improves the overall average. These results demonstrate that Enlightenment is a more effective and robust post-tuning strategy than ZT, delivering clear and stable gains across different VLM architectures and benchmark types.

Furthermore, we conduct per-category analysis. The primary evaluation results demonstrate that Enlightenment achieves competitive performance on the MMBench dataset when integrated with the Qwen2-VL-7B model. Since MMBench is a comprehensive benchmark, we conduct a per-category analysis to characterize the specific strengths and limitations of this method. Table 11 breaks down the MMBench performance gains by L3 category (spanning 20 categories across the full 1,292-sample set). Notably, Enlight-enment yields the most substantial improvements on perception and recognition tasks: image emotion (\(+16.67\%\)), attribute recognition (\(+6.67\%\)), and image style (\(+6.45\%\)). At the L2 aggregation level, “coarse perception” improves by \(+6.08\%\), while “fine-grained perception (instance-level)” increases by \(+4.50\%\). Conversely, performance declines in relational reasoning categories: nature_relation (\(-11.29\%\)), future_prediction (\(-4.00\%\)), and the L2 aggregate “relation reasoning” drops by \(1.15\%\). This category-level performance profile explains different benchmark-level generalization abilities of Enlightenment: it benefits MMBench because the dataset is dominated by perception tasks, whereas showing a slight degrades performance on RealworldQA due to its inherent focus on relational reasoning.

We also provide qualitative examples comparing Baseline and Enlightenment on the MMBench dataset. The model is Qwen2-VL-7B-Instruct (Qwen2VL architecture, 28 decoder layers, hidden dim 3584, 8.29B parameters), and loaded at BF16 precision, greedy decoding, 10 generated tokens.

Question: Two magnets are placed as shown in the Figure 5 (both sets of arrows point in the same direction inside each magnet, indicating the same pole orientation). Will these magnets attract or repel each other?

Options:

• A. Repel.

• B. Attract.

Correct answer: A (Repel.)

The model output is presented in Table 12. The baseline model incorrectly predicts “Attract” despite the poles being aligned in the same direction. Enlightenment’s residual damping suppresses the over-active sublayer outputs that contribute to this visual reasoning error, allowing the model to correctly identify the physical relationship. This example is representative of the pattern we observe across the MMBench dataset: Enlightenment improves perceptual and physical reasoning accuracy while preserving the model’s overall knowledge.

5.3 Sensitivity Analysis↩︎

Hyperparameters. Enlightenment introduces only a single hyperparameter \(\eta\). To evaluate its impact, we conduct a sensitivity analysis of \(\eta\) on MMBench-300, as summarized in Figure 6. Most models exhibit stable performance across different values of \(\eta\). Qwen2-VL-7B is the sole exception, displaying a clear inverted-U pattern that peaks at \(\eta=0.5\) with a 6% performance gain. In contrast, the performance of Qwen2-VL-2B, Qwen3-VL-8B, and MiMo-VL-7B are subjected to slight changes across the interval \(\eta \in [0.4, 0.8]\). This highlights that Enlightenment is highly robust to hyperparameter selection, making it an ideal plug-and-play, zero-tuning approach to boost the performance of base VLMs.

Different layer ranges. We conduct experiments to analyze the impact of applying Enlightenment in different layer ranges. These results are summarized in Table 13. First, the MMBench results show that gating any single layer range in isolation degrades performance; early layers are particularly sensitive (-12%), while late layers also experience a decline (-7%). Conversely, results on the AI2D dataset reveal a different pattern: gating only the middle layers (L10–19) improves accuracy by 4.9%, suggesting that visual diagrammatic reasoning benefits from mid-layer residual damping even in isolation. On the other hand, late-layer gating on AI2D causes the most severe performance drop among all configurations, indicating that diagram comprehension relies heavily on late-layer representations. Uniformly gating all layers yields an improvement of 1.49% on AI2D, consistent with the full-benchmark evaluation. Overall, an all-layer application is necessary to achieve general improvements across different benchmarks, and targeted layer selection offers a promising avenue for task-specific fine-tuning to achieve optimal performance.

5.4 Ablation Study↩︎

Our method uses Enlightenment on both attention and FFN layers. We conduct experiments to show their individual contributions.

Table 14 reveals that neither attention-only nor FFN-only gating works in isolation. Scaling only attention degrades MMBench by approximately -18%, and scaling only FFN degrades MMBench by about -25%. Only when both sublayer types are suppressed together does the accuracy improves on MMBench. This demonstrates that applying Enlightenment to attention and FFN residuals forms a coupled system: the scalar modulation’s effectiveness derives from uniform suppression of both sublayer types simultaneously.

6 Discussion↩︎

6.1 Swapping Enlightenment Strategies Between VLM and LLM↩︎

In this section, we examine why distinct Enlightenment strategies are needed for VLMs and LLMs. We evaluate both HeadMix and scalar-modulated shortcuts on two baseline models: Qwen2-VL-7B (a VLM) and Mimo-7B-RL (an LLM). The results are reported in Table 15.

For the VLM, HeadMix degrades performance, lowering MMBench accuracy from 79.3% to 76.5%. In contrast, the scalar-modulated residual achieves 85.3%, a gain of 6.0 percentage points over the baseline. For the LLM, the pattern reverses: scalar-modulated residuals cause accuracy to drop to 89.5%, indicating that this mechanism, despite its effectiveness in multimodal contexts, does not benefit text-only reasoning. HeadMix raises accuracy to 91.7%, confirming its alignment with the structural characteristics of LLMs.

Taken together, these findings reveal a clear cross-architec-ture asymmetry: the two Enlightenment strategies exhibit pronounced model-type specificity. HeadMix introduces new scaling and additive computation within the attention module, which proves beneficial for LLMs yet disrupts the inherently more complex multimodal attention patterns that jointly process text and visual tokens, thereby degrading VLM performance. Conversely, scalar-modulated residuals regulate feature flow through a lightweight gating mechanism that selectively attenuates extraneous signals entering the transformer decoder, reducing cross-modal noise and yielding stable, substantial gains for VLMs. This reciprocal behavior underscores the necessity of tailoring Enlighten- ment strategies to each model family.

6.2 Output Distributions↩︎

We use the Qwen3.5-9B-Base model with the prompt “Classify the sentence into one of the following categories: positive or negative. Sentence: A breathtaking masterpiece of modern cinema. Answer: ...”. A forward hook was registered on the output self-attention module of the 15\(^{th}\) layer (one of the target attention layers) to capture the post-softmax attention weight tensor of the shape \([1, 16, 28, 28]\) (batch, head groups, query length, key length). 16 head groups correspond to the KV-head dimension after GQA expansion in Qwen3.5 (32 query heads, 8 KV heads, with the query heads grouped into 16 tracked slices by the patched attention function). The “BOS attention” for each head group is computed as \(\frac{1}{Q} \sum_{q=0}^{Q-1} \mathbf{A}_{h,q,0}\), i.e., the mean attention weight that each query position assigns to the BOS key position (\(k=0\)), averaged over all \(Q=28\) query positions.

Specifically, we note that generic scaling rate assigns \(0.68, 0.68, 0.66, 0.86, 0.51\) to the 3\(^{rd}\), 7\(^{th}\), 15\(^{th}\), 23\(^{th}\), and 31\(^{st}\) layers. All five rates are below \(1.0\), meaning that suppressing the initial attention at target layers yields better accuracy on tasks. This is consistent with the “attention sink” phenomenon, where pretrained transformers allocate excessive attention to the initial token, and reducing this baseline improves task-specific focus.

Moreover, Figure 7 illustrates the effect of Enlighten- ment on BOS attention across heads in a targeted layer. Unlike previous methods that globally reduce initial attention values, Enlightenment selectively decreases BOS attention only on heads 0 and 8, while other heads remain slightly decreased or nearly unchanged. This phenomenon indicates that the injection of new shortcuts operates through a mechanism distinct from attention-based modification. Our method can dynamically control attention across heads rather than globally suppressing BOS attention.

6.3 Activation Analysis↩︎

To directly verify the over-active sublayer hypothesis in VLM, we compare the hidden state norms of all 28 decoder layers on the same input. First, Figure 8 present the L2 Norm of decoder layers and actication reduction by Enlightenment. We have the following findings. (1) Baseline activations explode in late layers. The L2 norm grows from 47 at layer 0 to 371 at layer 27, which is a nearly 8\(\times\) increase. This suggests that sublayer outputs in deep layers accumulate redundant information rather than refining the residual stream. (2) Enlightenment suppresses late layers proportionally more. Layers 0–2 see only \(5\)–\(10\%\) reduction, while layers 20–27 see \(40\)–\(47\%\) reduction, consistent with the baseline trend that over-activation concentrates at depth.

Moreover, we present the VLM decoder activation outputs correspondence to token position in Figure 9. The results confirm the “attention sink” phenomenon in the baseline model, where the initial token dominates the attention values. In contrast, our scalar-modulated method successfully suppresses this effect, substantially reducing the initial token’s outsized activation. Finally, the activation profile directly supports the mechanism: pretrained sublayer outputs become increasingly noisy at depth, and a single \(\eta\) suppresses that excess while preserving the residual path.

7 Conclusion↩︎

In this paper, we have introduced Enlightenment, a novel, training-free finetuning paradigm designed to facilitate zero-shot self-improvement in large-scale models. Drawing inspiration from the neurological phenomenon of enlightenment during resting states, our method modifies shortcuts within pre-trained transformers without requiring any data, gradient computation, or weight update. Enlightenment effectively regularizes over-active sublayer outputs and optimizes token-level attention allocation. Extensive empirical evaluations across multiple VLMs and LLMs demonstrate that Enlightenment consistently elevates baseline performance across a wide array of multimodal, perception, and reasoning benchmarks. Moreover, the method exhibits a markedly low sensitivity for hyperparameters. Our work confirms the promising trajectory of “NeuroAI” principles. Future work will be digging deeper into the brain’s mechanisms to boost the self-improvement of large models.

References↩︎

---

1. https://github.com/EleutherAI/lm-evaluation-harness↩︎