1 Introduction↩︎

Diffusion models [1], [2] have become a dominant paradigm for high-quality visual generation. With the adoption of Transformer denoisers [3], [4], Diffusion Transformers (DiTs) now underpin many strong text-to-image and text-to-video systems, including SD3, Flux, and recent video DiTs [5]–[9]. At the same time, pretrained diffusion models are increasingly reused as visual feature extractors for dense correspondence and perceptual tasks [10]–[13]. These two trends place DiTs in a dual role: they are powerful generators, and they also contain representations that may support visual understanding. However, the internal activation components that determine these two capabilities remain insufficiently understood.

A particularly important but underexplored component is Massive Activations (MAs), i.e., hidden activations whose magnitudes are much larger than the remaining activations. MAs have been studied in large language models, where they are related to contextual knowledge and long-context modeling [14], [15], and in Vision Transformers, where similar activation patterns are associated with background or register-like tokens and global semantic processing [16]. DiTs differ from both settings. They perform iterative denoising over a spatial grid, where every image token participates in reconstructing local content and the hidden state evolves with the denoising timestep. Therefore, MAs in DiTs cannot be fully understood as token-localized language outliers or recognition-specific global tokens. Their role must be examined in the context of dense visual generation.

In this work, we systematically analyze MAs in representative DiTs. We find that MAs exhibit a DiT-specific structure: they are distributed across spatial tokens, yet concentrated in a small set of fixed feature dimensions (2). This spatially distributed and channel-concentrated pattern suggests that MAs act as shared high-magnitude channels for dense visual computation, rather than as isolated outliers at a few special positions. We further show that the MA dimensions are closely aligned with the residual scaling factors produced by Adaptive Layer Normalization (AdaLN) (4). Their magnitudes are primarily shaped by the denoising timestep, while text conditions have much weaker influence. These observations indicate that MAs are structured, AdaLN-related, and stage-dependent activation components inside pretrained DiTs.

This structure leads to different consequences for generation and understanding. For visual generation, activation intervention shows that disrupting MA dimensions substantially weakens fine-grained local details, such as textures and subtle object parts, while largely preserving object identity, layout, and global semantic content. In contrast, perturbing the same number of non-MA dimensions has much weaker visual effect. This indicates that MAs are not the primary carrier of prompt semantics; instead, they are closely tied to local detail synthesis. Thus, suppressing MAs gives a natural way to construct a detail-deficient counterfactual branch whose residual from the original prediction isolates a detail-specific guidance direction.

For visual understanding, the same MA structure creates a different problem. Dense matching and perception require spatial descriptors whose directions reflect local semantic and geometric differences. However, since MAs occupy fixed high-magnitude dimensions across many spatial tokens, raw DiT features become biased toward a shared dominant direction, making local descriptors overly similar under cosine-based comparison. This weakens spatial discrimination even though the underlying DiT is a strong generative backbone. Importantly, MA dimensions should not simply be discarded: their spatial maps still exhibit meaningful response patterns over image tokens. A useful representation should therefore reduce the directional dominance of MAs while preserving their spatial structure.

Motivated by these findings, we introduce Eliciting Massive Activation (EMA), a unified training-free framework that modulates MAs to improve both generation and understanding in DiTs, as shown in 1. For generation, EMA introduces MA-driven Detail Guidance (DG), which enhances fine-grained rendering by contrasting the original prediction with a detail-deficient counterfactual branch. Specifically, DG suppresses MA dimensions at an intermediate DiT block to obtain the detail-deficient prediction, and then guides the original prediction away from this branch toward more detail-rich outputs. This hidden-state intervention naturally supports efficient partial-forward inference: the original and counterfactual branches share the computation before the intervention block, so only the remaining blocks need to be recomputed after MA suppression. Moreover, applying MA suppression only to selected spatial tokens yields Local DG, enabling targeted refinement of regions with weak details. DG is complementary to Classifier-Free Guidance (CFG) [17]: while CFG primarily strengthens semantic alignment with conditioning signals, DG explicitly improves fine-grained visual detail quality. For understanding, EMA introduces MA-modulated REPresentation extraction (MREP). MREP uses pretrained AdaLN channel-wise modulation to reduce MA directional dominance, and concatenates spatially normalized MA maps to retain their useful spatial cues. In this way, EMA treats MAs in a task-aware manner: their suppression defines detail guidance for generation, while their modulated and spatially normalized form improves dense representations. Our main contributions are summarized as follows:

• We provide a systematic study of MAs in DiTs and show that they are spatially distributed, channel-concentrated, AdaLN-related, and primarily timestep-modulated activation components.

• We reveal the dual role of MAs: they support fine-grained local detail synthesis during generation, but their shared high-magnitude directions impair the discriminability of raw DiT features for dense understanding.

• We introduce EMA, a training-free MA modulation framework with two task-specific modules: DG for detail-faithful image and video generation, and MREP for semantically discriminative DiT representation extraction.

• We demonstrate that EMA improves image generation, video generation, local detail refinement, and dense visual understanding, establishing MA modulation as a unified tool for eliciting stronger DiT capabilities.

2 Related Work↩︎

2.1 Diffusion Transformers for Visual Generation↩︎

Diffusion models [1], [2], [18], [19] have become a dominant paradigm for high-quality visual synthesis. In particular, latent diffusion models [1] improve computational efficiency by performing denoising in a compressed latent space, while large-scale text-to-image systems further demonstrate the importance of scaling model capacity, data, and conditioning signals [2]. Recently, Transformer architectures [3], [20] have been increasingly adopted as denoising backbones. DiT [4] introduces a pure Transformer formulation for diffusion models, and a series of DiT-based systems, including PixArt-\(\alpha\) [21], SD3 [5], and Flux [6], have demonstrated strong performance in text-to-image generation. Beyond image synthesis, Transformer-based diffusion models have also become central to text-to-video generation, such as CogVideo [8], CogVideoX [7], and Wan [9]. Despite their strong empirical success, most prior work primarily focuses on architecture design, training strategies, and scaling laws. In contrast, our work shifts the focus to the internal computations of DiT hidden states, aiming to better understand their role in enabling both visual generation and representation learning.

2.2 Sampling Guidance for Diffusion Models↩︎

Sampling guidance provides a practical mechanism for steering pretrained diffusion models without retraining the generator. Among existing methods, classifier-free guidance (CFG) [17] has become the de facto standard. It linearly extrapolates between unconditional and conditional predictions, thereby strengthening conditioning signals and improving prompt or class alignment. However, large guidance scales can lead to over-saturation and visual artifacts [22], which motivates improved variants such as CFG++ [23]. Beyond CFG-style interpolation, another line of work constructs guidance signals from internal model behavior. Auto-guidance [24] steers sampling using a deliberately degraded version of the same model, while self-attention guidance [25], perturbed-attention guidance [26], and spatiotemporal skip guidance [27] modify attention maps or intermediate activations to obtain improved sampling directions. Our method follows the general philosophy of self-guidance, but differs in both motivation and construction.

2.3 Diffusion Model Representations↩︎

Beyond image synthesis, pretrained diffusion models have demonstrated strong potential as visual representation learners. During large-scale generative training, these models implicitly acquire spatial structure and compositional semantics, making their intermediate features useful for a range of downstream perception tasks [28]–[31]. In visual correspondence, a line of work explores feature extraction from pretrained Stable Diffusion (SD) models for semantic or geometric matching across images [10]–[13], [32]. These approaches are often evaluated in comparison or combination with discriminative representations such as DINO and DINOv2 [33], [34], while recent studies further enhance geometric consistency via adaptive pose alignment [35]. More recently, several works have begun to repurpose Diffusion Transformers (DiTs) as general-purpose feature extractors for correspondence and perception tasks [36], [37], suggesting that stronger generative backbones may also yield more expressive visual representations. However, despite these encouraging results, the internal representational properties of DiTs remain significantly less understood.

2.4 Massive Activations and Internal Outliers↩︎

Massive activations (MAs) refer to unusually large hidden activations that concentrate in a small subset of feature dimensions or tokens. They were first systematically studied in large language models, where they often emerge at fixed dimensions of low-information tokens and have been linked to contextual knowledge modeling and long-context capabilities [14], [15], [38], [39]. Subsequent studies further associate such activation concentration with positional mechanisms such as rotary position embeddings [40]. Similar phenomena have also been observed in Vision Transformers [14], [16], [41], where high-magnitude activations are frequently associated with background or register-like tokens that help organize global semantic structure. Beyond discriminative models, massive or outlier activations have also been reported in Diffusion Transformers (DiTs), particularly in the context of quantization, acceleration, and feature extraction [36], [42]–[44]. Notably, DiTF [36] observes that DiTs exhibit fixed-dimensional massive activations across spatial tokens when used as feature extractors, while DG [45] further suggests that such activations are closely related to local detail synthesis in DiT-based generation. However, despite these empirical observations, the role of MAs during the denoising process and representation formation in DiTs remains poorly understood.

3 Preliminaries↩︎

3.1 Diffusion Transformer Architecture↩︎

Following DiT [4], we consider a latent diffusion model composed of a variational autoencoder (VAE) and a denoising Transformer. The VAE encoder \(\mathcal{E}\) maps an image \(x\) into a latent variable \(z_0=\mathcal{E}(x)\), and the VAE decoder maps the denoised latent back to pixel space. The Transformer denoiser \(D_\theta\) operates on a noised latent \(z_t \in \mathbb{R}^{C \times H \times W}\) at timestep \(t\) under condition \(c\). Let \(D_\theta=\{D_k\}_{k=1}^{N}\) contain \(N\) Transformer blocks, and denote the hidden state after the \(k\)-th block as \(z_t^k\). Each block updates the hidden state through a residual connection [46]: \[z_t^k=z_t^{k-1}+\alpha_{k-1} D_{k-1}(z_t^{k-1},t,c), \label{block}\tag{1}\] where \(\alpha_k \in \mathbb{R}^{C}\) is a dimension-wise residual scaling factor produced by the adaptive layer normalization (AdaLN) module [47]. The \(\operatorname{AdaLN}\) layer encodes the input timestep \(t\) and the additional conditioning information \(c\) (e.g., class or text embedding) into channel-wise scale and shift parameters \(\gamma_k\) and \(\beta_k\). It then performs Adaptive Layer Normalization (AdaLN) on the hidden state \(z_t^k\): \[\hat{z}_t^k = \bigl(1+\gamma_k\bigr)\,\operatorname{LayerNorm}(z_t^k) + \beta_k, \label{AdaLN}\tag{2}\] where \(\gamma_k, \beta_k\) are regressed by the MLP networks of AdaLN layer: \[\begin{gather} \gamma_k, \beta_k, \alpha_k = \operatorname{MLP}_k(t, c) \label{regress} \end{gather}\tag{3}\] where \(\alpha_k\) scales the \(k\)-th residual connection.

3.2 Diffusion Transformer Representation↩︎

Besides generating samples, a pretrained DiT can be used as a feature extractor by forwarding a real image through the denoising Transformer and collecting intermediate hidden states. Given an image \(x\), we first obtain its latent \(z_0=\mathcal{E}(x)\) and add noise at a selected timestep \(t\) to produce \(z_t=z_0+\sigma(t)\epsilon\). We then feed \(z_t\) into the pretrained DiT under a fixed condition \(c\) and extract the visual representation \(F_{k,t}(x)\) from a chosen block \(k\): \[z_t^k=z_t^{k-1}+\alpha_{k-1} D_{k-1}(z_t^{k-1},t,c),\quad F_{k,t}(x)=z_t^k . \label{eq:dit95feature95extract}\tag{4}\] The resulting feature map \(F_{k,t}(x)\in\mathbb{R}^{C\times H\times W}\) preserves the spatial token layout of the latent representation and can be used as a dense visual representation. The choice of timestep \(t\), block index \(k\), and feature normalization determines what visual information is exposed by the representation.

3.3 Diffusion Sampling and Classifier-Free Guidance↩︎

Diffusion models [18], [48] generate samples by progressively denoising an initial Gaussian variable \(z_T \sim \mathcal{N}(0,I)\). Given a clean sample \(x \sim p_{\mathrm{data}}(x)\), the forward corruption process can be written as \(z_t=x+\sigma(t)\epsilon\), where \(\sigma(t)\) is the noise schedule and \(\epsilon \sim \mathcal{N}(0,I)\). A denoising network \(D_\theta(z_t,t,c)\) is trained to predict the denoising direction conditioned on timestep \(t\) and condition \(c\), such as a class label or text prompt. Classifier-Free Guidance (CFG) [17] improves conditional generation by extrapolating from the unconditional prediction to the conditional prediction: \[\hat{D}_{\theta}\left(z_t, c\right)=D_{\theta}\left(z_t,c\right)+w\left(D_{\theta}\left(z_t, c\right)-D_{\theta}\left(z_t,\hat{c}\right)\right) \label{eq:cfg95prelim}\tag{5}\] where \(\hat{c}\) denotes the unconditional input and \(w\) is the CFG scale. The difference \(D_\theta(z_t,c)-D_\theta(z_t,\hat{c})\) captures the effect of the external condition and therefore primarily strengthens prompt or class alignment. However, semantic alignment and local detail fidelity are not identical objectives; strong semantic guidance can still leave textures and small object parts under-synthesized.

4 Massive Activations in DiTs↩︎

To systematically investigate the internal behavior of Diffusion Transformers (DiTs), we analyze the hidden states of several representative models, including SD3 and Flux. As shown in 2, we consistently observe the presence of Massive Activations (MAs): a small subset of feature dimensions exhibits activation magnitudes that are substantially larger than those of the remaining dimensions across different models. This phenomenon suggests that DiT representations are highly non-uniform and structured in magnitude distribution, raising the question of what functional role these massive activations play in model behavior. In the following sections, we conduct a detailed analysis to understand (i) where and how these activations emerge, and (ii) how they influence both visual generation and representation learning in DiTs.

4.1 Spatial and Channel Properties of MAs↩︎

We first investigate where MAs appear in DiTs. As shown in 3, MAs appear throughout the internal layers of DiTs, emerge early during training, and persist across different model sizes. We observe the same layer-wise behavior in SD3, SD3.5, and Flux, which further confirms that MAs are not isolated artifacts of a single architecture, checkpoint, or layer. We then examine their token and channel distributions. Unlike MAs in LLMs or ViTs [14], [16], which often concentrate on special tokens or background/register-like tokens, MAs in DiTs appear across all spatial patch tokens. At the same time, they are concentrated in a few fixed feature dimensions, such as dimension 810 in SD3. This spatially distributed but channel-concentrated pattern is consistent with the dense generative objective of DiTs: every spatial token must participate in reconstructing local image content, while a small set of feature dimensions can act as shared high-magnitude channels for the internal computation of DiTs. This property makes MAs fundamentally different from token-localized outliers in LLMs or ViTs and suggests that they may act as shared activation components for dense visual processing.

4.2 MAs Are Modulated by AdaLN↩︎

To understand why MAs appear in fixed feature dimensions, we compare the hidden-state activation distribution with the AdaLN residual scaling factor \(\alpha_k\). Recall from 1 that the hidden state is updated as \[z_t^{k+1}=z_t^k+\alpha_k D_k(z_t^k,t,c), \quad \alpha_k=\operatorname{MLP}_k(t,c). \label{computation}\tag{6}\] As shown in 4 (a), peaks in \(\alpha_k\) align with the dimensions where hidden-state MAs concentrate. This indicates that the scaling factor \(\alpha_k\) governs the channel pattern of MAs. Since \(\alpha_k\) is regressed from timestep \(t\) and condition embeddings \(c\), we further analyze their respective impact.

As shown in 4 (b), changing text prompts has little influence on the magnitude of MAs; the activation values remain nearly constant across 1,000 prompts. In contrast, the timestep has a strong effect: MA magnitudes progressively increase as sampling moves from high-noise timesteps toward low-noise timesteps. Similar trends are observed for SD3.5, Flux, and DiT-XL. These results suggest that MAs are primarily stage-dependent rather than prompt-dependent. Since the timestep encodes the noise level and denoising stage, MAs can be understood as activation components modulated by the progression of the denoising trajectory rather than by the external semantic condition.

4.3 Generative Impact of MAs↩︎

The structural and modulation properties discussed above reveal where MAs emerge and how they are controlled, but do not directly explain their functional role in DiTs. To investigate their impact on visual generation, we perform activation intervention [14] by selectively disrupting MA dimensions in a single DiT layer and propagating the modified hidden states through the remaining blocks.

4.3.0.1 MAs are critical for visual detail synthesis.

As shown in 5, disrupting MAs leads to a pronounced degradation of fine-grained local details, including textures and subtle object parts such as eyes and hair. In contrast, perturbing the same number of randomly selected non-MA dimensions has negligible impact on the generated outputs. This comparison indicates that MAs play a critical role in synthesizing local visual details during the generative process of DiTs.

4.3.0.2 MAs have limited influence on semantic content.

Despite the degradation of local details, images generated with disrupted MAs largely preserve global semantics, including object identity, color composition, and overall layout, and remain semantically consistent with the original outputs (5). This observation is further supported by prompt-alignment metrics: compared with the original outputs, the MA-disrupted model achieves comparable BLIPScore and CLIPScore win probabilities of 0.462 and 0.512, respectively (6). These results suggest that MAs primarily affect fine-grained visual detail rather than overall semantic content, which is consistent with our finding in 4.2 that text embeddings have negligible influence on MAs.

4.4 Representational Impact of MAs↩︎

The same activation structure leads to a distinct effect when DiTs are repurposed as feature extractors. Dense visual understanding requires spatial tokens to preserve discriminative local information, enabling features from different object parts or background regions to be reliably distinguished. However, since MAs concentrate in a few fixed dimensions and appear across nearly all spatial tokens, they introduce a shared high-magnitude direction into token representations. This dominant common direction makes features from different locations overly similar in the latent space, particularly under cosine-based comparison, while diminishing the contribution of locally informative non-MA dimensions. Consequently, raw DiT hidden states can become spatially indiscriminative and less effective for dense perception tasks, as shown in 7.

5 Methodology↩︎

Based on the above analysis, MAs are not incidental numerical outliers, but structured and stage-dependent activation components that shape how DiTs process visual information. This section formally presents EMA, which leverages MAs to enhance both the generative and representational capabilities of DiTs, as shown in 8. The core idea is to modulate MAs within the internal computations of DiTs, enabling targeted control over both the sampling process and representation extraction. For generation, EMA introduces MA-driven Detail Guidance strategy (DG), which constructs a degraded, “detail-deficient” DiT by disrupting MAs and uses it to guide sampling toward detail-rich regions (see [sec:detail_guidance,sec:partial_dg,sec:local_detail_guidance,sec:cfg_dg]). For understanding, EMA introduces MA-modulated REPresentation extraction method (MREP), which adaptively suppresses MA-dominated channels using the pretrained AdaLN operator, thereby recovering more spatially discriminative DiT representations (see 5.5).

5.1 Detail Guidance↩︎

Motivated by the analysis in 4, we propose Detail Guidance (DG), a concise, training-free self-guidance strategy that leverages MAs to enhance fine-grained detail synthesis in DiTs. DG follows the general principle of self-guidance [24], where sampling is guided by contrasting the original model with a deliberately degraded counterpart. Unlike generic degradation, DG requires a counterfactual branch that specifically weakens local details while preserving the semantic content of the original conditional prediction. This design ensures that the resulting residual provides a detail-specific guidance direction rather than an entangled semantic-detail signal.

To construct this branch, we disrupt MA patterns in the DiT hidden states of \(z_t\). Formally, let \(D_\theta\) denote the original pretrained DiT model, and let \(z_t^k \in \mathbb{R}^{C \times H \times W}\) be the hidden state output of the \(k\)-th block. We obtain the modified hidden state \(\hat{z}_t^k\) by adaptively suppressing MA dimensions in \(z_t^k\) conditioned on timestep \(t\), and then propagate it through the remaining blocks to produce the detail-deficient prediction.

MA dimension selection. For each candidate block \(k\), we identify MA dimensions from activation statistics collected once on a small calibration set. Let \(a_{k,i}\) denote the mean absolute activation magnitude of channel \(i\) at block \(k\), averaged over calibration samples, timesteps, and spatial tokens. We define the median channel magnitude at this block as \[\mu_k=\operatorname{median}_{j}\;a_{k,j}. \label{eq:ma95median}\tag{7}\] A channel is selected as an MA dimension if its average magnitude exceeds the median magnitude by a ratio threshold \(\tau_{\mathrm{MA}}\): \[\mathcal{I}_{\mathrm{MA}}^k =\left\{i\mid a_{k,i}>\tau_{\mathrm{MA}}\mu_k\right\}. \label{eq:ma95selection}\tag{8}\] The selected MA dimensions are fixed after this calibration and reused for all prompts during sampling.

Adaptive MA Suppression. We suppress MAs adaptively along the denoising trajectory when constructing the detail-deficient branch. Since fine visual details are also primarily formed in the late denoising stage (see 4), the counterfactual branch should preserve MAs at early timesteps for global structure formation, but strongly attenuate them at late timesteps to remove local detail synthesis. Let \(k\) be the intervention block and \(\mathcal{I}_{\mathrm{MA}}^k\) be its selected MA dimensions. We define a timestep-dependent retention coefficient \(\rho_t\in[0,1]\) and apply it only to MA dimensions: \[\hat{z}_{t,i}^{k}= \begin{cases} \rho_t z_{t,i}^{k}, & i\in\mathcal{I}_{\mathrm{MA}}^k,\\ z_{t,i}^{k}, & i\notin\mathcal{I}_{\mathrm{MA}}^k. \end{cases} \label{eq:adaptive95ma95suppression}\tag{9}\] Here \(z_{t,i}^{k}\) denotes the activation values of channel \(i\), with the coefficient broadcast over spatial tokens. We use a sigmoid decay schedule over the normalized timestep \(t\in[0,1]\): \[\rho_t=\sigma\!\left(\lambda_{\rho}(t-\tau_{\rho})\right), \qquad \sigma(x)=\frac{1}{1+\exp(-x)}. \label{eq:rho95schedule}\tag{10}\] where \(\tau_{\rho}=0.5\) controls the transition timestep and \(\lambda_{\rho}=12\) controls the sharpness of the decay.

Detail Guidance formulation. After adaptive MA suppression, the modified hidden state \(\hat{z}_t^k\) is propagated through the remaining DiT blocks, yielding a detail-deficient prediction \(D_{\theta}(\hat{z}_t,c)\). The intervention block \(k\) is selected via grid search. Since \(D_{\theta}(\hat{z}_t,c)\) preserves the semantic content while weakening local details, the residual between the original and detail-deficient predictions provides a detail-specific guidance direction. DG is therefore defined as \[\hat{D}_{\theta}(z_t,c) = D_{\theta}(z_t,c) + w\left( D_{\theta}(z_t,c)-D_{\theta}(\hat{z}_t,c) \right), \label{eq:dg95from95branch}\tag{11}\] where \(w\) controls the strength of detail guidance.

5.2 Efficient Partial-Forward Detail Guidance↩︎

A naive implementation of DG would require two complete model evaluations: one for the original conditional prediction and another for the counterfactual detail-suppressed prediction. However, DG admits a more efficient partial-forward implementation. Since the counterfactual branch shares the same computation as the original branch before the intervention block \(k\), we only need to recompute the intervened hidden state and the remaining DiT blocks after MA suppression. As a result, the additional inference cost is approximately proportional to \((N-k)/N\) of a single DiT forward pass, rather than a full extra evaluation as in standard guidance schemes such as CFG. This design makes DG training-free, parameter-free, and computationally efficient.

5.3 Local Detail Guidance↩︎

The spatially distributed nature of MAs further enables token-level control. While DG suppresses selected MA dimensions over all spatial tokens, Local DG constructs the detail-deficient branch by suppressing MAs only at target tokens. This produces a local counterfactual prediction in which fine details are weakened inside the specified region, while the remaining tokens follow the original DiT computation.

Formally, let \(\mathcal{R}\subseteq\{1,\ldots,H\}\times\{1,\ldots,W\}\) denote the target token set, which can be obtained from a mask, a bounding box, or selected patches. For the intervention block \(k\), we apply the same MA dimensions \(\mathcal{I}_{\mathrm{MA}}^k\) and retention coefficient \(\rho_t\) as in DG, but only for tokens in \(\mathcal{R}\): \[\hat{z}_{t,i,u}^{k}= \begin{cases} \rho_t z_{t,i,u}^{k}, & i\in\mathcal{I}_{\mathrm{MA}}^k,\; u\in\mathcal{R},\\ z_{t,i,u}^{k}, & \text{otherwise}. \end{cases} \label{eq:local95ma95suppression}\tag{12}\] Here \(u\) indexes a spatial token. The modified hidden state is then propagated through the remaining DiT blocks to obtain the local detail-deficient prediction \(D_{\theta}(\hat{z}_t^{\mathcal{R}},c)\). The resulting Local DG prediction is \[\hat{D}_{\theta}^{\mathcal{R}}(z_t,c) = D_{\theta}(z_t,c) + w\left( D_{\theta}(z_t,c)-D_{\theta}(\hat{z}_t^{\mathcal{R}},c) \right), \label{eq:local95dg}\tag{13}\] where the residual mainly reflects the missing details of the selected tokens. Thus, Local DG keeps the training-free formulation of DG while turning MA suppression into a token-specific detail-control mechanism. In practice, this allows DG to serve as a local refinement operator: when a generated image contains regions with insufficient texture or weak fine details, Local DG can selectively enhance those regions while keeping the rest of the image largely unchanged.

5.4 Integration with CFG.↩︎

Classifier-free guidance (CFG) [17] is a standard technique that enhances semantic alignment by extrapolating between conditional and unconditional predictions. Our DG method is naturally complementary to CFG: whereas CFG strengthens semantic fidelity, DG explicitly enhances local detail quality (see ¿tbl:tab:discussion?). The combined guidance is expressed as \[\begin{align} \hat{D}_{\theta}(z_t, c) &= D_{\theta}(z_t, c) + \lambda \big( D_{\theta}(z_t, c) - D_{\theta}(z_t, \hat{c}) \big) \\ &\quad + w \big( D_{\theta}(z_t, c) - D_{\theta}(\hat{z}_t, c) \big) \end{align}\] where \(\lambda\) and \(w\) are the guidance scales of CFG and DG, respectively. \(\hat{c}\) denotes the unconditional prompt and \(\hat{z}_t\) denotes \(z_t\) with MAs-disrupted hidden state

5.5 MA-modulated Representation Extraction↩︎

Beyond generation, MAs also affect the quality of DiT representations. As shown in 4.4, raw hidden states \(z_t^k\) contain MA-dominated channels shared by many spatial tokens, which weakens token-level discrimination for dense understanding. Meanwhile, our analysis in 4.2 shows that these MA dimensions are closely aligned with the AdaLN residual scaling factor \(\alpha_k\). This suggests that the pretrained AdaLN operator can be reused as a channel-wise modulator to suppress MA dominance in features. Based on this observation, we introduce MA-modulated REPresentation extraction (MREP), a training-free method that first modulates DiT hidden states with AdaLN and then extracts dense visual representations.

Adaptive MA modulation with AdaLN. Given an image \(x\), we encode it as \(z_0=\mathcal{E}(x)\) and add noise at timestep \(t\) to obtain \(z_t\). We then forward \(z_t\) through the pretrained DiT and collect the hidden state \(z_t^k\) at block \(k\). Instead of directly using \(z_t^k\) as the visual feature, MREP applies the pretrained AdaLN channel-wise weights from the same block. Following [AdaLN,regress], the AdaLN parameters are \[\gamma_k,\beta_k,\alpha_k=\operatorname{MLP}_k(t,c), \label{eq:mrep95adaln95params}\tag{14}\] where \(\gamma_k,\beta_k,\alpha_k\in\mathbb{R}^{C}\) are channel-wise parameters. Since the MA dimensions are aligned with large responses in the AdaLN scaling factor \(\alpha_k\), these pretrained weights provide a built-in localization and modulation signal for MA-dominated channels. We obtain the AdaLN-modulated hidden state as \[\hat{z}_t^{k,\mathrm{Ada}} = \operatorname{AdaLN}(z_t^k;\gamma_k,\beta_k) = \left(1+\gamma_k\right)\operatorname{LayerNorm}(z_t^k)+\beta_k . \label{eq:mrep95adaln95modulation}\tag{15}\] This channel-wise modulation reduces the dominance of MA channels while preserving the spatial token structure needed for dense visual understanding. After AdaLN modulation, MREP extracts representations from the modulated hidden state rather than the raw DiT hidden state: \[\hat{F}_{k,t}^{\mathrm{Ada}}(x)= \operatorname{Norm}\!\left(\hat{z}_t^{k,\mathrm{Ada}}\right), \label{eq:mrep95feature}\tag{16}\] where \(\operatorname{Norm}(\cdot)\) denotes the feature spatial normalization. This AdaLN-modulated branch suppresses the shared MA-dominated direction and better preserves token-level semantic differences, making DiT features more suitable for dense visual understanding.

Preserving Spatial Structure in MA Dimensions. Although MA dimensions can dominate feature directions and weaken spatial discrimination, they are not purely detrimental to representation learning. As shown in 10, MA channels still exhibit structured responses across visual tokens, indicating that they encode useful spatial cues. Therefore, instead of discarding MA dimensions entirely, MREP explicitly preserves their spatial structure as complementary information.

Specifically, we collect the spatial maps corresponding to the calibrated MA dimensions: \[\mathcal{M}_{k,t}(x) = \left\{z_{t,i}^{k}\mid i\in\mathcal{I}_{\mathrm{MA}}^k\right\}. \label{eq:mrep95ma95maps}\tag{17}\] However, due to their large magnitudes, directly using these MA maps can again dominate feature similarity. We therefore normalize them over spatial tokens: \[\hat{\mathcal{M}}_{k,t}(x) = \operatorname{Norm}\!\left(\mathcal{M}_{k,t}(x)\right), \label{eq:mrep95spatial95norm}\tag{18}\] where \(\operatorname{Norm}(\cdot)\) denotes spatial normalization within each MA channel. This operation preserves the relative spatial response pattern of each MA channel while removing its excessive magnitude scale.

MREP Feature Integration. The final MREP representation combines the AdaLN-modulated representation with the normalized MA spatial maps: \[\hat{F}_{k,t}(x) = \operatorname{Concat}\!\left( \hat{F}_{k,t}^{\mathrm{Ada}}(x),\, \hat{\mathcal{M}}_{k,t}(x) \right). \label{eq:mrep95final95feature}\tag{19}\] This design separates and reuses the two effects of MAs in a controlled manner. AdaLN modulation reduces the undesired directional dominance of MA dimensions, while spatial normalization preserves their useful spatial response patterns. As shown in 7, by integrating these two complementary components, MREP produces DiT representations that are both less dominated by global MA directions and more informative for dense visual understanding.

6 Experiments↩︎

6.1 Experimental Setup↩︎

Model variants. We evaluate EMA across a broad range of image and video Diffusion Transformers. For text-to-image generation, we consider four representative large-scale DiT-based models: SD3-Medium [5] (SD3), SD3.5-Large [5] (SD3.5), Flux1 [6], and Flux2 [6]. Since Flux is released as a CFG-distilled model, we use its de-distilled variant when evaluating DG independently from CFG. To examine whether DG depends on a particular DiT implementation, we further test it on PixArt-\(\alpha\) [21], which uses shared AdaLN layers, and SANA [49], which adopts linear attention and convolutional local modeling. For text-to-video generation, we evaluate EMA on two video DiTs: Wan-1.3B, and Wan-14B [50]. In addition, we assess EMA on class-conditional generation with DiT-XL/2 [4] on ImageNet at \(256\times256\) resolution.

Visual generation evaluation. For text-to-image generation, we evaluate EMA on two standard prompt benchmarks: the Pick-a-Pic “test unique” split [51] and HPSv2.1 [52]. To quantify prompt alignment, we compute CLIPScore [53] using clip-vit-large-patch14 and BLIPScore [54] using blip-itm-large-coco. CLIPScore measures global text-image semantic consistency, while BLIPScore provides a complementary image-text matching signal. To assess perceptual quality and visual preference, we report HPSv2.1 [52] and LAION-Aesthetics [55]. For ImageNet class-conditional generation, we follow the standard DiT evaluation protocol and report FID, IS, Precision, and Recall. For text-to-video generation, we adopt the standard prompts and evaluation metrics from VBench [56].

Visual understanding evaluation. To assess the representation quality of EMA, we conduct experiments on three dense visual perception tasks: semantic correspondence, semantic segmentation, and depth estimation. For semantic correspondence, we evaluate on three widely used benchmarks: SPair-71k [57], PF-Pascal [58], and AP-10K [35]. AP-10K is a large-scale semantic correspondence benchmark built upon the AP-10K dataset [59], comprising 2.61 million training pairs and 36,000 test pairs across three settings: intra-species, cross-species, and cross-family matching. Following prior work [11], [12], we report the percentage of correct keypoints (PCK). For semantic segmentation, we evaluate on ADE20k [60] and report mean intersection over union (mIoU). For depth estimation, we evaluate on NYUv2 [61] and report standard metrics, including absolute relative error (Abs Rel), root mean squared error (RMSE), and threshold accuracy \(\delta_i\).

Computational resources. All quantitative experiments are conducted on a single NVIDIA L40S GPU with 48GB memory. To measure computational overhead, we generate 100 images at \(1024\times1024\) resolution and report the average latency and memory usage.

6.2 Main Results on Image Generation↩︎

6.2.1 Text-to-image generation↩︎

Evaluation of DG. ¿tbl:tab:dg95pick? summarizes the performance of Detail Guidance (DG) on three pretrained DiTs. DG consistently improves both prompt alignment and fine-grained detail quality across all models. For example, on SD3.5, DG increases the BlipScore from 70.09 to 83.66 while improving the Aesthetic score from 5.94 to 6.16. The qualitative comparisons in 11 further demonstrate that DG enhances local textures and structural details while preserving the overall image composition. Notably, the consistent improvements on Flux indicate that DG generalizes beyond the SD3 family and remains effective for stronger large-scale DiTs. Compared with CFG, DG achieves higher detail-quality metrics (e.g., Aesthetic 6.01 vs. on SD3), whereas CFG provides stronger prompt alignment. As illustrated in 11, DG produces richer local textures, while CFG better preserves semantic consistency with the text prompt. These observations suggest that the two guidance strategies improve complementary aspects of generation: DG primarily enhances fine-grained detail synthesis, whereas CFG strengthens semantic alignment.

Integrating DG with CFG. DG is fully compatible with CFG and consistently improves detail quality when combined with it, as shown in ¿tbl:tab:dg95pick?. The visual results in 12 further show that the combined guidance produces more refined local structures and higher overall image quality. This complementary effect is also reflected quantitatively: CFG mainly improves prompt alignment, whereas DG consistently boosts detail-quality metrics.

Comparison with state-of-the-art. We further compare DG with representative guidance methods, including APG [22], CFG++ [23], CFG-Zero [62], and \(S^2\)-Guidance [63]. As reported in ¿tbl:tab:guidance?, DG achieves the highest Aesthetic score among standalone guidance methods without requiring an unconditional branch. Compared with PAG, which constructs a degraded branch by perturbing attention maps, DG delivers superior detail-quality performance. Moreover, when combined with CFG, DG achieves the best average HPSv2.1 score, further validating our central hypothesis that DG provides a detail-oriented guidance direction that complements semantic guidance from CFG.

6.2.2 Class-to-image generation↩︎

We further evaluate whether DG transfers from text-conditioned generation to class-to-image generation. As reported in 1, DG consistently improves the main sample quality metrics under both unconditional and class-conditional sampling. In the unconditional setting, DG reduces FID from 16.95 to 9.68. Under class conditioning, the improvement becomes more pronounced: FID decreases from 9.52 to 5.77 and IS increases from 122.79 to 179.26. These results indicate that DG is not tied to text prompts or text-image alignment metrics; instead, suppressing detail-deficient MA directions provides a broadly useful refinement signal for DiT-based visual generation.

6.3 Main Results on Video Generation↩︎

We further evaluate DG on Wan video models [9]. Compared with image generation, video generation poses a more challenging setting, as enhancing local details must also preserve temporal consistency across frames. As reported in ¿tbl:tab:video95dit?, DG consistently improves performance across both model scales. On Wan1.3B, DG outperforms standard CFG and other competing guidance methods in terms of the overall score, while DG+CFG achieves the best overall performance. A similar trend is observed on Wan14B, where DG surpasses the strong Wan14B baseline, and DG+CFG attains the highest total, quality, semantic, aesthetic, object, and appearance scores. These consistent improvements demonstrate that the MA-based guidance is effective not only for spatial tokens in image DiTs but also for spatiotemporal tokens in video DiTs. The comparison between DG and DG+CFG further reveals their complementary roles. CFG primarily improves prompt adherence and high-level semantic alignment, whereas DG enhances fine-grained visual details and appearance fidelity.

6.4 Main Results on Visual Understanding↩︎

6.4.1 Semantic correspondence↩︎

We first evaluate whether MREP improves the spatial discriminability of DiT features on semantic correspondence. This task is particularly sensitive to whether the representation preserves object-part geometry, since the model must match keypoints across instances, categories, and appearance changes. As shown in ¿tbl:tab:semantic95correspondence?, the raw DiTF representation already forms a strong unsupervised baseline, especially on PF-Pascal. However, removing the MREP modulation leads to a substantial degradation across all datasets, showing that MA-dominated directions can obscure spatially meaningful correspondences. With MREP, the model consistently improves over DiTF under the unsupervised setting, achieving the best results on SPair-71k, AP-10K, and PF-Pascal. These results suggest that MREP does not merely improve global image semantics; it also strengthens the local part-level structure required for dense matching.

6.4.2 Semantic segmentation↩︎

We further evaluate the learned representations on ADE20K semantic segmentation. Compared with semantic correspondence, segmentation requires the representation to support dense category prediction over the entire image, making it a useful test of whether MREP preserves both local boundaries and semantic region consistency. As reported in ¿tbl:tab:segmentation95pretraining?, the MREP-enhanced features achieve 54.9 single-scale mIoU and 56.1 multi-scale mIoU, outperforming the raw DiTF features and remaining competitive with strong diffusion-based pre-training baselines. In contrast, the variant without MREP drops notably, which again indicates that directly using MA-dominated DiT features can hurt dense perception. These gains show that MREP provides a task-useful representation refinement for semantic understanding, rather than only improving feature matching on correspondence benchmarks.

6.4.3 Depth estimation↩︎

Finally, we evaluate MREP on NYUv2 depth estimation to test whether the same representation benefits geometric prediction. Depth estimation differs from the previous semantic tasks because it relies more strongly on layout, object scale, and 3D structure. As shown in ¿tbl:tab:depth95estimation?, MREP obtains the best overall performance among the compared methods, reducing RMSE to 0.220 and REL to 0.060 while improving threshold accuracy. The comparison with the w/o MREP variant is especially informative: without the modulation, performance falls behind strong diffusion-based baselines, whereas MREP yields clear gains across both error and accuracy metrics. This consistency across correspondence, segmentation, and depth estimation supports our hypothesis that modulating massive activations helps convert generative DiT features into more reliable representations for visual understanding.

6.5 Additional Applications↩︎

6.5.1 Efficient Partial-Forward Detail Guidance.↩︎

We evaluate the computational efficiency of DG using the partial-forward implementation described in 5.2. As reported in 2, DG consistently reduces inference latency relative to CFG while achieving higher aesthetic quality. For example, on SD3.5, DG reduces the inference time from 15.7 s to 10.6 s while improving the Aesthetic score from 6.01 to 6.16. These results demonstrate that DG offers a favorable efficiency-quality trade-off, enhancing visual fidelity without incurring the full computational overhead of an additional denoising branch.

6.5.2 DG on Non-DiT Architectures.↩︎

To examine whether DG relies on the standard DiT architecture, we further evaluate it on PixArt-alpha [21] and SANA [49]. PixArt-alpha injects text conditioning via cross-attention with shared AdaLN layers, whereas SANA replaces standard attention with linear attention and incorporates local convolutional operations. Following the same MA suppression strategy, we suppress dimension in PixArt-alpha and SANA. As shown in ¿tbl:tab:non95dit?, DG consistently improves detail quality under both conditional generation and CFG. These results indicate that DG captures a general detail-guidance mechanism rather than exploiting architectural characteristics specific to DiTs.

6.5.3 Token-specific Local DG.↩︎

Our spatial analysis of MAs suggests that they function as token-wise modulators of local detail synthesis. To validate this hypothesis, we construct Local DG by suppressing MAs only for spatial tokens corresponding to a target region, instead of applying suppression to all spatial tokens. As illustrated in 13, Local DG selectively enhances details within the target region while leaving surrounding objects and the background largely unchanged. This result further supports the interpretation that DG provides spatially controllable detail guidance.

6.6 Ablation Studies↩︎

6.6.1 Core components in DG.↩︎

We ablate the core design choices of DG on SD3 conditional generation. As shown in ¿tbl:tab:ablation95dg?, simply constructing a guidance branch from random channels brings only limited gains, indicating that the selected MA dimensions are important for isolating detail-related changes. Replacing the adaptive timestep schedule with a fixed suppression coefficient improves over the baseline, but remains weaker than the full design. This suggests that preserving MAs during early structure formation and suppressing them more strongly in later denoising stages is beneficial. The full DG variant, which combines calibrated MA dimension selection, adaptive MA suppression, and residual detail guidance, achieves the best detail-quality metrics.

6.6.2 Core components in MREP.↩︎

We further ablate the representation-side design of MREP. The method contains two complementary components: AdaLN-based modulation reduces the directional dominance of MA channels, while normalized MA spatial maps preserve useful spatial response patterns. As reported in ¿tbl:tab:ablation95mrep?, using AdaLN modulation alone already improves dense prediction over raw DiT features, showing that suppressing MA dominance is important. Adding MA maps without spatial normalization is less stable, because their large magnitudes can again dominate feature similarity. The full MREP representation combines AdaLN-modulated features with normalized MA maps and obtains the best overall results across correspondence, segmentation, and depth estimation.

6.6.3 Intervention depth.↩︎

We examine the effect of the intervention depth \(k=m\) of massive activations, as shown in 14 (a). Our DG strategy achieves the best performance when applied to intermediate blocks (e.g., \(k\) ranging from 4 to 10). We hypothesize that early blocks mainly contain heavy noise and lack even coarse image structures, making disruption there uninformative, while applying disruption in late blocks occurs too close to the final output and thus has minor impact on generation. Based on these observations, we primarily perturb massive activations in the intermediate layers and set the default \(k=6\) for the SD3 model. For SD3.5 and Flux, we use \(k=20\) and \(k=22\), respectively, following their layer-wise MA distributions.

6.6.4 Guidance scales.↩︎

We present the quantitative results across different scales in 14. Our DG consistently achieves stable and high Aesthetic (AES) scores (14 (b)) and CLIPScore (14 (c)). When combined with CFG, it further boosts AES performance (14 (d)). These results highlight the effectiveness and robustness of our approach in enhancing fine-grained details. Moreover, DG integrates seamlessly with CFG, enabling joint improvements in local-detail fidelity and prompt alignment.

6.6.5 User study.↩︎

We conduct a user study to evaluate the benefits of our DG strategy from three key aspects: prompt alignment, color consistency, and detail preservation. For each model, 20 annotators compare 100 image pairs generated by CFG and CFG+DG. As shown in 15, CFG+DG obtains higher preference rates, especially on detail preservation and color consistency, demonstrating that the automated detail-quality gains are aligned with human judgment.

7 Conclusion↩︎

In this paper, we presented a systematic study of Massive Activations in Diffusion Transformers and showed that they form a distinctive activation structure for dense visual computation: they are spatially distributed across image tokens, concentrated in a small set of feature dimensions, closely related to AdaLN modulation, and primarily governed by the denoising stage. This structure gives MAs a dual role. For generation, they are crucial for fine-grained local detail synthesis while having limited impact on global semantics. For understanding, their shared high-magnitude directions can dominate raw DiT features and weaken spatial discrimination.

Based on these observations, we introduced EMA, a training-free framework that elicits the capabilities of pretrained DiTs through task-aware MA modulation. For visual generation, DG constructs a detail-deficient counterfactual branch by suppressing MA dimensions and uses it to guide sampling toward more detailed outputs. DG is compatible with CFG, supports partial-forward inference, and can be applied locally to selected spatial regions. For visual understanding, MREP reduces MA directional dominance with AdaLN-based modulation while preserving spatial MA cues, producing more discriminative dense representations. Experiments across image generation, video generation, local detail refinement, semantic correspondence, semantic segmentation, and depth estimation demonstrate that EMA consistently improves both the generative quality and representation utility of DiTs. We hope this work encourages further exploration of internal activation structures as controllable and reusable signals in large visual generative models.

References↩︎