1 Introduction↩︎

Denoising Diffusion Models (DDMs) [1] and Latent Diffusion Models (LDMs) [2] have gained significant popularity in the generative modeling landscape due to their ability to synthesize high-quality, high-resolution images across diverse domains [1], [2]. Their success has led researchers to increasingly leverage them for image editing tasks, ranging from text-driven modifications [3] to layout-based control [4]. However, a fundamental limitation persists in these workflows: human-crafted text descriptions are often too coarse to capture the nuanced visual differences, or “deltas”, that define precise, fine-grained edits. While a pair of images can effortlessly demonstrate a subtle semantic shift, translating that same transformation into a textual prompt is inherently constrained by the descriptive bottleneck of natural language. This creates a significant gap between the user’s intent and the model’s output, as the expressive power of language is frequently surpassed by the complexity of visual information.

This gap is further widened by the fact that diffusion models often remain black boxes, limiting the fine-grained interpretability required for disentangled control. Advancing such control requires understanding their semantic structures, particularly how specific attributes can be manipulated in isolation. In this regard, Generative Adversarial Networks (GANs) have proven to be useful models for achieving interpretable and disentangled edits due to their well-structured latent spaces that offer rich, linearly applicable semantics. By leveraging this organized architecture, specialized discovery frameworks [5], [6] have uncovered significant amounts of semantically meaningful directions [7], allowing for highly nuanced control. In contrast, current research on diffusion models has only uncovered a handful of disentangled latent directions [8], [9]. This disparity arises from the recursive noise estimation and the inherently complex management of variables across multiple timesteps in diffusion architectures, making it difficult to map the fine-grained visual deltas that are easily represented in GANs.

To overcome these limitations, we introduce ViDiT (Visual Direction Transfer for Diffusion), a framework that expands the existing editing vocabulary by capturing latent semantics directly from image-edit pairs. Operating on a “Learn Once, Edit Anywhere” principle, ViDiT moves away from a total dependency on human-crafted text by learning to capture latent semantics from a small set of before-and-after visual examples. Rather than a simple feedforward mapping, ViDiT learns the underlying transformation by optimizing a single, global, and continuous editing direction that transfers the visual delta into the diffusion model’s conditioning space. This allows for detailed attribute control that text alone cannot easily describe, all without requiring base-model finetuning or per-sample test-time optimization.

Our main contributions are outlined as follows:

• We present a universal framework that learns continuous editing directions from visual examples, effectively expanding the editing vocabulary of pre-trained diffusion models without requiring base model finetuning.

• We demonstrate the versatility of our direction transfer by successfully drawing from diverse sources, including unconditional generative spaces (e.g., StyleSpace [7]), domain adaptation models (e.g., StyleGAN-NADA [10]), and diffusion-based editors (e.g., Prompt2Prompt [3]).

• We showcase the “Edit Anywhere” capability of our approach by transferring a wide range of fine-grained semantics across diverse categories, including human faces, artistic styles, and extending these to real-world, in-the-wild images.

• Our evaluations demonstrate that ViDiT achieves superior disentanglement and identity preservation compared to state-of-the-art text-based diffusion editors, while providing granular control over edit intensity.

• We share our source code to facilitate further research in visual direction transfer for diffusion models.

2 Related Work↩︎

2.0.0.1 Latent Space Exploration of Diffusion Models.

Text-to-image diffusion models utilize pre-trained text encoders to represent semantically rich information within their latent spaces, enabling high-quality image generation from textual conditions [2]. Recent research has sought to uncover the underlying semantic structure of this feature space to enable more precise control. Some approaches focus on specific architectural components, such as the representations encoded in the bottleneck block of the denoising network [9], while others investigate the transformation of representations across different imaging domains [11]. Inspired by latent space exploration in GANs, researchers have attempted to identify directions targeted by specific input semantics [12]; however, these methods often struggle to generalize to large-scale models like Stable Diffusion, as the diffusion latent space lacks the compact, linear structure that makes GAN spaces amenable to direction discovery. Recent efforts have addressed this by decomposing latent variables into energy functions of semantic significance [13] or using contrastive learning to discover unsupervised disentangled directions [8]. Despite these advances, such methods remain constrained by the inherent complexity of the diffusion latent space and are limited to directions already present within the model’s learned representation. ViDiT bridges this gap by transferring well-understood semantics from diverse external sources directly into accessible, continuous editing directions.

2.0.0.2 Editing Visual Semantics with Diffusion Models.

Editing with diffusion models typically relies on text prompts or structural guidance to modify attributes [3], [4]. While effective, these methods are often limited by the descriptive bottleneck of natural language. SEGA [14] utilizes semantic guidance to steer the diffusion process without additional training, while InstructPix2Pix [15] learns to follow human-edited instructions through a conditional diffusion model. Structure-based methods such as PnP-Diffusion [16] and MasaCtrl [17] inject intermediate features to preserve layout and identity during editing, and Asyrp [9] and DiffAE [18] explore semantic latent spaces within the denoising architecture itself. Recent works have also explored diverse conditioning strategies: IP-Adapter [19] injects image-based prompts via decoupled cross-attention, while Weights2Weights [20] and Attribute-Control [21] identify linear directions in the weight or CLIP embedding spaces, respectively. However, all of these approaches either depend on coarse textual descriptions, require large-scale training data, or are restricted to directions expressible through text-level features alone. ViDiT bridges these paradigms by learning the desired edit once from visual examples and applying it as a global, continuous editing direction, an approach that captures attribute-level semantics that are self-evident in visual pairs but remain difficult for textual or structural instructions to define precisely.

2.0.0.3 Combining Structured Generative Models and Diffusion.

Recent literature has explored hybridizing the structured control of GANs with the generative power of diffusion models [22]. More relevant to our work, \(\mathcal{W}+\) Adapter [23] and Concept Sliders [24] leverage the disentangled capabilities of structured latent spaces to enhance diffusion-based editing. Concept Sliders [24] utilize low-rank adapters (LoRAs) to learn specific concepts, often employing samples from a pre-trained StyleGAN as a source for semantic directions, but require training a separate LoRA per concept, which is again bounded by a text prompt. The \(\mathcal{W}+\) Adapter [23] introduces an additional architectural module to map StyleGAN’s \(W+\) latent space directly to the diffusion model, enabling the transfer of GAN-based edits to individual images. Similarly, DiffAE [18] learns a semantic encoder alongside the diffusion decoder to enable structured latent manipulation, though this requires training a dedicated architecture from scratch. In contrast, ViDiT eliminates the need for both model fine-tuning and the introduction of additional adapter modules. By learning a single, global, and continuous editing direction directly from visual deltas, our framework enables a “Learn Once, Edit Anywhere” capability applicable to any image in a zero-shot manner. This bypasses the overhead of training per-concept LoRAs or specialized latent mapping layers, providing a more lightweight and scalable solution for incorporating complex visual semantics into pre-trained diffusion models.

3 Method↩︎

3.1 Preliminaries↩︎

Diffusion Models. Diffusion models [1], [2] learn to generate data through an iterative reverse denoising process. Given a clean image \(x_0\), the forward process adds Gaussian noise \(\epsilon \sim \mathcal{N}(0, 1)\) over \(T\) timesteps to produce a noisy latent \(x_t\). A denoising network \(\epsilon_\theta\) is trained to predict this noise via the objective: \[\label{eqn:denoising95network} \mathcal{L}_{DM} = \mathbb{E}_{x_0, \epsilon, t} [ ||\epsilon - \epsilon_\theta(x_t, t)||^2_2 ]\tag{1}\] During inference, sampling begins at \(x_T \sim \mathcal{N}(0, 1)\) and iteratively approaches \(x_0\). To steer this process, Classifier-Free Guidance (CFG) [28] computes a weighted combination of conditional and unconditional noise predictions: \[\label{eqn:cfg} \tilde{\epsilon}_\theta(x_t, c) = \epsilon_\theta(x_t, \phi) + \lambda_g (\epsilon_\theta(x_t, c) - \epsilon_\theta(x_t, \phi))\tag{2}\] where \(c\) is a text-derived conditioning embedding, \(\phi\) is the null-text embedding, and \(\lambda_g\) is the guidance scale. A key observation is that the CFG residual \((\epsilon_\theta(x_t, c) - \epsilon_\theta(x_t, \phi))\) acts as a directional signal in the noise-prediction space that steers generation towards \(c\). ViDiT exploits this structure by learning a continuous direction \(\mathbf{d}\) that replaces the text conditioning signal entirely, allowing semantic edits to be expressed as directions in this space rather than as natural language descriptions.

3.2 ViDiT: Transferring Visual Deltas↩︎

Motivation. Existing text-driven editing methods rely on natural language to specify semantic changes [3], [14], [15]. While convenient, this imposes a linguistic bottleneck: many fine-grained visual concepts, precise facial structure, subtle texture shifts, nuanced style changes, are self-evident in images but extremely difficult to describe in text. A prompt like “a person with a beard” captures only a coarse prior and inevitably entangles the edit with unrelated attributes such as age or skin tone. By contrast, a set of before-and-after image pairs \(\{(x, x')\}\), where \(x' = x + \Delta x\) is obtained by applying an edit \(\Delta x\) through any editing method, encodes the precise visual delta without linguistic ambiguity. Our key insight is that this delta can be transferred into a compact, continuous direction \(\mathbf{d}\) that lives in the diffusion model’s conditioning space and faithfully replicates the transformation on any new image.

Overview. ViDiT identifies a continuous latent direction \(\mathbf{d}\), which acts as a transferable visual prompt, by optimising it against a small set of \(N\) image-edit pairs \(\{\mathcal{X}_\text{input}, \mathcal{X}_\text{edited}\}\). The direction \(\mathbf{d}\) is initialised randomly within the model’s text-embedding space and refined using a dual-objective loss that enforces alignment at two complementary levels of abstraction: globally, via semantic feature matching, and locally, via dense latent-space correspondence. Crucially, the denoising network \(\epsilon_\theta\) and the vision-language encoder \(E_I\) are both kept frozen throughout; only \(\mathbf{d}\) is updated. This design ensures that the model’s generative prior is not disturbed and that \(\mathbf{d}\) genuinely captures the target transformation rather than overfitting to the specific images seen during training.

3.2.0.1 Semantic Alignment Loss.

To provide a global semantic anchor, we leverage a frozen Vision-Language Encoder \(E_I\) [29]. For \(\mathbf{d}\) to represent the intended semantic shift, it must be geometrically close to the edited images in the feature space of the encoder, while remaining distant from the originals. We formulate \(\mathcal{L}_\text{sem}\) as a contrastive objective: \[\label{eqn:sem-align} \mathcal{L}_\text{sem} = 1 - \text{cossim}(E_I(x'), \mathbf{d}) + \text{cossim}(E_I(x), \mathbf{d})\tag{3}\] This loss places \(\mathbf{d}\) in the correct conceptual neighbourhood of the target concept. However, because CLIP operates on globally pooled, high-level embeddings, it is insensitive to fine-grained spatial structure. A direction satisfying \(\mathcal{L}_\text{sem}\) alone may capture the right concept yet still alter unrelated facial features, since CLIP cannot distinguish between a change in beard texture and a correlated change in jaw shape. This motivates our second loss term.

3.2.0.2 Latent Alignment Loss.

To capture high-frequency structural details and enforce pixel-level correspondence between pairs, we introduce \(\mathcal{L}_\text{latent}\). Unlike GANs, which condition on compact low-dimensional latent codes, diffusion models produce spatially-aligned, high-dimensional noise-prediction maps, \(\epsilon_\theta(x_t, \mathbf{d})\), that encode dense per-pixel information. We exploit this property by maximising the difference between noise predictions for the noised edited image \(x_t'\) and the noised original \(x_t\) at random timesteps \(t \sim \mathcal{U}(1, T)\): \[\label{eqn:latent-align} \mathcal{L}_\text{latent} = -\mathbb{E}_{x_0, \epsilon, t} [ ||\epsilon_\theta(x_t', \mathbf{d}) - \epsilon_\theta(x_t, \mathbf{d})||_2^2 ]\tag{4}\] Intuitively, this encourages \(\mathbf{d}\) to produce predictions that maximally distinguish the edited from the unedited image in the denoising network’s internal feature space, grounding the direction in the model’s native representation. In practice, we observe that \(\mathcal{L}_\text{latent}\) is critical for preserving fine-grained identity cues such as hair follicle structure, specular reflections on glasses, and skin texture, details that \(\mathcal{L}_\text{sem}\) alone cannot recover. The two losses are complementary: \(\mathcal{L}_\text{sem}\) steers \(\mathbf{d}\) towards the correct semantic concept, while \(\mathcal{L}_\text{latent}\) sharpens its spatial precision. The total training objective is \(\mathcal{L} = \mathcal{L}_\text{sem} + \mathcal{L}_\text{latent}\). We provide the training algorithm for ViDiT in Alg. 4.

3.3 The “Edit Anywhere” Principle↩︎

Once \(\mathbf{d}\) has been optimized, a one-time process per editing concept that requires no fine-tuning of the base model, it effectively expands the diffusion model’s vocabulary with a new, reusable semantic token. At inference time, \(\mathbf{d}\) can be injected into the CFG formulation to steer the denoising trajectory of any image towards the target concept, regardless of domain, style, or content.

Editing via Direction Injection. We modify the inference-time CFG to incorporate the learned direction: \[\label{eqn:editing} \bar{\epsilon}_\theta(x_t, c, \mathbf{d}) = \tilde{\epsilon}_\theta(x_t, c) + \lambda_e (\epsilon_\theta(x_t, \mathbf{d}) - \epsilon_\theta(x_t, \phi))\tag{5}\] where the additional term replaces the standard text-conditioning residual with the visual-delta residual, and \(\lambda_e \in \mathbb{R}\) provides continuous, granular control over edit intensity. Setting \(\lambda_e = 0\) recovers the unedited output, while increasing \(\lambda_e\) progressively strengthens the effect. Crucially, this requires no per-image optimization or test-time adaptation: the same \(\mathbf{d}\) applies zero-shot to any input.

3.3.0.1 Multi-directional Editing.

A key advantage of the additive formulation in Eq. 5 is that multiple independently learned directions can be composed at inference time without retraining. Given a set of directions \(D = \{d_1, \dots, d_k\}\) with corresponding editing scales \(\{\lambda_{e_1}, \dots, \lambda_{e_k}\}\), the noise prediction generalizes to: \[\label{eqn:multi-edit} \hat{\epsilon}_\theta(x_t, c, D) = \tilde{\epsilon}_\theta(x_t, c) + \sum_{i=1}^{|D|} \lambda_{e_i} (\epsilon_\theta(x_t, d_i) - \epsilon_\theta(x_t, \phi))\tag{6}\] This allows complex combinations of attributes, for example, simultaneously applying a smile direction and an age direction, while preserving strong disentanglement between each component, since each \(d_i\) was optimized independently to isolate a single semantic concept.

3.3.0.2 Real Image Editing.

ViDiT directions generalize robustly to real-world photographs. To edit a real image, we first invert it into the model’s latent trajectory using DDPM Inversion [30], which produces a sequence of noisy latents \(\{x_T, \dots, x_0\}\) that approximately reconstruct the input when denoised. We then perform the reverse process with the direction-injected noise prediction: \[\label{eqn:real-edit} \bar{\epsilon}_\theta(x_t, \mathbf{d}) = \epsilon_\theta(x_t, \phi) + \lambda_e (\epsilon_\theta(x_t, \mathbf{d}) - \epsilon_\theta(x_t, \phi))\tag{7}\] The success of this zero-shot transfer to real images confirms that \(\mathbf{d}\) encodes a fundamental semantic vector within the diffusion model’s latent space, rather than an overfitted representation tied to the distribution of training pairs. As demonstrated in 3, the learned directions transfer faithfully across synthetic portraits, animal photographs, and stylized artwork alike.

4 Experiments↩︎

To evaluate the effectiveness of ViDiT in transferring semantically rich latent directions, we conduct experiments across human faces, cats, and dogs. We demonstrate that by transferring visual deltas into a continuous direction \(\mathbf{d}\), our framework achieves a “Learn Once, Edit Anywhere” capability that generalises across synthetic, real-world, and stylised images.

4.0.0.1 Experimental Setup.

We utilise Stable Diffusion [2] for all experiments. For supervision, we primarily use off-the-shelf StyleGAN2 [25] models trained on FFHQ [26], AFHQ-Cats, and AFHQ-Dogs [27]. GANs are chosen as the primary supervision source because they currently provide the highest-quality disentangled image-edit pairs; however, ViDiT is source-agnostic and we additionally demonstrate directions transferred from StyleGAN-NADA [10] and Prompt2Prompt  [3] in 3. We optimize \(\mathbf{d}\) using \(N{=}1000\) samples with the AdamW optimiser [31] for 1000 iterations. Once learned, \(\mathbf{d}\) acts as a new “token” in the model’s vocabulary, enabling zero-shot editing in \({\sim}5\) seconds on a single NVIDIA L40 GPU.

4.1 Ablation Studies↩︎

We evaluate four core design choices: the inference timestep \(T\), the number of training pairs \(N\), the choice of inversion method, and the contribution of each loss term. Identity preservation is assessed via LPIPS [32], DINO [33], and DreamSim [34], and semantic alignment via CLIP-T [29] and SigLIP-T [35]. Qualitative results are shown in 5 7 and quantitative results in 1.

Ablation on Inference Timestep. The injection timestep \(T\) controls the trade-off between edit strength and identity preservation. When \(\mathbf{d}\) is applied at the full denoising timestep, the edit is introduced early in the generative process, allowing the model to gently incorporate the semantic change while maintaining structural coherence. Reducing to \(0.4T\) applies the direction later, where the denoising network operates on lower-noise latents and produces more aggressive edits, useful when a strong effect is desired, but at the cost of noticeable identity drift, as shown in 5 (a). In our default configuration we use the full timestep, which offers the best balance between edit fidelity and identity preservation.

Ablation on Sample Size. A key practical question is how many image-edit pairs are needed to learn a reliable direction. As shown in 1, ViDiT already captures meaningful semantic content with as few as \(N{=}10\) pairs, demonstrating that the method is not data-hungry. However, DINO and LPIPS scores improve consistently as \(N\) increases, with \(N{=}1000\) yielding the best identity preservation overall. This suggests that a larger pool of pairs provides broader coverage of the target concept’s variation, leading to a more robustly disentangled direction that generalises better at inference time. Notably, CLIP-T peaks at \(N{=}100\), indicating a mild trade-off between semantic alignment and structural faithfulness at very large \(N\), which we address through our dual-loss formulation.

Ablation on Inversion Method. For real image editing, we ablate the choice of inversion strategy used to map the input into the model’s latent trajectory. As shown in 7 (b), DDIM Inversion [36] introduces noticeable drift in fine facial details, degrading structural consistency between the original and edited outputs. DDPM Inversion [30], by contrast, produces a more faithful latent trajectory that preserves high-frequency identity cues such as hair texture, glasses frames, and skin tone, yielding significantly cleaner edits. We therefore adopt DDPM Inversion as our default for real image editing throughout all experiments.

Ablation on Loss Terms. We isolate the contribution of \(\mathcal{L}_\text{latent}\) by training with only the semantic alignment loss \(\mathcal{L}_\text{sem}\). While \(\mathcal{L}_\text{sem}\) alone is sufficient to identify the target concept, the resulting direction moves the output in the correct semantic direction, the absence of \(\mathcal{L}_\text{latent}\) leads to entangled edits that undesirably alter facial structure and skin tone, as visible in 5 (b). This is because \(\mathcal{L}_\text{sem}\) operates on globally pooled CLIP features, which lack the spatial resolution to enforce pixel-level identity consistency. Adding \(\mathcal{L}_\text{latent}\) grounds \(\mathbf{d}\) in the denoising network’s dense, spatially-aligned prediction space, yielding a marked improvement in LPIPS and DINO without sacrificing semantic alignment, confirming that the two losses are complementary rather than competing.

4.2 Qualitative Results↩︎

3 demonstrates ViDiT’s editing capabilities across a wide range of semantics and domains. Our method transfers fine-grained directions spanning facial attributes such as age, gender, beard, and eyeglasses, as well as animal domains including cats and dogs. A key property of our edits is their high degree of disentanglement: only the targeted attribute is modified while all other aspects of the image, including identity, background, and artistic style, are faithfully preserved. Furthermore, the learned direction \(\mathbf{d}\) generalizes robustly beyond its supervision source: edits optimized on structured GAN pairs apply equally well to stylized illustrations, classical paintings, and real-world photographs, confirming that \(\mathbf{d}\) encodes a fundamental semantic vector within the diffusion model rather than an overfitted representation tied to the source domain.

7 (c) further demonstrates the compositional power of ViDiT. Since each direction is learned independently and injected additively at inference time, multiple directions can be combined simultaneously without retraining, producing coherent multi-attribute edits such as simultaneously applying Asian, Beard, and Smile while preserving strong disentanglement between each component.

4.3 Quantitative Comparisons↩︎

Comparison with Visual Conditioning Methods. To quantify the advantage of visual-delta supervision over text-based and visual conditioning alternatives, we evaluate all methods on 100 generated samples. As shown in 2, text-based and weight-space methods suffer from significant semantic entanglement, unintentionally altering subject identity to match coarse linguistic or model priors. By contrast, ViDiT outperforms all competing methods on identity-preservation metrics (CLIP-I, SigLIP-I, DINO) while remaining competitive in semantic alignment.

Comparison with Text-driven Editing Methods. We benchmark ViDiT against text-driven editing methods, SEGA [14], Prompt2Prompt [3], InstructPix2Pix [15], and Concept Sliders [24], on 200 input-edit pairs across “Asian” (Race #1) and “Smile” semantics. Identity preservation is evaluated with LPIPS  [32] and DINO [33], and semantic alignment with CLIP-T [29] and SigLIP-T [35]. Importantly, our edits are applied without access to any target text prompt, relying solely on the direction transferred from visual pairs. As shown in 3, ViDiT significantly outperforms all text-driven baselines on identity preservation while remaining competitive in semantic alignment. An extended qualitative comparison against a broader set of baselines, including PnP-Diffusion [16], MasaCtrl [17], Asyrp [9], and DiffAE [18], is provided in 7 (a), further confirming the superiority of ViDiT in preserving identity and disentangling targeted attributes.

User Study and Rescoring Analysis. We conducted a perceptual study with 40 participants on Prolific.com on real images, comparing ViDiT against SEGA [14], InstructPix2Pix [15], Prompt2Prompt [3], and Concept Sliders [24]. Participants rated 60 input-edit pairs on a 1–5 scale for semantic success and disentanglement quality. As reported in ¿tbl:tab:userstudy?, ViDiT achieves the highest mean preference score, confirming that our zero-shot edits are perceived as more disentangled and semantically faithful than existing text-driven alternatives.

To further assess disentanglement, we perform a rescoring analysis measuring how CLIP classification probabilities for specific attributes shift following an edit, following [8], [37]. We apply four editing directions, Asian (Race #1), Smile, Gender, and Beard, to 100 Stable Diffusion-generated images and report the resulting shifts in [tbl:tab:rescoring]. Targeted edits strongly increase the probability of the corresponding attribute (diagonal entries), while off-diagonal interactions remain minimal for unrelated attributes, confirming disentanglement. Some correlated shifts are semantically expected: the Gender edit towards femininity notably reduces the Beard score, and the Beard edit slightly diminishes Smile, consistent with known biases in the underlying generative model.

5 Limitations↩︎

While ViDiT faithfully represents the semantic transformation encoded in the input-edit pairs, the disentanglement of the learned direction is inherently bounded by the disentanglement of the supervision source. When pairs exhibit residual entanglement, the learned direction may reflect those characteristics. Additionally, due to biases present in CLIP and Stable Diffusion, certain attributes such as age can become entangled with correlated visual cues (e.g., white hair or eyeglasses). At present, ViDiT optimizes a single global direction per concept, which may limit expressiveness for highly localized or spatially varying edits where different regions of the image require different degrees of modification. As the broader ecosystem of editing methods continues to improve, ViDiT stands to directly benefit from higher-quality supervision sources, and exploring this further remains an interesting avenue for future work. Similarly, extending the framework to richer semantic concepts beyond facial and artistic attributes such as object-level or scene-level edits represents a natural and compelling direction for future investigation.

6 Ethics Statement↩︎

ViDiT does not introduce any new bias of its own. It learns directions by transferring semantics from pretrained models, including the supervision sources and the CLIP [29] and Stable Diffusion [2] models it builds on, and therefore inherits whatever biases those models already encode rather than creating additional ones. Any demographic correlations observed in our edits, such as the residual cross-attribute shifts in our rescoring analysis (Tab. [tbl:tab:rescoring]), originate in these pretrained components.

We anonymize racial attributes throughout: directions are referred to only by non-semantic numeric labels (e.g., “Race #1”, “Race #2”), which identify directions inherited from the source latent space and carry no ordering, ranking, or preference for any group. We deliberately avoid explicitly naming any race, and our use of these directions is intended purely to demonstrate transfer capability across diverse semantics.

Finally, like all image synthesis and editing technologies, ViDiT carries a risk of misuse for the generation of deceptive or manipulated media [38]. We acknowledge this concern and encourage responsible deployment, including the development of appropriate detection and attribution tools alongside such capabilities.

7 Conclusion↩︎

We introduced ViDiT, a framework that expands the editing vocabulary of pre-trained diffusion models by transferring visual deltas directly from image-edit pairs into continuous, transferable latent directions. Operating on a “Learn Once, Edit Anywhere” principle, ViDiT learns a single global direction from a small set of before-and-after examples without fine-tuning the base model or requiring per-image optimization at inference and applies it zero-shot to any image, including real photographs and stylized artwork.

By framing the problem as visual-delta transfer rather than text-prompt engineering, ViDiT overcomes the descriptive bottleneck of natural language and achieves significantly stronger identity preservation than text-driven and instruction-following baselines. The dual-objective training, combining semantic alignment via a frozen vision-language encoder with latent alignment via the denoising network’s native prediction space, is key to achieving disentangled edits that preserve fine-grained identity cues. Quantitative benchmarks and a user study with 40 participants confirm that our zero-shot edits are perceived as more disentangled and semantically faithful than existing alternatives.

While we leverage structured generative models as a high-quality supervision source, the framework is entirely source-agnostic: directions learned from domain-adaptation models, diffusion-based editors, or any method producing before-and-after image pairs are equally well supported, pointing toward a general paradigm for incorporating complex visual semantics into diffusion models from any generative source.

8 Acknowledgements↩︎

This research is supported by the National Science Foundation under Grant No..

9 Qualitative Comparisons with Diffusion-based Editing Methods↩︎

To qualitatively compare the edits performed with diffusion-based editing methods and the edits performed by ViDiT, we provide additional results here. In addition, we further demonstrate the editing capabilities of our framework, such as edit strength interpolation and generalization over out-of-domain images (e.g.images with style components) to demonstrate how the edit strength can be controlled and applied to a diverse set of imaging settings.

To compare the edits performed by ViDiT with diffusion-based editing methods, we benchmark our approach against several recent approaches including Concept Sliders [24], SEGA [14], InstructPix2Pix [15] and Prompt2Prompt (P2P) [3]. Here we use Null-text inversion (NTI) to adapt Prompt2Prompt for real image editing (NTI + P2P). In particular, SEGA, Prompt2Prompt, and InstructPix2Pix often result in substantial alterations to the input image for edits like “Race #1”. In addition, Prompt2Prompt and SEGA lead to entangled edits for edits such as “Gender”, where alterations on attributes such as age are also visible, whereas InstructPix2Pix produces results that are deteriorated in terms of image quality. As illustrated in Fig. 5 of the main paper, ViDiT surpasses these alternatives in maintaining semantic accuracy and in its ability to execute disentangled edits. Concept Sliders face challenges in applying multiple edits simultaneously, such as combining Race and Beard modifications, resulting in significant deviations from the original input image, whereas ViDiT can successfully combine multiple edits.

In addition, we provide comparisons with LEDITS++ [39], which requires text prompts to perform edits within the Stable Diffusion model. We perform these comparisons on Beard, Gender, and Race semantics in Figs. 8, 9 and 10. As can be seen from the results, our method performs more disentangled edits compared to LEDITS++. Furthermore, we provide comparisons with Asyrp [9] and DiffAE [18], PnP-Diffusion [16] and MasaCtrl [17] in main paper Fig. 6.

10 Supplementary Editing Results and Details↩︎

In addition to the results provided in the main paper, we provide additional editing results in the supplementary material. Specifically, we provide edit interpolation results on generated images in Fig. 11, editing results on images generated by Stable Diffusion in Fig. 16, editing results on artistic paintings in Fig. 17, and editing results on real images in Fig. 18. Furthermore, we provide additional details on the user study setup in Fig. 13.

11 Failure Cases↩︎

Extending our discussion on the limitations of ViDiT, we provide examples of failure cases in Fig. 12. When the supervision pairs encode very nuanced visual changes (e.g.minor modifications to nose shape), the learned direction \(\mathbf{d}\) may fail to capture sufficient semantic signal to reproduce the edit faithfully. As demonstrated qualitatively, even in such cases ViDiT can still localize the region to be edited; the limitation lies in the resolution of detail that can be encoded within the diffusion model’s conditioning space.

12 Comparisons with \(\mathcal{W}+\) Adapter and Concept Sliders↩︎

We provide a qualitative comparison between the \(\mathcal{W}+\) Adapter [23] and ViDiT in Fig. 14. ViDiT learns continuous editing directions from before-and-after image pairs and applies them to any given image without modifying the base model. In contrast, the \(\mathcal{W}+\) Adapter requires fine-tuning Stable Diffusion by training a separate adapter module. Our results demonstrate that ViDiT more accurately preserves the integrity of the input image while implementing the desired edits, such as changes in Gender or Age.

Moreover, we provide qualitative comparisons with Concept Sliders [24] in Figs. 8, 9 and 10, where ViDiT surpasses [24] both in disentanglement and representation quality without the need to train separate LoRA models per direction.

13 Supervision Pairs vs.Transferred Edits↩︎

To illustrate how faithfully ViDiT transfers the semantic transformations encoded in the supervision pairs, we show side-by-side comparisons between the input-edit pairs used for training and the corresponding edits produced by ViDiT on new images in Fig. 15.

References↩︎