1 Introduction↩︎

Identifying the generator that produced a synthetic image has become an important problem in multimedia forensics, known as synthetic image attribution (SIA). Many existing attribution methods rely on supervised fingerprint learning, in which a classifier or feature extractor is trained to distinguish among a fixed set of known generators. Although effective in closed-set settings, such methods are difficult to scale when new generators continuously emerge, proprietary models provide only limited samples, and retraining a global attribution model is impractical.

This motivates reference-based attribution, where a query image is attributed by comparing it with references known to come from each candidate source, rather than by training a task-specific classifier. Reference-based attribution methods differ in how such references are obtained. White-box or black-box methods can exploit model access to construct query-specific references through inversion or reconstruction. In contrast, sample-based methods assume no access to a model and rely only on a small set of references from each candidate source, where each reference is a generated sample with a known source label. In this work, we focus on this sample-based setting and study how the choice of references interacts with off-the-shelf representation spaces.

Reference-based attribution is governed by two coupled factors: the representation space where images are compared and the references used to characterize each candidate source. These factors are not independent, since pretrained representations usually entangle source-related artifacts with semantic information. Semantic cues are not purely a nuisance: some may be source-discriminative, as different generators may exhibit task- or content-specific priors and failure modes. However, uncontrolled semantic variation can also confound attribution by making query-reference similarity depend on content mismatch rather than source identity. In the following, we use reference selection to broadly indicate the way the reference set is built. Its role is not to eliminate semantics, but to control query-reference semantic alignment so that useful source-related cues are preserved while accidental semantic bias is reduced.

This paper provides a controlled analysis of training-free reference-based SIA from a joint representation-reference perspective. We focus on reference selection and off-the-shelf representations, without training or fine-tuning a task-specific classifier or fingerprint extractor. For the reference selection, we compare three methods with decreasing semantic constraints: resynthesized references, semantically aligned references, and arbitrary references. For representation, we analyze features extracted from different layers of pretrained vision encoders, with CLIP as the main representation and DINOv2 used to examine whether the observed trend is encoder-specific.

Our analysis leads to three main observations. First, intermediate representations are more suitable for attribution than final semantic embeddings, since they better preserve source-discriminative cues before semantic abstraction dominates. Second, reference selection strongly affects performance: semantically constrained references reduce query-reference mismatch and improve attribution, especially under limited reference budgets. Third, query-specific resynthesis is useful when only one or two references are available. In contrast, semantically aligned references provide a better accuracy-cost trade-off once a moderate-sized reference pool is available.

Our contributions are summarized as follows:

• We provide a controlled analysis of training-free reference-based SIA by jointly studying off-the-shelf representation spaces and reference selection.

• We construct two attribution datasets, BC-Attr-6 and COCO-Attr, to evaluate reference selection under controlled semantic categories and diverse caption-derived content, respectively.

• We disentangle semantic alignment from test-time resynthesis by comparing resynthesized, semantically aligned, and arbitrary references under different reference budgets.

• We show that intermediate representations are consistently more effective than final semantic embeddings, and that the remaining semantic bias makes semantic-aware reference selection crucial for reliable attribution.

2 Related Work↩︎

Existing SIA methods are commonly organized into two paradigms: direct source attribution and reference-based source attribution [1]. Direct methods learn source-discriminative traces from query images and predict source labels with a trained decision function, whereas reference-based methods attribute a query by comparing it with source-specific reference information.

2.1 Direct Source Attribution↩︎

Direct source attribution methods assume that images from the same generator share characteristic artifacts. Early studies exploited image-domain residuals, frequency patterns, or other forensic traces introduced by generative pipelines [2]–[5]. Later works learned more structured representations, including disentangled fingerprints, contrastive embeddings, and architecture-level forensic features [6]–[8]. Recent methods have further broadened the range of feature sources used for attribution. DE-FAKE [9] combined BLIP captions [10] with CLIP visual features [11]; MAID [12] and MADE [13] exploited diffusion-model representations; OFA [14] learnt a universal fingerprint extractor from simulated forensic traces. Despite their differences, these methods require attribution-specific training or adaptation, making them less flexible when the candidate sources change.

2.2 Reference-based Source Attribution↩︎

Reference-based methods avoid training a dedicated attribution classifier by comparing a query with source-specific references. These references can be obtained under different access assumptions. With white-box access, query-specific references are produced by reconstructing or inverting each candidate generator. Attribution is then obtained by selecting the generator with the highest reconstruction quality, measured in pixel, perceptual, or deep feature spaces [15]–[17]. With black-box access, candidate generators can be queried at inference time. Bongini et al. [18] derive a textual description from the query image, generate one reference with each candidate generator, and compare the query with these references in the final-layer CLIP feature space. Under sample-only access, candidate generators are not queried; attribution instead relies on pre-collected source samples. The \(k\)NN variant of Cioni et al. [19] compares samples in a pretrained feature space, whereas Wang et al. [20] learn a low-bit-plane attribution encoder for query-sample matching. Other methods aggregate samples into source-level fingerprints [21] or model source-specific forensic distributions [22].

These works show that reference-based attribution is governed by two coupled factors: how source-specific references are constructed and in which representation space query-reference similarity is measured. However, the relationship between these two factors remains largely unexplored. Existing studies typically instantiate them as part of a complete method, without explicitly analyzing their individual contributions or their interactions. As a result, it remains unclear whether attribution performance is mainly limited by semantic mismatch in the reference set, by the inability of the representation to preserve source-specific cues, or by the compatibility between the two. This work addresses this gap by factorizing reference-based attribution into reference selection and representation choice, and studying how these two factors jointly affect source attribution.

3 Reference-Based Source Attribution↩︎

We analyze training-free synthetic image attribution as a query-reference matching problem. Given a query image, the goal is to identify its source by comparing it with references from each candidate generator in a selected representation space. As illustrated in Fig. 1, this framework decomposes training-free attribution into reference selection, representation extraction, and query-reference matching. This formulation exposes two controllable axes: the reference selection method \(\pi\), which determines how source-specific references are obtained, and the representation space \(\rho\), which determines the space where similarity is measured. Other source representations, such as white-box inversion [15]–[17], learned fingerprint galleries [21], or source-level density models [22], are complementary but outside the scope of our work.

3.1 Unified Formulation↩︎

Let \(\mathcal{G}=\{G_1,\ldots,G_K\}\) denote the set of candidate generators. Given a query image \(x_q\), closed-set attribution aims at identifying the source \(G^\ast\in\mathcal{G}\) that was used to generate it. Under reference selection method \(\pi\), each candidate generator \(G\) is associated with a reference set \[\mathcal{R}^{\pi}_{G}(x_q;M) = \{r^{G,\pi}_{1},\ldots,r^{G,\pi}_{M}\},\] where \(M\) is the number of references per source. The dependence on \(x_q\) indicates query-dependent reference selection; for query-independent sampling, the reference set reduces to \(\mathcal{R}^{\pi}_{G}(M)\).

A representation space \(\rho\) specifies the feature mapping used for comparison. In our analysis, \(\rho\) is instantiated by a frozen visual backbone \(b\) and a selected layer \(l\), \[\rho=(b,l), \qquad \phi_\rho(x)=\phi_{b,l}(x).\]

For a query-reference pair, we compute a pairwise similarity score \[a_\rho(x_q,r) = \mathrm{sim}\big(\phi_\rho(x_q),\phi_\rho(r)\big),\] where \(\mathrm{sim}(\cdot,\cdot)\) is a generic similarity function.

The pairwise scores for a candidate source are aggregated into a source-level score: \[s_{\rho,\pi,M,\alpha}(x_q,G) = \alpha\big(\{a_\rho(x_q,r): r\in R^{\pi}_{G}\}\big),\] where \(\alpha\) is a fixed, non-learned score aggregation rule, such as mean or maximum aggregation. Attribution is then performed as

\[\hat{G} = \arg\max_{G\in\mathcal{G}} s_{\rho,\pi,M,\alpha}(x_q,G).\] This formulation separates the effects of the representation \(\rho\), the reference selection method \(\pi\), the number of references \(M\), and the aggregation rule \(\alpha\).

3.2 Instantiation of Reference-based Attribution↩︎

We instantiate the framework described above along two factors: the representation space used for query-reference comparison and the reference selection for each candidate source.

Representation space. Since attribution is based on query-reference matching, the representation defines which image properties determine the similarity score. An effective representation should preserve source-discriminative cues while reducing irrelevant semantic variation. We therefore use frozen off-the-shelf visual encoders and treat layer-wise features as representation probes. This allows us to measure how attribution performance changes across feature spaces with different abstraction levels, without assuming that a specific layer directly encodes source fingerprints.

Reference selection. The second factor is how references are obtained at inference time. We consider three methods, summarized in Table 1. Under black-box resynthesis, candidate generators can be queried, but their parameters and internal states are unavailable, ruling out inversion or reconstruction-based references. We instantiate this setting by caption-guided resynthesis, where a description of the query image is used to generate references from each candidate source. Under sample-only access, generators cannot be queried, and references must be selected from pre-collected source-specific image pools. We consider two sample-only methods: semantically aligned retrieval, which selects references semantically close to the query, and arbitrary sampling, which ignores query-reference semantic correspondence. These methods define a semantic-constraint hierarchy, \(\mathrm{resynthesis} \succ \mathrm{semantic\text{-}aligned} \succ \mathrm{arbitrary}\), where \(\succ\) indicates stronger query-reference semantic alignment. This hierarchy allows us to analyze how attribution performance changes as semantic control over the references is progressively relaxed.

4 Dataset Construction↩︎

We evaluate reference-based attribution on three datasets with increasing semantic diversity: the external face-only benchmark [18], denoted as FaceResyn, and two datasets constructed for this work, BC-Attr-6 and COCO-Attr. BC-Attr-6 stands for Bias-Controlled Attribution-6: it uses balanced semantic categories and prompts shared across generators to reduce source-semantic bias. COCO-Attr uses MSCOCO-derived captions as prompts to evaluate attribution under a more heterogeneous image distribution.

FaceResyn Dataset. FaceResyn is the 10-generator core resynthesis benchmark introduced in [18]. It contains 1,000 head-and-shoulders portrait query images generated from 100 character descriptions. For each query and each candidate generator, the dataset provides one resynthesis produced from an alternative textual description of the same portrait, yielding 10,000 query-specific references. We use the released resyntheses as a constrained face-only setting.

BC-Attr-6 Dataset. BC-Attr-6 extends the face-only setting to controlled multi-category attribution. We use ten text-to-image generators: \(\mathcal{G}\)={², ³, ⁴, ⁵, ⁶, ⁷, ⁸, ⁹, ¹⁰, ¹¹}. Prompts are grouped into six semantic categories: {, , , , , }. For each generator-category pair, we generate 200 images, yielding \(10 \times 6 \times 200 = 12{,}000\) images. We use 20 images per pair as queries and the remaining 180 as references, resulting in 1,200 queries and 10,800 references. The category labels support both semantically aligned references, selected from the query category, and arbitrary references, selected without category matching. Fig. 2 shows examples of reference selection on BC-Attr-6.

COCO-Attr Dataset. COCO-Attr uses the same ten generators as BC-Attr-6, but replaces predefined semantic categories with captions sampled from MSCOCO and used as generation prompts. This removes the fixed category structure and produces a broader distribution of objects, scenes, and compositions while keeping the candidate sources unchanged. For each generator, 100 generated images are held out as queries, and the remaining images are used as references, yielding 1,000 query images in total.

5 Experimental Results↩︎

Experimental setup. All experiments follow the training-free reference-based protocol described in Section 3. We compare three reference selection methods with increasing query-reference semantic alignment: arbitrary, semantically aligned, and resynthesis. Unless otherwise specified, each candidate source is represented by \(M=10\) references. For resynthesis, FaceResyn provides one query-conditioned reference for each query-source pair, whereas for BC-Attr-6 and COCO-Attr, we first describe the query image with LLaVA-v1.5-7B [23] and then use this description to query each candidate generator. Since FaceResyn uses ChatGPT-derived query descriptions while our datasets use LLaVA-derived descriptions, we analyze trends within each dataset rather than comparing absolute performance across datasets.

For representations, we use frozen ViT-L/14 visual encoders, including CLIP-ViT-L/14@336 [11] and DINOv2-ViT-L/14 pretrained on LVD-142M [24], both with 24 transformer layers. Each image is represented by the CLS token from the selected layer, which empirically outperforms patch-token pooling in our preliminary comparisons. We report Top-1 accuracy (ACC) under closed-set attribution, where each query belongs to one candidate generator. The main experiments use max cosine similarity as the score aggregation rule, assigning each query to the source whose reference set contains the most similar image in the chosen representation space.

5.1 Layer-wise Behavior on BC-Attr-6↩︎

Fig. 3 analyzes attribution accuracy on BC-Attr-6 across CLIP and DINOv2 layers, reference selection methods, and reference budgets. The main trend is consistent: both encoders perform best at intermediate layers, while accuracy drops at both shallow and final layers. This indicates that source-discriminative cues are best exposed before the representation becomes either too low-level or too semantic.

Reference selection also matters. Arbitrary references give the weakest results, whereas semantically constrained references improve attribution. Resynthesis is competitive with small reference budgets, while semantically aligned references become stronger when more references are available. Increasing \(M\) improves accuracy but does not alter the layer-wise trend, confirming that representation choice and reference selection are complementary factors. CLIP achieves higher absolute accuracy than DINOv2, but both exhibit the same intermediate-layer behavior.

Separability diagnostic. To interpret the layer-wise behavior, we train diagnostic linear classifiers on frozen CLIP features at each layer. These probes are used only for analysis: semantic separability is measured by predicting the semantic category, while generator separability is measured by predicting the source generator. As shown in Fig. 4, semantic separability increases toward deeper layers, whereas generator separability peaks at intermediate layers and then decreases. This supports the attribution results: the final CLIP representation is the most semantic, but not the most source-discriminative. This behavior suggests a trade-off between semantic abstraction and generator discriminability. In shallow layers, source-related cues may still be mixed with many low-level variations and are not yet well organized for comparison. In final layers, high-level semantics dominate and can suppress generator-related differences. Intermediate layers provide the best compromise.

The same observation also explains why reference selection matters. Intermediate representations are not source-only: they still encode semantic information. When query images and references differ in content, this semantic component can bias similarity. Semantically constrained references reduce this mismatch, making generator-related differences more accessible for attribution.

5.2 Cross-Dataset Analysis↩︎

Table 2 summarizes the results on the three datasets using three representative CLIP layers: early (\(L6\)), middle (\(L12\)), and final (\(L24\)). For FaceResyn, arbitrary references are omitted because all images are face portraits, making random references already semantically homogeneous. For COCO-Attr, semantically aligned references are not reported because the dataset has no predefined semantic categories.

The intermediate layer \(L12\) gives the best accuracy in every dataset and reference setting, while the final layer is usually much weaker. This confirms that the intermediate-representation advantage is not specific to BC-Attr-6. The results also show the benefit of semantic control: semantically aligned or resynthesized references outperform arbitrary references whenever the comparison is available. However, resynthesis is not always the best option; on FaceResyn and BC-Attr-6, semantically aligned references achieve higher \(L12\) accuracy than resynthesized ones. Thus, query-conditioned resynthesis can reduce semantic mismatch, but it does not guarantee better source matching.

5.3 Ablations and Robustness↩︎

We further examine whether the main trends depend on the reference budget, the score aggregation rule, or common post-processing operations.

We fix the representation to CLIP \(L12\) and vary the number of references per candidate source on BC-Attr-6. For resynthesis, we evaluate up to \(M=10\) due to generation cost. Table 3 shows that the preferred reference selection method depends on the available budget. With very few references, resynthesis performs best, reaching \(60.00\%\) and \(68.08\%\) accuracy at \(M=1\) and \(M=2\). This confirms the value of query-specific semantic alignment when the reference set is too small to cover the query content.

As \(M\) increases, semantically aligned references become more effective, outperforming resynthesis from \(M=4\) onward and reaching \(95.75\%\) at \(M=100\). Arbitrary references also benefit from larger budgets, but remain less sample-efficient: semantically aligned references with \(M=4\) already outperform arbitrary references with \(M=10\) (\(76.33\%\) vs. \(67.08\%\)). This indicates that semantic mismatch can interfere with similarity-based attribution and that semantic-aware reference selection reduces this interference. Overall, resynthesis is most useful under severe reference scarcity, whereas semantically aligned retrieval offers the best accuracy-cost trade-off when a moderate reference pool is available.

The previous experiments use max similarity, that is, each candidate source is scored according to the most similar reference. To check whether the trends we observed depend on this choice, Table 4 compares alternative aggregation rules with the same representation and reference budget. Besides max and mean similarity, we consider a softmax-weighted mean, which assigns each reference similarity \(s_i\) a weight proportional to \(\exp(\tau s_i)\) within the same source and uses the weighted mean as the source score. We also consider top-\(5\) \(k\)NN voting, which predicts the source by majority vote among the five nearest references.

The ranking of reference selection methods remains stable: semantically aligned references perform best, while arbitrary references are again the weakest. Max similarity and top-\(5\) \(k\)NN voting achieve the highest accuracy, while the softmax-weighted mean provides slightly lower but comparable performance. Mean similarity performs worse, especially with arbitrary references, because averaging all references can dilute the most relevant matches and amplify semantic mismatch.

We finally test whether the layer-wise trend persists under common post-processing operations. Fig. 5 reports attribution accuracy on BC-Attr-6 after JPEG compression, resizing, and Gaussian blur, using \(M=10\) references per source. Across all perturbations, accuracy still peaks at intermediate CLIP layers and drops toward the final layer. JPEG compression and Gaussian blur cause larger degradation as severity increases, whereas resizing has a milder effect. Thus, post-processing changes absolute accuracy but does not alter the main conclusion that intermediate representations are preferable for reference-based attribution.

6 Conclusion↩︎

This paper analyzes training-free reference-based synthetic image attribution from the joint perspective of representation and reference selection. Across pretrained encoders, including CLIP and DINOv2, intermediate representations are generally more suitable for attribution than final representations, as they retain source-discriminative cues before high-level semantic abstraction dominates. However, these representations are not semantically neutral, making reference selection critical. Semantically constrained references reduce query-reference mismatch and improve attribution, especially under limited reference budgets. These findings provide practical guidance for the design of source attribution systems: resynthesis is preferable when only very few references are available, whereas semantically aligned retrieval offers a better accuracy-cost trade-off when a moderate reference pool can be collected.

References↩︎