1 Introduction↩︎

With recent advancements in diffusion models [1], [2], generative AI now produces high-quality, diverse, and realistic visuals that are indistinguishable from real images. The spread of platforms and services makes this technology accessible to a wide audience.

This rise in generated images has prompted regulatory entities to react. The EU AI Act [3], the White House executive order [4], and Chinese AI governance [5] require that AI-generated content be easily identifiable and traceable. Among existing methods, such as metadata [6] and forensics [7], digital watermarking stands out as a key approach to address the issue.

This leads to a revival of interest in the design of new watermarking techniques. Classical methods modified Fourier or wavelet coefficients [8], [9] to leverage the theoretical literature on watermarking designs [10], [11]. This approach combined knowledge from signal processing [12], information theory [13], and statistical theory [14] in order to inform their design. The modern approach, on the other hand, is now fundamentally based on machine learning, leveraging the flexibility of deep neural networks [15]–[17].

The need to trace AI-generated content spurred a line of research specifically dedicated to take advantage of the generation process itself [18]–[20]. However, the difficulty of integrating these methods into existing infrastructure has led the industry to favor post-hoc methods [16], [17], [21]. Post-hoc methods embed the watermark in content after it has been generated, meaning they cannot reliably be used for open-source/open-weights workflows. This is of no consequence for proprietary models: the generation pipeline is kept behind an API. In this setting, modern post-hoc methods offer a plug-and-play approach with detection performance that can easily be tuned.

What we term the modern approach in this paper can be characterized by three main ingredients: 1) a multi-bit watermarking framework, 2) an embedder and a decoder designed as deep-neural networks, and 3) a training pipeline that explicitly includes robustness and quality considerations by introducing image-processing augmentations as well as psycho-visual masking [22]. The simplicity and flexibility of this recipe allowed the rapid refinement of embedders and decoders since the seminal HiDDeN architecture [15].

Nevertheless, the modern approach vastly increases the attack surface on the watermarking channel. First of all, they inherit the vulnerability of deep-neural networks to adversarial examples [23], making detectors extremely brittle against white and black-box attacks [24]. Secondly, modern detectors lack the concept of the watermarking key. Combined with the first vulnerability, this means that the watermarking channel is fully compromised as soon as the detector is made public, since erasure and copy attacks are then trivial to perform. This is fundamentally a security problem, which is vastly more understood and studied in the classical approach [25]–[27].

Another arresting decision with the modern approach is its focus on multi-bit watermarking. In practice, content traceability does not necessitate decoding a message. Indeed, re-framing watermarking as a detection problem – i.e. zero-bit watermarking – is a proven way to tackle the traceability problem [28], [29]. We further discuss the question of multi-bit and zero-bit watermarking in Section 2. For now, it suffices to observe that modern multi-bit decoders can be converted to zero-bit detectors easily – see [20], which demonstrates the excellent performance of such conversion.

Stemming from these observations, this work aims to challenge the assumed superiority of the modern approach by asking the following question: Does modern post-hoc watermarking consistently outperform classical approaches in a zero-bit setting? We answer this question through the following contributions: We establish a framework to compare methods and attacks under zero-bit assumption, we provide a fair comparison of modern and traditional watermarking, and we compare recent attacks within the same realistic framework.

2 Zero-Bit Watermarking↩︎

Zero-bit watermarking is fundamentally different from multi-bit watermarking: the former is a detection problem, whereas the latter is a communication problem. We frame it as a hypothesis test between two hypotheses: \(\mathcal{H}_0\) the image is not watermarked, \(\mathcal{H}_1\) the image is watermarked. Zero-bit watermarking aims to embed a signal in the host such that the power \(\mathrm{P}_{\mathrm{D}}\) of the detector is maximized under a guaranteed, fixed, probability of false-alarm \(\mathrm{P}_{\mathrm{FA}}\).

The watermarked detector is a function \(\phi : \mathbb{R}^{D} \times \mathcal{K} \rightarrow \mathbb{R}\) that takes an image \(\mathbf{x}\in \mathbb{R}^{D}\) and a key \(k \in \mathcal{K}\) as input. The test \(\delta_k(\mathbf{x})\) then decides on either hypothesis by comparing the detector’s output against a fixed threshold \(\tau \in \mathbb{R}\): \[\label{eq:stat-test} \delta_k(\mathbf{x}) := \phi(\mathbf{x}, k) \lessgtr^{\mathcal{H}_0}_{\mathcal{H}_1} \tau.\tag{1}\]

In practice, the distribution of the detector’s output is often known under \(\mathcal{H}_0\). It is then simpler to work directly with the \(p\)-values \(p_{\phi}(\mathbf{x},k) := F^{-1}_{\phi}\left(\phi(\mathbf{x},k)\right),\)where \(F^{-1}_{\phi}\) is the quantile function associated with the distribution of \(\phi(\mathbf{x},k)\).

When \(F_{\phi}\) is absolutely continuous, the distribution of the \(p\)-values is uniform. Replacing \(\phi\) by \(p_{\phi}\) in Eq.@eq:eq:stat-test , one guarantees a level \(\alpha \in [0,1]\) for the test simply by setting \(\tau = \alpha\).

2.1 The hypercone detector↩︎

Let \(\mathbf{k}\) be a normalized secret vector in a (secret or not) \(M\)-dimensional subspace \(\mathcal{S}_{\mathbf{k}} \subset \mathbb{R}^{M}\). We can build a detector \(\phi\) in two steps. Let \(\mathbf{x}\in \mathbb{R}^{D}\) be an input image and \(P_{\mathbf{k}}: \mathbb{R}^{D} \rightarrow \mathcal{S}_{\mathbf{k}}\) the projection to the key subspace. We can compare the extracted vector \(\mathbf{r}:= P_{\mathbf{k}}(\mathbf{x})\) to the secret vector \(\mathbf{k}\) by computing the angle \(\theta\) between the two, \(c(\mathbf{r}, \mathbf{k}) := \frac{|\mathbf{r}^{T} \mathbf{k}|}{||\mathbf{r}|| } = \cos \theta\).

Now, if we construct \(P_{\mathbf{k}}\) such that the distribution of \(P_{\mathbf{k}}(\mathbf{x})\) is isotropic under \(\mathcal{H}_0\), the probability that \(\mathbf{r}\) falls inside the hypercone of axis \(\mathbf{k}\) and half-angle \(\theta\) is given by:

\[\label{eq:hypercone-pfa} \mathrm{P}_{\mathrm{FA}}= 1-I_{\cos^2\theta}\left(1/2, (M-1)/2\right).\tag{2}\]

This is indeed the probability of false alarm of the test based on the detector \(\phi_{\mathrm{hypercone}}\) defined as \(\phi_{\mathrm{hypercone}}(\mathbf{x},\mathbf{k}) := c\left(P_{\mathbf{k}}(\mathbf{x}), \mathbf{k}\right).\)

2.2 Applications↩︎

2.2.0.1 Broken-Arrows

In the original work [9], the authors use \(\phi_{\mathrm{hypercone}}\) as the detector. Both the secret vector \(\mathbf{k}\) as well as the projection \(P_{\mathbf{k}}\) are secret. The projection is composed of two steps. The first step performs a (public) wavelet transform on the input and conserves the first \(N_f\) coefficients. These coefficients are then projected into the secret subspace using \(M\) secret (pseudo)-orthogonal basis vectors. Within this subspace, \(N_c\) hypercones are defined, using the closest one to the host is selected for embedding: this is the secret vector \(\mathbf{k}\). The host vector is then pushed as far as possible from the detection border in order to maximize robustness. Note that due to the use of several hypercones, the \(\mathrm{P}_{\mathrm{FA}}\) is slightly higher than reported in Eq. 2 . Using a simple union bound, the \(\mathrm{P}_{\mathrm{FA}}\) for Broken-Arrows is such that \(\mathrm{P}_{\mathrm{FA}}^{(\mathrm{BA})} \leq N_c \mathrm{P}_{\mathrm{FA}}\).

2.2.0.2 Modern multi-bit approach

The hypercone detector can be trivially applied to modern multi-bit watermarking by observing that their decoder are all based on projecting an input into a \(M\) dimensional subspace, \(M\) being the number of bits in the message. The message is then decoded by thresholding the value of each vector element, usually assigning \(0\) and \(1\) to negative and positive elements respectively. By skipping the thresholding step, we obtain the projection \(P_{\mathbf{k}}\) for free. It remains to define the secret vector \(\mathbf{k}\). To do so, notice that the original multi-bit schemes embed a message \(\mathbf{m}\in \{0,1\}^{M}\). We can construct the secret vector by modulating \(\mathbf{m}\) with antipodal modulation: we map 0s to -1s and 1s to 1s. Finally, we normalize the modulated message to obtain the final secret vector \(\mathbf{k}\in \{-1/\sqrt{M}, 1/\sqrt{M}\}^M\). The isotropy assumption must be checked carefully for this approach – see [20][Appendix C] for a generic methodology to enforce this assumption.

3 Evaluation methodology↩︎

The recent watermarking reference [27] defines a watermarking system as: “[...] the embedding of a robust, imperceptible and secure information.”. These properties were given thorough definitions in [30]: “Robust watermarking is a mechanism to create a communication channel that is multiplexed into an original content [where] the perceptual degradation of the marked content [...] with respect to the original content is minimal and [where] the capacity of the watermark channel degrades as a smooth function of the degradation of the marked content.” and “Watermark security refers to the inability by unauthorized users to have access to the raw watermarking channel.”.

We herein propose a methodology to compare and rank the performance of each watermarking based on these three axes. We motivate our decision in each case, with the goal to design an evaluation which is fair and relevant to a realistic implementation of watermarking systems.

A first important choice is the nature of images to use. Experiments are conducted on \(1024 \times 1024\) natural images to align with the scale-dependent capacity of watermarking and the high-resolution standards of generative models and digital media.

3.1 Detectability/Robustness↩︎

We can rank the robustness of watermarking schemes according to two main methods. The first is by computing some statistics on the \(p\)-values of each scheme across different attack scenarios. The second is by setting a detection level \(\alpha\) guaranteeing a chosen \(\mathrm{P}_{\mathrm{FA}}\). In practical scenarios, a detection threshold is always fixed, we thus focus on the second method. The hypothesis test is thus simply: \(p(\mathbf{x}) \lessgtr^{\mathcal{H}_0}_{\mathcal{H}_1} \alpha\).

How low should we set the level \(\alpha\)? We can observe a race towards higher capacities in modern watermarking schemes, with TrustMark [16] starting at 100 bits, followed by Videoseal [17] at 256 bits, and most recently ChunkySeal [31] at 1024 bits. When no attack is performed, more capacity usually translates to higher detectability after converting into a zero-bit detector. In a real-world scenario, we argue that guaranteeing extremely low \(\mathrm{P}_{\mathrm{FA}}\), say \(10^{-100}\), under a few image processing attacks is far less valuable than guaranteeing a modest \(\mathrm{P}_{\mathrm{FA}}\), say \(10^{-6}\), across all possible attack scenarios.

3.1.0.1 Decision

We follow the BOWS-2 competition [9] and set \(\alpha = 10^{-6}\) so that in expectation, one image out of one million is a false positive.

3.2 Imperceptibility↩︎

In this work, we compare the quality between two images using the standard choice of the \(\mathrm{PSNR}(\mathbf{x}, \mathbf{y}) = -10 \cdot \log_{10} \left( \frac{1}{D}\sum_{i=1}^{D}(x_i - y_i)^2 \right).\) We are aware of the use of perceptual metrics such as the LPIPS [32] and psycho-visual masks (JND) [22] during the training of modern watermarking encoders. However, in order to be fair with respect to detectability, we prefer working at a fixed watermark signal power which makes the \(\mathrm{PSNR}\) a natural choice in this case.

To ensure fairness, we thus fix the \(\mathrm{PSNR}\) of watermarking signal to a fixed level \(\eta\) by scaling the signal after embedding such that:

\[\mathbf{s}:= \psi(\mathbf{x}) - \mathbf{x};\quad \mathbf{x}_{\mathrm{wm}} = \mathbf{x}+ \mathbf{s}/||\mathbf{s}||\sqrt{D}10^{\frac{\eta}{20}}.\]

3.2.0.1 Decision

We measure the impact of watermarking on image quality using the PSNR between the original and the watermarked images. We fix \(\eta=\)​42dB. We measure the impact of an attack using the PSNR between the watermarked and the attacked images.

3.3 Security↩︎

We now propose four scenarios where the attacker’s goal is to erase the watermark of the image. More precisely, an attack is deemed successful if the attacked image \(p(\mathbf{x}_{\mathrm{atk}}) < \alpha\), that is, if its p-value is below the level \(\alpha\) of the detector. We present each scenario in decreasing order of knowledge of the attacker about the detection process.

3.3.0.1 White-box scenario

We provide the attacker with all the knowledge pertaining to the detection process. For the modern approach, this corresponds to access to the detector as a white-box. This most closely reflects the standard Kerchoff’s principle, where everything should be public but the key.

For the modern approach, we formulate the problem as the optimization of an adversarial perturbation: \[\label{eq:adv-opt} \begin{cases} \min_{\boldsymbol{\epsilon}\in\mathbb{R}^D} &||\boldsymbol{\epsilon}||^2_2\\ \mathrm{s.t.} &p(\mathbf{x}+ \boldsymbol{\epsilon}) > \alpha \end{cases}\tag{3}\] This is usually solved iteratively, with a fixed model query budget of \(Q\). For this paper, we settled on the boundary projection DDN Attack [33] with gradient.

For Broken-Arrows, we also disclose the key to the attacker. Though unfair to Broken-Arrows, this allows us to use closed-form formulas to compute the optimal attack using Eq.(11) and Eq.(12) in [9].

3.3.0.2 Black-box scenario

The attacker only has access to the detector as a black box: they can query it and obtain a yes/no answer, but the gradient cannot be computed exactly through backpropagation. This attacker still aims to solve the optimization problem in Eq. 3 . We settled on CGBA [34], which is currently the most efficient black-box attack for locally linear classifiers.

3.3.0.3 Oracle scenario

Contrary to previous scenarios, the attacker does not have access to the detector. They instead use an oracle \(O : \mathbb{R}^D \rightarrow \mathbb{R}\) which serves as a proxy to the detector’s output. We selected two attacks as representative of this scenario, with two different approaches. The first, WmForger [35], uses a preference model as an oracle and performs a gradient ascent on the image. In other words at the \(i\)-th iteration, we have: \[\label{eq:wmforger} \mathbf{x}_{\mathrm{atk}}^{(i)} = \mathbf{x}_{\mathrm{atk}}^{(i-1)} - \nabla_{\mathbf{x}_{\mathrm{atk}}^{(i-1)}}O\left(\mathbf{x}_{\mathrm{atk}}^{(i-1)}\right).\tag{4}\] The attack stops after a fixed number of iterations \(Q\). The second attack, Watermark In the Sand (WIS) [36], uses the oracle as a stopping condition. It locally modifies the image at each step and continues until the condition is reached. We found the design in the original paper – based on inpainting and GPT-as-a-judge oracle for quality – computationally intensive and inefficient. We simplified the design for this paper, purifying local patches using a VAE and using the PSNR as an oracle. This resulted in far better attack performance at a fraction of the computational cost – see Appendix 6 for details about our implementation.

3.3.0.4 Blind scenario

The attacker has no access to the detector and uses no feedback from an oracle. This is the classic robustness scenario in watermarking that tests detectability against sets of classic image processing transformations – e.g. JPEG compression, sharpening, gamma transform, etc. However, recent attacks based on the use of diffusion models and VAEs, such as Purification [37], also fall in this category since the attacker must set the number of steps blindly. For this paper, we made the decision to focus exclusively on valuemetric operations. The reason is that the robustness against geometric operations such as crops, rotations, perspective shifts, etc, can be generically improved by decomposing the image into patches – this is the approach recently taken by SynthID [21] – and synchronization techniques [[38]][8][Section 9.3]. Since we want to focus on the performance of the baseline watermarking systems at a fixed image size, we forego the comparisons to these operations.

4 Results↩︎

4.1 Experimental Settings↩︎

4.1.0.1 Dataset

We perform watermark embedding and attacks on 200 natural images from the MFlickr [39] dataset. All images were resized to \(1024 \times 1024\) using bilinear interpolation, with further cropping applied to preserve the aspect ratio.

4.1.0.2 Modern watermarking

We chose Videoseal [17] and TrustMark (without error correcting codes) [16] as modern representatives of post-hoc methods. They were converted to a zero-bit watermarking scheme using the methodology in Section 2. The detectors were whitened using the methodology presented in Appendix C of  [20] in order to match the isotropy assumption of the hypercone detector. The dimensions of the watermark subspace for Videoseal and TrustMark are \(M=256\) and \(M=100\) respectively.

4.1.0.3 Classical watermarking

We chose Broken-Arrows [9] as the state-of-the-art of the classical approach. We perform a three levels wavelet decomposition on the GPU using the Pytorch Wavelet Toolbox library¹ with Daubechies-9 wavelets. We embed only within the first \(N_f=60492\) coefficients. We set the dimension of the subspace as \(M=128\) and use \(N_c=50\) secret hypercones.

For all methods, the key is randomized for each image, and the watermarking signal is scaled to achieve a fixed PSNR of 42dB with respect to the original image.

4.1.0.4 White-box scenario

We performed the DDN Attack [33] on modern watermarking techniques with a fixed query budget \(Q=250\). For Broken-Arrows, we perform the optimal attack as described in Section 3.3.

4.1.0.5 Black-box scenario

We perform the state-of-the-art CGBA attack [34] for both modern and classic watermarking with a fixed budget of \(Q=2000\) queries. We restrict the search space by keeping only the lowest \(1.25\%\) DCT frequencies (including DC) – see Section 5.2 in [40]. Preliminary experiments showed an average search overhead of \(10\) queries to find the adversarial point at each iteration, irrespective of the method. Following the theoretical recommendations in [41], we thus set the number of queries for estimating the gradient to \(2\).

4.1.0.6 Oracle scenario

We use two novel methods: WmForger [35] and our version of the Watermark In the Sand (WIS) attack as defined in Appendix 6. For the latter, the image is downsampled to \(512\times 512\) before being passed through the VAE. We use the SANA [42]².

4.1.0.7 Blind scenario

We use the SANA and Stable-Diffusion-2.1³ diffusion models for purification. The Flow-Matching Euler Discrete scheduler and Euler Discrete schedulers are used respectively. For basic image processing valuemetric operations, we only succeeded to attack the detectors with JPEG compression at a quality factor of 5.

We reported examples of all used attacks with their residue compare to the watermarked image in figure 1.

4.1.0.8 Metrics

We compute the attack success rate of each attack as \(ASR = \frac{\sum_{i=1}^N 1(p_\phi(\texttt{atk}(\mathbf{x}_i), k_i) > \alpha)}{N}\), where \(N\) is the number of images \(x_i\) and \(\texttt{atk}\) is the evaluated attack. The threshold for attack success is set at \(10^{-6}\). We measure the distortion due to the attack using the PSNR with respect to the watermarked image.

4.2 Experimental Results↩︎

We report the ASR at a threshold of \(\alpha=10^{-6}\) as a function of PSNR for each scenario and watermarking technique in figure 2.

4.2.0.1 White-box scenario

Broken-Arrows clearly outperforms TrustMark and Videoseal. For the latter, the attack achieves a distortion nearing the sub-quantization limit around 60dB, whereas the true optimal attack for Broken-Arrows achieves \(100\%\) only around 40dB. Due to their finite training set, the detection region of DNN-based detectors in pixel space contains many "blind spot" making them vulnerable to small perturbations [23]. The possibility of computing gradients makes these blind spots easy to identify with a gradient projection attack. The way Broken-Arrows detection is built precludes the existence of such adversarial signals

4.2.0.2 Black-box scenario

Both Videoseal and TrustMark are once again far more vulnerable than Broken-Arrows. The gap between the modern and classical methods is surprisingly large: nearly 15dB separates the \(100\%\) ASR threshold between the two approaches. The reason for the lack of success against Broken-Arrows is surprising as it fully meets the assumptions of the attack, notably the linearity of the detector. From the results in the white-box scenario, we have an upper-bound on the smallest distortion for a successful attack. Leveraging the theoretical study in [41], we can estimate the expected distortion at each attack iteration depending on the distortion after one iteration. The main insight from this analysis is that if finding a good adversarial example is difficult early on, convergence to the optimum will be slow. Figure 3 presents the PSNR distribution for the initial adversarial points and their state after 2000 queries across both watermarking techniques. The results indicate that initial boundary searches against Broken-Arrows incur a higher distortion cost compared to Videoseal. We hypothesize that Videoseal susceptibility to find a better initial boundary point in the pixel space arises from the inherent geometry of DNN detectors.

4.2.0.3 Oracle scenario

For lower distortion attacks (\(PSNR \geq 35\) dB), Broken-Arrows is the most secure method. If we accept higher distortion, Videoseal becomes better. The figure 4 shows the details of each attack depending on the watermarking methods. It shows that WmForger is the most efficient oracle attack against TrustMark, making it the least secure attack for \(PSNR \geq 28\) dB. This attack is less efficient against Videoseal and has no effect on Broken-Arrows watermarked images. Our WIS attack has a lot more impact on Broken-Arrows. The VAE purification + downsampling operation results in systematic removal of the mid-frequency wavelet coefficients (\(LH, HL, HH\)), effectively filtering out the watermark signal embedded in those sub-bands. A possible defense is to concentrate the signal in the lowest frequency coefficients – this would further improve robustness, but the resulting wavelet artifacts

4.2.0.4 Blind Scenario

The results in Figure 2 show the best performance of Regeneration, VAE Purification, and JPEG with a quality factor of 5, as it was the only classic value-metric augmentation that had an impact on the watermarking methods. In this scenario, the three methods are robust, with at least \(PSNR \leq 30\) dB required to erase the watermark. Notably, the Regeneration attack has the same efficiency across all methods. A first difference between the methods lies in the VAE purification, which has a greater impact on Broken-Arrows than on modern methods for the same reasons as for WIS. Secondly, TrustMark is not robust against strong JPEG compression, unlike Videoseal and Broken-Arrows. Even if Videoseal stands out as the most robust method in this scenario, the gain compared to Broken-Arrows is marginal. TrustMark is the overall worst choice of the three.

5 Discussion and Conclusion↩︎

Our findings reveal a critical trade-off between security and robustness when comparing modern and traditional watermarking. While modern DNN-based methods offer no significant gain in robustness under the tested conditions, they exhibit a significant decline in security. However, modern techniques are primarily optimized for high-capacity multi-bit payloads rather than the zero-bit identification scenario addressed here. Furthermore, this study does not account for geometric distortions, typically handled by modern methods.

6 Our Watermark in The Sand↩︎

We developed a custom implementation of the WIS attack, adapting the original concept to improve visual fidelity and efficiency. The original WIS method relies on an iterative patch-inpainting process governed by a GPT-as-a-Judge oracle. This approach often introduces significant semantic distortions due to the inpainting and the stochastic nature of this type of oracle. To mitigate these issues, our implementation introduces two key modifications. First, we substitute the oracle with a distortion-based metric relative to the original image. Second, we employ a VAE-purification process at the patch level. The complete procedure is formalized in Algorithm 5.

References↩︎