1 Introduction↩︎

The remarkable performance of deep neural networks has led to a widespread increase in demand for their adoption in various domains ranging from image classification and  [1], face recognition [2] to malware detection [3]–[5]. However, deep neural networks continue to be vulnerable to a range of threats, including poisoning [6]–[8], model extraction [9], and adversarial example attacks [10]–[19], raising serious concerns regarding their safe deployment. Adversarial examples mislead the models by adding tiny perturbations to an input, causing misclassification by the target model. For example, in a classification task, adversarial examples can cause the label assigned by the model to change to an incorrect one by applying a carefully crafted perturbation to an input image, while making sure that the new adversarial input is highly similar to the original one to the human eye.

Adversarial examples were initially introduced in a white-box scenario [10], [11] in which the adversary has full access to the model’s architecture and parameters. Another scenario is the black-box threat model [12]–[14], [16], a more realistic threat model, where the attacker can only observe the model outputs. Even in the restrictive conditions of black-box assumption, many powerful adversarial example attacks have been proposed, such as  [12], [14], that can generate adversarial examples with only dozens of queries, a modest cost for any capable adversary. The dangers of adversarial attacks have led to a plethora of work on defenses against these attacks [10], [11], [20]–[25]. A significant amount of work has been dedicated to improving model robustness using adversarial training (AT) [10], [11], [20], [26]–[28]. However, it has been shown that the improvement in adversarial robustness provided by AT comes with a drop in the model accuracy. Furthermore, the additional overhead of adversarial training makes AT a less practical solution. Ensemble methods [22], [29] are another defensive approach against adversarial examples. In this approach, the defender aggregates the predictions of an ensemble of ML models when responding to user queries. A critical disadvantage of ensemble methods is that their maintenance, storage, computation, and deployment costs increase with the number of models in the ensemble.

As another approach to defeat query-based black-box attacks, several proposals have been made to leverage randomization for defense [23], [25], [30]–[33]. Notably, it has been demonstrated that simply adding random noise to the input is effective against black-box score-based attacks [25] at the cost of a slight decrease in the model accuracy; this is a common issue with other proposed randomization-based defenses. The methods often achieve better robustness at the cost of degrading the model accuracy.

This work presents two key contributions. First, we propose a plug-and-play, zero-effort, and easily deployable defense against black-box score-based attacks, which preserves model accuracy while significantly reducing the effectiveness of state-of-the-art black-box score-based attacks. Second, we introduce the Pendulum attack, an adaptive attack against AAA [34], a state-of-the-art (SOTA) non-randomized defense. AAA modifies output logits to reshape the attacker’s objective function into a sine-like or linear function with an opposing slope to the original loss, while also optimizing for reduced calibration error.

Our proposed defense, Random Logit Scaling (RLS), confuses black-box attackers by outputting confidence scores resulting from randomly scaled logits. We show that our defense substantially reduces the success rate of SOTA attacks compared to state-of-the-art randomization-based defenses and also AAA by conducting experiments on CIFAR-10 and ImageNet datasets and four image classifiers; notably, our defense improves robustness against Square attack [14] by up to  80%. Moreover, we show that, unlike the SOTA randomization-based defenses, our defense maintains accuracy and minimizes distortion to the confidence scores at the same time. RLS allows the deployment of test-time defenses without the additional overhead for training, preserves model accuracy, and therefore eliminates the trade-off between robustness and performance.

The contributions of this work can be summarized as follows:

• We propose RLS, a lightweight, zero-effort defensive method against black-box score-based adversarial example attacks, and provide an analysis of the effect of logit-scaling on black-box score-based optimizations that attackers use, showing how RLS makes it difficult for the attackers to tackle the optimizations and how it can preserve the model accuracy.

• We show that RLS significantly reduces the attack success rate of black-box score-based attacks through experiments conducted on ImageNet and CIFAR-10 datasets, evaluating \(\ell_\infty\), \(\ell_2\), and also \(\ell_0\) sparse attacks. In these experiments, we use six SOTA attacks and compare RLS with AAA and three SOTA randomization-based defenses. RLS shows improved robustness over the baselines while maintaining accuracy and disturbing confidence scores less than the baseline randomization-based defenses. Additionally, we use Expectation over Transformations (EOT) as an adaptive attack against the randomization-based defenses, showing that RLS outperforms other randomization-based defenses against the adaptive attacker and significantly increases the costs for considered adaptive attacker scenarios.

• We propose a novel adaptive attack targeting AAA [34], a SOTA non-randomized defense against score-based attacks, demonstrating how randomized defenses like RLS can provide greater advantages for defenders.

2 Related Work↩︎

This section outlines the related work in adversarial example attacks and defense research. We discuss the related work under two broad topics, namely, query-based attacks and defense methods.

2.1 Query-based attacks↩︎

Query-based adversarial attacks repeatedly query the victim model and adjust the adversarial perturbation based on the model output to craft adversarial examples. These attacks are either score-based [12], [14], [35] or decision-based [36]–[38]. Score-based attacks assume an attacker has access to all or some elements of the probability vector, contrary to the decision-based attacks where only the predicted labels are available. Lacking access to score information, decision-based attacks are often more challenging to conduct, thus increasing the number of queries required to create adversarial examples and raising the cost of the attack. We only focus on defending score-based attacks where powerful adversarial example attacks exist that can generate an adversarial example with only dozens of queries compared to thousands that decision-based attacks require.

Black-box score-based attacks lack access to the model parameters; therefore, they cannot compute input gradient of the loss function. Black-box attackers try to estimate the gradient or use a proxy of the gradient. [39] proposed a Zeroth Order Optimization (ZOO) method for gradient estimation using the model outputs and then generate an adversarial image using the estimated gradient. Borrowing from conventional derivative-free optimization research, Natural Evolution Strategies (NES) [13] as another approach to estimate the gradient. Bandit [16] uses the similarity of gradients from one step to the next in an iterative attack, and the similarity between neighboring pixels. Bandit exploits these correlations as prior information for more efficient gradient estimation.  [12] proposed SignHunter which improves attack efficiency by eliminating the gradient estimation and only relying on the gradient sign. [14] proposed a very powerful and query-efficient adversarial attack that uses square-shaped adversarial noise to solve the black-box optimization using random search. Square attack demonstrates a high success rate while requiring only a few dozen queries. BruSLe [35] is a SOTA sparse score-based attack that employs a Bayesian framework and a learnable mask to identify sparse perturbations under an \(\ell_0\) constraint.

2.2 Defense Methods↩︎

Numerous efforts have been made to mitigate the effectiveness adversarial example attacks. One category is adversarial training, where DNNs are trained on both clean and adversarial samples [26]–[28], [40]. Although adversarial training has been proven effective to some extent, it has several limitations, including its high computational cost, reduced accuracy for benign samples, and the risk of overfitting to adversarial examples. Ensemble methods [22], [29] aggregate the prediction of an ensemble of models in their response but increase the cost of storage and computation. [34] introduce AAA and propose optimizing output logits so that the output loss takes the shape of a target curve (specifically, a sine function or a linear function with a slope opposite to the true loss function’s slope) to confound the attacker. We show how the deterministic nature of AAA can be used by attackers to evade this defense by proposing a simple adaptive attack against AAA-sine. Additionally, AAA introduces additional overhead due to its logit optimization process, which can be problematic in applications that rely on fast and real-time predictions, e.g., self-driving cars speeding on highways.

Randomization-based defenses introduce randomness into the model predictions by randomizing inputs [25], outputs [30], parameters [32], [33], or the intermediate output features [23] of the model. These methods improve robustness against adversarial attacks, but they can have a negative effect on the model performance by degrading its accuracy. Random Noise Defense [25] proposes randomly adding a small Gaussian noise to the inputs. Boundary defense [30] proposes adding a random noise to logits. [23] investigate the effect of adding small random noise to the output of intermediate layers of neural networks and concludes that adding a small noise to the outputs of the penultimate layer improves robustness against adversarial example attacks.

Existing defensive methods involve significant trade-offs, either in terms of computation or the model’s accuracy. One of the key challenges tackled by our proposed method is to balance robustness against attacks and its performance. In other words, our lightweight method can defend against score-based attacks without affecting the model’s accuracy and performance.

3 Preliminaries↩︎

This section discusses some crucial preliminary concepts related to adversarial example attacks. Section §3.1 discusses adversarial examples and their various categories, Sections §3.1.1 and §3.1.2 introduce two popular white-box adversarial example attacks, and Sections §3.2.1 and §3.2.2 discuss black-box score-based and decision-based attacks, respectively.

3.1 Adversarial Examples↩︎

An adversarial example \(x_{adv}\) results from carefully perturbing a sample input such as \(x\), where \(x_{adv}\) leads to an incorrect prediction. In other words, for a model \(f_w\), where \(w\) denotes the model parameters, an adversarial example \(x_{adv}\) can be defined as \[\begin{align} & x_{adv} = x + \delta \\ \text{s.t.} \quad & \bar{f}_w(x_{adv}) \neq \bar{f}_w(x) \\ \end{align}\] where \(\bar{f}_w(x)\) denotes the predicted label for input \(x\) and \(\delta\) represents the carefully crafted perturbation by the attacker.
Untargeted and Targeted Attacks: Adversarial example attacks are either untargeted or targeted. For targeted attacks, an adversarial example \(x_{adv}\) is created for an original sample \(x\) with label \(y\), such that \(x_{adv}\) is classified as a specific class \({y_{trg}}\) where \({y_{trg}} \neq y\). In the untargeted setting, however, the attacker’s only goal is that \(x_{adv}\) is classified as \(y_{adv}\) where \(y_{adv}\) can be any class different from the correct one: \(\bar{f}_w(x_{adv}) \neq \bar{f}_w(x)\).

3.1.1 Fast Gradient Sign Method (FGSM)↩︎

FGSM [10] creates the \(x_{adv}\) for a sample \(x\) with true label \(y\) by moving in the input gradient direction \[x_{adv} = x + \alpha \cdot \text{sign}(\nabla_x(\ell(f_w(x), y)))\] where \(\ell\) is the loss function (e.g. cross entropy) , \(\alpha \geq 0\) controls the magnitude of the step, \(\text{sign}\) denotes the sign function, and \(\nabla_x(\ell(f_w(x), y))\) represents the input gradient of the loss function. In other words, the idea behind FGSM is to use this gradient to craft an adversarial sample \(x_{adv}\) that maximizes the loss value for the true label \(y\), causing the model \(f_w\) to misclassify \(x_{adv}\).

3.1.2 Projected Gradient Descent↩︎

Madry et al. [11] generalized FGSM by introducing an iterative adversarial example generation method named Projected Gradient Descent (PGD) formulated as \[x_{t+1}^{\prime} = \text{PGD}_{x, \epsilon} \left( x_{t}^{\prime} + \alpha \cdot \text{sign}\left( \nabla_{x} \ell(f_w( x_{t}^{\prime}), y) \right) \right)\] where \(x_{t}^{\prime}\) denotes the adversarial example at iteration \(t\), \(\text{PGD}_{x, \epsilon}\) denotes the projection function within the \(\epsilon\)-ball of \(x\), it ensures that at each step \(||x - x^{\prime}_t||_p \leq \epsilon\).

One benefit of PGD is that its iterative nature and projection mechanism lead to more refined perturbations and prevent significant changes to the resulting adversarial example by keeping the intermediate samples within the \(\epsilon\)-ball around the original \(x\).

3.2 Black-box Adversarial Example Attacks↩︎

While PGD and FGSM assume white-box access, in black-box settings the attacker only has an oracle access, meaning the attacker can only observe the output for any query \(q\) sent to the model. This lack of access to the parameters prevents black-box attacks from computing the input gradient. However, black-box attacks use alternative methods to overcome this challenge. Black-box attacks are further categorized as either score-based or decision-based attacks, as discussed in §3.2.1 and §3.2.2.

3.2.1 Black-Box Score-Based Attacks↩︎

Score-based attacks assume access to confidence scores [13], [14], [16], [41]. Here, we discuss two common approaches to score-based attacks.
Zero-Order Optimization: An alternative approach to direct gradient computation is to estimate the gradient of the adversary’s objective function \(\mathcal{J}\) using the output values of model \(f\) for an input \(x\) and slightly modified versions of \(x\) [13], [16]. This gradient estimator can be formulated as \[g(x) = \frac{\mathcal{J}(f(x+\mu u))-\mathcal{J}(f(x))}{\mu}u\] in which \(u\) is drawn from the Normal distribution \(\mathcal{N}(0,I)\) and \(\mu \geq 0\) determines the size of the perturbation.
Random Search: An alternative approach to gradient estimation is using search-based methods [14], [17]. These methods iteratively sample a candidate direction \(u\) from a pre-defined distribution, apply it to the sample \(x\), and test the effect of moving the sample in the direction of \(u\) by computing \(h(x)\) defined as \[h(x) = \mathcal{J}(f(x + \mu u)) - \mathcal{J}(f(x))\] in which \(\mu \geq 0\) is step magnitude. At each iteration, search-based methods replace \(x\) by \(x + \mu u\) if \(h(x) \leq 0\), i.e., if \(x + \mu u\) causes the prediction score of the correct class to decrease.

3.2.2 Black-Box Decision-Based Attacks↩︎

Decision-based black-box attacks differ from score-based attacks in that they assume the model \(f\) outputs only the final prediction [37], [38]. In other words, it only outputs the label of the top scoring class. Decision-based attacks usually start with a significantly perturbed and misclassified sample \(x_{adv}\), then aim to reduce the dissimilarity between \(x_{adv}\) and the original \(x\) while ensuring that the resulting sample is still misclassified.

4 Attacking AAA↩︎

[34] proposed Adversarial Attack on Attackers (AAA) for defense against black-box score-based attacks. AAA optimizes output logits such that the objective loss function computed by attackers is transformed into a sine-like shape (AAA-sine) or a linear shape with an opposite slope to the true loss. It was shown that AAA-linear is vulnerable to a simple adaptive attack [34] (the attacker can simply estimate the gradient and step in the opposite direction or search for the opposite direction in search-based attacks). However, here we propose an effective adaptive attack against AAA-sine to show that AAA is vulnerable to adaptive attackers.

We propose Pendulum attack, an adaptive attack against AAA-sine that repeatedly changes the direction the attacker follows. Both random-search and gradient-based attacks generally follow the direction that minimizes the loss, hence, AAA-sine misleads attackers by reversing the gradient direction in some parts of the loss curve, effectively trapping them in local minima. Our Pendulum adaptive attack works as follows: when the attacker fails to find a direction that reduces the loss for \(k\) attack iterations, they shift to the opposite direction, one that increases the loss. After several iterations, however, the attacker may become stuck at a local maximum of the sine-like loss curve. To overcome this, after another \(k\) iterations with no success in moving up the loss curve, the attacker again changes direction. By iteratively switching directions upon \(k\) unsuccessful consecutive attack iterations, the attacker will be able to climb up the local minima in the loss curve that AAA-sine generates and climb back down from local maxima again when it fails to find an upward direction for \(k\) attack iterations. Our results show that the attack success rate of attacks incorporating the adaptive Pendulum attack improves by more than 35% (as detailed in Section 6.2).

5 Random Logit Scaling↩︎

In this section, we propose Random Logit Scaling (RLS) as a defense against black-box score-based adversarial example attacks. We highlight that since RLS preserves predicted label and only perturbs confidence scores, it is not effective against decision-based attacks.

5.1 Objective Functions of Black-Box Score-Based Attacks↩︎

The form of the adversary’s objective function \(\mathcal{J}\) varies depending on whether the attack is targeted or untargeted. In the targeted adversarial example attacks, the attacker aims at generating a perturbed sample \(x^\prime = x + \delta\) from an original sample \(x\) with true label \(y\) such that \(x^\prime\) is classified as \(t\) where \(t \neq y\). The optimization in the targeted scenario can be formulated as \[\label{eq:targeted} \operatorname*{argmin}_{\delta} \max_{k \neq t} f_{k}(x + \delta) - f_{t}(x + \delta)\tag{1}\] where \(f_{i}(x)\) denotes the score of class \(i\) for input \(x\).

In the untargeted setting, the attacker only wishes to generate a perturbed sample \(x^\prime\) such that it is not classified as the true class \(y\). In other words, the attacker only aims for \(x^\prime\) to be misclassified as any class other than \(y\) \[\label{eq:untargeted} \operatorname*{argmin}_{\delta} f_{y}(x + \delta) - \max_{k \neq y} f_{k}(x + \delta)\tag{2}\]

In both classes of score-based attacks discussed in Section 3.2.1, namely search-based and gradient estimation-based attacks, the attacker essentially modifies output vectors \(f(x^\prime)\) by applying carefully crafted perturbations to \(x^\prime\) and using \[\begin{align} \mathcal{J}(x') = &\max_{k \neq t} f_{k}(x^\prime) - f_{t}(x^\prime) \label{eq:targeted95value952} \end{align}\tag{3}\] for targeted attacks and \[\begin{align} \mathcal{J}(x') = &f_{y}(x^\prime) - \max_{k \neq y} f_{k}(x^\prime) \label{eq:untargeted95value952} \end{align}\tag{4}\] for untargeted attacks to guide the minimization objectives in Eq. (1 ) and Eq. (2 ), respectively.

We highlight that the objective functions in Eq. (1 ) to (4 ) are representative margin loss objectives rather than an exhaustive description of all score-based attacks. Generally speaking, score-based attacks may use surrogate losses or alternative optimization procedures that essentially pursue the same goal. However, RLS generalizes across score-based attacks because it corrupts the score information on which both exact and surrogate objectives rely.

5.2 The Method Overview↩︎

The intuition behind our proposed defense is that by modifying \(f(x^\prime)\) in a deceptive way, the attacker will not be able to tackle either of the optimizations Eq. (1 ) or Eq. (2 ). More concretely, a defender could blind an attacker by not producing the true logit values, resulting in falsified confidence scores. Outputting a greater or smaller logit value for each class \(i\) leads to a falsified score \(f_i(x)\) and a false value for the objective functions Eq. (1 ) and Eq. (2 ). In other words, masking the scores in a way that falsifies the gaps in Eq. (3 ) and Eq. (4 ) undermines an attacker’s effort to minimize the objective values. However, this masking should be done with care since arbitrary modification of logits could lower the model accuracy as the value of each logit directly reflects the ranking of the respective class. A possible approach is to scale output logits by a positive factor \(m\). Scaling logits with a positive value can falsify the values in Eq. (3 ) and Eq. (4 ) while preserving the ranking of output classes as explained in Sections 5.2.1 and Appendix 10.

5.2.1 The Effect of Logit Scaling↩︎

Using different values of \(m\) to scale the output logits, results in large fluctuations in the value of the objective functions used by the attacker, hence confusing the attacker and making it difficult for both gradient-based and search-based black-box attacks to tackle the optimization problem:

Untargeted Attacks Eq. (4 ): As long as the untargeted attack is unsuccessful (as long as Eq. (4 ) is non-negative), \(\max_{k \neq y} f_{k}(x^\prime)\) represents the runner-up class score, and the top-scoring class is the true class (\(f_y(x^\prime)\)). Hence, Eq. (4 ) effectively subtracts the runner-up class score from the top-scoring class score, i.e., it uses the gap between the two classes to guide the optimization in Eq. (2 ). However, as the analysis in Appendix 10 and Appendix 11 show, scaling the logits with the random coefficient \(m\) can cause a falsified lower or higher value for the scores and the gap between the two classes used in Eq. (4 ).

Targeted Attacks Eq. (3 ): Scaling logits by a positive \(m\) affects Eq. (3 ) similarly to the untargeted attacks. As long as the targeted attack is not successful, \(\max_{k \neq t} f_{k}(x^\prime)\) represents the top-scoring class, and the target class score \(f_t(x)\) is a lower-ranked class. As we show in Appendix 10, the gap between any two classes is affected by the fluctuations caused by logit scaling in a similar manner.

We investigate the score fluctuations of any two classes in Appendix 10 and also the fluctuations to the top-scoring class in Appendix 11. As a visual example, Figure 1 demonstrates how scaling the logits with different values of \(m\) affects the probability scores of the two top-ranking classes. Here, both \(a\) and \(b\) are positive values with \(a < b\). As \(m\) moves toward \(b\), the gap between the two classes increases. Generally, when a larger \(m\) is selected, i.e., when \(m > 1\), the absolute value of all the class logits is increased. As a result, the confidence score of the true class increases. On the other hand, when \(m\) approaches zero, the probability vector will get smoothed out, which causes the probability of the top-scoring class to decrease while the lower-scoring classes will generally have higher confidence scores but still lower than the original top-ranked class.

5.2.2 Random Logit Scaling↩︎

We propose Random Logit Scaling (RLS) to blind the attacker and mask the confidence scores while maintaining the model accuracy. RLS scales the logits in order to cause a false increase or decrease in the objective values utilized by an attacker, as explained in Section 5.2.1. RLS scales logits \(z\) by a coefficient \(m\) sampled from a continuous uniform distribution \(\mathcal{U}\) with a range \((a,b)\) where \(a > 0\) and \(b > 0\) \[\begin{align} z^\prime &= m \times z \\ \text{s.t.} \;\;\;m &\sim \mathcal{U}(a,b) \end{align}\]

We found uniform distribution to work surprisingly well, outperforming other randomization-based defenses in means of reducing the attack success rate, preserving the model accuracy, and minimizing distortion to the confidence score. However, we present a brief ablation study on the selection of the probability distribution for RLS and its parameter values in Appendix 12.

RLS Effect on Gradient Estimation and Random Search: The variations to the confidence scores, caused by sampling a different \(m\) each time the model is queried, mask the true value of \(f(x^\prime)\). Therefore, when the attacker attempts to estimate the gradient of \(f(x^\prime)\), the computations will be based on falsified scores, resulting in a falsified estimate of the true gradient. Similarly, when the attacker uses search-based methods, such as random search used by Square [14] attack, a query with a small (large) \(m\) outputs a fabricated (high) low score for \(h(x)\), misleading the attacker to select (skip) the query for the next stage of the attack, as explained in Section [subsub:random95search].

5.3 Threat Model↩︎

We consider a black-box scenario in which the model is deployed as a service and users have only oracle access to the model. Specifically, the model returns the scaled logits produced by the defense. We assume that the adversary is aware of the deployed defense and its implementation details; however, the scaling factor m is not known. The attacker is allowed to query the model adaptively and repeatedly on the same input and is subject to no API restrictions other than a query budget of 10,000 queries, which is used throughout all of our evaluations.

6 Experiments↩︎

In this section, we evaluate and compare the performance and robustness of RLS against various SOTA attacks and defenses.

6.1 Experimental Setup↩︎

Six SOTA attacks are used in the experiments, namely NES [13], Bandit [16], Square [14], SignHunter [12], and ZO-signSGD [19]. Additionally, we use BruSLe attack [35] for evaluations in \(\ell_0\) space (as detailed in Appendix 14.2). The exact settings and parameter values for each attack are outlined in Appendix 9. We compare the results of our defense with that of Randomized Featured Defense [23] (RFD), Random Noise Defense (iRND) [25], Boundary Defense (oRND) [30], and AAA. The exact settings for all the defense methods is outlined in Appendix 9.

Our experiments use CIFAR-10 [42] and ImageNet [1] as benchmark datasets. For the experiments on ImageNet, we use Pytorch Torchvision’s ResNet-50 with its pre-trained weights. For CIFAR-10, we use VGG-16, ResNet-18, and WideResNet-28-10. We train them and achieve accuracy scores of \(91.80\%\), \(95.53\%\), and \(96.20\%\), respectively. We use a randomly selected set of 1,000 test samples for conducting experiments on CIFAR-10 (unless specified otherwise). For ImageNet we use 1,000 samples from the validation set (same set of samples as in iRND [25]). We conduct our experiments over \(\ell_\infty\), \(\ell_2\), and \(\ell_0\) norm spaces. For \(\ell_\infty\) we use a perturbation budget of \(\epsilon = 0.05\) (\(\epsilon = 0.1\) for evaluations in Appendix 14.3). Settings for \(\ell_2\) and \(\ell_0\) attacks are respectively detailed in Appendices 14.1 and 14.2

6.2 Results of the Pendulum Adaptive Attack Against AAA↩︎

Following  [34] that evaluated their adaptive attack against AAA-linear by incorporating it into Square attack [14], we incorporate the Pendulum attack into Square to adaptively attack AAA-sine.

We report the attack success rate and the average number of queries of successfully crafted adversarial examples for our adaptive attack at values of \(k=5\) and \(k=10\) against AAA-sine in Table 1. Our adaptive attack is referred to as Pen-Square-\(k\). The evaluations use ResNet-18 and the randomly selected 1,000 test samples from CIFAR-10. We use the \(\ell_\infty\) with a perturbation budget of \(\epsilon =0.05\).

The results highlight the vulnerability of AAA to our proposed adaptive attack. Nonetheless, we provide a comparison between RLS and AAA in Appendix  14.4 demonstrating that RLS outperforms AAA even in scenarios where no adaptive attack is applied to AAA.

6.3 Confidence Score Distortion and Calibration Error↩︎

Randomized-defenses cause distortion to model outputs and scores. Scaling logits distorts the confidence scores too, which can be harmful, especially in applications where the magnitude of the prediction’s confidence score is crucial—scaling logits up results in a misleadingly higher confidence score for the top-scoring class while shrinking logits results in a lower score for the class (see Appendix 11). This can pose a challenge in applications that rely on the accuracy of the gap between the model’s top and runner-up classes’ confidence scores (for instance, in out-of-distribution detection [43], out-of-distribution samples typically exhibit a smaller gap and in medical applications [44], it is essential to rely on outputs with high confidence for decision-making). We test the degree of confidence score distortion in the presence of our defense and compare our results to Randomized Feature Defense (RFD) [23], Random Noise Defense [25] (iRND), and boundary defense [30] (oRND).

In contrast to other randomized defenses, our method keeps the predicted class unchanged. This means that the model accuracy does not decrease in the presence of RLS. On the other hand, iRND, RFD, and oRND can cause the defended model to misclassify some input samples, hence lowering the model accuracy.

Table ¿tbl:table:distortion? reports the distortion of true probability values in the presence of our defense, iRND, RFD, and oRND. We used all the 10,000 test samples of CIFAR-10 to conduct this experiment. We first collect the original probability vectors for all the samples for an undefended ResNet-18 model. Then, we implement each defense and collect the probability vectors for the defended model. Finally, we compute the \(L_2\) norm of the difference between the two vectors for each data sample and report the maximum, average, and minimum taken over all the differences vectors.

The results show that RLS causes a lower level of distortion on average compared to iRND, oRND, and RFD for \(m \in (0.5, 10)\), and even when sampling from a large range of possible coefficients, such as \((0.5, 1000)\), RLS outperforms both iRND and RFD but distorts the score slightly more than oRND. The maximum distortion that our method causes is also lower than that of other defenses since other defenses can cause a sample to be misclassified.

Additionally, in Table [tab:table953], we evaluate the expected calibration error of RLS and compare it with other defensive methods on ResNet-18. Our evaluations show that RLS (0.5, 10) and RLS (0.5, 1000) cause an average of 1.3% and 1.8% increase in the ECE, respectively. This means RLS increases ECE more than RFD, AAA, and oRND but less than iRND.

Due to the confidence score distortion introduced by RLS, it creates a trade-off between robustness and confidence score preservation in applications that heavily rely on accurate confidence estimates. RLS is therefore most appropriate in settings where preserving top-1 predictions and improving robustness are prioritized, whereas applications that depend heavily on well-calibrated confidence scores may require additional calibration techniques or a carefully selected distribution for the scaling factor.

6.4 Results of Defense against Score-based Black-box Attacks↩︎

In this section, we evaluate the resistance of RLS against five state-of-the-art black-box score-based attacks. We compare our results with Random Noise Defense [25] (iRND), Randomized Feature Defense (RFD) [23], and Boundary defense [30] (oRND). We report the performance of RLS on CIFAR-10 and ImageNet in Tables ¿tbl:table:cifar10? and  ¿tbl:table:imagenet?, respectively. Additional results for attacks in the \(\ell_2\) norm space, SOTA sparse attack in the \(\ell_0\) space [35], evaluation for a larger perturbation budget of \(\epsilon=0.1\) in the \(\ell_\infty\) space, and comparison with AAA are respectively available in Appendices 14.1, 14.2, 14.3, and  14.4. Additional results for attack success across different query budgets of 1000, 2000, 5000, and 10000 is also avaialble in Appendix 14.6.

As reported in Table ¿tbl:table:cifar10?, RLS drastically reduces the attack success rate in almost all cases while maintaining the model accuracy. For NES [13], RLS outperforms SOTA randomization-based methods by at least 10%, except for WideResNet-28-10, where iRND works slightly better. However, iRND decreases the model accuracy by 3.95%, while RLS preserves the accuracy. For Bandit [16], Square [14], SignHunter [12], and ZO-signSGD [19], RLS consistently outperforms iRND, RFD, and oRND by at least 10% and up to more than 90%. Notably, RLS reduces the attack success rate for the powerful Square attack to only around 21% on VGG-16 and around 40% for both WideResNet-28-10 and ResNet-18.

We also evaluated our defense on a pre-trained ResNet-50 on ImageNet. The results are reported in Table ¿tbl:table:imagenet?. The results reveal that RLS achieves the best and the second-best performance with respect to attack success rate in all the cases except for the Bandit attack, where iRND achieved the second-best performance but at the cost of a 0.9% drop in the model accuracy.

Figure 2 visualizes the effect of RLS on an attacker and how our defense causes the objective function values computed by an attacker to fluctuate. Figure 2a illustrates the normalized loss values for a randomly selected CIFAR-10 image in a few dozen iterations of Square attack. The dashed plot indicates the normalized loss values in the presence of our defense, whereas the solid one plots the values for an undefended model. The plot in Figure 2b, indicates the sampled \(m\) value for each query iteration.

6.5 Adaptive Attack against RLS↩︎

To evaluate the robustness of our defense against adaptive attacks, we use Expectation over Transformation (EOT) [45] attack. EOT uses \(k\) queries for each sample at each attack iteration and takes an average over the outputs to cancel out the randomness introduced by randomized defenses. We report our results in Table ¿tbl:table:adaptive? for \(k=1\), \(k=5\), and \(k=10\) while additional results for \(k=25\), \(k=50\), and \(k=100\) are presented in Appendix 14.5. We use Square attack [14] and limit the number of attack iterations to 1,000 (5,000 queries for \(k=5\) and 10,000 queries for \(k=10\)) and use a randomly selected set of 2,000 test samples from CIFAR-10 to attack our trained ResNet-18 model. As the results show, our method can withstand the adaptive attack better. While the adaptive attack can achieve almost 80% success rate against iRND and 70% against oRND and RFD, our method can still decrease the attacker’s success rate by at least around 45%.

The results also show that when the defender uses a wider range of \(m\) values, the attacker’s success rate decreases. This is because as the range of possible \(m\) values increases, the attacker has to use more queries when taking the average to estimate the true confidence scores. In other words, the attacker has to increase \(k\) when the defender uses a wider range of \(m\) values. In Appendix [apx:adaptive95loss95viss], we have plotted correlation between true loss and loss under RLS for \(k=5, 10, 20\), and \(100\).

7 Discussion↩︎

Here we discuss the limitations of RLS. First of all, RLS preservers the predicted class and primarily only perturbs the confidence scores, therefore it has limited effectiveness against decision-based attacks that only rely on the model’s final prediction. Nevertheless, existing state-of-the-art decision-based attacks require substantially more queries than score-based attacks. Overall, RLS should be viewed as a defense mainly against score-based adversaries rather than a comprehensive defense against all black-box attacks.

Secondly, RLS is inherently a randomized defense and provides no certified robustness guarantees. Therefore, a sufficiently capable attacker with a large enough number of repeated observations for the same input can potentially eliminate the effect of the randomness. However, this would substantially increase the number of required queries and the attack cost, while making the detection of adversarial attempts easier.

8 Conclusion↩︎

In this work, we proposed Random Logit Scaling (RLS), a randomization-based, lightweight, and powerful defense against black-box score-based adversarial examples attacks and proposed an effective adaptive attack against AAA. Through extensive experiments on two benchmark datasets, six SOTA attacks, four state-of-the-art defenses, and four image classifiers, we showed that RLS preserves the model accuracy, minimizes distortion to the confidence scores, and outperforms other defenses against black-box score-based attacks. Furthermore, we showed that RLS is more robust against the EOT adaptive attack. We also analyzed how scaling logits affects a model and black-box adversaries and how it can be an effective defense against adversarial example attacks. The significant resistance of RLS against the most powerful black-box attacks, combined with its zero-effort implementation and preservation of the model accuracy, makes it a very powerful defense against black-box adversarial attacks.

9 Attack and Defense Configurations↩︎

For the Square [14], Bandit [16], NES [13], ZO-signSGD [19], and SignHunter [12] attacks, we use the implementations provided by  [46] in all of our experiments. The exact values of hyperparameters for all of the attacks are reported in Table ¿tbl:table:hyperparameters?. Hyperparameter values for BruSLiAttack [35] are also outlined in the table.

For the defensive methods used in all of our experiments, we use the configurations as outlined in the following. For iRND [25] we use the two original configurations of \(\nu = 0.01\) and \(\nu =0.02\). For oRND [30] we use \(c = 1\). For AAA [34] we use the configurations provided by the official implementation, specifically, we use 4 for attractor interval, 5 for regularization weight, learning rate of 0.1, and 100 iterations for optimization. However, authors of RFD [23] do not report the specific hyper-parameter values (\(\sigma\)) but instead, report the average accuracy drop of the configurations they use. Interestingly, this is because the \(\sigma\) value used by RFD has to be tuned for each model specifically. We conducted a search for each of our models and selected the best performing \(\sigma\) that provided a higher robustness against the majority of the attacks. We use \(\sigma = 0.35\), \(\sigma = 2.5\), \(\sigma = 0.7\), and \(\sigma = 3.0\) respectively for VGG-16, ResNet-18, WideResNet-28-10, and ResNet-50.

10 The RLS Formulation↩︎

We provide a formulation of RLS and how it affects the vector of confidence scores and the attacker’s efforts to generate adversarial examples. Specifically, we show how different values of \(m\) causes the gap between any two class scores to fluctuate, hence having a direct impact on the values in Eq. (3 ) and Eq. (4 ) of Section 5.1.

Notation. For any arbitrary input sample such as \(x\), \(Z\) denotes the original logit vector and \(z_i \in Z\) represents the vector’s \(i^{\text{th}}\) element. \(m \sim \mathcal{D}\) represents a randomly selected coefficient \(m\) from the distribution \(\mathcal{D}\). \(p_i \in P\) represents the true probability vector’s \(i^{\text{th}}\) element, while \(p^\prime_i \in P^\prime\) represent the \(i^{\text{th}}\) element of the probability vector resulting from multiplying \(Z\) by \(m\). Finally, \(S\) and \(S^\prime\) represent the normalization factor of the softmax function for the true and scaled logits, respectively.

Assuming \(p_i \in P\) and \(p_j \in P\) are the probability scores of two randomly selected classes \(i\) and \(j\), we have \[\begin{align} p_i = \frac{e^{z_i}}{S}, \qquad p_j = \frac{e^{z_j}}{S} \end{align}\] assuming \(i\) is the class with a higher confidence score, since \(0 < p_i < 1\) for any \(i\), we can write \[\label{eq:original95probs} \frac{p_i}{p_j} = \frac{e^{z_i}}{e^{z_j}} > 1\tag{5}\] After multiplying the logits by \(m\), the resulting class probabilities are \[\begin{align} p^\prime_i = \frac{e^{mz_i}}{S^\prime}, \qquad p^\prime_j = \frac{e^{mz_j}}{S^\prime} \end{align}\] by dividing the probabilities in the same manner, we have \[\label{eq:defense95probs} \frac{p^\prime_i}{p^\prime_j} = \frac{e^{mz_i}}{e^{mz_j}} = (\frac{e^{z_i}}{e^{z_j}})^m\tag{6}\] by combining Eq. (5 ) and Eq. (6 ), we have \[\label{eq:combined} \frac{p^\prime_i}{p^\prime_j} = (\frac{p_i}{p_j})^m\tag{7}\] based on Eq. (5 ) and Eq. (7 ), we can conclude that for \(m > 0\) \[\begin{align} \frac{p^\prime_i}{p^\prime_j} > 1 \\ p^\prime_i > p^\prime_j \end{align}\] therefore, the ranking of the classes is preserved. Additionally,

1. for \(0 < m < 1\), the value of \(\frac{p^\prime_i}{p^\prime_j}\) decreases towards one

2. for \(m > 1\) the value of \(\frac{p^\prime_i}{p^\prime_j}\) gets larger

3. for \(m < 0\), the value of \(\frac{p^\prime_i}{p^\prime_j}\) moves below \(1\), which means \(p^\prime_j > p^\prime_i\), hence the accuracy can drop

Since our method only considers positive values for \(m\), we only focus on the first two cases. As \(m\) grows larger, \({{p^\prime_i}/{p^\prime_j}}\) moves toward infinity, intuitively, causing \(p_i^\prime\) and \(p_j^\prime\) to get farther away from each other and resulting in the gap between the probabilities having a higher value than the true one. When \(m\) is below one and approaches zero, \({{p^\prime_i}/{p^\prime_j}}\) decreases towards one. This means the values of the manipulated probabilities get closer to each other. In Appendix 11, we prove that in both cases, the changes to the gap affect the top-class probability, hampering the effectiveness of many of the gradient estimation and search-based methods, as explained in Sections 5.2.1 and 5.2.2.

11 The Proof of Fluctuations to The Top-Scoring Class↩︎

We assume \(t\) represents the top class for a classification task with \(n\) classes. We use the notations from Appendix 10. First, we have \[\label{eq:inequality} e^{z_t} > e^{z_i}, \;\;\;\forall \;\;\;i \neq t\tag{8}\] since \(m > 0\) and \(e^x > 0\) for any \(x\), we can derive \[\begin{align} e^{mz_t} > e^{mz_i}, \;\;\;\forall \;\;\;i \neq t \end{align}\] \(S\) and \(S^\prime\) are defined as follows \[\begin{align} S &= e^{z_1} + \cdots + e^{z_n} \\ S^\prime &= e^{mz_1}+ \cdots + e^{mz_n} \end{align}\] for \(t\) we have \[\begin{align} p_t = \frac{e^{z_t}}{S} \end{align}\] since \(p_i\) is positive for any \(i\), we can take the reciprocal of the equation \[\begin{align} \frac{1}{p_t} = \frac{S}{e^{z_t}} = \frac{e^{z_1} + \cdots + e^{z_t} + \cdots + e^{z_n}}{e^{z_t}} \end{align}\] in other words \[\begin{align} \frac{1}{p_t} = 1 + \sum_{\substack{i=1 \\ i \neq t}}^{n} \frac{e^{z_i}}{e^{z_t}} \end{align}\]

similarly, for \(p^\prime_t\) we have \[\begin{align} \frac{1}{p^\prime_t} = 1 + \sum_{\substack{i=1 \\ i \neq t}}^{n} (\frac{e^{z_i}}{e^{z_t}})^m \end{align}\]

Based on (8 ) \[\begin{align} \frac{e^{z_i}}{e^{z_t}} < 1, \;\;\;\forall \;\;\;i \neq t \end{align}\] which means for \(m > 1\) \[\begin{align} \sum_{\substack{i=1 \\ i \neq t}}^{n} (\frac{e^{z_i}}{e^{z_t}})^m &< \sum_{\substack{i=1 \\ i \neq t}}^{n} \frac{e^{z_i}}{e^{z_t}} \\ \frac{1}{p^\prime_t} &< \frac{1}{p_t} \\ p^\prime_t &> p_t \end{align}\] On the other hand, for \(0 < m < 1\) \[\begin{align} \sum_{\substack{i=1 \\ i \neq t}}^{n} (\frac{e^{z_i}}{e^{z_t}})^m &> \sum_{\substack{i=1 \\ i \neq t}}^{n} \frac{e^{z_i}}{e^{z_t}} \\ p^\prime_t &< p_t \end{align}\]

12 Ablation Study on RLS Defense Components↩︎

The backbone of RLS lies in two key components: the distribution probability used for sampling the coefficient \(m\) and the parameters of candidate distributions. We initially focused on the Gaussian and uniform distributions, and found that uniform distribution works surprisingly well, as demonstrated in Section 6.4, outperforming other defenses by a large margin, notably reducing the success rate of the powerful Square attack by at least 40%.

Table ¿tbl:table:ablation? summarizes the results of our experiments on the main components of RLS. We run the Square attack [14] against our trained ResNet-18 and use all the 10000 testing samples of CIFAR-10 to attack the model. We implement RLS using Gaussian and uniform distributions. The attacker’s query budget is set to 10000 queries, and the detailed settings for the Square attack can be found in Appendix 9. It should be noted that since sampling from the Gaussian distribution can result in negative values for \(m\), we have set a minimum sampling threshold of \(0.2\) for all of the experiments conducted using Gaussian distribution.

The results show that the uniform distribution outperforms the Gaussian distribution in most cases. However, Gaussian distribution’s effectiveness against adversarial example attacks is comparable to that of uniform distribution, specifically when \(\mu=1 \;\text{and} \; \sigma=100\). The results also indicate that for the uniform distribution, the attack success rate drops as the upper-bound \(b\) (lower-bound \(a\)) increases (decreases). For all our evaluations in this work, we employed the uniform distribution for RLS, leaving a more comprehensive investigation into the choice of distribution for future work.

13 Adaptive Attack Loss Correlation↩︎

Figures 3 indicates how increasing \(k\) could lead to a better correlation between the true loss and the loss under RLS. Figures 3 (a), 3 (b), and 3 (c) demonstrate the correlation between true loss and the loss value computed by an adaptive attacker for four different values of \(k\). As the graphs show, increasing \(k\) results in more correlation between the loss value computed by the adaptive attack and the true loss value of the undefended model. However, increasing \(k\) makes the attack more expensive for an attacker, i.e., the attacker has to use \(k\) queries at each iteration of the attack. For instance, when \(k = 100\), a budget of 10,000 queries is equivalent to only 500 attack iterations. Figures 3 (d), 3 (e), and 3 (f) plot normalized loss values for the true loss (black), loss in the presence of RLS with no adaptive attacker (magenta), and loss in the presence of RLS and an adaptive attacker with \(k = 100\) (green). The figures show how a large \(k\) could improve the correlation of the loss computed by an attacker with the true loss values.

14 Additional Experiments↩︎

14.1 Defense Performance against Attacks in \(\ell_2\) Space↩︎

The experiments in Section 6 are all conducted in \(\ell_\infty\) space. Here we provide additional experiments in the \(\ell_2\) space and compare the robustness of our defense with other randomized defenses. The results indicate that our defense consistently outperforms other methods in the \(\ell_2\) space too. The perturbation budget for experiments in \(\ell_2\) is set to 255 (or 1) for CIFAR-10 and 1275 (or 5) for ImageNet. We use the same samples from CIFAR-10 and ImageNet that we used for experiments in the \(\ell_\infty\) space (outlined in  6.1) and the same attack and defense settings outlined in Appendix 9. We use Bandit [16], Square [14], and Zo-signSGD [19] attacks and victim models are ResNet-18 and VGG-16 for CIFAR-10, and ResNet-50 for ImageNet. Tables 2, 3, 4 report ASR and average number of queries for ResNet-18, VGG-16, and ResNet-50, respectively.

14.2 Defense Performance against the SOTA Sparse Attack (\(\ell_0\))↩︎

We evaluate the robustness of RLS and other randomized defenses against BruSLeAttack [35], the SOTA sparse black-box score-based attack. In the evaluations, we use ResNet-18 and the same 1,000 randomly selected test samples from CIFAR-10 and also 200 random samples from the 1,000 ImageNet samples that we used in evaluations in Section 6. The hyper-parameter values that we use for BruSLeAttack follow the settings provided by the authors (outlined in Table ¿tbl:table:hyperparameters?). For evaluations on ResNet-18 (CIFAR-10), we use sparsity levels of \(1\%\) and \(2.9\%\) corresponding to 10 and 30 pixels, respectively. For evaluations on ResNet-50 (ImageNet), we use sparsity levels of \(0.2\%\) and \(0.6\%\) which respectively correspond to 100 and 300 pixels. The results are outlined in Tables 5 and 6 and show that RLS outperforms other defenses too.

14.3 Defense Performance against Larger Perturbation Size↩︎

The experiments in Section 6 all use a perturbation budget of \(\epsilon = 0.05\) or \(12.75\)/\(255\). Here we evaluate and compare the performance of RLS against other defenses under a larger perturbation budget of \(\epsilon = 25.5\)/\(255\). The results indicate that RLS consistently outperforms other defenses across a range of perturbation budgets and settings. We employ the Square [16], attack, Bandit [16] attack, and SignHunter [12] attack and report ASR and average number of queries. The evaluations are conducted on ResNet-18 and VGG-16 for CIFAR-10, and on ResNet-50 for ImageNet, with the results reported in Tables 8, 9, and 10 respectively.

14.4 Comparison of robustness between AAA and RLS↩︎

In Section 4, we proposed a novel adaptive attack against AAA and experimentally showed AAA’s vulnerability to this adaptive attack in Section 6.2. However, for completeness, we provide and compare the results on the defense robustness of AAA-sine and RLS. AAA-linear is excluded from this comparison, as it was demonstrated to be vulnerable to a simple adaptive attack in the original work by  [34]. We use the same samples and settings that we used in Tables ¿tbl:table:cifar10? and ¿tbl:table:imagenet?.

We use Square [14] and SignHunter [12] attacks, the two most efficient attacks based on our results in Tables ¿tbl:table:cifar10? and ¿tbl:table:imagenet? and report the results on ImageNet and CIFAR-10 in Table 7. Each cell in the table represents ASR and average number of queries for successfully crafted adversarial examples. The results of Table 7 demonstrate that RLS outperforms AAA-sine across all the models and against bot attacks.

14.5 Experiments with Larger \(k\) for EOT Adaptive Attack↩︎

Table ¿tbl:table:larger95adaptive? shows the results for larger \(k\) values of the EOT adaptive attack using Square Attack against ResNet-18 across our defense method and the baseline defenses. The results show that, although the overall attack success rate increases with larger \(k\) values, RLS continues to outperform the baselines, even under the extreme and computationally expensive setting of using \(k=100\) EOT queries per sample.

14.6 Attack Success Rates Across Query Budgets↩︎

Table 11 shows the robustness of RLS and other defenses across query budgets of 1000, 2000, 5000, and 10000. As shown in the table, the attack success rate generally increases with larger query budgets. However, the increase for RLS is substantially smaller than that of the baseline defenses, exhibiting only an approximately 2% increase in ASR from a query budget of 5000 to 10000. These results suggest that RLS provides genuine robustness against the attack rather than simply postponing successful attacks to higher query budgets.

References↩︎