Adversarial Attacks on Online Handwriting using Salience-based Temporal Editing
July 14, 2026
Abstract
Deep learning models for online handwriting recognition have been shown effective and are increasingly deployed in practical applications. However, their vulnerability to adversarial attacks is still a challenge. Existing adversarial methods are predominantly designed for image-based inputs and typically rely on additive spatial perturbations. When applied to online handwriting, which is inherently represented as a time series of pen trajectories, such perturbations often introduce high-frequency jitter and visibly unnatural stroke artifacts. In this work, we propose a novel adversarial attack framework for online handwriting recognition based on salience-guided temporal editing. Instead of adding noise, the proposed method generates adversarial examples by inserting and deleting points at time steps selected according to temporal salience, preserving the shape and smoothness of the original handwriting. Temporal salience is estimated using gradient-based activation mapping, which guides edits toward time steps that strongly support the original class prediction. We evaluate the proposed approach on the Unipen and CASIA-OLHWDB datasets under both white-box and one-shot black-box attack settings. Experimental results demonstrate that while conventional image-based attacks achieve strong white-box performance, they exhibit poor transferability across models. In contrast, the proposed temporal editing attack achieves stronger one-shot black-box transferability while preserving the visual structure of the handwriting. These results indicate that temporal editing is a relevant threat model for online handwriting recognition, particularly in one-shot black-box transfer settings.
1 Introduction↩︎
Online handwriting recognition differs from offline handwriting recognition in that the input is a sequence of pen-tip coordinates rather than a raster image. Modern online handwriting recognizers build on sequence models such as Recurrent Neural Networks (RNN) [1], Temporal Convolutional Neural Networks (CNN) [2], and Transformers [3], and have achieved high accuracy on this input modality [4]–[8]. However, adversarial robustness for online handwriting remains less explored than for image-based recognition models.
As these systems are deployed in real-world applications, concerns about their reliability and security have become more pronounced. In particular, adversarial attacks are a widely known weakness of neural networks [9], [10]. Adversarial attacks work by adding small perturbations to the input with the aim of causing misrecognition by the models.
However, most adversarial attacks were originally developed for image-based classifiers with perturbations constrained by pixel-wise \(L_p\) norms. For example, various gradient-based attack methods have been proposed, such as Fast Gradient Sign Method (FGSM) [11], Projected Gradient Descent (PGD) [12], and the Carlini-Wagner (CW) attack [13]. These methods are designed to degrade recognition performance while remaining imperceptible in the image domain. In comparison, attacks on time series are much less explored, and most adversarial attacks on time series are adapted from image-based methods [14], [15].
The online handwriting data considered in this study is represented as a sequence of two-dimensional pen-tip coordinates, where each time step \(t\) contains a coordinate vector \(\mathbf{x}_t \in \mathbb{R}^2\). In the image domain, small perturbations can appear as inconspicuous noise due to the texture present in natural images. However, directly applying image domain adversarial noise to time series data frequently results in abrupt temporal fluctuations and high-frequency artifacts, which create unnatural deformations in online handwriting trajectories, as illustrated in Fig. 1. This indicates that even when perturbations satisfy small \(L_p\) norms, the resulting time series may violate the kinematic smoothness and structural regularities inherent to human handwriting. In other words, the implicit assumption in the image domain that “small noise is difficult for humans to perceive” is not necessarily valid for time series data. This discrepancy highlights a fundamental limitation in current adversarial attacks on time series.
Beyond naturalness issues, most gradient-based adversarial attacks operate under a white-box assumption, requiring full access to model parameters and gradients. In practice, however, deployed handwriting recognition systems are often accessible only through prediction interfaces, making black-box adversarial attacks a more realistic threat model. Commercial handwriting recognition and authentication services typically expose only final recognition outputs, rendering white-box assumptions impractical in many real-world scenarios. Therefore, evaluating robustness under black-box conditions is essential for understanding the true vulnerability of online handwriting recognition systems.
To address these challenges, we propose Adversarial Iterative Temporal Editing (AITE), a novel adversarial attack method that generates adversarial examples through discrete editing operations in the time dimension rather than additive noise in the spatial dimensions. Specifically, AITE iteratively performs element-wise insertion and deletion on the input sequence based on the discriminative importance, or salience, of individual time steps. Salience is estimated using Gradient-weighted Class Activation Maps (Grad-CAM) [16], allowing the attack to target temporally critical regions while preserving the overall geometric structure and kinematic smoothness characteristic of natural handwriting trajectories.
The contributions of this study are summarized as follows:
We highlight the inadequacy of spatial perturbation-based adversarial noise for online handwriting.
We propose a novel adversarial attack framework based on salience-guided temporal editing.
The proposed method is evaluated under both white-box and one-shot black-box attack settings on the Unipen [17] and CASIA-OLHWDB [18] online handwritten character datasets. Experimental results demonstrate that temporal editing achieves stronger visual similarity than conventional spatial perturbation-based noise attacks.
The black-box attack settings demonstrate that the salience-based editing is effective at attacking unseen models.
Furthermore, for reproducibility, we will release the source code at GitHub1.
2 Related Work↩︎
2.1 Adversarial Attacks↩︎
Adversarial attacks were first extensively studied in the context of image classification, where small, carefully crafted perturbations can induce misclassification while remaining visually imperceptible [10]. The vulnerability of deep neural networks to adversarial perturbations was first systematically demonstrated by Szegedy et al. [19], who formulated adversarial example generation as an optimization problem that searches for minimal input perturbations capable of changing a model’s prediction. To reduce the computational cost of this approach, Goodfellow et al. [11] proposed FGSM, which approximates the optimization by applying a single gradient step in the direction of the loss. Since then, many methods, such as Basic Iterative Method (BIM) [20], PGD [12], and the CW attack [13], have improved upon gradient-based adversarial attacks through iterative updates and regularized perturbation objectives. A separate line of work targets the black-box transferability of adversarial examples between models. MI-FGSM [21] accumulates a momentum term over the iterative input gradients to stabilize the update direction, while NI-FGSM [22] extends MI-FGSM with Nesterov accelerated gradients by evaluating the gradient at a look-ahead point. TI-FGSM [23] convolves the input gradient with a smoothing kernel before each update so that perturbations become approximately translation-invariant and transfer better across architectures. While these methods form the foundation of adversarial attack research, they were all originally developed for image-based classifiers and rely on additive, pixel-wise perturbations.
2.2 Adversarial Attacks on Time Series↩︎
Compared to image recognition, research on adversarial attacks against time series data is an emerging area. Carlini et al. [24] demonstrated that hidden commands can be embedded into automatic speech recognition systems. Fawaz et al. [15] evaluated FGSM and BIM on time series classification datasets and reported the vulnerability of deep learning–based time series classifiers. Karim et al. [14] proposed an Adversarial Transformation Network (ATN) using a distilled model to act as a surrogate to mimic the behavior of classical time series classification models. In addition, there have been black-box models that have been proposed for time series classification [25]–[27]. However, these methods still apply perturbations which may not be appropriate for all time series. Furthermore, many of these methods require feedback from the attacked model [28]. Developing attack methods that take into account structures specific to time series data (e.g., temporal dependency and sequence smoothness) remains an important research challenge [29].
2.3 Adversarial Attacks on Handwriting Recognition↩︎
Adversarial attacks on handwriting recognition have mainly been studied in the context of offline handwriting recognition, where adversarial attacks on offline handwriting have been shown to be effective on offline signature verification [30]–[32] and character recognition in various languages [33]–[35].
In contrast, research on adversarial attacks against online handwriting recognition is very limited. Yamashita et al. [36] showed that spatial perturbation-based attacks work on online handwriting and proposed a defense based on time series transformations. Lopresti and Raim [37] showed that a generative attack on online handwriting is possible.
3 Adversarial Attacks↩︎
3.1 Threat Model↩︎
We consider adversarial attacks against online handwriting recognition systems whose inputs are pen-tip trajectories of handwritten characters. In this study, each character is represented as a time series \(\mathbf{X}\) with ground-truth label \(y\): \[\mathbf{X} = (\mathbf{x}_1, \dots, \mathbf{x}_t, \dots, \mathbf{x}_T),\] where \(T\) is the sequence length and \(\mathbf{x}_t \in \mathbb{R}^d\) denotes the pen-tip coordinate vector at time step \(t\). In our experiments, we use two-dimensional coordinate sequences, i.e., \(d=2\). A recognition model \(f_\theta\) with trained parameters \(\theta\) maps \(\mathbf{X}\) to class logits \(f_\theta(\mathbf{X}) \in \mathbb{R}^{C}\). The predicted class label is \[\hat{y} = \arg\max_{c} f_\theta(\mathbf{X})_c .\] The goal of an adversarial attack is to find an adversarial example \(\mathbf{X}_{\mathrm{adv}}\) such that \[\hat{y}_{\mathrm{adv}} = \arg\max_{c} f_\theta(\mathbf{X}_{\mathrm{adv}})_c, \quad \hat{y}_{\mathrm{adv}} \neq y,\] while the resulting handwriting remains visually plausible.
Adversarial attacks can be broadly categorized into white-box and black-box attacks, and we evaluate the proposed method under both settings. In the white-box setting, the attacker has full access to the model architecture and parameters \(\theta\) and the gradients of \(f_\theta(\mathbf{X})\). This allows the attacker to design \(\mathbf{X}_\mathrm{adv}\) so that it directly exploits the model’s decision boundary. In the black-box setting, the attacker has no access to the model’s parameters or gradients. This setting reflects realistic deployment conditions, as malicious attackers would not have access to the model directly.
Furthermore, as illustrated in Fig. 2, we consider one-shot transferable black-box attacks. In this setting, the adversarial attack method is permitted only a single attempt and receives no feedback from the target model. Transferability is particularly important for real-world scenarios, as deployed handwriting recognition systems typically expose only final predictions, preventing iterative probing or gradient access by an adversary [38]. Our black-box evaluation is therefore transfer-based and one-shot: adversarial samples are generated using a surrogate model and then submitted once to unseen target models. We do not consider query-based black-box attacks, where the adversary adaptively updates the input using repeated feedback from the target model.
3.2 Spatial Perturbation-based Adversarial Attacks↩︎
Spatial perturbation-based adversarial attacks typically generate adversarial examples by adding small-magnitude noise to input representations. As a representative example, FGSM [11] creates an adversarial example \(\mathbf{x}_{\mathrm{fgsm}}\) by: \[\mathbf{X}_{\mathrm{fgsm}} = \mathbf{X} + \epsilon \cdot \mathrm{sign}\!\left(\nabla_\mathbf{X} L(f_\theta(\mathbf{X}), y)\right), \label{eq:fgsm}\tag{1}\] where \(L(f_\theta(\mathbf{X}), y)\) is the loss function between the model prediction for input \(\mathbf{X}\) and the label \(y\), \(\nabla_\mathbf{X} L(f_\theta(\mathbf{X}), y)\) is the gradient of the loss with respect to the input, \(\mathrm{sign}(\cdot)\) is the element-wise sign function, and \(\epsilon\) is a hyperparameter controlling the perturbation magnitude. Iterative methods, such as BIM [20] and PGD [12], repeat this process given constraints.
4 Proposed Method↩︎
We propose a temporal editing attack that generates adversarial examples by modifying the time series in the time dimension rather than adding spatial noise. The key idea is to induce misclassification through insertion and deletion of time steps while preserving the overall shape and smoothness of the handwriting. As shown in Fig. 1, this design directly addresses the limitation of image-based adversarial perturbations, which often produce unnatural artifacts when applied to time series, including online handwriting.
4.1 Temporal Editing↩︎
In order to modify the time series in the time dimension, we define two basic editing operations, temporal insertion and temporal deletion.
4.1.1 Temporal Insertion↩︎
Given input time series \(\mathbf{X}\) with \(\mathbf{x}_t\in\mathbb{R}^d\), temporal insertion adds a new point between two adjacent points \((\mathbf{x}_i,\mathbf{x}_{i+1})\) with \(i\in\{1,\ldots,T-1\}\). Interpolation is used to preserve local continuity and the new sequence becomes: \[\label{eq:insertion} \mathbf{X}_\mathrm{ins}=(\mathbf{x}_1,\dots,\mathbf{x}_{i},\frac{\mathbf{x}_i + \mathbf{x}_{i+1}}{2},\mathbf{x}_{i+1}\dots,\mathbf{x}_T),\tag{2}\] increasing the number of time steps by one.
4.1.2 Temporal Deletion↩︎
Temporal deletion removes one time step from the sequence. Since removing an endpoint may severely distort the overall shape, we exclude endpoints from deletion candidates. Given a selected time index \(j\), deletion removes the \(j\)-th element from \(\mathbf{X}\) to form: \[\label{eq:deletion} \mathbf{X}_\mathrm{del}=(\mathbf{x}_1,\dots,\mathbf{x}_{j-1}, \mathbf{x}_{j+1},\dots,\mathbf{x}_T).\tag{3}\] The resulting \(\mathbf{X}_\mathrm{del}\) has one fewer time step than the original \(\mathbf{X}\).
4.2 Adversarial Iterative Temporal Editing (AITE)↩︎
To create adversarial examples, an iterative approach is taken. In one step of AITE, we select an insertion position \(i\) and a deletion position \(j\), and apply the edit to the current sequence to generate a new sequence. By applying insertion and deletion in equal numbers, the sequence length is always kept at \(T\) while locally modifying only the trajectory shape. This editing step is repeated until the predicted label no longer matches the true label \(y\) (attack success), or until the maximum number of iterations \(K\) is reached.
The following constraints are imposed to ensure stable editing:
In each iteration, we perform one insertion and one deletion so that the sequence length remains \(T\). This allows the generated adversarial examples to be used for any model without interpolation or similar adjustments.
The element inserted in the same iteration is not allowed to be deleted. This constraint prevents iterations without changes.
Endpoints are excluded from deletion to prevent excessive distortion of the trajectory.
4.3 Salience Estimation for Temporal Editing↩︎
In order to select the locations of the inserted and deleted elements, we estimate temporal salience using Grad-CAM [16]. Grad-CAM is a post-hoc explainability method that highlights regions of an input that are most influential for a model’s prediction by leveraging gradient information from intermediate feature representations. Originally proposed for images, in this work, we adapt Grad-CAM to be used for time series by estimating the element-wise salience instead of pixel-wise salience.
Let \(\mathbf{A}\in\mathbb{R}^{M\times T'}\) denote an intermediate temporal feature map, where \(M\) is the number of channels and \(T'\) is the temporal length of the feature map. We write \(A^k\in\mathbb{R}^{T'}\) for the \(k\)-th channel of \(\mathbf{A}\). Because temporal pooling layers in the CNN can change the temporal resolution, \(T'\) may differ from the input length \(T\). Let \(z_c=f_\theta(\mathbf{X})_c\) denote the pre-softmax logit for class \(c\). In Grad-CAM, channel weights are defined as: \[\label{eq:Grad-CAM-alpha} \alpha_k^{(c)} = \frac{1}{T'}\sum_{\tau=1}^{T'}\frac{\partial z_c}{\partial A_\tau^k}.\tag{4}\]
The feature-level temporal contribution is then obtained as: \[\tilde{s}_\tau^{(c)}=\sum_{k=1}^{M}\alpha_k^{(c)} A_\tau^k,\quad\tau=1,\ldots,T'.\] Unlike the original Grad-CAM formulation, AITE does not apply the ReLU non-linearity to \(\tilde{s}_\tau^{(c)}\), so that both positive and negative contributions to the original class are preserved for temporal edit selection.
Since \(\tilde{\mathbf{s}}^{(c)}\) has the temporal resolution of the intermediate feature map, we resize it to the input length before selecting edit positions. Specifically, we linearly interpolate \[\tilde{\mathbf{s}}^{(c)}=(\tilde{s}_1^{(c)},\ldots,\tilde{s}_{T'}^{(c)}),\] to length \(T\), yielding the input-level salience sequence \(\mathbf{S}^{(c)}=(s_1^{(c)},\ldots,s_T^{(c)}).\)
When \(T'=T\), this interpolation is the identity operation. The input-level salience sequence \(\mathbf{S}^{(c)}\) assigns a score to each original input time step and is used to rank temporal locations for editing.
4.3.1 Temporal Insertion with Grad-CAM↩︎
For temporal insertion, we choose the interval between two adjacent input time steps that has the lowest average salience: \[\label{eq:insertion95grad} i^\ast=\arg\min_{i\in\{1,\ldots,T-1\}}\frac{s_i^{(y)}+s_{i+1}^{(y)}}{2},\tag{5}\] where \(y\) is the original class. We insert the midpoint of the selected interval \((\mathbf{x}_i,\mathbf{x}_{i+1})\) and update the sequence according to Eq. 2 .
4.3.2 Temporal Deletion with Grad-CAM↩︎
For temporal deletion, after excluding endpoints, the deletion index is selected as: \[\label{eq:deletion95grad} j^\ast = \arg\max_{t\in\{2,\ldots,T-1\}} s_t^{(y)}.\tag{6}\]
5 Experimental Results↩︎
5.1 Datasets↩︎
To evaluate the proposed method, we used two publicly available online handwriting datasets. The first is Unipen [17], for which we used three subsets: numerical digits (Unipen 1A), uppercase letters (Unipen 1B), and lowercase letters (Unipen 1C). Each sample is represented as a sequence of 2D coordinates, and the sequence length was normalized to \(T=50\) time steps. Unipen 1A has \(11{,}650\) training and \(1{,}300\) test samples, while Unipen 1B and Unipen 1C have \(11{,}063\) training and \(1{,}235\) test samples.
The second is CASIA-OLHWDB 1.1 [18], an online handwritten Chinese character dataset with \(3{,}755\) classes from \(300\) writers. For CASIA, we remove stroke-end and character-end sentinel coordinates, concatenate the strokes into a single 2D point sequence, and normalize the coordinates to \([-1,1]\) per sample and axis. Unlike Unipen, CASIA samples are kept at their original variable lengths. The test split contains \(224{,}559\) samples.
5.2 Architecture and Settings↩︎
For the experiments, four classifiers are used.
1D CNN-3: A CNN with three convolutional blocks. Each block consists of a 1D convolution, BatchNorm [39], a Rectified Linear Unit (ReLU), and max pooling. The first block uses 64 filters with kernel size 3 and stride 1, while the subsequent two blocks use 128 filters. The network is followed by two fully connected layers: one hidden layer with 512 units and dropout with probability 0.5, and one output layer.
1D CNN-4: This network has the same architecture as 1D CNN-3, except that it contains four convolutional blocks instead of three.
BLSTM-2: To test transferability to a different network structure, we use a Bidirectional LSTM (BLSTM) [40]. Specifically, we use a two-layer BLSTM with 100 units per layer, following the recommendation of [41].
Transformer: To test transferability to an attention-based architecture, we use an encoder-only Transformer [3]. The model consists of a linear input projection, positional encoding, and six Transformer encoder blocks with four attention heads each. The encoder output is averaged over time and passed through a small MLP head to produce the class logits.
For CASIA-OLHWDB, we use analogous classifiers adapted to variable-length inputs and 3,755 classes: a 1D CNN with masked global average pooling in place of 1D CNN-3, together with BLSTM-2 and Transformer counterparts. For CASIA-OLHWDB, we additionally train a second 1D CNN as a black-box target (channels 64→128→128→128, kernel size 3 throughout, 80 epochs, clean test accuracy 94.8%), providing a within-CNN-family transfer target with a different filter configuration from the surrogate (64→128→256→256).
In both white-box and black-box settings, adversarial samples are generated using the gradients or salience of a 1D CNN surrogate model: 1D CNN-3 for Unipen and the analogous 1D CNN for CASIA-OLHWDB. In the white-box setting, the generated adversarial samples are used to attack the same surrogate model. In the black-box setting, they are transferred to different target models: 1D CNN-4, BLSTM-2, and Transformer for both datasets. We only consider one-shot black-box attacks, where the attacker receives no feedback from the target classifiers.
All models are trained using the Adam optimizer [42] with an initial learning rate of \(10^{-3}\) and 10,000 training iterations.
5.3 Comparison Methods↩︎
To evaluate the proposed method, we compare to seven well-established adversarial attacks: four classical attacks (FGSM, BIM, PGD, CW) and three transfer-oriented variants (MI-FGSM, NI-FGSM, 1D-TI-MI-FGSM). The hyperparameters for each attack are as follows:
FGSM [11]: \(\epsilon=0.1\) under the \(L_\infty\) constraint.
BIM [20]: \(\epsilon=0.1\), \(L_\infty\), step size \(\epsilon_{\mathrm{iter}}=0.001\), and \(I=600\) iterations.
PGD [12]: same as BIM.
CW [13]: \(L_2\) constraint, \(I=600\) iterations, learning rate \(5\times10^{-3}\), 7 binary search steps, initial constant \(5\times10^{-3}\), confidence \(0\), and early stopping.
MI-FGSM [21]: BIM with an accumulated momentum term over the input gradients to stabilize the update direction. Decay factor \(\mu=1.0\); remaining settings as BIM.
NI-FGSM [22]: MI-FGSM with the gradient evaluated at a Nesterov look-ahead point before each momentum update. Settings as MI-FGSM.
1D-TI-MI-FGSM [23]: MI-FGSM in which the per-step gradient is convolved with a 1D Gaussian kernel (size \(7\), \(\sigma=3.0\)) along the time axis. Remaining settings as MI-FGSM.
Since inputs are normalized to \([-1,1]\), adversarial examples are also clipped to \([-1,1]\). For CASIA-OLHWDB, \(\epsilon=0.03\) and \(I=400\) are used to account for the larger number of classes; the CW binary search steps are reduced to \(3\).
5.4 Results↩︎
| White-Box | One-Shot Black-Box | ||||
|---|---|---|---|---|---|
| 3-3 (lr)4-6 Dataset | Attack | 1D CNN-3 | 1D CNN-4 | BLSTM-2 | Transformer |
| Unipen 1A | No Attack | 98.9 | 98.8 | 98.8 | 98.2 |
| FGSM | 73.8 | 82.8 | 95.6 | 92.9 | |
| BIM | 10.9 | 72.9 | 93.9 | 91.3 | |
| PGD | 12.4 | 74.2 | 95.1 | 92.0 | |
| CW | 0.69 | 93.9 | 98.2 | 97.1 | |
| MI-FGSM | 10.9 | 70.8 | 93.2 | 91.0 | |
| NI-FGSM | 9.1 | 70.5 | 93.7 | 91.0 | |
| 1D-TI-MI-FGSM | 67.5 | 71.2 | 80.6 | 83.3 | |
| AITE (Ours) | 25.5 | 58.2 | 70.2 | 68.8 | |
| Unipen 1B | No Attack | 98.1 | 97.5 | 97.7 | 95.5 |
| FGSM | 68.7 | 71.5 | 92.6 | 88.3 | |
| BIM | 11.2 | 75.4 | 91.1 | 85.8 | |
| PGD | 13.7 | 75.8 | 92.5 | 86.2 | |
| CW | 4.08 | 95.6 | 97.1 | 93.4 | |
| MI-FGSM | 11.2 | 71.3 | 91.1 | 84.5 | |
| NI-FGSM | 9.0 | 73.4 | 91.6 | 85.0 | |
| 1D-TI-MI-FGSM | 67.7 | 73.5 | 78.8 | 75.9 | |
| AITE (Ours) | 15.0 | 61.8 | 60.3 | 65.0 | |
| Unipen 1C | No Attack | 97.1 | 97.7 | 96.5 | 94.8 |
| FGSM | 64.5 | 79.4 | 90.8 | 85.2 | |
| BIM | 20.2 | 80.1 | 89.6 | 85.7 | |
| PGD | 30.4 | 85.8 | 93.6 | 89.1 | |
| CW | 5.96 | 96.9 | 96.1 | 93.8 | |
| MI-FGSM | 7.5 | 74.2 | 88.9 | 84.1 | |
| NI-FGSM | 6.2 | 75.6 | 88.7 | 85.3 | |
| 1D-TI-MI-FGSM | 55.6 | 68.9 | 76.7 | 73.4 | |
| AITE (Ours) | 18.4 | 70.5 | 59.7 | 62.0 | |
| CASIA | No Attack | 95.0 | 94.8 | 96.5 | 96.3 |
| FGSM | 65.2 | 78.5 | 90.1 | 90.7 | |
| BIM | 53.5 | 75.3 | 89.6 | 90.4 | |
| PGD | 53.8 | 75.2 | 89.7 | 90.4 | |
| CW | 0.71 | 74.4 | 90.5 | 93.0 | |
| MI-FGSM | 54.3 | 75.4 | 89.6 | 90.4 | |
| NI-FGSM | 53.6 | 75.2 | 89.6 | 90.4 | |
| 1D-TI-MI-FGSM | 85.3 | 89.9 | 94.2 | 93.5 | |
| AITE (Ours) | 29.2 | 52.5 | 53.6 | 47.9 | |
2.5pt
Table 1 reports recognition accuracy under adversarial attacks for both white-box and one-shot black-box settings across the three Unipen subsets and CASIA-OLHWDB. In the white-box scenario, gradient-based attacks, specifically BIM, PGD, and CW, drastically reduce classification accuracy on the surrogate, with CW achieving the lowest accuracy on every dataset (0.69–5.96% on Unipen and 0.71% on CASIA). MI-FGSM and NI-FGSM behave similarly to BIM in this setting, while 1D-TI-MI-FGSM is comparatively weaker on the surrogate because the smoothing of the input gradient sacrifices in-model optimality in exchange for transferability.
However, strong performance in the white-box setting does not necessarily carry over to the black-box scenario. Although CW is highly effective when applied directly to the surrogate, it transfers poorly to unseen targets: its impact on the 1D CNN-4, BLSTM-2, and Transformer columns is consistently the smallest among all attacks (e.g., the recognition accuracy on the Transformer target stays above 93% on every dataset). This is because gradient-based attacks rely on perturbations optimized with respect to the surrogate model’s gradients, which may not align with the decision boundaries of an unseen target, and the emphasis of CW on minimizing the perturbation magnitude further limits its transferability. MI-FGSM and NI-FGSM, which were designed for improved black-box transferability, do improve over plain BIM/PGD in some cases (e.g., Unipen 1A and 1B on 1D CNN-4), but they still leave the BLSTM-2 and Transformer targets largely intact, with accuracies above 84% on every Unipen subset and CASIA target. 1D-TI-MI-FGSM is the strongest baseline overall and even slightly outperforms AITE on Unipen 1C against 1D CNN-4 (68.9 vs.). However, it remains substantially weaker than AITE on the BLSTM-2 and Transformer targets across all datasets.
In contrast, the proposed AITE achieves the lowest recognition accuracy in almost every one-shot black-box column, including transfer to the attention-based Transformer target that was never used during attack generation. On CASIA-OLHWDB, AITE reduces the 1D CNN-4, BLSTM-2, and Transformer accuracies to 52.5%, 53.6%, and 47.9%, respectively. Notably, AITE transfers more effectively to the within-CNN-family target (CNN-4) than across architectures, consistent with the intuition that CNN-to-CNN perturbations share more decision-boundary structure. The strongest additive baseline (1D-TI-MI-FGSM) only reduces the three black-box targets to 90.0%, 94.2%, and 93.5%, respectively. Because AITE selects insertion and deletion positions from salience rather than from raw input gradients, the resulting temporal edits target class-discriminative structure that tends to be shared across architectures, including CNN-to-BLSTM and CNN-to-Transformer transfers.
5.5 Visual Similarity↩︎
Attack success is usually quantified by the change in the model’s prediction. However, for handwriting recognition, misclassification alone does not characterize the practical relevance of an attack: handwritten inputs are interpreted by humans, and visibly distorted adversarial samples are unlikely to represent plausible handwriting. We therefore also assess the visual similarity between original and adversarial samples.
| Dataset | Metric | FGSM | BIM | PGD | CW | MI-FGSM | NI-FGSM | 1D-TI-MI-FGSM | AITE |
|---|---|---|---|---|---|---|---|---|---|
| Unipen 1A | \(L_2\) \(\downarrow\) | 22.0 | 20.9 | 21.0 | 13.8 | 21.8 | 21.4 | 22.8 | 14.3 |
| SSIM \(\uparrow\) | 0.550 | 0.577 | 0.574 | 0.760 | 0.554 | 0.563 | 0.518 | 0.746 | |
| Unipen 1B | \(L_2\) \(\downarrow\) | 23.5 | 22.4 | 22.5 | 15.0 | 23.2 | 22.9 | 24.3 | 11.7 |
| SSIM \(\uparrow\) | 0.497 | 0.525 | 0.524 | 0.725 | 0.502 | 0.512 | 0.466 | 0.808 | |
| Unipen 1C | \(L_2\) \(\downarrow\) | 22.1 | 19.2 | 19.4 | 13.1 | 21.7 | 21.4 | 22.7 | 11.7 |
| SSIM \(\uparrow\) | 0.551 | 0.627 | 0.624 | 0.775 | 0.558 | 0.567 | 0.527 | 0.809 | |
| CASIA | \(L_2\) \(\downarrow\) | 17.8 | 17.6 | 17.6 | 11.1 | 17.6 | 17.6 | 18.3 | 15.9 |
| SSIM \(\uparrow\) | 0.585 | 0.593 | 0.594 | 0.812 | 0.592 | 0.593 | 0.564 | 0.618 |
3pt
To this end, we compare the visual similarity of the rendered characters. Table 2 reports quantitative results across all attacks. In this table, the online handwritten digits and characters are rendered into \(64\times64\) grayscale images with black strokes and no points. For the visual similarity measures, we use the pixel-space \(L_2\) distance and the Structural Similarity Index Measure (SSIM) between the original samples and the adversarial samples. \(L_2\) measures the strict pixel-wise differences between the images, whereas SSIM emphasizes structural consistency, which better reflects human visual perception.
FGSM, BIM, PGD, MI-FGSM, NI-FGSM, and 1D-TI-MI-FGSM consistently produce large \(L_2\) distortions and low SSIM scores across all datasets. As shown in Fig. 3, this reflects the large amount of jittering in the generated strokes of the characters. In the rendered examples, these attacks introduce visible jitter and local stroke irregularities. The transfer-oriented variants (MI-FGSM, NI-FGSM, and 1D-TI-MI-FGSM) do not qualitatively change this behavior: they share the additive \(L_\infty\) formulation of BIM/PGD, so the rendered trajectories exhibit similar jittering and remain easily detectable. In contrast, the perturbations in CW are minimal, but as discussed above, this adversely affects black-box transferability because the minimal perturbations only exploit surrogate-specific decision boundaries.
In contrast, AITE preserves the overall structure of the handwriting while still being effective under black-box transfer. Although online handwriting is represented as a time series, temporal-domain edits such as point insertion or deletion alter the temporal structure of the sequence while leaving the rendered spatial trace largely unchanged, and are typically less perceptible than the spatial jitter introduced by additive perturbations.
5.6 Gradient Versus Salience for Temporal Edit Selection↩︎
As a baseline ablation, we replace the Grad-CAM salience with gradient-based importance: \(s^{grad}_t = \lVert\nabla_{\mathbf{x}_t}\mathcal{L}(f_\theta(\mathbf{X}), y)\rVert_2\), where larger values indicate higher sensitivity of the loss to time step \(t\).
| Visual | White-Box | One-Shot Black-Box | ||||
| Similarity | 1D CNN-3 | 1D CNN-4 | BLSTM-2 | |||
| Dataset | Selection | \(L_2\) \(\downarrow\) | SSIM \(\uparrow\) | Acc (%) \(\downarrow\) | Acc (%) \(\downarrow\) | Acc (%) \(\downarrow\) |
| Unipen 1A | Gradient | 9.92 | 0.849 | 64.6 | 76.8 | 88.8 |
| Salience | 14.3 | 0.746 | 25.5 | 58.2 | 70.2 | |
| Unipen 1B | Gradient | 9.81 | 0.846 | 50.1 | 64.8 | 72.8 |
| Salience | 11.7 | 0.808 | 15.0 | 61.8 | 60.3 | |
| Unipen 1C | Gradient | 9.46 | 0.855 | 53.9 | 82.1 | 81.6 |
| Salience | 11.7 | 0.809 | 18.4 | 70.5 | 59.7 | |
A comparison of the results is shown in Table 3. In the results, the gradient-based selection creates more similar adversarial samples but has a smaller effect on the accuracy. The reason for this is because the edited time steps tend to be adjacent to each other in salient regions. Conversely, time steps selected by gradient magnitudes tend to be distributed more equally over the character, allowing for less disruptive edits. However, both the white-box and black-box attacks with gradient-based selection are not as effective as salience-based selection. This suggests salience is more suitable than gradients for temporal editing.
6 Examination of the Temporal Edits↩︎
6.1 Single-Iteration Adversarial Samples↩︎
An interesting observation from our experiments is that some handwriting samples can be misclassified after only a single iteration of temporal editing. In other words, the class of a character is able to be changed with the removal of a single time step and the addition of a single time step. As demonstrated in Fig. 4, these cases are nearly identical to the original samples. These cases are of interest because they show samples whose correct classification relies on a single time step.
6.2 Class Sensitivity to Temporal Edits↩︎
Figure 5: Comparison of the confusion matrices on Unipen 1A under different attacks. Darker cells indicate larger counts.. a — AITE (Ours), b — PGD, c — CW
Fig. 5 compares the confusion matrices on Unipen 1A under AITE, PGD, and CW. PGD and CW, being gradient-based, produce similar confusion patterns, whereas AITE differs substantially: for instance, “2”\(\to\)“4” under AITE versus “2”\(\to\)“7” under the gradient methods, suggesting that “2”s are temporally closer to “4”s but spatially closer to “7”s. “9”s also tend to transition to “1”s or “7”s under AITE, while a few classes behave similarly across methods (e.g., “0”\(\to\)“8” and “8”\(\to\)“5”).
The matrices also reveal class-wise susceptibility: “1” and “5” are the hardest digits to attack with AITE because their single-stroke shapes leave little room for class change without spatial distortion, while the more complex “6”, “8”, and “9” are the easiest. This is consistent with the stronger attack effectiveness on the more complex Unipen 1B and 1C subsets in Table 1.
7 Conclusion↩︎
This paper examined adversarial attacks on online handwriting recognition from a temporal perspective. By formulating adversarial example generation as a sequence editing problem, the proposed approach modifies handwriting trajectories through point insertion and deletion guided by temporal salience, rather than through additive spatial perturbations. Experimental evaluations under both white-box and one-shot black-box settings demonstrated that temporal editing produces adversarial examples that preserve the structure and smoothness of natural handwritten characters while remaining effective across different model architectures. Future work will extend this preliminary study to word- and sentence-level online handwriting recognition. Human evaluation should also be conducted to assess the perceived naturalness of adversarial handwriting, and defenses against temporal editing attacks should be investigated.
Acknowledgements↩︎
This research was partially funded by JSPS Grant Number 25K22845 and JST CRONOS-JPMJCS24K4.