1 Introduction↩︎

Touch is a critical modality for human interaction with the environment, providing rich information about geometry, texture, and contact forces, and is therefore increasingly exploited in robotic manipulation [1], [2]. Vision-based tactile sensors (VBTS) [3], which use a camera to capture fine-grained elastomer deformations at high resolution and low cost, have emerged as the dominant tactile sensing paradigm. However, their limited durability and the laborious real-world acquisition pipeline render tactile data substantially scarcer and noisier than visual data [4], [5]. This bottleneck has spurred growing interest in synthesizing high-fidelity tactile images as a scalable surrogate for physical collection, lowering the training cost of downstream tasks such as robotic manipulation [6] and texture recognition [7]–[10].

The fundamental paradigm of tactile image generation is to first extract diverse tactile feature representations from existing data and then synthesize tactile images that satisfy specific scene requirements. Although previous work has made significant progress [11], two challenges remain: the need for large-scale and diverse datasets for pretraining each sensor type [12], [13], which results in significantly degraded performance in environments where tactile data acquisition is difficult; and the inability of learned features to generalize to image generation tasks involving other sensors. This naturally leads to the following question: Can we learn a data-efficient tactile representation that facilitates tactile generation across sensors and scenarios?

To address this challenge, we introduce ********, a tactile generation framework that supports cross-sensor and multi-scenario applications. First, to model the low-variance latent space of tactile images while preserving their rich informational content, we adopt a VQGAN-based architecture, incorporating deformable convolutions and a multi-scale fusion module to extract both macro-level deformation patterns and micro-level texture features from tactile images. This enhances its applicability to vision-based tactile sensors (VBTS). Additionally, we propose a few-shot mixed training approach that enables the transfer of learned tactile representations from existing sensor data to image generation tasks for new sensors, facilitating data-efficient cross-sensor generalization. Based on these learned representations, we further train a discrete diffusion model to support multiple input modalities and perform conditional generation of corresponding tactile images. Our contributions are summarized as follows:

• We introduce DM-VQGAN, an efficient tactile representation learner that captures generalizable, information-rich features tailored to tactile data.

• We propose a unified multimodal conditional generation framework that supports multiple input modalities, including tactile images, visual images, and semantic labels, to address diverse tactile generation requirements.

• We model tactile sensors at the sensor-family level and use clustering and few-shot mixed training to transfer features from one sensor to its family, eliminating full multi-dataset training.

• **** outperforms state-of-the-art models in tactile image reconstruction and generation, excelling in vision-limited scenarios.

2 Methodology↩︎

Overview. In this section, we elaborate on ********, whose pipeline is shown in Fig.2. It consists of three components: (i) DM-VQGAN, a tactile-specific encoder that models and amplifies geometric variations to produce generation-friendly latent representations (Sec.2.1); (ii) a Few-shot Mixed Training scheme that transfers these representations across sensor families for data-efficient cross-sensor generalization (Sec.2.1); and (iii) a unified discrete-diffusion generative framework that conditions on multimodal inputs and uses the pretrained encoder to synthesize tactile images (Sec.2.2).

2.1 DM-VQGAN↩︎

Vision-based tactile images discard color while preserving essential cues such as morphology, geometry, and force distribution [14]. Exploiting this low intrinsic variance together with VQGAN’s strong latent-space modeling [15], we adopt a discrete codebook to obtain structured representations under limited data. Vanilla VQGAN, however, relies on fixed convolutions and thus struggles to capture the non-rigid deformations and multi-scale patterns characteristic of VBTS [14], [16]. We therefore propose DM-VQGAN, which integrates deformable and multi-scale dilated convolutions to strengthen feature extraction and representation, as shown in Fig. 2 (b).

Training Process of DM-VQGAN. Similar to the classic VQGAN [17], DM-VQGAN consists of an encoder \(E\), a decoder \(G\), a patch-based discriminator \(D\), and a codebook \(Z = \{ z_k \}_{k=1}^{K}\), containing \(K\) discrete codes, as shown in Stage1 of Fig. 2 (a). Given an input image \(x \in \mathbb{R}^{H \times W \times 3}\), the encoder \(E\) first extracts a latent feature representation \(\hat{z} \in \mathbb{R}^{H_f \times W_f \times n_z}\), where \(n_z\) and \(f\) represent the dimensionality of the latent features and the spatial compression ratio, respectively. Next, the feature vector at each spatial location \((i, j)\) is quantized to the closest discrete code from the codebook via nearest neighbor search: \[z_q = q(\hat{z}) := \left( \arg \min_{z_k \in Z} \| \hat{z}_{ij} - z_k \| \right) \in \mathbb{R}^{H_f \times W_f \times n_z}, \label{eq6}\tag{1}\] The decoder \(G\) then decodes the quantized features back into the image space, i.e., \(\hat{x} = G(z_q)\). The training objective of DM-VQGAN is to minimize the difference between the input image and the reconstructed image, and the corresponding loss function can be expressed as: \[\begin{align} L_{\text{total}} =\; & \underbrace{ \| x - \hat{x} \|_1 + \| \text{sg}[E(x)] - z_q \|_2^2 + \beta \| \text{sg}[z_q] - E(x) \|_2^2 }_{L_{\text{vq}}} \\ & + L_{\text{per}} + L_{\text{adv}}, \end{align} \label{eq7}\tag{2}\] where \(L_{per}\) and \(L_{adv}\) represent the perceptual loss and the adversarial loss. In the reconstruction loss \(L_{vq}\), \(\text{sg}[\mathpunct{\cdot}]\) denotes the stop-gradient operation, and \(\beta \| \text{sg}[z_q] - E(x) \|_2^2\) is known as the commitment loss, where the commitment weight \(\beta\) is set to 0.25 [18], [19].

Deformation Pattern Extraction: In each encoder and decoder, we introduce deformable convolutions [16] to adaptively learn dynamic spatial offsets for convolution kernels, enabling the model to better capture local, complex, and irregular deformation patterns present in tactile images. The structure of the deformable convolution layer, denoted as: \[y(p_0) = \sum_{p_k \in \mathcal{G}} w_k \cdot x \left( p_0 + p_k + \Delta p_k \right), \label{eq8}\tag{3}\] where \(y(p_0)\) is the output feature vector at position \(p_0\), and \(p_k \in \mathbb{R}^2\), \(\Delta p_k \in \mathbb{R}^2\) represent the pre-defined offset and learned offset for the \(k\)-th sampling point. This flexible convolutional sampling mechanism facilitates effective learning of dynamic spatial relationships and feature variations, thereby enhancing sensitivity to localized deformations during tactile representation learning.

Multi-scale Fusion: To effectively capture the rich, multi-scale features inherent in tactile images, we design a multi-scale fusion module that employs parallel dilated convolutions with varying dilation rates (i.e., 1, 2, and 4), denoted as: \[y = \mathcal{F}_{\text{conv3x3}} \left( \text{Concat} \left( \left\{ \sigma \left( \mathcal{F}_{\text{dilate}}^{(d_i)} (x) \right) \right\}_{d_i \in \{1, 2, 4\}} \right) \right). \label{eq9}\tag{4}\]

This design expands the receptive field, enabling the simultaneous extraction of macro-level global deformations and micro-level textural features. Consequently, the DM-VQGAN effectively addresses the limitations of traditional convolutional structures in representing complex dynamic deformations and multi-scale features of tactile images, enhancing tactile representation and improving tactile image generation.

2.2 Few-shot Mixed Training for Cross-sensor Generalization↩︎

Many studies [20], [21] have observed substantial discrepancies among tactile sensors, which hinder the generalization of tactile representation models. Existing methods, such as UniTouch [20] and T3 [13], typically learn sensor-specific features from multiple large-scale datasets. Although effective, such a strategy is data-intensive and potentially redundant.

By analyzing mainstream tactile datasets [12], [22], we identify two major sources of variation in tactile images: intrinsic sensor properties and environmental factors. The former are determined by sensor design and manufacturing and remain relatively stable, while the latter, including illumination, calibration, and material degradation, introduce appearance shifts that may obscure consistent tactile patterns. We therefore hypothesize that sensors \(A_1\) and \(A_2\) with shared intrinsic properties but different environmental conditions belong to the same sensor family \(\textit{A}\), whose representations can be jointly learned. Consequently, a model trained on \(A_1\) can be adapted to \(A_2\) with only minimal additional data.

To identify the family of an unseen sensor, we propose an unsupervised clustering strategy. Specifically, DM-VQGAN is used to extract tactile features from images of multiple sensor types, followed by cross-sensor clustering. For each cluster, we compute its prototype vector by iteratively estimating feature centroids, and define the inter-cluster distance as \[D(C_i, C_j) = \left\| \frac{1}{|C_i|} \sum_{x \in C_i} x - \frac{1}{|C_j|} \sum_{y \in C_j} y \right\|, \quad i,j \in \mathbb{N}, \label{ynojiwac}\tag{5}\] where \(|C_i|\) and \(|C_j|\) denote the numbers of samples in clusters \(C_i\) and \(C_j\), respectively, and \(x\) and \(y\) are their corresponding feature vectors. Given an unseen sensor, we compute its prototype vector and assign it to the nearest sensor family. We then construct a small-sample mixed dataset to train DM-VQGAN for tactile representation learning, avoiding full retraining on all datasets.

2.3 Multimodal Conditional Generation Framework↩︎

Although DM-VQGAN learns effective tactile representations, its original decoder cannot flexibly incorporate external prompts such as images. We therefore adopt discrete diffusion [23], which naturally operates in discrete latent spaces, to build a unified multimodal conditional generation framework. As shown in Stage 3 of Fig. 2 (a), the diffusion decoder is conditioned on a single modality at each training step: \[\boldsymbol{c} = f_{m}(y_{m}), \quad m \in \{\text{\textit{visual}},\,\text{\textit{tactile}},\,\text{\textit{label}}\}, \label{eq10}\tag{6}\] where \(f_m(\cdot)\) denotes the encoder for modality \(m\), and \(\boldsymbol{c}\) is the corresponding conditional latent feature. This design enables flexible tactile image generation from visual, tactile, or label conditions according to task requirements. For visual conditioning, we use a frozen pretrained CLIP encoder [24], i.e., \(\boldsymbol{c}_{\text{visual}} = f_V^{\text{CLIP}}(y_{\text{visual}})\). For tactile conditioning, the input \(y_{\text{tactile}}\) is mapped to discrete indices using the sensor-specific codebook \(\mathcal{Z}=\{z_k\}_{k=1}^{K}\) learned by DM-VQGAN: \[\boldsymbol{c}_{\text{tactile}} = q\left(f_{\text{DM-VQGAN}}(y_{\text{tactile}}),\,\mathcal{Z}\right). \label{eq12}\tag{7}\] The selected conditional feature \(\boldsymbol{c}\) then interacts with the noisy latent representation \(\boldsymbol{z}_t\) through transformer modules in the diffusion decoder, enabling condition-specific tactile synthesis.

3 Experiments↩︎

3.1 Dataset and Baselines↩︎

We train and evaluate the proposed model on four public tactile datasets, namely Touch and Go [12], FabricVST [25], VisGel [26], and HCT [22], covering GelSight, GelSight Mini, and DIGIT sensors. We compare our method with representative tactile generation approaches [20] across multiple downstream tasks. For fairness, all experiments are repeated three times, and the average results are reported.

3.2 Results↩︎

Effectiveness Analysis of DM-VQGAN for Tactile Representation Learning. To evaluate the effectiveness of DM-VQGAN for tactile representation learning, we conduct reconstruction experiments on the FabricVST dataset [25] under varying training-sample scales, using MAE [27] and VQGAN [17] as baselines. As summarized in Table 1, DM-VQGAN consistently achieves superior reconstruction quality, preserving fine tactile structures across all data scales. In contrast, MAE fails to recover detailed textures (Fig. 3 (b)), and its performance with \(N=10\mathrm{k}\) remains inferior to DM-VQGAN trained with only \(N=100\) samples. Furthermore, latent-space visualization (Fig. 3 (a)) shows that, unlike CNN autoencoders that mainly retain color distributions, DM-VQGAN learns more structured and compact representations that emphasize salient tactile features while suppressing background variations. This demonstrates its effectiveness in learning discriminative and low-variance tactile representations.

Evaluation Across Multiple Tasks and Applications. We evaluate the proposed framework across data-limited learning, multimodal generation, and downstream recognition tasks. Under limited samples and categories, our method demonstrates superior few-shot representation learning and zero-shot generalization to unseen objects, consistently outperforming competing approaches (Tab. 4, Fig. 3 (c)). We further assess vision- and label-conditioned tactile image generation using real-world scene images and semantic labels. As shown in Fig. 4 and Tab. 2, **** produces tactile images that maintain strong cross-modal consistency, structural fidelity, and semantic alignment. Finally, classifiers trained on generated tactile images achieve higher object recognition accuracy than baseline methods (Tab. 3), demonstrating the practical value of the generated data for downstream applications.

Validation of Transferability and Generalizability across Datasets and Sensors. As discussed in Sec. 2.2, we improve the generalizability of **** across different sensors and datasets within the same sensor family through few-shot mixed training. Specifically, a small number of tactile images from unseen sensors are combined with existing datasets from their corresponding sensor families to train DM-VQGAN. As illustrated in Fig. 5, the learned codebook can be directly transferred to tactile image reconstruction for unseen sensors, demonstrating the effectiveness of our approach in enabling cross-sensor generalization under limited-data settings.

4 Conclusion↩︎

In this work, we propose ****, a novel and data-efficient framework for tactile image generation. **** establishes a new paradigm for few-shot tactile generation by leveraging the representation learner DM-VQGAN and a unified conditional diffusion encoder, enabling cross-sensor compatibility and vision-free multimodal-to-tactile generation. Our framework demonstrates superior performance over SOTA methods while requiring limited training data. The proposed sensor-family clustering and mixed training strategy further enhance generalizability. These findings highlight the potential of **** as an effective solution for scalable tactile generation and efficient tactile perception in robotics and human computer interaction applications.

References↩︎