1 Introduction↩︎

Industrial anomaly detection (AD) is fundamental to modern manufacturing, ensuring product quality, operational reliability, and safety. Automated visual inspection systems prevent defective products from reaching customers, thereby reducing financial losses, reputational risks, and safety hazards [1], [2], while reducing reliance on human operators and supporting earlier intervention in production pipelines. However, real-world industrial inspection involves more than simply deciding whether an image is normal or not. Instead, in practice, users often care about four closely related questions: 1) Is there an anomaly? 2) Where is it? 3) What type of anomaly is it? 4) Why are they anomalies? Therefore, a practical system should not only detect anomalies, but also localize them, recognize their types, and provide evidence for decisions.

Despite this progress, most methods address only part of these needs. Recent zero-shot approaches [3]–[7] build on CLIP [8] to distinguish normal from anomalous samples and derive anomaly maps, but remain focused on visual discrimination and localization, with limited support for semantic understanding; even multi-type extensions like MultiADS [7] do not explain why a region is anomalous.

Large instruction-following VLMs (e.g., LLaVA [9], Qwen-VL [10], and NVILA [11]) offer a complementary direction with strong multimodal reasoning, but their potential for industrial anomaly understanding is underexplored. Existing VLM approaches such as AnomalyGPT [12] obtain localization from an external detector rather than the VLM itself, so prior methods emphasize either localization or language reasoning without tightly unifying detection, segmentation, type recognition, and explanation in one model.

To address this gap, we present GenAU, a Generalist vision-language framework for industrial Anomaly Understanding. GenAU unifies anomaly detection, anomaly segmentation, multi-type anomaly recognition, and defect analysis within a single instruction-following architecture and training recipe (cf. Figure 1). Specifically, we extend an instruction-following VLM with two dedicated segmentation tokens, [SEG_defect] and [SEG_normal], which act as text-based anchors for defective and normal regions. By aligning their embeddings with multi-scale visual features from the visual encoder, GenAU produces fine-grained anomaly maps while preserving the model’s ability to recognize and describe specific defect types in natural language. In this way, GenAU combines the reasoning capabilities of large VLMs with pixel-level localization, moving beyond text-only defect recognition toward joint semantic understanding and spatial grounding.

We construct instruction-response data from standard benchmarks, training on MVTec-AD [13] and evaluating cross-dataset on VisA [14], MPDD [15], and Real-IAD [16] under distribution shift. At inference GenAU is single-image, requiring no reference images or task-specific retraining. Our contributions are summarized as follows:

• We introduce GenAU, a generalist vision-language framework for industrial anomaly understanding. Through a unified instruction interface, GenAU supports image-level anomaly detection, pixel-level anomaly segmentation, multi-type anomaly detection, and textual defect analysis through a single instruction-following architecture and training recipe.

• We propose a language-grounded segmentation mechanism that extends an instruction-following VLM with dedicated [SEG_defect] and [SEG_normal] tokens. The hidden states of these tokens are aligned with multi-scale visual features, enabling the model to ground defect-aware language representations into pixel-level anomaly maps.

• We evaluate GenAU on public industrial anomaly benchmarks under cross-dataset distribution shift, covering binary anomaly detection, anomaly segmentation, and zero-shot multi-type anomaly detection. The results show that a single autoregressive VLM can jointly perform detection, segmentation, multi-type recognition, and defect reasoning (reaching the best image-level detection among CLIP-based zero-shot methods on VisA and Real-IAD), and we explicitly characterize the segmentation–reasoning trade-off incurred when the same recipe additionally learns to reason.

2 Related Work↩︎

2.1 CLIP-based Prompt Adaptation for Anomaly Detection and Segmentation↩︎

Most zero-shot industrial AD adapts CLIP [8], despite it not being designed for inspection. Prompt-based methods craft or learn normal/abnormal prompts: WinCLIP [17] (window aggregation), AnomalyCLIP [5] (object-agnostic prompts), and PromptAD [18]/AdaCLIP [19] (learnable prompt tokens). For localization, APRIL-GAN [6], AnoCLIP [20], and SimCLIP [21] improve patch-level visual–text alignment; CLIP-SAM [22] and FiLo [23] add external segmentation/grounding modules; and MultiADS [7] adds defect-aware supervision for multi-type segmentation, while DAPO [24] learns hybrid defect-aware prompts via progressive tuning for multi-type detection and localization under distribution shift. Others rely on patch comparison or reconstruction [25]–[27]. These methods largely depend on CLIP-style embedding comparison or external modules, making them less suited to unified defect reasoning and analysis.

2.2 Autoregressive VLMs for Anomaly Detection and Reasoning↩︎

Autoregressive VLMs offer a different paradigm, enabling open-ended visual reasoning and instruction following instead of fixed-prompt embedding comparison. AnomalyGPT [12] is an early effort, but it relies on an external CLIP-based anomaly map combined with the input image and fed to the LLM, so the VLM mainly serves as a reasoning interface over a separately generated map. More recently, Anomaly-OV [28] specializes MLLMs for zero-shot anomaly detection and reasoning via the Anomaly-Instruct-125k instruction set, but targets image-level detection and explanation, with pixel-level segmentation not grounded in the decoding process.

In contrast, GenAU treats the autoregressive VLM as the core anomaly understanding model rather than as a reasoning interface over externally generated anomaly maps. Building on the reasoning-segmentation paradigm of LISA [29], in which a VLM emits a segmentation token whose hidden state drives a mask, GenAU uses decoder hidden states of dedicated [SEG_normal] and [SEG_defect] tokens as language-grounded anchors, aligning them directly with frozen multi-scale visual features (without an external mask decoder) to generate pixel-level anomaly maps. This design unifies image-level anomaly detection, anomaly segmentation, multi-type anomaly detection, and textual defect analysis in a single instruction-following framework. Grounding visual features in language is consistent with broader evidence that aligning visual representations with language or structured knowledge improves generalization under distribution shift [30] and open-domain visual recognition [31]. An extended discussion is provided in Appendix.

3 Preliminaries↩︎

We address three tasks under a common cross-dataset protocol. A model is trained only on a source dataset \(\mathcal{D}_{\text{train}}=\{(\boldsymbol{x}_i,\boldsymbol{y}_i)\}_{i=1}^{N_1}\) with inputs \(\boldsymbol{x}_i\sim\mathcal{P}(\mathcal{X})\), and evaluated, without retraining, on a target dataset \(\mathcal{D}_{\text{target}}\) whose inputs are drawn from a different distribution \(\mathcal{P}(\mathcal{X}')\nsim\mathcal{P}(\mathcal{X})\) (a distribution shift). At test time the model may optionally use target-domain metadata such as a product description, but no target labels. What the literature calls zero-shot anomaly detection/segmentation is, more precisely, binary detection/segmentation under this category-level distribution shift; we use the terms interchangeably and retain zero-shot for comparability with prior work.

Binary anomaly detection. Each image \(\boldsymbol{x}\in\mathbb{R}^{H\times W}\) carries a label \(y\in\{0,1\}\) indicating anomaly presence; the model classifies each target image as normal or anomalous.

Binary anomaly segmentation. Each image is paired with a pixel mask \(\boldsymbol{y}\in\{0,1\}^{H\times W}\); the model classifies each target pixel as normal or anomalous.

Zero-shot multi-type detection. Each image carries a multi-label vector over one normal class and \(K\) defect types \(\{d_1,\dots,d_K\}\). The target defect vocabulary partially overlaps the training one, \(C_{\text{train}}\cap C_{\text{target}}\neq\emptyset\) and \(C_{\text{target}}\setminus C_{\text{train}}\neq\emptyset\), so the target mixes seen and novel defect types. The model predicts \(\hat{\boldsymbol{y}}'\in\{0,1\}^{K_2+1}\) over the target vocabulary. Full formal definitions are given in Appendix 14.

4 Methodology↩︎

We present an overview of the proposed framework, followed by detailed descriptions of the anomaly detection and anomaly segmentation components, the training objectives, and the inference process.

4.1 Framework Overview↩︎

GenAU couples two jointly-trained pathways (Figure 2): (1) a text pathway that generates text conversation and (2) a visual grounding pathway that uses the hidden states of the generated [SEG_normal]/[SEG_defect] tokens as language-grounded anchors, aligned with multi-scale visual features to produce pixel-level anomaly maps; the image-level score fuses the map maximum with the decoder’s normal/defect decision.

4.2 Anomaly Segmentation↩︎

GenAU performs anomaly localization by grounding visual features with the hidden states of generated segmentation tokens. Given an image \(\mathbf{X}_v\) and an instruction \(\mathbf{X}_q\), the autoregressive decoder generates a structured response containing the special tokens [SEG_normal] and [SEG_defect]. Let \(\mathbf{h}_{\mathrm{normal}}, \mathbf{h}_{\mathrm{defect}}\in\mathbb{R}^{D}\) denote their final-layer decoder hidden states. These two vectors serve as language-grounded anchors for the normal and defective states: \[\mathbf{T} = [\mathbf{h}_{\mathrm{normal}}, \mathbf{h}_{\mathrm{defect}}] \in \mathbb{R}^{2\times D}.\]

The input image is processed by the frozen vision encoder to obtain multi-scale patch features from intermediate and final stages, \(\{\mathbf{V}_i\}_{i=1}^{m}\), where \(\mathbf{V}_i\in\mathbb{R}^{H_iW_i\times C_i}\) denotes the patch features from the \(i\)-th stage. Since the visual features and decoder hidden states lie in different representation spaces, each stage is equipped with its own linear projection layer \(\phi_i:\mathbb{R}^{C_i}\rightarrow\mathbb{R}^{D}\), producing \[\mathbf{Z}_i = \phi_i(\mathbf{V}_i) \in \mathbb{R}^{H_iW_i\times D}.\]

For each stage, we compute cosine-similarity logits between every projected patch feature and the two segmentation-token anchors: \[\mathbf{S}_i = \gamma \cdot \frac{\mathbf{Z}_i}{\|\mathbf{Z}_i\|_2} \left( \frac{\mathbf{T}}{\|\mathbf{T}\|_2} \right)^{\top}, \qquad \mathbf{S}_i\in\mathbb{R}^{H_iW_i\times 2},\] where \(\gamma\) is a fixed temperature scale. The two channels of \(\mathbf{S}_i\) correspond to the normal and defect scores. After reshaping \(\mathbf{S}_i\) to spatial resolution \(H_i\times W_i\), the maps are upsampled to the target mask resolution and normalized with a channel-wise softmax. The defect-channel probability gives the pixel-level anomaly map: \[\mathbf{A}_i = \mathrm{Softmax}(\mathrm{UP}(\mathbf{S}_i))_{[\mathrm{defect}]} \in [0,1]^{H\times W}.\] The stage-wise maps are aggregated by summation across the \(m\) stages to form the final anomaly map \(\mathbf{A}\), where higher values indicate a higher probability that the corresponding pixels belong to defective regions.

4.2.0.1 AnyRes high-resolution readout.

The backbone encodes high-resolution inputs with an AnyRes scheme that, besides a global low-resolution view, splits the image into a grid of higher-resolution tiles. Applying the readout above to the global view alone yields a coarse low-resolution grid. To recover fine spatial detail, GenAU reassembles the patch features of the AnyRes tiles into a single high-resolution feature grid and computes the same anchor-grounded similarity maps on it. At inference, the high-resolution map \(\mathbf{A}^{\mathrm{hi}}\) and the global-view map \(\mathbf{A}^{\mathrm{g}}\) are fused per stage as \(\mathbf{A}=(1-\beta)\,\mathbf{A}^{\mathrm{g}}+\beta\,\mathbf{A}^{\mathrm{hi}}\) with a fixed weight \(\beta\). This high-resolution readout is used for all reported segmentation results.

4.3 Image-level Anomaly Scoring↩︎

Building on the segmentation map above, GenAU computes the image-level anomaly score without training a separate image-level classifier. Let \(\mathbf{A}\) denote the final anomaly map of Section 4.2 (the stage-aggregated defect response). The image-level anomaly score is its spatial maximum, \[s_{\mathrm{img}} = \max_{(u,v)}\mathbf{A}(u,v),\] where a larger \(s_{\mathrm{img}}\) indicates a higher probability that the image contains an anomaly. This score introduces no additional trainable parameters and is obtained entirely from the language-grounded segmentation readout.

4.3.0.1 Fused image-level score.

The map maximum \(s_{\mathrm{img}}\) is a purely visual readout. Because the decoder additionally emits an explicit normal/defect decision, the image-level score we report is \[s_{\mathrm{fused}} = \tfrac{1}{2}\,\tilde{s}_{\mathrm{img}} + \tfrac{1}{2}\,\tilde{\ell}_{\mathrm{nd}},\] where the visual score \(\tilde{s}_{\mathrm{img}}\) and the decoder’s normal/defect log-odds \(\tilde{\ell}_{\mathrm{nd}}\) are each standardized by a frozen calibration estimated once on MVTec (Appendix 9); no target-domain labels are used. Unless stated otherwise, all reported image-level results use \(s_{\mathrm{fused}}\).

4.4 Textual Reasoning and Multi-Type Anomaly Detection↩︎

For zero-shot multi-type anomaly detection, given a target image \(\mathbf{X}_v\), a language instruction \(\mathbf{X}_q\), and the target-domain defect vocabulary \(C_{\mathrm{target}}=\{d'_1,\ldots,d'_{K_2}\}\), GenAU generates a structured response \(\mathbf{X}_a\) with an autoregressive VLM. A deterministic parser \(\pi(\cdot)\) maps the response to a multi-type anomaly label \(\hat{\mathbf{y}}'\in\{0,1\}^{K_2+1}\), where the entries correspond to the normal class and the \(K_2\) target defect types.

4.5 Training Objective↩︎

GenAU is optimized with a joint objective that supervises textual reasoning and pixel-level anomaly segmentation. For each training sample, the model receives an image \(\mathbf{X}_v\) and an instruction \(\mathbf{X}_q\), and is trained to generate a structured response \(\mathbf{X}_a=(a_1,\ldots,a_T)\) containing the predicted defect type and the grounding tokens [SEG_normal] and [SEG_defect]. The overall objective is \[\mathcal{L} = \mathcal{L}_{\mathrm{txt}} + \lambda_{\mathrm{seg}}\mathcal{L}_{\mathrm{seg}}, \label{eq:overall-loss}\tag{1}\] where \(\lambda_{\mathrm{seg}}\) balances the language-modeling and pixel-level supervision terms. The image-level anomaly score is read off the segmentation map at inference and thus requires no dedicated training term.

4.5.0.1 Text generation loss.

The textual reasoning objective is the standard autoregressive cross-entropy loss over the target response tokens: \[\mathcal{L}_{\mathrm{txt}} = - \sum_{t=1}^{T} \log p_{\theta} \left( a_t \mid a_{<t}, \mathbf{X}_v, \mathbf{X}_q \right). \label{eq:text-loss}\tag{2}\] This term trains the model to generate domain-specific inspection responses, including defect-type descriptions and the required segmentation tokens. Prompt tokens and inserted visual tokens are excluded from the language modeling loss.

4.5.0.2 Pixel-level segmentation loss.

For anomaly segmentation, GenAU predicts stage-wise similarity maps \(\{\mathbf{S}_i\}_{i=1}^{m}\), where each \(\mathbf{S}_i\) contains normal and defect logits for visual patches at stage \(i\). Let \(\mathbf{A}_i\) and \(\mathbf{A}_i^{\mathrm{n}}\) denote the defect- and normal-channel probability maps after upsampling and channel-wise softmax. Given a binary ground-truth mask \(\mathbf{M}_x\in\{0,1\}^{H\times W}\), the segmentation loss is summed over all visual stages: \[\begin{align} \mathcal{L}_{\mathrm{seg}} = \sum_{i=1}^{m} \Big( & \mathcal{L}_{\mathrm{focal}} \big( \mathrm{UP}(\mathbf{S}_i), \mathbf{M}_x \big) + \mathcal{L}_{\mathrm{Tversky}} \big( \mathbf{A}_i, \mathbf{M}_x \big) \notag \\ & + \mathcal{L}_{\mathrm{dice}} \big( \mathbf{A}_i^{\mathrm{n}}, \mathbf{1}-\mathbf{M}_x \big) \Big). \label{eq:seg-loss} \end{align}\tag{3}\] The focal term mitigates the severe foreground/background imbalance; the Tversky term on the defect channel up-weights false negatives over false positives, improving region recall and AUPRO; and the Dice term on the normal channel suppresses false positives. For anomalous images \(\mathbf{M}_x\) is the annotated mask; for normal images it is all-zero, giving negative supervision.

4.5.0.3 Trainable parameters.

We use parameter-efficient fine-tuning: the vision encoder and cross-modal projector stay frozen, while LoRA [32] on the decoder, the token-embedding table and LM head (for the two segmentation tokens), the \(m\) stage-wise projectors \(\{\phi_i\}\), and the decoder layer-norms are trained jointly under Eq. 1 .

4.6 Inference↩︎

GenAU uses the same instruction format as in training. For binary detection and segmentation the instruction names the object category and asks, via natural-language paraphrases, ‘‘Is there any anomaly on the {object}? If yes, segment it.’’ For zero-shot multi-type detection it is instead conditioned on the product type and lists the target object’s candidate defect types (its coarse classes under the taxonomy of Appendix 11), so different products receive different option sets: ‘‘Identify the defect type. Choose exactly one option from this list: {defect list}.’’

The model autoregressively generates a textual answer immediately followed by the two segmentation tokens, e.g.{defect} [SEG_defect] [SEG_normal] for an anomalous image, and a statement of normality followed by the same tokens for a normal one. Crucially, both [SEG_normal] and [SEG_defect] are always emitted, so their final-layer hidden states are available as language-grounded anchors regardless of the textual prediction. From these anchors we read off the three task outputs: (i) binary detection thresholds the fused image-level score \(s_{\mathrm{fused}}\), while threshold-free metrics such as image-level AUROC are computed directly from it; (ii) segmentation forms the AnyRes-fused defect-probability map of Sec. 4.2, where high responses mark defective regions and normal images are expected to yield low maps everywhere (they are supervised with all-zero masks during training); and (iii) a deterministic parser \(\pi(\cdot)\) maps the multi-type response to \(\hat{\mathbf{y}}'\in\{0,1\}^{K_2+1}\) over the normal class and the \(K_2\) target defect types.

5 Experiments↩︎

In this section, we describe the experimental setup, datasets, and evaluation metrics used to assess the performance of GenAU. We also present the baseline methods and key implementation details.

5.1 Datasets↩︎

We use four public industrial anomaly benchmarks (MVTec-AD [13], VisA [14], MPDD [15], and Real-IAD [16]), plus VisA-D&R [28] for defect reasoning, covering the three tasks of Sec. 3 and defect reasoning/analysis. Additional details are in Appendix.

5.2 Experimental Settings↩︎

We build GenAU on LLaVA-OneVision [33], which pairs a Qwen2 language backbone with a frozen SigLIP-so400m-patch14-384 vision encoder, and report results at the 0.5B and 7B scales. Following prior zero-shot protocols [5]–[7], GenAU is fine-tuned on the test split of MVTec-AD (its only split carrying anomaly masks and defect-type labels, the train split being normal-only) and evaluated, without retraining, on VisA, MPDD, and Real-IAD; MVTec-AD is thus purely the source domain and is disjoint from every evaluation target. High-resolution inputs use AnyRes (anyres_max_9, up to a \(6\times6\) grid of \(384\)-px tiles); the segmentation readout reassembles the tiles into a high-resolution feature grid, fused with the global view (\(\beta=0.25\), temperature \(20\)) using patch features from vision-encoder layers \(6,12,18,27\). We adapt only LoRA [32] (\(r=16\), \(\alpha=32\)) on the LLM, the token embeddings and LM head (to add the two segmentation tokens), and the per-stage projectors and decoder layer-norms; the encoder and cross-modal projector stay frozen. The segmentation loss combines focal, Tversky (\(\alpha=0.3\), \(\beta=0.7\)), and Dice, with \(\lambda=1\) (Eq. 1 ) and no separate image-level loss; the image score fuses the defect-map maximum with the decoder log-odds via a frozen MVTec calibration (mixing weight \(0.5\)). We train one epoch at effective batch size \(32\) (\(lr=10^{-4}\) cosine, warmup \(0.03\), no weight decay; \(8\) GPUs, ZeRO-3). Further details, including the multi-type prompt and parser, are in Appendix.

5.3 Evaluation Metrics↩︎

We report image-level AUROC and AP for detection; pixel-level AUROC and AUPRO for segmentation; weighted Precision/Recall/F1 for multi-type detection (to handle class imbalance); and, for reasoning, the VisA-D&R GPT-Score [28], where GPT-4o rates each response \(1\)–\(10\) for low-level (perception) and complex (causal-reasoning) questions separately (full definitions in Appendix).

5.4 Baselines↩︎

All baselines, like GenAU, access only the query image at test time: no reference, support, or exemplar images. As CLIP-based zero-shot detectors and segmenters we compare against CLIP, CLIP-AC [8], WinCLIP [17], APRIL-GAN [6], CoOp [34], CoCoOp [35], AnomalyCLIP [5], AdaCLIP [19], FiLo [23], and MultiADS together with its variant MultiADS-F [7], the only CLIP-based baseline that additionally performs multi-type detection. For the multi-type and reasoning tasks we further compare against vision–language baselines: the anomaly-reasoning specialist Anomaly-OV [28] and our LLaVA-OneVision [33] backbone, both at the \(0.5\)B and \(7\)B scales. Methods without public implementations (e.g., SimCLIP [21]) are excluded for reproducible comparison.

5.5 Results↩︎

Next, we present and discuss results for binary anomaly detection and segmentation under distribution shift, multi-type zero-shot anomaly detection, and defect reasoning, the four tasks GenAU addresses within a single architecture and training recipe.

5.5.1 Binary Detection and Segmentation under Distribution Shift↩︎

Table 1 reports the performance of GenAU on VisA, MPDD, and Real-IAD, compared with state-of-the-art CLIP-based methods. Pixel-level metrics (AUROC, AUPRO) and image-level metrics (AUROC, AP) are reported.

On VisA, the 7B model achieves the best image-level AUROC (\(87.6\%\)) among the CLIP-based zero-shot methods in Table 1, together with competitive pixel-level scores (\(94.7\%\) AUROC); the VLM reasoning specialist Anomaly-OV-7B reports a higher VisA image-AUROC (\(88.9\%\)) but is trained under an instruction-tuning rather than our cross-dataset protocol and produces no segmentation, so we note it here only for context. We note, further, that GenAU’s pixel-level AUROC does not surpass the strongest CLIP-based segmenters on any of the three datasets. The 0.5B variant remains competitive. On MPDD, the 7B model attains high pixel-level AUROC/AUPRO but lower image-level AUROC (\(70.3\%\), against \(79.7\%\) for MultiADS-F), reflecting the logical and large-area anomalies characteristic of this dataset. On Real-IAD, the 7B model reaches an image-level AUROC of \(84.7\%\). Across datasets, pixel-level AUPRO is consistently the lowest-scoring metric. Overall, GenAU demonstrates that a single instruction-following VLM can achieve strong binary detection and segmentation performance under distribution shift.

1pt

Qualitative results of GenAU and the best overall performer, MultiADS, are shown in Figure 3 (and, for a failure case, Figure 4). GenAU accurately segments visually salient defects such as hole, broken, and crack, and produces cleaner, lower-noise segmentation maps than MultiADS. However, global-context or logic-dependent defects such as missing components remain challenging. For these cases, similarity maps are low-confidence, often resulting in incomplete localization or misclassification of normal regions for both models (Figure 4).

Ablation Studies. Table 2 varies one factor at a time from our default GenAU-7B on VisA. Removing the LM fusion (map-max only) lowers image-AUROC by \(2.3\) points (\(87.6\!\rightarrow\!85.3\)); the segmentation-map maximum alone is already a competitive detector (above the CLIP baselines), with the decoder log-odds adding a consistent refinement rather than carrying the score. Adding the reasoning instruction mix to training (the setting also used for defect reasoning in Table 4) costs only \(2.9\) image-AUROC and \(2.6\) pixel-AUPRO, so the model retains most of its detection and localization while gaining reasoning, the segmentation–reasoning trade-off. Dropping LoRA on the LLM costs a further \(2.2\) image-AUROC and \(1.1\) pixel-AUPRO, confirming its contribution to detection and localization. The ablation studies of hyper-parameters are in Appendix.

5.5.2 Zero-Shot Multi-Type Detection↩︎

We evaluate zero-shot multi-type defect classification on VisA and MPDD (Table 3), conditioning the model on the target object’s candidate defect list (a candidate-list-conditioned zero-shot setting in Appendix). F1 ranges from \(\sim\) \(29\%\) on VisA to \(\sim\) \(70\%\) on MPDD, the latter higher owing to its fewer defect types. Most prior methods target only binary detection, limiting comparison; MultiADS is the only multi-type-capable baseline. GenAU-7B achieves the best F1 on both datasets, outperforming MultiADS and the reasoning specialist Anomaly-OV at both scales (Anomaly-OV-0.5b/7b are our MVTec-trained reproductions, prompted in a two-turn detect-then-classify format), despite spending no training budget on reasoning; Anomaly-OV is comparatively strong on MPDD (few types, where Anomaly-OV-7B even leads on recall) but weaker on VisA, whose finer taxonomy is harder. One caveat tempers the MultiADS comparison: MultiADS, designed for multi-type segmentation, is converted to detection by thresholding its per-type channels at a single fixed value of \(0.5\), an operating point that may understate it. GenAU’s advantage is driven mainly by precision (e.g., \(50.3\) vs MultiADS’s \(17.4\) on VisA and \(90.4\) vs \(34.0\) on MPDD), reflecting fewer spurious defect predictions under candidate-list-constrained decoding. Absolute scores remain low partly because VisA has 14 defect types and, with multi-defect images, 56 unique combinations to predict.

5.5.3 Defect Reasoning and Analysis↩︎

Beyond detecting and localizing defects, GenAU answers open-ended questions about them. We evaluate on VisA-D&R [28] (low-level perception and complex causal-reasoning questions) against the reasoning specialist Anomaly-OV [28]; here GenAU augments its segmentation training with a reasoning instruction mix, so one model both localizes and explains, whereas Anomaly-OV produces no segmentation.

As shown in Table 4, GenAU improves markedly over its own LLaVA-OneVision backbone (at 7B, low-level perception rises from \(3.57\) to \(5.22\) and complex reasoning from \(5.44\) to \(6.43\)) and exceeds the dedicated specialist Anomaly-OV at both scales. Strikingly, even GenAU-0.5B (\(5.01\)/\(6.13\)) surpasses the 7B specialist (\(4.26\)/\(6.34\)) on low-level perception, showing the reasoning instruction mix is effective even at small scale. This indicates that augmenting a segmentation-capable VLM with a lightweight reasoning instruction mix yields strong conversational ability without training a dedicated reasoning model, while still emitting the segmentation tokens needed for localization. These baseline scores, including our LLaVA-OneVision backbone, are taken from [28] under a separate GPT-4o judge instance, so the absolute comparison is indicative rather than strictly controlled; the consistent margin across both question types and both scales is what we rely on. More importantly, once trained with the reasoning instruction mix, GenAU is the only model in our study that performs binary detection, pixel-level segmentation, multi-type recognition, and defect reasoning (Table 2): the CLIP baselines emit no language and the reasoning specialist no masks. This supports our central claim that one architecture and training recipe, with an explicit segmentation–reasoning trade-off, covers the full inspection stack rather than a bespoke model per task.

6 Discussion↩︎

This section provides the performance analyses of GenAU, highlighting challenges and insights that are often unexplored in prior studies.

6.1 Local vs. Global Context Segmentation↩︎

GenAU excels at visually salient anomalies (distinct texture or color deviations), where strong local cues align with the defect-token anchors. Segmentation degrades for globally-defined or logical anomalies (e.g., missing or misplaced components; Figure 4), which look normal at the patch level and, in the zero-shot setting, give no prior on expected placement; few-shot or domain priors could help.

6.2 Challenges in Defect-Type Classification↩︎

Fine-grained classification exposes the limits of the VLM’s pre-trained knowledge: the same label varies widely across objects (a missing cable is a conspicuous void, a missing LED blends in; cracks range from large openings to fine lines), so generalization is hard when training shows one instance of a defect. Logical defects further require reasoning about expected structure that the model cannot encode without domain knowledge; broader exposure to each defect would help.

7 Conclusion↩︎

We presented GenAU, a single instruction-following architecture and training recipe that unifies the four questions of industrial inspection (is there an anomaly, where, what type, and why). Across VisA, MPDD, and Real-IAD it achieves competitive zero-shot detection and segmentation under distribution shift and, when the same recipe additionally learns to reason, adds multi-type recognition and defect analysis at a small, quantified cost. Globally-defined anomalies and fine-grained, object-dependent defect types remain open, pointing to few-shot adaptation and richer domain priors.

Acknowledgments↩︎

The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Hongkuan Zhou and Jingcheng Wu.

8 Datasets↩︎

Due to space limitations in the main manuscript, we provide here a detailed description of the industrial anomaly detection datasets used in our experiments: MVTec-AD [13], VisA [14], MPDD [15], and Real-IAD [16]. Key dataset statistics, including the number of defect types, object categories, and the proportion of single- and multi-defect images, are summarized in 5. For all datasets, each defect instance is mapped to a unified defect taxonomy as defined in . MVTec-AD includes common industrial items such as bottles, cables, zippers, and hazelnuts. VisA contains a mixture of everyday products such as chewing gum and macaroni, as well as more complex components like printed circuit boards. MPDD focuses primarily on industrial parts, including metal brackets, tubes, and machined components. Real-IAD contains a broad range of real-world consumer and industrial items such as audio jacks, switches, bottle caps, and plastic toys.

9 Implementation Details↩︎

We build GenAU on the LLaVA-OneVision family [33], which couples a Qwen2 language backbone with a SigLIP-so400m-patch14-384 vision encoder, and report the 0.5B and 7B scales. High-resolution inputs use the AnyRes scheme (anyres_max_9) with an image grid of up to \(6\times6\) tiles of \(384\) pixels, in addition to a global low-resolution view. For the segmentation readout we take patch features from four vision-encoder stages (layers \(6\), \(12\), \(18\), and the final layer) and equip each stage with its own linear projection into the decoder hidden space.

9.0.0.1 Trainable parameters.

The vision encoder and the original cross-modal projector are frozen. The LLM is adapted with rank-stabilized LoRA (rank \(r=16\), \(\alpha=32\), dropout \(0\)). The token-embedding table and language-modeling head are fully fine-tuned to integrate the two added segmentation tokens [SEG_normal] and [SEG_defect], and the four segmentation projectors together with the decoder layer-norms are trainable.

9.0.0.2 Optimization.

Training runs for a single epoch with an effective batch size of \(32\) (per-device batch \(1\), gradient accumulation \(4\), across \(8\) GPUs with DeepSpeed ZeRO-3), a learning rate of \(10^{-4}\) with cosine decay, a warmup ratio of \(0.03\), and no weight decay. The segmentation objective sums, over the four stages, a focal term on the upsampled logits, a Tversky term (\(\alpha=0.3\), \(\beta=0.7\)) on the defect channel that up-weights false negatives, and a Dice term on the complementary normal channel; the text-to-segmentation weight is \(\lambda=1\).

9.0.0.3 High-resolution readout and image-level fusion.

At inference the high-resolution and global-view anomaly maps are fused per stage as \(\mathbf{A}=(1-\beta)\,\mathbf{A}^{\mathrm{g}}+\beta\,\mathbf{A}^{\mathrm{hi}}\) with \(\beta=0.25\) and a similarity temperature of \(20\). The image-level score fuses the spatial maximum of \(\mathbf{A}\) with the decoder’s normal/defect log-odds: the two are each \(z\)-score standardized with statistics estimated once on the MVTec calibration split and summed with a mixing weight of \(0.5\); the standardization is frozen and shared across target datasets, so no target-domain labels are used.

10 Training Data Construction↩︎

Training data are derived from established anomaly detection and segmentation benchmarks [13], [14], [16]. We retain the original images and segmentation masks and augment them with instruction–response pairs designed for instruction tuning; this augmentation is the only modification to the original data. Each sample thus consists of the original image, its segmentation mask, a textual instruction, and the corresponding response. The instruction asks, via natural-language paraphrases, whether the named object contains an anomaly and requests its localization, e.g.‘‘Is there any anomaly on the {object}? If yes, segment it.’’

The response describes the anomaly in natural language and terminates with the two segmentation tokens [SEG_defect] and [SEG_normal], from which the localization read-out is taken; for a normal image it instead states that no anomaly is present. An example response is:

“Yes, there is an anomaly on the capsule. Specifically, there is a dent on the orange side, affecting the clarity and smoothness of the numeral ‘0’ in the printed number ‘500.’ [SEG_defect] [SEG_normal]”

11 Defect Taxonomy↩︎

Industrial AD datasets often define highly specific defect categories, which can lead to overfitting and inconsistent label usage [36]. To improve generalization, we map product-specific labels to a unified, coarser taxonomy, following MultiADS [7]: e.g., liquid and metal contamination are merged into contamination, and broken large/broken small are unified as broken. The taxonomy is defined per object, so the candidate defect set differs across product categories. This reclassification promotes abstraction over product-specific anomalies and mitigates label imbalance, and it does not affect comparisons with prior binary-detection methods, whose metrics are independent of class granularity. The complete per-object mapping for all four datasets is given in .

12 Evaluation Protocol↩︎

All models are evaluated under single-image inference: only the query image is available at test time, with no reference, support, or exemplar images. GenAU is fine-tuned on the MVTec-AD test split (the only MVTec split carrying anomaly masks and defect-type labels, as the train split contains normal images only) and evaluated, without retraining, on the test splits of VisA, MPDD, and Real-IAD; MVTec thus serves purely as the source domain and is disjoint from all evaluation targets, so no test-time information leaks into the reported numbers. For Real-IAD we use the single-view setting.

12.0.0.1 Unified inspection prompt.

For binary detection and segmentation, the instruction names the object category and asks whether the image contains an anomaly, requesting its segmentation. The model generates a textual answer immediately followed by the two segmentation tokens, e.g.{defect} [SEG_defect] [SEG_normal] for an anomalous image and a statement of normality followed by the same tokens for a normal one; both [SEG_defect] and [SEG_normal] are always emitted, and we extract their final-layer hidden states as language-grounded anchors.

12.0.0.2 Product-conditioned multi-type detection.

For zero-shot multi-type detection, the instruction is conditioned on the product type: it lists the candidate defect types of the target object (its coarse classes under the taxonomy above) and asks the model to select one. A deterministic parser maps the generated response to a multi-type label over the target-domain defect vocabulary. We report weighted Precision, Recall, and F1 to account for the strong class imbalance across defect categories.

13 Additional Ablations↩︎

Table 13 ablates, on MVTec-AD\(\rightarrow\)VisA, four training choices relative to our default GenAU-7B: the text-to-segmentation loss weight \(\lambda\), the number of intermediate vision-encoder stages \(m\) supplying the segmentation readout, whether the cross-modal projection to the LLM is trainable, and whether the vision encoder itself is unfrozen. Each row changes a single factor; the shaded default row corresponds to the GenAU-7B configuration of Table 1.

Raising the text-to-segmentation weight to \(\lambda=5\) trades a small pixel-AUROC gain for lower image-level detection (image-AUROC \(87.6\rightarrow87.0\)), so we keep the balanced \(\lambda=1\). Reducing the segmentation readout to \(m=3\) intermediate stages costs \(4.1\) points of pixel-AUPRO (\(88.2\rightarrow84.1\)) and about a point of image-AUROC, confirming that the multi-stage read-out is load-bearing for localization. Making the cross-modal projection trainable is close to neutral on the pixel metrics but lowers image-AUROC by \(1.5\) points; with only the MVTec-AD test split available for fine-tuning, the added capacity yields no gain, so we keep the projection frozen. Finally, unfreezing the vision encoder is markedly harmful, collapsing pixel-AUPRO to \(80.6\) and image-AUROC to \(73.5\) — far below every other configuration — as the encoder over-fits the small fine-tuning set and forgets its general visual features; freezing it is therefore essential.

3pt

14 Formal Task Definitions↩︎

We give the full definitions of the three tasks summarized in Section 3.

14.0.0.1 Binary anomaly detection under distribution shift.

Let \(\mathcal{D}_{\text{train}}=\{(\boldsymbol{x}_i,y_i)\}_{i=1}^{N_1}\) and \(\mathcal{D}_{\text{target}}=\{(\boldsymbol{x}_i,y_i)\}_{i=N_1+1}^{N_1+N_2}\) be two image–label datasets drawn from different input distributions \(\mathcal{P}(\mathcal{X}),\mathcal{P}(\mathcal{X}')\): \(\forall i\in\{1,\dots,N_1\}\), \(\boldsymbol{x}_i\sim\mathcal{P}(\mathcal{X})\), and \(\forall i\in\{N_1+1,\dots,N_1+N_2\}\), \(\boldsymbol{x}_i\sim\mathcal{P}(\mathcal{X}')\), where \(\mathcal{P}(\mathcal{X})\nsim\mathcal{P}(\mathcal{X}')\). Each image \(\boldsymbol{x}\in\mathbb{R}^{H\times W}\) is labeled with \(y\in\{0,1\}\) indicating anomaly presence. A binary detector \(\hat{f}(\cdot)\) is trained solely on \(\mathcal{D}_{\mathrm{train}}\); at test time it is optionally adapted to \(\hat{f}'(\cdot)\) using auxiliary target-domain metadata such as a product description. For each \(\boldsymbol{x}\in\mathcal{D}_{\mathrm{target}}\), the detector classifies it as normal or anomalous.

14.0.0.2 Binary anomaly segmentation under distribution shift.

With \(\mathcal{D}_{\text{train}}\) and \(\mathcal{D}_{\text{target}}\) as above and \(\mathcal{P}(\mathcal{X})\nsim\mathcal{P}(\mathcal{X}')\), each image \(\boldsymbol{x}\in\mathbb{R}^{H\times W}\) is paired with a binary mask \(\boldsymbol{y}\in\{0,1\}^{H\times W}\) indicating the presence of anomalies at each pixel. The segmentation model \(\hat{f}(\cdot)\) is trained solely on \(\mathcal{D}_{\mathrm{train}}\) and optionally adapted to \(\hat{f}'(\cdot)\) with target-domain metadata. For each pixel of \(\boldsymbol{x}\in\mathcal{D}_{\mathrm{target}}\), it classifies the pixel as normal or anomalous.

14.0.0.3 Zero-shot multi-type anomaly detection.

Let \(\mathcal{D}_{\text{train}}=\{(\boldsymbol{x}_i,\boldsymbol{y}_i)\}_{i=1}^{N_1}\) and \(\mathcal{D}_{\text{target}}=\{(\boldsymbol{x}_i,\boldsymbol{y}'_i)\}_{i=N_1+1}^{N_1+N_2}\), with inputs from \(\mathcal{P}(\mathcal{X})\nsim\mathcal{P}(\mathcal{X}')\). Each training image carries \(\boldsymbol{y}\in\{0,1\}^{K_1+1}\), whose entries index one normal class and \(K_1\) defect types \(\{d_1,\dots,d_{K_1}\}\); each target image carries \(\boldsymbol{y}'\in\{0,1\}^{K_2+1}\) over one normal class and \(K_2\) defect types. Writing \(C_{\mathrm{train}}=\{d_1,\dots,d_{K_1}\}\) and \(C_{\mathrm{target}}=\{d'_1,\dots,d'_{K_2}\}\), we have \(C_{\mathrm{train}}\cap C_{\mathrm{target}}\neq\emptyset\) and \(C_{\mathrm{target}}\setminus C_{\mathrm{train}}\neq\emptyset\): the target dataset mixes defect types previously seen during training with novel types never seen by the model.

15 Extended Related Work↩︎

This expands the related-work overview of the main text.

15.0.0.1 CLIP-based prompt adaptation for anomaly detection and segmentation.

Recent advances in industrial anomaly detection/segmentation have largely built upon CLIP due to its strong zero-shot transfer ability, although CLIP was not originally designed for industrial inspection. A major line of work adapts CLIP to industrial AD through normal/abnormal prompts, learnable prompt tokens, or improved visual–text alignment. WinCLIP [17] introduces normal and abnormal textual prompts together with window-based feature aggregation for zero-shot anomaly detection. AnomalyCLIP [5] learns object-agnostic prompts to capture generic semantics of normality and abnormality, while PromptAD [18] improves state discrimination by adding learnable prefix and suffix tokens. AdaCLIP [19] further combines static and dynamic learnable prompts for adaptive inference across object categories. Beyond prompt learning, CLIP-based methods have been extended to localization and segmentation by improving patch-level alignment or incorporating external modules. APRIL-GAN [6], AnoCLIP [20], and SimCLIP [21] enhance cross-modal alignment for pixel-level anomaly mapping, while CLIP-SAM [22] and FiLo [23] combine CLIP with external segmentation or grounding models, and MultiADS [7] introduces defect-aware textual supervision for multi-type anomaly detection. Other approaches, such as MuSc [25], MAEDAY [26], and Aota et al. [27], rely on patch comparison, feature uniformity, or reconstruction difficulty. Despite their effectiveness, these methods largely depend on CLIP-style embedding comparison, external localization modules, or non-generative anomaly scoring, making them less suited for unified defect reasoning and analysis.

15.0.0.2 Autoregressive VLMs for anomaly detection and reasoning.

Autoregressive VLMs provide a different paradigm for anomaly understanding by enabling open-ended visual reasoning and instruction-following interaction. Instead of comparing an image embedding with a fixed set of CLIP prompts, they can generate textual descriptions, answer inspection questions, and reason over defect attributes in natural language. AnomalyGPT [12] is an early effort: it introduces an instruction-following VLM for industrial anomaly reasoning and lets users query the model about abnormal regions, but it relies on an external CLIP-based anomaly map combined with the input image and provided to the language model, so the VLM mainly serves as a reasoning interface on top of a separately generated map. More recently, Anomaly-OV [28] specializes multimodal large language models for zero-shot anomaly detection and defect reasoning via Anomaly-Instruct-125k, a large-scale visual instruction-tuning dataset for anomaly-centric understanding; however, it mainly targets image-level detection and explanatory reasoning, while pixel-level segmentation is not explicitly grounded in the autoregressive decoding process.

16 Detailed Evaluation Metrics↩︎

We adopt standard metrics for each task. For image-level anomaly detection, we report the Area Under the Receiver Operating Characteristic Curve (AUROC) and Average Precision (AP). For segmentation, we report the pixel-level AUROC and the Area Under the Per-Region Overlap (AUPRO). For multi-type anomaly detection, we use weighted Precision, Recall, and F1-score to account for the strong class imbalance across defect categories. For defect reasoning, we follow the VisA-D&R protocol [28] and report a GPT-Score in which GPT-4o rates each response on a \(1\)–\(10\) scale, separately for low-level (perception) and complex (causal-reasoning) questions.

17 Qualitative Results↩︎

In this section, we provide additional visualizations of anomaly segmentation results together with example text responses generated by the model. We present eight samples from diverse products, including tubes (5) from MPDD, capsules (6) and pipe fryum (7) from VisA, and mounts (8) and rolled strip base (9) from Real-IAD.

For the tubes, the model precisely identifies the deformed section and preserves clean boundaries, introducing almost no spurious activations elsewhere in the images. A similar pattern holds for the capsules, where the model distinguishes among different defect manifestations, correctly localizing internal bubbles, subtle surface scratches, and areas showing signs of leakage. For the pipe fryum, the model maintains high confidence across several defect types, producing focused masks for color specks, fine scratches, and fractured edges, and doing so consistently across varying shapes and orientations. On the Real-IAD mounts, the model is able to localize defects but introduces substantial prediction noise. It also occasionally assigns elevated similarity scores to background regions, even when those regions are uniform black and contain no structural cues. For the rolled strip base, the model effectively captures the primary defect, although the anomaly map exhibits considerable background noise and incorrectly attributes medium similarity scores to structural features such as the central diagonal ribs, despite these regions being non-defective.

In 10, we show examples of how the model classifies defects across different products together with the corresponding segmentation masks. For the cashew sample, it correctly notes the presence of a dark region on the surface, corresponding to a color defect. For the pipe fryum, it identifies that several pieces are touching and classifies the issue as misplaced. For the PCB, it detects the bent defect by recognizing the curved pins. Combined with the segmentation masks, these textual explanations provide detailed and actionable information that can support process optimization.

References↩︎