1 Introduction↩︎

The research community aims to establish a validation method for safety case fragments, assuring over time-changing safety-critical systems that adapt to evolving safety boundaries [1], [2]. The pace of adaptation required is determined by the exposure of entities in, for example, autonomous driving fleets. These kinds of systems demand robust safety cases, enhanced by runtime monitoring feedback [3], thereby enabling advanced system-level assurance [4]. Defeaters, in the domain of safety assurance, refer to arguments or evidence that undermine the assurance claims made within the safety case [5]. Identifying and understanding these defeaters are essential for validating a safety case’s resilience against real-world scenarios. Manual creation and validation in this context is not scalable and relies on subjective expert judgment [6]. Generating defeaters using large language models (LLMs) increases confidence in the argumentation [7], thereby making safety assurance scalable and responsive.
In Viger et al. (2024) [7], the generated defeaters were manually reviewed by two safety experts. The experts only agreed on the quality of the defeaters, with a low level of inter-rater agreement calculated by cohens kappa [8]. Low inter-rater agreement indicates that expert judgments vary due to differing interpretations and domain knowledge. Furthermore, manual validation is resource-intensive and lacks scalability, particularly for dynamic systems with evolving safety boundaries, such as adaptive software. These limitations hinder the efficiency and reliability of assurance synthesis, making automated approaches necessary to enhance defeater validation. This work addresses the issue of subjectivity in manual defeater validation, as illustrated by the schematic figure 1.
In Section 3, we propose an NLP-based method that uses BERT embeddings [9] and meta-classifiers to objectively evaluate LLM-generated defeaters, enabling scalable assurance. Our experiment, detailed in Section 4, compares this approach with human evaluations and demonstrates improved consistency and reduced subjectivity in expert dissent. The results in Section 5 provide further support for the conclusions in Section 6 with an outlook on future work.

2 Related Work↩︎

LLM-generated Assurance Fragments Recent studies, such as Sivakumar et al. (2023) [12], demonstrated GPT-4’s moderate success in generating Goal Structuring Notation (GSN)-based safety cases, a graph structure style, limited by incomplete domain knowledge [13]. Kurzidem et al. (2023) [14] used meta-models to enable a holistic safety analysis to quantify the uncertainties. Gohar et al. (2024) [15] showed preliminary success in using LLMs to identify defeaters, while Shahandashti et al. (2024) [16] highlighted GPT-4 Turbo’s capability to generate domain-specific defeaters like software bugs or environmental risks, thereby further advancing automation. Graydon (2025), appendix, finds the redefinition of defeaters as hazards or failure conditions, distinct from logical flaws, suggesting that generic defeater searches may not replace safety analysis [17]. Defeaters are reasons that challenge a claim through defeasible reasoning [18].

Embeddings for Assurance Cases Fragments The Transformer architecture [19] has revolutionized natural language processing by enabling large language models (LLMs), such as GPT-4, to generate contextual embeddings that capture intricate semantic and syntactic relationships [9]. These embeddings serve as a foundation for advanced techniques like Named Entity Recognition (NER) [20]. By leveraging the contextual richness of LLM-generated embeddings, assurance case development can produce clear, domain-specific narratives within frameworks like Goal Structuring Notation (GSN), ensuring both structural integrity and semantic accuracy [13]. Existing research often overlooks automated quality assessments of generated claims and the use of advanced classification techniques to validate them. Recent studies that apply LLMs to the analysis of safety-critical systems include goal modeling [21], automated software proof generation [22], and LLM-driven defeater identification [7]. Odu et al. (2025) [23] compared LLM outputs and assurance cases, finding that outputs are "relatively good" but not sufficient for certifying mission-critical systems without human intervention. Varadarajan et al. (2024) [24] propose a continuous assurance framework that emphasizes validity and soundness, employing LLMs to extract properties and environmental information from assurance case texts, processed by Prolog-based reasoning engines for semantic and logical analysis. While effective for structured validation, such hybrid approaches often rely on LLMs for natural language syntactic parsing rather than direct context semantic analysis.

Verification and Validation Eliminative Argumentation (EA) in this regard provides a structured approach to constructing and validating assurance arguments by eliminating defeaters [18]. This iterative process builds confidence in claims by systematically addressing doubts, making it suitable for automation through NLP. Ensuring LLM reliability requires robust validation against real-world data, expert evaluations, and alignment with standards like ISO 26262, ISO 21448, and UL4600 [25]–[27]. These standards enhance the administration of domain-specific safety case generation and independent body compliance, confirming stakeholder acceptance. Rushby et al. [6] emphasize evaluating assurance arguments beyond compliance, advocating for semantic robustness. We assume that semantic robustness can be enhanced through precise claim localization in DAGs and objective defeater embedding analysis by NLP techniques. Validation strategies, as discussed in [28], address risks such as output inaccuracy, lack of systematic coding frameworks, and ethical concerns in the validation and verification of LLM-generated code, emphasizing the need for rigorous processes to ensure reliable and safe outcomes. An iterative process builds confidence [6] in safety-critical systems, such as autonomous driving, by addressing doubts to support claims like “the system is safe”. Viger et al. (2024) [7] leveraged EA to generate defeaters for assurance cases, providing input for our work. Building on the defeaters generated by Viger et al., our approach uses the BERT architecture [9] to automate their evaluation and validates it with a trained model on expert judgment. This case study lays the foundation for experiments that can automatically evaluate LLM-generated defeater performance in safety-critical contexts.

Summary Existing studies focus on generating assurance case fragments but do not address the systematic evaluation of their quality, leaving a gap in reliable and objective validation. In this work, we focus on automating the quality assessment of synthesized assurance arguments and the use of classifier strategies to validate defeaters.

3 Method↩︎

This section presents the methodology for assessing the quality of defeaters within assurance cases using Natural Language Processing (NLP) techniques. Our methodology extends the AI-Supported Eliminative Argumentation (AI-EA) framework by Viger et al.[7] comprising three phases: (1) the foundation experiment establishing a baseline through manual review, we will present a small insights from the provided dataset from Viger et al. to clarify the manual review task; (2) an inter-rater agreement analysis to quantify consensus and dissensus of the manual review; and (3) our novel extention, the automated review process.

3.1 Manual Review Process↩︎

The AI-Supported Eliminative Argumentation (AI-EA) study dataset includes 171 AI-generated defeaters from two industrial assurance cases (ACs): an Adaptive Cruise Control (ACC) system from the automotive domain including (317 nodes) and the CERN Large Hadron Collider Machine Protection System (MPS) from the nuclear domain with (506 nodes) [29]. Each generated defeater was structurally analyzed as a textual counterargument within the Eliminative Argumentation framework [18], comprising three components: “What” (the flaw), “Where” (the affected claim or system component), and “Why” (the rationale for the challenge). Reviewers scored these components as 0 (absent or non-compliant), 1 (present but generic), or 2 (specific and reasonable). Addressing defeaters strengthens safety arguments, thereby enhancing AC confidence. Below, we provide two example defeaters to illustrate their structure and the manual analysis process, as detailed in the study by Viger et al..

• ACC Defeater ID 1: “The effectiveness of the black box radar sensing and Machine Learning system in maintaining safe distance is uncertain; Where: reliance on a system whose internal functioning is not analyzed; Why: without understanding the system’s internal functioning, potential failure modes may not be predictable or preventable.”
  Analysis: Rated 1 (generic) for “What” (vague uncertainty), 0 (non-compliant) for “Where” (lacks specific claim reference), and 2 (reasonable) for “Why” (clear failure mode rationale). This defeater prompts analysis of system behavior to address black box uncertainties.

• CERN Defeater ID 1: “What: Total reliance on the Beam Loss Monitoring System (BLMS) for detecting intolerable beam loss; Where: CHILD CLAIM 9; Why: assumes BLMS always provides timely indications, risking system failure if BLMS faults.”
  Analysis: Rated 2 (reasonable) for “What” and “Where” (specific system and claim), but 1 (generic) for “Why” (broad fault assumption). This defeater flags a single point of failure, necessitating reliability evidence.

These examples demonstrate how defeaters were manually analyzed individually for component quality and how addressing them (e.g., through evidence or refined claims) strengthens safety case confidence.

3.2 Consensus and Dissensus↩︎

Following the manual review process described in Section 3.1, we assessed the inter-rater agreement between the two human reviewers (H1 and H2) to quantify the consistency of defeater ratings for the “What,” “Where,” and “Why” components across the ACC and CERN datasets. This analysis establishes the baseline variability in expert judgments, motivating the need for automated validation. We calculated Cohen’s kappa categorical using the cohen_kappa_score function from the scikit-learn library [30], accounting for agreement by chance. The methodology involved the following steps:

1. Observed Agreement (\(p_o\)): Computed as the proportion of defeaters where H1 and H2 assigned the same rating (0, 1, or 2) for each component.

2. Marginal Probabilities: Probabilities that each rater assigns a specific rating to a component, regardless of the other rater’s choice. Obtained from the rating distributions.

3. Expected Agreement (\(p_e\)): Calculated as the sum of the products of corresponding marginal probabilities for each rating level.

4. Cohen’s Kappa (\(\kappa\)): Determined using the formula: \[\kappa = \frac{p_o - p_e}{1 - p_e}\] as described in [8].

For example, the ACC dataset with the “Why” component, reviewers agreed on 42 out of 75 defeaters, with an observed agreement (\(p_o = 0.56\)), and an expected agreement of (\(p_e \approx 0.57\)). The rating distributions were:

• Reviewer 1: {0: 1, 1: 19, 2: 55}

• Reviewer 2: {0: 1, 1: 24, 2: 50}

The kappa values were:

• What: \(\kappa = 0.383\), indicating fair agreement.

• Why: \(\kappa = -0.024\), implying disagreement below chance.

• Where: \(\kappa = 0.442\), indicating moderate agreement.

For the CERN dataset, the “Why” component, here the reviewers agreed on 61 out of 97 defeaters (\(p_o \approx 0.629\)), with \(p_e \approx 0.565\). The rating distributions were:

• Reviewer 1: {1: 32, 2: 65}

• Reviewer 2: {0: 2, 1: 26, 2: 69}

The kappa values were:

• What: \(\kappa = 0.167\), indicating slight agreement.

• Why: \(\kappa = 0.147\), indicating slight agreement.

• Where: \(\kappa = 0.329\), indicating fair agreement.

These results highlight significant inter-rater variability, particularly the negative kappa for the ACC “Why” component, underscoring the subjectivity in manual evaluations. This variability underscores the need for an automated, structured review process that follows qualitative, objective rating metrics. This will help reduce bias and improve consistency.

3.3 Quality Assessment of Defeaters↩︎

As discussed in Subsection 3.2, manual validation of defeaters often exhibits significant inter-rater variability, reflecting inconsistencies in the subjective judgements of evaluators. This variability undermines the reliability and reproducibility of assurance case assessments, particularly in safety-critical domains where objective rigour is paramount. To address these issues, we have formalised a set of quality attributes for defeaters derived from established manual evaluation practices. These attributes are designed to be machine-processable, enabling standardised, automated assessment. Table 1 presents these attributes (left), their Interpretation Description (center), and how these attributes can be quantified by our NLP Metric (right) to enable machine-processable assessment. The attributes are: Correctness aligns automated ratings with human judgments using F1-scores. Stability ensures consistent labels for similar defeaters via BERT embeddings’ cosine similarity. Relevance reflects the defeater’s affiliation within the assurance case graph. Completeness verifies component presence through text parsing, informing field detection. Novelty assesses uniqueness with inverse similarity, diversifying features. This variability-driven approach shapes our automated review design, enhancing the reflective NLP features extraction for the machine learning models.

3.4 Automated Review Process↩︎

From the identification of the machine-processable quality attributes of the previous subsection 3.3, we derived the Automated Review Process. Figure 2 illustrates this process chart with three primary nodes: Manual Review Process, Automated Review Process, and Evaluation, arranged to depict the transition from human assessment to automated validation and its performance assessment. The details of each node and its subcomponents are provided below, along with their positions and functionalities within a machine-learning cycle. This cycle is clustered on the upper rim of the process chart figure.

3.4.1 Data Collection↩︎

The Manual Review Process, positioned as the leftmost node in Figure 2, encompasses the initial human-reviewed defeater data mentioned in subsection 3.1, comprising the defeaters for each component. The Human node includes two sub-categories: Consensus Defeaters, where reviewers agree (H1=H2), and Dissensus Defeaters, where they disagree (H1xH2), reflecting inter-rater variability as discussed in section Consensus and Dissensus 3.2. This bridges to the preprocessing node of the Automated Review Process, the central node in Figure 2.

3.4.2 Preprocessing↩︎

The embedding preprocessing employs the bert-base-uncased model [31] from Hugging Face, a transformer-based architecture with 12 layers and a 768-dimensional output representation, as the initial choice. To generate embeddings for defeaters, the input text is tokenized with a maximum sequence length of 512 tokens. The token-level embeddings produced by the model are aggregated into a single vector representation for each defeater by computing the mean across the token dimension. For creating the semantic relation of embeddings and the top-down hierachical structure, a directed graph of assurance case elements was constructed. This included nodes representing claims, evidence, strategies, contexts, or defeaters, and edges denoting hierarchical dependencies, facilitating the contextual analysis. This data preparation underpins the feature engineering.

3.4.3 Feature Engineering↩︎

Validity features were engineered to enhance classification, derived from the quality attributes outlined in section 3.3 Table 1. These features support the automated validation process, interpreting the expert judgment in a machine-quantifiable format. Thereby, it provides a transparent, reproducible factor to enable objective evaluation.

• Is Linked: A binary indicator (1 if the defeater references a linked assurance case element, 0 otherwise), derived from the Relevance and Completeness attribute to assess applicability to the case claims.

• Linked Element: Cosine similarity between the defeater’s BERT embeddings and the embedding of its linked assurance case element (if present, Is Linked=1), supporting Relevance by quantifying semantic alignment.

• Similarity Path: Cosine similarity between the defeater’s BERT embedding and the concatenated text of the shortest path from the assurance case graph’s root to the linked element. Thereby reflecting the Stability attribute by ensuring consistent predictions for related elements.

• Similarity Average: Mean cosine similarity between the defeater’s BERT embedding and the assurance case embeddings, enhancing Relevance by capturing overall contextual fit and its opposite Novelty to define whether defeaters are novel.

Following, we will introduce the models that were trained on the features presented.

3.4.4 Model Selection and Training↩︎

Two classification approaches were implemented using scikit-learn:

• Simple Classifier: The Logistic Regression (LR) employs multinomial softmax regression with L2 regularization for multiclass classification and balanced class weights compensating for data imbalances. LR serves as a baseline or complementary model. Secondary Support Vector Machines (SVMs) that find the optimal hyperplane to separate classes in a feature space were trained on BERT embeddings for the What and Why components. The hyperparameters (kernel: {linear, rbf}, C: {0.1, 1, 10, 100}, gamma: {scale, 0.001, 0.01, 0.1}) were tuned using GridSearchCV (e.g., all combinations of kernel, C, gamma) with 5-fold stratified cross-validation (or fewer if class sizes were small ex. Why 0 class ratings), optimizing for weighted F1-score. These hyperparameters are tuned to find the best SVM configuration for classifying defeater components, balancing fit and generalization. Class imbalance was addressed using class weights, and probabilities were calibrated with
  CalibratedClassifierCV (sigmoid method), to better reflect true confidence levels.

• Meta-Classifier: LR and SVMs combinations were trained on a feature set combining simple classifier probabilities and validity features (is linked, linked element, path similarity, similarity average) as introduced in subsection Feature Engineering 3.4.3. The same hyperparameter tuning and calibration process was applied as for the Simple Classifier. To address class imbalance, the Synthetic Minority Oversampling Technique SMOTE [32] was used. Additionally, a self-training approach using the SelfTrainingClassifier [33] with a confidence threshold of 0.9 was applied. This threshold determines the minimum confidence required to assign pseudo-labels to unlabeled data, reducing the risk of incorporating incorrect labels, enhancing model robustness.

3.4.5 Evaluation↩︎

With the method section 3 establishing the automated review process, we will apply it to the ACC and CERN datasets [29], evaluating the performance of the models based on the scikit-learn, providing F1 scores and cohen_kappa_score Agreement Rates.

Improving objectivity in the task performance by leveraging the graph-based embeddings for consistency, to prove the feasibility of AI-aided defeater validation for assurance synthesis.

4 Experiment↩︎

This experiment addresses the context gap between manual and automated validation of generated textual output, specifically defeaters, which are core elements in the argumentation of system safety. Defeaters require subjective review due to their role in challenging safety claims, and this study seeks to recreate such validation using Natural Language Processing (NLP) methods as elaborated in 1, to achieve objective, bias-independent results. The forthcoming experiments extend the foundational research executed by Viger et al. [7], which supplies the ACC and CERN assurance cases as well as their corresponding defeater datasets (ACC and CERN) [29].

4.1 Hypotheses and Research Questions↩︎

We hypothesize that a vector-based embeddings approach outperforms subjective expert reviews as outlined in [7], in terms of consistency (measured by F1-score) and objectivity (measured by agreement on dissensus defeaters). This addresses the inter-rater variability observed in Section 3.2. The experiment investigates two research questions:

1. RQ1: Is the validation of defeaters using an embedding vector-based method comparable to human expert evaluations in terms of accuracy and consistency?

2. RQ2: Does the embedding-based approach reduce subjectivity by providing an objective perspective on safety case defeater ratings?

4.2 Experimental Setup↩︎

The data [29] was obtained from the experimental results of Viger et al. [7], in which two reviewers collaborated with domain experts to independently analyse the 172 defeaters (75 for ACC and 97 for CERN) generated by GPT-4. These defeaters, covering nine argument fragments, were prompted by three strategies: background information, generic doubts, and context-specific challenges informed by Eliminative Argumentation (EA) [18]. Each defeater comprises “What” (the flaw), “Why” (the rationale), and “Where” (the affected claim or system component). The automated validation rates the “What” and “Why” components as 0 (non-compliant), 1 (generic), or 2 (reasonable), using expert annotations as the ground truth.

The methodology, detailed in Section 3.4, employs open-source BERT embeddings (“bert-base-uncased”) [9] to transform defeater texts and assurance case structures into 768-dimensional vectors. Logistic Regression (LR), Support Vector Machine (SVM), and meta-SVM classifiers predict ratings. Therby incorporating defeater text and structural context in the DAGs (e.g., claim hierarchies via networkx). The experiment proceeds in two rounds: (1) using defeater text alone, and (2) integrating structural context via features to enhance predictions. Performance is evaluated against dissensus annotations (H1\(\neq\)H2) using precision, recall, and F1-score, addressing the quality attributes in Table 1.

4.3 Data Preprocessing↩︎

The ACC and CERN datasets, stored in ACC-CERN_defeater_dataset.xlsx [29], contain 172 defeaters annotated by two human raters (H1, H2) for “What”, “Why”, and “Where” components on a {0, 1, 2} scale, yielding 688 samples (172 defeaters \(\times\) 2 reviewers \(\times\) 2 components). The dataset includes columns for Defeater ID, Defeater text, component ratings, and rater information. We split the data into:

• Consensus Sets: Defeaters where H1 and H2 agree on “What” and “Why” ratings (H1=H2). For ACC, 32 defeaters (31 initial, plus 1 dissensus for class balance); for CERN, 42 defeaters (40 initial, plus 2 dissensus).

• Dissensus Sets: Defeaters with mismatched ratings (H1\(\neq\)H2) or insufficient samples, used for evaluation (42 for ACC, 50 for CERN after transfers to consensus sets).

To address class imbalance in the training sets, we applied SMOTE [32] for the “What” and “Why” components. For each dataset, if a class had fewer than 5 samples, SMOTE was applied with a \(k\)-nearest neighbors parameter set to \(\min(3, n-1)\) a common default as it balances local information and generalization. Where \(n\) is the number of samples in the minority class. If \(n = 1\), SMOTE cannot be applied (since \(k = n-1 = 0\)), and the script falls back to generating synthetic defeaters using nlpaug with WordNet-based synonym augmentation. The final training sets, after SMOTE and synthetic augmentation, included: for ACC, 32 consensus defeaters supplemented with up to 5 synthetic samples per underrepresented class in “What” and “Why”; for CERN, 42 consensus defeaters similarly augmented. The evaluation sets remained unchanged in size, with 42 defeaters for ACC and 50 for CERN.

For the context-informed evaluation step 2), a directed graph of assurance case elements (claims, evidence, strategies) was constructed using networkx to capture dependencies, with nodes and edges derived from the safety case CSV files (ACC.csv, CERN_LHC.csv) [29]. Outputs, including raw datasets, consensus and dissensus sets, and training/evaluation splits, are saved as CSV files for reproducibility.

4.4 Embeddings and Features↩︎

We use the bert-base-uncased model [9] to generate
768-dimensional embeddings for defeater texts, computed as the mean of the last hidden state. Empty or invalid inputs return zero vectors. For the context-informed approach, a meta-classifier integrates the validity features introduced in section 3.4.3.

• Is Linked: Binary indicator

• Cosine Similarity: Between defeater & linked element.

• Path Similarity: Between defeater & path route.

• Average Cosine Similarity: Mean graph similarity.

These features are normalised using min-max scaling and combined with the probabilities of the simple classifiers reassembled for the meta-classifier predictions, as illustrated in the ‘meta-classifier node’ of Figure 2 in the method section 3.

4.5 Model Description↩︎

The experiment trains the two types of classifiers detailed in Subsection 3.4.4 for “What” and “Why” components:

• Simple Classifiers: LR and SVM models predict labels {0, 1, 2} using BERT embeddings of defeater texts.

• Meta-Classifiers: LR and SVM meta-classifiers refine predictions by integrating simple classifier probabilities and validity features as mentioned in the paragraph above. Labels are encoded with LabelEncoder, and min 2-fold cross-validation ensures balanced data training.

Feature importance is computed via coefficients for linear kernels and permutation importance for non-linear kernels.

4.5.1 Extraction Logic↩︎

Defeaters were matched to safety case nodes using regex to identify ID-based references (e.g., "Child Claims 5000", "Context 3001"). The logic targeted terms like "Text Strategy", "Text Context", "Parent", "Child", "Evidence", or numeric IDs in the "Where" field. Defeaters with descriptive ‘Where’ fields lacking structured references were rated 0 for positioning due to insufficient linkage to safety case nodes and were excluded from semantic analysis. Positional verification was performed by searching against the related safety case claims.

4.6 Evaluation↩︎

The evaluation is conducted in two rounds:

1. Text-Only: Simple classifiers predict ratings using only defeater text embeddings.

2. Context-Informed: Meta-classifiers incorporate structural DAG context (validity features) to improve predictions.

Performance is assessed against dissensus annotations (H1\(\neq\)H2) using precision, recall, and weighted F1-score, as defined in Table 1. The weighted F1 score is a metric that computes the F1 score for each class and then averages them, weighted by the number of samples in each class (support). Correctness is evaluated by comparing predictions to H1 and H2 ratings, while Stability, Relevance, Completeness, and Novelty are computed as described in Section 3.3. An ablation study was conducted to evaluate the different configurations for class imbalance mitigation: baseline (SVM simple SVM Meta), SMOTE, non-consensus data inclusion, self-training, and SMOTE with non-consensus data. We followed the default random state 42 for data splitting and initialization to produce the same result base across runs.

4.7 Limitations↩︎

The experiment faces several limitations:

• Data Quality: Inconsistent or incomplete annotations, particularly for the “Where” component, impact embedding reliability. Class imbalance in training (e.g., few “0” ratings) is mitigated with dissensus integration; the small sample sizes limit robustness.

• Human Bias: Variability in expertise for a domain influences annotations (Section 3.2), introducing bias in the training data, which is partially addressed by focusing on consensus-defeater training and extracting objective context from the structure within the assurance case.

• Model Complexity: BERT’s 768-dimensional embeddings capture rich semantics but could be enriched with more dimensionality. The LR and SVM classifiers may oversimplify nuanced defeater semantics based on the chosen parameters.

5 Results↩︎

4pt

5pt

4pt

3pt

This section assesses classifier configurations for automated defeater evaluation, addressing RQ1 (accuracy and consistency with human raters) and RQ2 (subjectivity reduction) from Section 4.1. To our knowledge, it is the first approach automating the evaluation of generated defeaters; for that, we tested Simple LR, SVM, and meta-classifiers (LR and SVM variants) on “What” and “Why” components against human annotations (H1, H2). The “Where” component was evaluated with the extraction logic explained in Subsection 4.5.1. We show below the weighted F1-score results in Table 2 for general match against both raters and in Table 3 for individual matches, additionaly listed the imbalance mitigation strategies introduced in Subsection 3.4.4; Table 4 and 5 show the agreement results and class distribution the SVM simple SVM meta model achieved.

5.1 RQ1↩︎

For RQ1, embedding-based classifier models demonstrate comparable consistency with human expert ratings, as evidenced by weighted F1 scores in Tables 2 and 3. For the ACC dataset, the SVM meta, LR simple models achieve the highest General What F1 score of 0.85, with a strong What H2 Individual F1 of 0.65. Simple SVM excels with What H1 Individual F1 of 0.65 and Why H2 Individual F1 of 0.60. For the CERN dataset, Simple LR achieves a general What/Why F1 score of 0.86. In contrast, LR Meta and SVM Simple lead with a What H2 individual F1 score of 0.71 and a Why H2 individual F1 score of 0.61. SVM Meta and SVM Simple top the Why H1 individual F1 score at 0.60, which is balanced for both raters and shows individual accuracy. The SVM Meta and SVM Simple models achieve an F1 score of 0.84 across the ACC and CERN datasets, thereby also supporting RQ1, which demonstrates the accuracy of the artificial ratings.

5.2 RQ2↩︎

For RQ2, classifier models reduce the impact of subjectivity in defeater validation, particularly in cases of significant disagreement, as demonstrated by the F1 scores in Table 3. Balanced ratings across reviewers highlight this effect. The imbalance mitigation method involving self-training with iteratively refined predictions does not equalise performance among raters. Synthetic Minority Oversampling Technique (SMOTE) with artificial minority labels performs better, but the remaining imbalance is worse than with the SVM Meta and SVM Simple approaches. Adding additional non-consensus defeaters does not improve the balance of performance against both reviewers. The subjectivity problem of independent raters is evident, shown by the class distributions of dissensus data in Table 4. A deeper analysis of the agreement rates for the best overall performance model (SVM Meta and SVM Simple) in Table 5 reveals further insights. The initial negative kappa values: ACC Why (-0.50), CERN Why (-0.40), and CERN What (-0.18) indicate worse-than-chance agreement in the human ratings. Using the automated review approach incorporating the model improved the values by approximately 40%. The ‘Where’ component did not improve much but is included to provide a complete overview. It is not representative due to its static filter mechanism, which was explained in Subsection 4.5.1, not considering general system descriptions.

5.3 Result Discussion↩︎

These findings suggest that embedding-based classifiers can mitigate the scalability and subjectivity issues of manual defeater validation, as outlined in Section 1. By leveraging structured domain-specific embeddings, the models capture nuances relevant to automotive (ACC) and nuclear (CERN) assurance cases (ACs) [7], enhancing defeater evaluation shown by the results in Table 2 and 3. However, the models lack human-like contextual understanding, occasionally missing critical domain-specific information (e.g., unmodeled risks in complex ACs) due to limitations in embedding patterns. This is noticeable in lower Individual F1 scores (e.g., 0.43 for ACC Why H1 from Table 3), where nuanced defeater evaluation requires deeper contextual insight. The size of the validation datasets with Dissensus Defeaters is limited. The performance of the models could be statistically supported by larger validation sets.

6 Conclusion↩︎

Results from the model comparison Table 2 confirm that the models, trained on embeddings from the domains, can predict defeater quality and reflect expert tendencies, addressing Research Question 1. The meta-classifier integrates assurance case features, thereby enhancing contextual understanding; yet, it still exhibits training imbalances. Despite balanced class weights, LR models, including the meta-classifier, tend to favor majority classes due to imbalanced data distributions, aligning predictions more with Reviewer 2’s labels than Reviewer 1’s. Notably, SVM models showed slight bias, demonstrating consistent performance against both reviewers in the individual performance comparison Table 3.

For Research Question 2, this automated review approach reduces subjective uncertainties, as demonstrated by the improvement in inter-agreement rates shown in Table 5. It provides an objective perspective on safety case defeaters by using semantic feature embeddings to achieve contextually informed evaluation. While this method has its advantages, like all supervised methods, it relies on expert-curated data and is not excluded from data change effects.
Future Work Advanced LLMs or improved embedding techniques, such as fine-tuning on domain-specific datasets, could address contextual understanding limitations, opening directions for research to enhance classifier performance and calibrate appearing generalization problems for the generation task. Furthermore, future work should explore hybrid expert/embedding-based approaches in the safety engineering process to enhance validation reliability and scalability for dynamic systems at runtime.

References↩︎