1 Introduction↩︎

Organizations increasingly deploy AI-powered conversational agents in regulated domains such as financial services, healthcare, and telecommunications [1]. These agents must comply with strict operational standards—identity disclosures, authorization protocols, and outcome classification—defined by Standard Operating Procedures (SOPs). Manual quality assurance is labor-intensive, subjective, and does not scale.

Frontier LLMs (GPT-4o, Claude Opus) offer strong general-purpose capabilities but present operational challenges for domain-specific structured evaluation: (1) high per-query costs at scale, (2) 4–10\(\times\) higher latency than self-hosted models, (3) accuracy degradation when domain rules are embedded in large context windows [2], and (4) data privacy concerns when sensitive transcripts leave organizational boundaries.

We demonstrate that a domain-adapted small language model (8B parameters) with hybrid post-processing can match or exceed frontier model accuracy on structured compliance evaluation while providing order-of-magnitude improvements in cost and latency. Our contributions are:

1. Data-efficient domain adaptation: Fine-tuning LLaMA 3.1 8B via LoRA [3] with only 219 examples, including targeted hard-negative augmentation for critical decision boundaries.

2. Hybrid neural-deterministic inference: A two-stage pipeline decomposing evaluation into contextual understanding (neural) and invariant rule enforcement (symbolic).

3. Production evaluation: Blind assessment on 53 production transcripts validated by human experts, with detailed cost and latency analysis against frontier API alternatives.

2 Related Work↩︎

Parameter-efficient fine-tuning. LoRA [3] introduces low-rank weight decompositions, reducing trainable parameters by orders of magnitude. QLoRA [4] adds 4-bit quantization. These have been applied to legal [5], medical [6], and financial [7] domains. We extend PEFT to multi-label compliance evaluation with extreme data scarcity.

Long-context limitations. Liu et al. [2] show 20–40 point accuracy drops when relevant information is in the middle of long contexts, motivating weight-encoded domain knowledge over context-window approaches.

Structured output generation. Constrained decoding [8] enforces syntactic validity but not semantic correctness. Our hybrid approach addresses both via fine-tuning and rule-based post-processing.

Data augmentation. We draw on contrastive learning principles [9] but operate at the input-output pair level, creating near-miss variants for critical compliance patterns.

3 Problem Formulation↩︎

3.1 Task Definition↩︎

Given a transcript \(T = \{(s_i, u_i)\}_{i=1}^{n}\) and disposition \(d\), produce a structured vector \(\mathbf{y} = (y_1, \ldots, y_{18})\) where each \(y_k \in \mathcal{Y}_k\). The 18 fields span four types:

Type I (Critical): Agent Disclosure \(y_{\text{AD}} \in \{P, F\}\), Payment Disclosure \(y_{\text{PD}} \in \{P, F, N/A\}\), Disposition \(y_{\text{D}} \in \{P, F\}\), Compliance Result \(y_{\text{CR}} \in \{P, F\}\).

Type II (Behavioral): Threats, Unprofessionalism, Transfer Accuracy, etc.

Type III (Informational): Product Name, Balance Due, Amount Collected.

Type IV (Free-text): Overall Feedback, Expected Responses.

3.2 Inter-Field Constraints↩︎

The Compliance Result is deterministically derived: \[y_{\text{CR}} = \begin{cases} \text{Fail} & \text{if } y_{\text{AD}}{=}F \vee y_{\text{PD}}{=}F \vee y_{\text{D}}{=}F \\ \text{Pass} & \text{otherwise} \end{cases}\]

Agent Disclosure requires exact substring matching against a canonical phrase \(\phi^{*}\): \[y_{\text{AD}} = \begin{cases} \text{Pass} & \text{if } \phi^{*} \sqsubseteq T \\ \text{Fail} & \text{otherwise} \end{cases}\]

4 Methodology↩︎

4.1 System Architecture↩︎

Fig. 1 shows the two-stage hybrid inference pipeline. The fine-tuned model generates a structured JSON evaluation (Stage 1), which is then validated and corrected by deterministic SOP rules applied over the original transcript (Stage 2).

4.2 Multi-Stage Data Curation↩︎

Stage 1: Noise removal (209 \(\to\) 199). Removed 10 examples with encoding artifacts, fragments \(<\)​30 characters, or system errors.

Stage 2: Hard-negative augmentation (199 \(\to\) 219). Agent Disclosure requires the exact phrase \(\phi^{*}\). Near-miss variants (e.g., omitting “AI” or substituting “automated”) must be classified as failures. We synthesized 20 adversarial examples from 10 near-miss templates across three violation categories:

• Partial omission: “I am a Collections agent” (missing “AI”)

• Synonym substitution: “I am an automated Collections agent”

• Complete omission: “I am here to help you”

Each hard negative preserves the full transcript context but modifies only the disclosure phrase, creating minimally contrastive pairs with opposite labels.

Stage 3: Exhaustive SOP audit (219 examples). Automated validation of all fields: regex-based amount extraction, authorization pattern matching, disposition-outcome consistency, and cross-field constraint enforcement per Eq. (1).

4.3 LoRA Fine-Tuning↩︎

For pre-trained weight \(W_0 \in \mathbb{R}^{d \times k}\), LoRA constrains the update: \[W = W_0 + \Delta W = W_0 + BA\] where \(B \in \mathbb{R}^{d \times r}\), \(A \in \mathbb{R}^{r \times k}\), \(r \ll \min(d, k)\). The forward pass: \[h = W_0 x + \frac{\alpha}{r} BA x\]

We apply LoRA to all seven projection matrices \(\{W_q, W_k, W_v, W_o, W_{\text{gate}}, W_{\text{up}}, W_{\text{down}}\}\) in each of the 32 transformer layers with \(r = 64\), \(\alpha = 128\), yielding 167.8M trainable parameters (2.05% of 8.2B).

4.4 Training Configuration↩︎

Table 1 summarizes the training setup. We use label masking to compute loss only on assistant (JSON output) tokens: \[\mathcal{L} = -\frac{1}{|\mathcal{A}|} \sum_{t \in \mathcal{A}} \log p_\theta(x_t \mid x_{<t})\] where \(\mathcal{A}\) is the set of assistant token positions and \(\theta\) denotes LoRA parameters. System/user tokens are masked with label \(-100\).

4.5 Deterministic Post-Processing↩︎

The post-processing layer \(g(\tilde{\mathbf{y}}, T) \to \hat{\mathbf{y}}\) enforces five rules:

R1. Agent Disclosure: exact substring match for \(\phi^{*}\), overriding the model unconditionally.

R2. Payment Disclosure: conjunction of authorization script detection (5 patterns) and post-authorization verbal confirmation.

R3. Disposition: bankruptcy dispositions \(\to\) Fail; transfer-disposition consistency enforcement.

R4. Compliance Result: recomputed via Eq. (1).

R5. Feedback regeneration when corrections are applied.

This decomposition assigns contextual reasoning (professionalism assessment, payment intent inference) to the neural model and deterministic rules (exact matching, cross-field arithmetic) to the symbolic layer.

5 Experimental Setup↩︎

5.1 Dataset↩︎

219 labeled transcripts from a production conversational AI system in financial services. Split: 188 train / 21 eval (90/10, seed=42). Token lengths: min 1,067, max 2,472, mean 1,350. Compliance distribution: 83 Pass (39.7%) / 126 Fail (60.3%). 20 hard-negative augmented examples (9.1% of dataset).

5.2 Blind Test Set↩︎

53 production transcripts completely unseen during training, each independently reviewed by a human compliance expert who marked predictions as “Correct” or “Incorrect” with error annotations.

5.3 Deployment↩︎

Inference on a single NVIDIA A100 80GB GPU ($134/day). Greedy decoding (temperature=0, repetition penalty=1.1, max 1024 tokens).

6 Results↩︎

6.1 Training Dynamics↩︎

Fig. 2 shows the loss curves. Training loss drops from 0.322 to 0.017 (94.7% reduction) while validation loss bottoms at 0.1305 (step 60) before rising. The \(\sim\)​9\(\times\) train-val gap at step 100 confirms that LoRA’s parameter efficiency and early stopping are both critical for this low-resource regime.

6.2 Blind Test Results↩︎

Table 2 presents the main results on 53 blind production transcripts.

Key findings: (1) Agent Disclosure achieves 100%, validating hard-negative augmentation. (2) JSON validity is 100%, versus typical 85–95% for zero-shot frontier models. (3) Disposition (90.6%) is the primary error source, with 5 of 9 errors being false positives.

6.3 Error Analysis↩︎

The model predicted Disposition=Pass for all 53 cases, never predicting Fail—revealing training distribution bias (67% correct dispositions). Targeted augmentation analogous to the successful Agent Disclosure hard negatives would likely address this.

6.4 Cost and Latency Analysis↩︎

Table 4 demonstrates 46–76% cost reduction versus per-transcript API alternatives and \(>\)​97% versus in-context learning. Monthly savings versus GPT-4o: $3,480.

6.5 Impact of Hard-Negative Augmentation↩︎

The 100% accuracy on Agent Disclosure—versus 90.6% on Disposition (which lacked targeted augmentation)—provides direct evidence that hard-negative augmentation at critical decision boundaries is highly effective. Just 20 synthetic examples (9.1% of the dataset) eliminated all disclosure classification errors.

7 Discussion↩︎

7.1 Small Models vs. Frontier APIs↩︎

Three mechanisms explain why our 8B-parameter model matches frontier models on this task:

Weight-encoded knowledge. Fine-tuning encodes domain rules in model weights, accessible at \(O(1)\) cost per inference. In-context learning encodes rules in the context window, subject to attention degradation [2].

Schema consistency. Fine-tuning on structured JSON with label masking teaches the exact output format, achieving 100% JSON validity versus 85–95% for zero-shot approaches.

Decision boundary calibration. Hard-negative augmentation directly calibrates boundaries for critical rules—impossible with zero-shot prompting alone.

7.2 The Hybrid Decomposition Principle↩︎

Our architecture embodies a general principle for regulated domains: decompose tasks into contextual reasoning (neural) and constraint enforcement (symbolic). The neural model need not be perfect on every field—deterministic fields are corrected at zero cost. This concentrates model capacity on genuinely difficult subproblems requiring transcript understanding.

7.3 Limitations↩︎

Statistical power. 53-example blind test yields \(\pm\)​10% confidence intervals at 95% level.

Disposition bias. The model never predicted Disposition=Fail, revealing training distribution bias addressable via targeted augmentation.

Class imbalance. Extreme imbalance (Threats: 98% Pass) limits minority-class evaluation.

Post-processing brittleness. Regex-based rules may fail on novel conversational patterns.

8 Conclusion↩︎

We demonstrated that a domain-adapted small language model (LLaMA 3.1 8B, 2.05% trainable via LoRA) fine-tuned on only 219 examples, combined with deterministic post-processing, achieves 83% human-validated accuracy on multi-label compliance evaluation—with 100% JSON validity, \(\sim\)​2-second latency, and $0.013 per evaluation. Compared to frontier API alternatives, this represents 2–5\(\times\) latency reduction, 46–76% cost savings, and full data privacy through on-premise deployment.

Our hard-negative augmentation strategy achieves 100% accuracy on the most critical field using just 20 synthetic examples, demonstrating that targeted data curation can substitute for model scale. The hybrid neural-deterministic architecture provides a general blueprint for deploying language models in regulated industries where both contextual understanding and strict rule compliance are required.

References↩︎