1 Introduction↩︎

Modern data pipelines face a persistent dilemma, large web corpora demand aggressive deduplication for efficiency and data quality, yet strict lexical filtering often removes useful semantic variants while semantic methods are usually too expensive at scale. This tension has become more important as recent language model studies show that data redundancy strongly affects memorization, optimization, and downstream quality [1].Similar concerns about resilience under dynamic operating conditions have also been studied in logistics and routing systems [2]. At the same time, open pretraining corpora continue to grow in size and heterogeneity, making scalable curation a first order systems problem [3].

Current solutions struggle because they optimize only one side of the trade off. Exact and near exact fingerprinting methods are fast, but they are brittle under paraphrase, template wrapping, and small character perturbations. Pure embedding based retrieval is more semantically aware, but its computational and thresholding costs are difficult to control across billions of documents.Related efforts on efficient large model inference also explore multi-stage compression and acceleration strategies that balance accuracy with deployment cost, including adaptive quantization, token pruning, and decode-time optimization [4]. These challenges are amplified in modern corpora that mix long documents, short snippets, boilerplate heavy pages, and containment relationships within the same pipeline [5].

We address this problem with SemHash LLM, a unified framework that combines learned semantic hashing, attention weighted lexical sketching, adaptive contrastive decision boundaries, and selective LLM based refinement. Our design breaks the conventional efficiency versus accuracy compromise by using a cascaded pipeline for pruning, multi granularity fusion for robustness, and uncertainty aware routing so that expensive reasoning is applied only to rare borderline cases.

2 Related Work↩︎

Recent work on retrieval and matching has shown that contextualized token interaction can improve fine grained similarity estimation beyond bag of words fingerprints. COIL demonstrated that exact lexical matching can be strengthened with contextualized token representations, while ColBERTv2 improved efficiency through lightweight late interaction for large scale retrieval [6] [7]. These studies motivate deduplication systems that preserve lexical precision while incorporating richer semantic structure.

Efficient semantic modeling has also advanced through compact encoders and contrastive representation learning.SimCSE further established that simple contrastive objectives can produce strong sentence embeddings with clear similarity structure [8]. Our method builds on these ideas by learning binary semantic projections and adaptive duplicate boundaries rather than relying on fixed embedding thresholds alone.

A third line of work concerns model based judgment and confidence aware evaluation. MT Bench and Chatbot Arena provided evidence that strong language models can act as practical judges for nuanced comparisons [9]. G Eval showed that carefully designed prompts can turn LLMs into reliable structured evaluators [10]. SelfCheckGPT highlighted the value of uncertainty signals when trusting generative model outputs [11]. We extend this direction to document deduplication by using LLM judgment only for uncertain candidate pairs identified by the automated pipeline.Related multi-agent LLM systems also show the value of hierarchical specialization and evidence-grounded reasoning in complex troubleshooting workflows [12].Related work on dynamic retrieval and selective tool invocation also shows that LLM-based systems can route between retrieved evidence and external APIs when evidence is insufficient, as demonstrated by DynaRAG [13].

3 Methodology↩︎

In this section, we present SemHash-LLM, a semantic-aware hierarchical deduplication framework that bridges lexical fingerprinting with deep semantic understanding through LLM integration. As illustrated in Fig. 1, our framework integrates five core components operating across multiple representational granularities—from character perturbations to token patterns to document-level semantics—with adaptive weighting determined by the redundancy type encountered. Semantic Projection Hashing learns locality-sensitive hash functions in the LLM embedding space, balancing similarity preservation with quantization loss and orthogonality constraints. Attention-Weighted MinHash addresses template pollution by extracting importance weights from transformer attention patterns into the consistent weighted sampling framework. Contrastive Boundary Learning provides self-supervised adaptive decision boundaries through margin-based optimization over document pair distributions. For borderline cases with high uncertainty, an LLM-as-Judge mechanism fuses structured model judgments with automated predictions via confidence-weighted ensemble. These components are unified within a Multi-Granularity Fusion Network employing gated attention to aggregate character-level, token-level, and semantic-level features.These components are unified within a Multi-Granularity Fusion Network employing gated attention to aggregate character-level, token-level, and semantic-level features [14]. Trillion-scale efficiency is achieved through a cascaded filtering pipeline combining Bloom filters, semantic hash blocking, and attention-weighted LSH. The framework handles five deduplication categories: template-wrapped near-duplicates, hot-spot documents with distribution skew, short texts with character perturbations, parent-child containment hierarchies, and high-frequency viral fragments.

3.1 Semantic Projection Hashing↩︎

Traditional locality-sensitive hashing methods such as MinHash and SimHash operate on discrete token or n-gram representations, fundamentally limiting their ability to capture semantic relationships. A document about “automobile manufacturing” and another about “car production” would produce entirely different hash signatures despite conveying nearly identical meaning. This limitation motivated our development of Semantic Projection Hashing, which learns hash functions that operate directly in the continuous LLM embedding space. The complete SPH training pipeline is depicted in Fig. 2.

3.1.1 Embedding Extraction and Efficiency Considerations↩︎

Given a document \(d\) with content \(c\), we first obtain a semantic representation through a pre-trained LLM encoder. The straightforward approach of using the final hidden state of a special classification token proved suboptimal in our experiments, as it tends to capture global document statistics rather than fine-grained semantic content. Instead, we employ mean pooling over all token representations:

\[\mathbf{e} = \frac{1}{T} \sum_{t=1}^{T} \mathbf{h}_t^{(L)} \label{eq:embedding}\tag{1}\]

where \(\mathbf{h}_t^{(L)} \in \mathbb{R}^{d}\) is the hidden state at the final layer \(L\) for token \(t\), and \(T\) is the sequence length. However, applying a full-scale LLM encoder to billions of documents would be computationally intractable.This efficiency bottleneck is consistent with roofline-style analyses of memory-bound LLM inference on Arm CPUs [15]. We address this through knowledge distillation, training a lightweight student encoder to mimic the semantic representations of a larger teacher model:

\[\mathcal{L}_{\text{distill}} = \text{KL}\left(p_{\text{tea}}(\mathbf{e}) \| p_{\text{stu}}(\mathbf{e})\right) + \alpha \|\mathbf{e}_{\text{tea}} - \mathbf{e}_{\text{stu}}\|_2^2 \label{eq:distill}\tag{2}\]

The combination of KL divergence and L2 distance proved crucial for maintaining both distributional consistency and point-wise accuracy. Using either term alone resulted in degraded performance on downstream deduplication tasks, with KL-only training producing embeddings that preserved ranking but lost magnitude information important for threshold-based decisions.

3.1.2 Learnable Hash Function Design↩︎

With semantic embeddings in hand, we need to convert them into compact binary codes suitable for efficient similarity search. We learn a family of \(K\) hash functions, each implemented as a hyperplane partition of the embedding space:

\[h_k(\mathbf{e}) = \text{sign}\left(\mathbf{w}_k^{\top} \mathbf{e} + b_k\right) \label{eq:hash}\tag{3}\]

where \(\mathbf{w}_k \in \mathbb{R}^{d}\) and \(b_k \in \mathbb{R}\) are learnable parameters. The complete hash code concatenates all \(K\) bits:

\[\mathbf{z} = [h_1(\mathbf{e}), h_2(\mathbf{e}), \ldots, h_K(\mathbf{e})] \in \{-1, +1\}^K \label{eq:code}\tag{4}\]

A critical design decision involves the training objective for these hash functions. Random hyperplanes, as used in traditional SimHash, provide theoretical guarantees but ignore the specific structure of the document embedding distribution. We instead optimize the hash functions to satisfy a semantic preservation property:

\[\Pr\left[h_k(\mathbf{e}_i) = h_k(\mathbf{e}_j)\right] \approx \frac{1 + \cos(\mathbf{e}_i, \mathbf{e}_j)}{2} \label{eq:property}\tag{5}\]

This is achieved through a similarity preservation loss that encourages similar documents to produce similar hash codes:

\[\mathcal{L}_{\text{sim}} = \sum_{i,j} \left(S_{ij} - \frac{1}{K}\mathbf{z}_i^{\top}\mathbf{z}_j\right)^2 \label{eq:sim95loss}\tag{6}\]

where \(S_{ij} = \cos(\mathbf{e}_i, \mathbf{e}_j)\) is the target semantic similarity. However, directly optimizing this objective encounters the gradient discontinuity problem due to the sign function. We address this through the straight-through estimator during training, using \(\tanh\) with temperature annealing as a smooth approximation:

\[\tilde{h}_k(\mathbf{e}) = \tanh\left(\beta \cdot (\mathbf{w}_k^{\top} \mathbf{e} + b_k)\right) \label{eq:smooth}\tag{7}\]

The temperature \(\beta\) is gradually increased during training, starting from 1.0 and reaching 10.0, which smoothly transitions from soft assignments to hard binary decisions.

3.1.3 Orthogonality and Information Maximization↩︎

A subtle but important issue arises when learning multiple hash functions jointly: without explicit constraints, they tend to converge to similar hyperplanes, producing redundant bits that waste the limited code capacity. We enforce diversity through orthogonality regularization:

\[\mathcal{L}_{\text{orth}} = \left\|\mathbf{W}^{\top}\mathbf{W} - \mathbf{I}_K\right\|_F^2 \label{eq:orth}\tag{8}\]

where \(\mathbf{W} = [\mathbf{w}_1, \ldots, \mathbf{w}_K] \in \mathbb{R}^{d \times K}\) stacks all hyperplane normal vectors. Additionally, we observe that hash codes often suffer from bit imbalance, where certain bits are predominantly positive or negative across the corpus, reducing effective capacity. We add a balance regularization term:

\[\mathcal{L}_{\text{bal}} = \sum_{k=1}^{K} \left(\frac{1}{n}\sum_{i=1}^{n} \tilde{h}_k(\mathbf{e}_i)\right)^2 \label{eq:balance}\tag{9}\]

The complete training objective for Semantic Projection Hashing combines these components:

\[\mathcal{L}_{\text{SPH}} = \mathcal{L}_{\text{sim}} + \beta_1 \mathcal{L}_{\text{orth}} + \beta_2 \mathcal{L}_{\text{bal}} + \beta_3 \|\mathbf{W}\|_F^2 \label{eq:sph95total}\tag{10}\]

where the final term provides standard L2 regularization to prevent overfitting. Through grid search on a validation set, we found \(\beta_1 = 0.1\), \(\beta_2 = 0.01\), and \(\beta_3 = 0.001\) to provide a good balance across diverse document types.

3.2 Attention-Weighted MinHash↩︎

While Semantic Projection Hashing excels at capturing document-level semantic similarity, it may overlook fine-grained lexical patterns important for certain deduplication types. Standard MinHash addresses this through n-gram fingerprinting but treats all n-grams equally, which proves problematic when documents contain substantial boilerplate. Consider a news article wrapped in website navigation menus, cookie consent notices, and advertisement blocks: the discriminative news content may constitute only a small fraction of the total text, yet standard MinHash would give equal weight to boilerplate n-grams. Fig. 3 illustrates how attention-derived importance weights.

3.2.1 Extracting Attention as Importance Signal↩︎

Our key insight is that transformer attention patterns naturally encode token importance: tokens receiving high attention from many other tokens are likely central to the document’s meaning, while peripheral tokens like stopwords and boilerplate receive diffuse, unfocused attention. We extract multi-head attention scores from the LLM encoder:

\[\mathbf{A}^{(l,h)} = \text{softmax}\left(\frac{\mathbf{Q}^{(l,h)}{\mathbf{K}^{(l,h)}}^{\top}}{\sqrt{d_k}}\right) \label{eq:attention}\tag{11}\]

Rather than using attention from a single layer, which we found to be noisy and inconsistent, we aggregate across all layers and heads to obtain a robust importance estimate:

\[\bar{a}_t = \frac{1}{LH} \sum_{l=1}^{L} \sum_{h=1}^{H} \sum_{t'=1}^{T} A_{t',t}^{(l,h)} \label{eq:agg95attn}\tag{12}\]

This aggregated attention score indicates how much other positions attend to position \(t\), serving as a proxy for content importance. An interesting finding during development was that earlier layers tend to capture syntactic patterns while later layers focus on semantic content. Uniform averaging across layers implicitly balances these complementary signals.

3.2.2 Weighted MinHash Formulation↩︎

For each character 5-gram \(g\) spanning token positions \([i, j]\), we compute an importance weight that combines attention-derived importance with inverse document frequency:

\[w(g) = \left(\frac{1}{j-i+1} \sum_{t=i}^{j} \bar{a}_t\right) \cdot \log\frac{N + 1}{\text{df}(g) + 1} \label{eq:gram95weight}\tag{13}\]

The IDF term down-weights common n-grams that appear across many documents, complementing the attention signal which identifies within-document importance. We then apply consistent weighted sampling to generate MinHash signatures that respect these importance weights:

\[h_k^{\text{AW}}(d) = \arg\min_{g \in G_5(d)} \frac{-\log(U_{g,k})}{w(g) + \epsilon} \label{eq:weighted95minhash}\tag{14}\]

where \(U_{g,k} \sim \text{Uniform}(0,1)\) is a random value determined by applying hash function \(k\) to gram \(g\), and \(\epsilon = 10^{-8}\) prevents numerical instability. This formulation ensures that high-weight grams have proportionally higher probability of being selected as the minimum, effectively focusing the signature on discriminative content.

The resulting signatures satisfy a weighted Jaccard estimation property:

\[\Pr\left[h_k^{\text{AW}}(d_1) = h_k^{\text{AW}}(d_2)\right] = J_w(d_1, d_2) \label{eq:wj95property}\tag{15}\]

where the weighted Jaccard similarity is defined as:

\[J_w(d_1, d_2) = \frac{\sum_{g \in G_5(d_1) \cap G_5(d_2)} \min(w_1(g), w_2(g))}{\sum_{g \in G_5(d_1) \cup G_5(d_2)} \max(w_1(g), w_2(g))} \label{eq:weighted95jaccard}\tag{16}\]

3.2.3 Adaptive LSH Configuration↩︎

Standard LSH uses fixed band and row configurations across all documents, but we observed that documents with different attention entropy profiles require different sensitivity settings. Documents with concentrated attention on few key phrases exhibit clearer duplicate boundaries, allowing for more aggressive band configurations, while documents with diffuse attention over many equally important segments need finer discrimination.

We quantify this through attention entropy:

\[H_{\text{att}}(d) = -\sum_{t=1}^{T} \tilde{a}_t \log \tilde{a}_t \label{eq:entropy}\tag{17}\]

where \(\tilde{a}_t = \bar{a}_t / \sum_{t'} \bar{a}_{t'}\) is the normalized attention distribution. The number of bands is then adapted:

\[b(d) = \text{clip}\left(b_0 + \lfloor \delta \cdot (H_{\text{att}}(d) - \bar{H}) \rfloor, b_{\min}, b_{\max}\right) \label{eq:adaptive95band}\tag{18}\]

where \(b_0\) is the base band count, \(\bar{H}\) is the corpus-wide mean entropy, and \(\delta\) controls adaptation strength. The clipping ensures configurations remain within reasonable bounds to prevent extreme sensitivity variations.

3.3 Contrastive Boundary Learning↩︎

Fixed similarity thresholds (e.g., 0.8 for Jaccard, 0.9 for containment) assume uniform document characteristics. In practice, optimal thresholds vary significantly: technical documents tolerate higher lexical overlap, while generic web content requires stricter thresholds.

3.3.1 Learning Adaptive Decision Boundaries↩︎

We learn thresholds from data through a contrastive approach. A projection network maps document representations to a space where duplicates cluster together:

\[\mathcal{L}_{\text{con}} = -\sum_{(i,j) \in \mathcal{P}^+} \log \frac{\exp(\text{sim}(\mathbf{z}_i, \mathbf{z}_j)/\tau)}{\sum_{k \neq i} \exp(\text{sim}(\mathbf{z}_i, \mathbf{z}_k)/\tau)} \label{eq:contrastive}\tag{19}\]

where \(\mathbf{z}_i = g_\theta([\mathbf{e}_i; \mathbf{f}_i^{\text{lex}}])\) combines semantic and lexical features, with temperature \(\tau = 0.07\). A boundary network then predicts duplicate probability:

\[p_{\text{dup}}(d_i, d_j) = \sigma\left(f_\psi\left(\mathbf{z}_i \oplus \mathbf{z}_j \oplus (\mathbf{z}_i \odot \mathbf{z}_j) \oplus |\mathbf{z}_i - \mathbf{z}_j|\right)\right) \label{eq:boundary}\tag{20}\]

where multiplicative interaction and absolute difference capture complementary similarity patterns.

3.3.2 Type-Specific Threshold Optimization↩︎

The five deduplication types exhibit distinct similarity distributions. We learn type-specific thresholds through margin-based optimization:

\[\begin{align} \mathcal{L}_{\text{margin}} = \sum_{t} \Bigg[ & \sum_{(i,j) \in \mathcal{P}_t^+} [m - (s_{ij} - \tau_t)]_+ \\ & + \sum_{(i,j) \in \mathcal{P}_t^-} [m - (\tau_t - s_{ij})]_+ \Bigg] \end{align} \label{eq:margin}\tag{21}\]

where \([x]_+ = \max(0, x)\) is the hinge function and margin \(m\) provides robustness against label noise.

3.3.3 Uncertainty Quantification↩︎

For ambiguous pairs, we quantify uncertainty through Monte Carlo dropout:

\[\hat{p}_{\text{dup}} = \frac{1}{M} \sum_{m=1}^{M} p_{\text{dup}}^{(m)}(d_i, d_j), \quad \sigma_{\text{dup}}^2 = \frac{1}{M} \sum_{m=1}^{M} \left(p_{\text{dup}}^{(m)} - \hat{p}_{\text{dup}}\right)^2 \label{eq:uncertainty}\tag{22}\]

High variance indicates conflicting signals across feature types; these uncertain cases are routed to a more sophisticated adjudication mechanism.

3.4 LLM-as-Judge Refinement↩︎

For borderline cases where automated methods exhibit high uncertainty, we leverage the reasoning capabilities of instruction-tuned large language models. This design was motivated by the observation that humans can often resolve ambiguous duplicates through careful reading and reasoning, capabilities that LLMs increasingly approximate. The dual-stream decision architecture with uncertainty routing is shown in Fig. 4.

3.4.1 Structured Prompt Engineering↩︎

Given a candidate pair \((d_i, d_j)\), we construct a structured prompt that presents both documents along with explicit deduplication criteria:

\[\mathcal{T}(d_i, d_j) = [\mathcal{I}; \text{``Document A:''}, c_i; \text{``Document B:''}, c_j; \mathcal{Q}] \label{eq:prompt}\tag{23}\]

The instruction \(\mathcal{I}\) encodes domain knowledge about duplicate types, and the query \(\mathcal{Q}\) requests a structured judgment. We experimented with various prompt formulations and found that including concrete examples of each duplicate type significantly improved judgment accuracy, consistent with findings in few-shot prompting literature.

The LLM generates a judgment with associated probability:

\[p_{\text{LLM}}(y | d_i, d_j) = \text{LLM}_{\text{gen}}(\mathcal{T}(d_i, d_j)) \label{eq:llm95judge}\tag{24}\]

where \(y \in \{\text{duplicate}, \text{distinct}, \text{contained}\}\). We extract confidence from the token probability of the first generated token, which we found to correlate well with judgment quality.

3.4.2 Confidence-Weighted Ensemble↩︎

The final decision fuses automated and LLM judgments through adaptive weighting:

\[p_{\text{final}} = \frac{\text{conf}_{\text{auto}} \cdot p_{\text{dup}} + \text{conf}_{\text{LLM}} \cdot p_{\text{LLM}}}{\text{conf}_{\text{auto}} + \text{conf}_{\text{LLM}}} \label{eq:ensemble}\tag{25}\]

where confidence scores are derived from prediction certainty:

\[\text{conf}_{\text{auto}} = 1 - 2\sigma_{\text{dup}}, \quad \text{conf}_{\text{LLM}} = |p_{\text{LLM}} - 0.5| \cdot 2 \label{eq:conf}\tag{26}\]

This formulation naturally trusts the more confident predictor while gracefully degrading when both are uncertain. Due to the computational cost of LLM inference, we only invoke the LLM judge when automated confidence falls below a threshold, processing approximately 3% of candidate pairs in practice.

3.5 Multi-Granularity Fusion Network↩︎

Different deduplication types require different analysis granularities: Type-C duplicates exhibit character-level perturbations, Type-A template pollution requires token-level analysis, and Type-D containment is best detected at the semantic level. Our Multi-Granularity Fusion Network hierarchically combines features from all three levels.

3.5.1 Character-Level Feature Extraction↩︎

Character-level features capture fine-grained perturbations through multi-scale convolutions:

\[\mathbf{f}_{\text{char}} = \text{MaxPool}\left(\text{ReLU}\left(\sum_{k \in \mathcal{K}} \mathbf{W}_k * \mathbf{E}_{\text{char}} + \mathbf{b}_k\right)\right) \label{eq:char}\tag{27}\]

where \(\mathbf{E}_{\text{char}} \in \mathbb{R}^{|c| \times d_c}\) is the character embedding matrix and \(\mathcal{K} = \{3, 5, 7\}\) specifies filter widths capturing perturbations from single character substitutions to short phrase variations.

3.5.2 Token-Level Feature Extraction↩︎

Token-level features capture lexical and template patterns through bidirectional recurrence:

\[\mathbf{f}_{\text{token}} = [\overrightarrow{\mathbf{h}}_T; \overleftarrow{\mathbf{h}}_1] = \text{BiLSTM}(\mathbf{E}_{\text{token}}) \label{eq:token}\tag{28}\]

The sequential inductive bias of LSTM naturally identifies template patterns at document boundaries.

3.5.3 Semantic-Level Feature Extraction↩︎

Semantic features project LLM embeddings through a two-layer network with GELU activation and layer normalization for training stability:

\[\mathbf{f}_{\text{sem}} = \text{LayerNorm}\left(\mathbf{W}_2 \cdot \text{GELU}(\mathbf{W}_1 \mathbf{e} + \mathbf{b}_1) + \mathbf{b}_2\right) \label{eq:sem}\tag{29}\]

3.5.4 Gated Fusion Mechanism↩︎

We employ gating to adaptively weight different granularities:

\[\mathbf{g} = \sigma\left(\mathbf{W}_g [\mathbf{f}_{\text{char}}; \mathbf{f}_{\text{token}}; \mathbf{f}_{\text{sem}}; \mathbf{f}_{\text{meta}}] + \mathbf{b}_g\right) \label{eq:gate}\tag{30}\] \[\mathbf{f}_{\text{fused}} = \mathbf{g} \odot [\mathbf{f}_{\text{char}}; \mathbf{f}_{\text{token}}; \mathbf{f}_{\text{sem}}] \label{eq:fused}\tag{31}\]

where \(\mathbf{f}_{\text{meta}}\) includes document length and vocabulary richness. Learned gate values show interpretable patterns: character-level features dominate for short documents, semantic features for long documents, and token-level features when template indicators are detected.

3.6 Cascaded Filtering Pipeline↩︎

Our cascaded pipeline progressively filters candidate pairs through increasingly sophisticated stages. Fig. 5 visualizes the filtering funnel.

3.6.1 Stage 1: Bloom Filter Pre-screening↩︎

A Bloom filter with \(k = 7\) hash functions provides \(O(1)\) exact duplicate n-gram testing at \(\text{FPR} < 0.01\), eliminating approximately 40% of the corpus:

\[\text{FPR} = \left(1 - e^{-kn/m}\right)^k \label{eq:bloom}\tag{32}\]

3.6.2 Stage 2: Semantic Hash Blocking↩︎

Semantic Projection Hashing with \(K = 64\) bits and Hamming radius \(r = 8\) partitions documents into buckets, reducing the candidate space by approximately 95%:

\[\mathcal{B}(\mathbf{z}) = \{d : \text{Hamming}(\mathbf{z}_d, \mathbf{z}) \leq r\} \label{eq:blocking}\tag{33}\]

3.6.3 Stage 3: Attention-Weighted LSH↩︎

AW-MinHash signatures further refine candidates through banded comparison, eliminating pairs with high Bloom filter overlap but low weighted Jaccard similarity (typically shared boilerplate):

\[\Pr[\text{candidate} | J_w] = 1 - \left(1 - J_w^r\right)^b \label{eq:lsh95prob}\tag{34}\]

3.6.4 Stage 4: Neural Verification↩︎

The Multi-Granularity Fusion Network performs final verification with decision threshold \(\tau_{\text{dec}}\). The compound pass rate \(\text{PassRate} = \prod_{s=1}^{4} r_s \approx 0.7\%\) (with \(r_1 = 0.6\), \(r_2 = 0.05\), \(r_3 = 0.3\), \(r_4 = 0.8\)), ensuring neural computation applies to less than 1% of all pairs.

3.7 Containment Detection and Fragment Removal↩︎

3.7.1 Semantic-Enhanced Containment↩︎

We augment lexical containment with semantic verification to avoid false positives from shared common phrases:

\[C_{\text{sem}}(d_i, d_j) = \frac{|G_5(d_i) \cap G_5(d_j)|}{|G_5(d_i)|} \cdot \mathbb{1}\left[\cos(\mathbf{e}_i, \mathbf{e}_j) > \tau_c\right] \label{eq:sem95contain}\tag{35}\]

Document \(d_i\) is contained in \(d_j\) only when both n-gram overlap exceeds 0.9 and semantic similarity confirms meaning preservation. The parent document maximizes total coverage:

\[d_{\text{parent}}^* = \arg\max_{d \in \mathcal{C}} \sum_{d' \in \mathcal{C} \setminus \{d\}} C_{\text{sem}}(d', d) \cdot |c_{d'}| \label{eq:parent}\tag{36}\]

3.7.2 Viral Fragment Detection↩︎

Type-E fragments are detected via suffix arrays over the concatenated corpus. For each high-frequency substring \(f\), a semantic relevance score distinguishes meaningful content from boilerplate:

\[\text{Rel}(f, d) = \cos\left(f_{\text{LLM}}(f), f_{\text{LLM}}(c_d \setminus f)\right) \label{eq:relevance}\tag{37}\]

Fragments with \(\text{Rel}(f, d) < \tau_{\text{rel}}\) averaged across host documents are removed; high-relevance fragments are preserved.

4 Evaluation Metrics↩︎

The overall performance is measured by a weighted composite score:

\[\text{Score}_{\text{final}} = 100 \times \sum_{t \in \{A,B,C,D,E\}} w_t \cdot S_t \label{eq:final95score}\tag{38}\]

where \(w_A = 0.25\), \(w_B = 0.15\), \(w_C = 0.20\), \(w_D = 0.20\), \(w_E = 0.20\).

For near-duplicate categories (Types A, B, C), deletion recall is defined as:

\[S_t = \frac{\sum_{g \in \mathcal{G}_t} \text{Del}_{\text{correct}}(g)}{\sum_{g \in \mathcal{G}_t} (|g| - 1)}, \quad t \in \{A, B, C\} \label{eq:deletion95recall}\tag{39}\]

Type-D containment accuracy measures parent preservation and child removal:

\[S_D = \frac{\sum_{g \in \mathcal{G}_D} |\text{Children}_{\text{del}}(g)| \cdot \mathbb{1}[\text{Parent}(g) \in \mathcal{D}^*]}{\sum_{g \in \mathcal{G}_D} |\text{Children}(g)|} \label{eq:containment95acc}\tag{40}\]

Type-E fragment removal rate:

\[S_E = \frac{\sum_{f \in \mathcal{F}} \text{Removed}(f)}{\sum_{f \in \mathcal{F}} \text{Occurrences}(f)} \label{eq:fragment95rate}\tag{41}\]

A circuit breaker enforces Type-F preservation:

\[\text{Score}_{\text{final}} = 0 \quad \text{if} \quad \frac{|\mathcal{D}_F \cap \mathcal{D}^*|}{|\mathcal{D}_F|} < 0.90 \label{eq:circuit95breaker}\tag{42}\]

5 Experiment Results↩︎

Experiments were conducted on 100GB web content from RedPajama using the MaxFrame platform. Table 1 presents the comparison with baseline methods and ablation results.

6 Conclusion↩︎

This paper presented SemHash-LLM, a semantic-aware deduplication framework integrating large language model capabilities with efficient hashing techniques. The framework achieves state-of-the-art performance across all five deduplication categories while maintaining computational efficiency at trillion-scale.

References↩︎