1 Introduction↩︎

Retrieval-augmented generation (RAG) systems ground their answers in externally retrieved evidence, making the retrieval corpus a direct attack surface [1]–[3]. Existing multi-passage attacks [1] generate adversarial texts without an explicit lexical-diversity constraint; the resulting passages exhibit incidental similarity from shared retrieval-condition components and remain susceptible to near-duplicate corpus hygiene. We contrast the polymorphic regime against a monomorphic baseline (lexically near-identical, mean Token Jaccard \(\approx 1.00\)) constructed here as the worst-case lexical-similarity limit (§3.3).

We introduce polymorphic sybil poisoning (Figure 1): \(S\) passages that jointly support an attacker-chosen target answer, maintain low pairwise token overlap (\(\tau_{\text{lex}}{=}0.8\) as a generation-time soft constraint; achieved mean 0.32, max 0.60; §3), and pass a verifier-LLM quality gate. The released benchmark fixes \(S{=}6\), chosen to dominate top-10 retrieval slots (§[sec:setup]). A lexical near-duplicate filter (Token Jaccard \(\geq 0.60\)) yields a binary separation between polymorphic (0%) and monomorphic (100%) clusters; embedding-based filtering admits no operating point matching this gap (Table 1, §3.3). A monomorphic–polymorphic ablation under Forced Exposure isolates the diversity dimension: monomorphic copies register only 4.0% as hijack while polymorphic surface diversity recovers 22.8%—a \(+\)​18.8pp amplification (95% paired bootstrap CI \([+15.4, +22.4]\), \(B{=}5{,}000\); §3.4). Because polymorphic sybils present diverse, seemingly independent evidence, reader outputs under attack distribute across four categories—gold, hijack, abstention, drift—rather than collapsing onto the hijack axis. ASR captures only hijack; the remaining 47–66% of output mass falls outside both ASR and ACC. We propose a failure-mode-aware evaluation framework with a four-way partition, paired clean-to-poison transitions, and a Forced Exposure protocol isolating reader-side conflict resolution.

1.0.0.1 Contributions.

(1) Polymorphic sybil poisoning, a coordinated attack class defeating lexical near-duplicate filtering and forcing a precision–recall trade-off on embedding-based filtering (§3, §3.3). (2) A four-way evaluation framework with instance-level paired clean-to-poison transition matrices and the Forced Exposure protocol isolating reader-side conflict resolution (§3.4, §5). (3) A frozen benchmark (3,145Q, 2,982 retained; cross-dataset on TriviaQA and 2WikiMultiHopQA) with an official evaluator and reference results across five readers, two retrievers, three conditions (§7, §8).

2 Related Work↩︎

2.0.0.1 Retrieval poisoning attacks.

[4] craft adversarial passages via HotFlip-style perturbations to maximize retrieval similarity. PoisonedRAG [1] injects \(N{=}5\) passages per target question and reports 90%+ ASR with Contriever+PaLM 2; its black-box variant generates passages via LLM stochastic sampling, yielding intermediate lexical diversity from sampling randomness rather than explicit pairwise enforcement. Extensions add trigger activation [2], backdoors [3], and single-text variants [5]; all measure effectiveness by ASR alone, leaving non-hijack failure modes unreported. Our polymorphic class adds an explicit pairwise-diversity constraint (\(\tau_{\text{lex}}\)) and verifier gate, producing a binary gap against lexical filters and shifting embedding-based detection onto a precision–recall trade-off (§3.3).

2.0.0.2 RAG evaluation frameworks.

Prior frameworks evaluate RAG along complementary axes: RGB [6], RAGChecker [7], SafeRAG [8], PoisonArena [9], and ClashEval [10]. RSB [11] sweeps 13 attacks \(\times\) 7 defenses reporting ACC, ASR, F1 independently—informationally equivalent to our four-way partition in static reporting. Our novel component is the instance-level paired transition matrix resolving which clean-correct outputs each attack converts into which failure mode, combined with adversarial multi-document injection and Forced Exposure.

2.0.0.3 Abstention in QA.

AbstainQA [12] and Abstain-QA [13] define benign-setting abstention frameworks without adversarial injection. We decompose their “answered incorrectly” category into targeted hijack (attacker-intended) vs.drift (undirected)—a distinction critical for defense design.

3 Polymorphic Sybil Attack↩︎

3.1 Threat Model↩︎

We adopt a gray-box threat model [1]: the attacker can inject passages into the corpus but not modify existing ones, has black-box access to reader and retriever, and aims to make the system output an attacker-chosen target \(t \neq g\) on question \(q\).

3.2 Formal Definition↩︎

In the released benchmark, \(S{=}6\), \(\tau_{\text{lex}}{=}0.8\), and \(\theta_{\text{qc}}(6){=}3\). \(\tau_{\text{lex}}{=}0.8\) is a generation-time soft constraint on the paraphrase-and-resample loop (groups exceeding it are regenerated); the retained distribution is much tighter (mean pairwise Token Jaccard 0.32, max 0.60). Passages are generated by Llama-3.1-8B-Instruct (temp\(=\)​0.8) and verified by Qwen2.5-72B-Instruct.

3.2.0.1 Diversity axes.

The definition operates on two axes: lexical surface form (Condition 2: \(\tau_{\text{lex}}\) pairwise enforcement) and group-level coordination (Condition 3: verifier rejection of groups whose constituent passages disagree on the target). Prior multi-passage attacks [1] produce surface paraphrastic variation incidentally from sampling randomness but constrain neither worst-case pairwise overlap nor within-group target-support consistency. Semantic and evidence-source diversity emerge as paraphrase-loop byproducts (§9).

3.3 Why Polymorphic Sybils Are Harder to Defend↩︎

We compare polymorphic sybil groups (\(\tau_{\text{lex}}{=}0.8\)) against the monomorphic worst-case baseline (\(S{=}6\), verifier gate, diversity filter disabled, 500-question subset; mean Token Jaccard 1.00) under four corpus-level detection strategies (Table 1). Token Jaccard \(\geq 0.60\) yields a binary gap (mono 100%, poly 0%); E5 cosine flags monomorphic at 100% at every threshold but trades polymorphic detection sharply against FPR on legitimate same-topic pairs (capturing the residual 14.2% at \(\geq 0.95\) raises FPR from 0.2% to 1.8%, a 9\(\times\) increase). Perplexity and query-paraphrase flag both variants at \(\geq 95\)% and do not separate them.

3.4 Monomorphic vs.Polymorphic Outcome Redistribution↩︎

Beyond detection evasion, polymorphic diversity affects how the reader responds to the attack. Table 2 compares four-way outcome distributions under Forced Exposure on the 500Q ablation subset (Qwen2.5-72B reader). The two retrievers agree to three decimals (Forced Exposure pins placements deterministically); generalization is discussed in §9 (iv).

The two variants separate on all four channels under Forced Exposure (all CIs excluding zero): polymorphic groups achieve \(+\)​18.8pp higher hijack, suppress abstention by \(-\)​7.4pp, reduce drift by \(-\)​14.2pp, retain marginally more gold (\(+\)​2.8pp). The same six target positions delivered through monomorphic copies register only 4.0% as hijack despite 83.6% of outputs deviating from gold (mass absorbed by abstention 38.2% and drift 41.4%); polymorphic surface diversity recovers 22.8%—a 5.7\(\times\) amplification of ASR-visible hijack.

4 Benchmark Construction↩︎

4.1 Task, Pipeline, and Release Statistics↩︎

The benchmark operates in a grounded QA setting: for each question \(q\), the system retrieves passages from a fixed corpus \(C\) and produces \(\hat{y}\); evaluation uses strict exact match (EM) against canonicalized gold alias sets \(G(q)\). The attacker selects a target answer \(t \notin G(q)\), constructs a polymorphic sybil group (Definition 1), and injects all \(S\) passages into \(C\) without access to reader parameters, retriever internals, or the evaluator. Target answers are sampled from dataset-conditional plausible alternatives; the target-sampling protocol is released with the code package.

4.1.0.1 Source datasets and splits.

Host questions are drawn from validation/dev splits of four open-domain QA datasets (seed\(=\)​42 fixed random samples): NQ-open (nq_open, 3,610\(\to\)​1,145) [14], [15]; HotpotQA distractor (hotpot_qa/distractor, 7,405\(\to\)​2,000) [16]; TriviaQA (trivia_qa/unfiltered.nocontext val, 11,313\(\to\)​2,000) [17]; 2WikiMultiHopQA (Alab-NII dev, 12,576\(\to\)​3,000) [18]. Exact identifiers and text are frozen in the released manifest with SHA-256 checksums; reproducibility does not depend on upstream snapshot stability.

The main manifest is built on NQ and HotpotQA via a three-stage filter pipeline (all stage manifests released): (1) qc-v2 (\(\tau_{\text{lex}}{=}0.6\) \(+\) semantic verifier \(+\) target–gold disjoint check): 11,015 \(\to\) 4,022; (2) class balance (seed\(=\)​42, HotpotQA capped at 2,000): 4,022 \(\to\) 3,145 (1,145 NQ \(+\) 2,000 HotpotQA); (3) strict release (\(\tau_{\text{lex}}{=}0.8\) \(+\) \(\theta_{\text{qc}}(6){=}3\)): 3,145 \(\to\) 2,982 (94.8%; 96.6% NQ, 93.8% HotpotQA; mean pairwise Jaccard 0.32, max 0.60). Generator: Llama-3.1-8B-Instruct (\(Q8\_0\), temp\(=\)​0.8); verifier: Qwen2.5-72B-Instruct (\(Q4\_K\_M\), temp\(=\)​0)—distinct families to avoid generator–verifier collapse; 5.2% rejection at the strict release filter indicates substantive filtering.

4.1.0.2 Cross-dataset validation layer.

For generalizability we apply the same pipeline to TriviaQA (2,000\(\to\)​1,398; factoid; 69.9% retention, below 89.9–96.6% elsewhere due to short factoid prompts giving the verifier more headroom on ambiguous gold) and 2WikiMultiHopQA (3,000\(\to\)​2,696; multi-hop). Released as a separate layer (§8.3); a 2Wiki full-scale 18,000-passage QC audit is also released.

5 Evaluation Framework↩︎

5.1 Operational Partition and Official Evaluator↩︎

Let \(\hat{y}\) denote the canonicalized evaluator output, \(G\) the canonicalized gold alias set, \(T\) the canonicalized target alias set, and \(A\) the abstention marker set. We define four mutually exclusive and collectively exhaustive categories: gold retention (\(\hat{y} \in G\)), targeted hijack (\(\hat{y} \in T \setminus G\)), abstention (\(\hat{y} \in A\)), and third-answer drift (\(\hat{y} \notin G \cup T \cup A\)). This extends AbstainQA’s four-scenario framework [12] to the adversarial setting by decomposing “answered incorrectly” into targeted hijack (attacker-intended) and drift (undirected), a distinction critical for defense evaluation.

The official evaluator applies strict EM against canonicalized gold and target alias sets (lowercase, article strip, non-alphanumeric removal) uniformly across tracks (full canonicalization in App. 10.3). When an extracted answer contains both gold and target aliases the evaluator raises a conflict flag resolved via first-entity rule; flag rates are a descriptive signal varying by two orders of magnitude across readers (per-reader breakdowns in output CSVs). A lenient alias-substring variant raises ACC by 4.6–13.4pp but shifts paired metrics by at most \(\pm\)​2pp.

5.2 Metrics, Paired Analysis, and Forced Exposure↩︎

5.2.0.1 Metrics and statistics.

ACC (full-manifest gold), ASR (attack-condition fraction in \(T\setminus G\) on the retained-poison subset), abstention rate, drift rate; paired metrics (hijack, drift, attack-induced abstention) on the clean-correct, retained-poison subset. We report 95% percentile bootstrap CIs (\(B{=}5{,}000\)) on per-cell rates and paired bootstrap CIs on differences (mono–poly \(\Delta\), retriever \(\Delta\)); retriever comparisons across 5 readers \(\times\) 3 tracks (15 tests) use Bonferroni \(m{=}15\), \(\alpha{=}0.0033\). Clean and attack/forced outputs are matched on question identifier; conditional transition matrices on the clean-correct subset quantify what fraction shifts to each failure mode (§8.1).

5.2.0.2 Forced Exposure.

The \(S{=}6\) sybil passages are placed at protocol-defined top-10 positions alongside gold-supporting passages drawn under a shared rule from the retained candidate pool. This eliminates retrieval stochasticity and measures reader behavior when poison and gold evidence coexist; empirical retrieval-reader decoupling is reported in §8.2. Output schema contracts (raw / eval / final / abstain / conflict-flag fields per question) are specified in the released evaluator code.

6 Experimental Setup↩︎

The released benchmark fixes the sybil group size at \(S{=}6\), chosen to dominate top-10 retrieval slots under attack while leaving room for organic gold-supporting passages elsewhere in the candidate pool; \(S\)-sensitivity is left as future work.

Five main LLM readers span 7B–120B parameters across open and proprietary families: Qwen2.5-7B/72B-Instruct [19], Llama-3.1-70B-Instruct [20], GPT-OSS-120B (Unsloth release; Harmony reasoning tags stripped post-hoc), and GPT-4o-mini. Open-source readers run Q4_K_M GGUF via llama.cpp; GPT-4o-mini uses the OpenAI Chat Completions API. All readers share a byte-identical few-shot prompt (SHA-256 App. 10.3); decoding is temperature\(=\)​0, max_tokens\(=\)​128 except GPT-OSS-120B (max_tokens\(=\)​256 for Harmony-format reasoning; truncation analysis §8.3).

Retrieval uses Wikipedia DPR 100-word split [21] (21M passages). Main pipeline: BM25\(\to\)ColBERTv2 [22] (top-1000\(\to\)top-10); cross-retriever: E5-large-v2 [23] (FAISS) top-200 \(\to\) ms-marco-MiniLM-L6-v2 \(\to\) top-10. Three conditions: clean (organic); attack (frozen \(P_q\) prepended pre-rerank, displacing lowest-ranked organic); Forced Exposure (top-10 = \(S{=}6\) sybil \(+\) 2 gold \(+\) 2 organic filler; slot-assignment rule in the harness). The 6:2:2 composition is fixed a priori; ratio sensitivity is future work.

7 Results↩︎

7.0.0.1 Outcomes redistribute beyond hijack under both attack and Forced Exposure.

Across five readers and both retrievers (Table 3, Fig. 2), clean-to-attack gold drops span 8.5–15.5pp; hijack gains \(+\)​13.5 to \(+\)​20.9pp; abstention shifts \(-\)​1.9 to \(-\)​11.5pp; drift shifts \(+\)​1.3 to \(+\)​5.3pp. Abstention and drift together account for 47.4–65.9% of outputs under attack—unmonitored by ACC\(+\)ASR. Under Forced Exposure the ASR-blindness sharpens: two readers can land at near-identical hijack yet differ sharply on which non-target channel absorbs the displaced mass (Qwen2.5-72B vs.GPT-OSS-120B on Hotpot+NQ\(\times\)E5+CE: hijack 0.212/0.210 but abstention 0.360/0.195 and drift 0.238/0.410—a 16.5pp/17.2pp inversion at near-identical hijack). The attack\(\to\)forced shift moves mass primarily into abstention or drift in reader-specific patterns; ASR-visible hijack remains tightly clustered.

8 Analysis↩︎

8.1 Paired Clean-to-Forced Transitions↩︎

Three reader profiles emerge (Table 4, Fig. 2). Abstention-dominant (Qwen2.5-7B, Llama-3.1-70B, GPT-4o-mini, Qwen2.5-72B): forced abstention 36.0–48.7% on Hotpot+NQ over drift 19.4–23.8%, with the same ordering replicated on TriviaQA and 2Wiki. Drift-dominant (GPT-OSS-120B): drift 41.0% on Hotpot+NQ over abstention 19.5%, drift 31.1% on TriviaQA over abstention 12.0% (consistent with 14–23% Harmony-format truncation, §8.3). Highest gold preservation: Qwen2.5-72B preserves the most gold on Hotpot+NQ and TriviaQA (18.9% and 29.0%); 2Wiki gold is uniformly low (6.9–10.4%). Aggregate ACC drops obscure these splits: Qwen2.5-72B vs. GPT-OSS-120B differ by 0.2pp in forced hijack but 16.5pp in abstention and 17.2pp in drift on Hotpot+NQ\(\times\)E5+CE—a qualitative profile difference invisible to ASR\(+\)ACC.

8.2 Retrieval Substrate Decoupling↩︎

Forced Exposure pins sybil and gold placements deterministically, so \(|\Delta(\text{E5+CE} - \text{ColBERT})| = 0\) holds by construction; we verify this as a sanity check (\(|\Delta| \leq 0.004\) across error-free cells, most exactly \(0.000\)). The substantive finding is on clean/attack: E5+CE yields 1.9–4.9pp higher gold rates than ColBERT (Bonferroni \(m{=}15\), \(\alpha{=}0.0033\)), reflecting the retrieval choice’s effect on which passages reach the reader before the placement protocol takes over.

8.3 Cross-Dataset Validation↩︎

Three replicated findings (Table 5): (i) forced hijack ranges across the three datasets span 2.8–16.0pp per reader (Llama-3.1-70B 2.8pp, Qwen2.5-72B 6.2pp, GPT-OSS-120B 12.4pp, Qwen2.5-7B 15.8pp, GPT-4o-mini 16.0pp)—a spectrum from near-invariance to substantial dataset sensitivity; (ii) TriviaQA produces the highest mean forced hijack (28.2%; Hotpot+NQ 19.6%, 2Wiki 17.6%), suggesting factoid surfaces leave readers more susceptible than multi-hop chains; (iii) retriever decoupling replicates (\(|\Delta(\text{E5+CE} - \text{ColBERT})| \leq 0.004\) across error-free cells). Reader profiles also survive: GPT-OSS-120B shows the highest drift on every dataset (29.7–41.0%), Llama-3.1-70B the highest abstention (48.7–68.5%), Qwen2.5-72B the highest gold on Hotpot+NQ and TriviaQA (18.9% and 29.0%; 2Wiki gold uniformly low, 6.9–10.4%).

8.3.0.1 GPT-OSS-120B decode-budget truncation.

GPT-OSS-120B’s Harmony output exceeds the 256-token decode budget on 14.3–23.3% of Hotpot+NQ, 10.3–12.7% of TriviaQA, and 14.1–19.5% of 2Wiki queries; the evaluator scores these truncated outputs as drift, contributing to GPT-OSS-120B’s elevated drift fraction. We treat this as a reader-specific output behavior under a fixed decode budget; other readers at max_tokens\(=\)​128 do not exhibit comparable truncation.

8.4 Clean Drift Audit↩︎

To separate genuine reader reasoning errors from evaluator-side extraction artifacts, an LLM-based classifier (Qwen2.5-72B-Instruct, also used as verifier and a main reader; §9 (v)) labels 1,356 clean-condition drift samples on the reconciled manifest as GENUINE (reasoning error), EXTRACTION (gold present but canonical extraction fails), or DATASET (gold annotation ambiguity); automatic composition 89.6%/3.2%/7.0% (Table 8, App. 11). One author re-labeled a 250-instance stratified subsample blind to the classifier; human composition 84.4%/15.2%/0.4% with raw agreement 82.8%, Cohen’s \(\kappa{=}0.262\) (fair, Landis–Koch). Excluding GPT-OSS-120B’s truncated reasoning outputs raises raw agreement to 88.5% but lowers \(\kappa\) to 0.194 due to rising expected-by-chance agreement (76.7% to 85.7%) under a more class-imbalanced GENUINE-dominant distribution, not deteriorating alignment (Table 7). The classifier under-detects EXTRACTION (auto 3.2% vs.human 15.2%) and over-attributes DATASET, particularly on GPT-OSS-120B (Harmony-format truncation scored as dataset-ambiguity: auto 19–21% DATASET on GPT-OSS subsample vs.human 0%); the audit’s GENUINE fraction remains an upper bound. Cross-reader drift comparisons in Table 3 use the four-way evaluator and are unaffected by classifier bias.

9 Limitations and Ethics↩︎

9.0.0.1 Limitations.

(i) Single attack class with two enforced diversity axes (lexical pairwise overlap, group-level target-support consistency); semantic and evidence-source diversity emerge as paraphrase-loop byproducts; generator/verifier are both LLMs (partially controlled by distinct model families). (ii) Forced Exposure measures reader-side conflict resolution rather than deployed effectiveness; the fixed 6:2:2 (sybil:gold:filler) composition maximizes contention and ablation channel-shift magnitudes (§3.4) depend on this ratio. (iii) NQ/HotpotQA contamination risk, partially mitigated by cross-dataset validation. (iv) Ablation uses only Qwen2.5-72B on 500Q against a worst-case monomorphic baseline; generalization across readers and intermediate baselines is open. (v) Qwen2.5-72B serves three roles (verifier, reader, drift-origin classifier); verifier–reader overlap is partially controlled by three non-Qwen readers, classifier role is not (human-classifier \(\kappa{=}0.262\) fair, raw 82.8%; raw 88.5%, \(\kappa{=}0.194\) excl.GPT-OSS; non-Qwen replication deferred to camera-ready). (vi) GPT-4o-mini\(\times\)ColBERT attack retains 9.1% MCAR API errors (App. 12); paired analysis fixes all three conditions to \(n_{\text{eff}}{=}2{,}711\), inducing \(\leq 0.4\)pp deviation from retriever equality in forced cells. GPT-OSS-120B cells contain 3–6 [ERROR] outputs per cell from Harmony-format parsing (evaluated as drift by the official rule). (vii) A small fraction of retained groups contain a residual generator-preamble fragment (one of six positions): main 32/2,982 (1.07%), TriviaQA 10/1,398 (0.72%), 2Wiki 34/2,696 (1.26%). Manifest-level Jaccard unaffected within \(\pm 0.001\). Detector \(+\) flagged IDs in audit/detect_preamble_fragments.py. (viii) Two manifest-quality processes for v1.0: (a) target metadata reconciliation reassigned target answers to the sybil-dominant entity in \(\sim\)​33% of qc-v2 groups (3,181 across four manifests; per-instance labels released); (b) release-stage audit (\(n{=}100\) random, seed\(=\)​4202) finds 76% strict / 92% majority / 4% degenerate (§10.6); strict subset hijack averages \(+\)​3.6pp above full-sample, matching the polymorphic ablation \(\Delta{=}+18.8\)pp direction (§3.4). All paper statistics use the verifier-accepted reconciled manifest; per-instance audit labels released for downstream filtering.

9.0.0.2 Ethics.

Released artifacts contain adversarial passages for robustness evaluation; the attack class adds no capabilities beyond prior work [1], [4]. We release evaluation-time materials and prompts but not orchestration tooling; the Datasheet (§14) excludes production red-teaming and attack-model training.

9.0.0.3 Release.

The complete benchmark will be publicly released upon publication. Benchmark proper (code repository): frozen manifest (3,145Q, 2,982 retained, \(\sim\)​8.6 MB), four-way evaluator, paired-transition utilities, generator/verifier prompts with SHA-256, cross-dataset and ablation layers, drift audit, Datasheet (§14), Croissant metadata (manifest/croissant.json). Prebuilt retrieval indexes (BM25 \(\sim\)​11 GB, E5 FAISS \(\sim\)​81 GB) over Wikipedia DPR 100w—redistributed for reproducibility convenience. ColBERTv2 weights load from the official upstream checkpoint. Repository and dataset URLs will be added in a future revision.

10 Artifact Specifications and Reproducibility↩︎

10.1 Verifier Configuration↩︎

Verifier \(V\) is Qwen2.5-72B-Instruct GGUF (Q4_K_M; bartowski/Qwen2.5-72B-Instruct-GGUF) served via llama.cpp (context 8,192; temperature\(=\)​0; max_tokens\(=\)​32). The prompt (SHA-256 0a4a02c9…) elicits a per-passage structured binary judgment (supports_gold, supports_target). For a candidate group \(P{=}\{p_1,\ldots,p_S\}\), let \(n_{\text{gold-only}} = |\{i: \texttt{supports\_gold}_i \land \lnot \texttt{supports\_target}_i\}|\); we retain \(P\) iff \(n_{\text{gold-only}} < \theta_{\text{qc}}(S)\), with \(\theta_{\text{qc}}(6){=}3\) (implementation: verifier_acceptance.py).

10.2 Polymorphic Sybil Generation Pipeline↩︎

Sybil passages are generated by Llama-3.1-8B-Instruct (Meta-Llama-3.1-8B-Instruct-Q8_0.gguf) via a llama.cpp completion endpoint, deliberately distinct from the Qwen2.5-72B verifier to avoid generator–verifier identity collapse (the observed 5.2% verifier rejection rate, well above 0%, indicates substantive filtering rather than rubber-stamping). For each manifest question \(q\), the generator produces \(S{=}6\) candidate passages supporting a target answer \(t \notin G(q)\); target answers are sampled from dataset-conditional plausible alternatives (target-sampling code and seed released). The generator prompt (SHA-256 37eb61bc…) instructs the model to produce lexically diverse passages supporting the target while avoiding verbatim copies; an LLM-based paraphrase stage (temperature\(=\)​0.8, default top-p, held constant across questions and datasets) yields lexically distinct variants from a shared semantic template.

10.3 Reader Configurations↩︎

All readers share a common few-shot QA template (COMMON_QA_SYSTEM_PROMPT) with three concise demonstrations (1–3 token answers) plus an Unknown demonstration anchoring abstention; byte-identical across readers, SHA-256 bfda716d…. Local GGUF readers use llama.cpp text completion; GPT-OSS-120B uses llama.cpp chat completion with Harmony-format post-processing; GPT-4o-mini uses the OpenAI Chat Completions API. Each inference run emits per question the five canonical fields answer_raw / answer_eval / answer_final / abstain / conflict_flag supporting deterministic re-scoring (Table 6).

4pt

\(^{\dagger}\)Observed snapshot that the gpt-4o-mini alias resolved to during our runs; the alias itself is not guaranteed to route to this snapshot for future callers (see §9).

10.4 Pipeline Specification↩︎

10.4.0.1 Retrieval.

Corpus: wikipedia-dpr-100w, 21,015,324 passages (DPR conventions). Main substrate: Lucene BM25 (\(\sim\)​11 GB index) top-1000 \(\to\) ColBERTv2 (dim 128; compressed MaxSim index from HF hub) rerank \(\to\) top-10. Cross-retriever substrate: E5 (intfloat/e5-large-v2, dim 1,024, FAISS IndexFlatIP, \(\sim\)​81 GB) top-200, reranked by the ms-marco-MiniLM-L6-v2 cross-encoder (\(\sim\)​22.7M params) \(\to\) top-10.

10.4.0.2 Sybil injection.

Attack: frozen group \(P_q\) prepended to the candidate pool before reranking, displacing the lowest-ranked organic candidates to maintain the retriever-specific candidate cap (top-1000 for BM25, top-200 for E5). Forced Exposure: top-10 slots are deterministically composed as 6 sybil passages, 2 gold-supporting passages drawn from the retained candidate pool, and 2 organic filler passages from the same pool; the slot-assignment rule (consistent with §[sec:setup]) is applied identically across retrievers and readers and is included in the released evaluation harness.

10.4.0.3 Note on the sybil_in_top10 diagnostic.

The released artifacts include a per-query counter sybil_in_top10 that records, for each query, the number of sybil passages present in the top-10 slots delivered to the reader. Under attack on Hotpot+NQ, the mean of this counter is 4.000 for ColBERT (100% of queries contain \(\geq 1\) sybil) and 3.75 for E5+CE (91.9% of queries contain \(\geq 1\) sybil), confirming that both retrievers expose readers to the injected sybils at comparable rates. Gold-supporting passages reach the top-10 in 90.2% of queries under both retrievers. Under Forced Exposure the count is 6 by construction (sybil placement is deterministic per the protocol; §5.2); the released runner does not recompute the diagnostic in this condition and emits 0.000 as a not-applicable placeholder in the per-query log—this value indicates “not measured,” not “not exposed.”

10.5 Checksums and Licenses↩︎

10.5.0.1 Checksums.

Prompt-level SHA-256 is fixed: verifier 0a4a02c9…, common reader bfda716d…, generator 37eb61bc…. File-level checksums for all data and software artifacts are committed as SHA256SUMS in the release repository (verification: sha256sum -c SHA256SUMS).

10.5.0.2 Licenses.

Main-release data (manifest, poison artifacts, monomorphic baseline) under CC BY-SA 4.0 (share-alike inherited from NQ and HotpotQA); software under MIT; TriviaQA samples and 2WikiMultiHopQA samples under Apache 2.0 (inherited from upstream). Users redistributing must comply with the upstream source license in addition.

10.6 Audit, Compute, and LLM Usage↩︎

10.6.0.1 Manual audit.

Authors conducted blind audits at two pipeline stages. Stage 1 (qc-v2): 100 accepted and 50 rejected groups from the qc-v2 pool (\(n{=}4{,}022\), post semantic-verifier; seed\(=\)​42) scored under a framing-aware criterion (passage substantively presents \(t\) as the answer to \(q\) in context); composition: 61% strict, 86% majority, 12% degenerate. Stage 2 (release): an independent 100-group audit drawn directly from the released manifest (frozen_manifest.jsonl, \(n{=}2{,}982\); seed\(=\)​4202, balanced 51 hotpot \(+\) 49 nq) yields 76% strict, 92% majority, 4% degenerate. The release-stage strict rate is higher because class-balance and strict-\(\tau_{\text{lex}}{=}0.8\) filtering steps (§9 (viii)) preferentially remove lower-quality groups. On the strict-support subset of the release audit (\(n{=}76\)), forced hijack averages \(+\)​3.6pp above full-sample across the five readers (Qwen2.5-72B \(+\)​7.7, Llama-3.1-70B \(+\)​5.0, GPT-4o-mini \(+\)​4.1, GPT-OSS-120B \(+\)​2.7, Qwen2.5-7B \(-\)​1.7), with strong readers shifting from drift to hijack (\(-\)​4 to \(-\)​6pp drift)—direction matches the polymorphic ablation \(\Delta{=}+18.8\)pp (§3.4). Per-instance audit labels released (audit/sybil_qc_human_audit.json, sybil_qc_v2_release_human.json, results/strict_patch_v2_release.csv).

10.6.0.2 Compute.

2\(\times\) NVIDIA A100 80GB GPUs for generation, verification, and inference; CPU retrieval indexing with \(\geq 256\) GB RAM (the reference machine had 376 GiB physical RAM; FAISS index loading and Pyserini BM25 dominate at \(\sim\)​120 GB peak resident, with the remainder reserved for OS and runner overhead). Approximate wall-clock: generation 48 GPU-h, verification 6 GPU-h, main-grid inference 96 GPU-h, cross-dataset 30 GPU-h, ablation 4 GPU-h (total \(\sim\)​200 GPU-h).

10.6.0.3 LLM usage.

LLMs serve as (1) QC verifier (Qwen2.5-72B; §10.1), (2) sybil generator (Llama-3.1-8B; §10.2), (3) main readers (§10.3), and (4) drift-origin audit classifier (Qwen2.5-72B; §11). Qwen2.5-72B’s verifier–reader–classifier triple role is discussed in §9. Any LLM use for writing or editing did not affect the core methodology and is therefore not separately declared, consistent with the NeurIPS policy.

11 Drift Origin Audit Details↩︎

We sample 1,356 clean-condition drift instances on the reconciled manifest (post-Phase-1 pool, seed\(=\)​42); per-cell counts vary (Table 8) reflecting per-cell drift incidence. An LLM classifier (Qwen2.5-72B-Instruct Q4_K_M, temperature\(=\)​0; same model as the construction verifier, §10.1) labels each as GENUINE (substantively different answer), EXTRACTION (gold present but canonical extraction fails), or DATASET (gold annotation incorrect). The use of Qwen2.5-72B in this third role is acknowledged as a limitation (§9, item v).

11.0.0.1 Inter-rater agreement.

To validate the automatic classifier, one author re-labeled a 250-instance subsample stratified across all 10 reader\(\times\) retriever cells (seed\(=\)​42), blind to the classifier output. Human-labeled composition is 84.4% GENUINE / 15.2% EXTRACTION / 0.4% DATASET, against automatic 89.6% / 3.2% / 7.0% (post-Phase-1 reconciled manifest); raw agreement is 82.8% and Cohen’s \(\kappa{=}0.262\) (fair agreement on the Landis–Koch scale). Per-class disagreement is concentrated on GPT-OSS-120B: when the 50 GPT-OSS-120B subsample instances (across both retrievers) are excluded from the comparison, raw agreement rises to 88.5% but \(\kappa\) falls to 0.194; this \(\kappa\) decrease reflects rising expected-by-chance agreement (76.7% to 85.7%) under a more class-imbalanced GENUINE-dominant distribution, not a substantive deterioration in classifier-human alignment. The classifier under-detects EXTRACTION (auto 3.2% vs.human 15.2%, a 12.0pp gap on the full sample) and over-attributes DATASET-issue (auto 7.0% vs.human 0.4%, a 6.6pp gap), particularly on GPT-OSS-120B where the classifier scores Harmony-format truncated reasoning as dataset-ambiguity (auto 19–21% DATASET on GPT-OSS subsample vs.human 0%). The audit’s GENUINE fraction therefore remains an upper bound; the human-labeled estimate is closer to 84% GENUINE / 15% EXTRACTION. Per-instance disagreement breakdowns, per-class confusion matrices, and the labeling protocol are released with the audit artifacts. This audit does not modify the official evaluator and is independent of the four-way main-grid metric.

12 API-Error Missingness and MCAR Verification↩︎

One main-grid cell (GPT-4o-mini\(\times\)ColBERT attack) retains residual OpenAI API errors after rerun attempts (271 errors, 9.1%). The forced-condition runs themselves were API-error-free in the top-10 forced exposure pass; however, for paired clean–attack–forced analysis on GPT-4o-mini\(\times\)ColBERT all three cells are reported on the same \(n_{\text{eff}}{=}2{,}711\) question subset (excluding the attack cell’s error-affected questions across all three conditions). This induces a \(\leq 0.4\)pp deviation from the by-construction retriever equality in GPT-4o-mini’s forced cells. For the attack cell itself, validity of rate estimates depends on missingness being uncorrelated with reader outcome (MCAR).

Clean-condition gold rates differ by 1.1pp between error and non-error subsets (Table 9); \(\chi^{2}\) fails to reject independence (\(p{=}0.78\)), supporting MCAR at the resolution of available evidence. MCAR is testable only on clean-condition outcomes and is not directly verifiable on attack outcomes; the verification is therefore necessary but not sufficient (§9, item vi). Full re-evaluation against a more stable endpoint is deferred to the camera-ready revision.

13 Bootstrap Confidence Intervals↩︎

95% bootstrap CIs (percentile, \(B{=}5{,}000\)) for main-grid four-way partition metrics are reported in Table 10. Cross-dataset CI half-widths are 1.7–2.6pp on TriviaQA (\(n{=}1{,}398\)) and 1.0–1.9pp on 2Wiki (\(n{=}2{,}696\)); cross-reader differences discussed in §8.3 exceed CI overlap.

14 Datasheet for Datasets↩︎

We follow the Datasheet for Datasets template (abridged; full template in the release repository).

14.0.0.1 Motivation & composition.

The benchmark supports failure-mode-aware evaluation of RAG systems under polymorphic sybil retrieval poisoning. Contents: 3,145 questions with paired polymorphic sybil groups (\(S{=}6\); 2,982 retained). Main sources: NQ-open validation (1,145 sampled), HotpotQA distractor dev (2,000 sampled). Validation layer: TriviaQA unfiltered.nocontext validation (2,000 sampled; 1,398 retained), 2WikiMultiHopQA dev (3,000 sampled; 2,696 retained). All samples use seed\(=\)​42 from the respective pool (§4). Ablation: monomorphic baseline (500Q subset). Target answers are drawn from dataset-conditional plausible alternatives; sybils are generated by Llama-3.1-8B-Instruct under the acceptance-filter protocol (§10.1); aliases follow source conventions.

14.0.0.2 Uses, distribution, maintenance.

Intended for evaluating RAG robustness to coordinated retrieval poisoning; not intended for training attack or defense models (risk of distribution collapse) or for red-teaming production systems. Released via public repository under licenses in §10.5; integrity verification via SHA-256. Versioned frozen release; errata published as patch versions without modifying the evaluator or official scoring rule.

References↩︎

---

1. Corresponding author.↩︎