1 Introduction↩︎

Pseudo-Relevance Feedback (PRF) is a widely adopted technique for enhancing retrieval effectiveness by expanding queries with terms extracted from top-ranked documents [1], [2]. Although PRF often results in average relevance gains, it suffers from a well-known robustness issue: query drift, where non-relevant expansion terms shift the ranking away from the user’s original intent [3], [4]. This drift creates an asymmetric per-query impact: while some queries benefit, others suffer from ranking degradation [5]. This per-query asymmetry is hidden at two levels. For system designers, aggregate offline metrics such as nDCG average performance over queries and conceal the specific queries that are degraded by query expansion as well as the magnitude of such a degradation [5], [6]. For end users, the system gives no signal that expansion was applied at all, and no explanation for why their search results have changed. We argue that this two-sided opacity is a core explainability problem for automatic query expansion in Information Retrieval (IR): making PRF accountable requires per-query explanations that are interpretable to both system designers (who decide whether to ship expansion) and end-users (who must trust the results that are returned).

Selective PRF (sPRF) aims to mitigate query drift by applying expansion only to the queries that are predicted to benefit in terms of retrieval performance, typically using Query Performance Prediction (QPP) methods [7]. However, QPP methods lack robustness across diverse retrieval tasks [8] and are validated against the same offline metrics that often fail to reflect actual user satisfaction [5], providing no transparent justification for when or why expansion is applied to a given query. The applied selection mechanism, therefore, inherits, rather than resolves, the explainability gap.

We propose a two-stage audit-then-automate framework that closes this gap by producing explanations that are useful to system designers and end-users. In Stage 1, a participatory audit [9] transforms user interactions into transparent, query-level signals that reveal which queries PRF harms and why, providing system designers with auditable evidence of when expansion fails. In Stage 2, an automated closed-loop predictor repurposes LLM-based document rerankers as system preference predictors that replicate these user-derived signals at deployment time, so that selective PRF decisions can be made for new queries without re-running user studies, while remaining traceable to documents that an end-user can inspect.

2 Stage 1: Participatory Audit of PRF↩︎

Our participatory audit is designed to make the per-query impact of PRF visible by collecting preference signals from users’ natural search behaviour, without requiring users to complete an effortful relevance annotation task [9]. For each of the 43 TREC Deep Learning 2019 queries, we generate dual rankings using ColBERT [10] (baseline) and ColBERT-PRF [11] (expanded), and construct a fair document set using Team Draft Interleaving (TDI) [12], which treats the two rankings as ‘captains’ in a sports draft and alternates selections to produce a combined set with an equal number of documents from each ranking. We present this combined document set in a 3\(\times\)​3 grid using Latin Square ordering [13], [14] to mitigate position bias. As shown in Figure 1, users perform a two-stage interaction: first clicking a document card to inspect its content in a pop-up, then selecting a specific passage to confirm relevance, allowing us to capture high-confidence preference signals from natural browsing behaviour. Each document’s originating system is recorded but hidden from users, such that user preference can be attributed to retrieval quality rather than rank position.

We conduct two sequential user studies, each with 54 mutually exclusive participants [9]. The first study audits per-query PRF impacts by collecting preferences over interleaved baseline/PRF rankings, identifying which queries have benefited, or were harmed, by PRF. The second study then validates the effectiveness of selective PRF. Together, the user studies address three explainability questions: which queries are harmed by PRF, whether selectively not applying PRF on harmed queries contributes more than applying it on beneficial ones, and when expansion should be applied.

First, only 20.9% of queries benefit from PRF, while 25.6% suffer degraded user experience, and the majority (53.5%) show no clear preference; this heterogeneity, entirely hidden from aggregate offline evaluation, becomes visible and explainable at the per-query level through the audit. Second, eliminating harmful expansions (i.e., reverting to the baseline ranking for queries that are degraded by PRF, i.e., PRF-Hurt queries) yields a 68.5% relative improvement in user engagement, nearly twice the 37.0% gain obtained by applying PRF only to the queries identified as beneficial (PRF-Benefit queries), which provides a concrete, user-grounded explanation for why sPRF should be preferred: avoiding query drift is more valuable than exploiting successful expansion, consistent with prior evidence that users are disproportionately sensitive to query failures [5]. Third, applying PRF only to queries for which the audit signals indicate benefit achieves a significant gain of +10.5 adjusted selections per query, averaged across the 20 selective queries (11 PRF-Hurt + 9 PRF-Benefit) (\(p\) = 0.032, Cohen’s \(d\) = 1.9) over universal PRF application. This result is validated through a Difference-in-Differences [15] analysis with Neutral queries as a control (those queries whose PRF preference ratio falls within the [0.45, 0.55] tolerance band, with no clear user preference; the interleaved ranking is held constant across both user studies, so any change in engagement reflects cohort-level drift rather than the sPRF decision). Together, these results show that transforming users’ natural interactions into structured audit signals produces per-query explanations of PRF impact that are both transparent to system designers (which queries fail) and actionable for selective expansion (when to apply PRF).

3 Stage 2: Closing the Loop with LLM Rerankers↩︎

The participatory audit produces ground-truth, user-derived explanations, but cannot be repeated for every new query. To scale the approach, we argue that LLM-based document rerankers can serve as inspectable surrogates for user preference. The key idea is to repurpose rerankers as system preference predictors: rather than asking an LLM to classify whether PRF helps or harms a query (an opaque process), we propose to present the LLM with the same interleaved document pool that users saw in Stage 1, and use its ranking of these documents to reveal which system it prefers.

This design enables explainability and transparency in selective PRF. The LLM’s output ranking is itself the explanation: an sPRF decision can be justified by pointing to specific documents that the LLM ranked highly from each system, mirroring how users in Stage 1 revealed preferences through their selections rather than through explicit judgements. This supports inspection at two levels: for system designers, an auditable trace of which documents drove each sPRF decision; for end-users, we propose an in-interface rationale, for example by highlighting the supporting passages that distinguish the chosen system from its alternative, thereby replacing an opaque expansion on/off switch with a grounded reason for the displayed results. We see this as a path from a one-off audit study to an always-on transparency mechanism in deployed retrieval systems, and as a step towards selective PRF, whose decisions are explainable by construction rather than through post-hoc rationalisation.

4 Discussion: Explainability Through Auditing↩︎

Our framework contributes to explainability in IR in two complementary ways. First, it makes per-query harms visible: standard offline evaluation averages over all queries and conceals that over a quarter of queries are degraded by PRF, while our audit produces per-query labels that explicitly identify which queries are harmed and to what extent they are harmed, providing the granular transparency that regulatory frameworks such as the EU Digital Services Act increasingly require [16]. Second, our generated explanations are derived directly from user behaviour, i.e., users selecting more useful passages from one system’s documents than from the other’s, and the same evidence base, namely the interleaved documents themselves, is then re-used by the LLM reranker at deployment time to replicate this judgement. Every automated sPRF decision is therefore traceable end-to-end, from a query’s interleaved documents to a user-grounded preference signal to an inspectable predictor output, constituting a form of process-level explainability that is rare in current IR systems despite its importance.

5 Conclusions↩︎

We have presented a two-stage audit-then-automate framework that transforms Pseudo-Relevance Feedback from an opaque retrieval component into one whose per-query decisions are transparent, user-grounded, and automatically reproducible. Our participatory audit with 108 users shows that avoiding harm is nearly twice as valuable as exploiting successful expansion on the queries that benefit from it, while our initial LLM-based automation shows that these user-derived preferences can be replicated at scale. By mapping each sPRF decision to those documents that users inspected in the audit, and that the reranker preferred at deployment, the framework provides explanations that are auditable to system designers and interpretable to end-users. We see this audit-then-automate paradigm as a step towards explainability in IR that does not stop at rationalising algorithmic outputs in isolation, but instead explains an algorithm’s impact on real users.

References↩︎