1 Introduction↩︎

Large language models (LLMs) have increasingly become the backbone for much of natural language processing and there has been a push to formulate a wide range of tasks as sequence to sequence transduction. However, when LLMs need to interact with non-parametric knowledge in the form of an external evidence corpus, the typical approaches chain LLM generations with calls to a separate retrieval model as part of a multi-system pipeline. In this paper we introduce a new approach, Riches (Retrieval Interlaced with Sequence Generation) which can natively interleave text generations with retrievals from an evidence corpus using a single LLM and decoding process.

Riches builds on previous work that demonstrated the application of constrained decoding to retrieval over a corpus [1], [2] but extends this work to support multiple retrievals, entwined in a standard text generation procedure. In this approach, we retrieve documents by directly decoding their contents or related natural language retrieval keys that point to the documents they were generated from. For example, Figure 1 illustrates a solution from Riches to multi-hop question answering [3], where evidence must be retrieved from multiple separate documents, by iteratively generating an unconstrained ‘thought’ about what needs to be retrieved and then generating a supporting proposition derived from an evidence corpus and tied to an original piece of supporting text. Riches executes this task in a single decoder pass. For this example task, which is evaluated alongside others in Section 6, we have built on recent advances in chain-of-thought reasoning via prompting alone [4] but have directly integrated the retrieval step without needing to account for any interaction with an external retrieval system.

The observations we build this work on are:

1. LLMs are knowledge warehouses: They internalise and generalise over vast quantities of training data and are often able to generate surprisingly accurate knowledge in response to complex inputs [5]. However they are also susceptible to hallucination and cannot account for fresh knowledge, not available at the time of training. That is where retrieval shines.

2. LLM decoding is a search process: Language model decoders search for a single sequence in the set of all possible token sequences [6]. Retrievers just need to constrain this search space to those sequences that are known to exist in a corpus of interest.

3. Unifying tasks unlocks rapid development via prompting By unifying retrieval with generation in a single decoder pass, we create a system that can be adapted to diverse new tasks via prompting alone, directly benefiting from the advances in instruction following. We later show that Riches works with an off-the-shelf instruction-tuned model, without any additional training. This is in contrast to pipelines that need to be rebuilt/retrained on a task-by-task basis.

There is an another advantage of using language models as search agents. Of the two core operations in retrieval, indexing and search, indexing is constrained by corpus size, while search typically depends only on the index structure. Using large language models for indexing billion-token corpora is highly expensive, but search does not face the same bottle-neck. This enables us to unlock the knowledge stored in very large models for retrieval.

This work overlaps with a variety of related work focusing on retrieval, retrieval augmented generation [7], reasoning in language models, and open domain question answering. We discuss their connections to Riches in Section 2, then introduce the key components of the generalizable Riches approach in Section 3.

While Riches is applicable to any task that can be reduced to an interleaved generation of unconstrained text and pre-defined retrieval keys, we validate the approach with tasks in open domain question answering and show how it natively supports single-hop question answering, including the case where attribution to a source text is required; multi-hop question answering; and interleaving retrieval with ‘planning steps’ that enhance the retrieval performance. Results are presented in Section 6.2 along with qualitative examples and analysis in Section 6.3 to help motivate the approach.

2 Related Work↩︎

2.0.0.1 Retrieval Augmented Generation (RAG)

ODQA tasks predominantly employ the RAG approach [7] where typically a dense retriever [8] retrieves documents from an evidence corpus and feeds to a language model for the final answer. These pipelines involve switching between heterogeneous models and are hard to train in concert. Moreover, Dense retrievers fail to generalize out-of-domain [9].

2.0.0.2 Generative Retrieval

[10] techniques shifting the onus of Search from non-parametric nearest neighbor scan to language models. Differentiable Search Index [11] memorizes a mapping of query to opaque document identifiers, however memorization struggles to generalize to unseen corpus [12]. An alternative approach is to use natural language keys as document identifiers, where keys are constrained decoded to lie in the corpus [2], [13]. These systems still need an external model to generate answers. 1-Pager [1] unifies evidence and answer generation, by generating a sequence of keywords that map to a document. However, isolated keywords limit context understanding and suffer similar pitfalls as lexical matching.

2.0.0.3 Recitation

Separate from retrieval augmentation, language models have been shown to recite entire passages from memory [5], [14]. But these passages are prone to hallucination. Our aim is to intersect contextual passage generation with corpus grounding. GopherCite [15], a noteworthy work in this direction, generates quotes verbatim from a small set of documents using constrained decoding. Riches aims to scale this to a billion-token corpus.

2.0.0.4 Iterative reasoning and Search

In recent times, there have been several efforts to improve multi-hop question answering by better reasoning [16] and planning [4], [17]. Language models have also been applied to the task of search to explore alternative paths [18], [19].

Our work builds on these advances in reasoning while integrating search within generation.

3 Retrieving while Generating↩︎

We present a method of interleaving unconstrained text generation with the generation of retrieval keys that point into a retrieval corpus. For example, Figure 1 shows generations that interleave unconstrained ‘thoughts’ with evidence sentences drawn from a predefined corpus for a multi-hop question answering task. Later in this section we’ll introduce a number of different choices of retrieval key as well as a variety of tasks that benefit from interleaved generation and retrieval. However, for now we simply define a retrieval key as a sequence of tokens that exists in a pre-defined finite set of sequences \(K\) where every entry is associated with one or more documents in an underlying corpus \(C\).

Formally, we focus on the sequence to sequence transduction task where we predict an output sequence \(\mathbf{y} = [y_0, \dots, y_n]\) conditioned on an input sequence \(\mathbf{x} = [x_0, \dots, x_m]\) and we mark the start and end of a retrieval key in \(y\) with special markers << and >>. If we let \(Q(\mathbf{y})\) be a function that returns all retrieval key spans from \(y\) (i.e. \((i, j) \in Q([y_0, \dots, \texttt{<<}{}, y_i, \dots, y_j \texttt{>>}{}, \dots, y_n])\)) then we can update the standard autoregressive language modeling probability

\[\begin{align} P_{\theta}(\mathbf{y}|\mathbf{x}) = \prod_{i=0}^{|\mathbf{y}|} P(y_i|y_0,\dots,y_{i-1}, \mathbf{x}, \theta) \label{eqn:lm} \end{align}\tag{1}\] to include the indicator function \(\mathbb{1}_{K}(\mathbf{q})\) that maps elements of \(K\) onto one and otherwise to zero. \[\begin{align} \begin{aligned} P_{\theta}(\mathbf{y}|, \mathbf{x}, K) & = \frac{1}{Z}\prod_{\mathbf{q}\in \mathcal{Q}(\mathbf{y})} \mathbb{1}_{\mathcal{K}}(\mathbf{q}) \\ & \times \prod_{i=0}^n P(y_i|y_0,\dots,y_{i-1}, \mathbf{x}, \theta) \end{aligned} \label{eqn:constlm} \end{align}\tag{2}\]

where \(Z\) is a normalizing term that accounts for the probability mass assigned by Equation 1 to disallowed sequences. In practice, we do not need to compute \(Z\) and can sample from Equation 2 in the usual way, one token at a time, by simply zeroing out the probability of disallowed continuations as presented in Section 3.1.

3.1 Constrained Beam Decoding↩︎

We opt for Beam Search [6] as our decoding strategy to simulate a heuristic Best-first search. Here, the action or next node space is the entire vocab. At each time step, the LLM estimates the value of each node (token) given the paths explored so far and adds them to the fixed-size queue (Beam). 2 visualizes how the beam progresses over decoding timesteps. Unlike regular beam decoding where the top decoded sequences have only small variations, constraints impose sparsity over the search space resulting in diverse beams. In 3.3, we discuss how beam can hurt unconstrained generation and suggest hybrid decoding strategy as workarounds. Constrained decoding can also gain from more sophisticated algorithms such as value-based decoding [20], look-ahead scoring and planning [19], [21].

3.2 Efficient Constraints via the FM-Index↩︎

During decoding, model outputs are constrained to the corpus by masking out any continuation not in the corpus. To compute the continuations of a sequence, we use FM-index [22], a compressed suffix array augmented with additional data structures to support fast substring search operations. Unlike a Trie structure, it is also highly space economical due to the compression. Given a prefix, FM-Index can efficiently compute the next allowed tokens in O(Vocab), independent of the corpus-size. Below is the pseudo code for the modified decoding process.

def constrain(input_prefix):
  # Fetch continuations for prefix
  allowed_tokens = fm_index.get_continuations(input_prefix)
  # Get next token probabilities
  logprobs = LLM.logprobs(input_prefix)
  # Disallowed tokens are set to -inf
  for i in logprobs:
    token = vocab[i]
    if token not in allowed_tokens:
        logprobs[i] -= np.inf
  return logprobs

3.3 Adaptive Beam Size↩︎

In Section 5.2 we introduce some tasks that interleave constrained and unconstrained generation. The constrained generations must be precise—to match the target retrieval key exactly. The unconstrained generations are generally more robust to small variations in surface form—these only need to convey the correct information to a reader, or to provide the model room for a ‘thought’ trace when reasoning about a response.

To ensure that Riches can properly make use of beam search, which is here intended to ensure the model does not get stuck irretrievably after generating an incorrect constrained prefix, we introduce an adaptive decoding strategy that switches between full beam decoding for the sensitive constrained sequences but opts for greedy decoding when unconstrained. In practise, a constrained prefix is expanded to next beam-size probable tokens while an unconstrained prefix is expanded to only the next one token. This is expected to provide room for rest of the beam to be utilized largely for constrained sequences. 6.1.0.3 shows experiments with multiple decode modes.

3.4 Indexing Strategies↩︎

The FM-Index used by Riches supports efficient indexing of all sub-strings in a corpus, which is useful when we want to generate corpus text verbatim. However, it is not clear that this is the best option of retrieval key for the auto-regressive decoder in Section 3.1. A key question in index construction is the document representation used in indexing. In traditional lexical-based retrieval systems, documents are represented by the terms in it, with transformations such as stemming, weighing by corpus statistics [23]. Neural retrieval systems transform raw text into dense vector representations and offload representation computation to the neural network. But even in this case, proper document chunking and/or multi-vector document significantly impact final performance [24], [25].

In this section, we introduce a few different choices of retrieval keys, including a propositional index that requires indexing time neural computation. A key consideration here is the interplay between the retrieval index and the search strategy.

3.4.0.1 Document Title and Section Headers

Many retrieval corpora such as Wikipedia have consistent structures in the form of titles and sometimes sub-titles and metadata. This provides a hierarchical structure such that one can first decode titles, sub-titles and then the document.

3.4.0.2 Paragraph Sub-string

A natural option for retrieval key is any sub-string of the unit of text being indexed itself. In most open domain question answering approaches, paragraph is the de-facto unit of evidence. We can index paragraphs efficiently using the FM-index (Section 3.2) and decode sub-strings directly with Riches to get pointers into the retrieval corpus. It should be noted that this yields an inherently many-to-many mapping between paragraphs and retrieval keys, but that the mapping is in-effect one-to-one for longer sequences of tokens.

3.4.0.3 Sentence Sub-string

Similarly, individual sentences form a natural retrieval key. Sentence are smaller units of information than passage, but may not be interpretable or stand-alone.

3.4.0.4 Propositional Index

The above choices do not perform any non-trivial indexing step, unlike standard approaches in information retrieval where documents are mapped to sparse or dense vectors. The omission of this indexing step may be desirable but it also forces Riches to deal with the non-uniform and diffused information in raw text. An alternative that is closer, in intent, to the offline indexing step used by other IR systems, is to map each indexed chunk to a set of uniformly structured propositions [26], [27]. A proposition is a stand-alone unit that efficiently encodes small, atomic chunks of factual information. For example, instead of the sentence "He has 7M followers on Twitter" a proposition would be decontextualized to "Tom Cruise has 7M followers on Twitter." We adopt a pre-existing propositional index from [28] described in 5.1.

6.1.0.1 compares various Retrieval keys for the ODQA task with illustrations in 10.4.

4 Interleaving Retrieval and Generation↩︎

We have presented a method of interleaving unconstrained text generation with constrained generation of retrieval keys. In this section we introduce a handful of tasks that make use of this interleaving either as a core task requirement, or as a means to an end by interleaving ‘thoughts’ with retrieval actions to help guide search.

4.0.0.1 Attributed Question Answering

We apply Riches to the open domain question answering (ODQA) task where we score both the ability to correctly predict a short answer string and retrieve attribution for that answer [29]. See ¿tbl:tab:wins? for examples.

4.0.0.2 Multi-hop Question Answering

Interleaving between generation and retrieval can be powerful in multi-hop reasoning, where the model needs to retrieve and stitch together knowledge from multiple sources. Examples of Riches outputs for multi-hop QA are given in ¿tbl:tab:win95multihop?.

4.0.0.3 "Thinking" for Retrieval

Multi-step questions often require breaking down a query into smaller steps and reasoning or planning what to retrieve next. Foreshadowing retrieval with thoughts is crucial in this context. It helps direct the retrieval process, avoid repetitions, and, more importantly, allows for iterating upon and correcting previously erroneous retrievals. A few such demonstrations can be found in ¿tbl:tab:win95multihop?.

5 Experimental Setup↩︎

5.1 Datasets↩︎

5.1.0.1 Queryset

Our experiments are focused on open domain question answering tasks including both single and multi-hop benchmarks. For single-hop, we use the Open-NQ [30] dataset. To evaluate multi-hop reasoning, we look into Hotpot-QA [3] and Musique-Ans [31]. The latter includes varying hops and different composition operations, offering a rich test-bed for how well Riches can generalize across a diverse range of queries.

5.1.0.2 Corpus

3.4 describes multiple strategies to index the corpus. Each type of retrieval key needs to be accompanied with its own corpus. Title, passage and sentence keys are derived from the Wikipedia corpus presented in [29]. For propositions, we re-use the Factoid-Wiki corpus built by [28]. This is derived from [29] by decomposing passages into smaller, compact propositions using a finetuned Flan-T5-large [32] model. We drop the titles from Factoid-Wiki and only use the propositions (See 10.2).

5.2 Evaluation↩︎

The standard metric for ODQA benchmarks has predominantly been F1 answer match accuracy. However, language models are prone to hallucinate and F1 stand-alone can be misleading as the answer may not be conditioned on the evidence. Attribution [33] helps us trade-off answer accuracy for faithfulness to the evidence. Thus, we measure two competing metrics: i) end-to-end answer accuracy with F1 and ii) attribution of the answer to evidence using AutoAIS [29]. AutoAIS, or AIS for short, is automatically computed by classifying whether the evidence text entails the question and predicted answer pair. We re-use the NLI scorer and formulation from [29] (See details in 10.2). The evidence text here is the concatenation of all retrieval keys in the Riches output. The unconstrained thoughts are discarded from evaluation. Only the top beam output is considered for evaluation.

5.3 Models and Inference↩︎

Throughout our experiments, we use off-the-shelf instruction-tuned models in a few-shot setting, without any fine-tuning. We test the instruction-tuned versions of PaLM2-M and its larger variant PaLM2-L [34] based on stacked Transformer architecture. We use 3 example demonstrations in our prompt (10.1), with different sets of examples for single-hop (NQ) and multi-hop (Hotpot, Musique) datasets. The unconstrained sequences or thoughts are formulated as hint keywords. Our final setup uses a beam of 10 with constrained decoding (3.1), adaptive beam size (3.3) and propositions as retrieval keys. Later in 6, we ablate these choices. Note that only the top-beam result is considered for evaluation.

5.4 Baselines↩︎

We experiment with 3 types of baselines: no retrieval, the standard dense retriever and an iterative retriever suited for multi-hop QA. We opt for baselines that test the out-of-domain system performance, similar to Riches.

5.4.0.1 No Retrieval

We compare to a few-shot unconstrained baseline using the same 3-shot prompt as Riches allowing for chain-of-thought reasoning. The setup generates an answer along with hallucinated evidences, not grounded to a corpus. This is a measure of model’s memorization capabilities.

5.4.0.2 Generalized Dense Retriever

For single-hop QA, we compare our approach against the Generalized T5 retriever (GTR-xxl, 11B variant) [35]. GTR undergoes multi-staged training, first on unsupervised web-mined corpus and then supervised search datasets including NQ. It has been shown to generalize well out-of-domain. However, GTR and other conventional dense retrievers provide only retrieved documents, not the answers themselves. To extract answers, we use the PaLM2-M model in a few-shot setting (see 10.1 for the details).

Since Riches generates a single output with a varying number of interleaved documents, direct comparison with dense retrievers that fetch a fixed top-k documents is challenging. We set k to a value equivalent to the mean documents Riches fetches for single-hop. When retrieval keys are different, such as passages vs propositions, we approximately match the tokens used by both setups. In our final experiments, we compare against k=1 passage and k=2 propositions for GTR-xxl.

5.4.0.3 Iterative Retrieval (Iter)

For Multi-hop QA, we adopt a popular method where question is decomposed into sub-queries [36]. At each step, passages are retrieved for a sub-query and fed as input for the next query, until one converges to an answer. The method has the same surface form as Riches, except for the key distinction that each step requires switching between a heterogeneous mix of models. In our experiments, we retrieve top-1 document with GTR-xxl and use PaLM2-M few-shot for both decomposing the query and generating the final answer (See prompt at 10.1). Max allowed steps is set to 4 where most of the queries converge.

6 Results↩︎

In the following sections, we investigate the key building blocks of Riches: i) indexing strategies (3.4) amenable to auto-regressive decoding ii) effect of beam decoding (3.1) iii) suitable mechanisms to interleave thoughts and retrieval keys (3.3). Finally, we compare Riches against conventional retrieval systems. We also draw a detailed analysis of wins and losses to fathom the strengths and pitfalls of the system.

6.1 Riches building blocks↩︎

6.1.0.1 Retrieval Keys

We explore the following retrieval key candidates as detailed in 3.4: a) Title: Wikipedia page and section titles, ranking paragraphs within the section using TF-IDF scores. b) Paragraph with Title: Decodes the page title, section title, and full paragraph. c) Paragraph: Decodes the paragraph only. d) Sentence: Uses individual sentences. e) Proposition: Uses atomic information units derived from paragraphs. Table ¿tbl:tab:search95unit? shows that among the retrieval keys explored, the propositional index is best aligned with our decoding search strategy, perhaps its compact nature is most suited for autoregressive decoding. An in-depth analysis of retrieval keys is provided in 10.4. In the following experiments, we use proposition as our retrieval key.

6.1.0.2 Effect of Beam size

¿tbl:tab:beam? shows how greedy decoding can get stuck with poor retrieval keys. A larger beam enables better search space exploration, albeit with diminishing returns. In our final experiments, we use a beam of 10.

6.1.0.3 Interleaving with Adaptive Beam

¿tbl:tab:blend? shows the impact of interleaving thoughts with retrieval keys. First, we note that an adaptive beam is crucial for interleaving unconstrained and constrained sequences. Without an adaptive beam, minor irrelevant variations in unconstrained thoughts can consume and overwhelm the available space in the beam. By greedily decoding unconstrained sequences, the beam space is preserved for backtracking during document search. Once we have an adaptive beam in place, the insertion of keywords enhances both answer and retrieval performance, reminiscent of chain-of-thought technique to enable better retrieval.

6.2 Overall Results↩︎

¿tbl:tab:results? shows the overall performance of Riches across various datasets. First, we compare with the no-retrieval baseline and observe that jointly retrieving and answering does not negatively impact model’s answering capabilities. For single-hop tasks, Riches competes well with dense retrievers, offering higher answer accuracy at the expense of attribution. In multi-hop QA, Riches excels, outperforming iterative baselines by +15 F1 points on Hotpot and +11 on Musique, with comparable or better attribution. The increase in answer accuracy with the larger PaLM2-L model suggests improved performance with larger model sizes. Notably, Riches achieves these results with a single inference pass, unlike the Iterative baseline, which requires a model call at each sub-query step.

6.3 Qualitative analysis↩︎

We inspect 50 win and loss examples each to analyze the strength and weaknesses of the system.

6.3.0.1 Wins

Several properties distinguish Riches from dense retrievers: a) Riches allows large language models to utilize their parametric knowledge for retrieval. Since the search operation in Riches is independent of corpus size, it can employ much larger models at query time. b) The inherent alignment of instruction-tuned models enables them to retrieve contextually relevant passages, whereas dense retrievers may sometimes latch onto keywords. c) The interleaved thoughts guide the model toward more accurate retrievals. ¿tbl:tab:wins? demonstrates these scenarios for single-hop retrievals and ¿tbl:tab:win95multihop? for multi-hop retrievals.

6.3.0.2 Can the model retrieve what it doesn’t know?

A language model may hold stale or incorrect information. However, Riches can often override model’s pre-existing knowledge and generate correct answers by constraining on the corpus (10.3)

6.3.0.3 Losses

We inspect 50 failed queries and categorize the losses (¿tbl:tab:loss95category?) as follows: a) Index failure: the proposition is absent from the index or not decontextualized. b) Search failure: Proposition exists in the index, but could not be generated c) Attribution failure: The answer is partially attributed, with LLM hallucinating based on partial evidence. (see 10.3 for examples)

7 Conclusion↩︎

Retrieval has so far been alienated from the rapid progress in instruction tuning. This work makes the following contribution: i) an approach that can seamlessly integrate retrieval with generation. ii) a thorough investigation of indexing and search strategies that enable such an approach to be effective. iii) proof-of-concept of the capabilities of such a system on a variety of QA tasks. We hope the ideas introduced in this work fuel progress in aligning retrieval to generation and simplifying it.

8 Limitations↩︎

First we note the limitations in our experimental setup. All our experiments are based on Wikipedia, a corpus heavily seen during pre-training. This work does not analyze how Riches fares on corpora unseen during pre-training. Furthermore, we only examine a handful of factoid question-answering tasks due to the lack of objective evaluations. Performance on tasks such as long-form QA is deferred for future work. There are also certain inherent limitations with Riches. It forces verbatim emission of corpus text, which might be an overkill for tasks where a similarity-based metric is sufficient. Riches lacks the ability to retrieve dozens of documents, a necessity for certain summarization tasks. For long documents with diffused information, rewriting into propositions adds complexity and can be cumbersome. Lastly, while Riches’s search operation is independent of corpus size, the use of beam search and communication between the FM-index and Transformer model can slow down inference.

9 Ethical Considerations↩︎

All artifacts used in this paper, including models, datasets, and baselines, are under permissive licenses and publicly available. We have attempted to provide detailed information to facilitate the reproduction of our results.

Our findings are based on English-language data from Wikipedia, and we have not tested the generalizability of our claims to other languages or domains.

Lastly, the datasets used in this work are not expected to contain any offensive content. However, it is important to note that Large Language Models (LLMs) can exhibit biases related to gender, race, and region, and are also prone to hallucination. Although Riches aims to ground its generation in an external corpus, some biases may still be present.

10 Appendix↩︎

10.1 Experiment Details↩︎

10.1.0.1 In-context prompts

We use 2 different sets of few-shot demonstration for single-hop (NQ) and multi-hop (Hotpot, Musique) datasets displayed in ¿tbl:prompt:proposition? and ¿tbl:prompt:single95hop? respectively. Both prompts carry the same instruction, but the multi-hop variants provides demonstrations with multiple evidence passages.

10.1.0.2 Computing constraints

An example of constrained decoding is illustrated in 3.

10.1.0.3 Baselines

For the dense-retriever baseline, answers are extracted from retrieved passages with an external reader. We use PaLM2-M with a few-shot prompt (¿tbl:prompt:reader?).

For iterative retrieval baseline, we use PaLM2-M for both query decomposition and answering. At each step, the model can choose to generate a sub-query or the final answer. The unified prompt is provided at ¿tbl:prompt:iterative?.

10.2 Evaluation↩︎

10.2.0.1 Datasets

We use Musique-Ans [31] subset of Musique which consists of answerable queries. Details of query sets evaluated can be found in ¿tbl:tab:dataset?. To make retrieval challenging, we use the full Wikipedia corpus for retrieval (¿tbl:tab:corpus?). This is different from the typical Hotpot and Musique setting which use the first Wikipedia paragraph (5M documents) and documents associated with query-set (1.3M) respectively.

10.2.0.2 Metrics

AutoAIS is an automated way of measuring AIS (Attributable to Identified Source) [33]. AutoAIS formulates evaluation as a Natural Language Inference task that asks a model whether the question and answer are entailed by the provided evidence. We re-use a T5-11B checkpoint finetuned on a collection of NLI tasks from [29]. Question answering is formulated into NLI task as follows:

hypothesis: <retrieved evidence1> <retrieved evidence2> ... premise: The answer to the question ’<question>’ is ’<predicted answer>’ The NLI scorer provides a calibrated score between 0 (not attributed) to 1 (completely attributed) which is averaged over the dataset to get the final AutoAIS score.

10.3 Extended Qualitative Analysis↩︎

We provide examples for loss categories defined in 6 in ¿tbl:tab:losses?. ¿tbl:tab:override? showcases a few selected examples where the unconstrained model emits incorrect answer, but constraining on the corpus guides it towards correct answer.

10.4 Index representation qualitative analysis↩︎

In this section we analyse retrievals from a few select examples from the OpenNQ development dataset. We compare retrievals using different document representations in the index, including: title + tf-idf, passage, sentence and proposition (see Section 3.4).

10.5 Computations involved↩︎

Evaluating the precise compute cost for Riches depends on the specific implementations of the decoding algorithm, but we can sketch the key operations involved in retrieval: indexing and search. Indexing depends on the number of items in the corpus \(|D|\). We use a model of size \(\mathcal{M}\) to rewrite each passage (average length \(|p|\)) into propositions. The overall indexing cost is proportional to \(O(D\mathcal{M}p^2)\), similar in magnitude to the cost for encoding the corpus in dense retrieval, differing only by a constant factor. Note that our experiments use a T5-large backbone (770M) for Riches much smaller than T5-xxl (11B) used in the dense baselines.

Now let’s look at the search operation. At each auto-regressive step, besides standard decoding, the only additional operation is computing FM-index constraints, which consumes CPU resources. However, while the index is efficient, communication between the index on the host and the Transformer model on the GPU/TPU adds latency to the decoding step. In contrast, RAG systems retrieve documents from index using nearest neighbor scan in a single go. But even there, the documents need to encoded as input to the language model.

References↩︎