1 Introduction↩︎

Legal question answering is an attractive use case for retrieval-augmented generation [1] because legal answers should be grounded in authorities that can be inspected and cited. In recent years, many RAG-based legal assistants have been developed to help laypeople and self-represented litigants understand their rights while enabling legal professionals to conduct research more efficiently. A useful system may retrieve relevant laws or case laws, identify the legal context, and answer in language appropriate to the user. However, prior work showed that LLM-based legal assistants hallucinate unsupported claims and readily accept users’ incorrect assumptions [2]. At best, these errors reduce the assistant effectiveness by forcing legal professionals to spend significant time verifying the accuracy of claims. At worst, they risk distorting justice.

To measure progress on that front, the NLP community has built benchmarks for legal language understanding [3], [4] and case retrieval and entailment [5]. Recent benchmarks have begun to address retrieval and end-to-end grounded generation [6]–[8]. However, most use artificial queries – such as reproducing references from citations – rather than realistic legal questions. Moreover, Canadian law is underrepresented in these evaluations. Canada’s legal system has distinctive legal sources, court hierarchies, tribunal systems, and federal and provincial jurisdictional boundaries, necessitating dedicated benchmarks. Despite the availability of Canadian legal corpora, there is no public dataset designed specifically for legal RAG in Canadian law [9]–[11]. To bridge this gap, we propose CanLegalRAGBench, a benchmark for Canadian legal question answering with realistic user queries and expert-annotated answers grounded in Canadian case law.

We evaluated a range of RAG methods on our benchmark. Our findings show that dense retrieval and enhancements such as reranking significantly beat sparse indexing, chunk size matters significantly, and open-source embedding models are competitive with close-source models. However, standard automatic retrieval metrics are limited, and expert evaluation finds many relevant documents outside the gold set.

2pt

Generated answers differed substantially from the gold standard answers, especially when they were based off retrieved documents. Manual analysis revealed that models generated long answers that often contained irrelevant, overly-detailed claims that miss the point of the query. Finally, at least 20% of the generated claims were unsupported by the documents they were referencing. Many such claims were due to reiterating the facts of the query or other connecting ideas, though there were various categories of hallucination such as misapplication of law and misunderstanding of scope and authority. We hope this benchmark will support future work in addressing remaining gaps in legal RAG systems.¹

2 Background↩︎

Many existing legal benchmarks focus on narrow legal contexts – such as merger agreements in the US [15]; but there are also evaluation suites such as LexGLUE [3], LegalBench [4], and LawBench [16] that test various legal contexts and capabilities such as legal classification, extraction, NLI, and QA. Classification tasks most commonly involve legal judgment prediction [17]–[19]. Information extraction tasks include named entity recognition and terminology extraction [20]–[22], and extraction of structured information from documents [23]–[25]. As models’ reasoning abilities improved, benchmarks were developed to measure legal reasoning, for instance identifying legal violations [26].

More recent work has shifted toward legal question answering. BarExamQA [7] consists of multiple-choice US bar exam questions with supporting gold passages from case laws, online resources, and textbooks. Of particular relevance to our work are benchmarks that evaluate retrieval and answer generation. We situate our benchmark against those in Table 1. Notably, few benchmarks focus on legal case retrieval [8], [14], which is a crucial component of legal reasoning in common law jurisdictions, where outcomes rely on precedent.

Those same two benchmarks constructed pseudo queries based on citations in the source document. This design supports creating much larger-scale datasets, but the resulting queries don’t represent the type of questions that users pose to legal AI systems. Natural-language legal queries [12], [13], [27] are therefore important for practical evaluation. Our benchmark addresses this gap by combining natural queries with Canadian case law retrieval documents, enabling more realistic evaluation of legal retrieval-augmented generation.

3 CanLegalRAGBench↩︎

We introduce CanLegalRAGBench, a benchmark comprising of legal queries grounded in Canadian case law. The goal of CanLegalRAGBench is to evaluate RAG-based systems on realistic queries that users of a legal research tools would ask. To create the data, we employed a 6-step process, illustrated in Fig. 1 and detailed below. First, we sample a random case law and a user persona, and condition on them to generate a seed query. Next, we generated variations of the query to test models’ sensitivity to point of view (e.g., layperson vs. lawyer) and small but meaningful factual changes. We used an LLM-as-a-judge to filter out low quality queries. Finally, we collected the answers to the queries from legal experts. We report data statistics in Table 2.

3.0.0.1 Sample a case law.

Each seed query in our benchmark is based on an existing case. We use the A2AJ Open Canadian Legal Data [9], a publicly available collection of 200k+ Canadian court decisions. To remove noisy documents and keep the computational requirements reasonable, we filtered out any documents in the top or bottom quartile in terms of length, keeping only the middle 50%. We selected a subset of the courts and tribunals that represent common legal disputes encountered by the general public (BCSC, SST, CHRT, TCC), and sampled documents equally from each.

3.0.0.2 Sample a user persona.

User personas were sampled using low, medium, and high amounts of each personality from the Big Five personality traits [28]. Different prompts given to the query generator were used to vary this. For all query generations we use DSPy [29] to template our prompts.

3.0.0.3 Generate seed query.

Given a case law and a user persona, we prompt Gemini 2.5 Flash to simulate a legal query from this user that can be answered by the given case law. See Prompt [prompt:query_generation] in the appendix. To get provincial coverage, we sampled from all provinces and territories except Quebec which follows civil law (see Appendix 9.2 for the composition). To add provinces to queries, we simply prepend “I am in {province}.”

3.0.0.4 Generate query variations.

We prompted Gemini 2.5 Flash to generate 8 variations of each seed query. We generate four variations where the facts remain the same – thus a robust system is expected to retrieve the same case laws – but where the focus of the question or the user persona change. For example, the seed query “What are my rights if my landlord enters my apartment without notice?” in Fig. 1 can also be asked from the point of view of a landlord checking if they are allowed to enter their apartment when the tenant is not there. The other four variations change a small fact in the seed query, which may require systems to retrieve different case laws. Finally, within each batch of 8 queries, half of the queries simulate a legal professional and half simulate a lay person. See the appendix for prompts [prompt:query_variation_alternative_facts] and [prompt:query_variation_alternative_situation], and query variations (Appendix 9.3).

3.0.0.5 Verify with LLM-as-a-judge.

We used Gemini 2.5 Pro to filter out \(\sim\)​20% of queries that are self-contradictory, nonsensical, overly complex, hinting towards the document with references, or otherwise unfit for a retrieval setup. See Prompts [prompt:judge-system] and [prompt:judge-user] in the appendix.

3.0.0.6 Collect answers from legal experts.

We hired 3 final year law students from Canadian universities and a Canadian paralegal from Upwork to provide the ground truth answers for our benchmark. Annotators took 2-3 hours to complete a batch of 8 queries, and were paid $20 CAD per hour. Due to the high cost of expert annotation, each example was annotated by a single annotator. To ensure the quality of annotations, we calculated n-gram overlap between queries, answers, and snippets and manually reviewed and filtered around 30 flagged examples with obvious errors, such as answers that fail to target the query. Our evaluation in Sec. 4 consists of both automatic evaluation that treats the expert annotation as reliable ground truth, and expert evaluation that explores the extent that experts miss relevant information.

4 Retrieval↩︎

We present the document collection (§4.1), the baselines (§4.2), evaluation metrics (§4.3), and results (§4.4) for the retrieval task, and analyze the performance and errors (§4.5).

4.1 Document Collection and Chunking↩︎

To facilitate testing RAG systems on our benchmark, CanLegalRAGBench includes additional distractor case laws, randomly selected in equal proportions from CLD and a second dataset, which have been preprocessed and filtered as described in Sec. 3. Since at the time of construction, CLD only comprised 11 sources (generally covering BC, Ontario, federal, and refugee law), we supplemented it with a second dataset of public court decisions across Canadian jurisdictions, which was provided to the authors by Caseway.² This dataset covers additional jurisdictions and more diverse legal issues. We applied various strategies to remove duplicate case laws, including citation normalization, case normalization, stripping whitespaces, and manually comparing documents from the same year and similar lengths. All chunking is implemented as follows. We test various values of max character sizes (1024–8192). We chunk documents via recursive text splitting, always with 128 character overlap. The text splitter uses hierarchical text breaks such as newlines, periods and spaces.

4.2 Baselines and Evaluated Models↩︎

We benchmark various retrieval systems on CanLegalRAGBench, including sparse retrieval, dense retrieval, and enhancements such as reranking and IterRetGen, which are detailed below. Table 10 lists all the models we evaluated.

4.2.0.1 Sparse Retrieval.

We use BM25, an established keyword-based retrieval algorithm which is still considered a strong baseline [30]. Since legal documents are generally long, we use the BM25L variant which is specifically designed for long documents [31].

4.2.0.2 Dense Retrieval.

Dense retrieval works by embedding document chunks and retrieving based on vector similarity [1]. Our choice of embedding models is based on high ranking on the legal embedding benchmark [32], and includes generally strong open source models (Qwen 3 Embedding 8B[33] and EmbeddingGemma [34]); closed models (Gemini Embedding 001, Gemini Embedding 2 [33]); and closed legal model (Kanon 2 [32]).

4.2.0.3 Reranking.

Reranking is a common way to improve retrieval methods. A typical reranker takes the query and retrieved documents and processes them together using a cross-encoder transformer, to output a relevance score for each pair. We use Kanon 2 Reranker , which is proprietary. We implement reranking by retrieving \(4k\) documents, reranking, and retaining the top k for evaluation.

4.2.0.4 Hybrid Retrieval.

We retrieve the top k documents for each of the sparse and dense retrievers, deduplicate them, rank using reciprical rank fusion [35] or the Kanon-2 reranker, and retain the top k documents.

4.2.0.5 Iterative Retrieval.

Iter-RetGen [36] is a retrieval enhancement that iteratively generates an answer and retrieves relevant documents using the query-answer concatenation as the new query. The generated answer should share more similarity with documents than queries and improve retrieval. We do all runs at 3 iterations, and use Gemini Flash-2.5 for answer generation (see Prompt [prompt:iter-retgen] in the appendix). For brevity we will abbreviate as IRG.

4.3 Evaluation Metrics↩︎

4.3.0.1 Automatic Evaluation.

We report recall@k, MRR [37], and nDCG [38]. These metrics measure the ability to find correct documents and rank them appropriately. Following prior work [8], [12], we consider recall and nDCG as the primary metrics (nDCG is more nuanced than MRR as it looks at the ranking of all top-k retrieved documents; and recall is complementary to all other metrics). We report all metrics for the main experiments, and perform analysis based on nDCG and recall, with \(k=10\). All aggregated metrics are macro averages as calculated in Eq. 1 : \[\mathrm{Score}= \frac{1}{|Q|}\sum_{q \in Q} M(q, D_r^q, D_{GT}^q) \label{eq:macro}\tag{1}\] where \(Q\) is the set of queries in CanLegalRAGBench, \(M\) is the metric, and \(D_r^q\) and \(D_{GT}^q\) are the retrieved and gold standard documents for each query. We map retrieved chunks to their documents and report metrics at the document level.

4.3.0.2 Expert Evaluation.

While automatic evaluation provides a general comparison of baselines, it is possible that automatic evaluation unnecessarily penalizes retrieving relevant documents not in the gold set. To investigate this possibility, we perform manual evaluation. We made a stratified sample of 30 queries by province first and by legal issue second using Gemini-2.5-Pro for categorization. We have our expert annotators evaluate the retrieved documents of 3 best performing configurations according to the automatic evaluation (Qwen 8192 chunk size IRG + Rerank, Gemini-Embedding-2 4096 chunk size IRG, and Kanon2 8192 chunk size + rerank). We select a configurations of different embedding models for diverse evaluation. We instructed the expert annotators (§3) to judge retrieved documents for relevance to the query. Documents judged as relevant are added to the gold standard documents, and we report recalculated metrics after expert evaluation for this subset.

4.4 Results↩︎

We report the overall retrieval performance on CanLegalRAGBench, and look into the effect of retrieval method and embedding model on the performance.

4.4.0.1 General Trends.

We tested 46 combinations of method, embedding model, and chunk size. The complete results of all evaluated configurations are reported in 11 in Appendix 11.1. Here, we report the best performing configuration for each embedding model family in 3, based on the average across evaluation metrics.

The best dense retrieval based methods dramatically outperform sparse retrieval, and the hybrid approach seems to also fall behind dense retrieval. Enhancements such as IterRetGen and reranking improve upon a vanilla dense retrieval approach but inconsistently. Large chunk sizes generally perform better (8192 performs very well) – suggesting a good trade off of preserving legal context while adding noise – but it is also model and method dependent. Finally, with respect to model family, Gemini-2 performs best across most evaluation metrics, with Gemma and Qwen closely following, suggesting that open-source embedding models are close in quality to closed source models. Interestingly, the Kanon-2 embeddings, which is specialized for the legal domain, slightly underperforms other embedding models.

4.4.0.2 Best retrieval method.

We report the performance of each retrieval method averaged across embedding model and chunk size in Fig. 2. Reranking, IterRetGen, and their combination, perform highly. Vanilla dense retrieval is marginally behind. Sparse and hybrid retrieval underperform. The hybrid method performs worse than dense retrieval on both metrics due to poor BM25 performance.

Table  11 shows that the effect of reranking on performance is configuration-dependent. It consistently improves Kanon-2 across all chunk sizes and metrics – this is unsurprising given that the reranker is based on Kanon, so they are presumably designed to work well together. However, it is mostly useful for Qwen and Gemma with larger chunks – potentially because they can be noisier and reranking can help filter the noise. IRG provides the highest gains when the base model is already strong (e.g., the Gemini family). Combining IRG with reranking causes a small improvement in recall at the cost of nDCG (Fig. 2).

4.4.0.3 Expert evaluation.

Fig. 3 displays the recall@10 and nDCG@10 of the best configurations before and after expert evaluation of document relevance on a subset of 30 queries. Recall marginally improved or stayed the same except for Kanon-2; Conversely, nDCG improves substantially, since high ranking documents that were considered irrelevant may now be contributing to the DCG (the numerator of nDCG). Overall, the trend suggests that a non-trivial amount of penalized system-retrieved documents were indeed relevant. In terms of comparison between models, Kanon-2 emerges as a competitive model, in contrast to the results in Table 3. However, looking at only the evaluated subset of 30 queries suggests that the automatic and expert evaluation predict a similar trend, ranking Gemini and Kanon-2 close to each other and Qwen slightly behind on both metrics. Overall, these results suggest that automatic metrics predict a similar trend as expert evaluation, but underestimate the absolute performance of systems.

We also re-evaluate the expert annotators from Sec. 3. Trivially, adding previously missed relevant documents leads to recall dropping significantly. This highlights the limitation of relying on experts to provide gold standard retrieval annotations, and underscores the value of iterative evaluation.

2pt

4.5 Analysis↩︎

4.5.0.1 Case Analysis.

Table 4 demonstrates a retrieval example that highlights various error cases such as retrieving documents that are topically similar to the query but differ in legal issue, party roles, and procedural posture. The query asks what happens when an ex-partner repeatedly enters your home despite being unwelcome, even after police involvement. The Qwen and Kanon 2 configurations retrieve the first chunk which discusses the police entering an accused person’s home during a murder investigation, reversing the roles (police entering the home of the accused). The second retrieved passage concerns intimate partner violence and threats in an apartment building without discussing entrance. This example motivates better retrieval methods that measure not only topical similarity, but also specific legal logic needed to answer the query.

4.5.0.2 Analysis of performance per query type.

Table 5 presents the performance based on query type (legal expert or lay person) and province. Across methods, retrieval performance is generally worse on BC and Alberta and best on Ontario, but Gemini has the least variation between provinces showing robustness to legal domains. Comparing the query types, the nDCG is higher for legal experts across models but recall is mixed. We compare the difference in query style performance deltas between base and enhanced configurations in Fig. 4 and Fig. 5 (Appendix 11.1). In base configurations, performance is always better on queries simulating legal experts, with large gaps in nDCG@10. This illustrates how embeddings are not robust to the difference between layperson language and legal documents. Enhancements bridge the gaps in nDCG@10, likely due to IRG writing intermediate answers in legal style.

5 Generation↩︎

We present the baselines (§5.1), evaluation metrics (§5.2), and results (§5.3) for the generation task, and analyze the performance (§5.4).

5.1 Baselines and Evaluated Models↩︎

Table 10 shows the models we use for generation. We select one strong proprietary model (Gemini Flash 2.5) and two strong open source models (Qwen 3.5-9B and Gemma 4-E4B) for answer generation (Prompt [prompt:answer-generation] in the appendix). We use temperature of 0.³ Each generator is paired with retrievers based on embeddings of the same model family.

5.2 Evaluation Metrics↩︎

We test the generated answers along two dimensions: (a) how similar they are to ground truth answers, and (b) how grounded they are in retrieved documents. The latter measures the inverse of hallucination rate. We use a variation of FActScore [39] together with Gemini-2.5-Pro. We decompose the generated answer \(\hat{y}_q\) into a set of atomic claims \(A(\hat{y}_q)=\{a_{q,1},\ldots,a_{q,m_q}\}\). We then look at the proportion of claims in \(A(\hat{y}_q)\) that are supported by the evidence set, which is defined differently for each dimension below.

5.2.0.1 Accuracy against ground truth answers.

The evidence set is \(A(y_q)\), which is obtained by using the same LLM to decompose the gold standard answer \(y_q\) into atomic claims. We then compute the accuracy of claims as follows: \[\scalebox{0.85}{ \displaystyle Acc(\hat{y}_q, y_q) = \frac{1}{|A(\hat{y}_q)|} \sum_{a \in A(\hat{y}_q)} \mathbb{I} \left[ \max_{e \in A(y_q)} \operatorname{Supp}(a,e) =1 \right] }\] where \(\operatorname{Supp}(a,e)\) is an entailment-based support judgment returned by the factuality evaluator (see Prompts [prompt:factscore_groundedness]–[prompt:factscore_entailment] in the appendix). A claim is counted as grounded if at least one claim in the evidence set \(A(y_q)\) entails it. Following FActScore and X-FactAlign [40], this metric measures the fraction of atomic claims in the generated answer that are supported by the provided evidence, rather than independent legal correctness.

5.2.0.2 Groundedness in documents.

The evidence set is \(E_q\), under condition \(c \in \{\mathrm{\small oracle}, \mathrm{\small pipeline}\}\). In the pipeline condition, documents are gathered from retrieval methods: \(E_q=D_q^{r}\). In the oracle condition, we confine retrieval to only chunks from ground truth documents: \(E_q^{c}=D_q^{GT}\).⁴ The pipeline condition measures the impact that retrieval has on final answers and partially measures the ability of retrieval methods despite missing documents in the gold set. Query-level groundedness is thus similarly defined as: \[\scalebox{0.85}{ \displaystyle G(q,c) = \frac{1}{|A(y_q)|} \sum_{a \in A(y_q)} \mathbb{I} \left[ \max_{e \in E_q^{c}} \operatorname{Supp}(a,e) =1 \right] }\]

5.3 Results↩︎

6 presents groundedness and accuracy across generations and conditions. For each model family, we used the top configuration (Table 3).

5.3.0.1 Models generate claims unsupported by documents.

Across all setups, at least 20% of generated claims are unsupported by the documents.

In both conditions, Qwen creates the highest number of unsupported claims. This could be explained by generating longer answers averaging 14,685 characters, compared to 6057 for Gemma and 11,550 for Gemini. Qwen also produces many thinking tokens which sometimes get stuck on a wrong idea.

To investigate the nature of unsupported claims, we sampled 25 unsupported claims from the best performing model in each condition: Gemma in the oracle condition and Gemini in the pipeline condition, and categorized them in Tables 13 and 14 in Appendix, respectively. For Gemma oracle condition, approximately 38% of ungrounded statements are valid statements that establish the context of the query or finalize an answer, both trivially lacking from retrieved documents. Yet, 54% capture different hallucinations types such as confusing between general law vs. case specific statements, misapplication of legal tests, misattributing content between documents, and lack of sensitivity to details such as “shall not” vs. “may not”. In the Gemini-2 pipeline condition, around 60% of the claims were trivially failed, mostly from synthesizing texts and creating grounded statements too semantically far from the original texts. The other 40% of the ungrounded claims were hallucinations. Extrapolating this to results in Tab  6, estimate a lower bound of 8% hallucination rate with an upper bound of 29%.

5.3.0.2 Generated answers differ substantially from the gold standard, especially when based on different documents.

The accuracy of generated answers in the oracle condition is 64-76%, demonstrating substantial deviation. We analyze unsupported claims in Appendix 11.2. The accuracy is much lower in the pipeline condition, when it is based on different documents than the gold standard answer. These findings illustrate the importance of a good retrieval component.

5.4 Analysis↩︎

Table 5 shows that subgroup trends in generation do follow retrieval trends. Legal-expert queries have higher nDCG@10 for all three family-matched pipelines, but this does not translate uniformly into better answers. Gemini and Qwen improve on legal-expert queries for both groundedness (0.747 to 0.784, and 0.704 to 0.717) and accuracy against ground truth answers (0.419 to 0.492, and 0.423 to 0.457). Gemma, however, has higher groundedness and much higher accuracy for layperson queries (0.767 and 0.603) than legal-expert queries (0.723 and 0.445), despite similar nDCG@10. Thus, legal phrasing helps retrieval ranking, but improvements on generation quality remains to be seen. Our previous categorization of groundedness errors shows there can be a lot of noise in judging.

Jurisdictional results show a similar decoupling. Gemini is most accurate and grounded on the ‘Other’ subset of provinces, and performs worst on Ontario for accuracy (0.284), even with strong retrieval scores. Gemma is weakest on British Columbia across retrieval and generation, and strongest on Alberta for generation. Qwen is more stable across jurisdictions in generation. Across each configuration, there is high deviation across performance on different jurisdictions which highlights the need for more robust systems or individual systems tailored for each jurisdiction.

6 Conclusion↩︎

We introduced CanLegalRAGBench, a benchmark for evaluating RAG over Canadian case law. CanLegalRAGBench differs from prior benchmarks by focusing on realistic queries from users of public-facing legal assistants. Evaluation on various RAG systems reveals a high hallucinate rates, but it also highlights the limitations of automatic evaluation of relevance and accuracy. Future work should develop additional benchmarks to cover other aspects of Canadian law such as case laws in French and Quebec civil-law sources. By releasing the code and data, we aim to support transparent comparison and more reliable development of RAG systems for legal information tasks.

Limitations↩︎

6.0.0.1 Data Coverage.

CanLegalRAGBench only covers Canadian case law in select courts and provinces, and only focuses on English case laws. Future work can develop benchmarks that cover bilingual Canadian law, and Quebec civil-law sources, statute and regulation. In addition, due to the cost of annotations, our dataset is only meant for testing rather than training models.

6.0.0.2 Evaluation.

Our findings highlighted the challenge in evaluating document relevance and answer correctness automatically. To overcome these limitations, we employed both expert evaluation and various types of analyses. However, expert annotation is costly and doesn’t scale. Future work should focus on developing reliable evaluation metrics for legal RAG. Moreover, our evaluation largely ignored the temporal relevance and the precedential value of courts, which could be important to determine document relevance. We leave this challenge to future work.

Ethics Statement↩︎

6.0.0.3 Data Access.

We used two sources of case laws: the CLD dataset, which is publicly available, and a dataset which was provided to us by Caseway with permission to publish it. The benchmark itself combines queries generated by an LLM with answers collected from experts, and we make it publicly available along with our code under an open MIT license. Data from courts are subject to their own licenses which have limitations for commercial use. By releasing the data, prompts, model outputs, and evaluation code, we aim to support transparent comparison and more reliable development of legal RAG systems for Canadian legal information tasks.

6.0.0.4 Data Privacy.

While case laws contain names and other personal information, all data we use in this work was already public record.

6.0.0.5 Expert Annotation.

The work was approved by our institutional review board (IRB). We hired annotators internally at our institution and through Upwork and compensated them $20 CAD per hour for students and $40 per hour for the professional paralegal. All are higher than the minimum wage. We explained to them the goal of improving automated systems for legal research and answer synthesis. To demonstrate how costly it is to collect a moderate size dataset like ours while still paying experts fairly, overall we spent around $6,000 CAD on data collection and manual evaluation. We collected consent forms from annotators where we detailed the context of our research and publication goals. We highlighted the risk of viewing sensitive content in court cases which they agreed to.

6.0.0.6 Implications.

Performance on CanLegalRAGBench is not directly indicative of real-world performance, and we do not advocate for replacing legal expertise with AI. People should still consult lawyers when they face legal issues, and lawyers are still responsible for the quality and accuracy of their legal research, whether or not it was supported by AI tools.

Acknowledgments↩︎

We would like to thank Alistair Vigier and Ajay from Caseway for their domain expertise and support, and the legal expert annotators: Hugo Mak, Hooria Hayat, Michelle Lee, Kiona Choi Smith, Ryan Elmhirst, Manatsa Munyukwi, and Cynara Onyemordi, for their contribution to our benchmark. This work was funded by NSERC Alliance and Mitacs Accelerate grants in collaboration with Caseway. The authors are also supported by the Vector Institute, Canada CIFAR AI Chairs program, and NSERC Discovery grants. This research was enabled in part by computational resources and services provided by the Digital Research Alliance of Canada and free credits from Isaacus AI to use the Kanon 2 Embedder.

7 Annotation Details↩︎

We accepted students who had above-average grades and/or prior work experience under a lawyer. Annotators were provided a UI that presented all the queries for a given seed query. They were instructed to first flag and skip nonsensical, ill-formed or very complex queries. Then, they were tasked with searching for relevant case laws. We allowed annotators to use common search tools that they would normally use in legal research, such as the Canadian Legal Information Institute (CanLii) and Thomson Reuters’ Westlaw.⁵ For each case law they retrieved, they were asked to select relevant text of each document. After completing the retrieval stage, they moved on to the answer phase. To speed up the annotation process, instead of asking annotators to phrase an answer from scratch, we provided the query and the selected relevant texts to Gemini 2.5 Pro to generate an answer, and instructed the annotators to verify or edit to a final answer. This design was informed by authors who are legal experts, who indicated that searching for relevant case laws is the most challenging step of legal research. To prevent AI influence in the retrieval stage, once the annotators moved on to the answer generation stage, the retrieval stage was locked.

The following are the original instructions given to instructors. We received feedback from the annotators and emailed clarifications and updates throughout. For example, part way through, we lifted the max document limit while stating they should still aim to include less than 10 documents per query on average. We also allowed them to use statutes part way through to improve the speed, though we still required at least 1 case law per query.

8 Prompt Templates↩︎

9 Additional Information↩︎

9.1 Court Statistics↩︎

9.2 Province Statistics↩︎

9.3 Query Variations↩︎

10 Models Used↩︎

2pt

11 Additional Results↩︎

11.1 Full Results↩︎

11.2 Categorization of Unsupported Claims (Accuracy)↩︎

Of the three models, Qwen achieves the highest accuracy by a large margin. With that said, to better understand the type of errors that Qwen made, we sampled 25 unsupported claims and manually analyzed them (12). We found that Qwen’s answers were significantly longer than the gold standard answers (that had an average of 3883.5 characters), and often contained claims that were either irrelevant, too detailed, or relevant but miss the point of the question (such as talking about factors that impacted other applications for employment insurance when it’s already established the client is ineligible). A small number of errors include hallucinations (i.e. claims unsupported by the documented they cite), false positives (claims that appeared in the gold answer), and relevant claims missing from the gold answer.

11.3 Categorization of Unsupported Claims (Groundedness)↩︎

References↩︎

---

1. Code and data are available at https://github.com/NLP-UBC/CanLegalRAGBench.↩︎

2. The portion of the data used in this work will be made available upon publication.↩︎

3. We re-run some examples for Qwen with a temperature of 0.05 when it entered thinking loops with a temperature of 0.↩︎

4. We used the Qwen Embedder 8192 chunk size with reranking configuration across oracle conditions for equal evaluation.↩︎

5. While the search in these tools may involve AI, annotators were still responsible for constructing the queries and manually selecting among the retrieved cases. To improve the completeness of the final answers, we also allowed them to also cite laws, but we required that each query have at least one case law.↩︎