1 Introduction↩︎

With the rapid advancement of AI technologies, large language models (LLMs) have been increasingly applied to a wide range of domain-specific applications, such as legal [1], [2], medical [3], and finance [4]. In these knowledge-intensive fields, real-world problems often require the integration of extensive prior knowledge together with complex logical reasoning. Enabling LLMs to handle professional tasks in such domains, therefore, necessitates the coordinated use of multiple capabilities, including knowledge retrieval [5], reasoning [6], planning [7], and decision making [8].

In the legal domain, LLMs have been explored as tools to assist with real-world legal tasks traditionally performed by legal practitioners. Practical legal scenarios are highly diverse and complex, requiring the ability to retrieve and apply legal knowledge, identify and clarify legally relevant facts, plan multi-step problem-solving strategies, conduct structured legal reasoning [9], and verify and analyze conclusions [10], [11]. This raises a fundamental question: To what extent can LLMs handle complex legal tasks and reason like human practitioners?

Prior research has attempted to evaluate LLMs on legal tasks [12]–[16]. However, the construction of these benchmarks often fails to reflect the practical demands of realistic legal workflows. Specifically, existing work suffers from the following three limitations:

1.0.0.1 Lack of Realistic Task Construction.

Some previous benchmarks are directly derived from judicial or bar examination questions (e.g., LAIW [17], LEXAM [18]), while others contain tasks that are deliberately simplified (e.g., LegalAgentBench [7], Legalbench [19]). In real-world legal settings, user queries are frequently ambiguous, incomplete, or strategically framed, with key facts omitted or obscured. Only by incorporating such characteristics can evaluations more accurately capture LLM performance when acting as legal practitioners.

1.0.0.2 Insufficient Modeling of Legal Reasoning.

Logical and structured legal reasoning steps are essential when LLMs handle legal problems. Prior benchmarks claim to assess reasoning abilities. LexEval [20] evaluates “Logic Inference” capability while their designs overlook the distinctive nature of legal reasoning. Simple syllogistic reasoning [21] or IRAC (Issue, Rule, Application, and Conclusion)-style [22] evaluations primarily capture surface structure rather than substantive legal inference [23]. Even recent benchmarks, such as PRBENCH [24], rely on general-purpose scoring frameworks that are not tailored to the unique reasoning mechanisms of the legal domain. As a result, these approaches fail to adequately assess the complex, context-dependent reasoning required in real legal practice.

1.0.0.3 Coarse and Uniform Evaluation Metrics.

Moreover, existing evaluations typically rely on coarse criteria such as whether the final conclusion is correct, whether relevant statutes are cited, or whether a syllogistic structure is followed  [15], [20], [25]. However, such uniform evaluation metrics fail to capture the diversity and complexity of legal practice across different scenarios. Fine-grained and case-specific evaluation rubrics are essential for realistically assessing legal practitioner LLMs.

Due to these limitations, existing benchmarks have not achieved a comprehensive, in-depth, or a realistic assessment of LLM capabilities in legal practice. To address this gap, we propose a Practical Law Benchmark (PLawBench) designed to evaluate LLMs acting as legal practitioners under realistic, knowledge-intensive, and reasoning-driven conditions. This work has three main contributions:

1.0.0.4 Authentic Simulation of Legal Practice.

We evaluate LLMs using a hierarchical framework adapted from authentic legal workflows, comprising three levels: Public Legal Consultation, Practical Case Analysis, and Legal Document Generation (see Figure 1) ³. To mirror the cognitive challenges of real-world practice, we adapt tasks from actual cases and deliberately incorporate realistic noise — such as vague queries, emotional narratives, and omitted facts. This design moves beyond idealized benchmarks to rigorously test whether models can distill relevant legal facts from the ambiguity inherent in professional environments.

1.0.0.5 Fine-Grained Reasoning Steps.

Beyond evaluating the final conclusions, our benchmark explicitly incorporates fine-grained legal reasoning steps into task design and evaluation. This allows us to examine whether LLMs can perform multi-stage legal reasoning, including issue identification, fact clarification, legal analysis, and conclusion validation, rather than relying on shallow pattern matching or surface-level reasoning.

1.0.0.6 Task-Specific Rubric.

Our framework employs personalized, task-specific rubrics to assess substantive legal reasoning beyond mere outcomes. Legal experts follow a two-stage annotation process: first defining a reasoning-oriented framework for each task type, then tailoring specific criteria to individual case scenarios. This approach ensures evaluation is both principled and context-sensitive, enabling a fine-grained and realistic assessment of LLM performance in practical settings.

2 Related Work↩︎

Existing research has proposed a variety of benchmarks to evaluate AI systems in legal domains, primarily focusing on models’ ability to understand and apply legal knowledge. Early efforts assess legal AI systems through tasks such as legal information extraction and legal reasoning. Representative benchmarks include JEC-QA [27], which evaluates multi-hop reasoning based on Chinese bar examination questions; CaseHOLD [28], which focuses on identifying legal holdings in U.S. appellate decisions; [29], which evaluates the identification of overruling relationships in U.S. Supreme Court cases; and SARA [30], which examines statutory reasoning in U.S. tax law. Many of these benchmarks adopt syllogistic reasoning structures or legal writing frameworks such as IRAC.

More recently, comprehensive benchmarks have been introduced to provide multi-dimensional evaluations of legal AI systems. For example, LawBench [15] evaluates legal LLMs across cognitive levels, including memorization, understanding, and application; LegalBench [19] comprises a wide range of tasks spanning multiple forms of legal reasoning; LexEval [20], the largest Chinese legal benchmark to date, covers diverse legal tasks at scale; and LAiW [17] explicitly structures legal reasoning into layered components aligned with legal practice logic.

Despite their contributions, existing benchmarks often fail to reflect the complexities of real-world legal practice. First, tasks like statutory memorization or exam-based QA (e.g., LawBench [15]) diverge significantly from authentic scenarios where practitioners must distill relevant facts from ambiguous, incomplete client narratives. Second, current benchmarks typically isolate interdependent tasks — such as separating statutory retrieval from judgment prediction — ignoring the holistic reasoning chain required in practice. Furthermore, evaluation metrics for document generation (e.g., ROUGE-L) fail to capture professional legal standards. In contrast, our benchmark integrates three representative authentic tasks with fine-grained, expert-designed rubrics, ensuring comprehensive and practically relevant evaluation. A comprehensive comparison of PLawBench with existing legal benchmarks is illustrated in Figure 2, while a more in-depth analysis of existing benchmarks is provided in Appendix 8.

3 PLawBench↩︎

3.1 Framework↩︎

3.1.0.1 Overall Framework

As illustrated in Figure 1, we organize legal tasks into three hierarchical levels based on the real-world workflow of legal practitioners: public legal consultation, practical case analysis, and legal document generation. These three tasks are designed to evaluate distinct but complementary capabilities of LLM-based agents: identifying legal issues and clarifying key facts, analyzing disputes through legal reasoning, and summarizing and presenting legal claims in a structured and formalized manner, respectively.

3.1.0.2 Rubric Construction

Our rubric design adopts a risk-tiered perspective. It is grounded in principles emphasized by the EU AI Act’s [31] risk classification framework, AI auditing standards, and legal risk management practices: the risks posed by AI systems must be evaluated in relation to specific tasks, concrete contexts, and affected stakeholders, rather than relying solely on abstract accuracy metrics [32], [33].

In legal settings, traditional notions of “accuracy” are insufficient to capture task complexity. In many scenarios, information may be factually correct yet operationally unusable, legally accurate but high-risk in terms of rights implications, or correct but lacking essential procedural guidance — all of which can result in substantial harm to users.

Accordingly, we abstract task-level rubric items into six unified evaluation dimensions, each corresponding to a fundamental component of legal practice: (1) Issue and Fact Identification, (2) Legal Reasoning, (3) Legal Knowledge Application, (4) Procedural and Strategic Awareness, (5) Claim and Outcome Construction, and (6) Professional Norms and Compliance as shown in Table 1. These dimensions are designed to capture both substantive legal reasoning and practical decision-making, rather than surface-level task performance. Building on the six unified evaluation dimensions, we map them onto specific legal task settings and integrate them into rubric design to ensure that risk assessment aligns with task‑specific legal risk points.

As shown in Figure 3, this unified rubric design allows PLawBench to move beyond fragmented task evaluations and provides a principled framework for assessing whether LLMs exhibit the core competencies required in real-world legal practice.

3.1.1 Public Legal Consultation Task↩︎

3.1.1.1 Task Definition

The public legal consultation tasks primarily evaluate a model’s ability to identify legal issues and clarify key facts. In real-world consultations, users often describe their situations ambiguously, omit critical information, or intentionally conceal unfavorable details. In some cases, emotional expressions overlap with criminal law terminology, for example, users may claim that they were “defrauded”, even when the statutory elements of fraud under criminal law are not satisfied. In other scenarios, users’ subjective perceptions diverge substantially from legally accurate classifications; for instance, in cases involving illegal fundraising, users may believe they were “scammed”, while the applicable criminal offense is illegal fundraising rather than fraud.

Consequently, effective legal consultation relies on iterative questioning to clarify the user’s intent and reconstruct legally relevant facts. This requires models to demonstrate the ability to actively elicit key information by posing appropriate follow-up questions, gradually identifying the facts that materially affect legal outcomes.

3.1.1.2 Rubric

Compared to traditional legal QA tasks, which are based on pre-structured, exam-style questions and primarily emphasize answer correctness, the public legal consultation task is designed to assess a model’s ability to identify legal risks that emerge in real-world practice. Such risks must be inferred from the model’s performance in issue and fact identification, legal reasoning, and legal knowledge application when responding to user narratives that are ambiguous, emotionally charged, or incomplete. These deliberately omitted facts represent the core legal risk points in real consultation scenarios, and the model must identify and clarify them to detect and manage such risks.

To enable systematic evaluation, we design task-specific rubrics focusing on these critical aspects. Each rubric targets a indispensable fact that is not explicitly stated in the prompt, must be clarified through inquiry, and plays a decisive role in legal qualification or the allocation of rights and obligations. A detailed description of the rubric design and examples is provided in the Appendix 10.1.

3.1.2 Practical Case Analysis Task↩︎

3.1.2.1 Task Definition

The practical case analysis tasks focus on evaluating LLMs’ capabilities of analyzing disputes through legal reasoning. We assess LLMs from two perspectives.

First, all questions in this task are simulated with scenarios that users are likely to encounter in practice. To ensure comprehensive coverage, we construct a legal knowledge framework spanning multiple legal domains. Second, given the importance of statutes of limitations and legal theory application in practice, we introduce two dedicated scenarios to explicitly evaluate models’ capabilities in handling temporal legal constraints and doctrinal reasoning.

Recognizing the diversity and complexity of real-world legal consultation demands, we adopt a practice-oriented classification system. We systematically analyze multiple layers of reference materials, including official business classification systems of top-tier law firms in China, categorization standards from global legal service rankings, and business structures used by consumer-facing legal consultation platforms (e.g., China Legal Service Network⁴).

3.1.2.2 Rubric

Unlike law‑retrieval tasks that merely restate statutory provisions or legal judgment prediction tasks that output opaque conclusions, legal case analysis requires coherent integration of Legal Reasoning, Legal Knowledge Application, and Claim & Outcome Construction. In real legal practice, the factual background of a case is often relatively clear, yet the application of legal rules, element assessment, allocation of rights and obligations, and procedural requirements remains highly complex.

In designing the rubrics, we require models to grasp the conditions under which legal rules apply, identify the facts that are legally determinative, and preserve both logical and doctrinal coherence throughout the reasoning process.

To move beyond overly formalistic legal reasoning evaluation, we revise our rubric design to emphasize substantive legal reasoning. Recognizing that law is fundamentally a reasoning-driven practice, we adapt classical legal syllogism into a framework consisting of Legal Statutes, Case Facts, Reasoning, and Conclusion. While Legal Statutes and Case Facts function as the major and minor premises, respectively, we highlight the critical role of the reasoning step that bridges abstract legal concepts and concrete factual descriptions.

In practice, legal premises rarely align directly with case facts, requiring explicit deductive justification to subsume specific facts under legal concepts. This deductive reasoning step is therefore treated as a core evaluative component and assigned a higher weight than traditional elements. As shown in Figure 4, scoring strictly enforces logical coherence: credit for reasoning is granted only when the response follows a correct logical sequence. In contrast, conclusions, facts, and legal provisions are evaluated with lower weights.

Finally, to better serve lay users, we adopt an output structure that presents the claim first, followed by the complete reasoning process, ensuring transparency and persuasive clarity. A detailed explanation of the reasoning framework and scoring criteria is provided in the Appendix 10.2.

3.1.3 Legal Document Generation Task↩︎

3.1.3.1 Task Definition

The legal document generation tasks evaluate the ability to summarize and present legal claims in a structured legal format. Unlike prior benchmarks[34] that provide structured case descriptions and directly request document drafting, our dataset emphasizes simulation of the full real-world legal workflow.

Specifically, we intentionally increase the complexity of legal elements by introducing multiple related parties, ambiguous responsibility boundaries, layered rights and obligations, and scenarios involving the application of both legacy and newly enacted laws. In addition, we innovate in prompt presentation by replacing standardized case descriptions with disorganized narratives from the client’s perspective. These narratives incorporate unreasonable demands, incorrect legal terminology, and redundant or misleading information to approximate real-world lawyer-client interactions.

Under this setting, evaluation extends beyond document drafting. Models are required to complete the entire workflow of information filtering, legal relationship analysis, litigation strategy formulation, and document drafting. This enables a comprehensive assessment of the model’s problem-solving ability in realistic legal practice scenarios.

3.1.3.2 Rubric

The legal document generation task primarily corresponds to the dimensions of Claim & Outcome Construction, Procedural & Strategic Awareness, and Normative Compliance in our framework.

In designing the rubrics, the model must identify legally relevant facts and claims, make sound procedural and strategic judgments, construct structured and persuasive legal claims, and ensure that the document adheres to required formal and normative standards.

A detailed specification of rubric components and penalty rules is provided in the Appendix 10.3.

3.2 Dataset↩︎

3.2.1 Data Procurement and Filtering↩︎

We present a fine-grained annotated dataset derived from two primary sources: expert-edited judgment documents from China Judgments Online⁵, individual consultation records, and complex cases curated by leading Chinese law firms. These open-ended cases represent more intricate disputes that exceed the complexity of standard exam-based questions. Following initial collection, we performed a rigorous screening and anonymization process to finalize 850 representative cases covering diverse legal scenarios.

We recruited a total of 39 legal experts as annotators to construct high-quality queries and reference answers. These experts include legal practitioners from law firms and Ph.D. candidates from top-tier universities. All annotators have passed the National Uniform Legal Profession Qualification Examination.

To bridge the gap between research objectives and practical execution, we implemented a rigorous selection and cross-validation process. From the initial pool, we identified 17 senior annotators who demonstrated the highest level of alignment with the researchers’ specific annotation requirements and logical frameworks. This selected group led the validation phase to ensure consistency across the dataset. Given that legal reasoning seldom admit a single correct answer, we prioritized the soundness of the analytical process and adherence to legal logic over traditional redundant labeling schemes. This high-intensity process requires an average of three hours per data instance. Detailed information regarding the annotation process is provided in Appendix 9.1.

3.2.2 Dataset Statistics↩︎

The benchmark comprises 850 questions covering a wide range of legal functions. Table ¿tbl:tab:dataset95stats95transposed? presents a summary of key statistics across the three legal tasks, including the number of questions, average prompt length, and the distribution of rubric items. These statistics illustrate the differences in scale, task complexity, and evaluation granularity among the tasks. A detailed description of the dataset composition and additional statistics is provided in Appendix 9.

4 Experiment↩︎

4.1 Experimental Setup↩︎

We conduct experiments across all three tasks: Public Legal Consultation (PLC), Practical Case Analysis (PCA), and Legal Document Generation (LDG). Following previous work [7], we select 24 representative LLMs, covering the GPT [35] , Gemini [36], Claude [37], Qwen [38], DeepSeek [39] , and Kimi [40] series. All LLMs are accessed through API calls. We utilize default hyperparameters for reproducibility, solely adjusting the max_token limit to 16k / 32k to ensure response completeness. To ensure structured reasoning, we design task-specific prompts for each task, with full model and experimental details provided in Appendix 11 and 12.

4.2 Judge Model Selection↩︎

Given the high complexity of manual evaluation, we adopt an LLM-as-a-Judge approach. To ensure reliability, we conduct a pilot study to identify the optimal evaluator.

We sample at least 10% of the questions from each legal domain. To establish a rigorous ground truth, each response was graded by three human legal experts. We then calculate the alignment (Pearson and Spearman correlations) between human scores and candidate LLM judges (Gemini-3.0-Pro-Preview, GPT-5.1, and Qwen3-Max). As shown in Table 3, Gemini-3.0-Pro-Preview demonstrate the highest degree of concordance with human experts and is substantially selected as our final judge. Details of the pilot study and scoring prompts are provided in Appendix 13 and 12.2.

4.3 Evaluation Metrics↩︎

We adopt a fine-grained evaluation approach based on specific rubrics. For Task 2 PCA, we evaluate performance across four dimensions: Conclusion, Case Facts, Reasoning, and Legal Statutes. For Task 3 LDG, we distinguish between Plaintiff and Defendant documents. To normalize performance across questions with varying difficulty, we report the Scoring Rate, calculated as the aggregate of item-level scores: \[\text{Scoring Rate} = \frac{S_{\text{model}}}{S_{\text{total}}} = \frac{\sum_{i=1}^{N} s_{i}}{\sum_{i=1}^{N} m_{i}},\] where \(N\) is the number of rubric items, \(s_i\) is the score obtained for the \(i\)-th item, and \(m_i\) is its maximum possible score.

4.4 Experimental Results↩︎

4.4.1 Overall Performance↩︎

Table 2 summarizes the overall performance of the evaluated models. We find that:

1) Task-Dependent Capability Profiles. Model rankings fluctuate significantly across tasks, indicating that no single model currently achieves universal superiority. While GPT-5.2 and Claude-4.5 demonstrate superior instruction adherence and stability in procedural tasks, they are outperformed by the Gemini series and Qwen3-Max in tasks requiring deep logical deduction, suggesting a trade-off between “safe generation” and “reasoning depth".

2) Distinct Family Characteristics & Hallucinations. We observe distinct error patterns: The Gemini series excelled in constructing reasoning chains but occasionally struggled with evidence complexity in consultation tasks. Conversely, the GPT series showed exceptional summarization skills but exhibited a specific type of hallucination: citing repealed or outdated statutes (as detailed in Table 4), likely due to temporal misalignment in pre-training data.

3) Scaling and Generational Effects. Performance correlates positively with model scale and iteration. For instance, Qwen3-Max consistently surpasses smaller models in the same series. Notably, GPT-4o lags significantly behind the GPT-5 series, ranking as the lowest-performing frontier model in our set, which highlights the substantial generational leap in legal reasoning capabilities.

4.4.2 Task-Specific Analysis↩︎

To diagnose the root causes of performance gaps, we conduct a comprehensive error analysis on low-scoring responses. Tables ¿tbl:tab:task195errors?, 4, and 5 summarize the distribution of error types across models (see Appendix 14 for qualitative details). While frontier models achieve comparable aggregate scores, this deeper inspection reveals significant divergences in their failure patterns:

1) The “Fluency-Logic” Gap in Consultation. In Task 1 PLC, while models generated fluent responses, they suffer from Insufficient Grasp of Factual Causality (Table ¿tbl:tab:task195errors?). They often fail to link reported events to legal consequences accurately, missing the necessity of corroborating key facts before offering advice.

2) Fragility of Legal Reasoning Chains. In Task 2 PCA, score rates for Statute and Reasoning were consistently lower than the average. Our detailed analysis reveals that this is not merely an accuracy issue but a structural one: models frequently fail to follow the legal syllogism (major premise → minor premise → conclusion). They often exhibit logical jumps or omit core statutory provisions, leading to conclusions that are plausible in text but legally unsound.

3) “Blind Compliance” in Document Generation. In Task 3 LDG, scores for Plaintiff drafting were notably lower than for Defendant drafting. This disparity stems from a lack of legal judgment. As shown in error analysis, models — particularly DeepSeek and Kimi — are prone to Subject Suitability Errors (Table 5). They tend to follow user instructions blindly (e.g., filing a lawsuit for a client with no standing or in the wrong jurisdiction) rather than proactively identifying procedural flaws, a core requirement for the autonomous Plaintiff role.

Figure 5 reveals a universal performance drop in sequential tasks, confirming that ordered, multi-step logic significantly increases difficulty. This degradation is most pronounced in the Reasoning dimension, where top models (e.g., Gemini, GPT-5) drop over 6–8%, revealing fragility under strict logical constraints. An exception is GPT-4o, which shows a minimal drop (0.2%). Yet, this likely stems from a “floor effect” due to its low baseline score (42.00), rather than genuine adaptability.

5 Conclusion↩︎

We present PLawBench, a benchmark evaluating LLM agents within realistic legal workflows. Constructed with legal experts, it spans three core tasks covering 13 scenarios, 850 questions, and approximately 12,500 fine-grained rubric items. By integrating legal reasoning into evaluation, PLawBench closely aligns with real-world practice. Experiments reveal that current models struggle with legal knowledge and reasoning, achieving suboptimal performance. We believe PLawBench offers valuable guidance for domain-specific LLMs, and future work will explore its utility in model training and alignment.

6 Ethical Considerations↩︎

PLawBench introduces a benchmark for evaluating LLMs’ capabilities in the legal domain. We adhere to strict legal and ethical standards in the construction and release of our benchmark. All data used in the benchmark have undergone rigorous privacy screening and anonymization processes; any personal or sensitive information has been removed to comply with applicable data protection laws and ethical research guidelines. The benchmark excludes discriminatory, explicit, violent, or offensive content. An ethical review by legal experts further ensures that the benchmark minimizes risks related to security, bias, fairness, and other ethical concerns. During the preparation of this work, we used AI tools only for language polishing and editing. Our work aims to provide a legally compliant and ethically sound foundation for advancing AI capabilities in law, upholding the principles of fairness, transparency, and social responsibility.

7 Limitations↩︎

We acknowledge several limitations of this work. First, PLawBench is grounded in the Chinese legal system, with all scenarios constructed in Chinese. Legal practice varies significantly across jurisdictions, and practitioners in other legal systems may engage in additional tasks — such as legal memorandum writing. This design choice reflects a deliberate trade-off. Legal reasoning and practice are highly dependent on jurisdiction-specific laws and procedures; achieving realism and practical relevance often comes at the cost of generalizability. In future work, we plan to collaborate with Legal AI researchers from other countries to extend this benchmark to additional legal systems and explore cross-jurisdictional legal agent capabilities.

Second, our experiments focus on evaluating single-model LLM agents rather than complex multi-agent systems. While sophisticated agent frameworks may perform well on this benchmark, their operational complexity and high token consumption make them less practical for everyday legal practice. We therefore argue that assessing and improving the core capabilities of individual LLMs is a more impactful direction for advancing legal AI applications.

Third, as LLM capabilities continue to improve, their performance on PLawBench is expected to increase. To support ongoing evaluation, we plan to launch an online platform that continuously benchmarks state-of-the-art models and updates results accordingly.

Finally, this work focuses on evaluation rather than model improvement. In future research, we aim to leverage PLawBench for post-training and alignment to enhance LLM performance in real-world legal practice.

8 Survey on Legal Benchmarks↩︎

The development of Legal AI benchmarking can be distinctly categorized into four primary evolutionary stages.

The Era of Early Benchmarking and Classification Tasks (2014-2020): During this foundational period, research was predominantly centered on Legal Judgment Prediction (LJP) and basic information extraction, with models typically operating within text classification frameworks to identify factual features. For instance, the COLIEE competition (est. 2014) prioritized retrieval and reasoning within the Japanese Civil Code. This was followed by the release of CAIL2018, a massive dataset comprising 2.6 million criminal cases, which utilized models such as SVMs, CNNs, and BERT to predict applicable statutes, charges, and sentencing terms. Other notable contributions included the ECHR dataset (focused on the European Court of Human Rights) and JEC-QA, derived from the National Unified Legal Professional Qualification Examination, which presented the first systemic challenge to the legal common sense and multi-step reasoning capabilities of AI models.

The Transition to Specialization and Retrieval Benchmarks (2021-2022): Research subsequently shifted toward domain-specific skills such as legal case retrieval and summarization. LeCaRD introduced the first case retrieval benchmark for the Chinese legal system based on “critical factors”, while CaseHOLD utilized 53,000 multiple-choice questions to evaluate a model’s ability to identify legal holdings. Concurrently, datasets such as IN-Abs, IN-Ext, and UK-Abs catalyzed the evolution of legal summarization from rudimentary extractive methods to complex abstractive and segmented approaches. Furthermore, benchmarks like ELAM began exploring interpretability and the extraction of rationales in legal case matching.

The Period of LLM Reasoning and Comprehensive Evaluation (2023-2024): With the rapid proliferation of LLMs, evaluation dimensions became increasingly stratified and cognitive-oriented. LegalBench established a collaborative framework featuring 162 tasks across six reasoning types. Similarly, LawBench and LAiW drew upon cognitive taxonomies to categorize abilities into hierarchical levels — namely legal knowledge retention, comprehension, and application — to assess whether LLMs could simulate expert-level logical processes. During this stage, LexEval expanded the evaluation scale and introduced legal ethics and privacy as independent metrics, while EQUALS, LLeQA, and CLSum focused on generating evidence-based legal responses supported by large-scale knowledge bases.

The Frontier of Logical Process and Reasoning Chain Evaluation (2025-Now): The latest benchmarks have transcended simple output comparison to scrutinize the internal quality of a model’s reasoning process. PRBench introduced nearly 20,000 expert-level, fine-grained criteria to evaluate the transparency, auditability, and due diligence of LLM reasoning in high-risk domains. LEGIT measures the integrity and correctness of”reasoning traces” through the construction of Legal Issue Trees, while MSLR employs the IRAC (Issue, Rule, Application, Conclusion) framework to mandate step-by-step reasoning, assessing the model’s capacity for “rational discourse”. Finally, LegalReasoner utilizes step-by-step verification and error-correction mechanisms to further address the logical fragility of LLMs when analyzing complex legal cases (See More Details in Table [tab:survey]).

The evaluation of AI’s Legal Capabilities has pivoted from outcome-based metrics toward a rigorous scrutiny of the reasoned elaboration underlying judicial decisions. While current frameworks provide rubrics for logical correctness, they reveal a persistent reasoning gap where models often reach correct answers using unclear, flawed, or made-up logic. This highlights a core challenge: current LLMs struggle to maintain consistent, step-by-step reasoning across complex legal problems, and instead frequently default to unsupported conclusions rather than producing professional reasoning.

To evaluate the distinctive contributions of PLawBench, we select a representative set of existing legal benchmarks for a comprehensive comparative analysis, as summarized in Table 6. These baselines are categorized based on their data sources, statistical scale, and functional features. Unlike traditional benchmarks that predominantly rely on academic textbooks or standardized bar exams, PLawBench distinguishes itself by focusing entirely on practical legal scenarios. Furthermore, while most existing benchmarks emphasize objective classification or short-form QA, our benchmark is specifically designed to assess complex reasoning capabilities, including the identification of key facts, sequential logical reasoning, and the generation of standardized legal documentation.

9 Details of Dataset↩︎

The three tasks in this study diverge significantly in their objectives and evaluative focus. Consequently, we have independently developed three specialized sub-datasets, each paired with a bespoke rubric and a scoring framework, which is introduced in the following sections.

9.1 Annotation Process↩︎

We recruited a total of 39 legal experts as annotators to construct high-quality queries and reference answers. These experts include legal practitioners from law firms and Ph.D. candidates from top-tier universities. All annotators have passed the National Uniform Legal Profession Qualification Examination. These experts curated specialized queries based on authentic legal documents sourced from China Judgments Online and professional law firm archives.

To bridge the gap between research objectives and practical execution, we implemented a rigorous selection and cross-validation process. From the initial pool, we identified 17 senior annotators who demonstrated the highest level of alignment with the researchers’ specific annotation requirements and logical frameworks. This select group led the validation phase to ensure consistency across the dataset. Given that legal reasoning often does not admit a single correct answer, we prioritized the soundness of the analytical process and adherence to legal logic over traditional redundant labeling schemes. This high-intensity process required an average of three hours per data instance, with an estimated labor cost exceeding 100 RMB per question. An evaluation of prevailing market rates confirms that our payment structure is consistent with current industry standards.

9.2 Construction Methodology↩︎

Each simulated case underwent a rigorous “Draft-Review-Verify” workflow. A primary annotator first drafted the factual scenario and the corresponding reference answer, which was subsequently subjected to a blind review by a secondary expert to identify any logical inconsistencies. Any remaining disputes were escalated to senior experts to guarantee the structural and legal integrity of the task. Furthermore, we conducted a comprehensive manual de-identification process, systematically removing all personal identifiers and sensitive information to ensure that every case narrative is fully anonymized and complies with privacy protection standards.

Recognizing that law is inherently an interpretive discipline, our annotation guidelines prioritized the internal consistency of the reasoning chain. Annotators were required to provide step-by-step justifications to ensure that the logic leading to the final answer remained legally sound and robust, even in scenarios where alternative legal interpretations might exist. Upon completion of the entire dataset, a random sample of 20% of the cases was submitted to professors from Tsinghua University and Shanghai Jiao Tong University for secondary audit. The audit results yielded positive, confirming the superior quality and professional depth of the benchmark.

9.3 Dataset Characteristics↩︎

Figure 6 provides a comprehensive hierarchical breakdown of the dataset composition. The inner ring delineates the three primary task categories, highlighting the dataset’s focus: Practical Case Analysis (n=750) constitutes the largest portion, followed by Public Legal Consultation (n=60) and Legal Document Generation (n=40). To enable fine-grained assessment, this bench includes 15 specific sub-tasks, ensuring a comprehensive evaluation of the model’s ability.

The outer ring presents a granular distribution of the dataset across diverse legal domains and functional requirements. It encompasses core civil and commercial sectors. Additionally, the dataset incorporates procedural elements like Litigation and Arbitration and specific drafting tasks such as Statements of Claim and Defense. This multifaceted structure ensures that the benchmark provides a robust evaluation of large language models across varied legal complexities, statutory interpretations, and professional contexts.

• Task 1: Public Legal Consultation Task
  This task focuses on user-centric inquiries, featuring an average prompt length of 1,171 characters. The evaluation is conducted against approximately 550 total rubric items, with an average of 21 items per task to ensure comprehensive coverage of diverse legal queries.

• Task 2: Practical Case Analysis Task
  Representing the most extensive category in terms of evaluation granularity, this task involves an average prompt length of 692 characters. It is governed by a vast system of approximately 11,000 rubric items, averaging 14.1 per task, to facilitate deep logical reasoning assessments.

• Task 3: Legal Document Generation Task
  As the most context-heavy category, this task requires processing extensive prompts averaging 2,574 characters. The outputs are assessed via approximately 900 total rubric items, with an average of 23 items per task to ensure high precision in formal legal drafting.

10 Details of Rubrics↩︎

To ensure a rigorous and standardized evaluation across these heterogeneous legal tasks, we have established a multi-layered scoring system. Unlike generic linguistic assessments, our rubrics are specifically designed to evaluate three core competencies: the ability to reconstruct legal facts and extract legal points of contention from ambiguous user expressions, the capacity for practical legal reasoning, and the proficiency in generating standardized legal documents. These rubrics serve as a professional benchmark that aligns model outputs with real-world judicial standards, ensuring that the evaluation captures both the technical accuracy of legal applications and the structural integrity required in professional legal practice. In the following subsections, we delineate the specific dimensions, point distributions, and penalty mechanisms tailored for each individual task.

10.1 Rubrics of Public Legal Consultation Task↩︎

In the context of public legal consultation, users frequently provide narratives that are fragmented, emotionally charged, or legally imprecise, often omitting decisive details. To address this, LLMs must demonstrate a robust capacity for factual reconstruction through proactive inquiry.

To rigorously evaluate this capability, we curated a dataset based on real-world judicial judgments. Legal experts adapted these cases into simulated consultee prompts, intentionally embedding informational pitfalls such as latent factual contradictions, misleading subplots, and the tactical concealment of critical evidence. Through these high-pressure scenarios, we assess whether the model can:

• Identify latent contradictions or logical inconsistencies within the user’s narrative;

• Detect missing facts that have been intentionally omitted or obscured;

• Strategically formulate follow-up questions that target the core legal relationships of the case.

Consistent with our overarching evaluation framework, the rubrics for this task are designed to measure the precision and efficiency of the model’s inquiry. Each scoring point is governed by three rigorous criteria (See Table [tab:rubrics95task1]):

• Uniqueness of Critical Fact: The inquiry must target a unique, irreplaceable factual element rather than posing generalized or redundant questions.

• Necessity of Inquiry: The points of inquiry must be demonstrably absent from the initial prompt and essential for filling a specific “informational gap.”

• Impact on Legal Determination: The inquiry must focus on circumstances that hold a pivotal position in determining the nature of the legal relationship or the allocation of liabilities.

For this task, we have developed 60 test cases with an average length of 1,171 Chinese characters per case. The evaluation framework is granular, featuring an average of 8.7 rubric items per case and totaling approximately 550 individual rubric scoring points across the entire dataset.

10.2 Rubrics of Practical Case Analysis Task↩︎

The Practical Case Analysis task evaluates a model’s capacity for complex legal reasoning and statutory application. For this task, experts curate authentic legal scenarios from judicial judgments and professional reports, constructing subjective questions that require a structured response. Each evaluation is grounded in a reference answer comprising four essential components: Legal Conclusion, Fact Interpretation, Reasoning Process, and Statute Location.

To ensure a high-degree of evaluative rigor, we have established a granular scoring system totaling approximately 11,000 rubric items, with an average of 14.1 items per task (See Details in 7). These rubrics are meticulously categorized into Sequential Reasoning and Non-sequential Reasoning dimensions to reflect the inherent logic of legal practice. The formulation of these rubrics follows four core standards:

• Granularity: Each item represents a single, discrete scoring point (e.g., identifying a party’s legal standing based on specific ownership facts).

• Precision: Scoring points must be legally accurate and formulated to eliminate interpretive ambiguity.

• Exclusivity: Rubric items are non-redundant and collectively encompass the entire evaluative scope of the case.

• Quantification: Points are allocated according to a predefined distribution, with the final score determined by the aggregate of validated rubric items.

Furthermore, we define specific structural requirements for different reasoning patterns to assess the model’s logical integrity (See Details in Table 7):

• Non-Sequential Reasoning Requirements: This applies when the order of argumentation does not affect the validity of the conclusion. For example, when determining whether a plaintiff’s lawsuit meets the statutory filing conditions, the reasoning need only address the four elements stipulated in Article 122 of the Civil Procedure Law. There is no mandatory sequence for evaluating these elements. When drafting, do not label steps (e.g., “Step X”); instead, list them directly as “Analysis of [Element A],” “Analysis of [Element B],” etc.

• Sequential Reasoning Requirements: This applies to issues that must follow a strict argumentative progression to maintain logical integrity. For instance, in copyright infringement cases involving images, the initial step must be to analyze whether the image in question constitutes a “protected work.” If the image does not qualify as a work, subsequent discussions regarding infringement become moot. Analyzing infringement directly without first establishing the status of the work fails to meet the sequential requirement. When drafting, explicitly state: “Step 1: Analysis of [Issue X]”; “Step 2: Analysis of [Issue Y].”

Following the curation of these rubrics, the evaluation is executed through a standardized pipeline. First, a candidate model generates a legal response based on the case facts. Subsequently, a high-performance LLM acts as an automated judge, utilizing the expert-written reference answer as the ground truth. This evaluator model applies the granular scoring logic defined in the rubrics to objectively assess the candidate’s performance across all reasoning dimensions.

10.3 Rubrics of Legal Document Generation Task↩︎

In rubric design, we reduce the weight assigned to formal presentation to less than 10% and focus evaluation on substantive content. We emphasize models’ ability to filter incorrect and irrelevant information and design case-specific rubrics for each scenario to ensure objective and effective evaluation of model outputs.

Specifically, the rubric of Plaintiff focuses on:

• Procedures And Jurisdiction: Identify the Applicable Court, Determine Proper Party, Select the most Favorable Cause of Action

• Key Claims and Best Strategy of Claim: Ensuring Reasonableness and Completeness of the Prayer for Relief, Capability to Accurately Calculate Amounts

• The Construction of Facts by Evidence: Logical Structuring of Facts and Reasoning, Preliminary Organization of Evidence

The rubric of the Defendant focuses on:

• Selection of Defense Strategies: Restructuring of Legal Relationships, Determining Core Defense Strategies, Demonstrates Solid and Coherent Argumentation

• Procedural Defenses of Jurisdiction and Admissibility: Jurisdictional Objections, Challenges to Legal Standing, Defenses Based on Procedural Timelines, Procedural Dismissal Motions

• Defenses Against Damages and Compensation Claims: Arguing of causation or intervening factors, Excluding non-compensable losses such as indirect damages, Challenging the calculation of plaintiff’s claimed amount

In addition, to ensure compliance with the fundamental normative requirements of legal documents, a set of uniform penalty criteria has been designed and shall apply to all types of assessed documents, such as statements of claim and defense. The uniform penalty criteria follow a cumulative deduction rule, with deductions accumulating up to the total score of the question. The specific criteria are as follows:

• Accuracy of Provisions Citation: The serial numbers and content of legal provisions cited in the document must be authentic, valid, and accurate. Each error in a legal provision (e.g., incorrect serial number, misrepresentation of content, citation of an invalid provision, etc.) will result in a 10-point deduction.

• Compliance with Instructions: The document must be generated strictly in accordance with user instructions or the template of the specified document type. If the document type generated does not match the required type (e.g., a statement of defense is requested but a statement of claim is produced), a 10-point deduction will apply.

• Document Standards: The following standards must be met simultaneously. For each standard not satisfied, a 10-point deduction will apply:

  • The format complies with statutory format requirements for the corresponding type of legal document.

  • Wording is precise and standardized, using formal legal terminology without idiom or ambiguous expressions.

  • All essential elements of the document are included, with no omission of core components (e.g., absence of party information, or lack of a clear statement of defense).

11 Model Details↩︎

In this section, we provide brief introductions to the 10 large language models evaluated in our experiments.

Claude-Sonnet-4.5

 [37] The latest iteration in Anthropic’s Claude series, Sonnet-4.5 balances high performance with cost-efficiency, demonstrating enhanced capabilities in nuanced instruction following and complex reasoning tasks.

Gemini-2.5-Pro

 [36] An upgraded version of Google’s multimodal model, Gemini-2.5-Pro features improved long-context understanding and more robust performance across diverse logic and coding benchmarks.

Gemini-3.0-Pro-Preview

 [56] The next-generation preview model from the Gemini family, designed with advanced native multimodal integration and superior reasoning depth for complex agentic tasks.

GPT-4o-20240806

 [57] A highly optimized snapshot of OpenAI’s GPT-4o, known for its rapid inference speed and state-of-the-art performance in multimodal processing and structured output generation.

GPT-5-0807-Global

The foundational version of the GPT-5 series, representing a significant leap in general intelligence, with substantial improvements in world knowledge and reasoning stability compared to its predecessors.

GPT-5.2-1211-Global

An iterated and refined version of the GPT-5 architecture, offering enhanced instruction adherence and reduced hallucination rates, specifically tuned for complex problem-solving.

DeepSeek-V3.2

 [39] Developed by DeepSeek, this model utilizes a Mixture-of-Experts (MoE) architecture to achieve competitive performance with top-tier proprietary models while maintaining high inference efficiency.

Kimi-K2

 [40] The second-generation model from Moonshot AI, Kimi-K2 specializes in lossless long-context processing, capable of handling massive information retrieval and synthesis tasks effectively.

Qwen3-235b-Think

 [38] The latest version of the Qwen model family. We use qwen3-235b-a22b-think, a flagship model featuring a unified framework that dynamically allocates a “thinking budget” to balance latency and performance, specifically optimized for complex multi-step reasoning.

Qwen3-Max

The most capable model in Alibaba’s Qwen3 lineup, designed to handle the most demanding tasks with comprehensive knowledge coverage and superior complex agent planning abilities.

12 Prompt Engineering↩︎

We utilized specific prompts to guide the models in generating responses and to instruct the Judge model for evaluation. Tables 11, [tab:prompt95task2], 12,through 13, 14, 15 display the core content of these prompts.

12.1 Response Generation Prompts↩︎

12.1.1 Prompt Design for Task 1: Public Legal Consultation↩︎

The prompt for Task 1 transitions the LLM from a passive information provider to an active legal interrogator by enforcing a rigorous fact-finding protocol (As shown in Tables 11). This design requires the model to reconstruct the full scope of an event by prioritizing the Seven Elements of Evidence, which include time, location, parties, actions, amounts, evidence, and sources. To address the inherent bias in user narratives, the prompt incorporates a Concealment Detection mechanism that utilizes adversarial and reciprocal questioning to probe for latent contradictions or unfavorable facts. Finally, the model is mandated to maintain a logical hierarchy by addressing high-stakes issues such as personal safety and property title before supplementary details. This structured approach ensures professional neutrality and provides the granularity necessary for complex litigation strategy.

This is an example of adapted case (Shown in Table [tab:adapted95case]). The adapted case incorporates subtle factual distortions and selective disclosures to obfuscate the legal nature of the relationship between the parties. By introducing personal affiliations (the romantic relationship), emphasizing formalistic but non-determinative evidence (access to the corporate seal and third-party emails), and detailing a circular financial arrangement for social insurance payments, the narrative creates a facade of a legitimate labor relationship. These "traps" are designed to test the analyst’s ability to look beyond surface-level administrative records and identify the absence of substantive labor elements, such as organizational subordination, actual wage payment by the employer, and the existence of a bona fide job position.

12.1.2 Prompt Design for Task 2: Practical Case Analysis↩︎

The prompt for Task 2 evaluates the ability of Large Language Models to perform systematic legal deconstruction by enforcing a structured cognitive framework aligned with professional judicial standards (As shown in Tables [tab:prompt95task2]). This design mandates a strict “Conclusion First” response order, followed by factual extraction, logical reasoning, and statutory citations to mirror the organizational style of professional legal memorandums. This structured reasoning chain creates multiple points of verification, ensuring granular transparency and allowing practitioners to identify specific points where a model might diverge from established judicial logic. Furthermore, the step-by-step requirement necessitates a direct alignment between case facts and specific legal provisions, providing an explicit mapping that allows experts to audit the precision of the reasoning. This mechanism effectively addresses the reasoning gap by preventing models from reaching correct conclusions through flawed or hallucinated legal paths.

12.1.3 Prompt Design for Task 3: Legal Document Generation↩︎

The prompt for Task 3 evaluates the proficiency of Large Language Models in professional legal drafting by requiring the transformation of informal client narratives into formal legal instruments (As shown in Tables 12). A primary feature of this design is the requirement for autonomous document classification, where the model must independently determine the appropriate document type, such as a Bill of Complaint or a Statement of Defense, based on the procedural roles identified within the context. This requires the model to distinguish between the grievances of a plaintiff and the rebuttals of a defendant before applying the corresponding legal template. Furthermore, the prompt mandates a critical analysis of user input to identify and correct factual inconsistencies or potential traps resulting from low legal literacy. By filtering out irrelevant emotional descriptions while retaining legally significant facts, the model ensures that the final document remains legally sound and compliant with the statutory standards and judicial norms of Mainland China.

12.2 Judge Model Scoring Prompts↩︎

12.2.1 Scoring Prompt 1: Public Legal Consultation↩︎

To provide an objective and scalable assessment of model performance, we developed a standardized scoring protocol for Task 1 that transitions the evaluator into a neutral judge (As shown is 13). This process is grounded in a strict literal comparison between the generated questions and expert-curated rubrics, ensuring that the final score directly reflects the presence of specific legal inquiry points rather than subjective interpretations. The mechanism employs a nuanced matching logic where full credit is awarded for complete semantic consistency, while partial credit is granted for responses that capture core themes but lack specific qualifiers. Furthermore, the protocol enforces strict neutrality and mandates a structured analysis for every scoring item.

12.2.2 Scoring Prompt 2: Practical Case Analysis↩︎

The evaluation protocol for Task 2 employs a rigorous item-by-item inspection mechanism to assess the quality of complex legal case analyses (As shown is 14). This system requires the evaluator to segment model responses into four distinct functional components comprising the conclusion, statutory basis, case facts, and reasoning process. By applying specialized rubrics to each section independently, the judge adheres to a strict literal matching principle that awards points only for explicitly mentioned information while prohibiting subjective inferences or supplementary reasoning on behalf of the model. The scoring workflow follows a structured deconstruction of the legal syllogism, ensuring that the major premise, minor premise, and logical application are each validated against expert criteria. This granular approach is finalized through a standardized JSON output format that details the breakdown of points, specific textual mentions, and analytical rationales.

12.2.3 Scoring Prompt 3: Legal Document Generation↩︎

The scoring protocol for Task 3 evaluates the proficiency of Large Language Models in generating formal legal instruments that adhere to professional drafting standards (As shown is 15). This evaluation process assesses the model’s ability to autonomously select the correct document type, such as a Bill of Complaint or Statement of Defense, based on the procedural context of the user narrative. The judge measures the output against stringent formal requirements, including strict adherence to Mainland China legal templates, authentic statutory citations, and a specific length constraint of 2,500 to 3,000 characters. Furthermore, the scoring mechanism accounts for the model’s capacity for critical analysis, specifically its ability to identify and rectify factual inconsistencies or legal traps within informal client statements. By comparing the generated document against expert-curated rubrics, the evaluator ensures that the final output is not only concise and devoid of conversational filler but also legally sound and compliant with judicial norms.

13 Evaluation Methodology Details↩︎

13.1 Judge Model Validation Results↩︎

To identify the most accurate judge model, we calculated the alignment between scores from human experts and various LLMs to determine which model most closely approximates human judgment. The evaluation dataset consisted of the following:

1. Case Analysis: A random 10% sample of all items, paired with responses from a randomly selected model.

2. Document Drafting: Ten cases each for Bill of Complaint and Statement of Defense, paired with responses from a randomly selected model.

3. Legal Consultation: Responses to 20 test items generated by four specific models (Claude 3.5 Sonnet, GPT-5-0807-Global, Qwen-Max, and Gemini 3 Pro Preview).

These responses were distributed to both human experts and LLMs for scoring. To ensure reliability, each response was evaluated by at least three human experts. We compared individual expert scores against the group average; the Pearson correlation coefficients between the three experts’ scores and the human average ranged from \(0.8391\) to \(0.8841\), with a Mean Absolute Error (MAE) between \(5.1937\) and \(7.9817\). Table 3 presents the alignment results between candidate Judge models and human experts.

Following current research trends that employ high-performance models as evaluators, we tested Gemini 3 Pro Preview, GPT-5.1-1113-Global, and Qwen3-Max as potential judge models. Based on their comprehensive performance across different tasks, we selected Gemini 3 Pro Preview as our final judge model. Although it did not achieve the highest alignment in the Document Drafting task, it demonstrated superior performance in the remaining two categories.

13.2 Scoring Mechanics↩︎

To prevent scoring discrepancies, we utilize a scoring rate (Earned Points / Total Points). We established eight comprehensive metrics. Since LLMs are prone to calculation errors, we employ regular expressions to extract raw scores from the JSON or list outputs generated by the Judge model (as defined in the scoring prompts) and perform the final summation programmatically.

14 Detailed Error Analysis↩︎

14.1 Error Analysis of Public Legal Consultation↩︎

For Task 1, we analyzed 183 low-scoring responses and identified four common categories of errors along with eight specific issues. The common problems observed in current LLMs during public legal consultation include:

• Insufficient grasp of facts and causal relationships. Models often struggle to accurately link reported events to their legal consequences.

• Shallow understanding of evidence. Models frequently fail to build a closed logical loop of questioning around evidence, missing the necessity of corroborating key facts.

• Weak identification of specific evidence types. There is a general lack of sensitivity to the specific forms of evidence required for different legal claims.

• Defects in role adherence. Some models exhibit limitations in their ability to discern and analyze the overall case context due to persona setting issues.

Beyond these commonalities, distinct model-specific weaknesses were observed. For instance, GPT-5.2 tends to deviate from key facts during follow-up questioning, resulting in divergent lines of inquiry. Gemini-3.0 struggles with cases involving complex evidence, often failing to understand probative requirements and asking questions without sufficient legal basis.

Overall, the performance of current LLMs in simulating real-world legal consultation remains insufficient. Whether in fact organization, causal inference, or evidence analysis, models generally exhibit shallow understanding and fragmented reasoning. This is particularly evident in complex cases with multiple causes or weak evidence chains, where misjudgments and omissions of key points are frequent.

14.2 Error Analysis of Practical Case Analysis↩︎

In Task 2, we examined 200 low-scoring cases and summarized the errors into four dimensions covering 16 specific issues. The common issues across models are:

• Conclusion: Fundamental errors in legal characterization.

• Case Facts: Omission of key facts that affect qualitative analysis, as well as loss of important details such as time, amount, and subject identity.

• Reasoning Process: Failure to follow the required legal reasoning sequence (e.g., syllogism).

• Statutory Basis: Absence of core legal provisions.

Model-specific analysis reveals that GPT-5.2 has significant deficiencies in legal retrieval and application, frequently missing important legal bases or citing incorrect versions. Gemini-3.0 shows defects in the integrity of reasoning chains and logical rigor, alongside serious issues with missing core statutes.

In summary, current LLMs have not yet reached a professional level in practical case analysis. While they can generate structured text, they exhibit obvious shortcomings in verifiable analysis: often omitting key facts, reasoning with improper order or broken chains, and misapplying laws (e.g., mixing new and old laws, or failing to distinguish between general and special provisions).

14.3 Error Analysis of Legal Document Generation↩︎

For Task 3, an analysis of 183 low-scoring responses revealed two common issues:

• Inaccurate Legal Application: Insufficient grasp of the timeliness, relevance, and citation principles of laws.

• Weak Jurisdiction Review: Insufficient sensitivity and capability in reviewing jurisdictional rules.

These problems are prevalent across all models. Both international state-of-the-art models (e.g., GPT-5.2, Claude-4.5) and other strong models (e.g., DeepSeek, Qwen) struggle to perfectly resolve issues related to legal "hallucinations" and procedural logic.

Regarding individual differences, GPT-5.2, Claude-4.5, and Gemini-3.0 make fewer errors in basic legal logic such as "proper party" determination and "fact summarization," indicating stronger instruction following and stability. Conversely, models like DeepSeek, Kimi, and Qwen3-Max are more prone to hallucinations and blindly following client instructions even when they are legally unsound.

Overall, while current models are mature in text organization, they lack substantive quality. They often fail to judge the suitability of subjects (blindly following wrong instructions), lack sensitivity to the statutory scope of claims (accepting unreasonable claims or omitting lawful ones), and violate the rigor required for legal documents.

References↩︎

---

1. These authors contributed equally to this work.↩︎

2. Corresponding authors.↩︎

3. Adapted from Maughan and Webb [26]: Consultation corresponds to "Interviewing" (Ch.5), Case Analysis to "Case Management" (Ch.10), and Document Generation to "Drafting" (Ch.7-8).↩︎

4. https://www.12348.gov.cn/#/homepage↩︎

5. https://wenshu.court.gov.cn/↩︎