1 Introduction↩︎

Planning has become a central challenge for evaluating and improving the ability of large language models (LLMs) to solve complex real-world problems. This capability reflects a foundational aspect of general intelligence because it requires models to reason about future consequences, allocate limited resources, and pursue long-term goals under constraints. Real-world planning tasks therefore provide a natural setting for both evaluation and training, as they require complete solutions that are executable and verifiable under coupled constraints [1]–[3]. These requirements distinguish planning from open-ended generation and localized reasoning benchmarks, where individual outputs or intermediate steps can often be evaluated in isolation [4], [5]. In planning, a locally reasonable decision may invalidate the global plan, and a feasible plan may still be far from optimal [6]. As frontier LLMs continue to improve, scalable sources of diverse and verifiable planning data are needed not only to faithfully evaluate their planning abilities, but also to further train models toward more generalizable planning behavior [7].

A growing body of work has advanced LLM planning evaluation through realistic and verifiable benchmarks [7]–[13]. Recent studies also show that planning data can support model development beyond static evaluation [14]–[17]. These advances highlight the value of planning data, but also reveal limitations of benchmark-driven design. Existing efforts often define fixed collections of planning tasks rather than scalable mechanisms for generating data with high diversity, controlled difficulty, and automatic verification. This limits coverage and diagnostic value, since many benchmarks focus on a small number of domains and test only a limited range of planning structures [11], [13], [18]. Difficulty is also often approximated by surface-level proxies such as task length or number of requirements, which may not capture structural sources of planning difficulty such as task-specific design, hierarchy, and constraint coupling. Consequently, existing benchmarks may not provide a fine-grained view of frontier LLM capability limits or reliable verification signals for planning-oriented training [12], [19].

To address these limitations, we propose PlanningBench as a synthetic planning data generation framework for evaluating and training LLM planning abilities. PlanningBench starts from real planning scenarios abstracted by domain experts. These abstractions organize practical planning workflows into a taxonomy of task types, subtasks, constraint families, and difficulty factors. The taxonomy covers more than 30 planning tasks, each with multiple subtasks and a rich pool of task-specific constraints that can be systematically composed into diverse problem instances. In this way, PlanningBench preserves realistic task grounding while shifting planning data construction from collecting fixed benchmark instances to controllable data generation.

PlanningBench implements this controllable generation process through a constraint-driven synthesis pipeline. The pipeline samples structured task-constraint configurations and instantiates each configuration as a self-contained planning problem. It then applies controlled difficulty enhancement and post-hoc quality filtering. These steps enable controlled diversity across domains, subtasks, constraints, and solution structures. They also support task-specific and hierarchical difficulty control by varying constraint tightness, resource scarcity, objective conflicts, subtask dependencies, and global coordination requirements rather than only surface complexity. Each instance is paired with automatic verification rules that check constraint satisfaction and objective quality. Figure 2 provides an overview of the PlanningBench construction pipeline.

With this generated and automatically verifiable planning data, we study LLM planning from both evaluation and training perspectives. We systematically test open-source and closed-source frontier LLMs to examine the current boundary of planning capability. Even the strongest evaluated models achieve limited solve rates on PlanningBench, showing that the generated evaluation suite remains far from saturated. We further analyze performance across task topics, prompt lengths, and the number of checklist items, revealing how scenario structure and constraint complexity affect planning success. For training, we use verified PlanningBench data for GRPO-based reinforcement learning and evaluate transfer beyond the training distribution. The trained models improve on unseen planning benchmarks such as TravelPlanner [12] and ChinaTravel [18], with further gains on broader instruction-following and complex reasoning benchmarks, including Multi-Challenge [20], Inverse IFEval [21], and Collie [22]. These gains suggest that PlanningBench does not merely teach models to imitate fixed task templates. Instead, its verification-driven training signals encourage constraint integration, multi-step reasoning, and global consistency maintenance [23]. Our training analysis further suggests that planning instances with more determinate optimal solutions provide more stable and directional reward signals than instances that admit many locally acceptable answers.

Overall, this work makes three main contributions.

• We introduce PlanningBench, a synthetic planning data generation framework grounded in real planning scenarios. It generates scalable, diverse, and verifiable planning data with broad task coverage, task-specific constraints, and controllable difficulty.

• We show that verified PlanningBench data can support planning-oriented reinforcement learning. Training on PlanningBench improves performance on unseen planning benchmarks and broader instruction-following and reasoning tasks. We further identify reward determinacy as an important factor for stable and transferable planning-oriented training.

• We systematically evaluate open-source and closed-source frontier LLMs on PlanningBench. The results show strong discriminative power across models while the generated evaluation suite remains challenging even for the strongest evaluated models.

5pt

2 Related Work↩︎

Recent work has introduced many benchmarks for evaluating LLM planning. A large body of work focuses on travel planning and its variants, including retrieval-based planning, realistic user requests, infeasible tasks, dynamic changes, reward construction, and multimodal or web-based environments [8], [11]–[13], [18], [24]–[28]. Other benchmarks extend planning evaluation to verifiable constraints, cost-sensitive tool sequencing, progressive constraint revelation, retail and service scenarios, execution-based simulation, and agent reliability [7], [9], [10], [29]–[35]. These benchmarks provide valuable testbeds, but they mostly treat planning data as fixed task collections or simulated interactions. Their difficulty is often controlled through general proxies such as prompt length, number of requirements, interaction depth, or tool-use complexity, which only indirectly capture structural sources of planning difficulty.

PlanningBench treats planning data construction as a controllable generation problem. It abstracts reusable task and constraint structures from real scenarios and synthesizes self-contained instances through task-specific constraint composition and hierarchical difficulty control. This design covers more than 30 task types and pairs each instance with verification checklists, allowing the same data to support complete-solution evaluation and planning-oriented training. In this sense, PlanningBench complements prior benchmarks by providing a scalable source of diverse and verifiable planning data rather than another fixed evaluation set. Table 1 compares representative benchmarks along these dimensions. Domain coverage counts task types or domains. Tool-free planning excludes external tools and environment interaction. Determinate objectives refer to unique, determinate, or well-specified optima.

Alongside benchmark development, recent methods improve LLM planning through task decomposition, multi-agent collaboration, constraint repair, reinforcement learning, human-in-the-loop workflows, and synthesized experience [14]–[17], [36]–[40]. These methods mainly focus on designing stronger planners, agents, or tool-use systems. PlanningBench is complementary because it focuses on generating structured and verifiable planning data for evaluating and training models. Its Generator, Responder, and Critic are closed-loop synthesis components for data construction, not a new agent architecture.

3 Data Construction↩︎

3.1 Definition and Scope of PlanningBench↩︎

PlanningBench targets text-based planning problems in which all information needed to construct and verify a plan is provided in the input. Each generated instance is represented as \((x_i,c_i)\), where \(x_i\) denotes a self-contained planning task and \(c_i\) denotes the verification checklist used for evaluation and training [41], [42]. The checklist specifies the requirements a model-generated plan must satisfy, including relevant context, constraints, output conditions, and task objectives. Under this formulation, PlanningBench asks whether a model can integrate the provided information into a concrete, executable, and verifiable plan. When an instance specifies a determinate optimum, verification further checks whether the plan reaches that optimum.

3.2 Overview of the Construction Pipeline↩︎

PlanningBench constructs scalable and verifiable planning data through real-scenario abstraction, task-constraint taxonomy, and constraint-driven synthesis. It abstracts reusable planning schemas from representative planning scenarios, where each schema specifies the planning target, constraint space, and controllable difficulty factors.

As shown in Figure 2, construction has two macro stages. The first builds task and constraint taxonomies from real scenarios, covering task types, subtask variants, general constraints, task-specific constraints, and specialized stateful constraints. The second synthesizes self-contained instances by sampling task-constraint configurations, attaching verification checklists, and adjusting difficulty through closed-loop feedback.

Candidate instances are further filtered and revised for clarity, consistency, and verifiability. The resulting verified data pool supports both model evaluation and planning-oriented training. This process preserves realistic task grounding while turning planning data construction from fixed collection into controllable generation.

3.3 Task and Constraint Taxonomy↩︎

The first macro stage converts representative planning scenarios into task and constraint taxonomies. Its purpose is not to catalog domains, but to define a reusable design space for planning data generation. To construct this space, twenty professional annotators with planning experience, together with algorithm researchers, review representative scenarios from real applications. They abstract stable problem structures, including planning targets, candidate actions, resources, time windows, dependencies, and required plan conditions. These structures are organized into two layers. The task taxonomy captures what type of plan to construct, while the constraint taxonomy captures feasibility, optimization, and state-dependent requirements that make the plan verifiable. Together, these taxonomies provide the structured elements from which PlanningBench samples and composes diverse planning instances.

3.3.1 Task Taxonomy↩︎

The task taxonomy is organized by planning structure rather than surface application labels. Based on annotated scenarios, we group planning tasks into six families that capture different sources of structural planning complexity:

1. Scheduling and timetabling covers temporal conflicts, time-window matching, and execution ordering.

2. Allocation and matching focuses on resource assignment under compatibility and capacity constraints.

3. Shift and workforce scheduling emphasizes coverage, rotation fairness, and workforce availability.

4. Routing and travel involves route selection, spatiotemporal coordination, and multi-leg transfers.

5. Project and production operations captures milestones, dependencies, capacity limits, and execution continuity.

6. Emergency response and public service focuses on timeliness, priority-based allocation, and plan restructuring.

This organization allows PlanningBench to cover a systematic range of planning structures while keeping the generation space controllable. At a finer granularity, each family contains concrete tasks derived from planning scenarios, and each task is divided into 5–10 subtasks on average to capture variants of the same planning problem type. A detailed task taxonomy with representative subtasks is provided in Appendix 6 Table 1.

3.3.2 Constraint Taxonomy↩︎

Constraints define the feasibility and optimization requirements that make planning instances verifiable and controllable during synthesis. Because task and constraint structures are closely coupled, we model constraints together with task types rather than adding them as post-hoc annotations. This produces a structured task-constraint specification in which each task is associated with reusable constraint elements.

We organize the extracted constraints into three categories:

• General constraints are shared across tasks and capture common planning requirements, such as complete input information, time-window satisfaction, resource boundaries, and infeasibility recognition.

• Task-specific constraints are tied to a particular task structure and define requirements that arise from that task.

• Specialized stateful constraints depend on previously executed actions, accumulated state, or triggered conditions.

At the taxonomy level, these are reusable meta-constraints rather than rules tied to a single problem, allowing PlanningBench to scale generation beyond fixed manually written instances.

To support difficulty control, we organize constraints into three levels for each task. The basic level covers fundamental feasibility requirements, including complete input, resource boundaries, time windows, capacity limits, and executability. The medium level introduces optimization objectives, such as fairness, balance, multi-objective trade-offs, and load balancing. The hard level captures demanding structural requirements, including infeasibility recognition, exception recovery, robustness design, and coordination under conflicting objectives. Specialized stateful constraints are maintained as a separate optional layer and sampled independently when needed.

During data construction, the pipeline samples a task and subtask variant, selects associated constraints, and instantiates the resulting configuration with difficulty control and closed-loop enhancement. In this way, the taxonomy connects real-scenario grounding with scalable synthetic data generation. Detailed constraint descriptions and representative examples are provided in Appendix 7 Tables 2 and 3.

3.4 Constraint-driven Synthesis and Difficulty Enhancement↩︎

Given the task and constraint taxonomies, PlanningBench synthesizes instances by instantiating structured task-constraint configurations as self-contained problems. We implement this process with three closed-loop components, namely a Generator, a Responder, and a Critic, as shown in Figure 2. The Generator creates candidate instances, the Responder attempts solutions, and the Critic verifies them. This loop generates diverse and verifiable planning instances while increasing difficulty when the current Responder fully solves a generated problem.

For a sampled task \(\tau\) and subtask variant \(\sigma\), the Generator draws constraints from task-specific basic, medium, and hard constraint pools, denoted by \(\mathcal{C}_b(\tau)\), \(\mathcal{C}_m(\tau)\), and \(\mathcal{C}_h(\tau)\). To keep each instance manageable while allowing controlled variation in difficulty, the initial counts satisfy \[N_b\in\{1,2,3\}, \quad N_m\in\{0,1,2\}, \quad N_h\in\{0,1\}.\] After sampling the task, subtask, and constraints, the Generator instantiates a self-contained problem \(x\) and its associated verification checklist \(c\). Additional sampling and subset construction details are provided in Appendix 8. Given the generated problem \(x\), the Responder outputs a candidate plan \(\hat{y}\). The Critic evaluates \(\hat{y}\) against the checklist and a prompt-based verifier, producing a verification score \(\rho\in[0,1]\) and a binary all-pass indicator \[u=\mathbb{I}\{\hat{y}\;\text{satisfies all constraints and passes final verification}\}.\] The indicator \(u\) is used as feedback for subsequent synthesis rather than as a final benchmark score. When \(u=1\), the current Responder fully solves the instance, so the pipeline keeps the same task and shifts the constraint composition toward higher difficulty. When \(u=0\), the instance remains challenging under the current configuration.

Let \(\mathbf{p}^{(k)}=(p_b^{(k)},p_m^{(k)},p_h^{(k)})\) denote the sampling probabilities over basic, medium, and hard constraints at iteration \(k\). When a harder variant is requested, the next distribution is updated by \[\mathbf{p}^{(k+1)}= \mathrm{Normalize}( \mathbf{p}^{(k)} \odot \exp(\eta[-\alpha,\beta,\gamma])), \qquad \alpha,\beta,\gamma,\eta>0 .\] This update decreases the weight of basic constraints and increases the weights of medium and hard constraints. The updated probabilities are then projected back to admissible discrete constraint counts before the next generation step, as detailed in Appendix 8. Overall, PlanningBench performs adaptive search over structured planning configurations, turning the task and constraint taxonomies into a controllable mechanism for scalable planning data generation.

3.5 Automatic Verification and Quality Control↩︎

After the generation pipeline, each generated instance includes a verification checklist that converts prompt-level constraints and objectives into instance-level checks. These checks cover input conditions, resource and time constraints, output format, and determinate objectives when available. This design makes PlanningBench instances directly usable for both model evaluation and planning-oriented training, since the same verification structure can provide feedback and reward signals.

After automatic verification, we conduct a human quality-control audit to assess data usability. Twenty professional annotators review synthesized samples and assign each instance to one of four outcomes: direct retention, retention after minor revision, retention after source correction, or discard. Recoverable samples are revised and retained, while irrecoverable samples are removed. In the audited batch, 86.15% of synthesized samples require no revision or only minor revision, and 13.85% require additional source correction before retention. No sample in this batch is directly assigned to the discard category. These results suggest that the synthesis pipeline usually produces recoverable planning instances, while human revision further improves the clarity, consistency, and verifiability of the final data. Details of the audit categories and common revision types are provided in Appendix 9.

3.6 Preference for Determinate Optimal Solutions↩︎

Beyond making each instance verifiable, PlanningBench favors determinate or well-specified optimal solutions whenever the task permits. This design principle is useful for both planning-oriented training and evaluation [43]–[45]. In planning tasks, loose answer spaces can allow many outputs to be partially feasible. If verification rewards such outputs too generously, a model may learn local constraint satisfaction without learning to coordinate time, resources, dependencies, and objectives at the global level. The resulting reward signal can be positive but diffuse. It may indicate how to obtain partial credit, but not how to construct a globally consistent plan. A more concentrated answer space also strengthens evaluation by distinguishing models that merely satisfy local requirements from those that integrate constraints and optimize the full plan.

This preference is informed by early data construction and training practice. In an early batch of about 1k synthesized examples, GRPO training did not yield clear gains and degraded performance on general instruction-following benchmarks. We conjecture that loose optima and permissive verification made the reward less directional. Later batches therefore strengthen the preference for determinate or otherwise well-specified optimal solutions whenever possible. Subsequent evaluations show clearer gains, although more systematic experiments are still needed. We further analyze this effect in Section 4.2.

4 Experiments↩︎

We evaluate PlanningBench from two complementary perspectives. First, we study whether the generated data forms a challenging and discriminative evaluation suite for current LLMs. Second, we study whether the same automatically verifiable data can support planning-oriented reinforcement learning and transfer beyond the training distribution. These two perspectives correspond to the central goal of PlanningBench, which is to generate scalable, diverse, and verifiable planning data for both evaluation and training.

The experiments are organized as follows. Section 4.1 evaluates representative open-source and closed-source models on PlanningBench using All-pass and Avg-pass, and further analyzes performance variation and semantic failure patterns. Section 4.2 studies whether PlanningBench data can support planning-oriented training by evaluating transfer to external planning benchmarks and broader instruction-following benchmarks.

4.1 PlanningBench as an Evaluation Suite↩︎

We first evaluate whether PlanningBench can serve as a challenging and diagnostic evaluation suite for structured planning. The goal is to test whether current LLMs can integrate all provided information into an executable and verifiable plan, rather than only satisfy isolated local requirements. Since planning failures often arise from global inconsistency rather than complete task misunderstanding, we evaluate both complete-solution success and partial checklist satisfaction. The following subsections report model performance, analyze factors that influence difficulty, and examine the semantic error types behind failed outputs.

4.1.1 Evaluation Setup↩︎

During synthetic evaluation data construction, we use strong models as the Generator to inform the human annotation process, through which we obtain the final evaluation set. We then us Qwen-A3B-30B as the Responder and GPT-oss-120b as the Critic model. We evaluate representative open-source and closed-source LLMs on the PlanningBench evaluation set, which contains 467 planning instances. All models follow the same evaluation protocol and are queried with default inference parameters unless otherwise specified. We use GPT-oss-120b as the judge model. For each instance, the judge assesses the model output against its verification checklist.

We report two complementary metrics. All-pass measures whether a plan satisfies all checklist items in a single response, capturing complete-solution success under coupled constraints. Avg-pass measures the fraction of checklist items satisfied by the output, capturing partial progress toward a valid plan. The gap is important because a model may satisfy many local requirements while still failing to produce a globally consistent solution.

4.1.2 Performance of Open-Source and Closed-Source Models↩︎

Table 2 reports the performance of representative LLMs on PlanningBench, with models ordered by All-pass. PlanningBench remains challenging even for the strongest evaluated models. GPT-5.4-xhigh achieves the best performance, with 63.17% All-pass and 92.35% Avg-pass, but still fails to satisfy the full verification checklist on more than one third of the evaluation set. Gemini-3-1-pro follows with 53.25% All-pass and 88.36% Avg-pass. Several other models, including Seed-2.0-pro-high, DeepSeek-V3.2-thinking, Hy3-Preview, and Qwen3.5-plus-thinking, achieve relatively high Avg-pass scores but substantially lower All-pass scores. These results indicate that the generated evaluation suite is far from saturated.

5pt

The gap between All-pass and Avg-pass shows that local constraint satisfaction does not necessarily imply complete planning success. For example, GPT-5.4-medium reaches 90.03% Avg-pass but only 58.09% All-pass. Similarly, Seed-2.0-pro-high reaches 84.02% Avg-pass but only 44.33% All-pass. These gaps suggest that models often satisfy many checklist items while still failing to produce a globally valid plan because of a small number of critical errors. PlanningBench therefore distinguishes partial compliance from the ability to integrate locally correct decisions into a complete and consistent plan.

For weaker models, exact solving ability drops sharply. Qwen3-30b-moe achieves only 12.15% All-pass, while Qwen3-32b, Qwen3-14b, and Qwen3-8b are nearly unable to solve full instances under the All-pass metric. Their Avg-pass scores show that some local requirements can still be satisfied, but the strict global-solving metric exposes a larger capability gap. Overall, PlanningBench provides useful discrimination across model capability levels and reveals substantial remaining limitations in structured planning.

4.1.3 Influencing Factor Analysis↩︎

We further analyze model performance by task type, prompt length, and number of checklist items. These factors reflect planning structure, information load, and constraint density. They do not fully determine instance difficulty, but provide useful views of where PlanningBench becomes challenging. Figure 3 shows the results.

Performance varies substantially across task types, and models show distinct task-specific profiles. For example, GPT-5.4-xhigh performs best on Hiring Plan and Material Allocation, reaching 80.0% and 69.2% All-pass. In contrast, Gemini-3-1-pro achieves its highest score on Student Grouping, with 72.7% All-pass. Some task types also expose larger cross-model gaps. On Hiring Plan, performance ranges from 80.0% to 6.7%, while on Power Dispatch Planning, it ranges from 53.9% to 7.1%. These results suggest that PlanningBench captures task-dependent differences in models’ ability to handle complex constraints and maintain planning consistency.

Longer prompts are generally more difficult, although prompt length is not explicitly controlled during construction. GPT-5.4-xhigh declines from 64.0% in the Very Short group to 50.6% in the Very Long group, and Gemini-3-1-pro drops from 61.5% to 43.8%. The trend is not strictly monotonic across intermediate buckets, but the overall pattern suggests that larger information load makes it harder to satisfy all requirements consistently.

A larger number of checklist items also makes All-pass harder to achieve, especially in the highest-count bucket. GPT-5.4-xhigh reaches 63.4% All-pass in the 2–6 item group, but drops to 40.0% in the 15+ item group. Gemini-3-1-pro similarly decreases from 62.1% in the single-item group to 20.0% in the 15+ item group. Other models show similar declines in this setting. Overall, these results suggest that PlanningBench difficulty reflects both information load and the need to jointly satisfy multiple constraints while preserving global coherence.

4.1.4 Error Analysis↩︎

To better understand model failures beyond aggregate pass rates, we analyze All-pass failure cases, excluding exact refusals and blank or no-answer outputs. Each remaining case is assigned to one primary semantic error type. Constraint Omitted refers to missing required constraints or details. Wrong Calculation / Assignment covers numerical, temporal, scheduling, allocation, or logical errors. State Tracking refers to failures to preserve or update an evolving solution. Format / Structure captures violations of the required output schema. Missing Rationale refers to missing explanations, comparisons, or verification steps when required. The distribution is reported in Table ¿tbl:tab:error95analysis?, with a representative failure case shown in Appendix 10 Table 4.

3pt

Wrong Calculation / Assignment dominates across all models, accounting for 60.9%–83.5% of semantic errors. This indicates that the main bottleneck is often not format following, but making correct numerical, temporal, scheduling, allocation, and logical decisions under constraints. Constraint Omitted accounts for 4.2%–19.9% of failures, State Tracking for 5.3%–11.3%, Format / Structure for only 0.7%–3.4%, and Missing Rationale for 4.6%–10.7%.

Model-specific patterns further clarify these failures. Gemini-3-1-pro has the highest share of Constraint Omitted errors at 19.9%, while GPT-5.4-xhigh has the highest State Tracking rate at 11.3%. Qwen3-32b concentrates most failures in Wrong Calculation / Assignment at 83.5%. Overall, PlanningBench exposes weaknesses in constrained reasoning, multi-constraint integration, and stateful plan revision, rather than formatting compliance.

4.2 PlanningBench as Training Data↩︎

Beyond evaluation, PlanningBench is designed to provide automatically verifiable planning data for training. We study whether GRPO-based reinforcement learning on verified PlanningBench data improves performance beyond the training distribution. This section evaluates transfer to external planning benchmarks and broader instruction-following benchmarks, and then analyzes training dynamics under different data construction settings. The goal is not to show that planning data improves all general abilities uniformly, but to test whether verifiable planning data provides useful reward signals for constraint integration, multi-step reasoning, and global consistency.

4.2.1 Training Setup↩︎

We use Qwen-A3B-30B as the base model and conduct GRPO-based reinforcement learning [46] on 300 PlanningBench training instances. We compare four settings. Base Model denotes the original model without reinforcement learning. Syn-PlanningBench uses PlanningBench data constructed with verification-oriented constraints and a preference for determinate or well-specified optimal solutions. Syn-NotDetOptimal uses synthetic planning data that does not emphasize determinate optimality to the same extent. To contextualize the effect of structured synthesis, we additionally include Human-Authored as a comparison baseline. This baseline is independently written from scratch by the same pool of twenty professional annotators, without using the PlanningBench taxonomy, constraint-driven synthesis pipeline, or generated instances. The three reinforcement learning datasets are controlled to be of comparable size.

We evaluate trained models on external planning and general instruction-following benchmarks. The planning benchmarks include ChinaTravel [18] and TravelPlanner [12]. ChinaTravel contains Easy, Medium, and Human subsets, and we refer to the last as ChinaTravel-Human to avoid confusion with Human-Authored. TravelPlanner contains TP-Train, TP-Val, and TP-Test. The general benchmarks include Multi-Challenge [20], Inverse IFEval [21], and Collie [22], which test instruction integration, constraint following, and global consistency. Detailed training hyperparameters are reported in Appendix 11.

4.2.2 Generalization to External Planning Benchmarks↩︎

We first evaluate planning-specific transfer on ChinaTravel [18] and TravelPlanner [12]. Table 3 reports Avg-pass and All-pass on both benchmarks. This comparison tests whether reinforcement learning on PlanningBench data improves planning performance beyond the training distribution.

2.8pt

Syn-PlanningBench consistently improves over Base Model on both external planning benchmarks. On ChinaTravel, Avg-pass increases from 84.88% to 88.12%, and All-pass from 50.92% to 58.36%. Even on the ChinaTravel-Human subset, where the base model performs strongly, Syn-PlanningBench improves All-pass from 86.76% to 93.18%. The transfer effect is stronger on TravelPlanner, where Avg-pass increases from 80.84% to 91.93% and All-pass from 28.85% to 46.86%. All-pass gains are consistent across TP-Train, TP-Val, and TP-Test, with the largest on TP-Val at +22.78 points. These results show that PlanningBench training transfers to external planning benchmarks, especially when tasks require stronger constraint coupling and global consistency.

Compared with Human-Authored, Syn-PlanningBench shows stronger transfer on both benchmarks. On ChinaTravel, Human-Authored brings 1.49 points of average All-pass improvement, compared with 7.44 points from Syn-PlanningBench. On TravelPlanner, Human-Authored reaches 33.86% average All-pass, while Syn-PlanningBench reaches 46.86%. Although Human-Authored provides valid training data, its examples are not constructed through taxonomy-guided constraint composition or controlled difficulty progression. This comparison highlights the value of structured and verification-oriented synthesis for planning-oriented training.

4.2.3 Transfer to General Instruction-following Benchmarks↩︎

We further evaluate whether PlanningBench training transfers beyond planning-specific benchmarks. Table 4 reports results on Multi-Challenge, Inverse IFEval, and Collie. These benchmarks are not planning benchmarks in the same form as ChinaTravel or TravelPlanner, but they require models to integrate instructions, track constraints, and maintain consistency across complex responses.

2.8pt

Overall, Syn-PlanningBench achieves the strongest transfer performance. It obtains the best or tied-best scores on all three benchmarks, improving the average score from 38.74% to 45.80%, with a gain of 7.06 points over Base Model. The improvement is especially pronounced on Collie, where the score increases from 38.33% to 53.17%. These results suggest that verified PlanningBench data can provide training signals that transfer beyond planning-specific evaluation settings.

The contrast with Syn-NotDetOptimal further suggests that broader transfer depends on the quality of the planning signal. Syn-NotDetOptimal brings only a marginal average gain of 0.75 points and slightly decreases performance on Inverse IFEval. This indicates that synthetic planning data without sufficiently clear optimality and verification signals provides less stable transfer. These results do not imply that planning data improves all general capabilities uniformly. Rather, they suggest that verification-driven PlanningBench data can provide useful training signals for tasks where models must coordinate multiple requirements across a complex response.

4.2.4 Training Dynamics and the Role of Determinate Optima↩︎

To examine how different data construction strategies affect reinforcement learning effectiveness, we compare the training dynamics of three models trained with different data sources mentioned above: Human-Authored, Syn-NotDetOptimal, and Syn-PlanningBench, in terms of solve-none ratio, solve-all ratio, and critic reward, as shown in Figure 4. Syn-PlanningBench shows the most favorable dynamics. Its solve-none ratio decreases fastest and converges to the lowest level, while its solve-all ratio increases more consistently. Its critic reward curve is also smoother, suggesting more stable optimization signals. In contrast, Syn-NotDetOptimal shows weaker dynamics and maintains a low solve-all ratio, suggesting that partial checklist satisfaction does not reliably translate into complete solutions. Human-Authored performs between the two synthetic settings.

These trends align with the design preference in Section 3.6. Planning tasks require global consistency across many decisions, so permissive rewards may improve partial checklist satisfaction without reliably producing complete solutions. By favoring determinate or well-specified optima, Syn-PlanningBench provides clearer reward direction for complete planning success. Together with the transfer results, these dynamics suggest that reward determinacy is important for stable and transferable planning-oriented reinforcement learning.

5 Conclusion↩︎

We introduced PlanningBench as a synthetic planning data generation framework for LLM planning. Starting from real planning scenarios, it abstracts reusable task and constraint structures for constraint-driven synthesis, adaptive difficulty control, automatic verification, and quality filtering. This process generates scalable, diverse, and verifiable planning data for both evaluation and planning-oriented training. Experiments show that PlanningBench forms a challenging and discriminative evaluation suite, with even the strongest models leaving a substantial gap under the All-pass metric. Error analysis suggests that failures mainly reflect constrained reasoning and global consistency limitations rather than formatting issues. Beyond evaluation, GRPO-based training on verified PlanningBench data improves transfer to external planning and broader instruction-following benchmarks. The results further suggest that determinate or well-specified optimal solutions provide more stable and directional reward signals. Taken together, these findings show that controllable planning data construction, automatic verifiability, and reward determinacy are important for diagnosing and improving LLM planning abilities.

6 Task Taxonomy Summary↩︎

This section summarizes the task taxonomy underlying PlanningBench. Table 1 lists the main task families, representative subtasks, and the primary abilities each task is intended to test, providing a concise supplement to the task-construction discussion in the main text.

7 Constraint Taxonomy Summary↩︎

This section gives the full appendix-level summary of the constraint taxonomy used in PlanningBench. Table 2 presents the reusable general constraints shared across tasks, while Table 3 lists task-specific constraints for concrete planning scenarios; together they show how difficulty and verifiability are operationalized in the benchmark.

8 Implementation Details for Constraint Sampling↩︎

This appendix provides additional details on constraint-count sampling, subset construction, and projection in the synthesis pipeline. These details are kept in the appendix because the main text focuses on the high-level closed-loop synthesis procedure.

8.0.0.1 Initial constraint-count sampling.

For each sampled task \(\tau\) and subtask variant \(\sigma\), the Generator samples constraints from the task-specific basic, medium, and hard constraint pools. The initial number of sampled constraints from each pool is drawn from a fixed categorical prior: \[N_b \sim \{1\!:\!0.2,2\!:\!0.6,3\!:\!0.2\}, \quad N_m \sim \{0\!:\!0.25,1\!:\!0.55,2\!:\!0.2\}, \quad N_h \sim \{0\!:\!0.7,1\!:\!0.3\}.\] These priors ensure that each instance contains at least one basic constraint, usually includes medium-level requirements, and introduces hard constraints sparsely. This design keeps early generated instances manageable while allowing closed-loop difficulty enhancement to shift later samples toward harder constraint compositions.

8.0.0.2 Subset construction.

After drawing \(N_b\), \(N_m\), and \(N_h\), the Generator samples without replacement \[S_b \subset \mathcal{C}_b(\tau),\qquad S_m \subset \mathcal{C}_m(\tau),\qquad S_h \subset \mathcal{C}_h(\tau),\] with \(|S_b|=N_b\), \(|S_m|=N_m\), and \(|S_h|=N_h\). These sampled subsets are checked against task-template compatibility rules to avoid duplicate, contradictory, or underspecified requirements. Specialized stateful constraints are maintained as an optional layer and are sampled separately when the corresponding task template requires state-dependent behavior.

Together with a random variable \(z\) controlling wording, numerical instantiation, and contextual details, the sampled elements define the generation specification \[r=(\tau,\sigma,S_b,S_m,S_h;z).\] The Generator then produces a problem and checklist pair \((x,c)=G(r)\).

8.0.0.3 Projection after difficulty resampling.

After adaptive difficulty resampling updates the difficulty-level probabilities, the updated distribution is converted back into discrete constraint counts. Given the next-turn total constraint number \(M^{(k+1)}\), we compute the expected allocation \[\mathbb{E}[\mathbf{n}^{(k+1)}]=M^{(k+1)}\mathbf{p}^{(k+1)}.\] This expected allocation is projected onto the admissible count space \[\mathcal{S}=\{1,2,3\}\times\{0,1,2\}\times\{0,1\}\] through \(\Pi_{\mathcal{S}}(\cdot)\), yielding \[(N_b^{(k+1)},N_m^{(k+1)},N_h^{(k+1)}).\] This projection preserves the allowed count ranges for basic, medium, and hard constraints while reflecting the updated difficulty distribution. It also keeps the problem size controllable during adaptive resampling.

9 Human Quality-control Categories↩︎

During human quality control, annotators assign each synthesized sample to one of four categories:

1. No modification needed. The prompt, checklist, and reference answer are clear, consistent, and directly usable.

2. Minor revision with usable source data. The instance is generally valid, but minor edits are needed to improve wording, formatting, checklist coverage, or answer clarity.

3. Minor revision with source data not directly usable. The instance can be retained only after correcting or completing source information, such as missing rules, incomplete boundary conditions, or underspecified constraints.

4. Discard. The instance contains irrecoverable ambiguity, inconsistency, or verification failure and is removed from the final data.

Across the audited batch, 86.15% of synthesized samples fall into categories (1) and (2), while 13.85% require category-(3) revision. No sample in this batch is directly assigned to the discard category. Most revisions involve missing checklist items, incomplete constraint checks, ambiguous or underspecified prompts, unclear references, failures to follow required output structures, missing rules or boundary conditions, and occasional logical inconsistencies between the prompt and answer.

10 Break Case↩︎

This section provides a concrete break case used for qualitative inspection of model failures. Table 4 shows a production-planning example together with the gold solution, a model response, and the corresponding error analysis, illustrating the kinds of global-consistency mistakes that are penalized by PlanningBench.

11 Training Details↩︎

For GRPO-based reinforcement learning, we train with a batch size of 128 for 50 epochs. The actor learning rate is \(2\times10^{-6}\), and the KL coefficient is 0.001. We use 8-way rollout sampling during training. During inference, we use temperature=0.7, top-p=0.6, and top-k=20.

12 Prompt Templates for Task Synthesis and Critic Evaluation↩︎

This section lists the core prompt templates used in our data-construction pipeline. The first template is used to synthesize planning tasks and corresponding checklists which can then be edited manually, while the second is used by the critic model to perform rubric-based evaluation in a consistent format.

12.1 Prompt for Synthesizing the Task and Checklist↩︎

12.2 Prompt for GPT-OSS-120B Used as the Critic↩︎

References↩︎

---

1. https://github.com/Tencent-Hunyuan/PlanningBench. Correspondence to zhaoziliang@ruc.edu.cn↩︎