1 Introduction↩︎

Advanced Air Mobility (AAM) operations are projected to grow significantly in the near future, with unmanned aircraft systems (UAS) and eVTOL vehicles performing large volumes of flights for both cargo delivery and passenger transportation. Maintaining safe and efficient flight planning within such dense and complex airspace presents a major challenge. Flight restrictions, coupled with the often subjective and context-dependent nature of mission-specific requirements, make it difficult to define rigid, mathematical objectives that fully capture operational intent.

Classical path planning algorithms, such as A\(^*\) and rapidly-exploring random trees (RRT, RRT\(^*\)) [1], [2], can compute optimal, obstacle-free trajectories; however, they depend on explicitly defined mathematical objectives and constraints, which limits their ability to capture nuanced human pilot preferences, such as trading off flight duration against waypoint complexity. As the scale and complexity of eVTOL operations continue to grow, there is an increasing need for automated planning systems capable of interpreting such nuanced mission intent.

One promising technology to bridge this gap is to use LLMs for planning. For instance, in robotics and navigation, the SayCan framework [3] demonstrated the capability of LLMs to guide robots using natural language to do feasible and contextually appropriate actions. In planning, hybrid approaches have leveraged LLMs to generate subgoals for classical planners [4]–[6], while other methods integrate solver heuristics to better guide LLM generated plans [7], [8]. Researchers have also explored combining LLMs with reinforcement learning (RL) to enhance reasoning capabilities. For instance, S2RCQL [9] augments prompts using information derived through Q-learning, and another approach employs LLMs to generate semantic hints for contextual RL in motion planning [10]. Closer to our setting, recent work has brought LLMs directly into unmanned aerial vehicle (UAV) control: TypeFly [11] translates natural language into executable drone missions, and GSCE [12] introduces a structured prompt framework that improves the reliability of LLM-issued UAV commands.

Moreover, LLMs have also shown strong potential as proxies for human preference, which can be utilized to readily evaluate preference alignment [13], and recent work on introspective planning [14] shows that aligning an LLM’s uncertainty with task ambiguity yields safer downstream plans—motivating the coach/verifier agent we adopt in this work. A second complementary direction is retrieval-augmented planning, where an embodied agent conditions on trajectories or experiences retrieved from a memory store: P-RAG [15] introduces progressive retrieval for everyday planning tasks, and STRAP [16] retrieves sub-trajectories for robot policy learning. FRAMe adopts the same intuition for eVTOL route planning, retrieving preference-conditioned past flight plans rather than learning a policy from scratch.

Although LLMs have been explored extensively across these domains, their potential for real-world flight planning remains largely unexplored. To address this gap, prior work introduced an approach in which an LLM selected the route most aligned with human preference from multiple candidates [17]. That line of research was subsequently extended to an end-to-end flight planning generation system for eVTOL operations, employing various different chain-of-thought (CoT) prompting strategies [18], [19].

In this paper, we continue this progression by presenting an end-to-end LLM-based flight planning tool that incorporates a multi-modal coach agent and a memory system based on RAG. Specifically, we introduce FRAMe, a flight planning tool that enables human operators to generate eVTOL flight plans based on natural language preferences.

We propose a flight planning system that integrates a planner LLM with a multimodal coach agent and a RAG memory module. The LLM serves as the route planner, transforming textual prompts into sequences of geographic waypoints. The RAG module grounds the LLM’s reasoning by supplying relevant context from prior planning experiences and their evaluations. The coach agent then validates and assesses each generated plan through a three-stage review: first, geometric tools verify that the flight plan is physically valid; second, the agent evaluates alignment with operator preferences by inspecting the rendered flight plan image; and third, the operator can optionally provide additional feedback. The resulting record is stored in the memory database, progressively improving future RAG retrievals. Our key contributions are as follows:

1. FRAMe System: We design a flight planning framework that integrates a planner LLM with a RAG-based memory module and a multi-modal coach agent, enabling natural language-driven route generation that respects no-fly zones and aligns with operator preferences.

2. Annotator-Free Preference Evaluation: We propose a quantitative evaluation framework built on three measurable preference objectives: minimizing flight distance, minimizing waypoint count, and maximizing polygon clearance. Because each objective is computed geometrically, preference alignment can be assessed objectively without human annotators. We pair this with a two-phase protocol, a warmup phase that seeds the memory followed by a read-only ablation phase, which isolates the contribution of retrieval augmentation.

3. Multi-Model Ablation: We conduct a systematic evaluation across four LLMs (OpenAI’s o3-mini, o4-mini, GPT-5.4 [20], and DeepSeek-R1 [21]) and three scenario difficulty levels, benchmarking against a classical A* baseline. The ablation isolates each component: the coach provides a non-redundant validity lift on top of retrieval for every planner—most visibly on o3-mini, where RAG alone does not improve over Baseline.

The remainder of this paper is as follows. Section 2 describes the FRAMe system architecture, including the planner agent, multi-modal coach agent, and RAG-based memory module. Section 3 presents the web prototype and experimental evaluation, covering plan validity, preference-metric alignment, and comparison with an A* baseline across four LLMs. Section 4 discusses limitations and Section 5 concludes. The appendix provides prompting strategies (Appendix 6), the full coach vision agent prompt (Appendix 7), qualitative flight-plan examples (Appendix 8), a per-preference metric breakdown (Appendix 10), and a per-difficulty validity analysis (Appendix 9).

2 System Architecture↩︎

FRAMe’s architecture combines an LLM-based planning core with auxiliary modules for solution evaluation and knowledge retrieval. The system takes two main inputs: a flight scenario comprised of an origin, a destination, the flyable airspace, and a set of no‑fly zones encoded as polygons in a KML file, and a natural language prompt from the flight operator describing how the route should traverse the environment (for example, “Maximize clearance from hazardous polygons.”). KML is a file format that can visualize geographic data in earth browsers such as Google Earth [22]. The flowchart of the system architecture is depicted in Figure 1. We explain FRAMe components here.

2.1 Planner Agent↩︎

The planning process begins by extracting the origin, destination, flyable airspace, and no‑fly‑zone placemarks from the operator’s KML file and converting them into natural‑language messages. The system messages are generated according to the chosen prompting strategy, priming the LLM with general instructions. Tabrizian et al. [19] discussed different prompting strategies based on CoT and showed that the best results are achieved by using a customized version of CoT without providing any examples (Zero-shot). All of the prompting strategies available in FRAMe are described in the Description of Different Prompting Strategies section in Appendix. After system messages are constructed, they will be combined with user messages (natural-language information of the KML) and the flight operator preferences. This will be the input for the LLM-planner. If RAG is enabled, these messages will be augmented with prior successful flight plans based on the flight setting and operator preferences. With the fully assembled prompt, the LLM-planner produces a candidate flight plan which is a sequence of waypoints forming a flight path that begins at the origin, ends at the destination, avoids all no‑fly zones, and adheres as closely as possible to the operator’s preference. The planner will also generate a reasoning in natural language for better explainability purposes.

2.2 Multi-modal Coach Agent↩︎

The multimodal coach agent is implemented using o4-mini alongside rule-based geometric checks and flight operator’s feedback. The agent first verifies plan validity. A plan is considered valid if: (1) All generated waypoints remain within the flyable airspace, (2) the plan correctly connects the designated origin and destination based on the scenario data, and (3) no segment passes through a no-fly zone.

All criteria are evaluated using geometric tools. Following the validity check, the system renders an image of the full planning problem with the proposed solution overlaid (see Figure 2) and assesses preference alignment, confirming whether the generated route satisfies the operator’s directional or positional instructions using o4-mini’s vision capabilities (for complete coach vision agent prompt see Appendix 7). The vision agent has access to both the rendered solution image and the validity evaluation results. Finally, the flight operator can optionally provide free-form feedback on the proposed plan.

2.3 Retrieval Augmented Generation↩︎

To augment the prompt with proper context about the current flight planning problem, we implement the RAG module. When enabled, the system queries a vector database to retrieve past flight plans that most closely resemble the current setting and the operator’s stated preferences.

The process begins by passing the operator’s preferences into an embedding model, which converts the textual input into a numerical embedding vector. This vector is then used by the history retrieval module to identify the most similar prior flight plans based on the cosine distance between the current preference embedding \(\mathbf{e_p}\) and each other preference embedding in the database \(\forall \mathbf{e_h} \in \mathcal{H}\): \[\text{CosineDistance}(\mathbf{e_p}, \mathbf{e_h}) = 1 - \frac{\mathbf{e_p} \cdot \mathbf{e_h}}{\|\mathbf{e_p}\| \|\mathbf{e_h}\|},\] where \(\mathcal{H}\) denotes the set of preference embedding vectors corresponding to previously successful flight plans. Since lower cosine distances indicate greater similarity, the system selects records with the \(K\) lowest values. Note that the retrieval query restricts the candidate set to prior records with identical scenario geometry, that is, the same flyzone, no-fly polygons, origin, and destination. The cosine similarity over the operator’s natural-language preference text is then used only to rank within that geometrically identical set. Geometric relevance is thus guaranteed by exact scenario matching rather than inferred from the embedding distance, and the preference embedding only selects which past plan for the same problem best matches the current intent. The retrieved plans, referred to as the contextual history, represent flight plans most aligned with the current setting and preferences.

The contextual history is then used to augment the input prompt, providing relevant context and guidance for generating the new flight plan. The planner LLM processes this enriched prompt to produce a flight plan, which is presented to the operator for evaluation. If the plan is submitted for storage, the RAG module then collects the relevant data into the vector database, making it available for future retrieval. The complete planning workflow is detailed in Algorithm 3.

3 Web Prototype and User Interface↩︎

Conducting real flight tests with full-scale eVTOL aircraft is prohibitively expensive at this stage of research. Although FRAMe is designed and evaluated in the context of eVTOL mission planning, we deploy it as a web application targeting subscale UAV operations as a practical intermediate step. The web interface connects with small UAVs and ground control station software such as Mission Planner [23], a widely used platform for configuring, controlling, and monitoring UAVs. This is intended to allow validation of the planning pipeline in real flight conditions at a fraction of the cost, while the underlying planning logic and preference-alignment framework remain directly applicable to eVTOL operations. The backend is implemented in Python, integrating the LLM via an API and our custom modules for retrieval and validation. The front-end is a user-friendly interface that guides the operator through scenario setup, preference input, and reviewing the flight plan.

3.1 Experimental Setup↩︎

We evaluate FRAMe on flight scenarios in the Dallas–Fort Worth metropolitan area grouped into three difficulty levels—Easy, Medium, and Hard—that differ in the number and geometry of no-fly zones (2, 4, and 7 polygonal restrictions, respectively). Each difficulty level is paired with every combination of a fixed set of origins and destinations, producing a broad coverage of flight configurations.

Models. We benchmark four state-of-the-art LLMs as the planning module: OpenAI o3-mini, OpenAI o4-mini, DeepSeek-R1, and OpenAI GPT-5.4.

Conditions. Each scenario is evaluated under four conditions that reflect progressively richer augmentation. We use the same short names (A*, Baseline, +RAG, +RAG+Coach) throughout the text, tables, and figures:

1. A*: a classical A* path planner serving as a geometry-optimal baseline.

2. Baseline: the LLM planner alone, with no retrieval augmentation or coach feedback.

3. +RAG: the LLM planner augmented with the RAG module, retrieving the \(K{=}2\) most relevant prior plans from the database (no coach).

4. +RAG+Coach: the full FRAMe system, combining RAG-based retrieval with the multi-modal coach agent for validity checking and preference verification.

Preferences. To enable objective, quantitative evaluation of preference alignment, we use three measurable preference objectives: (1) minimize total flight distance, (2) minimize the number of waypoints, and (3) maximize clearance from hazardous polygons. For each preference, alignment is assessed by whether the model’s output improves the corresponding metric relative to the other preference conditions within the same scenario.

Warmup and ablation protocol. Before ablation, a warmup phase runs each planner over every scenario with a neutral preference prompt, both with and without the coach agent, retaining only the plans that pass the geometric validity checks, to seed the RAG vector database (150 runs). The ablation phase then fixes the database in read-only mode and sweeps each model across the full scenario–preference–condition grid, isolating the marginal contribution of RAG and of the coach.

3.2 Results↩︎

Plan validity. Figure 5 reports the validity rate for all four models averaged over difficulty levels, and Table 1 provides the corresponding preference-metric breakdown for valid plans. For every model, +RAG+Coach achieves the highest validity. The three reasoning models converge to 88.9–93.8% under the full system (o3-mini: 79.6%\(\rightarrow\)​79.1%\(\rightarrow\)​88.9%, o4-mini: 84.4%\(\rightarrow\)​88.4%\(\rightarrow\)​90.7%, DeepSeek-R1: 83.6%\(\rightarrow\)​87.1%\(\rightarrow\)​93.8%), while GPT-5.4 lags considerably, improving from a markedly lower Baseline of 37.3% to 55.6%—a gap that persists across all conditions and reflects weaker instruction-following on geometrically constrained scenarios. For o3-mini in particular, +RAG alone slightly regresses validity (79.6%\(\rightarrow\)​79.1%)—retrieval without validation surfaces misleading neighbors—and the coach is what recovers and extends the gain, indicating that the coach review is load-bearing rather than a redundant signal on top of retrieval.

2pt

Preference capture. Figure 6 traces how the preference-relevant metric evolves across conditions for valid plans, with reasoning models showing distinct preference-specific patterns. Under the waypoints preference, o3-mini is the strongest reasoning-model responder, reducing mean waypoint count from 4.06 (Baseline) to 3.58 (+RAG+Coach, \(-12\%\)); o4-mini and DeepSeek-R1 start already near their planning floor (3.49 and 3.32, respectively), leaving little room for further reduction. Under the clearance preference, o4-mini shows the clearest gain, increasing minimum clearance from 5.58 km to 6.07 km (\(+9\%\)); DeepSeek-R1 peaks at +RAG (7.50 km, up from 6.72 km at Baseline) but retreats slightly with the coach (6.84 km), suggesting that retrieval alone is the stronger signal for its spatial reasoning. Notably, o3-mini’s clearance declines monotonically with augmentation (5.88\(\rightarrow\)​5.35\(\rightarrow\)​4.52 km in Figure 6; the per-preference matrix in Appendix 10 reports the same downward trend with rounding to one decimal)—coach-driven revisions appear to prioritize correcting validity violations over maximizing polygon separation. The distance preference moves little for all three reasoning models (193–197 km), reflecting that they already operate close to their planning floor at Baseline. GPT-5.4, by contrast, is the only model that monotonically improves distance when the distance preference is active (190.5\(\rightarrow\)​190.0 km in Figure 6); we note this is a per-preference effect—in the all-preferences aggregate of Table 1 GPT-5.4’s mean distance rises slightly with augmentation (190.5\(\rightarrow\)​191.7\(\rightarrow\)​191.2 km), so the gain is conditional on the operator actually requesting distance. GPT-5.4 also shows comparable waypoint reduction (\(-13\%\)), though its lower overall validity limits the practical impact of these gains.

A* baseline comparison. The classical A* planner achieves 100% validity by construction, but it is preference-blind: its mean clearance of 1.91 km is well below what every LLM produces even in the Baseline condition (3.9–5.0 km), and its waypoint count (18.2) is \(3\)–\(5\times\) higher than every LLM configuration because A* tracks the grid discretization rather than the operator’s intent. FRAMe (even without RAG) already outperforms A* on the clearance and waypoint preferences, while accepting a small distance overhead over the preference-blind geodesic. Adding +RAG+Coach widens the clearance and waypoint margins and recovers a substantial part of the validity gap for the stronger planners, most clearly for DeepSeek-R1.

4 Limitations↩︎

Headroom-bounded preference capture. The distance preference does not move meaningfully under any condition (Table 1) because every model already plans within 6 km of the \(\approx\)​190 km geodesic at Baseline. The same headroom caveat partially applies to waypoint count for o4-mini and DeepSeek-R1, which sit near a 3.3–3.5 floor at Baseline and therefore have limited room to improve further. Our preference-capture evidence is therefore strongest for the clearance preference (and for waypoint count on planners that start above the floor, e.g.o3-mini and GPT-5.4) and weakest for distance—a consequence of the evaluation geometry rather than evidence that the framework ignores the preference.

Aggregate vs.per-preference effects. Several of our positive findings are clearest when conditioned on the matching preference (Figure 6). When metrics are averaged across all three preferences (Table 1), augmentation can move a non-target metric in the unfavorable direction—for example, aggregate clearance for o3-mini and GPT-5.4 declines under +RAG+Coach as those configurations correctly de-prioritize clearance when the operator did not request it. The aggregate column should therefore be read as a side-effect summary, not as the primary preference-capture signal.

LLM-as-judge bias in the coach. The preference-alignment step of the coach is performed by a multimodal LLM (o4-mini) evaluating the planner’s output, and prior work has shown that LLM judges can exhibit position, verbosity, and self-preference biases [13]. We mitigate this by combining the vision judgment with rule-based geometric checks for validity, but the alignment verdict itself inherits the hidden biases or priors the judge carries. A human-rater study on a subset of plans is left for future work.

Simulation-only evaluation. All experiments use synthetic polygonal no-fly zones; no live flight tests or real weather data are incorporated. As a next step, we plan to validate the approach through real flight tests with a subscale drone.

Constraint coverage. Our current formulation treats hazards as static no-fly polygons and does not model dynamic traffic from other UAVs or crewed aircraft, onboard energy or battery limits, or vehicle dynamics. These factors are safety critical for feasibility and collision avoidance in dense airspace, and incorporating them, for example by adding traffic and energy terms to the scenario and the validity checks, is important future work.

5 Conclusion↩︎

We presented FRAMe, an end-to-end flight planner that couples an LLM with a RAG memory of prior plans and a multi-modal coach agent that gates both validity and preference alignment. The framework uses geometric ground truth, so it needs no human annotators, and it is modular: retrieval and the coach can be ablated independently against a fixed warmup-seeded database. Across four state-of-the-art LLMs and three operator preferences, the coach review is load-bearing on top of retrieval, attaining the highest validity for every planner; for the weakest planner, retrieval alone can slightly regress validity, and the coach is what recovers it. Preference capture is real but model-specific: planners with headroom shift the requested metric in the operator-favored direction, while those already near their planning floor have little room to move, and gains can surface under retrieval alone or only once the coach is added. Finally, FRAMe is preference-aligned where A* is preference-blind, accepting a small distance overhead in exchange for higher clearance and substantially fewer waypoints.

6 Description of Different Prompting Strategies↩︎

Here we describe all of the prompting strategies available for FRAMe:

6.0.0.1 Raw Prompt:

This baseline uses minimal prompt engineering. The scenario and request are provided directly to the LLM without any additional guidance on the solution approach. Specifically, the system message is:

There is no task breakdown or few-shot example. This represents how the model might perform “out of the box” given a straightforward request.

6.0.0.2 Zero-Shot Prompt (Basic):

In the zero-shot setting, we append the phrase “Think step by step.” to the system messages, following [18], to prompt a strategic reasoning process. However, we do not specify the individual steps the LLM should follow for planning, nor do we supply any examples in this approach.

6.0.0.3 Zero-Shot Prompt (Customized):

This approach is a modified form of zero-shot prompting that provides more explicit guidance for reasoning. The system message is extended with instructions tailored for chain-of-thought reasoning in flight planning tasks. The following steps are incorporated into the system messages:

By clarifying the criteria and encouraging intermediate reasoning, this strategy is expected to help the LLM perform better. However, no example flight plan is given, so this remains a zero-shot approach (the model must draw on its internal knowledge and the on-the-fly reasoning).

6.0.0.4 One-Shot Prompt (Easy Example):

This method resembles the zero-shot (customized) approach, but additionally supplies the LLM with a single example problem–solution pair before presenting the new scenario. The example is a simple case (structurally similar but with different coordinates) that illustrates the format of a correct solution.

6.0.0.5 One-Shot Prompt (Hard Example):

This approach is analogous to the previous one, except that the provided example depicts a challenging scenario with multiple hazards, illustrating a more elaborate chain-of-thought process.

7 Coach Vision Agent Prompt↩︎

The multi-modal coach agent receives two inputs simultaneously: (i) a structured text prompt, shown below, in which {geometric_summary} is filled at runtime with the output of the rule-based geometric checks (validity flag, violated polygons, waypoints outside the flyzone, and origin/destination match), and {human_preference} with the operator’s stated preference; and (ii) the rendered flight plan image depicted in Figure 2. The agent first handles invalid plans (Step 0), then—only if the plan is geometrically valid—judges preference alignment from the image (Step 1), returning a structured verdict (aligned, evaluation, reasoning).

8 Extended Qualitative Flight-Plan Examples↩︎

The examples below show the real inputs and outputs of two valid and one invalid FRAMe runs with the o4-mini planner. Within each FRAMe Input box, three sections are color-coded: SYS (the fixed system message, shared across all runs), USER (the per-scenario user message built by prompt_generator), and OPERATOR (the operator preference, appended to the user message). The FRAMe Response box shows the waypoints and natural-language explanation returned by the planner. Finally, coach review is shown for the generated flight plan. Polygon labels such as poly7-1 and poly1-3 use the scenario identifier as a prefix; the leading number is a scenario ID, not the polygon count.

8.1 Easy scenario, clearance preference↩︎

8.2 Medium scenario, minimize distance preference↩︎

8.3 Hard scenario, minimize waypoints preference↩︎

9 Validity by Difficulty Level↩︎

Figure 10 breaks down the full system (+RAG+Coach) by difficulty level and reveals that the relationship between geometric complexity and validity is model-specific rather than uniform. DeepSeek-R1 achieves the highest validity on Easy and Medium scenarios (98.7% and 97.3%, respectively) but drops to 85.3% on Hard, suggesting that its strong internal reasoning becomes less reliable when navigating seven overlapping no-fly zones simultaneously. o4-mini exhibits the opposite pattern: its Hard validity (96.0%) exceeds its Easy (89.3%) and Medium (86.7%) rates, indicating that richer geometric structure helps anchor this model’s planning decisions rather than overwhelm them. o3-mini shows a monotone decline from Easy (93.3%) through Medium (92.0%) to Hard (81.3%), consistent with a planner that handles moderate complexity well but degrades under the densest airspace configurations. GPT-5.4 remains consistently the weakest planner across all difficulty levels (52–60%), with no clear sensitivity to difficulty; the coach partially compensates for its lower intrinsic planning capability but cannot fully bridge the gap to the other three models.

10 Per-Preference Capture Matrix↩︎

Figures 11 and 12 show the full per-preference breakdown of all three metrics (distance, waypoints, clearance) across the three conditions (Baseline, +RAG, +RAG+Coach) for each model. Each cell reports the mean value over valid plans only. Reading across a row shows how a given metric changes as augmentation increases; reading down a column shows how different preferences affect the same metric under the same condition. The diagonal entries (where the preference matches the metric, e.g.distance metric under the distance preference) are the primary preference-capture signal: a diagonal cell that moves in the desired direction (lower distance or waypoints, higher clearance) relative to its Baseline indicates that the corresponding condition captures the active preference for that model. Off-diagonal entries are informative as side-effect probes—e.g.asking for clearance can lengthen distance, or asking for fewer waypoints can reduce clearance—and should be read as costs incurred to satisfy a different active preference rather than as failures of preference capture.

References↩︎