PVLDB Reference Format:
Dong and Wang. Compilation-Based Execution of Semantic Operators. PVLDB, XX(X): XXX-XXX, 2026.
doi:XX.XX/XXX.XX

¹

1 Introduction↩︎

With the power of Large Language Model (LLM), semantic operators are proposed, which are data-processing operators whose behaviors are specified by natural-language (NL) predicates rather than explicit symbolic conditions. For example, a semantic filter can filter a course table to find “beginner-friendly courses about machine learning”. Commonly used semantic operators include semantic filter (sem_filter), semantic join (sem_join), semantic map (sem_map), semantic aggregation (sem_agg) and so on. With a series of builtin, first-class semantic operators, semantic operator systems [1]–[6] have emerged that enables native semantic query processing for complex data analytics. For example, LOTUS introduces semantic operators as a declarative interface that extends relational processing with AI-based operations over structured and unstructured data [1]. Palimpzest proposes a declarative system for optimizing AI-powered analytics over unstructured data [2]. These systems show that semantic operators are a promising abstraction for LLM-powered data analytics.

However, the mainstream execution strategy for semantic operators is expensive. Many implementations execute semantic operators using what we call interpretation-based execution. In interpretation-based execution, the LLM is invoked at runtime to interpret the semantic operator for each data item (cell or pair of cells). For example, a semantic filter such as sem_filter("beginner-friendly machine learning courses") will be executed by repeatedly sending the predicate together with one row to the LLM and asking whether that row satisfies the predicate, until all rows are checked. Similarly, a semantic map will send each row to the LLM to obtain a transformed value, while a semantic join may send each candidate pair of records to the LLM to determine whether the pair satisfies the semantic relationship indicated by the predicate.

In other words, the LLM sits inside the inner loop of query execution. A filter or map is executed conceptually as follows:

for row in table:
    output = LLM(semantic_condition, row)

A semantic join is even more expensive because the LLM may be invoked for many candidate pairs:

for l, r in candidate_pairs:
    output = LLM(join_condition, l, r)

This execution model is general, but it is also costly. Each LLM invocation incurs latency, monetary cost, and possible rate-limit overhead. For filters and maps, the number of LLM calls grows linearly with the number of rows. For joins, the number of calls grows with the number of candidate pairs, which could be much larger. As a result, interpretation-based semantic execution can become prohibitively slow and expensive on large datasets. Our initial evaluation proves this.

Inspired by the idea of compiling SQL into code to speed up query execution [7], we propose our vision for a complementary execution strategy: natural-language semantic operators should not always be treated as prompts to be repeatedly interpreted by LLMs at runtime. They can also be treated as high-level declarative programs that are compiled into lower-level executable code for accelerating the execution.

Specifically, we propose a new execution paradigm: compilation-based execution for semantic operators. The key idea is to move the LLM out of the runtime loop and into an one-time compilation phase. Given a semantic operator and its predicate, the compilation-based system invokes the LLM once to generate deterministic executable code that “mimics” the intended operator behavior. The generated code is then applied locally over the dataset without additional LLM calls. Figure 1 illustrates this idea with a semantic filter example, sem_filter("beginner-friendly courses related to machine learning"). An interpretation-based system (the upper workflow in the figure) asks the LLM to evaluate this predicate separately for every row/course. In contrast, a compilation-based system (the lower workflow in figure) first asks the LLM to synthesize a deterministic function that approximates the LLM evaluation logic for "beginner-friendly courses related to machine learning":

def compiled_filter(row):
    text = (row["title"] + " " + row["subject"])
    is_ml = any(t in text for t in [
        "Machine Learning",
        "ML", 
        "Deep Learning",
        "DL", 
        "Artificial Intelligence",
        "AI"
    ])
    is_beginner = row["difficulty"] == "beginner"
    return is_ml and is_beginner

In the example, the generated function uses text fuzzy matching to approximate the LLM judging logic for whether the predicate is satisfied. The code in Figure 1 is a shorter version of the code above due to space limit. The generated function is then executed over all rows without LLM calls. In this case, the LLM is used only once to interpret the semantic intent and synthesize code, rather than repeatedly to judge every row, avoiding the LLM call latency and cost per row. Therefore, compilation-based execution naturally takes less running time and less LLM cost than interpretation-based execution, especially on large datasets. The only problem is the approximation will unavoidably cause quality loss. Fortunately, our initial evaluation shows the loss is small. Table 1 summarizes the difference between the two execution models.

This paper explores the feasibility and implications of compilation-based semantic operator execution. We instantiate the idea on three common semantic operators: semantic filter, semantic map, and semantic join. For filters, the LLM compiles a natural-language predicate into a Boolean function. For maps, it compiles a natural-language transformation into a deterministic mapping function. For joins, it compiles a semantic relationship into a pairwise function over two records. In all cases, the generated code is intended to mimic the behavior of the corresponding interpretation-based LLM operator while avoiding repeated LLM calls during execution.

Our current design is intentionally simple. The system performs a direct two-stage process: first compile the semantic operator into code using an LLM, then execute the generated code locally. We evaluate this design in two settings. First, we compare compiled semantic operators against naive LLM-based implementations that apply the LLM directly to each row or candidate pair. Second, we integrate compilation-based execution into an existing semantic operator system, Palimpzest [2] and compare end-to-end performance and output quality with the original execution model in Palimpzest. Our results show that compilation-based execution can significantly reduce runtime and LLM usage, while incurring only modest quality degradation, proving compilation-based execution is not only a conceptual innovation but also a practical solution to improve semantic operators’ speed and cost efficiency. Furthermore, we believe our vision and finding will inspire a broader range of research directions beyond semantic operators, making LLM-powered approaches more efficient and less costly in more areas.

This paper makes the following contributions:

• We propose a new semantic operator execution paradigm, compilation-based execution, as a practical and effective solution to improve semantic operator system efficiency and cost usage.

• We implement a prototype of compilation-based execution and conduct initial evaluation, showing that it has significantly higher efficiency, lower cost, with only small approximation loss of quality.

• We discuss the research challenges of semantic operator compilation, including operator compilability, semantic fidelity, and future optimizer integration. We further introduce how our vision may impact broader research areas.

The broader message of this paper is that LLM-powered applications should not only optimize how LLM calls are scheduled, batched, cached, or pruned. They should also ask when LLM calls can be removed from execution entirely.

2 Related Work↩︎

Semantic Operator Systems: Recent work has proposed semantic operators as a declarative abstraction for LLM-powered data processing. LOTUS introduces semantic operators that extend relational-style processing with operations such as semantic filtering, joining, aggregation, and ranking over structured and unstructured data [1]. Palimpzest similarly proposes a declarative system for AI-powered analytics over arbitrary collections of unstructured data, allowing users to specify high-level analytical workflows while the system searches over model choices, prompts, and physical implementations [2]. Abacus further develops this direction by introducing a cost-based optimizer for semantic operator systems, where operators such as maps, filters, and joins may have multiple physical implementations with different cost, latency, and quality tradeoffs [5]. Our work is complementary to these systems. They primarily optimize how LLM-based operators are executed, for example through planning, model selection, prompt selection, batching, caching, etc. In contrast, we study whether the LLM can be moved out of the runtime execution loop entirely.

Database Query Compilation: Some modern query engines compile query plans into low-level code to reduce interpretation overhead and improve CPU efficiency. For example, HyPer compiles query plans into efficient machine code using LLVM, showing that compilation can substantially improve query execution performance on modern hardware [7]. Our work is inspired by this database tradition, but applies the interpretation versus compilation distinction to semantic operators. In traditional query compilation, the input is a formal query with precise relational semantics, and the compiler preserves exact query meaning. In our setting, the input is a natural-language semantic operator whose behavior is typically defined by an LLM oracle or by the user’s intended meaning. As a result, compilation is approximate: the generated code is intended to mimic the behavior of the interpretation-based LLM operator, but it may not be semantically equivalent. This creates new research questions around semantic fidelity, operator compilability, and quality–cost tradeoffs.

3 From Interpreter to Compiler↩︎

3.0.0.1 The interpretation-based execution model.

A semantic operator is specified by a natural-language string, like a filter predicate in semantic filter (sem_filter), a per-row transformation in semantic map (sem_map), or a join condition in semantic join (sem_join). In today’s systems [1], [2], the operators are interpreted: at execution time the engine forwards each data item (row or pair), together with the predicate, to an LLM and reads back the per-item result. A unary operator over \(N\) items therefore issues \(N\) LLM calls, while a semantic join over collections of size \(M\) and \(N\) issues up to \(M\!\times\!N\). So both latency and money cost grow with record cardinality and width. This execution model is expressive but slow, expensive, and hard to scale, where the LLM calls are the bottleneck.

3.0.0.2 The key observation.

For a large class of operators the LLM’s per-item behavior is not open-ended, instead, many operators output a fixed-scope result based on some implicit attributes of the data item. For example, sem_filter and sem_join output binary judgment (True/False) on the current row or pair of rows; sem_filter("popular courses") may implicitly relies on thresholding attribute “course rating” to determine the course popularity. Such semantic operators with close-ended output and relatively clear decision rules are possible to be simulated with a deterministic program, since they are semantic on the surface but mechanical underneath. For such operators, invoking an LLM once per item re-derives the same rule \(N\) times in total.

3.0.0.3 Compile once, execute locally.

Therefore, for such operators, we propose to use the LLM as a compiler rather than an interpreter. During a one-time compilation phase, a single LLM call extracts the operator’s implicit decision rules into a explicit, deterministic Python function, such that no future LLM calls are needed to re-derive the rules again and again. The function is re-used for every item by local execution without LLM. Execution cost collapses from \(O(N)\) (or \(O(M\!\times\!N)\)) LLM calls to a single compile call. The running time is therefore also reduced due to drastically less LLM calls.

4 Implementation↩︎

4.0.0.1 Compiled operators.

We instantiate compilation for the common per-item semantic operators, i.e., semantic filter, map and join, whose decision rules are easier to extract than multi-item operators like semantic aggregation. Compiled filter (compiled_filter(row)) receives each record and outputs True/False on whether the record satisfied the extracted filter rule. Compiled map (compiled_map(row)) transforms/maps each record to a new value based on the extracted mapping rule. Compiled join (compiled_join(row_l, row_r)) receives a pair of records and decides True/False on whether the two records match based on the extracted join rule, and only matched records are kept as results. In every case a single compile prompt asks the LLM to emit a named deterministic function, and forbids any LLM call inside the generated function. Figure 2 shows this compilation prompt for sem_filter to compiled_filter on the running example. The prompt is parameterized only by the natural-language predicate and the row schema.

4.0.0.2 The compiler sees global information

Compilation changes the information available to the LLM. The LLM interpreter can see just the current row or pair of rows, while LLM compiler can see more, including a few more samples of rows from the table. This is reasonable as compiler only call LLM once, so adding a few more samples to compilation prompt does not affect the overall cost and latency too much, and this can further improve the compiled function quality. As a test, when compiling sem_join, we additionally include several real sample rows from left and right tables in the compiler prompt, so that the model can observe value-distribution overlap and better choose a correct join key.

5 Preliminary Results↩︎

We evaluate compilation-based execution in three settings of increasing integration. First, we run each compiled operator as a standalone implementation, outside any engine, to isolate the per-operator behavior of compilation versus per-item interpretation (§5.1). Second, we integrate the compiled operators into Palimpzest and run each operator inside the engine, comparing with the engine’s native LLM-interpretation operator (§5.2). Third, instead of single operators, we run the pipelines consisting of multiple operators implemented for the complex queries in SemBench [8] (§5.3), a benchmark specifically for semantic operator systems.

Unless otherwise noted, all settings use gpt-5-mini for both compilation method and interpretation baseline. The first two single-operator evaluations have no pre-labeled ground truth, as there is no such dataset and we have to construct predicates for each operator ourselves, without labors to annotate the ground truth results. Therefore we treat the results of the interpretation approach as the reference and report agreement—the fraction of items (rows or pairs of rows) on which the compiled operator returns the same result as the interpreter. The third evaluation on SemBench (§5.3) ships ground truth, so we score our method and the baseline directly against the ground truth.

5.1 Standalone Operators (outside the engine)↩︎

We first implement and evaluate the three per-item operators, i.e., filter, map, and join, outside any existing semantic operator systems. The compiled and interpreted versions are both run and the latter is used as ground truth. Table 2 reports agreement (accuracy, precision, recall, F1) between compiled operator results and those of the ground truth. The latency is also reported in three parts: the interpreted operator end-to-end latency ("Base"), the compilation phase latency ("Comp") and the execution latency of the compiled operator ("Exec").

3.5pt

The compiled operators agree with the interpreted implementations on the large majority of data items: 91% (filter), 91% (map), and 99% (join) accuracy and mostly more than 90% F1. The efficiency improvement is significant: once compiled, each query (with only one operator) is evaluated in microseconds, and the total time of compiled operator (compilation time + execution time) has a up to 31x speedup to interpreted version. These prove that compilation execution achieves large acceleration with tiny loss of quality in algorithm level without system level factors.

5.2 Evaluation of Single Operator inside Engine↩︎

We then integrates each compiled operator into Palimpzest and compare them with the engine’s native LLM-backed operator (LLMFilter / LLMConvertBonded / NestedLoopsJoin) on a suite of 30 / 36 / 29 filter / map / join queries over a movie dataset. Here each query still include only a single operator, with unique predicate. Similar to standalone evaluation, we take the native operator’s output as ground truth, and measure agreement, LLM-call count, dollar cost, and end-to-end time. To make it more straightforward, Table 3 reports agreement in the absolute values, while reports the LLM calls, cost, and time as “how many times compilation saves in average”, i.e., the metric of native implementation over that of compiled operator.

The agreement of filter and join are still close to the ground truth, except map who has the lowest agreement (0.889 accuracy and 0.826 F1), caused by the open-ended characteristic of sem_map, which will be discussed later (§5.5). The LLM usage and cost are significantly reduced (saving up to 31.6x cost). The time is largely saved except for sem_join, where compiled_join is slightly slower than native Palimpzest sem_join (0.9x). This is because Palimpzest’s has internal join parallelism, which improve its efficiency. However, this optimization does not save LLM cost as the number of calls remains unchanged. In contrast, compilation-execution could save both latency and cost, which is one of our biggest advantages.

5.3 Evaluation on SemBench Queries↩︎

To evaluate the performance on complex queries rather than single operator, we use the Palimpzest-based implementations of compiled and native operators to answer the ten movie queries (Q1–Q10) in SemBench. SemBench has provided ground truth operator pipeline to answer each query, with ground truth answer to each query. If the ground truth pipeline includes any operator that has no compiled version, we simply use a native operator. Since this benchmark has ground truth answers, we score both the baseline (full native operators) and our method (using compiled operators as much as possible) against the ground truth answers and report accuracy, F1, LLM calls, cost and time.

As reported in Table 4, the compiled operators are as accurate as the baseline in most cases, and markedly more accurate on the join-heavy queries: on Q5–Q7 (which looks for pairs of data), compiled pipeline has significant higher quality than baseline (e.g., accuracy 0.90/1.00/1.00 VS 0.50/0.70/0.65), demonstrating that compilation execution not only effectively approximate the interpreted implementation, but also be able to improve the execution quality in some cases. The LLM calls and cost of compiled pipeline are always significantly lower than baseline, with a more than 2x speedup. These further strengthen the effectiveness of our idea. The queries marked with ’’’ are those cannot generate meaningful results, like both methods output no answers. So we exclude their quality scores.

5.4 Scaling with Input Size↩︎

The paradigm’s key promise is that compilation’s benefit grows with input size. To test this we use the Pzlimpzest-based implementation over different sizes of tables: filter and map vary the row count (\(N=30/50/100\)), and the join varies the candidate-pair count (\(N=100/400/900\)).

Table 5 clearly confirms compilation execution’s advantage grows with data scale. For any operator, when \(N\) increases, average LLM cost saving increases rapidly (like from 8x to 90x saving on join), speedup becomes larger (like from 3.6x to 14.2x on filter), and quality remains high and stable (F1 keeps more than 0.9 in most cases). So large-scale datasets are especially suitable to use compilation execution.

5.5 Limitation of Compilation Execution↩︎

Not every operator compiles cleanly. Operators whose semantics can reduce to string, numeric, or key logic compile almost exactly, like join and filter. Compilation execution does not perform that well on operators which require open-ended world knowledge or judgment. In contrast, the predicates of map operator are usually more flexible and the output is relatively broader than join and filter, so map is often the lowest quality operator among the three implemented. But this does not affect the effectiveness of our idea as map still keep a high quality.

Another limitation is the compilation phase unavoidably takes long time since the resulting code may be complex with many keywords and matching rules, so generating such long code becomes the bottleneck.

6 Research Opportunities↩︎

Compilation-based execution opens several questions at the intersection of query processing, program synthesis, and LLM systems, which we believe will inspire people from broader domains beyond semantic data systems.

• Fidelity and verification. When does a compiled operator faithfully reproduce its LLM counterpart, and can we estimate this before trusting it? Possible solutions include small-scale test on a validation set, attaching a confidence score to the generated code, and so on.

• Compilability and operator routing. Some predicates are “implicitly deterministic” and compile near-perfectly; others are irreducibly semantic. The optimizer must decide, per operator, whether to compile, interpret, or both. Accurately estimating compilability is a big challenge and should be a first-class property alongside selectivity and cardinality in cost model.

• Hybrid and partial compilation. Rather than one strategy per operator, an engine could compile a predicate’s deterministic skeleton and fall back to the LLM only on the residual hard cases, i.e., letting lightweight compilation-based path handles more tasks while keeping the capability to use LLM only when necessary.

• Reuse, drift, and recompilation. A compiled operator is a reusable artifact, cacheable across queries and sessions, which raises classic systems questions: when does data drift invalidate it, and how cheaply can it be recompiled or repaired? These questions need more exploration.

• Beyond tabular and text. Multimodal operators (e.g., predicates over images) are the hardest to compile to short deterministic code, as they are naturally more semantic than table and text. Whether compilation extends to them is open. This probably requires synthesizing pipelines over learned models to extract and match visual features.

• Compilation-based execution beyond semantic operators. There are many other LLM-powered tasks satisfying (1) fixed-scope output and (2) decision rules is implicit but easy to extract, like spam filtering, document ranking, even LLM-based Text-to-SQL, which are good candidates for compilation execution. How to extend compilation-based execution to speed up those tasks beyond semantic data systems? This needs exploration.

7 Conclusion↩︎

We argue that semantic operator systems should treat the LLM as a semantic compiler rather than a runtime interpreter: a single compilation call can translate a natural-language operator into deterministic code that runs locally, removing the LLM from the execution loop to reduce cost and time. Our initial results suggest that this preserves the interpreter’s behavior while cutting LLM calls and cost by one to two orders of magnitude and saving several times of latency. We finally introduce the research opportunities around query processing, program synthesis, LLM systems, and furthermore, the potential of our vision to impact broader areas.

References↩︎

---

1. This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing info@vldb.org. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment.
   

   Proceedings of the VLDB Endowment, Vol. XX, No. XISSN 2150-8097.
   doi:XX.XX/XXX.XX
   ↩︎