Not Your Usual Type(s)


Abstract

Composable data systems promise to let developers combine languages, engines, and catalogs without sacrificing a coherent user experience. In practice, however, pipeline-node boundaries remain weakly specified: transformations exchange tables through schemas that are often checked late, enforced unevenly across languages, and disconnected from the semantics business users care about. Based on over a year of operating millions of jobs in Bauplan, we share the design principles behind our new SDK, which treats data contracts as types for a composable, multi-language lakehouse. Users, whether humans or agents, annotate input and output tables with schema objects that encode column types, constraints, documentation, and lineage; Bauplan then interprets these annotations at different points in the execution lifecycle. We show how this design addresses common production failures, and how an “everything-as-code” philosophy enables both deterministic and non-deterministic reasoning over data flows across languages and engines.

VLDB Workshop Reference Format:
. . VLDB 2026Workshop: Fourth International Workshop on Composable Data Management Systems.

1

1 Introduction↩︎

The data lakehouse is today the standard architecture for analytics and AI workloads, combining object storage with open table formats and decoupled, multi-language compute [1], [2]. In many lakehouses, data pipelines – DAGs of transformations from raw to refined data assets – are the most common OLAP use case, measured by both usage and total cost [3], [4].

Historically, a large fraction of DAG errors are due to schema changes at the intersection of two nodes [5], as columns get dropped or replaced, types change, and semantics shift. The problem is even more acute in composable data systems [6], where multi-language, multi-runtime pipelines are the norm and the node interface may be fully unconstrained or unevenly enforced across languages and runtimes. As the majority of labor shifts from writing to verifying code, thanks to the rise of agents [7], self-documenting, “correct-by-design” data projects promise to bring to data engineering the same productivity boost experienced by software engineers [8].

As most data correctness errors are prima facie the same avoidable runtime errors that plague dynamic languages [9], we redesigned our pipeline SDK by borrowing a familiar solution from dynamic languages: type annotations [10]. In this paper, we present the design of the new Bauplan SDK (“SDK 2.0”), which defines a contract layer for composable lakehouse DAGs. This layer makes schemas, constraints, lineage, and documentation portable across languages, execution engines, and lifecycle stages. Once contracts are explicit and machine-checkable, both humans and agents can safely operate over the same dataflow interface. In particular, we summarize our contributions as follows:

  1. multi-language DAG types: thanks to our privileged position running Bauplan – an agentic lakehouse at scale with millions of DAGs in production – we motivate SDK 2.0 through real-world failures, and outline how the new abstractions address them;

  2. three-stage contract enforcement: we map contract validation to three distinct moments in pipeline execution across a distributed platform: local inference over code files, planning inference over lakehouse state, and runtime inference over data assets produced in a run. As with software engineering, our guiding principle for agentic data engineering is to “shift left” the burden of correctness and fail as early as possible with a clear, explicit error message;

  3. everything-as-code: table descriptions, downstream metrics, joinable columns and column metadata all live next to the code that generates the table, and persist into the table metadata. Data semantics (i.e. how tables connect to business entities) are essential both for ongoing maintenance and for translation of business questions into analytics code [11]: by “shifting left” the burden of documentation, we enable a full agentic loop on the code base over a longer time horizon.

Altogether, these capabilities continue the convergence of data and software engineering, which we, among others, pioneered before the explosion of coding agents. While Bauplan is the natural application ground for these ideas, our design sits more generally at the intersection of programming languages, data management and distributed systems. As such, the ergonomics introduced here are easily portable to other composable systems and the guarantees are easily generalizable to other runtimes. All in all, we believe our lessons from the trenches to be valuable to a broad set of data practitioners.

2 Platform design and failure modes↩︎

2.1 Bauplan overview↩︎

Bauplan is a composable data system built for AI agents as first-class citizens [8], [12], [13]. It is composable as it re-uses a mix of proprietary and open source modules [14], [15] assembled with clear interfaces, while offering users a seamless, cohesive experience. In particular, Bauplan is a lakehouse built on the separation of storage (Apache Iceberg [16] on object storage) and compute (SQL and Python sandboxes). While we refer the reader to [17][19] for a full system overview, we survey here the distinctive features that are relevant for our SDK re-design.

We introduce a running example both to illustrate Bauplan’s original SDK and to introduce the platform abstractions. With “Titanic” as the source dataset2, we consider a three-node DAG. Node f is a SQL transformation that produces table “f” (through a naming convention), Node g is a Python transformation that produces table “g”, and Node h is a Python transformation that produces table “h”. Listings [lst:f_untyped], [lst:g_untyped], [lst:h_untyped] show each transformation expressed with the original SDK.

Listing lst:f_untyped: Node \#1 (f) as a SQL transformation

-- bauplan: name=f
-- bauplan: materialization_strategy=REPLACE
  SELECT PassengerId AS pid, Pclass AS tclass, Fare
  FROM titanic WHERE Pclass IN (1, 2) AND Sex = 'female'

Listing lst:g_untyped: Node \#2 (g) as a Python transformation

@bauplan.python('3.13', pip={'polars': '1.37'})
@bauplan.model(materialization_strategy='REPLACE')
def g(
  adult_lodgings=bauplan.Model(
    'titanic',
    columns=['PassengerId', 'Pclass', 'Fare', 'Cabin'],
    filter='Age >= 18 AND Age <= 65',
  ),
):
  return (
    pl.DataFrame(adult_lodgings)
      .with_columns(cabin_level=(pl.col('Cabin').str.slice(0, 1)))
      .with_columns(mean_fare=(pl.col('Fare').mean().over('cabin_level')))
      .select(pl.col('PassengerId').alias('pid'),
              pl.col('Pclass').alias('tclass'),
              'Cabin', 'cabin_level', 'Fare', 'mean_fare')
      .to_arrow()
  )

Listing lst:h_untyped: Node \#3 (h) as a Python transformation

@bauplan.python('3.13', pip={'polars': '1.37'})
@bauplan.model(materialization_strategy='REPLACE')
def h(f_data=bauplan.Model('f', columns=['pid', 'tclass']),
      g_data=bauplan.Model('g')):
    return (pl.DataFrame(f_data)
              .join(pl.DataFrame(g_data), on=('pid', 'tclass'))
              .filter((pl.col('Fare') > pl.col('mean_fare')))
              .select('pid', 'tclass', 'Fare', 'Cabin')
              .to_arrow())

The abstractions are largely self-explanatory, so we highlight only their declarative aspects, which play an important role in enabling “DAG types”:

DAG Topology

Python function parameters and SQL query source tables collectively define the DAG topology; f \(\rightarrow\) h and g \(\rightarrow\) h for the above snippets. When considering a SQL query as a function (as dbt3 does), it is easy to realize that transformations f and g share the same language-independent shape: one or more input tables and a single output table. Restricting the signature does not significantly constrain pipeline topologies and enables language-agnostic enforcement of invariants at the boundaries (Section 3).

Infrastructure

Python version and packages are listed in a decorator per function, requiring no client-side tools (such as Docker) and no installation commands; it is the platform’s job to make sure that the sandbox has the required dependencies when running the function.

I/O

Physical data operations (e.g., reading source data from S3 in f and g, passing their outputs into h, writing transformed data, and applying filter and projection pushdown in g) happen in “platform space”, so that user space only contains transformation code that is executed in secure and sandboxed compute. Fig. 1 (bottom) showcases the “isomorphism” [20] between the declarative layer and the corresponding data changes.

We will refer to this example (and variations on it) in the rest of the paper.

Figure 1: Pipeline declarative code and lakehouse changes. We show a simplified two-node DAG (top) transforming data from a source table into table-1 and table-2: from the data asset perspective, the language and logic of the transformations are implementation details for the lakehouse (bottom), which is recording on a data branch the I/O operations specified in the code.

2.1.1 Execution model↩︎

Bauplan has a standard lakehouse architecture (Figure 2): a control plane, a FaaS data plane, and a local client (CLI, SDK) to trigger cloud operations [17]. Fig. 2 shows the logical flow of information when a DAG gets executed (a “run”)—from a user’s local environment to object storage, and back. A coding agent writes a Bauplan project locally and submits the project when triggering the run; the control plane parses the code into a plan and sends it to a worker for execution; the worker reads/writes data from/to S3 and streams logs and result tuples back. The mapping between user code and lakehouse changes is depicted for a linear DAG in Fig. 1: since tables are managed through Git-like abstractions [21], [22], every function execution in the DAG corresponds to a commit, i.e., an immutable reference to the state of the lakehouse at that time. For example, since g is declared with \(REPLACE\) semantics, the platform will drop and re-create the table during a run, then insert rows: what the user writes as a single function gets (deterministically) “compiled” as a series of physical data operations on the lake.

Importantly, the logical flow of a run identifies three key moments when executing a DAG: (\(1\)) the local code environment, before the run is triggered; (\(2\)) the control plane, when preparing the run (before DAG execution begins); and finally (\(3\)) the worker process, before and after physical data operations (persisting data is a specific operation). These roughly correspond to code compilation, CI/CD and runtime invariant checks in software development: as a general design principle, we want a system that fails as early as possible and with clear responsibilities at each stage of the process.

a

Figure 2: A run in Bauplan: 1) a user writes code locally, and triggers the run; 2) the control plane parses the code into a plan and sends it to a worker for execution; 3) the worker reads/writes data from/to S3 and 4) streams logs and results to the user..

2.2 Failure modes↩︎

Across millions of production jobs, “avoidable” schema mismatches at node boundaries have been a recurring and operationally important class of failure [5], [23]. Note that while these failures are already challenging for workloads run by human experts, their severity is amplified in a world in which untrusted agents operate on production data at scale [8]. To illustrate common failure modes, we consider a few scenarios:

columns dropped

If \(tclass\) is dropped from g without dropping references to it in h, we see an error in h at runtime.

null values present

The \(Pclass\) column, and therefore the derived \(tclass\) column, contains no \(NULL\) values, although both are declared nullable in Iceberg. If g filters out \(NULL\) values from its output, there is no way to communicate that intent to other DAG nodes: constraints cannot be defined, enforced, or even remembered for query results.

type change

If g were to change \(tclass\) to have string values such as “First” for first class instead of the numerical value \(1\), it would implicitly alter the column type. This would raise an error for h when it is used to join with the table “f” at runtime.

shifting semantics

Consider that \(tclass\) from f is filtered to specific values, whereas \(tclass\) from g is unfiltered. Downstream analyses can easily look correct but actually suffer from errors of omission due to confusing similarly named columns in (or after) a join operation. Column lineage can be used to disambiguate columns by origin (which DAG node they flow from).

Pushing semantic information to live next to everything else is part of Bauplan’s original vision of “everything-as-code”, in line with many modern Pythonic frameworks [24]. Additionally, beyond these failures, a second trend has been pushing us to revise SDK 1.0: as downstream consumers of refined assets are themselves agents free to explore the lakehouse, many of the original semantic layer [25] capabilities (such as query compilation and lineage) seem ripe for disintermediation by on-the-fly LLM reasoning.

3 DAG Types↩︎

In this section, we discuss our approach to evolving Bauplan in a principled way to address common challenges rooted in schema failures. Our approach follows a typical two-part pattern from dynamic languages such as Python: (\(1\)) optional syntax—annotations of code for both human and machine verification and (\(2\)) enforcement semantics—how a verifier interprets the annotations.

3.1 Annotation Syntax↩︎

We address hidden, unintentional schema failures by defining syntax for explicit, machine-checkable contracts. This allows developers to declaratively specify expectations between transformations and incrementally add them to their Bauplan projects to achieve schema and column lineage inference, as exemplified by the snippets below4.

Listing lst:schemas: Type contract syntax

class WomenByClass(TableSchema):
    """Schema for women passengers with 1st or 2nd class tickets."""
    pid:    Annotated[Int64, Required, Doc('Unique passenger ID')]
    tclass: Annotated[Int64, Required,
        Doc('Ticket class, filtered to 1st or 2nd class only.')
    ]
    Fare:   Annotated[Float64, Doc('Ticket cost (British pounds)')]

  class CabinCostSource(TableSchema):
    """Schema for cabin costs used for analysis."""
    # Same annotations as WomenByClass['pid', 'tclass']
    PassengerId: Annotated[Int64, ...]
    Pclass:      Annotated[Int64, ...]
    Fare:  Annotated[Float64, Doc('Ticket cost (British pounds)')]
    Cabin: Annotated[String,  Doc('Cabin number (C28 is on deck C)')]

  class CabinCostAnalysis(TableSchema): ... # Omitted for brevity
  
  class ExpensiveWomenCabins(TableSchema):
    """Schema for expensive cabins reserved by working-age women."""
    # Lineage by referencing schema column (`Schema['col']`)
    pid:    Annotated[Int64,   WomenByClass['pid']]
    tclass: Annotated[Int64,   WomenByClass['tclass']]
    Cabin:  Annotated[String,  CabinCostAnalysis['Cabin']]
    Fare:   Annotated[Float64, WomenByClass['Fare'],
        Doc('Cost of tickets that cost more than average (by deck).')
    ]

Listing [lst:schemas] illustrates how schema objects are defined and how they are associated with rich annotations. Then, Listings [lst:f_typed], [lst:g_typed], and [lst:h_typed] show how each node in the DAG directly associates its input tables (from upstream transformations) and its output table with defined schema objects. In this way, schema objects elegantly relate documentation and constraints to DAG nodes using a lightweight, extensible mechanism.

Schema Objects are defined using a Python-centric design to minimize the need for custom SQL and because we expect the syntax to be more ergonomic in Python.5 This means that we lean heavily into the use of standard type annotations using \(typing.Annotated\) and familiar Pydantic-style annotations [26] (type objects such as \(Required\) and classes such as \(Doc\) for objects that don’t support docstrings).

Bauplan uses the base class \(TableSchema\) to identify Bauplan schema objects and uses the special \(Annotated\) type to identify column annotations. The first annotation is the column’s data type and subsequent annotations may be rich metadata for “semantic layer” support (Section 4.2) or may be constraints for validation (Section 4.3). For column data types, we choose to accept generic names for supported types (such as \(String\) and \(Float64\)) and Bauplan defines a direct correspondence to PyArrow types: to maximize compatibility with other platforms (Section 4.4), we support the subset of Arrow types that intersects with Iceberg. To identify explicit column lineage, a schema column may be referenced directly with the syntax \(Schema['column\_name']\). For example, the annotation \(WomenByClass['pid']\) on \(ExpensiveWomenCabins.pid\) explicitly records lineage to the \(pid\) field in the \(WomenByClass\) schema, allowing the system and downstream tools to recover the corresponding upstream origin and metadata.

Listing lst:f_typed: Node \#1 (f) with output schema reference

-- bauplan: name=f
  -- bauplan: materialization_strategy=REPLACE
  -- bauplan: output_schema=WomenByClass
  SELECT PassengerId AS pid, Pclass AS tclass, Fare
  FROM titanic WHERE Pclass IN (1, 2) AND Sex = 'female'

Listing lst:g_typed: Node \#2 (g) with typing

@bauplan.python('3.13', pip={'polars': '1.37'})
  @bauplan.model(materialization_strategy='REPLACE')
  def g(
      adult_lodgings: Annotated[
          Table[CabinCostSource],
          Filter(Model('titanic'), 'Age >= 18 AND Age <= 65')
      ],
  ) -> Table[CabinCostAnalysis]:
    """Analyze the cabin cost by level for working-age adults."""
    # Function body remains unchanged
    return (pl.DataFrame(adult_lodgings).(...).to_arrow())

Listing lst:h_typed: Node \#3 (h) with typing

@bauplan.python('3.13', pip={'polars': '1.37'})
  @bauplan.model(materialization_strategy='REPLACE')
  def h(
      f_data: Annotated[Table[WomenByClass],     Model('f')],
      g_data: Annotated[Table[CabinCostAnalysis], Model('g')],
  ) -> Table[ExpensiveWomenCabins]:
    """Expensive cabins reserved by/for working-age women."""
    # Function body remains unchanged
    return (pl.DataFrame(f_data).(...).to_arrow())

Transformation Annotations are defined by parameter and return type annotations in Python and by a structured comment in SQL. In Python, a function parameter is annotated using \(Annotated\) where the first annotation is a \(Table[schema]\) type, meaning a tabular data structure (such as \(pyarrow.Table\)) whose schema matches the specified schema contract. The second annotation must resolve to a \(Model\) reference identifying an input transformation (an Iceberg table or DAG node), either directly or through a chain of operator functions such as \(Filter\).6 As syntactic sugar, the schema from the first annotation is applied as a projection on the \(Model\)’s output.

In SQL, a query is preceded by a structured comment that specifies a schema object, by name, to be associated with the SQL result set. While this lacks support in the client environment in Step 1 (Fig. 2), it allows the control plane to associate contracts with SQL transformations at Step 2. Local static checking is therefore currently available only for Python nodes; SQL contracts first participate in validation during control-plane planning, while Step 3 enforcement at the Arrow boundary is language-independent across supported runtimes.

3.2 Enforcement Semantics↩︎

Contracts are enforced by “fail fast” behavior where errors are lifted to the earliest possible point in the execution lifecycle. When the user authors code, local type checkers can catch obvious mismatches immediately (Step 1). Next, before scheduling any execution, the control plane parses the metadata (Step 2) and validates that adjacent nodes compose correctly and that the boundary between tables in the catalog and the assets defined in the DAG is specified correctly. Finally, at the worker (Step 3), runtime checks validate that the physical data conforms to its specified schema before execution (input) and before any results are persisted (output), ensuring that late-discovered schema problems do not leak inconsistent state into storage.

Enforcement at Steps 1 and 3 is relatively straightforward. Step 1 is entirely deferred to a local type checker as annotation syntax is purposefully designed to enable the use of standard type checkers. For example, \(Table\) is defined in a way so that a type checker recognizes it as having the same interface as a \(pyarrow.Table\), which prevents calling invalid methods on \(adult\_lodgings\) in g.

At Step 3, a worker uses information from the control plane to validate that an output table (resulting from the transformation) has a correct schema as specified in the Bauplan project. For example, when h returns an Arrow table:

\[pl.DataFrame(f\_data).select(...).to\_arrow()\]

then an appropriate error message will be raised at runtime if schema validation fails:

Listing lst:runtime_err: Example schema errors

# Some possible errors on column "Fare" after the transformation `h`
  Error: job failed: function failed due to user error:
  Schema contract validation failed on model [h]:
    - missing column "Fare" (expected double) (RuntimeTaskUserError)
    - column "Fare" has type int64, expected double (RuntimeTaskUserError)
    - unknown column "Fare" (has type int64) (RuntimeTaskUserError)

Other table validations can then be performed, such as checking for null values (Section 4.3). Finally, auxiliary logic may be executed, such as inserting column type casts when necessary.

Step 2 Enforcement involves a more complex analysis of the DAG types and how information and metadata flow across the DAG. We leverage a graph-based representation to reason about data, task, and infrastructure dependencies [27], turning consistency checks into graph queries (e.g. do projections in the child node correctly map to its parent’s output?). Only in Step 2 can the system link catalog information on source tables with the pipeline code, allowing deterministic type checks at the boundaries between DAG and catalog, and full reasoning over the data flowing from the lakehouse into the tasks being planned.

Listing lst:type_cast: New node with type-cast

class TotalFares(TableSchema):
    """Simple schema for ticket fare data."""
    fare_total: Annotated[Int64, Required,
      Doc('Total ticket prices, rounded up to the nearest British pound.')
    ]
  
  @bauplan.python('3.13', pip={'polars': '1.37'})
  @bauplan.model(materialization_strategy='REPLACE')
  def fare_sum(
      tcosts: Annotated[Table[WomenByClass], Model('f')],
  ) -> Table[TotalFares]:
    """Compute the total fare for first- and second-class women passengers."""
    # Function body remains unchanged
    return (pl.DataFrame(tcosts)
              .select(fare_total=pl.col('Fare').sum().ceil())
              .to_arrow())

With Listing [lst:type_cast] as a concrete example, we consider \(Fare\) and how it flows from titanic to f to fare_sum. In the catalog, titanic defines \(Fare\) as a \(double\), corresponding to \(Float64\). In Listing [lst:schemas], the schema \(WomenByClass\) preserves \(Fare\) as \(Float64\) and adds documentation. The new node fare_sum computes a sum, but wants the result to be rounded and returned as an \(Int64\), as specified by its output type \(TotalFares\). Before execution, the control plane determines that the output column \(fare\_total\) must have type \(Int64\) and includes this contract in the execution plan. At Step 3, the worker validates the produced Arrow column and, where permitted, applies the required cast before persistence.

4 Capabilities↩︎

SDK 2.0 is now in beta and will soon be included in our documentation: since our SDK is open-source, the client-side implementation is publicly available. In this section, we survey the consequences of this new design across four important dimensions: preventing the “avoidable failures”, powering semantic reasoning through in-code documentation, enforcing data quality concisely and declaratively, and enabling engine composability without giving up contractual guarantees.

4.1 Failure modes revisited↩︎

With our enforcement semantics defined, we are now in a position to revisit the failure modes of SDK 1.0 and show how the new annotations address those challenges.

columns dropped

If g’s declared output contract drops \(tclass\) while h still requires it, the control plane rejects the DAG at Step 2. If g’s implementation omits \(tclass\) while its declared contract still includes it, the worker rejects g’s output at Step 3, before persistence.

null values present

If g filters out \(NULL\) values from its output, we can explicitly declare the non-null constraint and relate any violation to the node that defines it.

type change

If g’s declared type for \(tclass\) changes from \(Int64\) to \(String\) while h still expects \(Int64\), the control plane rejects the DAG at Step 2. If g’s implementation produces strings while its contract still declares \(Int64\), the worker rejects g’s output at Step 3, before persistence.

shifting semantics

For a transformation using \(tclass\) to disambiguate between f and g, we declare the desired lineage directly as WomenByClass[‘tclass’] where \(WomenByClass\) is the schema for the output of f.

Each of the common failures now has a direct and concise solution.

4.2 Semantic reasoning↩︎

In line with the “everything-as-code” principle of modern data frameworks [24], SDK 2.0 couples transformation code with annotations describing the intended business meaning of the assets, as well as column-level metadata. In the write path, everything-as-code guarantees that data semantics and data transformations are always “in context”, which follows the best practices for AI-assisted coding and simplifies both human verification and ongoing code maintenance by coding agents.

What about the read path? The first step is therefore to make data semantics available outside the code base, without disconnecting them from the code that produced them. This is achieved at materialization time: when the materialization of assets occurs during a run, table and column annotations are persisted in versioned Iceberg table metadata associated with the materialized table state. When business users ask LLMs to translate English questions into queries (Figure 3), an agent using an MCP server that exposes Iceberg catalog and metadata APIs is able to retrieve asset descriptions and column information and put them into the LLM context for accurate SQL generation. In other words, annotations become part of the lakehouse exchange layer: any downstream system can recover the same semantic context, independently of what produced it. This turns metadata from local documentation into a composability mechanism across tools, engines, and agents.

Once again, the design stresses the outer composability of Bauplan (how Bauplan and other data systems interact at predefined, clean interfaces) on top of the usual inner composability (how Bauplan itself is built out of several modules exposed through vertical APIs).

a

Figure 3: A coding agent writes transformations and table metadata, which are persisted in Iceberg during a run. At read time, an agent uses an MCP server exposing standard Iceberg metadata APIs to retrieve that information and put it into the LLM context for SQL generation..

4.3 Data quality↩︎

Types also give a principled handle on data quality checks without additional code – such as manually writing expectations in SDK 1.07 – or additional tools. For example, Listing [lst:enum] illustrates an enumeration allowing declarative runtime checks without any user code to explicitly compare each value to \(1\) or \(2\). In particular, scalar constraints such as non-nullability and enumerated values can be checked directly on the Arrow buffer “in-flight”, without the need for materialization and wasteful computation as is typical in dbt-based setups.

Listing lst:enum: Supporting enum column types

class WomenByClass(TableSchema):
    """Schema for women passengers with 1st or 2nd class tickets."""
    pid:    Annotated[Int64, Required, Doc('Unique passenger ID')]
    tclass: Annotated[Int64, Required,
        Enum('tclass', names=[('1st', 1), ('2nd', 2)]),
        Doc('Ticket class, filtered to 1st or 2nd class only.'),
    ]

Following the “shift left” philosophy, we envision a near future in which the SDK provides even further guarantees in the form of “Dafny-style” preconditions and postconditions for SQL nodes in a DAG. As an example, consider an aggregation-type node such as SELECT col1, COUNT(*) FROM table GROUP BY ALL: static analysis alone guarantees that if no nulls are in table.col1, no new nulls will be created by this transformation – in turn, if we knew through annotation and type inference that table.col1 is indeed not null, we could conclude via modus ponens that no nulls will be returned at the end.

4.4 Composability↩︎

Through its connector semantics8, Bauplan supports replacing its proprietary sandboxes with alternative runtimes that implement the platform’s execution interface. Because contracts are expressed over Arrow-compatible types, any such runtime that consumes and produces Arrow data can enforce the same input and output boundaries, even when the transformation engine changes. Further, annotations may be persisted in an Iceberg catalog and propagated between independent data pipelines or shared with other data platforms and subsystems (Section 4.2).

As the composable data systems community expands, we envision independent platforms interpreting a common subset of these annotations, allowing contracts and semantic metadata to propagate across engines through Arrow and Iceberg interfaces.

5 Related work↩︎

Popular DAG frameworks provide partial forms of asset contracts. Dagster [28] provides Python-centric asset checks, while dbt model contracts [29] are limited to SQL models and perform a build-time preflight check of each model’s query output against its YAML declaration. Unlike Bauplan, they do not type-check contracts across SQL and Python nodes or compose pipeline contracts against the current lakehouse catalog state before execution.

Python libraries such as Pandera [30] and Patito [31] similarly provide class-based schemas and runtime validation for DataFrames. Bauplan differs by attaching contracts to multi-language DAG boundaries, checking their composition against catalog state before execution, and enforcing them through a shared Arrow boundary.

The extensive use of \(Annotated\) and the Pythonic DAG syntax are heavily inspired by Pydantic’s widely understood class-based annotation pattern [26]. Separately, the idea of column lineage and its benefits is based on the analysis used in SqueezeCache for “squeezing” [32]. Our usage of lineage is currently more generic (closer to dataflow).

Pythonic data-quality tools such as Great Expectations [33] provide a rich framework for validating data assets through expectations and validation suites. However, they add a separate validation layer: users must adopt new dependencies, learn a distinct API, and write quality checks in addition to the transformation code. Our approach instead makes common data-quality constraints part of the table type itself. Nullability, enums, and column types are declared once in the schema and then interpreted uniformly across the client, control plane, and data plane.

6 Conclusion↩︎

We presented Bauplan SDK 2.0, a contract-oriented extension of our lakehouse programming model. Starting from common failures in multi-language pipelines, we introduced DAG types: lightweight schema objects that attach column types, constraints, documentation, and lineage to pipeline inputs and outputs. Bauplan interprets these annotations across the execution lifecycle, enabling earlier validation in the client, stronger contract composition in the control plane, and runtime checks before invalid results are persisted. The result is a portable contract layer for composable data systems, where correctness and semantics live next to the code that defines the data flow.

Our next step is to push these checks further left. We plan to develop a Bauplan Language Server Protocol (LSP) implementation that brings schema validation and actionable diagnostics directly into the IDE. This would move more reasoning client-side, reduce feedback latency, and enable even faster and safer agentic exploration over production data.

References↩︎

[1]
M. A. Zaharia, A. Ghodsi, R. Xin, and M. Armbrust, “Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics,” in Conference on innovative data systems research, 2021.
[2]
D. Mazumdar, J. Hughes, and J. Onofre, “The data lakehouse: Data warehousing and more.” 2023, [Online]. Available: https://arxiv.org/abs/2310.08697.
[3]
C. G. Tapan Srivastava Jacopo Tagliabue, “Eudoxia: A FaaS scheduling simulator for the composable lakehouse,” Proceedings of Workshops at the 51st International Conference on Very Large Data Bases, 2025.
[4]
A. van Renen et al., “Why TPC is not enough: An analysis of the amazon redshift fleet,” in VLDB 2024, 2024.
[5]
H. Foidl, V. Golendukhina, R. Ramler, and M. Felderer, “Data pipeline quality: Influencing factors, root causes of data-related issues, and processing problem areas for developers,” Journal of Systems and Software, vol. 207, p. 111855, 2024, doi: https://doi.org/10.1016/j.jss.2023.111855.
[6]
P. Pedreira et al., “The composable data management system manifesto,” Proc. VLDB Endow., vol. 16, no. 10, pp. 2679–2685, Jun. 2023, doi: 10.14778/3603581.3603604.
[7]
R. Huang, A. Reyna, S. Lerner, H. Xia, and B. Hempel, “Professional software developers don’t vibe, they control: AI agent use for coding in 2025.” 2025, [Online]. Available: https://arxiv.org/abs/2512.14012.
[8]
J. Tagliabue and C. Greco, “Safe, untrusted, "proof-carrying" AI agents: Toward the agentic lakehouse.” 2025, [Online]. Available: https://arxiv.org/abs/2510.09567.
[9]
S. Sun, S. Zhang, J. Yan, J. Yan, and J. Zhang, “Co-evolution of types and dependencies: Towards repository-level type inference for python code.” 2025, [Online]. Available: https://arxiv.org/abs/2512.21591.
[10]
L. Di Grazia and M. Pradel, “The evolution of type annotations in python: An empirical study,” in Proceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering, 2022, pp. 209–220, doi: 10.1145/3540250.3549114.
[11]
C. Labadie, C. Legner, M. Eurich, and M. Fadler, “FAIR enough? Enhancing the usage of enterprise data with data catalogs,” in 2020 IEEE 22nd conference on business informatics (CBI), 2020, vol. 1, pp. 201–210, doi: 10.1109/CBI49978.2020.00029.
[12]
W. Sheng, J. Wang, M. Barros, A. Montana, J. Tagliabue, and L. Bigon, “Building a correct-by-design lakehouse. Data contracts, versioning, and transactional pipelines for humans and agents.” 2026, [Online]. Available: https://arxiv.org/abs/2602.02335.
[13]
J. Tagliabue, “Querying everything everywhere all at once: Supervaluationism for the agentic lakehouse.” 2026, [Online]. Available: https://arxiv.org/abs/2603.13380.
[14]
L. Bigon, J. Tagliabue, and S. Salihoğlu, “DAG lakehouse planning with an ephemeral and embedded graph database,” in VLDB 2025 workshop: Third international workshop on composable data management systems, 2025, [Online]. Available: https://www.vldb.org/2025/Workshops/VLDB-Workshops-2025/CDMS/CDMS25_13.pdf.
[15]
R. Curtin and J. Tagliabue, “The deconstructed warehouse: An ephemeral query engine design for apache iceberg,” in VLDB 2025 workshop: Third international workshop on composable data management systems, 2025, [Online]. Available: https://www.vldb.org/2025/Workshops/VLDB-Workshops-2025/CDMS/CDMS25_12.pdf.
[16]
Apache, “Iceberg,” GitHub repository. https://github.com/apache/iceberg; GitHub, 2024.
[17]
J. Tagliabue, T. Caraza-Harter, and C. Greco, “Bauplan: Zero-copy, scale-up FaaS for data pipelines,” in Proceedings of the 10th international workshop on serverless computing, 2024, pp. 31–36, doi: 10.1145/3702634.3702955.
[18]
J. Tagliabue, C. Greco, and L. Bigon, “Building a serverless data lakehouse from spare parts,” ArXiv, vol. abs/2308.05368, 2023, [Online]. Available: https://api.semanticscholar.org/CorpusID:260775634.
[19]
J. Tagliabue, R. Curtin, and C. Greco, FaaS and Furious: abstractions and differential caching for efficient data pre-processing ,” in 2024 IEEE international conference on big data (BigData), Dec. 2024, pp. 3562–3567, doi: 10.1109/BigData62323.2024.10825377.
[20]
N. R. Schneider, D. Ghilardi, G. Piccinini, and J. Tagliabue, “"Skill issues”: Data-centric optimization of lakehouse agents.” 2026, [Online]. Available: https://arxiv.org/abs/2606.01185.
[21]
J. Tagliabue and C. Greco, “Reproducible data science over data lakes: Replayable data pipelines with bauplan and nessie,” in Proceedings of the eighth workshop on data management for end-to-end machine learning, 2024, pp. 67–71, doi: 10.1145/3650203.3663335.
[22]
W. Sheng, J. Wang, M. Barros, A. Montana, J. Tagliabue, and L. Bigon, “GitLake: Git-for-data for the agentic lakehouse.” 2026, [Online]. Available: https://arxiv.org/abs/2607.08319.
[23]
D. Bhadauria, H. Harmouch, F. Naumann, D. Srivastava, and L. Ehrlinger, “A catalog of data errors.” 2026, [Online]. Available: https://arxiv.org/abs/2604.09277.
[24]
J. Tagliabue, H. Bowne-Anderson, V. Tuulos, S. Goyal, R. Cledat, and D. Berg, “Reasonable scale machine learning with open-source metaflow,” ArXiv, vol. abs/2303.11761, 2023.
[25]
M. Rumiantsau and I. Fokeev, “Semantic layers for reliable LLM-powered data analytics: A paired benchmark of accuracy and hallucination across three frontier models.” 2026, [Online]. Available: https://arxiv.org/abs/2604.25149.
[26]
S. Colvin et al., “Pydantic validation.” May 22, 2026, [Online]. Available: https://github.com/pydantic/pydantic.
[27]
S. S. Luca Bigon Jacopo Tagliabue, “DAG lakehouse planning with an ephemeral and embedded graph database,” Proceedings of Workshops at the 51th International Conference on Very Large Data Bases, 2025.
[28]
Dagster Labs, “Welcome to dagster.” https://docs.dagster.io/, Jan. 2026.
[29]
dbt Labs, Inc., “What is dbt?” https://www.getdbt.com/product/what-is-dbt, Jan. 2026.
[30]
P. Contributors, Accessed: 2026-07-14“Pandera: The open-source framework for dataset validation.” https://pandera.readthedocs.io/.
[31]
P. Contributors, Accessed: 2026-07-14“Patito: A data modelling layer built on top of polars and pydantic.” https://patito.readthedocs.io/.
[32]
X. Hao et al., “SqueezeCache: Beyond ”optimal” eviction for data analytics.” https://xiangpeng.systems/_app/immutable/assets/squeeze-cache.CgwfvKO-.pdf, 2026.
[33]
Great Expectations, Open-source data quality validation frameworkGreat Expectations Core.” https://github.com/great-expectations, 2026.

  1. This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing . Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment.

    Proceedings of the VLDB Endowment. ISSN 2150-8097.
    ↩︎

  2. https://www.kaggle.com/datasets/yasserh/titanic-dataset↩︎

  3. https://github.com/dbt-labs/dbt-core↩︎

  4. Please check the accompanying repository for the full-fledged example: redundant details are omitted here for brevity↩︎

  5. Note that while the design does not require a specific version of Python, we show syntax best supported in Python \(>=3.13\) for convenience↩︎

  6. The \(Annotated\) type requires at least 2 annotations and uses the first for type checkers.↩︎

  7. https://docs.bauplanlabs.com/concepts/expectations↩︎

  8. https://docs.bauplanlabs.com/integrations/warehouses-lakehouses/snowflake-outbound↩︎