1 Introduction↩︎

Multimodal large language model (MLLM) applications are becoming increasingly widespread. However, multimodal inference frameworks have not kept pace with this rapid evolution. Current MLLMs require multiple heterogeneous computational modules to collaborate during inference, including text encoders, vision encoders [1]–[3], audio encoders [4], [5], large language models (LLMs) for text generation or Diffusion Transformer (DiT) forward computation [6], [7], diffusion denoising modules [8]–[10], and variational autoencoder (VAE) encoders and decoders [11]. Multimodal model architectures are still evolving rapidly, and different tasks exhibit substantially different model topologies. Image understanding [12], [13], image generation [14], [15], speech recognition, and spoken dialogue employ different encoder and decoder structures as well as different execution schedules. At this stage, therefore, building a unified system framework that can efficiently accommodate diverse model architectures is more urgent than aggressively optimizing for a single fixed architecture.

As shown in Fig. 1, recent model designs represented by BAGEL [16] and HunyuanImage-3 [17] are increasingly unifying understanding and generation within a shared Transformer backbone [18]–[20]. The LLM performs cross-modal understanding, while the diffusion process reuses the same model weights during generation and shares the KV cache to avoid redundant GPU memory consumption. LongCat-Next [21], in contrast, adopts an architecture consisting of an LLM and multiple multimodal heads, where the text, image, and audio heads exhibit substantially different computational loads.

Any-to-any multimodal models compose multiple heterogeneous models into a single inference pipeline. Consequently, intermediate tensors and KV caches are no longer confined to a single inference engine, but must be efficiently shared and transferred across different computational roles. This shift introduces several new system requirements. Resource-constrained single-node deployments require local sharing of model weights and KV caches to avoid doubling GPU memory consumption. High-performance deployments spanning multiple machines require a unified mechanism for distributed KV-cache management. Meanwhile, continuously evolving multimodal model topologies demand a flexible workflow abstraction capable of accommodating diverse heterogeneous execution pipelines. Existing production-grade inference frameworks, however, lack a system-level abstraction that jointly addresses these requirements.

To address these challenges, we present Omni-Flow, a model-agnostic serving framework for heterogeneous multimodal inference. Omni-Flow organizes a unified multimodal inference system around three cooperating abstractions: Control Flow, Data Flow, and Compute Flow (overall architecture shown in Fig. 2). (1) Control Flow provides a unified representation of multimodal workflows and their execution logic. It models multimodal inference as a declarative directed graph with support for cycles, where nodes represent heterogeneous computational roles and edges represent data-transfer channels. The abstraction supports advanced workflow semantics, including OR-AND aggregation, multi-yield streaming, diamond, serial, and multi-stream joins. It also incorporates service discovery and multiple load-balancing policies, decoupling workflow topology from model implementation. (2) Data Flow provides unified management of intermediate tensors and distributed KV caches through a common data plane that supports data sharing and transfer across processes, machines, and storage tiers. For KV-cache management, Omni-Flow integrates direct prefill-to-decode (PD) transfer [22], [23], paged management [24], and hierarchical storage [25], [26] into a distributed KV-cache manager. This design restricts the inference engine to model forward computation while delegating cache allocation, migration, sharing, and reclamation across the L1/L2/L3 storage hierarchy to the data plane. It further supports direct data sharing and transfer across computational roles, storage tiers, and machines. The system also provides a zero-copy, low-latency communication channel with a tiered fallback mechanism, while supporting zero-copy sharing of model weights, KV caches, and input/output parameters among multiple processes on the same machine. (3) Compute Flow provides a unified execution interface for heterogeneous models. It supports complex multimodal prefix matching—spanning both full matching and incremental matching modes—to enable KV reuse across multi-turn dialogues involving heterogeneous modal outputs. It uses SGLang as a deeply customizable LLM inference kernel and takes control of the KV-cache pool and sampling post-processing through a unified interface. For compatible diffusion models, the DiT module can directly reuse the execution path and PagedAttention infrastructure [24] of the LLM inference engine. This design enables model weights and KV caches to be shared under unified parallel-execution semantics, reducing duplicated implementation effort and improving resource reuse across heterogeneous models. These three abstractions are decoupled yet cooperative, allowing model topology, data movement, and underlying computation to evolve independently. Together, they provide a unified, extensible, and efficient serving framework for rapidly evolving multimodal models.

To summarize, we make the following contributions.

• We propose a unified declarative workflow abstraction that models multimodal inference as an execution graph, supporting heterogeneous computation roles, conditional execution, multi-yield streaming, flexible join patterns, and iterative workflows with loops.

• We propose a unified data plane that manages the lifecycle of intermediate tensors and KV caches, unifying cache management, data transfer, hierarchical storage, and zero-copy sharing into a single framework for efficient cross-role and cross-machine resource sharing.

• We propose a unified heterogeneous computation abstraction that enables compatible Diffusion Transformers to reuse the execution infrastructure of LLM inference engines, reducing model adaptation overhead while improving resource reuse.

• We implement Omni-Flow on top of SGLang and validate its effectiveness, generality, and scalability on representative multimodal models. We also share our AI-assisted development practices and engineering lessons in Appendix 10.

2 Related Work↩︎

2.1 Multimodal Inference Serving and Workflow Orchestration↩︎

Modern LLM serving systems, such as vLLM [24] and SGLang [27], incorporate a mature set of optimizations, including paged KV-cache management [24], prefix caching [27], continuous batching [28], [29], prefill–decode disaggregation [22], [23], [30], hierarchical KV-cache storage [25], [26], and KV-cache compression for long contexts [31]–[33]. These techniques substantially improve the throughput and resource utilization of autoregressive inference, but are primarily designed around the execution of a single LLM and therefore do not readily generalize to any-to-any multimodal inference involving multiple heterogeneous models. Recent systems, including vLLM-Omni [34] and SGLang-Omni [35], extend LLM serving to multimodal workloads by decomposing a workflow into multiple stages. Their execution topologies, however, remain tightly coupled to model implementations. vLLM-Omni exposes only coarse-grained stages, leaving modality-specific components such as vision and audio encoders encapsulated within individual models. SGLang-Omni supports finer-grained stages, but distributes topology specifications across configuration, routing, and executor components, complicating extension and maintenance. Both systems also provide limited support for dynamic control flow: vLLM-Omni cannot skip stages on a per-request basis, whereas SGLang-Omni supports only coarse-grained path routing. Moreover, their streaming interfaces depend on the underlying communication mechanisms, and their execution models are restricted to directed acyclic graphs, precluding iterative workflows with loops. General-purpose orchestration frameworks such as LangGraph [36] and LangChain [37] expose explicit graph abstractions, but target application-level agent orchestration rather than the tensor-level scheduling required within a multimodal inference engine. Representative workflow definitions for these systems are provided in Appendix 9, and Table ¿tbl:tab:capability-comparison? summarizes the capability comparison.

2.2 KV-Cache Management for Multimodal Inference↩︎

The emergence of unified multimodal understanding and generation models, such as BAGEL [16], HunyuanImage-3 [17], Show-o [18], and Janus [19], has made cross-model KV-cache sharing increasingly important for efficient multimodal inference. Existing multimodal serving systems, however, provide no general mechanism for such sharing. vLLM-Omni implements dedicated support for models such as BAGEL and HunyuanImage-3, whereas the publicly available version of SGLang-Omni does not yet support this class of models. Existing approaches are model-specific: they typically maintain a private tensor cache within each model and transfer KV-cache states between the autoregressive model and diffusion module through manual serialization and layer-wise injection. Confining the KV cache to a single request and process prevents efficient sharing across nodes. The absence of reuse across requests and interaction rounds further requires common prefixes to be recomputed for each inference request. Moreover, because cache-transfer logic is tightly coupled to model architecture, supporting a new multimodal model requires a dedicated KV-cache migration mechanism. These limitations motivate a model-agnostic KV-cache abstraction that supports distributed transfer and reuse across models, requests, and inference rounds.

\begin{table}[t]

-0.1in \end{table}

3 Control Flow↩︎

The Control Flow layer abstracts the “multimodal, multi-heterogeneous computing unit inference pipeline” into a declarative graph DSL, with the framework layer taking over all scheduling and data transmission. This completely removes the burden of scheduling details from the business-side computational logic. The design pursues two key goals: universality—a single set of abstractions covering any arbitrary concatenation of heterogeneous roles such as LLM [7], [38], Diffusion [9], [15], Vision [1], [2], and Audio [4]; and performance—fully asynchronous and decentralized execution flow to maximize throughput in real-time systems.

First, taking the control flow example graph (Fig. 3) in this typical multimodal inference scenario as an example, we introduce two core capabilities and three types of dependency problems at the framework level, and subsequently present the solutions corresponding to each of the three dependency types.

The execution flow of this graph is as follows: A request enters from entry, and parse_request streams out data frame by frame, with each frame independently activating downstream stages without waiting for all frames to complete. When frame \(T_0\) produces text-related data, it is immediately encoded via embedding and activates llm.generate once. When frame \(T_1\) produces image-related data, it simultaneously triggers two encoding paths, vit_encode and vae_encode; the results from both paths join and then activate llm.generate again. Upon completion, llm.generate streams out token frames, which are streamed back to entry for the user in real time on one hand, and trigger diffusion to generate an image after all tokens are produced on the other hand. The image is then decoded by vae_decode and sent back to entry.

The aforementioned process reflects two core capabilities: (1) Streaming frame-by-frame activation—parse_request independently activates downstream stages as soon as each frame is produced, without waiting for all frames to complete; (2) Runtime dynamic branch selection—routing is automatically performed based on the output type of each frame (frame \(T_0\) takes the embedding path, while frame \(T_1\) simultaneously triggers both vit_encode and vae_encode), which also serves as the foundation for the framework to support cyclic graphs.

These two core capabilities are supported and completed by static and dynamic graphs, as detailed in Sections 3.3 and 3.4.

At the same time, implementing the aforementioned process poses three core scheduling challenges to the framework:

Problem 1: Multi-path convergence of single-source data—The Downstream Synchronization Problem. In frame \(T_1\), parse_request produces image-related data, which then simultaneously traverses two independent encoding paths: vit_encode and vae_encode. Ultimately, both results must converge into llm.generate to trigger its execution. Because vit_encode and vae_encode process data at different speeds, they cannot be guaranteed to arrive at the convergence node at the same time. Consequently, llm.generate must wait until both paths are ready to be activated, rather than being triggered as soon as either single path arrives.

Problem 2: Ordered streaming outputs but potential out-of-order arrival downstream—The Arrival Order Guarantee Problem. llm.generate streams out tokens, and each text frame is logically strictly ordered. However, the path from text to entry in the graph is a return edge. After these frames undergo network transmission and asynchronous scheduling, their arrival order at entry may not align with their production order—for instance, \(t_2\) might arrive before \(t_1\). If entry streams these out-of-order tokens back to the user, the sequence of text received by the user will deviate from the generated order.

Problem 3: The number of paths is only determined at runtime—The Dynamic Synchronization Problem. To increase throughput, the framework supports segmented multi-frame production, which conversely introduces the dynamic synchronization problem: diffusion.generate_image must synchronize text embeddings, conditional image encodings (from vit_encode and vae_encode), and diffusion parameters—none of which can be missing. However, the number of conditional image paths varies per request: it is 0 for text-to-image (t2i) generation, and \(N\) for mixed text-and-image (ti2i) generation. The framework cannot determine \(N\) during the parsing phase, yet it must correctly “synchronize all \(N\) paths before triggering” at runtime, avoiding both premature triggering and perpetual waiting.

These three problems can be categorized into three distinct dependency relationships: Problem 1 is a diamond dependency (single-source data splits and later converges downstream), Problem 2 is a sequence dependency (multi-frames produced in order may arrive downstream out of order), and Problem 3 is a multi-stream join (the number of single-source data paths is dynamic, requiring all data to arrive before activation). Our solutions targeting these three problems are detailed in Section 3.2.

Omni-Flow completely encapsulates scheduling complexity within the framework layer: users declare field-level data dependencies via a Python DSL, which the framework compiles into a static execution plan during the parsing phase. At runtime, this plan is driven by decentralized node schedulers through table lookups—leaving the business-side computational logic completely free from the burden of scheduling details.

3.1 Graph Definition↩︎

Users declare graph structures via a Python DSL, which consists of four core building blocks:

• Role: A computing role (e.g., LLM, Diffusion). _interface() declares input/output fields and scheduling strategies, while _graph_call() declares subgraph invocation entry points and completion conditions.

• Placeholder: External data fields injected at the graph entry point.

• GlobalParameters: Session-granularity global parameters (e.g., KV Cache) shared across roles, with their lifecycles managed by the framework.

• RoleRef / GlobalParametersRef: Cross-graph references, supporting a single role instance to host multiple graphs.

OR-AND Input Semantics. The inputs_or of each _interface() uses a two-dimensional list: the outer list represents the OR groups (triggered as soon as any group is ready), and the inner list represents the AND groups (triggered only when all fields within the group have arrived). This structure models the branching logic in inference without the need to write any explicit conditional judgment code—the framework automatically selects the branch based on “which OR group is fully satisfied first.”

3.2 Implementation of the Three Types of Dependencies↩︎

These three types of dependencies (as illustrated in Fig. 4) frequently co-occur and overlap in real-world pipelines (see Appendix 8). If one were to hand-write the scheduling logic for each case individually, the code would become extremely fragile and difficult to reuse.

To address these three types of dependencies, Omni-Flow abstracts a Bucket model. The definition of a bucket is given below:

Definition of a Bucket: A bucket is a container used at convergence points to synchronize multi-path inputs before triggering the forward pass of the current node. If a certain OR group of a node has \(\geq 2\) direct predecessors (i.e., multiple paths converge here, satisfying a diamond dependency, termed a convergence point), the framework allocates a bucket for it; conversely, directly connected nodes (intermediate nodes in a linear chain, source nodes, or single-predecessor nodes) do not hold a bucket, and a forward pass is triggered immediately upon the arrival of each upstream frame without waiting for synchronization. That is, buckets only exist at positions where multi-path convergence is required.

Identity of a Bucket: The Quadruple. Each bucket is uniquely identified by a quadruple \((\mathit{gc\_rid},\, \mathit{owner\_node},\, \mathit{or\_group},\, \mathit{activation\_id})\):

• \(\mathit{gc\_rid}\) — The associated graph_call invocation, used for cross-request isolation;

• \(\mathit{owner\_node}\) — Identifies the node to which the bucket belongs;

• \(\mathit{or\_group}\) — The OR group of the owning node (different OR groups of the same node hold independent buckets);

• \(\mathit{activation\_id}\) — Distinguishes buckets that belong to the same OR group of the same node but correspond to different frames.

The quadruple orthogonalizes each concurrency axis where data mismatches could potentially occur; the absence of any dimension would lead to cross-bucket data leakage (bucket collision). While the first three dimensions can be finalized during the parsing phase, activation_id is recursively calculated at runtime to ensure that the previous frame and the subsequent frame fall into different buckets, whereas data from the same source and the same frame falls into the exact same bucket.

Internal State and Triggering of a Bucket. Each path arriving at a bucket is named a static slot during the parsing phase. A bucket maintains four sets: required (the complete set of slots that must be synchronized), received (slots that have received data), slot_eos (slots that have received an end-of-stream indicator), and unreachable (slots reported as unreachable by upstream). The triggering criterion for a bucket is: \[\mathit{required} \subseteq \mathit{received\_satisfied} \cup \mathit{unreachable}\] when every required path has either arrived or been confirmed unreachable, the bucket fires and schedules the current node’s forward pass.

Bucket Lifecycle: Buckets are not pre-registered—the framework lazily creates the corresponding bucket only after an upstream node sends a frame from a specific path; a bucket can be reclaimed immediately after it is triggered or determined to be unreachable.

3.2.1 Diamond Dependency↩︎

Terminology: A diamond consists of three types of nodes—a fork source (source) (e.g., parse_request), intermediate nodes (pre-sink) (e.g., vit.encode), and a convergence point (sink) (e.g., llm.generate).

Core Mechanism: Using the identity of the fork source and the frame sequence number as the seed for the aid. The framework computes an aid for each message flowing toward a convergence point by hashing a specific set of identity information. To ensure that multiple paths from the same source fall into the same bucket, the key lies in the selection of the seed: all paths uniformly use the identity of their common fork source as the seed, rather than the identities of their respective direct upstream nodes. Consequently, regardless of which intermediate nodes a path traverses or how long the path is, the seeds substituted into the hash function are completely identical, yielding a consistent aid and thus placing the messages into the same bucket; conversely, the subsequent frame corresponds to a different fork source frame sequence number, resulting in a different seed and naturally falling into a new bucket.

3.2.2 Sequence Dependency↩︎

The framework concentrates order preservation at the SessionLoadBalance (SLB) nodes, while all other nodes are entirely free from queuing. For each queuing key \(q = (\mathit{session},\, \mathit{gc},\, \mathit{interface})\), the order-preserving node maintains an expected sequence number expected and enforces two invariants: (i) it only permits the passage of activations whose sequence numbers exactly match expected; and (ii) it advances expected only after the OR group corresponding to the current sequence number has been finalized.

This convergence holds because any node with intra-session state dependencies is inherently declared as an SLB node, whereas stateless nodes such as RoundRobin or Broadcast do not require order preservation. The order-preserving information is passed segment-by-segment via the sequence numbers reassigned when exiting an SLB node, allowing intermediate ordinary nodes to execute fully out of order. This approach avoids introducing redundant serialization points while preserving end-to-end ordering.

Source of Sequence Numbers: The seq and first_seq tags are embedded within messages. The seq in Algorithm 5 represents a tag (i.e., SeqId) written by upstream nodes and propagated hop-by-hop alongside the data frames.

3.2.3 Multi-Stream Join↩︎

For this type of node, activate_by_count is declared. Upon inbound arrival, the framework forcibly rewrites the bucket identities of these signals to the same constant, causing multiple paths to converge into a single bucket. This mechanism is processed in two steps:

Step 1: Merging Buckets. Under default conditions, different OR groups and different activations of the same node fall into their respective independent buckets. Since the semantics of an activate_by_count node dictate that “a single activation collects fields from all upstream paths,” the framework forcibly assigns all inputs into the exact same bucket upon inbound arrival, allowing multi-path inputs to accumulate uniformly at the node granularity.

Step 2: Triggering via Field-Level Thresholds. The merged bucket counts each trigger field independently. A field is considered satisfied once its arrived frame count reaches the declared threshold; the forward pass is triggered only when all required fields are satisfied.

3.3 Graph Parsing and Static Plan↩︎

The graph definitions written by users via the Python DSL are declarative descriptions, which the framework converts into data structures directly utilizable at runtime upon startup. This conversion yields two direct benefits: first, configuration errors are exposed before any requests arrive (fail-fast); second, the runtime hot path degrades into pure table lookups, eliminating the need for repetitive derivation.

The Graph Parser (GraphParser) takes the user-written graph definition module as input, extracts the interface declarations and field-level data dependencies of each node through static reflection, and outputs a structured Graph. In multi-graph scenarios, it is also responsible for routing based on gid and performing cross-graph consistency validation.

The Static Plan (StaticPlan) takes the parsed Graph as input and precomputes all indices required for runtime scheduling:

• Edge Table (Edge): Field-level directed edges {source} -> {destination} that describe the data flow.

• Control Edges (CtrlEdge): Specifies which downstream nodes an upstream node should send ARRIVED, SLOT_EOS, or UNREACHABLE signals to upon completing a yield.

• Diamond Roots: Locates the common ancestor of each convergence node via a reverse BFS, utilized for calculating activation_id.

• Critical Slots: Marks the slots of a field that have a unique upstream path; when this path becomes unreachable, the bucket can be directly determined as DEAD.

3.4 Dynamic Graph↩︎

The Dynamic Graph layer is responsible for translating the results of the Static Plan into actual request execution processes, concurrently handling multiple complex scenarios such as diamond synchronization, order preservation, and unreachability propagation. Its design goal is to cover all scenarios with a minimum number of orthogonal mechanisms, keeping the processing logic of each node simple and unified.

The Core Scheduler (NodeController) serves as the execution unit for each graph node, managing the complete lifecycle of a request from inbound arrival to outbound dispatch:

Inbound Routing: It determines based on the static plan whether an incoming message should enter a bucket for accumulation (at convergence nodes) or directly trigger a forward pass (at directly connected nodes).

Outbound Dispatch: It dynamically routes frames downstream according to the CtrlEdge after each execution—if an output field is present, it sends an ARRIVED signal to activate the corresponding downstream node; if a field is absent, it sends an UNREACHABLE signal to silently skip that branch, achieving runtime dynamic branch selection. Zero-yields, exceptions, and cancellations are all transparently propagated through the UNREACHABLE path.

The Bucket State Machine is responsible for data accumulation and triggering decisions at convergence nodes. Its core logic triggers a forward pass once “all required upstream paths have either arrived or become unreachable.” Standard join buckets make this determination based on upstream source granularity, whereas activate_by_count buckets make it based on field-level count thresholds.

The SeqId Segmented Order Preservation mechanism maintains the sequence information of streaming frames. The order-preserving tag, SeqId, is generated solely by multi-frame nodes and SLB nodes, while ordinary nodes merely propagate it transparently. This concentrates the order-preservation logic at the order-generating nodes without intruding upon each intermediate node.

3.5 Other Mechanisms↩︎

This section summarizes several relatively independent mechanisms and outlines their core concepts:

Unreachability Propagation and Cancellation: When a node encounters a zero-yield, an exception, a cancellation, or a field mismatch, it sends an UNREACHABLE signal (distinguishing among empty, error, and cancel) to its ctrl_edges. If a critical slot is hit, the bucket is marked as DEAD and this state propagates recursively downstream; if a non-critical slot is hit, it is merely recorded in unreachable_inputs, and the bucket can still be triggered normally.

Task Completion Determination (ActivityTracker): The ActivityTracker maintains a global inflight set; a node registers upon activation and deregisters after its forward pass and control signal dispatches are completed. The GC is determined to have concluded as soon as the inflight set becomes empty, which subsequently triggers system_finish.

Cross-GraphCall Serialization (GCScheduler): Concurrent graph_call instances within the same session will introduce race conditions over shared GlobalParameters (such as the KV cache and history). The GCScheduler enforces a session-level FIFO queue, permitting only the head of the queue to proceed and releasing the next one only after the preceding GC has completely exited; execution across different sessions remains non-blocking. By lifting the serialization point to the invocation entry layer, the framework completely avoids deadlocks caused by the interleaving of intra-node queuing and delayed callbacks.

Concurrency Model: The Control Flow layer does not employ business-level locks; instead, serialization is derived from a “single-threaded event loop + non-branching invocation chains.” Inbound messages unfold along a continuous await chain: send \(\rightarrow\) on_inbound \(\rightarrow\) bucket accumulation \(\rightarrow\) fire \(\rightarrow\) forward, without spawning separate new tasks. This guarantees that when an upstream send returns within a process, the downstream forward has inherently finished, fundamentally eliminating race conditions from out-of-order callbacks. Under cyclic topologies, UNREACHABLE signals are deduplicated based on \((\mathit{origin},\, \mathit{node},\, \mathit{or\_group})\) to prevent infinite recursion within the loop.

Service Discovery & Load Balancing: The ServiceRegistry implements distributed registration and discovery based on Redis [39], with heartbeats maintained over an independent connection to isolate Redis jitter from the main data path. Failed instances are evicted on timeout, and routing is automatically reconstructed during scaling. Built atop this registry, LoadBalance offers three strategies: RoundRobin for stateless dispatch, and SessionLoadBalance for session-affinity routing (affinity states are persisted in Redis for cross-process sharing, which also underpins sequence-level dependency tracking).

4 Data Flow↩︎

Current multimodal models are converging toward a unified backbone architecture [16]–[18]—where the LLM handles cross-modal understanding and the Diffusion model manages generation, both sharing the exact same set of Transformer weights and KV cache. Concurrently, predictable ultra-long-context tasks [40], [41] will inevitably catalyze requirements for KV cache compression and compaction [31]–[33]: the compressed KV must remain reusable by multiple downstream roles rather than being restricted to a single consumer. Consequently, the capability for multi-role sharing of the KV cache is no longer a luxury but an architectural imperative for evolutionary advancement.

Existing KV cache sharing schemes (as shown in Fig. 6) are relatively fragmented: prefill/decode separation [22], [23], [30] relies on dedicated point-to-point transfer channels, local capacity expansion leverages L2/L3 storage [25], [42], and cross-service sharing depends on L3/L4 global caching [26], [43]. These three mechanisms operate in isolation and are difficult to compose beyond a small number of fixed roles.

To address this, we propose a unified distributed KV cache abstraction centered on a Global Params Pool (as shown in Fig. 7). Allocation and reclamation across the L1/L2/L3 storage hierarchy are consolidated under a single data plane, and all data transfers—whether between different roles or between replicas of the same role—are uniformly routed through direct data-plane channels. Unlike prefill/decode separation schemes that only open a single fixed transfer path, the Global Params Pool establishes a fully connected data fabric: any role or replica can push to or pull from any other, without any hardcoded pairing assumptions. The inference engine remains solely responsible for forward computation, completely agnostic to data locality and ownership. This consolidated design naturally scales to an arbitrary number of collaborative roles without architectural modification.

The Data Flow subsystem adopts a Client-Server architecture: the MemoryManager serves as the core server component, uniformly managing all memory pools and handling RPC requests, while the MemoryManagerClient is embedded within each Role process to communicate with the server via ZMQ [44] IPC. Memory is categorized into three major pools based on usage:

• Global Params Pool: The designated destination for the KV Cache, utilizing a paged, tiered storage hierarchy [25] (L1 GPU / L2 CPU / L3 SSD) with automated data migration between tiers based on hot/cold data access patterns.

• Tmp Buffer Pool: The transit hub for intermediate data, nested within the system to automatically accomplish zero-copy, low-latency transfers of input and output tensors across different role interfaces.

• Weight Share: The sharing layer for model weights, which performs matching based on structured, generic names to enable multiple co-located roles on the same GPU to reuse the exact same physical weights, reducing the GPU memory overhead down to that of a single replica.

4.1 Global Params Pool↩︎

Distinct from existing KV cache schemes (where the engine internally manages a radix tree, maintains point-to-point prefill/decode direct transmission channels, and independently manages local L2/L3 storage), the core divergence of the Global Params Pool lies in its holistic takeover of management:

• All KV cache slot allocations and reclamations on High Bandwidth Memory (HBM) are uniformly managed by the data plane. The inference engine is solely responsible for forward calculation, remaining completely agnostic to data location and ownership.

• The L1/L2/L3 three-tier storage hierarchy is entirely registered within a distributed metadata center. Rather than using point-to-point prefill/decode direct transmission, the sharing channel is transformed into a subscribe-and-preload mechanism—notifications are automatically pushed upon data publication, and subscribers pull data on demand.

• The cross-replica retrieval path of KV cache shards is uniformly determined by the global metadata center based on layout information and a candidate-combination algorithm. Shards can be gathered from any arbitrary source node, independent of its specific role or replica index.

The subsystem is organized across four layers: the physical paging layer \(\rightarrow\) the multi-tier slot engine \(\rightarrow\) the distributed metadata coordination layer \(\rightarrow\) the complete dataflow closed-loop.

4.1.1 Physical Paging Layer↩︎

The Physical Paging Layer is responsible for low-level storage management across the three memory tiers (L1 GPU / L2 CPU / L3 SSD) and provides the cross-tier copy primitives used by the Slot Engine during migration. Rather than imposing a monolithic single-tensor layout, it supports flexible compositional definitions: each global parameter is described by a two-level metadata spec—an outer sub_name dimension (e.g., one entry per Transformer layer) and an inner slice_name dimension (e.g., per-head KV caches [45] buffers, or any future slice type)—so that heterogeneous model architectures can be accommodated without modifying any allocation code. Upper layers retrieve any tensor view with zero-copy overhead by supplying three coordinates \((\mathit{sub\_names\_tuple},\, \mathit{page\_idx},\, \mathit{slice\_idx\_tuple})\), whose Cartesian product automatically covers all layer \(\times\) page \(\times\) slice combinations.

4.1.2 Multi-Tier Slot Engine↩︎

The Slot Engine maps pages across three tiers of physical media. Eviction is guided by a TinyLFU [46] Count-Min Sketch that estimates per-page access frequency with periodic decay, feeding a four-tier frequency-aware LRU (cold / warm1 / warm2 / hot) that always reclaims from the coldest bucket. Rather than integer reference counters, each slot maintains four Set[str] trackers (active requests, write tasks, eviction tasks, and readback tasks), guaranteeing eviction safety through set idempotency and eliminating both negative-count and memory-leak bugs. Cross-tier migration uses a custom CUDA kernel for zero-copy format conversion between the GPU-optimized NHD block-stack layout and the flat serialization layout used by CPU/SSD. Finally, concurrent misses on the same page are deduplicated via an in-flight upload dictionary: the first requester acts as owner and performs the physical I/O, while latecomers wait and reuse the result—compressing N parallel uploads into a single operation and preventing thundering-herd amplification.

4.1.3 Distributed Metadata Coordination↩︎

With Redis [39] as the single source of truth, all write operations are executed atomically via Lua scripts and reads are batched through pipelines to minimize RTT. The coordination layer is organized around four functional blocks.

Layout Identification and Cross-Node Consistency. Each node encodes its physical layout metadata as a JSON document and takes the first 16 hex characters of its SHA-256 hash as a globally unique layout ID. When a single node covers only a partial page (e.g., node A holds K, node B holds V), a candidate combination algorithm enumerates all node-layout unions that cover a complete page, sorted by combination size, occurrence frequency, then lexicographic order.

Session Prefix Matching. Given a session ID and a query page sequence, the system performs longest-prefix matching against the cached page chains in Redis, automatically truncating expired tails and verifying replica availability of candidate combinations page by page—stopping at the first page with no valid provider.

Replica Publishing and Provider Selection. After completing computation, a node registers itself as a replica provider for the corresponding pages. During queries, the provider with the highest-priority candidate combination is selected, preferring nodes that cover more pages to minimize cross-machine transmission overhead.

Atomic Eviction with Three-Layer Protection. All eviction checks are executed within a single Lua script to eliminate TOCTOU races: (1) replica reference lock—does the target page have active locks from other nodes; (2) session run lock—are there ongoing inference requests associated with this page; (3) candidate combination integrity—does each combination still have at least one complete replica after removal. The script confirms all three conditions and executes deletion atomically, leaving no window between check and action.

4.1.4 Complete Data Flow Loop↩︎

Initialization. When the upper layer creates a global parameter, the API allocates the physical paging tensor, registers the memory region with the RDMA [47] engine, establishes the L1/L2/L3 slot engine structure, and registers the layout and its candidate combinations in Redis.

Inference Write \(\rightarrow\) Publication \(\rightarrow\) Preload. Upon completing computation, a request invokes the publication interface, which maps page IDs to local slots, registers the node as a replica provider, and appends the new pages to the session’s page chain. Downstream preload is triggered automatically with batched notification delivery to reduce redundant cross-node messages.

Cross-Node Demand Pulling via RDMA. On a preload notification or local cache miss, the subscriber first searches its local multi-tier storage. On a miss, it queries Redis for remote providers and initiates an RDMA bulk read encapsulated in a strict Search–Transfer–Release lifecycle: registering search markers on local slots, executing the cross-node direct read, and releasing markers upon completion.

Copy-on-Write and Fragmented Page Handling. When incremental data is appended to a page shared by multiple active requests (reference count \(> 1\)), a Copy-on-Write is triggered automatically. For partial writes that do not fill a complete page, a dirty-key tracking mechanism records which segments have been overwritten, ensuring subsequent reads return a correctly stitched tensor view. Fragmented pages in multimodal pipelines are always retained in full, because a fragmented page may span content produced by different model roles (e.g., KV cache written by the LLM component that the Diffusion component cannot recompute), making partial eviction or truncation unsafe.

Eviction Under Memory Pressure. When local HBM is under pressure, the slot engine identifies cold-priority candidates and submits a three-layer protected eviction request to Redis. Eviction decisions never block active inference: the four Set[str] reference trackers described in §4.1.2 provide automatic mutual exclusion between eviction and inference tasks.

4.2 Tmp Buffer Pool↩︎

Intermediate tensors (activation values, encoder outputs, etc.) transferred between workers are managed via a pre-allocated slab—a contiguous, byte-addressable tensor partitioned into fixed-size chunks that are carved out on demand—avoiding operating-system-level memory allocations on a per-request basis.

Reference Counting + LRU Dual-Track Reuse. Each allocated block operates on a reference count: initializing to 1 upon allocation; incrementing by 1 when downstream consumers reuse it via zero-copy add_ref; and physically reclaiming only when free decrements the count to 0. Concurrently, an OrderedDict-structured LRU cache is embedded—borrowing existing written blocks based on cache_key; upon a cache hit, ref increments by 1 but the block remains in the cache without being popped, allowing multiple concurrent requests to borrow the same read-only data. During OOM events, the engine exclusively evicts idle cache blocks where ref==0, while active blocks remain protected by reference counts. Additionally, find_alloc_by_ptr supports an O(\(\log N\)) reverse lookup via any arbitrary byte offset to trace back to its parent allocation block, resolving the provenance problem where an externally obtained tensor slice pointer falls into the middle of a block rather than at its starting address.

A key difference from conventional schemes is the pull-on-demand paradigm for cross-host transfers rather than a push paradigm—the data producer never proactively pushes data; instead, the consumer pulls it via RDMA only when needed, completely eliminating redundant transfers and the overhead of managing receiver-side buffer zones.

4.3 Weight and KV Cache Sharing↩︎

Model Weight Sharing. By monkey-patching PyTorch’s factory functions (torch.empty, torch.zeros, etc.), all parameter allocations are redirected to the meta device during model construction, resulting in zero GPU memory footprint at build time. Once construction is complete, the framework allocates or reuses cross-process IPC shared tensors keyed by names derived from the model architecture and TP/EP topology, with the Owner process writing checkpoint data into shared GPU memory and Follower processes using a SkipCopyParameter whose copy_() is a no-op—since all processes physically point to the same memory block, redundant copies would silently corrupt each other’s data. Shared-weight initialization uses client-side polling rather than server-side condition variables: if all followers blocked on a condition.wait(), they would saturate the MemoryManager thread pool and deadlock; polling once per second on the client side entirely avoids this. The overall effect is that multi-role deployments consume only a single replica’s worth of weight memory.

KV Cache Sharing. Multiple co-located roles (e.g., LLM and diffusion model) share a unified KV cache pool whose size is set to the minimum of all roles’ proposals, preventing any single role from over-allocating. A kv_pool_barrier resolves a subtle timing hazard: roles that finish weight loading earlier observe artificially high free memory and would over-estimate the available KV pool; the barrier ensures all co-located roles complete weight loading before the pool size is evaluated.

5 Compute Flow \(\times\) SGLang↩︎

The computation plane of the Native Multimodal Model (NMM) inference framework addresses a fundamental tension: model topologies are highly fluid across modalities (image understanding, generation, speech transcription, voice dialogue each have distinct encoders/decoders), yet inference execution must remain highly efficient. The strategy is to unify all modalities into a single HiddenState representation before they enter the LLM backbone, allowing proven LLM optimization techniques—such as KV cache reuse [24], [25], shared memory I/O [48], [49], speculative decoding [50], [51], and weight quantization [52], [53]—to apply uniformly across any modality. The compute_flow layer treats SGLang [27] as a deeply customizable LLM kernel, takes over its memory and sampling through a unified interface, and forces modality encoders and diffusion models to converge onto the same forward execution path. Three core mechanisms are described below: prefix matching, interface abstraction and KV takeover, and diffusion reusing the LLM path.

5.1 Prefix Matching↩︎

Traditional LLM prefix caching operates on token ID sequences, which is insufficient for multimodal inference where outputs (generated images, audio, video) cannot be trivially re-tokenized in subsequent turns. The framework addresses this by decomposing prefix matching into two modes that share a common underlying mechanism.

Full Matching applies to scenarios without complex modal outputs. The server runs all modality encoders to obtain exact segment lengths, concatenates the resulting hidden states into a complete sequence, and performs page-by-page prefix matching. To avoid redundant encoding, each encoder maintains a cross-request LRU cache keyed by the hash of the raw input payload (image URL, base64, etc.), so a cache hit skips the encoder forward pass entirely.

Incremental Matching applies when the input references prior modal outputs whose token IDs cannot be recovered by re-parsing. The client carries a step offset declaring its already-acquired data position; the server truncates history to that position and processes only the new incremental data. A negative or absent step degrades to an append semantic that consumes all current history.

Both modes share the same underlying Chained Paged Hashing mechanism: each page’s hash encodes the hash of its predecessor, so hashes are equal only when the entire prefix matches page-by-page. This compresses radix-tree semantics into a flat hash string that is directly comparable across processes and replicas without maintaining a shared prefix tree.

The prefix reuse of the LLM is further deconstructed into two semantic layers. Layer 1—Recomputable Complete Inputs (emb, small footprint, stored on CPU): all inputs from different modalities are uniformly abstracted into hidden states. Even if all GPU KV caches are lost, the system relies on this emb to execute a fresh prefill and reconstruct the KV, guaranteeing correctness and recoverability. Layer 2—KV-Cache Matching, “Use If Available”: KV cache hits on the GPU are purely opportunistic—the system reuses however much is hit, and any unmatched portions fall back to Layer 1 for recomputation, guaranteeing performance. Decoupling these two layers means the expensive volatile KV cache is reused only best-effort, while the inexpensive reliable emb serves as a safety net.

Rather than emitting a single frame per request, the prefix matching phase yields multiple frames on demand to maximize asynchronous concurrency. If any historical prefix is unmatched, a recomputation frame is emitted first to reconstruct the KV cache independently. Each modal segment is then yielded as its own frame with its offset, length, and hidden state slice. This segmented emission enables pipelined prefill: downstream nodes begin computing each segment as soon as its frame arrives, without waiting for the full sequence. Combined with OR-group activation, preparation and computation of different segments overlap in a pipeline, transforming serial within-request waiting into across-segment asynchronous parallelism.

5.2 Interface Abstraction over SGLang and KV Takeover↩︎

The framework does not use SGLang’s native memory and sampling logic directly; instead, it hollows out the SGLang runtime via a unified plugin singleton, transferring full control to the framework’s data plane. This refactoring covers two main areas.

Abstracted I/O Parameter Interface. Each request carries precise metadata (session ID, prefix/extension/prefill lengths, max new tokens, physical slots, page IDs, etc.). At execution time, input tensors such as cached hidden states are copied into persistent GPU buffers and attached to the forward batch; on completion, generated token and hidden state slices are copied back out. Which tensors flow in and out is driven by a configurable key list, making model I/O a declarative interface rather than hardcoded logic.

Holistic KV-Cache Takeover. SGLang’s native radix-tree insertion is entirely bypassed. Prefix matching directly returns physical slots precomputed by the framework, and request completion simply publishes status back to the data plane. True KV cache ownership resides in the framework’s global parameter memory pool: slot allocation, hit/miss delimitation, post-commit publication, and the translation from business-side slot IDs to engine page IDs are all managed by the data plane, while SGLang’s attention kernel is solely responsible for writing into those slots.

5.3 Diffusion Reusing the LLM Path↩︎

A key engineering decision is that the Diffusion (DiT) [9], [10] module does not use a native diffusion runtime, but instead reuses the full LLM infrastructure—the same engine and scheduling stack, the same paged KV cache, and the same shared weight mechanism. The reuse is structured as a shared base class with runtime plugin selection: both LLM and DiT inherit from the same KV-and-weight management base, and environment switches toggle their respective implementations at runtime. DiT denoising steps are organized as sequences that execute along the standard paged attention pathway, with image and timestep data passed as extra fields on the forward batch.

The motivation is parallelism alignment. Weight and KV cache sharing requires that both roles adopt identical splitting dimensions across TP [54], EP [55], [56], and other parallel strategies [57], [58]. Native diffusion stacks lack expert parallelism support, making cross-role sharing impractical. Transplanting DiT onto the LLM pathway forces both roles into the same parallelism semantics, enabling them to share physical weights and KV cache directly.

6 Deploy with Ray↩︎

Single Replica as the Resource Unit. Resource configuration operates at the minimum granularity of a single role replica [59]—users only declare the number of GPUs and nodes per replica, and the framework automatically infers the overall topology of the cluster. Replicas are homogeneous, physical card numbers are sliced automatically, and elastic scaling is achieved within a single resource group simply by modifying the replicas parameter.

Layer Abstraction—Physical Card Resource Reuse. Roles at layer: 0 determine the Placement Group (PG) topology and physically request GPU allocations; roles at layer >= 1 declare num_gpus=0, bypassing Ray resource quotas but reusing the physical cards already allocated by layer 0 through the injection of CUDA_VISIBLE_DEVICES. This decouples Ray’s resource bookkeeping from actual physical allocations, serving as the fundamental foundation for resource reuse in multi-role, co-located deployments.

7 Conclusion and Future Work↩︎

Omni-Flow provides a unified workflow orchestration, data transmission, and key-value cache sharing solution for multimodal inference scenarios through three layers of abstraction: Control Flow, Data Flow, and Compute Flow. The framework has successfully supported three typical scenarios: DeepSeek-V2 [60] pure LLM inference (PD, streaming token output), LongCat-Next [21] multimodal dialogue (LLM with different heads), and HunyuanImage-3 [17] image generation pipeline (LLM+DiT).

The framework is currently in its early stages, and future work directions include: (1) Performance optimization is the focus of the next stage, with the core goal of evolving towards real-time inference; (2) Deepening the distributed KV Cache—supporting more attention variants [49], [61], hierarchical transmission strategies, efficient transmission under different TP configurations, CP support, intelligent selection of KV shards, etc.; (3) More complex scheduling strategies, such as cache-aware scheduling, making routing decisions based on the KV Cache hit status of each replica; (4) RL integration [62], [63]—currently, model weights have been uniformly managed by the framework, and parameter synchronization in RL training will naturally benefit from this shared infrastructure; (5) Supporting more model forms and continuously expanding the model coverage of the framework.

8 Nested Scenarios of Diamond and Sequence Dependencies↩︎

The preceding sections independently introduced the basic mechanisms of the three dependency types. However, in real-world business graphs, multiple dependency types often co-occur (as shown in Fig. 8): taking HunyuanImage-3’s image generation pipeline as an example, the fork source parse_request yields multiple frames by modality (multi-frame), each frame passes through different encoders and converges at the same join point (diamond dependency), while the entire chain is nested within SessionLoadBalance’s order-preserving link (sequence dependency). This section demonstrates how the activation_id mechanism remains valid under overlapping scenarios without requiring special-case handling for each combination—first supplementing the generalized formula and per-frame metadata needed for generalization, then presenting three progressive cases showing the behavior when diamond and sequence dependencies overlap.

General Computation of aid: Per-Edge Binary Decision. aid is computed per outgoing edge: each time a node produces a frame and sends it along an outgoing edge to downstream \((v, g)\), the framework makes a binary decision based on whether the downstream has a bucket—downstream with bucket follows Rule C, downstream without bucket follows Rule A. The criterion examines the downstream node and is determined per edge; therefore, different outgoing edges from the same node may follow different rules.

Two branches of Rule C determine whether a diamond structure, when superimposed with multiple frames, results in accumulation or binning: when \(s\) hits, the seed takes the output sequence number from a further upstream fork source, and this sequence number remains fixed for the multiple frames of the current node \(\rightarrow\) multiple frames share the same aid \(\rightarrow\) accumulation; when \(s\) misses (the current node itself is the fork source), the seed takes the yidx of the current frame \(\rightarrow\) multiple frames have different aids \(\rightarrow\) binning.

The diamond root identity needed for seeding is propagated along with messages via diamond_roots (a diamond root state table, List[DiamondRootEntry], each entry containing the source node, source aid, and source yield index): bucket-less nodes pass through the incoming state table unchanged, while diamond root nodes append their own entry upon outgoing (deduplicated by source node); Rule C then looks up \(s\) by root from this table.

Case 1: Diamond root multi-frame, intermediate node single-frame—the convergence point splits buckets by the diamond root’s frames. Each time the diamond root yields a frame, its yield index differs; Rule C computes \(K\) distinct aids, causing the convergence point to split into \(K\) buckets, each corresponding to one yield from the diamond root, independently gathering inputs and firing. When the convergence point is an SLB node, ordering must also be preserved: each yield from the diamond root corresponds exactly to one sequence number, and SLB releases the \(K\) buckets in expected order, maintaining ordering.

Case 2: Diamond root single-frame, intermediate node multi-frame—multiple frames accumulate within the same bucket in order. The intermediate node’s diamond root is further upstream and yields only one frame, so its yield index is fixed. The aid computed across the intermediate node’s \(M\) yields is entirely identical, and all \(M\) frames fall into the same bucket for accumulation; meanwhile, the intermediate node’s multiple frames must also accumulate in order within the bucket.

Case 3: Diamond root multi-frame, intermediate node also multi-frame—two layers superimposed. The outer layer splits into multiple buckets by the diamond root’s frames per Case 1; within each bucket, the corresponding intermediate node’s multiple frames accumulate per Case 2. Both layers are recursively derived from the same formula: the diamond root’s yield index determines which outer bucket a frame falls into, while the intermediate node’s multiple frames share the same diamond root yield index and thus accumulate in order within the bucket.

9 Workflow Definition Examples for LangGraph / vllm-omni / sglang-omni↩︎

LangGraph↩︎

# Nodes = Python functions,
# Edges = add_edge / add_conditional_edges,
# State = TypedDict
import operator
from typing import TypedDict, Annotated, Any
from langgraph.graph import StateGraph, START, END

class S(TypedDict):
    prompt: str; image: Any; gen_audio: bool
    messages: Annotated[list, operator.add]
    streamed: Annotated[list, operator.add]
    audio: bytes

def preprocess(s):  return s
def img_enc(s):
    return {"img_emb": f"vit({s.get('image','')})"}
def embed(s):
    return {"txt_emb": f"emb({s['prompt'][:10]})"}
def thinker(s):
    return {
        "messages": [{"role":"assistant",
            "content":f"ans:{s['prompt'][:10]}"}],
        "streamed": ["tok1","tok2","tok3"]
    }
def talker(s):
    if not s.get("gen_audio"): return {}
    return {"audio":
        f"tts({s['messages'][-1]['content'][:10]})"
        .encode()}
def tools(s):
    return {"messages":
        [{"role":"tool","content":"result"}]}

g = StateGraph(S)
g.add_node("pre", preprocess)
g.add_node("img", img_enc)
g.add_node("emb", embed)
g.add_node("think", thinker)
g.add_node("tts", talker)
g.add_node("tools", tools)

g.add_edge(START, "pre")
g.add_edge("pre", "emb")
# Dynamic branching
g.add_conditional_edges("pre",
    lambda s: ["img","emb"]
        if s.get("image") else ["emb"],)
g.add_edge("img", "think")
g.add_edge("emb", "think")
g.add_edge("think", "tools")
g.add_conditional_edges("tools",
    lambda s: "think"
        if s["messages"][-1].get("tool_calls")
        else END,
    {"think": "think", END: END})
g.add_conditional_edges("think",
    lambda s: ["tts"]
        if s.get("gen_audio") else [END],)
g.add_edge("tts", END)
graph = g.compile()

async for e in graph.astream_events(
        INPUT, version="v2"):
    if (e["event"] == "on_chain_end"
            and "streamed" in
            e["data"].get("output",{})):
        print(e["data"]["output"]["streamed"])

vllm-omni↩︎

# Nodes = frozen dataclass,
# Edges = input_sources tuple,
# Bridge = function reference

P = PipelineConfig(model_type="m", stages=(
    StagePipelineConfig(
        stage_id=0, model_stage="thinker",
        execution_type=
            StageExecutionType.LLM_AR,
        input_sources=(),
        requires_multimodal_data=True,
        engine_output_type="text",
    ),
    StagePipelineConfig(
        stage_id=1, model_stage="talker",
        execution_type=
            StageExecutionType.LLM_GENERATION,
        input_sources=(0,),
        engine_output_type="audio",
        async_chunk_process_next_stage_input_func=
            "m.thinker2talker_chunk",
        sync_process_input_func=
            "m.thinker2talker_sync",
    ),
))

def thinker2talker_sync(src, **kw):
    return [OmniTokensPrompt(
        prompt_token_ids=[0],
        additional_information={
            "text": src[0].outputs[0].text})]

sglang-omni↩︎

# Nodes = JSON StageConfig + factory functions,
# Edges = next/route_fn/wait_for/stream_to

# pipeline.json
{
  "stages": [
    {"name":"pre","factory":"s.create_pre",
     "route_fn":"n.pre_next"},
    {"name":"img","factory":"s.create_img",
     "next":"agg"},
    {"name":"agg","factory":"s.create_agg",
     "wait_for":["pre","img"],"next":"think"},
    {"name":"think","factory":"s.create_think",
     "stream_to":["tts","dec"]},
    {"name":"tts","factory":"s.create_tts",
     "terminal":true},
    {"name":"dec","factory":"s.create_dec",
     "terminal":true}
  ]
}

# next_stage.py
def pre_next(rid, out):
    s = load_state(out)
    st = ["agg"]
    if s.has_image(): st.insert(0, "img")
    if s.has_audio(): st.insert(0, "aud")
    return st

# stages.py
def create_think(model_path):
    eng = create_sglang_ar_engine(...)
    def _stream(payload, token):
        return {"token_id": int(token)}
    return EngineExecutor(eng,
        stream_builder=_stream)

10 With AI Coding↩︎

Omni-Flow was designed in late 2025 with development starting in early 2026, totaling roughly 1–2 person-half-years of engineering effort, the vast majority of which was produced with AI assistance. The iteration process roughly fell into three phases: (i) humans design the architecture, AI handles local fill-in—early AI was not yet capable of global design, so architects drove the structure while AI filled in method implementations and boilerplate; (ii) AI fixes bugs, humans supervise—once the framework took shape, high-volume bug fixes and edge-case handling were delegated to AI, with humans reviewing correctness and consistency; (iii) humans provide design documents, AI refactors the framework—as AI reasoning improved and the design converged, AI progressively took ownership of refactoring, including the full rewrite of Control Flow v1\(\rightarrow\)v2 driven by detailed design documents.

Several recurring challenges emerged and shaped our practices: (1) AI tends toward local patching, accumulating framework band-aids—we mandate that every fix be evaluated for whether it addresses the root cause, rolling back mere relocations; (2) readability of AI-generated code continuously degrades—we require persistent human oversight, reading through each completed module to confirm alignment with the overall design; (3) correctness control under parallel AI submissions—AI produces code too fast for manual diff review, so every module requires unit tests and all new logic must come with corresponding tests, making test coverage the only reliable guardrail; (4) AI diverges on larger modules—we first converge requirements into a document specifying interface definitions, core logic, and boundary conditions, then have AI execute strictly against it.

References↩︎