1 Introduction↩︎

With the development of large language models (LLMs), agent systems built upon them have seen rapid adoption in a wide range of applications [1]–[3]. Building such systems usually involves various design aspects, including how different modules are composed, how the system workflow is executed, and how runtime search strategies are applied. These aspects are often closely related but serve for different purposes. It is useful to examine and modify them separately, since different applications and experiments may require different agent components, workflows, and search strategies [4]–[6].

Several existing frameworks and libraries have been proposed to support the development of LLM-based agent systems. For example, frameworks such as CAMEL [7], MetaGPT [8], and AutoGen [9] provide abstractions for multi-agent collaboration; libraries such as LangGraph [10] and Microsoft Agent Framework [11] support graph-based workflow orchestration; and runtime search strategies have been explored in research works such as tree-of-thought [12] and tree-search-based agents [13]. These systems have provided useful tools for building agent applications. However, many of them focus on specific aspects of agent system construction, and the boundaries between component composition, workflow control, and runtime search are often not made explicit. This makes it less convenient to conduct holistic agent evaluation under different workflows and search strategies, or analyze the efficiency of different agent designs [14]. In addition, some feature-rich frameworks introduce relatively large codebases and complex abstractions, while lightweight libraries such as smolagents [15] and LightAgent [16] mainly focus on simplifying agent construction and do not provide full support for complex workflow control, runtime search, and efficiency analysis.

In this work, we present MyAG (My AGent), a lightweight software framework for building and analyzing LLM-based agent systems. MyAG introduces graph-based abstractions that separate component composition, workflow control, and runtime search. Specifically, the component graph describes agents, environments, and system modules; the workflow graph specifies how these components are executed; and the search graph records the runtime execution and search process. This design allows users to keep the same set of components while being able to switch among different workflows or search strategies in a flexible way.

Building on these graph abstractions, we further support flexible composition of agent systems. We introduce a special type of node, called a system node, as the main container of the component, workflow, and search graphs. Since a system can itself be used as a component inside another system, we support hierarchical construction of agent systems and encourages modular reuse. In addition, our framework supports agent efficiency analysis by recording different types of execution costs, including LLM-side costs, such as model calls and token usage, as well as environment-side costs, such as environment actions and action latencies. These measurements allow users to evaluate different workflows and search strategies in terms of both task performance and execution efficiency [17].

The main highlights of our proposed framework are summarized as follows:

• We provide graph-based abstractions by separating component composition, workflow control, and runtime search. (§2.1)

• We support composable and hierarchical construction of agent systems. (§2.2)

• We provide efficiency-aware analysis tools, enabling comprehensive comparison of different workflows and search strategies. (§2.3)

• We provide monitoring and visualization tools for inspecting agent execution. (§5)

2 Framework↩︎

Figure 1 provides an overview of our framework. We focus on three design aspects: graph-based abstractions for component specification, workflow control, and runtime search; hierarchical composition of agent systems; and efficiency analysis based on execution records. Table 1 compares our framework with representative open-source LLM agent frameworks in terms of implementation size and supported features. In the following sub-sections, we will describe the details of these designs.

2.1 Graph-Based Abstractions↩︎

We use three graphs to represent different structural aspects of an agent system: the component graph, the workflow graph, and the search graph. The component graph specifies the available components in the system, such as agents, environments, and tools. The workflow graph indicates how these components are executed at runtime, including control flow and stopping conditions. The search graph records the detailed runtime execution process, such as states, branches, and trajectories. Analogous to a computer program, these three graphs can be viewed as describing variable declarations, program logic, and runtime states, respectively. In this sub-section, we use a web agent system as a typical running example to illustrate how these graphs are specified and used.

2.1.0.1 Component Graph.

The component graph specifies what components an agent system contains. It mainly includes agent nodes, which represent LLM-based decision-making modules, and environment nodes, which represent non-LLM environments or tools that the agents interact with. It can also contain system nodes, which wrap another agent system as a reusable component; we discuss this hierarchical composition in §2.2. An edge in the component graph indicates that one component can access or interact with another component. For example, an agent node may call functions exposed by an environment node, or one agent node can read the memory or output of another agent node. Figure 2 (left) shows the component graph of a typical web agent system. In this example, both the “Action” node and the “Finish” node can access the web browser environment, while “Finish” can also read the execution history of “Action” to produce the final formatted answer.

2.1.0.2 Workflow Graph.

The workflow graph specifies how an agent system is executed to complete a target task. A common agent workflow follows the ReAct-style pattern [18], which alternates between action decision and action execution. However, more flexible agent systems, especially multi-agent systems, often require more complex execution structures, such as branching, loops, and different stopping conditions. We represent such execution logic with a workflow graph. Each workflow graph contains a start node and an end node. During execution, we maintains a workflow-level variable map that stores intermediate information shared across workflow nodes. A workflow node is usually associated with an agent node in the component graph. When such a workflow node is reached, the corresponding agent node is executed with its own input preparation and output parsing for the LLM call. After the execution, we determine the next workflow node according to the conditions specified on the current node’s outgoing edges. Figure 2 (right) shows the workflow graph of the web agent. After the “Action” node is executed, its provides an output variable “has_result”, which decides whether the next step returns to action decision or proceeds to the “Finish” node.

2.1.0.3 Search Graph.

The search graph records the runtime execution process and supports the exploration of alternative execution paths. Many agent systems follow a greedy strategy, where the system proceeds along a single trajectory. However, prior work has shown that exploring alternative paths can be beneficial for complex tasks [12], [13], [19]. We support different search strategies by storing runtime states as nodes in the search graph and allowing execution to resume from a selected node through state restoration. This design makes it possible to implement different search strategies such as greedy search, beam search, or best-first search under the same component and workflow definitions. Some search strategies require additional support from the workflow and component graphs. For example, best-first search requires a scoring function for selecting the next node to expand. This scorer can be typically implemented as a specialized scoring node in the workflow graph, together with a corresponding scoring agent in the component graph. Typically, the scoring agent can use LLM-as-a-judge methods [20] to evaluate candidate nodes and provide scores to guide the search.

Overall, the three-graph abstractions allow the decoupling of what components are available, how they are executed, and how runtime trajectories are explored. This separation makes it easier to reuse the same agent components under different workflows and search strategies, and provides a basis for systematic comparison and evaluation.

2.2 Hierarchical Composition↩︎

Many agent tasks can be naturally decomposed into sub-tasks that require different tools, environments, or execution strategies. For example, consider a long-horizon task that asks an agent to find a paper on the Internet and then analyze its content. One sub-task focuses on locating and downloading the paper through a web environment, while another sub-task focuses on reading and analyzing the downloaded file. Although a single agent system may handle both sub-tasks, a more modular design is to build a web sub-system and a file sub-system separately, and compose them in a higher-level system. These sub-systems may share the same underlying LLM, but they can use different tools, environments, and workflows.

In our framework, we support such hierarchical composition through system nodes. A system node is a special node in the component graph that wraps another complete agent system. The node “SYS” in Figure 3 shows an example of a system node. Each system, including both the top-level system and any sub-system, contains its own component graph, workflow graph, and search graph. Therefore, a sub-system is not only a tool wrapper, but can also maintain its own internal components and execution logic. This allows users to package a group of components and workflows as a reusable module, while still retaining fine-grained control over how each system is executed.

To allow a top-level system to use its sub-system, we expose the sub-system through function-call interfaces. We provide two modes for managing the state of a sub-system across calls. In the stateless mode, the sub-system is reset before each call and behaves similarly to a tool. This is useful when the sub-system is expected to solve an independent sub-task given the current input. In the stateful mode, the sub-system keeps its own memory and runtime state across calls, allowing it to cooperate with the top-level system in a more persistent multi-agent manner. These two modes provide a simple way to reuse sub-systems under different scenarios.

2.3 Efficiency Analysis↩︎

Efficiency is an important consideration for LLM-based agent systems, especially when agents require many model calls or interact with slow external environments. However, compared with task performance, execution cost is often less systematically analyzed, and there is not yet a single widely adopted protocol for measuring agent efficiency [17], [21]. In practice, agent efficiency can depend on multiple factors, including the number of LLM calls, token usage of each call, and action latencies. We therefore provide execution-level tools to record and analyze these costs.

In our framework, we mainly consider two types of costs: the LLM inference cost (\(C_L\)), and the environmental interaction cost (\(C_E\)). Given an agent trajectory \(\tau = (s_0, a_0, s_1, a_1, \dots, s_T)\), these costs can be computed from the execution records stored during runtime.

The LLM inference cost \(C_L\) measures the cost of the agent’s internal decision process. We define it as the weighted sum of input and output tokens used by LLM calls along the trajectory: \(C_L = \sum_{t=0}^{T-1} (\alpha \cdot L_{t}^{\mathrm{in}} + \beta \cdot L_{t}^{\mathrm{out}})\), where \(L_t^{\mathrm{in}}\) and \(L_t^{\mathrm{out}}\) denote the number of input and output tokens at step \(t\), respectively. The coefficients \(\alpha\) and \(\beta\) are weighting factors for input and output tokens. They can be set according to API prices, empirical inference latency, or other user-defined cost models. This makes the metric adaptable to different model providers and deployment settings.

The environmental interaction cost \(C_E\) measures the cost of executing actions in external environments. This cost is important because environment actions may have different latencies or resource requirements. For example, in a web environment, scrolling a page is usually cheaper than loading a new page. We define it as: \(C_E = \sum_{t=0}^{T-1} w(s_t, a_t)\), where \(w(s_t, a_t)\) is a user-defined or empirically measured cost function for taking action \(a_t\) at state \(s_t\). In our implementation, this cost is instantiated by empirically measured action latency.

These two costs enable the analysis of agent executions from both the model side and the environment side. Together with task performance, these measurements support performance-efficiency trade-off analysis across different workflows and search strategies.

3 Evaluation↩︎

3.1 Settings↩︎

3.1.0.1 Setup

We evaluate our framework on two representative agent tasks: complex question answer, using the text subset of the GAIA benchmark [22], and web navigation, using Mind2Web-Live [23]. In our main experiments, we use Qwen3.5-27B as the primary underlying LLM and implement the automatic web browser environment with Playwright.² Each experiment is run three times and the averaged performance is reported, which could provide more stable results.

3.1.0.2 Comparisons

We compare several workflow and search strategies supported by our framework: 1) The Greedy strategy follows a ReAct-style execution pattern [18], where the agent perform one action at each step based on the latest state; 2) The Best-first strategy maintains a search tree and expands the highest-scored node at each step [13]; 3) The Rollback strategy allows the agent to decide whether to return to a previous state before taking the next action [24]; and 4) The Best-of-N strategy performs multiple parallel runs and uses an additional LLM judge to select the best trajectory [25]. Unless otherwise specified, we set the maximum step budget to 36, and we also vary this budget in our evaluation to study how different strategies scale with test-time computation.

3.2 Results↩︎

The main results are shown in Figure 4 and 5, where different strategies show different patterns. Overall, the simplest Greedy strategy is the most efficient in terms of computation usage, since it follows a single execution trajectory without additional scoring, rollback, or selection steps. The Rollback strategy obtains the best overall performance when sufficient budget is available, but it is slightly less efficient than Greedy due to the additional rollback decision and the more complex prompt. The Best-first and Best-of-N strategies can improve performance with more computation, but they also introduce larger overheads. To be noted, the effectiveness of some complex strategies depends on the quality of their verification modules, such as the final selector in Best-of-N or the state-value scorer in Best-first. In our experiments, we use the same policy LLM with specialized prompts for these modules, while stronger verification models or task-specific scorers may further improve these strategies. Our framework makes such comparisons straightforward, since different workflow and search strategies can be implemented and evaluated under the same framework.

3.3 Analysis↩︎

3.3.0.1 Component Cost

We first break down the overall latency by component for different strategies, and the results are shown in Table 2. Best-first spends a much larger portion of time on environment resetting, reflecting its frequent context switching. Although Rollback introduces an additional module for rollback decisions, this module is not triggered frequently and therefore does not introduce substantial overhead. Best-of-N spends more time on the Finish module, since multiple parallel trajectories are executed and then compared before producing the final answer.

3.3.0.2 Action Cost

We further break down the latency cost of environment actions, and the results are shown in Table 3. The main motivation for recording action-level costs is that different environment actions can have substantially different latency profiles. This difference is clearly reflected in our analysis. For example, the “scroll” action is much cheaper than other actions: although it accounts for nearly 30% of action calls, it contributes only around 11% of the total environment interaction time. In contrast, the “wait” action is the most expensive action, which explains why its time percentage is much higher than its call percentage.

3.3.0.3 Underlying LLMs

Finally, we evaluate different underlying LLMs to study how model capability affects agent performance and efficiency. We use models from the Qwen3.5 family and vary the model size. Results are shown in Appendix 6.

4 Conclusion↩︎

We presented MyAG, a lightweight graph-based framework for designing and analyzing composable LLM agent systems. It separates agent system construction into component graphs, workflow graphs, and search graphs, making it easier to reuse components under different execution and search strategies. It also supports hierarchical composition through system nodes and provides monitoring and analysis tools for inspecting agent execution. Experiments and evaluations show that our framework can support flexible system design and help analyze performance-efficiency tradeoffs.

Limitations↩︎

MyAG is designed as a lightweight, research-oriented framework rather than a full production platform. Therefore, it does not currently focus on production-level features such as distributed deployment, access control, or fault tolerance. In addition, the efficiency metrics used should be viewed as configurable analysis tools rather than standardized evaluation protocols, since the cost weights may depend on many factors, such as model providers and the hardware. Finally, our experiments aim to demonstrate the capability of the framework through representative agent applications. Broader evaluation across more environments and extensions to additional application scenarios are left to future work.

5 Monitoring and Visualization Tools↩︎

We provide monitoring and visualization tools for inspecting agent execution. The dashboard-style interface shows the step-by-step running process, including workflow transitions, agent calls, environment actions, intermediate outputs, and runtime metrics. This allows users to better understand agent behavior and analyze the trajectory of agent execution.

The dashboard interface is shown in Figure 7. On the left column, we display the system execution trace, which is arranged in tree structures, where each node represents a key execution step, such as an LLM call or an environmental action call. This hierarchical tree structure also supports navigation across nested systems, making it easier to inspect the execution traces of hierarchically composed agent systems. On the right panel, we show the detailed information of the selected execution node, which can be chosen by clicking a node in the trace tree. The information includes the running time information, the visualization of the component graph, the workflow graph, and the search graph of the current agent system. In addition, the workflow-level global variables and node-level local variables are shown below the graph visualizations. The dashboard also supports cost visualization, including pie charts for latency distribution and token usage.

Overall, this toolkit provides a practical interface for monitoring and analyzing agent execution. It allows users to inspect both high-level execution traces and fine-grained runtime details, which is useful for debugging agent behavior and understanding the characteristics of different strategies.

6 Extra Results↩︎

In the main experiments, we use Qwen3.5-27B as the underlying LLM and compare different workflow and search strategies. In this analysis, we instead vary the underlying LLM to investigate how model capability affects agent performance and efficiency. We consider models from the Qwen3.5 family³ with different sizes, including Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-27B, Qwen3.5-35B-A3B, and Qwen3.5-122B-A10B. We evaluate these models on GAIA-text using the simple Greedy strategy.

The results are shown in Figure 8. Smaller dense models have lower per-call computation but may require more interaction steps or fail more frequently, while larger models can provide stronger task performance at higher resource requirements. Directly comparing token usage across models does not fully reflect inference cost, since the cost of processing each token varies with model size and architecture. We therefore additionally consider a simplified cost model in which the per-token inference cost is assumed to be proportional to the number of activated parameters. Figure 6 presents the results under this relative cost estimate.

Under this cost model, the mixture-of-experts (MoE) models provide better performance-efficiency tradeoffs, since only a subset of their parameters is activated for each token. Nevertheless, this estimate does not account for all deployment costs. In particular, such models may still require substantial GPU memory to store the full model parameters, which is not captured by the activated-parameter-based cost model.

Overall, these results suggest that the choice of the underlying LLM can substantially affect the performance-efficiency tradeoff of an agent system. There is no single model that is optimal under all deployment constraints: the best choice depends on the available inference budget, memory capacity, latency requirements, and target performance. This analysis also demonstrates how our framework can combine execution records with configurable cost models to compare agent systems across different underlying LLMs.

References↩︎

---

1. Our code is publicly available at https://github.com/zzsfornlp/MyAG, and a demo video is provided at https://github.com/zzsfornlp/MyAG/blob/main/demo.mp4.↩︎

2. https://playwright.dev/python/↩︎

3. https://huggingface.co/collections/Qwen/qwen35↩︎