1 Introduction↩︎

Recent advancements in artificial intelligence have been significantly influenced by the development of LLMs such as GPT-4 [1] and DeepSeek-V3 [2]. These models, renowned for their ability to process and generate human-like text, have catalyzed innovations across various domains, including natural language understanding, generation, and reasoning. One promising application is the creation of LLM-based agents, which use natural language to autonomously interact with users and perform a wide range of tasks. Unlike traditional rule-based systems, LLM-based agents exhibit the flexibility to understand complex tasks and generalize to novel scenarios, enhancing human-computer interaction.

A notable development in this area is the emergence of LLM-based agents capable of operating graphical user interfaces (GUIs). Unlike Robotic Process Automation (RPA), which relies on predefined rules, these agents can engage with GUIs dynamically, mimicking human-like interactions by outputting low-level actions such as clicks, drags, and scrolls. This approach not only increases operational flexibility but also enables the execution of tasks that require adaptability and reasoning, without needing access to backend systems or APIs.

To achieve optimal performance, various mechanisms have been introduced to guide the LLMs in generating accurate and contextually appropriate actions. Techniques such as reflection[3] and chain-of-thought[4] reasoning have been employed to help the model carefully consider the implications of each action before execution. While these methods can significantly enhance the agent’s decision-making abilities, they often come at the cost of efficiency, particularly for tasks that do not require advanced reasoning. For example, as illustrated in Figure [fig:motivation95example], during simple tasks like search operations, the agent may need to reason each step, such as clicking the search box, typing text, and pressing submit, leading to unnecessary delays. While traditional RPA methods can execute these fixed steps rapidly, they lack the flexibility to handle tasks requiring intelligent judgment. This highlights the need for a more efficient, adaptable approach to automate routine tasks.

In response to this challenge, we propose a novel, evolving framework for GUI agents that aims to enhance both the efficiency and intelligence of the agent’s behavior.

Our approach enables the agent to learn from their past interactions and dynamically evolve more abstract, high-level actions, eliminating the need for repetitive low-level operations. Specifically, the agent will analyze its execution history to identify patterns in repetitive, low-intelligence actions, such as those involved in routine tasks. From this analysis, the agent can generate a high-level action, encapsulating a series of low-level actions, enabling it to perform tasks more efficiently. For example, in a search operation, the agent would automatically evolve a "search" action that directly executes the required sequence, improving both speed and accuracy.

Intuitively, our framework enables the agent to focus its reasoning capabilities on tasks requiring more intelligent judgments, while simplifying repetitive tasks into a more compact form. To support this approach, we design a knowledge base structured as a chain to record task execution history and facilitate the abstraction and evolution of behaviors. This knowledge base allows the agent to continuously improve its task execution strategies, further optimizing its performance and enabling the evolution of more intelligent, high-level actions over time.

Our approach relies entirely on visual information, eliminating the need for backend access or APIs. Extensive experiments show that it outperforms baseline and state-of-the-art (SOTA) methods in both efficiency and accuracy across several benchmark tasks.

The main contributions of this paper are summarized as follows:

• We propose an evolutionary mechanism that enables a GUI Agent to learn from its task execution history and improve its efficiency by abstracting repetitive operations.

• We design a chain-based framework for recording and optimizing the agent’s execution behavior.

• Our code will be open-sourced to facilitate further research in this area.

2 Related Works↩︎

Large language models. Recent advancements in large language models (LLMs) have significantly expanded the scope of AI-driven automation. Models such as GPT-4 [1] and DeepSeek-V3 [2] exhibit strong natural language understanding and reasoning abilities. These capabilities allow LLMs to process complex UI structures and facilitate interactive decision-making, forming the foundation for LLM-driven GUI Agents [5]. Unlike traditional script-based or rule-based approaches [6], [7], LLM-powered agents can generalize across diverse applications and dynamic interfaces without explicitly predefined rules. However, challenges remain in model efficiency, adaptability, and spatial reasoning, necessitating further optimization in both architectural design and training methodologies.

LLMs as Agents. LLMs have significantly advanced intelligent agents, enabling complex task execution. AutoGPT[8], AFlow[9], MetaGPT[10], and AutoAgent[11] exemplify autonomous task decomposition and execution, while Stanford Smallville[12] and Agent Hospital[13] showcase multi-agent simulations. LLM-driven multimodal agents also enhance perception and decision-making across domains. GPT-Driver[14] enables adaptive motion planning for autonomous driving, SmartPlay[15] improves agent intelligence in gaming, and MP5[16] integrates active perception for efficient task execution in robotics.

LLMs as GUI Agents. Beyond general agents, LLMs have also enhanced GUI automation, surpassing traditional script-based methods in flexibility and adaptability. WebVoyager[17], AppAgent[18], and MobileAgent[19] leverage multimodal perception for interactive interfaces, while UFO[20], AutoGLM[21], and MMAC-Copilot[22] improve cross-platform adaptability and multi-agent collaboration. To refine UI understanding, OmniParser[23] and Ferret-UI[24] enhance element recognition, while TinyClick[25] and CogAgent[26] improve interaction precision and vision-based task execution. Additionally, WebGUM[27], MobileVLMS[28], and DigiRL[29] optimize performance in dynamic web and mobile environments. However, most existing agents rely on static training data and lack continuous adaptation. To address this limitation, we propose AppAgentX, integrating task trajectory memory to enhance efficiency and adaptability in GUI automation.

3 Preliminary↩︎

Before delving into our proposed methodology, we first introduce a baseline method for LLM-based GUI agents. This baseline serves as a foundation for understanding the core components and tasks involved in enabling LLMs to control smartphones.

The process of utilizing LLMs for smartphone control involves two key stages: screen perception and action execution. The screen perception phase begins with capturing a screenshot of the device’s current interface. In order to accurately interpret this screenshot, we employ OmniParser [23] to detect and label all interactive elements within the interface, such as buttons and text boxes. OmniParser annotates these elements with tagged bounding boxes, which are subsequently overlaid onto the original screenshot for clear visualization. Following this, the annotated screenshot is passed to the LLM for action planning. At this stage, the LLM interprets the UI components and generates corresponding actions based on its understanding of the interface.

In the second stage, action execution, we follow AppAgent [18] to define a set of low-level actions that the agent can perform within the smartphone environment. These actions include common gestures such as tapping, long-pressing, swiping, text input, and navigating back. These actions collectively define a basic, app-agnostic action space to simulate typical human interactions with a smartphone interface. Formally, the low-level action space is defined as follows: \[\label{eq:low95level95action95space} \mathcal{A}_{\text{basic}} = \{a_{\text{tap}}, a_{\text{long\_press}}, a_{\text{swipe}}, a_{\text{text}}, a_{\text{back}}\},\tag{1}\] where \(\mathcal{A}_{\text{basic}}\) represents the set of atomic actions available to the agent.

The LLM employs a structured process of observation, reasoning, and function-calling to interact with the smartphone interface. In this process, the LLM iteratively analyzes the current UI, reasons about the appropriate next steps to achieve the desired task, and invokes the corresponding actions from the defined action space. This cycle continues until the task is completed successfully. An illustration of this process is shown in Figure 1.

4 Methodology↩︎

This section outlines the core methodology of the proposed evolutionary framework. The framework comprises three main components: a memory mechanism for recording the agent’s operational history, an evolutionary mechanism for improving performance, and an execution strategy for utilizing the evolved agent. We will detail each of these elements in the following subsections.

4.1 Memory mechanism↩︎

To support the agent’s evolution from inefficient to efficient operational modes, it is essential for the agent to retain a record of its past actions and the corresponding outcomes. This enables the agent to learn from its experiences and improve future interactions. To achieve this, we propose a memory mechanism designed to capture and store the agent’s trajectory during its interactions with the environment.

The agent’s interaction with the UI is modeled as a series of page transitions, where each UI page is represented as a “page node”. Interactions performed on these pages, such as button clicks or text input, lead to transitions between these nodes. Each page node is characterized by several key attributes, including:

Page Description: This attribute stores a text description of the entire UI page, initially setting it to empty at the beginning.

Element List: This property holds a JSON list containing information of all elements detected by OmniParser[23], such as the relative screen position, OCR results, etc.

Other Properties: The page nodes also include other properties required for logging, such as page screenshots, ID, timestamps, etc.

For more detailed information, we introduce “element nodes”. Each element node corresponds to a specific UI element, such as a button, text field, or icon. The interactions with these UI elements lead to the page transitions. Similarly, each element node encapsulates essential attributes:

Element Description: This attribute records a textual description of the element’s functionality, providing a semantic understanding of the element’s purpose within the UI. Similar to page descriptions, this attribute is initialized as empty.

Element Visual Embeddings: This property stores the identifier of the element’s screenshot in the vector database. The visual features are extracted using a pre-trained ResNet50 [30] model.

Interaction Details:This property includes information related to the basic action for the current element, e.g., tapping, along with the corresponding arguments and other relevant interaction details.

This structure enables the agent to record and learn from both the high-level transitions (from one page to another) and the low-level interactions (with individual UI elements) that lead to those transitions. Figure 2 (c) illustrates this process.

Extracting Features and Descriptions. We next leverage the LLM to generate functional descriptions of pages and individual elements based on observed action sequences. Specifically, we follow the approach in [18] by decomposing the agent’s trajectory into multiple overlapping triples. Each triple consists of a source page, an action performed on an element, and a target page. These triples capture the changes in page states before and after an action is executed.

The resulting triples are then passed to the LLM for reasoning. The model generates detailed descriptions and functionalities for both the page and element nodes based on the context of the action. This process, illustrated in Figure 2 (a), enables the system to build accurate and contextually aware descriptions.

Merging Overlapping Descriptions. Since the descriptions of page nodes are generated from multiple overlapping action triples, it is likely that the descriptions for the same page node will be generated twice, based on different contexts. Therefore, it is necessary to merge them to generate a unified description for each page node. To achieve this, we instruct the LLM to combine the descriptions generated from different triples, taking into account both the specific context of each individual action and the broader global task the agent is performing. This allows the LLM to generate a more enriched and complete description of the page, considering its function in the overall task. The merged descriptions provide a detailed, unified record of the agent’s interactions with the UI, contributing to a coherent chain of node attributes that document the agent’s progress as it completes the task.

4.2 Evolutionary Mechanism↩︎

The primary goal of the evolutionary mechanism is to improve the agent’s accuracy and efficiency in executing similar tasks by learning from its previous execution histories. In typical task execution scenarios, the action sequence may contain repetitive patterns that do not require deep reasoning at each step. For example, to search for a content item, the agent may repeatedly perform actions such as tapping the search box, inputting text, and tapping the search button. By identifying and exploiting these repetitive patterns, the agent can replace inefficient action sequences with higher-level actions that streamline the process, thereby improving overall task efficiency and accuracy. To achieve this, we introduce the concept of a shortcut node, which represents a high-level action that bypasses previously inefficient action sequences. The shortcut node is designed to streamline its decision-making process, eliminating unnecessary reasoning process and accelerating task completion.

To identify whether an action sequence contains repetitive patterns that can be replaced by shortcut nodes, we design mechanisms that evaluate the trajectory of each task. Initially, LLM, drawing from prior knowledge, is tasked with determining whether a given task contains repetitive patterns that may be optimized through the introduction of shortcut nodes. If the task is deemed to contain such patterns, the next step involves inspecting the actual trajectory data. The trajectory data, which includes the descriptions of all relevant pages and elements, is input into the LLM. Upon receiving this data, the model is prompted to generate the description of a shortcut node, specifying the scenarios in which the shortcut node can be invoked and the specific low-level action sequences that it is intended to replace. This process effectively constructs an evolved action space by integrating new, higher-level actions into the agent’s operational repertoire, allowing it to perform tasks more efficiently.

To formally define the construction of high-level actions and the expansion of the action space, we introduce the following representations. A high-level action \(\tilde{a}\) is composed by abstracting a sequence of low-level actions from the basic action set \(\mathcal{A}_{\mathrm{basic}}\). This abstraction allows the agent to perform more complex tasks with fewer steps, leveraging previously learned patterns of interaction. We then define the expanded action space \(\mathcal{A}_{\mathrm{evolved}}\), which incorporates both the original low-level actions and the newly introduced high-level actions: \[\label{eq:expand95action95space} \mathcal{A}_{\mathrm{evolved}} = \mathcal{A}_{\mathrm{basic}} \cup \{\tilde{a}\}. \tag{2}\]

Here, \(\mathcal{A}_{\mathrm{evolved}}\) represents the enriched action space that includes both basic, low-level actions and the newly composed high-level actions. This expanded action space enables the agent to perform tasks more efficiently by utilizing higher-level abstractions that reduce the need for repetitive action sequences.

4.3 Action Execution↩︎

Following the evolution of the action space, the agent can now select high-level actions from the expanded action space to efficiently execute tasks. The execution process begins after the agent captures a screenshot of the current page. At this stage, the system then matches the parsed elements on the page to the stored element nodes in memory, by comparing their visual embeddings. Next, we check whether these identified element nodes are associated with any shortcut nodes. If an association between the element nodes and shortcut nodes exists, the system leverages the LLM to determine whether the corresponding high-level actions can be executed. This decision is made by inspecting the description of the shortcut node in conjunction with the current task context. If the conditions for executing the high-level action are met, the LLM generates an action execution template. This template includes the sequence of low-level actions to be executed by the shortcut node, along with the corresponding function arguments necessary for each action. In cases where multiple elements on the page are associated with shortcut nodes, the system prioritizes executing the actions based on the order of matching elements. This order is determined by their execution sequence, ensuring that the actions are performed in a logical and efficient manner. If the conditions for executing a high-level action are not met, the agent will default to selecting actions from the basic action space \(\mathcal{A}\). This ensures that the agent can still operate effectively, even when high-level abstractions cannot be applied to the current task.

During the execution of high-level paths, operations that would traditionally require multiple reasoning steps by the LLM are transformed into a page-matching and retrieval-based process. This shift significantly enhances the overall execution efficiency, as the agent can bypass repeated reasoning processes and rely on pre-determined, efficient action sequences.

5 Experiments↩︎

In this section, we will present our evaluation of the evolutionary framework through a combination of various experiments on multiple benchmarks.

5.1 Experimental Setup↩︎

Evaluation metrics. To ensure a fair and accurate comparison of our proposed method with baseline models and existing works, we adopt several evaluation metrics, which have been commonly used in prior research [18], [19], [31]. The metrics we report and compare are as follows:

Average Steps per Task (Steps): This measures the average number of operations the agent performs to complete a task.

Average Overall Success Rate (SR): This metric evaluates the proportion of tasks completed successfully by the agent.

Average Task Time (Task Time): The total time taken to complete a task, starting from the initiation of the task execution process to the determination of task completion by the agent.

Average Step Time (Step Time): The average time consumed per operation (step) during task execution.

Average LLM Token Consumption (Tokens): The total number of tokens used by the language model (LLM) during the task execution, including both prompt tokens and completion tokens.

For time-related comparisons, we focus only on tasks that are successfully executed by all methods. This ensures that the comparisons are not skewed by failures due to early terminations or excessive retry attempts, which might distort the results. To calculate token consumption, we compute the total number of prompt and completion tokens used by the LLM for each task and report the average number across all tasks. To mitigate the impact of random fluctuations, each task is repeated five times by default, and we report the averaged results across these repetitions. This helps ensure that our findings are statistically robust and not affected by outlier performance from individual task executions.

Benchmarks. We evaluate the performance of our method on several widely used benchmarks to validate its effectiveness and generalizability. These benchmarks cover a diverse range of tasks and applications, providing a comprehensive assessment of our framework’s capabilities. The benchmarks used in our experiments include:

AppAgent Benchmark [18]: This benchmark consists of 50 tasks across 9 different applications, including 45 general tasks and 5 long-duration tasks.

DroidTask [32]: Comprising 158 high-level tasks derived from 13 widely used mobile applications.

Mobile-Bench [33]: This benchmark offers a larger and more comprehensive set of task records, spanning multiple applications across 832 data entries. It provides a more extensive evaluation of the agent’s performance, facilitating comparisons with a broader range of existing methods in mobile task management.

Implementation details. The implementation of our framework leverages several key platforms. For the foundational LLM, we selected GPT-4o [1] as the default model unless otherwise stated. The LangGraph framework [34] is used as the agent platform, providing essential features for LLM input-output parsing and process control. To implement the memory mechanism, we integrated Neo4j [35] for graph-based storage and retrieval and Pinecone [36] for vector search. For feature matching, we employed cosine similarity with embeddings derived from ResNet-50 [37], which enables effective task representation and retrieval. All experiments were conducted using the Android Debug Bridge (ADB), enabling on-device evaluations for mobile applications.

5.2 Experimental Analysis↩︎

In this part, the experiments are designed to validate AppAgentX’s advantages in terms of efficiency and accuracy. We conducted five experiments for the baseline comparison and two experiments for the large dataset to mitigate the effects of randomness in LLMs.

Comparison with baselines. A comparison of our experimental results with model variants is shown in Table 1. We report the results on the AppAgent benchmark. We begin with a baseline model that does not incorporate any memory mechanism and progressively introduce our proposed enhancements. The first model variant integrates a module that records only information about elements, providing a basic form of memory. Subsequently, we introduce a more advanced memory design that maintains a structured memory chain, enabling the model to retain a more comprehensive representation of past interactions. Finally, we augment the system with an evolutionary mechanism that expands the action space to include high-level actions, further optimizing task execution.

As the result shows, incorporating the memory mechanism significantly improves the task success rate. The baseline model, which lacks memory, achieves an SR of only 16.9%. Introducing the chain-structured memory significantly enhances the success rate, reaching 70.8%, highlighting the clear advantage of maintaining a more structured history of interactions. Furthermore, it proves to be more effective than element memory in facilitating task completion. Moreover, the integration of the evolutionary mechanism leads to substantial efficiency gains. Expanding the action space to include high-level actions decreases the average required number of steps from 9.1 to 5.7, while the step execution time is reduced from 23 to 16 seconds. Additionally, average token consumption is significantly minimized, dropping from 9.26k to 4.94k, indicating a more efficient decision-making process. Notably, this enhancement reduces computational overhead and improves the average success rate to 71.4%. These results show that our approach is effective. The memory mechanisms significantly improve task performance, while the evolutionary strategy boosts efficiency by reducing steps, execution time, and token use.

Task difficulties analysis. To further investigate the performance of AppAgentX across tasks of varying complexity, we categorize the tasks based on their ground-truth operation steps, annotated through expert demonstrations. Tasks with up to 5 steps are considered short, tasks with 6 to 10 steps are classified as medium-length, and tasks requiring more than 10 steps are categorized as long tasks. As shown in Figure 3, the violin plot provides an intuitive visualization of how task complexity affects performance. Notably, our method demonstrates a clear advantage in terms of task execution time as the task complexity increases. To account for random fluctuations, we conducted one-tailed paired t-tests [38] on each data group to verify that AppAgentX significantly outperforms the baseline in efficiency, with all p-values falling below 0.05. Our model consistently outperforms the baseline across all task lengths, resulting in reduced task completion times. This finding reinforces the robustness and efficiency of our approach, confirming its stability under varying task difficulties.

5.3 Comparison with other Frameworks.↩︎

To further highlight the advantages of the AppAgentX framework, we compare its performance to other state-of-the-art frameworks on additional benchmarks, including Mobile-Bench [33] and DroidTask [32]. As shown in Figure 2, AppAgentX outperforms AppAgent, in both task execution time and accuracy. Specifically, AppAgentX achieves significantly higher efficiency while maintaining higher task success rates across a broader range of applications.

Additionally, we compare the efficiency of AppAgentX against the Mobile-Agent-v2 framework [39] under different foundational LLMs, including GPT-4 [1], Claude 3.5 Sonnet [40], and Gemini 1.5 Pro [41]. Table 3 shows that AppAgentX consistently outperforms Mobile-Agent-v2[39] (previous sota) in terms of efficiency, highlighting its ability to optimize performance across various LLMs.

5.4 Qualitative Analysis↩︎

In Figures 4, we show a qualitative analysis of some of the actions that used high-level actions to operate the screen. Despite the different user interfaces and actions performed by these applications, AppAgentX successfully completed the given task.

6 Conclusion↩︎

We propose an evolving GUI agent framework that enhances efficiency and intelligence by abstracting high-level actions from execution history. Unlike rule-based automation, our approach generalizes task execution by compressing repetitive operations, balancing intelligent reasoning with efficient execution. A chain-based knowledge framework enables continuous behavior refinement, improving adaptability and reducing redundancy. Experiments show our framework outperforms existing methods in accuracy and efficiency.

Limitations↩︎

As the GUI Agent framework continues to evolve, the recognition and processing of screen content remain a primary bottleneck in current research. While target detection and segmentation techniques in computer vision have reached a relatively mature stage, the detection of interactive elements on screens is still in its early development and lacks sufficient stability and generalization capability. This limitation can introduce biases in the Agent’s perception of the environment, ultimately affecting the accuracy of its interaction decisions.

In this study, we attempt to address this challenge by employing a dynamic matching mechanism in conjunction with the state-of-the-art screen element detection method. However, due to the diversity of screen elements, the complexity of layouts, and the variability across different application interfaces, our approach still has limitations and does not fundamentally overcome the bottleneck of screen content understanding and localization. Therefore, future research should focus on developing more robust screen content understanding techniques, such as integrating large-scale pre-trained visual models, joint image-text modeling, and other advanced methods, to enhance the adaptability and generalization capability of GUI Agents in complex environments.

7 Appendix↩︎

7.1 Chain Design Details↩︎

In chain design, there are three types of relationships between nodes:

Page-HAS_ELEMENT\(\rightarrow\)Element

Type: HAS_ELEMENT

Direction: (Page) -[:HAS_ELEMENT]-> (Element)

Purpose: Indicates that a page contains a specific UI element.

Shortcut-COMPOSED_OF\(\rightarrow\)Element

Type: COMPOSED_OF

Direction: (Shortcut) -[:COMPOSED_OF]-> (Element)

Relationship Attributes:

• order: Integer, representing the execution sequence.

• atomic_action: Type of basic action (e.g., click, text input, etc.).

• action_params: JSON format, potentially including input text, click parameters, etc.

Purpose: Defines how a composite action consists of certain elements and their corresponding steps, which must be executed in a specific order.

Element-LEADS_TO\(\rightarrow\)Page

Type: LEADS_TO

Direction: (Element) -[:LEADS_TO]-> (Page) Purpose: Represents the linkage of a composite action.

7.2 Execution Process Flowchart↩︎

Figure 5 presents a detailed flowchart that delineates the task processing workflow. The primary components of this workflow include page analysis, element comparison, matching, and memory storage.

7.3 Reliability Analysis of Shortcut Designs↩︎

In this study, we implement a fault-tolerant design for high-level actions within the evolutionary mechanism to enhance the overall robustness of the model. As depicted in Figure 5, the system utilizes a hierarchical action framework comprising high-level nodes and basic action spaces. During the operation, if the matching retrieval process of a high-level node fails, the system seamlessly transitions to the corresponding basic action spaces. This fallback strategy ensures that the model remains operational even when high-level decision-making encounters issues, thereby improving its resilience to errors and unexpected scenarios.

References↩︎