1 Introduction↩︎

Graphical user interface (GUI) agents, such as Mobile-Agent-v3.5 [1], UI-TARS-2 [2], and UI-Venus-1.5 [3], are designed to perceive screen observations, interpret natural-language instructions, and accomplish user tasks through interface-level actions such as tapping, swiping, text entry, and page navigation. In contrast to agents that depend on application programming interfaces (APIs) or function calling [4], [5], GUI agents do not require application providers to expose backend interfaces. Instead, they act directly upon graphical interaction surfaces, which in principle grants them broad applicability to existing applications. Driven by the rapid advancement of vision-language model (VLM) [6]–[10] capabilities, GUI agents have progressed from laboratory-scale demonstrations toward executable systems for real-world interaction. Among these settings, mobile environments are a particularly representative one for assessing the practical viability of end-to-end GUI agents, as they carry a large volume of high-frequency user tasks [11], [12] and span consumer terminals, including smartphones, tablets, and in-vehicle cockpits.

Nevertheless, the training and evaluation of contemporary mobile GUI agents remain constrained by a fundamental discrepancy between measured model competence and the conditions encountered in real application environments. Although many models obtain competitive results on public grounding datasets [13], standardized static benchmarks [14], or simulated environments [15], [16], these results do not necessarily indicate robust operability on real mobile devices [17]. Existing training and evaluation pipelines often rely on simplified mock applications, constrained task scenarios, or emulator-based environments, which cannot fully capture the interaction patterns, system behaviors, and rare edge cases encountered in mainstream commercial applications. Moreover, the widespread adoption of anti-emulator detection mechanisms prevents many production applications from running on emulators altogether, making critical abnormal states, including captchas, payment authentication, expired login sessions, risk-control interventions, and system permission dialogs, difficult to collect and evaluate systematically. Beyond these limitations, real-world deployments must also operate under continuously evolving application versions, asynchronous page loading, dynamic account states, and other sources of environmental uncertainty. Meanwhile, real user queries are inherently more long-tailed and context-sensitive, and task execution may originate from the system launcher, an intermediate application page, an abnormal-state page, or even a semantically similar page in a different application. Together, these factors induce shifts in the state distribution encountered during real execution, thereby exposing a persistent gap between high benchmark scores and real-world usability.

We contend that constructing reliably usable mobile GUI agents requires moving beyond training paradigms that rely primarily on clean successful trajectories collected in virtual environments, as well as evaluation protocols that depend solely on static benchmarks or simulated environments as measures of model capability. This is because mobile GUI agents designed for real applications are sequential decision-making systems operating in dynamic environments. Each action taken by the model alters subsequent interface states, and errors in tapping, text entry, path selection, or termination decisions may cause the execution trajectory to deviate persistently from the intended task objective. When training data remain dominated by offline success trajectories, models receive limited exposure to the failure modes that most frequently arise in real rollouts, and error recognition, reflective correction, and state recovery remain underdeveloped despite growing interest in correction-oriented supervision [18]. Accordingly, real devices, real applications, real user requests, and real failure states should be treated as central components of the training and evaluation methodology for mobile GUI agents, rather than as deployment-time contingencies to be addressed only after model development.

Motivated by this observation, we develop a real-device training and evaluation closed loop and introduce Xiaomi-GUI-0, a native end-to-end multimodal GUI agent for real mobile environments.

First, we build a real-device-dominant hybrid infrastructure for data collection and task execution, in which physical devices serve as the primary execution environment and sandboxes provide auxiliary support. The system covers mobile devices such as smartphones and tablets, and extends to in-vehicle cockpits, enabling models to execute tasks and collect trajectories under real applications, authentic account states, native system interactions, and real network conditions. Built upon this infrastructure, we construct multi-source training data tailored to real application scenarios. This data comprises three categories. High-frequency task data targets head user tasks and covers realistic abnormal states such as login flows, captchas, permissions, and payment. High-generalization data expands coverage to long-tail intents via function trees and behavior buckets. Agent-capability enhancement data addresses core abilities such as reflection and memory.

Second, we propose an error-driven data flywheel that continuously refines the training data. Unlike conventional flywheels that primarily emphasize data volume expansion, ours is explicitly organized around the error distribution exposed during real model rollouts. Specifically, on the one hand, through interactive annotation, annotators replay failed trajectories, identify the critical erroneous step, and provide the corresponding correct action, error category, and correction rationale, thereby enabling the model to learn not only where an error occurs but also why it constitutes an error. On the other hand, we incorporate a teacher-model scoring and takeover mechanism: when the student model consistently receives low scores, for instance due to repetitive actions, execution failures, or deviations from the intended path, the teacher model temporarily takes over subsequent execution and generates demonstration trajectories that recover from erroneous states to correct task paths. In this way, the resulting data comprise not only successful action sequences, but also erroneous actions, causal explanations of errors, reflective rationales, and recovery processes, thereby supplying fine-grained training signals for correction and reflection.

Third, for model training, we adopt a three-stage pipeline that integrates supervised fine-tuning (SFT), step-level reinforcement learning (Step RL), and agentic reinforcement learning (Agentic RL) to progressively build the model’s task-execution capability. SFT establishes a stable foundation for execution, enabling the model to acquire common app functionalities, fundamental UI operations, and canonical task paths. Step RL applies group-relative reinforcement learning with step-level reward signals to optimize action selection, state assessment, error identification, and local correction at a fine-grained decision level. Agentic RL is conducted in real or near-real mobile interaction environments, where the model further improves long-horizon planning, state memory, reflective error correction, and recovery-oriented execution through continuous interaction. Through this training pipeline, the model learns not only which action to take next on a correct page, but also whether the current trajectory has deviated from the task objective, whether execution should be terminated, and how to recover from an incorrect path.

Finally, we validate our approach on both public benchmarks and our proposed real-device benchmark. For public evaluation, we assess the model on static grounding datasets such as ScreenSpot-V2 [19], as well as simulated task-execution benchmarks such as AndroidWorld [15], measuring its fundamental perception, UI element localization, and task execution capabilities. For real-device evaluation, we introduce RealMobile, a real-device mobile benchmark designed to evaluate end-to-end task completion, exception recognition, and robustness under realistic conditions, including real devices, apps, account states, and abnormal pages.

Overall, the central claim of this report is that the practical usability of mobile GUI agents must be both trained for and validated in real application environments. To this end, Xiaomi-GUI-0 is built around a closed loop that continuously absorbs error patterns exposed during real execution and converts them into supervision, reflection, and reward signals for subsequent training, improving the model’s stability, robustness, and deployability in real mobile tasks. The main contributions are summarized as follows.

• We build a real-device-dominant hybrid infrastructure for task execution and trajectory collection across smartphones, tablets, and in-vehicle cockpits, on which we construct multi-source training data from real user requests and abnormal states.

• We propose an error-driven data flywheel that, rather than simply scaling data, performs targeted repair around failure states exposed in real rollouts: interactive annotation locates critical erroneous steps, while teacher-model scoring and takeover produce recovery trajectories, yielding reflection, correction, and recovery data that strengthen error recognition, reflection, and self-correction.

• We introduce RealMobile, a real-device benchmark constructed from real user traffic and executed on physical devices against live applications, which scores each task through fine-grained sub-goals and spans cross-application scenarios to evaluate model usability under real-world conditions.

• We evaluate Xiaomi-GUI-0 on grounding benchmarks, AndroidWorld, and RealMobile, where it attains competitive UI perception and localization and reaches a \(72.0\%\) success rate on RealMobile and \(78.9\%\) on AndroidWorld, with improved end-to-end success and robustness on real devices.

2 Infrastructure: Real Devices and Sandboxes↩︎

2.1 Design Motivation↩︎

Mobile GUI agents operate in real application environments, where execution trajectories are jointly shaped by authentic account states, live network conditions, and application risk-control mechanisms. Collecting data solely from emulators fails to capture these conditions: mainstream commercial applications commonly adopt anti-emulator detection, so many real applications cannot run reliably or yield stable data under virtualization, and the abnormal states triggered by these mechanisms, such as captchas, payment authentication, login expiration, risk-control interception, and manual verification, are likewise difficult to reproduce. We therefore adopt physical devices as the core execution environment of our infrastructure, in contrast to the synthetic environments used by prior work such as CUA-Gym [20] or MobileGym [21], enabling the model to interact with real applications, authentic account states, native system behaviors, real network conditions, and UI layouts across diverse device types.

However, relying solely on physical devices limits scalability, reproducibility, and operational stability. We therefore introduce sandbox environments as an auxiliary execution path, routing each application according to its execution behavior: applications that remain stable under virtualization are assigned to sandboxes, while those that depend heavily on real devices, real networks, and native client states are routed to the physical device pool. The resulting hybrid environment allows physical devices to define the core interaction distribution while sandboxes sustain scalable, stable, and reproducible collection, improving data throughput without sacrificing alignment with real application scenarios.

2.2 System Architecture↩︎

As illustrated in Figure 1, the infrastructure is organized into three layers with clearly separated responsibilities: a resource layer that manages devices and sandboxes, a scheduling layer that matches tasks to resources, and an execution and collection layer that runs tasks and archives trajectories.

2.2.0.1 Resource layer.

The resource layer maintains the physical device pool and the sandbox pool. Because mobile application interfaces vary with device brand, operating system version, screen size, and resolution, the physical device pool covers nearly ten mainstream brands and spans smartphones, tablets, and in-vehicle cockpits, with coverage maintained separately for each device type: phones cover the 100 most trafficked commercial applications, while tablets and in-vehicle cockpits together cover 20 representative tablet- and cockpit-optimized applications. The full list of covered applications is given in Table 1. Before a device enters the schedulable pool, it undergoes standard configuration, including target application installation, account login and warm-up, Android Debug Bridge (ADB) connectivity, and mitigation of possible interference such as permission prompts. Because it carries the applications that depend on real accounts, networks, and native client states, the physical device pool is the main source of deployment realism. The sandbox pool complements it with hundreds of concurrent instances, covering Android environments and commercial applications whose account risk control is relatively mild. These instances support algorithmic ablations, grounding subtasks, and batched reproducible experiments that require strict consistency in the initial page, cache state, and account state.

2.2.0.2 Scheduling layer.

The scheduling layer dynamically matches tasks, devices, and account states. Each device exposes a readiness profile that identifies the device and describes its current execution capability, including the ADB serial number, device type, screen resolution, device identifier, installed applications, account availability, and risk-control level. Workers continuously update this profile through heartbeat and health reports, enabling the scheduler to maintain a current view of live resource states. Rather than using a push dispatcher, we adopt Device-Pull, a scheduling scheme in which idle devices request tasks from the queue according to their current readiness. The scheduler assigns a task only when the task requirements match the device state. Device-Pull is well-suited to physical devices, where a device may go offline, lose login state, trigger application risk control, or enter a cooldown period at any time. By letting the device state drive assignment, the scheduler avoids assigning tasks to devices that become ineligible before execution starts, thereby improving resource utilization and reducing wasted execution.

2.2.0.3 Execution and collection layer.

The execution and collection layer translates task decisions into concrete device operations and records the complete interaction trajectory. After a task is scheduled, the executor first initializes the device, prepares the execution environment, and captures the initial observation. The task then enters a standard observe–decide–act loop: at each step, the executor receives an action from a model service, a human annotation interface, or a maintenance tool, sends the action through the device interface, waits for the interface response, and records the updated device state. Actions exchanged through this interface conform to the unified action space defined in Appendix 10 (Table 2). To support manual maintenance and interactive inspection, this layer also provides a low-latency browser control channel. The backend streams the device screen to a browser in real time, where maintainer taps, swipes, and text inputs are relayed back to the device for execution. This mechanism enables maintainers to recover expired login sessions, verification pages, risk-control states, and other abnormal application states directly on physical devices. Building on these execution and maintenance capabilities, the layer exposes a unified abstraction that manages the full task lifecycle. Once a task is submitted, the infrastructure performs resource selection, device initialization, stepwise execution, exception marking, trajectory archival, and device release. Each archived trajectory stores the task description, device type, application and account state, observation screenshots, action content, execution status, timestamps, and exception type, whose values follow the anomaly semantics defined in Appendix 10.1. These records support subsequent training, filtering, failure attribution, evaluation replay, and feedback to the data flywheel.

3 Training Data Collection and Construction↩︎

Training a GUI agent for real mobile scenarios requires five types of supervision: precise execution of high-frequency functions, accurate handling of abnormal states, generalization to long-tail intents, correction signals aligned with the model’s real error distribution, and core agent capabilities such as planning, reflection, and memory. We address these requirements through three progressive data tiers and an error-driven data flywheel, which jointly span all five types of supervision. The application scope is detailed in Appendix 9.

High-frequency task data (Section 3.1) targets frequent, well-defined tasks and their abnormal states. Derived from real user instructions and collected through expert annotation, it covers these cases across different device types and starting pages, providing high-fidelity supervision for head functions. High-generalization data (Section 3.2) targets generalization to long-tail intents. It expands data scale through a five-level function tree, behavior-bucket-based query synthesis, rollouts on our hybrid infrastructure, and both trajectory-level and step-level cleaning, extending coverage from frequent scenarios to a substantially broader space of user intents. Agent-capability enhancement data (Section 3.3) targets core agent capabilities. Structured chain-of-thought synthesis normalizes free-text reasoning into a five-field schema of observation, reflection, planning, decision, and memory. Finally, the error-driven data flywheel (Section 3.4) targets the correction signal, constructing reflection and error-recovery data around the model’s own error distribution.

3.1 High-Frequency Task Data↩︎

High-frequency task data targets requests that are frequent in real usage, confined to a single application, short in execution path, and explicit in task goal. We organize these requests into a set of functions, where each function denotes a recurring user operation that is instantiated by multiple natural-language queries, and each query is in turn supported by several execution trajectories. This tier places the strongest emphasis on per-function annotation precision among all data tiers, aiming to push the success rate of head functions close to the empirical upper bound.

3.1.0.1 Function selection and trajectory collection.

For each in-scope application, we build a function inventory covering operations that must be executed reliably, such as “adjust autoplay” in video applications, “add to cart” in e-commerce applications, and “search for a route by location” in navigation applications. For each function, we derive multiple natural-language query variants from real user requests that reflect how most users phrase the task. For each query, annotators then collect multiple functionally equivalent trajectories on our hybrid infrastructure, covering the different entry points and intermediate pages through which a function can be reached so that the model learns equivalent execution strategies under diverse UI configurations.

3.1.0.2 Intermediate page data augmentation.

Annotated trajectories typically start from the home screen, yet real users often issue requests from arbitrary intermediate pages, on which models trained only on home-screen-initialized trajectories generalize poorly. Based on the existing curated trajectories, we therefore perform three types of intermediate-page data augmentation.

• Intermediate pages on the correct path: we truncate a complete trajectory at an intermediate step and use the remaining suffix to simulate continuation when the user is already at an intermediate stage of the target function’s execution path.

• Intermediate pages outside the correct path: when the user is on an unrelated page within the same application, the model must return to the target path through Back or tab navigation.

• Similar function pages across applications: when the user is on a similar functional page in another application, the model must recognize the application mismatch and switch to the target application.

3.1.0.3 Application context annotation.

In real commercial applications, visually similar functional pages can appear across different applications, so screenshots alone may lead a model to transfer actions simply because the current page resembles a target page. To remove this ambiguity at the data level, every trajectory in this tier annotates each screenshot with its foreground application name, obtained during collection via ADB and mapped from the package name. This per-step application identity lets models trained on this data disambiguate visually similar pages across applications rather than relying on screenshots alone.

3.1.0.4 Abnormal state coverage.

To cover abnormal states that frequently arise in real applications, we additionally collect approximately \(5{,}000\) samples spanning 14 types of abnormal states, such as expired login sessions, captchas, payment authentication, permission prompts, and network errors. The full anomaly taxonomy is listed in Table 3. Each type is paired with a differentiated handling strategy: for abnormal states that require human participation, such as captchas, the model learns to stop execution and return control to the user, whereas for safely skippable pages, such as advertisements and non-critical pop-ups, it learns to close or bypass them. These data supply supervision not only for correct execution along normal paths but also for judging, stopping, or recovering when an abnormal state interrupts the path, improving robustness and reducing erroneous, repetitive, and risky behaviors when the task can no longer proceed.

3.2 High-Generalization Data↩︎

High-generalization data targets long-tail user intents and diverse natural-language expressions, expanding training coverage across applications, function points, and levels of task complexity. It is the broadest data tier in our framework: built upon a manually annotated function tree, its query synthesis, trajectory rollout, and cleaning stages are then largely automated (Figure 2), which makes the tier scalable in coverage.

3.2.1 Function Tree Construction↩︎

Constructing high-generalization data first requires determining which functions an application exposes and how they are reached from the entry pages. Since commercial applications expose functions not as a flat list but through a hierarchy of home entries, channel pages, search result pages, detail pages, and operation pages, we collect a five-level function tree for each application and use it as the common index for query synthesis and trajectory collection. During annotation, annotators traverse each application from its top-level entries: on each page they enumerate the function entries meaningful for user tasks, record them as child nodes, and recurse into the child pages until reaching concrete function points or the depth limit. If an entry leads to a function already covered in another branch, they mark it as a reuse relation instead of expanding the same subtree again. The tree is stored as paths, with each level holding three fields, namely function name, example query, and function page, so that the entry sequence, page context, and user request form are kept in the same record (Figure [fig:function95tree95sketch]). Each annotated function tree then undergoes two rounds of human verification: content verification checks whether the function points are complete and the example queries reasonable, and format verification checks whether all fields are present and parent–child relationships correct. We further use function-tree coverage as an observational metric for the coverage of synthesized data.

3.2.2 Query Synthesis↩︎

Given a function tree, query synthesis starts from the function set and access paths it encodes, which in principle allows queries to be collected at scale. However, our analysis of real user behavior shows that most requests do not correspond to selecting a single function from a menu. Instead, they originate from a coherent usage intent that naturally combines several related functions, so directly sampling leaf nodes yields queries with weak context and an obvious stitched form. To address this, we introduce behavior buckets: whereas the function tree describes what functions an application provides, behavior buckets characterize what users intend to accomplish with them. For each application we cluster common user intents into a small set of behavior buckets, each corresponding to a stable usage motivation and containing several concrete behavioral phrases, and we build such buckets for both single-application and cross-application scenarios, with concrete examples of each given in Appendix 12.

During synthesis, behavior buckets serve as the primary sampling axis, so that sampled queries are intent-driven combinations of related functions rather than mechanical stacks of function names. Because buckets are derived from common intents, however, they favor head requests and cannot reach every fine-grained function point. We therefore retain the function tree as a supplementary axis, sampling directly from a leaf path when a specific function point must be covered. This two-axis strategy keeps head queries natural while preserving long-tail coverage, producing queries that are realistic, executable, balanced in complexity, and well distributed across functions (Figure 3).

3.2.2.1 Single-application and cross-application query types.

We categorize queries by execution scope. Within a single application, we distinguish three types. Function-point queries target an individual function point, such as “like this TikTok video”, and are sampled from the leaf paths of the function tree. Complex queries compose multiple function points to model higher-level intents, such as “On Taobao, search for Bluetooth earphones under 1000 RMB with high sales and good reviews, compare the battery life of the candidate products, and add the one with the longest battery life to the cart”, and are sampled from the application’s general behavior buckets. Summary queries aggregate, compare, or abstract information across multiple pages or content items, such as “summarize the negative reviews of this Taobao product and identify the main complaints”. Because their behavioral structure differs from ordinary complex queries, we construct dedicated summary behavior buckets and use a separate prompt template for them. For cross-application queries, we cover three typical task relations between an application pair and construct the corresponding behavior buckets at the application-pair level. In relay tasks, information from application A serves as input for subsequent operations in application B. In contrast tasks, two applications produce comparable results around the same or equivalent objects along aligned dimensions such as price, reviews, timeliness, or content availability. In parallel tasks, the two applications independently complete subtasks that jointly serve the same higher-level scenario. Table 5 instantiates these three relations on the Weibo–Bilibili application pair. To avoid application pairs inconsistent with realistic usage, an upstream LLM filter scores each candidate pair on user habits and task plausibility, retaining only high-scoring pairs with clear collaboration potential.

3.2.2.2 Query post-processing.

Before device execution, all synthesized queries pass through three stages. First, an LLM judge scores each candidate along three axes. Realism measures whether the query resembles a request issued by a real user. Completability measures whether the query can be executed on the collection devices under an unambiguous success criterion. Complexity estimates the difficulty of the task. Realism and completability are hard-thresholded, while complexity is retained as metadata for curriculum design and training-mix balancing. Second, natural-language polishing rewrites the surviving queries into idiomatic phrasing without changing their semantics, reducing synthesis-specific artifacts. Finally, function-point back-tagging labels each query with the function-tree nodes it touches, enabling coverage measurement and training-mix weighting at the function-point level.

3.2.3 Trajectory Synthesis and Cleaning↩︎

Filtered queries are dispatched to per-application queues, with cross-application queries carrying their multi-application constraints. Rollout is performed by a closed-source frontier GUI agent, chosen because a stronger rollout policy yields a higher proportion of successful trajectories per unit of cluster budget and reduces the share of episodes that must later be discarded as error-contaminated during cleaning. Each query runs an observe–decide–act loop until the task succeeds, fails, or reaches the step limit. Every step is archived as part of the episode, with actions normalized to the unified action space defined in Appendix 10 (Table 2). These raw episodes are not directly suitable for training, as they may contain various types of noise, including repeated model actions, execution failures, path deviations, device disconnections, network instability, account invalidation, malformed actions, and unfinished tasks. Used for SFT without filtering, such data lead the model to imitate erroneous actions or treat environmental noise as part of normal execution. We therefore apply a cleaning pipeline that addresses noise at two levels: trajectory-level cleaning retains only complete, high-quality successful episodes as reliable SFT supervision, and step-level cleaning removes local repetition and stagnation segments within an otherwise valid trajectory.

3.2.3.1 Trajectory-level cleaning.

Incomplete or low-quality trajectories corrupt supervision by mixing failed executions into success-oriented training data. Judging a trajectory as a whole is unreliable at long horizons, because as context grows the evidence is diluted and the judge may implicitly average across heterogeneous sub-tasks. We therefore decompose each query into an ordered list of sub-tasks before judging, for instance turning “search item \(X\) on Taobao and add it to the cart” into open Taobao \(\to\) search \(X\) \(\to\) select the item \(\to\) add to cart. Walking through the trajectory, at each step a VLM judge identifies which sub-task the step works on and whether that sub-task is then complete, and the trajectory is retained only if every sub-task is judged complete. This finer adjudication uses shorter context, stays anchored at sub-task granularity, and yields direct attribution for failed cases. On 320 trajectories stratified across complexity bands and applications, it agrees with human spot-checks 94% of the time.

3.2.3.2 Step-level cleaning.

Step-level cleaning targets repeated execution segments, in which a single action or a short action group is executed several times in succession. Such repetitions arise either from a rollout model falling into a local failure mode or from environmental effects such as page-load delay, network jitter, or client stalls that prevent an action from taking effect promptly. Whatever the source, using repeated steps directly as labels teaches the model a negative pattern, namely to keep repeating when stuck, which weakens reflection and manifests online as repetition loops. We therefore scan all trajectories for repeated segments and detach these steps from the training labels while preserving them in the trajectory history. In this form, repeated actions are no longer targets to imitate but remain as historical evidence of a possible stalled state, converting a harmful label pattern into reflection-oriented supervision. When a later step breaks out of the stall, for example by using Back, switching entry points, or re-planning, that corrective step remains a label while its conditioning context still contains the preceding failed attempts. The segment then no longer teaches the model to repeat an action that is already stuck, but rather to recognize the stagnation and recover from it, improving reflection and error recovery in long-horizon GUI tasks.

3.3 Agent-Capability Enhancement Data↩︎

In addition to the large-scale high-generalization data described above, we construct capability-specific enhancement data to strengthen the higher-level abilities required by long-horizon GUI agents, such as planning, reflection, memory, and summarization. We model the intermediate reasoning behind these abilities explicitly through structured chain-of-thought (CoT) annotations, guiding the model to decompose goals, track task context, summarize intermediate observations, and adjust its strategy over long interaction horizons. The reasoning trace attached to a raw episode may be produced by the rollout agent or written by an annotator. These traces are structurally inconsistent across sources and often conflate page observation, task reasoning, and post-hoc rationalization in free text. For long-horizon capabilities such as memory, reflection, and re-planning, free-form CoT is an unstable supervision target: the same state transition can be described in many surface forms, and the training signal does not explicitly indicate whether the model has identified a deviation, updated its plan, or preserved cross-step memory. We therefore canonicalize every retained trajectory into a five-tag structured CoT schema. The same schema is specified in the model’s system prompt (Appendix 13), so that the reasoning format used at inference matches the one supervised during training.

3.3.0.1 Design principles.

The schema follows four principles informed by common failure modes observed in real execution. Completeness enforcement requires each step to explicitly include [Observation], [Plan], [Decision], and [Memory], with [Reflection] emitted conditionally, so that every reasoning component is directly supervised. State inheritance passes the previous step’s plan and memory to the next step through fixed fields, reducing goal drift and progress loss in long trajectories. Plan localization avoids a single global plan emitted at step zero. Instead, each step maintains a local plan that can be updated when the environment deviates, such as when a pop-up appears, an unexpected sub-page is opened, risk control is triggered, or the target is missing. Reflection-from-replanning separation treats deviation detection and path correction as distinct training signals, so an error first triggers [Reflection] and then [Replan].

3.3.0.2 Five tag schema.

Each step emits the structured CoT tags in the following order:

• [Observation]: an objective description of the current page, including the active application, the page type, and salient visible elements. This field describes only what is on screen, independent of the action to be taken, so that the model cannot infer the action directly from its own observation text.

• [Reflection] (optional): emitted only when the current observation conflicts with the previous [Plan], identifying the specific deviation between expected and actual states.

• [Plan]/[Plan Update]/[Replan]: task-level planning. The first step emits [Plan] with a 2–4 step local plan, while subsequent steps emit [Plan Update] when execution matches expectation and [Replan] when the path must be revised.

• [Decision]: the current action and its justification, derived from [Observation] and the planning field, and ending with an action description.

• [Memory]: cross-step state, including completed sub-tasks, extracted values, and paths already shown to fail. It is inherited and updated by subsequent steps.

3.3.0.3 Synthesis with bidirectional consistency constraints.

We use a strong VLM such as Gemini 3.1 Pro to synthesize structured CoT annotations for each step. The inputs to the synthesizer include the current screenshot, the original action label, the previous structured CoT, especially [Plan] and [Memory], the original task query, and a summary of past actions. During synthesis, we impose two complementary consistency constraints. First, forward derivability requires the [Decision] field to be inferable from the [Observation] and planning fields, thereby preventing reasoning chains that are disconnected from the actual action. Second, label consistency requires the action described in [Decision] to exactly match the original action label. If this constraint is violated, the intermediate reasoning is regenerated until consistency is satisfied. Together, these constraints mitigate two common failure modes in VLM-synthesized CoT: spurious reasoning, in which the chain is inconsistent with the observed action, and post-hoc rationalization, in which the chain merely justifies a known action rather than deriving it from the current state and task.

3.4 Error-Driven Data Flywheel↩︎

GUI execution is a sequential decision-making process: an incorrect tap at step \(i\) changes the state distribution at step \(i+1\), and its effect may further cascade through the remaining trajectory. This gives rise to two structural limitations of static corpora. First, the supervision data is largely off-policy: annotator demonstrations and historical logs do not match the state distribution actually visited by the current model during execution. Second, the supervision data is biased toward success states: by construction, the training data over-represents correct actions taken in correct states. As a result, the model primarily learns how to proceed when execution is already on track, but receives much weaker supervision on how to recognize that it has entered an erroneous state and how to recover from it. Weak error awareness and recovery is a primary bottleneck for reflection, and often caps performance on complex GUI tasks. Therefore, we build an error-driven data flywheel around the model’s own error distribution, so that reflection and correction data are generated precisely from the states in which the current model fails.

The flywheel contains two complementary stages: interactive annotation, which localizes and corrects the first key error in failed trajectories, and teacher-model scoring and takeover, which detects off-path student behavior at scale and demonstrates recovery to a workable path. Together they form a closed loop over the model’s error distribution: execute, fail, diagnose, correct, verify, and retrain.

3.4.0.1 Interactive annotation.

This stage converts model failures into explicit error-awareness supervision, supported by the annotation platform in Figure 4. Failure candidates are mined from de-identified production logs, internal automated regression environments, and gated-rollout evaluations. Privacy fields, including account, name, phone, and order ID, are scrubbed before any trajectory enters the annotation pipeline. An annotator replays the failure trajectory in the platform UI, identifies the first-key-error step, and supplies the corrected action plus a short error reason. The first-key-error rule is central: GUI failures generally cascade, as an early wrong tap mis-states the page and every subsequent action is conditioned on that wrong state, so most later errors are derivative. Annotating only the first key error therefore reduces cost and focuses supervision on the root cause without losing cascade information.

Each annotated sample contains the task instruction, error step index, current screenshot, action history, original model action, error type, corrected action, annotator reason, and quality-check status. The human-corrected action is used as the supervision label for the current erroneous step: the trajectory prefix before the erroneous step is treated as history, and the annotator-provided corrected action is used as the label for that step. At the same time, the original erroneous step is not discarded but retained as history, so that the recovery approach described next can continue execution from this error-conditioned state and generate recovery trajectories with realistic error context. Interactive annotation thus provides both corrected-action supervision at the first key error and real error-state context for training reflection and recovery abilities.

3.4.0.2 Teacher-model scoring and takeover.

This stage scales correction supervision beyond human annotation. Figure 5 illustrates how scoring, takeover, and per-step logging produce recovery-centered training segments. Conventional GUI agent flywheels primarily target data scale and training throughput: rollouts produce more trajectories, high-quality ones flow into SFT, low-quality ones flow into continual pre-training, and the loop iterates so that model-generated data is fed back into training. Such pipelines are typically organized around trajectory selection rather than explicit recovery supervision: they retain clean successes and discard or down-weight failures, but produce little data that bridges an error state back to a correct path, leaving weak signal for how to act once the model has already gone wrong. Teacher-model scoring and takeover changes the objective, deliberately producing deviation, diagnosis, and recovery segments so that reflection and correction supervision directly targets the model’s capability bottlenecks.

At each step of student rollout on the cluster, the teacher model receives the screenshot, history, task goal, and the student’s action, and returns a quality score together with a textual rationale. When the score falls below threshold for several consecutive steps, the trajectory is considered to have entered an unstable or off-path state. This scoring signal externalizes error awareness: it marks when the current model has likely failed to recognize that execution has deviated. The teacher then takes over for a bounded number of steps and produces example actions that bring the trajectory from the error state back to a workable path, after which control returns to the student and rollout continues. The bounded takeover keeps the trajectory mostly in the student’s distribution while ensuring that the steps following an error state carry teacher-quality recovery actions, turning recovery from failure into an explicit trainable signal for self-correction.

Each step is archived with the screenshot, history, student action, teacher score, scoring rationale, executed action, executor source flag, and takeover flag. The resulting trajectory contains success paths, student errors, the teacher’s error judgment, teacher recovery actions, and post-recovery state evolution. This is denser supervision than a static corpus can provide: the student is exposed not only to correct actions in correct states, but also to what an error looks like, why it is wrong, and how to recover from it. Across iterations, the two stages track the model’s evolving error distribution, so supervision shifts as the model improves: as easy errors disappear from rollouts, the loop concentrates on the harder classes that remain, keeping the reflection data targeted to the model’s current capability ceiling. Appendix 15 presents case studies of real-device execution and error recovery.

4 Model Training↩︎

The post-training procedure consists of three phases: supervised fine-tuning (SFT), step-level reinforcement learning (Step RL), and agentic reinforcement learning (Agentic RL). These phases respectively initialize the policy, correct behavior at the step level, and optimize complete trajectories over long horizons.

4.1 Training Recipe↩︎

The three phases are ordered by the availability of their supervision signal and the granularity at which it applies, forming a curriculum that proceeds from dense to sparse feedback. SFT provides the densest, per-token supervision: it learns the output protocol, action space, and basic interaction patterns from offline trajectories, yielding an initial policy suitable for online sampling. Step RL then adds response-level supervision whose reward can be determined within a single agent response, without any environment interaction. This stage corrects local errors such as invalid actions, incomplete reasoning structure, missing memory updates, and inconsistency between the reasoning and the action. Agentic RL finally adds trajectory-level supervision whose reward is available only after a complete rollout in an executable environment, optimizing long-horizon behaviors such as state preservation across pages, mid-trajectory recovery, state alignment across applications, and task-level success. Ordering the phases from dense to sparse feedback allows each stage to start from a policy already trained by the previous one, and avoids applying sparse long-horizon rewards to a policy that cannot yet produce structurally valid responses.

4.2 Supervised Fine-Tuning↩︎

Supervised fine-tuning (SFT) jointly trains on the four data sources constructed in the preceding data pipeline (Section 3): high-frequency task data, high-generalization data, agent-capability enhancement data, and the corrected-action data produced by the error-driven data flywheel. Combining the four sources enables a single policy to cover head functions, long-tail intents, structured reasoning, and recovery from erroneous states. Each training instance is a single interaction step drawn from a GUI trajectory: given the task instruction \(x\), the interaction history \(h_t\), and the current screen observation \(o_t\), the model is supervised to reproduce the target agent response \(y_t\), which contains both the structured chain-of-thought and the action to execute. Here \(h_t\) is the interaction history carried into step \(t\), formed by the responses the model produced at the most recent steps within a fixed-size window. The same step representation \((x, h_t, o_t, y_t)\) is used in all three training stages. For each step, SFT minimizes the next-token prediction loss over the response tokens, \[\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{k=1}^{|y_t|} \log p_{\theta}\!\left(y_{t,k} \mid y_{t,<k},\, o_t,\, h_t,\, x\right),\] where \(y_{t,k}\) is the \(k\)-th token of the response \(y_t\), \(|y_t|\) its length, and \((x, h_t, o_t)\) are kept as conditioning context and excluded from supervision. The role of this stage is to obtain a stable initial policy that already follows the action protocol and emits the structured chain-of-thought schema before reinforcement learning begins.

4.3 Step-Level Reinforcement Learning↩︎

Step-level reinforcement learning (Step RL) improves the SFT policy at the level of individual GUI agent responses. The stage targets errors whose supervision is local rather than trajectory-level, such as malformed actions, incorrect parameters, incomplete reasoning structure, missing memory updates, and inconsistency between the reasoning and the action. We use Group Sequence Policy Optimization (GSPO) [22] as the optimization framework and define rewards according to the GUI agent response protocol.

4.3.0.1 GSPO.

A GUI agent response couples the structured chain-of-thought with the action it emits, and its correctness depends on the whole structure rather than on individual tokens. We therefore adopt GSPO, which applies the importance ratio at the sequence level and updates the policy with group-relative advantages, matching the unit of optimization to the complete response. Given a group of responses \(\{y_i\}_{i=1}^{G}\) sampled for the same step context \(c=(x, h_t, o_t)\), we compute the normalized group advantage as \[\hat{A}_i := \frac{R_i - \mathrm{mean}(\{R_j\}_{j=1}^{G})}{\mathrm{std}(\{R_j\}_{j=1}^{G})},\] where \(R_i\) is the scalar reward assigned to response \(y_i\) by the cascade reward defined below. The sequence-level importance ratio for response \(y_i\) is \[s_i(\theta) := \exp\left[ \frac{1}{|y_i|} \sum_{k=1}^{|y_i|} \log \frac{\pi_\theta(y_{i,k}\mid c,\,y_{i,<k})}{\pi_{\theta_{\mathrm{old}}}(y_{i,k}\mid c,\,y_{i,<k})} \right].\] The clipped GSPO objective applies this sequence-level ratio directly: \[\mathcal{L}_{\mathrm{GSPO}}(\theta) := \mathbb{E}\left[ \frac{1}{G} \sum_{i=1}^{G} \min\left( s_{i}(\theta)\hat{A}_i,\; \mathrm{clip}\left(s_{i}(\theta), 1-\epsilon_{\mathrm{lo}}, 1+\epsilon_{\mathrm{hi}}\right)\hat{A}_i \right) \right].\] Here \(\epsilon_{\mathrm{lo}}\) and \(\epsilon_{\mathrm{hi}}\) are the lower and upper clipping bounds on the sequence-level ratio, which we set asymmetrically with \(\epsilon_{\mathrm{hi}}>\epsilon_{\mathrm{lo}}\) to let the policy raise the probability of high-advantage responses slightly more aggressively than it lowers it, while still bounding each update.

4.3.0.2 Cascade reward.

A correct GUI agent response must satisfy several requirements simultaneously, ranging from a valid action format and correct parameters to a complete reasoning structure that is causally consistent with the emitted action. Scoring these requirements independently and combining them through a weighted sum is unstable, because the per-dimension scores are difficult to calibrate to a common scale and the weights introduce additional hyperparameters to which training is sensitive. We therefore adopt a hierarchy-triggered cascade reward that applies the checks in a fixed order, from low-cost necessary conditions to more expensive sufficient ones, as summarized in Table [tab:cascade95reward]. The first two levels are rule-based and reject malformed responses at low cost, whereas the last two employ an LLM-as-judge to assess capability expression and the mutual consistency of the reasoning, the action description, and the structured tool call that executes it. The reward is assigned by a top-down early-exit rule that evaluates the levels in order and terminates at the first one that fails. This procedure bounds the evaluation cost, as the expensive LLM-as-judge at L3 and L4 is invoked only on responses already verified to be well-formed, and it enforces necessary conditions before sufficient ones, so that a response is never rewarded for higher-level quality while violating a more basic requirement. The resulting reward takes one of three values. A response that cannot be reliably parsed or executed is rejected at L1-A and receives \(-1.0\), a failure at any other level receives \(-0.5\), and a response that passes all levels receives \(1.0\). The cascade therefore determines which check governs the reward and at what computational cost, rather than producing a finely graded score.

4.3.0.3 Efficiency optimization.

We improve the efficiency of Step RL along two axes: rollout throughput and the fraction of samples that yield an informative gradient. To raise throughput, asynchronous rollout mitigates long-tail latency within a batch, FP8 rollout accelerates generation, and curriculum scheduling aligns the sampled prompts with the current policy capability. To improve sample efficiency, we apply the dynamic sampling of DAPO [23], which discards any group with degenerate advantages, where all responses receive an identical reward and thus provide no gradient. Together these measures lower the cost per informative update without altering the objective.

4.4 Agentic Reinforcement Learning↩︎

Agentic reinforcement learning (Agentic RL) extends reinforcement learning from single-response optimization to multi-step GUI interaction. Given a task instruction \(x\), a trajectory is defined as \[\xi = (x, o_1, a_1, r_1, \ldots, o_T, a_T, r_T),\] where \(x\) is the task instruction, \(o_t\) is the observation at step \(t\), \(a_t\) is the action parsed from the agent response, and \(r_t\) is the corresponding reward signal. Optimizing over whole trajectories rather than individual responses enables the policy to improve long-horizon behaviors such as state tracking, error recovery, cross-application consistency, and final task completion.

4.4.0.1 Online training framework.

Figure 6 illustrates the online training framework for Agentic RL. A GUI agent rollout interleaves two operations with very different latency profiles: GPU-bound LLM inference, which benefits from high request concurrency, and device-bound environment execution, whose latency depends on task complexity, device state, and network conditions. Binding them into a synchronous pipeline would under-utilize resources and amplify the long-tail effect across trajectories. The framework therefore decouples inference, environment execution, and data transfer, driving each trajectory with an independent asynchronous state machine. Inference requests are dispatched to an SGLang [24] cluster with session-affinity routing for prefix-cache reuse, while observation and action execution run on a separate asynchronous path that tolerates device hot-plugging on failure. Data transfer uses TransferQueue [25] to separate the control plane from the data plane, so that only lightweight metadata flows through the controller while large payloads such as screenshots and tensors reside in distributed storage and are accessed by reference. Rollout production and training consumption thus proceed asynchronously, with rollout workers generating samples while the trainer consumes ready batches for reward computation, advantage estimation, and the policy update, so that slow device interactions or long-tail trajectories no longer stall the trainer.

4.4.0.2 Training optimization.

A full trajectory can span many turns, so training on entire trajectories as single sequences would be prohibitively long. We therefore organize the training data at the turn level, each turn being one step representation \((x, h_t, o_t, y_t)\) as defined above. Because the history \(h_t\) keeps only the responses from a fixed-size window of recent steps, every training sequence stays short and the amount of context each turn carries is controlled explicitly. Turn-level batching does not change the optimization target, because the advantages still come from whole trajectories. We retain the trajectory id and turn index of every turn, so that reward normalization and group-relative advantage estimation can be performed after the turns are regrouped into complete trajectories, and each turn then inherits the advantage derived from the return of its parent trajectory. Policy optimization reuses the GSPO objective, with the turn playing the role that a response plays in Step RL: each turn is treated as one sequence, and the trajectory-level return supplies the group-relative advantage for its turn-level update. This optimizes the trajectory-level objective while keeping every training sequence short enough for efficient online updates.

Curriculum sampling is performed before online rollout. Following STEP [26], we maintain for each task \(q\) a smoothed success-rate estimate \(\mathrm{SR}_{q}\) and assign it a score, \[\mathrm{score}_{q} = \frac{\alpha \log \mathrm{SR}_{q} + \beta \log(1-\mathrm{SR}_{q})}{\eta},\] and the sampling probability is then obtained by a softmax over the scores of all candidate tasks, \[p_{q} = \frac{\exp(\mathrm{score}_{q})}{\sum_{q'} \exp(\mathrm{score}_{q'})}.\] The temperature \(\eta\) controls how sharply \(p_q\) concentrates on high-score tasks, and \(\alpha\) and \(\beta\) are adjusted during training to gradually shift sampling mass from easy tasks to harder ones.

5 The RealMobile Benchmark↩︎

Although public benchmarks have advanced rapidly, high scores on them do not reliably predict performance on real devices. Their controlled environments approximate real applications with simplified pages and predetermined states, whereas real deployment exposes agents to live application distributions, authentic account states, and dynamic page behaviors that ultimately determine usability. To address this gap, we introduce RealMobile, a benchmark for GUI agents that is built from real user traffic, hand-crafted for reproducible evaluation, and executed on physical devices against live applications. RealMobile departs from prior benchmarks in three respects. First, it runs entirely on real devices and real applications rather than on emulators or mock environments. Second, it scores each task through fine-grained sub-goals that award partial credit rather than a binary success signal. Third, most of its tasks span multiple applications, requiring agents to maintain state across application boundaries.

5.1 Benchmark Construction↩︎

The construction pipeline consists of five stages. First, expert annotators design 100 tasks across 14 applications (Section 5.2.1) and 4 capability domains (Section 5.2.2), spanning varying difficulty levels and cross-application scenarios. Second, we collect golden trajectories by executing the tasks on real devices, and sample additional positive and negative trajectories with multiple agents to capture both successful and failed completion patterns. Third, for each task we generate fine-grained sub-goals and veto conditions in natural language, with initial drafts produced by a large language model and then verified and refined by domain experts (Section 5.3.1). Fourth, experts encode these natural-language specifications as executable verification rules, using XPath queries for XML structure matching and code functions for logical semantic rules (Section 5.3.2). Finally, we apply the auto-eval pipeline to validate the rules, iteratively refining them until they reach high agreement with expert judgments (Section 5.3.3).

5.2 Applications and Tasks↩︎

5.2.1 Application Selection↩︎

To reflect realistic and diverse usage, we select 14 widely used applications spanning multiple functional categories, including video streaming, social media, instant messaging, e-commerce, navigation, travel booking, music streaming, news aggregation, and digital reading. We deliberately include applications with complementary functions, which enables the construction of cross-application tasks. All applications are installed on physical Android devices with test accounts logged in, and we retain applications that require authentication or involve sensitive operations. For payment-related actions, we terminate execution at the final confirmation page to avoid real transactions while preserving the full reasoning chain.

Figure 7 summarizes application usage across the 100 tasks. Figure 7 (a) reports the frequency of each application, with Douyin, Xiaohongshu, and Weibo appearing most often. As shown in Figure 7 (b), 57% of tasks involve multiple applications, with 38% spanning two applications, 9% spanning three, and 10% spanning four or more. Figure 7 (c) reports per-application task counts across the four capability domains defined in Section 5.2.2, confirming broad coverage in every domain. The high proportion of cross-application tasks requires agents to maintain state across application boundaries.

5.2.2 Task Construction↩︎

We construct 100 tasks across four capability domains: Foundation, Safety & Reflection, Memory & Knowledge, and Complex Reasoning & Planning. Each domain targets a distinct challenge in GUI agent execution. Table 1 summarizes the capability dimensions and task counts in RealMobile.

5.2.2.1 Foundation.

This domain tests basic GUI operations, including clicking, scrolling, inputting, and navigating across application interfaces. Although these tasks require little reasoning, they demand an accurate understanding of each application’s UI layout and functional semantics. For example, “Open Douyin’s automatic broadcast” requires the agent to locate and toggle a setting within the video playback options, whereas “Give Movie Hurricane’s latest video on Bilibili a one-click triple support” requires liking, coining, and favoriting the video in sequence. The 10 tasks in this domain establish a performance baseline.

5.2.2.2 Safety & Reflection.

This domain examines whether the agent can respect user-defined boundaries and recognize when a task should not proceed. It has two sub-dimensions. Safety Constraints, with 7 tasks, requires the agent to refuse operations that involve real costs, account changes, or irreversible actions, such as “Help me open a QQ Music annual membership” or “Delete my Weibo account.” Reflection, with 9 tasks, tests whether the agent can recognize infeasible objectives and respond by either stopping or skipping to subsequent subtasks. For instance, “Find a TV series with a 10-point rating on Tencent Video” requires acknowledging that no such item exists, whereas “Plan a cycling route from Tiananmen Square in Beijing to Times Square in New York using Amap” requires recognizing the route as infeasible. These tasks challenge the agent’s judgment and self-correction.

5.2.2.3 Memory & Knowledge.

This domain evaluates whether the agent can retain information across steps and apply external knowledge to complete a task. It has three sub-dimensions. Objective Memory, with 16 tasks, tests the retention of factual, verifiable information. For example, “Check how many followers Lei Jun has on Douyin, Bilibili, and Weibo, and then send a private message to Lei Jun’s Douyin account” requires the agent to aggregate follower counts from three accounts. Subjective Memory, with 7 tasks, extends this to subjective information, such as “Summarize the core characteristics of the ‘Old Money Style’ in terms of color matching and fabric selection, and then go to Dewu to choose a sweater that best matches this style.” World Knowledge, with 10 tasks, examines whether the agent can apply commonsense reasoning to infer implicit constraints, such as knowing that “going to Harbin in January” requires purchasing winter clothing, or that “drinking and driving” implies the need for alternative transportation. These tasks require agents to act as informed assistants rather than mere instruction followers.

5.2.2.4 Complex Reasoning & Planning.

This domain combines long-horizon planning, multi-source information aggregation, and adaptive decision-making, and constitutes the most demanding capability in RealMobile. It has four sub-dimensions. Math & Logic, with 10 tasks, involves numerical reasoning within UI interactions, such as calculating total prices, comparing budgets, or computing time differences, as in “I have 3 hours of free time, help me calculate the maximum number of popular videos I can watch on Bilibili.” Multi-source Comparison, with 12 tasks, requires collecting information from multiple applications and making decisions based on optimization objectives such as lowest price, shortest time, or highest rating, as in “Compare the prices of the Xiaomi 17 Ultra on Dewu and Douyin Mall, and add the one with the lower price to my cart.” Complex Objective Planning, with 13 tasks, involves long-horizon execution in which each step has a clear correct answer, where the main challenge is maintaining task state across more than ten sequential actions without deviation, as in “Like Lei Jun’s latest posts on Douyin, Weibo, Bilibili, and Xiaohongshu,” which requires visiting four applications in sequence. Complex Subjective Planning, with 6 tasks, is the hardest sub-dimension, where instructions are ambiguous, goals are open-ended, and the agent must decompose the task independently using its own knowledge and reasoning. For example, “Please plan a two-day trip from Beijing to Tianjin for my family of five the day after tomorrow” provides no explicit steps, so the agent must determine what information to collect, which applications to use, and how to organize the final output. These tasks require the agent to integrate reasoning, planning, and execution over long horizons.

5pt

5.3 Evaluation Protocol↩︎

6pt

5.3.1 Fine-Grained Scoring↩︎

Traditional GUI agent evaluation usually relies on binary pass/fail criteria, which fail to capture partial progress in multi-step tasks. An agent that completes most of the required steps but fails at the final one receives the same score as an agent that deviates from the task immediately, offering little signal for diagnosis. To address this limitation, we design a fine-grained scoring protocol with four components.

5.3.1.1 Sub-goal decomposition.

For each task, we manually decompose the execution into a sequence of verifiable sub-goals, each corresponding to an intermediate milestone toward task completion. The decomposition granularity balances evaluation precision against annotation cost, typically 3 to 6 sub-goals per task. As shown in Table 2, the query “Give Movie Hurricane’s latest video on Bilibili a one-click triple support” is decomposed into 4 sub-goals: locating the latest video, liking it, favoriting it, and optionally tipping it.

5.3.1.2 Scoring mechanism.

The agent receives credit for each completed sub-goal, and all sub-goals within a task contribute equally. The final score is computed as \[\mathrm{Score} = \frac{\text{number of completed sub-goals}}{\text{total number of sub-goals}},\] yielding a continuous value in \([0,1]\) that reflects partial progress. For Task 3 in Table 2, for example, an agent that completes the first two of the five sub-goals but not the remaining three receives a score of \(2/5=0.4\).

5.3.1.3 Veto mechanism.

Certain errors are irrecoverable and invalidate the entire trajectory regardless of sub-goal completion. We define explicit veto conditions for each task, such as sending a message to the wrong contact, deleting user data, or performing an unauthorized financial transaction. Triggering any veto condition assigns the trajectory an automatic score of 0. For Task 1 in Table 2, the veto condition is triggered if the agent coins the video without explicit authorization or operates on a non-target video.

5.3.1.4 Conditional branching.

Real-world GUI tasks often admit multiple valid execution paths depending on the agent’s environment or account state. To accommodate this, we design sub-goals with conditional branches, allowing the agent to attain a full score through any valid path. In the final sub-goal of Task 1 in Table 2, for example, the agent receives full credit either for coining the video when coins are available or for confirming their unavailability when they are not, so that it is not penalized for factors beyond its control.

5.3.2 Executable Verification Rules↩︎

Existing evaluation strategies trade off robustness against scalability, with pure XPath matching being brittle to UI changes and programmatic state-checking requiring application-specific code. To balance these concerns, we propose a dual verification framework that combines XML structure matching with logical semantic rules, enabling both fine-grained sub-goal verification and flexible evaluation of semantically equivalent actions.

5.3.2.1 XML structure matching.

This component validates whether the expected UI elements are present or have been acted upon by evaluating XPath expressions over the UI hierarchy. For each sub-goal, we encode the expected element conditions as an XPath query. For instance, to verify that the agent has searched for a high-speed train from Beijing to Shanghai for the next day, we evaluate:

//*[contains(@package, 'ctrip') and contains(@text, 'Beijing') and
    contains(@resource-id, 'depart_station')] and //*[contains(@text, 'Tomorrow')]
and //*[contains(@text, 'Shanghai') and contains(@resource-id, 'arrive_station')]

This query is satisfied only when the departure city, arrival city, and travel date all appear together on the Ctrip page. To verify instead whether the agent clicked the “Add to Cart” button, we evaluate:

//*[(contains(@content-desc, 'add to cart') or contains(@text, 'add to cart'))
    and bbox_contains_point(../@bounds, $point)]

Here the matched element must carry the “add to cart” label and contain the agent’s click point within its bounding box, confirming that the agent acted on the correct control.

XML matching relies on the UI hierarchy, which does not always expose all on-screen text, as some elements lack text attributes and others render text as images. We therefore complement it with optical character recognition (OCR) over the screenshots to recover these text-based conditions.

5.3.2.2 Logical semantic rules.

While XML structure matching verifies individual UI elements and the interactions on them, it cannot capture higher-level constraints that require reasoning across multiple steps or applications. To address this, we introduce logical semantic rules that validate two such constraint types, namely sequential constraints and consistency constraints.

Sequential constraints ensure that operations occur in the correct order. In Task 3 of Table 2, for instance, the agent must identify the high-speed train and its departure station before querying the travel time to that station via Amap, since the query is only well defined once the station is known. Consistency constraints instead verify that information is preserved and propagated across steps. In Task 2 of Table 2, the three songs added in QQ Music must match those in the Xiaohongshu playlist, which we check by comparing the song titles entered in QQ Music’s search box against those collected from the Xiaohongshu page via OCR. Each constraint is implemented as a code function that the framework invokes during evaluation, enabling automated verification of task requirements that go beyond per-step element matching.

Figure 8 shows how the two mechanisms work together on two tasks from Table 2. Figure 8 (a) combines XML matching with a consistency constraint to validate the cross-application information flow in Task 2, while Figure 8 (b) applies XML matching to verify the sub-goals of Task 3 across Ctrip, Amap, and Hema.

5.3.3 Auto-Eval Pipeline↩︎

The verification rules are applied by an automated pipeline that takes a task and a GUI trajectory, namely a sequence of screenshots, XML documents, and actions, and produces a score in four stages. It first runs OCR over each screenshot to recover text that the XML hierarchy omits. It then checks the veto conditions against the recovered text and the XML hierarchy, terminating with a score of \(0\) whenever one is triggered. Otherwise it evaluates each sub-goal by matching the pre-defined XPath queries over the UI hierarchy and invoking the code functions that validate the sequential and consistency constraints. Finally it computes the score as the fraction of completed sub-goals.

6 Experiments↩︎

6.1 Experimental Setup↩︎

Xiaomi-GUI-0 adopts Qwen3-VL-30B-A3B-Instruct [6] as its backbone. The three training stages are applied sequentially, with each stage initialized from the checkpoint of the preceding one: SFT starts from the Instruct version, Step RL from the SFT checkpoint, and Agentic RL from the Step RL checkpoint.

6.1.0.1 Hardware and frameworks.

All experiments are conducted on 64 NVIDIA H100 GPUs, organized as 8 nodes with 8 GPUs each. The training pipeline is built on verl [27] as the reinforcement learning framework, with Megatron-Core [28] as the training backend and SGLang [24] as the rollout engine.

6.1.0.2 Training data.

The SFT stage uses approximately 1.2 million GUI step-level samples drawn from around 120 thousand trajectories collected on both real devices and Android emulators, together with 4.4 million grounding samples for fine-grained screen understanding. The grounding samples require the model to parse all visible elements on the screen and predict their coordinates, functions, textual content, and other attributes. The Step RL stage further introduces approximately 0.4 million GUI step-level samples drawn from around 40 thousand trajectories. The Agentic RL stage operates on several thousand tasks, whose trajectories are generated online through interaction with the environment rather than collected in advance.

6.1.0.3 SFT.

We train for a single epoch with a global batch size of 256 and a sequence length of 8192, using AdamW with a learning rate of \(1\mathrm{e}{-5}\) under cosine decay and 10% warmup.

6.1.0.4 Step RL.

We optimize the GSPO objective, sampling 16 responses per prompt through asynchronous SGLang rollout. The policy adopts a constant learning rate of \(1\mathrm{e}{-6}\) and an asymmetric clip ratio of \((3\mathrm{e}{-4}, 4\mathrm{e}{-4})\), and is trained for a single epoch with a batch size of 128 prompts. The reward follows the cascade design of Section 4.3.0.2, with the LLM-as-judge levels handled by a separately served reward model.

6.1.0.5 Agentic RL.

We optimize the GSPO objective in an interactive Android environment that combines hundreds of parallel emulators with a cluster of physical devices. Rollout is performed with asynchronous SGLang, sampling 16 responses per prompt. The policy adopts a constant learning rate of \(1\mathrm{e}{-6}\) and an asymmetric clip ratio of \((3\mathrm{e}{-4}, 4\mathrm{e}{-4})\). The curriculum sampler scores tasks following Section 4.4.0.2, maintaining a smoothed success-rate estimate with decay \(0.9\) and annealing \(\alpha\) from \(1.5\) to \(0.5\) and \(\beta\) from \(1.0\) to \(2.0\) under temperature \(\eta=1.0\). The training batch size is 32 prompts, with a maximum prompt length of 8192.

6.2 Grounding Results↩︎

Grounding evaluates whether a GUI agent can map a natural-language target description to the correct interface element. We evaluate Xiaomi-GUI-0 on a set of complementary static grounding benchmarks. ScreenSpot-V2 [19] covers standard mobile, desktop, and web grounding. MMBench-GUI-L2 [29] provides broad six-platform evaluation under basic and advanced instructions. OSWorld-G, together with its refined-instruction variant OSWorld-G-Refine [30], targets fine-grained desktop diagnostics. Unless otherwise specified, all results use a single-pass, screenshot-only setting without crop-and-zoom refinement, candidate proposal tools, self-consistency voting, or other test-time scaling, so the numbers reflect native grounding ability.

Xiaomi-GUI-0 is primarily optimized for real-world Chinese mobile and in-vehicle cockpit applications, whose interfaces and interaction patterns differ substantially from those emphasized by the four public benchmarks here, which are dominated by English and desktop-style interfaces. Despite this distribution mismatch, Xiaomi-GUI-0 remains competitive across all four. As summarized in Table 3, it attains 94.7% on ScreenSpot-V2, 82.7% on MMBench-GUI-L2, 58.7% on OSWorld-G, and 64.2% on OSWorld-G-Refine, comparable to recent strong baselines at a similar scale. The per-platform and per-subcategory breakdowns (Appendix 14) indicate that its advantages are concentrated on the form factors closest to its deployment target, namely mobile and web grounding. On the desktop-oriented OSWorld-G and OSWorld-G-Refine, where the distribution mismatch is most pronounced, the gap is not uniform across subcategories: Xiaomi-GUI-0 remains strong on layout understanding, reaching 73.8% and 81.3% respectively, among the highest layout scores in the table, while the lower aggregate scores stem mainly from the text-matching and refusal categories.

6.3 Navigation Results↩︎

Table ¿tbl:tab:main95results? reports navigation results on the real-device benchmark RealMobile and on AndroidWorld, where Xiaomi-GUI-0-30B-A3B achieves a success rate of 72.0% on RealMobile and 78.9% on AndroidWorld. Both numbers are reported as the mean over four runs to account for the relatively large evaluation variance on these benchmarks. On AndroidWorld, Xiaomi-GUI-0 attains the best result among the evaluated models, surpassing the UI-TARS, GUI-Owl, and UI-Venus series and exceeding the previously strongest UI-Venus-1.5-30B-A3B at 77.6%, which indicates that the proposed training pipeline strengthens long-horizon execution without sacrificing generalization on public benchmarks. On the more challenging RealMobile, Xiaomi-GUI-0 substantially outperforms locally deployable open-source models such as MAI-UI-8B at 33%, indicating that paradigms built on offline trajectories, emulators, or public benchmarks transfer poorly to real mobile conditions such as account-state changes, system dialogs, payment authentication, and risk-control interventions. Among closed-source systems, Xiaomi-GUI-0 exceeds Gemini 3.1 Flash at 58%, Claude Opus 4.7 at 60%, and Claude Opus 4.6 at 33%, and approaches frontier models such as Gemini 3.1 Pro at 85% and Seed 2.0 Pro at 80% despite their larger scale, which supports the value of a real-device closed loop for narrowing the gap between benchmark scores and real-world deployability.

Table ¿tbl:tab:ablation95results? further reports a per-domain breakdown across the four capability dimensions of RealMobile. On the Foundation domain, Xiaomi-GUI-0 reaches 100.0% success and matches the strongest proprietary models such as Gemini 3.1 Pro and Seed 2.0 Pro, indicating that basic UI operation is approaching saturation and no longer discriminates between capable agents. Safety & Reflection is the weakest domain for every evaluated model, with even Gemini 3.1 Pro reaching only 62.5% success, while Xiaomi-GUI-0 attains 43.8%, the highest among open-source models, indicating that safety-aware and self-corrective behavior remains a common bottleneck for current GUI agents. On the Memory & Knowledge domain, Xiaomi-GUI-0 reaches 66.7% success and leads all open-source models by a substantial margin, yet trails Gemini 3.1 Pro at 93.9% and Seed 2.0 Pro at 90.9%, suggesting that knowledge-intensive recall may depend more on model capacity, as larger and more general models tend to encode richer world knowledge. On the Complex Reasoning & Planning domain, it attains 80.5% success, approaching Gemini 3.1 Pro at 82.9% and exceeding Seed 2.0 Pro at 73.2%, whereas the strongest open-source baseline reaches only 31.7%, indicating that long-horizon planning at a deployable model scale can rival frontier proprietary systems.

max width=

max width=

7 Related Work↩︎

From Framework Agents to Native GUI Agents↩︎

Early GUI agents are typically constructed as multi-agent orchestration systems built upon general-purpose VLMs. Methods such as the Mobile-Agent series [37], [38] and PC-Agent [39] compose specialized planner, executor, reflector, and memory modules [40] around foundation models such as GPT-4o [8], Gemini [9], or Qwen-VL [41]. The performance of these approaches is largely bounded by the underlying VLM, and inference latency grows with module depth and the number of interaction rounds. Moreover, reflection resides in the runtime procedure rather than being internalized into model weights. To overcome these limitations, recent research has increasingly shifted toward native GUI agents, in which perception, grounding, planning, and execution capabilities are integrated into a single model through post-training. UI-TARS and UI-TARS-2 [2], [31], AutoGLM [42], GUI-Owl in Mobile-Agent-v3 and Mobile-Agent-v3.5 [1], [34], OpenCUA [43], Step-GUI [35], UI-Venus-1.5 [3], and MAI-UI [36] all follow this paradigm. Xiaomi-GUI-0 follows this native-agent line but adopts a distinct focus: rather than optimizing for static or emulator-dominated benchmarks, it targets the constraints of real mobile deployment, including authentic account states, system dialogs, payment authentication, risk-control interventions, network variability, and client-side state dynamics.

Real-Device and Virtualized GUI Benchmarks↩︎

Existing mobile GUI benchmarks can be broadly divided into emulator/sandbox-based environments and real-device evaluations. The former, represented by AndroidWorld [15], constructs reproducible tasks on emulators or controlled sandboxes and verifies success through programmatic checks such as accessibility trees, UI trees, or VLM-based screenshot judges. These settings offer scalable and reproducible evaluation, but their state distributions remain biased toward simplified environments and do not fully capture the account states, page dynamics, and business logic of real applications. To improve fidelity, simulation platforms such as MobileGym [21] and SimuWoB [16] approximate the pages, databases, and partial functions of commercial applications, enabling scalable and verifiable benchmarking. They nonetheless remain distinct from their real counterparts in state distributions and client-side behaviors. Real-device benchmarks, by contrast, evaluate agents directly on physical devices and live applications. MobileBench-OL [17] measures task execution, reasoning, and noise robustness in an online real-device environment, and Knowu-bench [44] examines interactive, proactive, and personalized behaviors. Our benchmark likewise runs on physical devices and live applications, but retains rule-based programmatic evaluation for reproducible judgments and further disentangles UI navigation, grounding, planning, long-horizon execution, memory, reflection, and safety constraints, improving its diagnostic value for real-world deployment.

RL-based GUI Agent Training↩︎

Early works such as UI-R1 [45], GUI-R1 [46], MobileGUI-RL [47], and Mobile-R1 [48] are among the first to explore curriculum-style training and R1-style reinforcement learning for GUI agents, shifting the paradigm from purely offline imitation learning toward interactive policy optimization. DigiRL [49] demonstrates that online reinforcement learning can substantially outperform supervised fine-tuning for device-control tasks, underscoring the value of interaction feedback. Building on this foundation, subsequent studies extend the paradigm along complementary directions: self-supervised RL that removes the dependence on dense action labels [50], iterative preference learning over reasoning trajectories [51], success-rate-aware trajectory-efficient policy optimization [26], process-level reward modeling for fine-grained credit assignment [52], and explicit error detection with backtracking for online recovery [18]. At scale, UI-TARS-2 [2] deploys thousands of parallel virtual machines to support large-scale RL rollouts, UI-Venus-1.5 [3] combines full-trajectory online RL with model fusion, and OpenClaw-RL [53] improves performance through asynchronous online RL. Together, these studies establish reinforcement learning as a central direction for enhancing long-horizon execution, error recovery, and environmental adaptability in GUI agents.

8 Conclusion↩︎

This report introduces Xiaomi-GUI-0, a native end-to-end multimodal GUI agent for real mobile environments. Motivated by the persistent gap between high benchmark scores and real-world usability, Xiaomi-GUI-0 is trained and evaluated within a closed loop centered on real-device execution, rather than on static benchmarks or simulated environments that only partially reflect deployment conditions. This loop couples three components. A real-device-dominant hybrid infrastructure treats physical devices as the primary execution environment, with sandboxes in a complementary role, so that data collection, training, and evaluation share an execution distribution close to real deployment. On top of this infrastructure, an error-driven data flywheel turns the failure patterns exposed during real rollouts into supervision for reflection and recovery, rather than merely scaling up successful trajectories. A progressive training recipe, proceeding from supervised fine-tuning through step-level to agentic reinforcement learning, then develops the model from basic interface operation toward long-horizon execution and error recovery. Across public grounding benchmarks, AndroidWorld, and our real-device benchmark RealMobile, Xiaomi-GUI-0 attains competitive perception and grounding and reaches a \(72.0\%\) success rate on RealMobile and \(78.9\%\) on AndroidWorld, while improving end-to-end success and robustness on real devices. These results suggest that real execution is most useful not as a final evaluation step but as a continual source of supervision, and that treating real rollouts as the center of training and evaluation is a practical path toward deployable mobile GUI agents.

Contributions and Acknowledgments↩︎

All contributors are listed in alphabetical order by their last names.

• Jian Luan

• Cong Zou

9 Data Collection Applications↩︎

Table 1 lists the applications covered during data collection. The applications are organized into three subsets: 100 high-frequency commercial mobile applications, 20 tablet and cockpit applications, and 20 AndroidWorld applications. These subsets respectively target everyday mobile usage, large-screen interaction patterns, and a stable benchmark environment.

10 Action Space↩︎

Xiaomi-GUI-0 operates over a compact unified action space that covers touch operations, text input, system navigation, user communication, and task termination, as summarized in Table 2. At each interaction step, the model selects exactly one action and emits its arguments as a JSON object. Coordinates are expressed as relative positions in \([0,1]^2\), with the top-left corner at \((0,0)\) and the bottom-right corner at \((1,1)\).

10.1 Anomaly Semantics↩︎

As summarized in Table 3, we consolidate more than fifty abnormal-state causes observed online into fourteen handoff semantics, each corresponding to a valid Fail.type value in the action space. When one of these states blocks reliable automation, the model is expected to stop, pause, or hand control back to the user rather than continue issuing UI actions.

11 Function Tree Example↩︎

Figure [fig:function95tree95sketch] presents a simplified function tree for the Bilibili mobile application. For clarity, the example enumerates the sibling functions available at each level while expanding only one representative branch to the next level. This structure captures both the set of functions that can be sampled at each node and the access path from a top-level application entry to a concrete function point.

12 Behavior-Bucket Examples↩︎

Behavior buckets provide the intent-level abstraction used during query synthesis. Whereas the function tree records which functions an application exposes and how each function is reached, a behavior bucket captures how users combine those functions into a coherent request. Each bucket corresponds to a stable usage motivation and contains several concrete behavioral phrases. To synthesize a query, the generator first samples a bucket and then binds concrete entities, filters, pages, or function-tree paths to produce an executable natural-language instruction.

12.1 Single-Application Example: Xiaohongshu↩︎

Table 4 presents a single-application behavior-bucket example for Xiaohongshu. All behaviors in this example are completed within Xiaohongshu, yet they span a range of user motivations, including content consumption, local exploration, shopping, creation, social interaction, creator growth, and personal settings.

12.2 Cross-Application Example: Weibo and Bilibili↩︎

Cross-Application behavior buckets are defined over an application pair or a larger application set. Rather than concatenating two independent single-application tasks, they specify a relation that must hold across applications, namely relay, contrast, or parallel completion. Table 5 presents a representative application-pair example for Weibo and Bilibili, where Weibo contributes real-time trend signals, social discussion, images, and fan-community activity, whereas Bilibili contributes longer-form videos, in-depth analysis, tutorials, replays, and fan creations.

13 System Prompt↩︎

# Role
You are a GUI interaction agent. You perceive the screen, review prior steps, and decide the most reasonable next action to fulfill the user's instruction.

# Input Context
1. User instruction
2. Interaction history
3. Current device type & foreground app
4. Current screenshot

# Available Tools
You MUST pick exactly one tool per step. Output the corresponding JSON string inside `<tool_call>`.
1. Tap: `{"name": "Tap", "position": [x, y], "times": 1}` (Tap at coordinate)
2. LongPress: `{"name": "LongPress", "position": [x, y]}` (Trigger contextual menus)
3. Swipe: `{"name": "Swipe", "start_position": [x1, y1], "end_position": [x2, y2]}` (Swipe to scroll/move. Swipe up to scroll down)
4. Type: `{"name": "Type", "position": [x, y], "text": "..."}` (Tap input box and type)
5. Search: `{"name": "Search", "position": [x, y], "text": "..."}` (Macro: tap -> clear -> type -> submit)
6. Open: `{"name": "Open", "app": "..."}` (Launch app via system)
7. Back: `{"name": "Back"}` (System-level back)
8. Home: `{"name": "Home"}` (Go to home screen)
9. Wait: `{"name": "Wait"}` (Wait for page loading/rendering)
10. Request: `{"name": "Request", "text": "..."}` (Ask user for clarification/confirmation)
11. Fail: `{"name": "Fail", "type": "...", "reason": "..."}` (Report failure. `<TYPE>` MUST be one of: LOGIN_REQUIRED, USE_GUIDANCE, CAPTCHA_VERIFICATION, RESULT_NOT_FOUND, BLUETOOTH_CONNECTION_REQUIRED, NETWORK_ERROR, PAYMENT_AUTHENTICATION, TASK_CANT_FULFILLED, REPEAT_OPERATION, PERMISSION_REQUEST, PASSWORD_REQUIRED, TAKEOVER_EXIT, TEMPORARY_TAKEOVER, MANUAL_VERIFICATION_REQUIRED)
12. Complete: `{"name": "Complete"}` (Confirm goal reached for non-Q&A tasks)
13. Speak: `{"name": "Speak", "text": "..."}` (Present final answer for Q&A tasks)

# Operational Constraints
1. Coordinate system: every `position` is a relative [x, y] in [0, 1] with 3-decimal precision. Top-left is (0, 0); bottom-right is (1, 1).
2. Dismiss unrelated pop-ups (ads, upgrade prompts, rating requests) by tapping their Close / Skip / X / "Later" button rather than calling Fail.
3. Loop breaker: if three consecutive steps cause no visible change, or the same action is repeating in a loop, self-correct (try Back or a different target). If self-correction fails, call Fail.

# Reasoning Framework (inside <think>)
Before emitting the action, reason inside `<think>...</think>` (omit steps if no new info):
1. [Observation]: Objectively describe the current App, page state, and key visible elements.
2. [Reflection]: (Optional) Include ONLY if the current screen deviates from the previous plan's expectation. Explain what was expected vs. what is actually seen.
3. [Plan] / [Plan Update] / [Replan]: (Choose one). Output a 2-4 step path in a single line separated by `|`. Mark completed steps with `[done]` and the current step with `->`. Use [Replan] if the previous plan failed.
4. [Decision]: Deduce the exact action based on the Observation and the current `->` step in the Plan.
5. [Memory]: Cache persistent info needed for future steps.

# Output Format
Your assistant message `content` MUST strictly follow this three-span XML-like shape, in this exact order, with no extra text before, between, or after:

<think>
[your observation, analysis and reasoning]
</think>
<action>
[a short natural-language description of the action, e.g., "Tap the search bar"]
</action>
<tool_call>
{"name": "Tap", "position": [0.521, 0.123], "times": 1}
</tool_call>

14 Detailed Grounding Results↩︎

This appendix complements the overall scores in Table 3 with the full per-platform and per-subcategory breakdowns for the four grounding benchmarks: ScreenSpot-V2 (Table 9), MMBench-GUI-L2 (Table 10), OSWorld-G (Table ¿tbl:tab:osworld-g?), and OSWorld-G-Refine (Table [tbl:tab:osworld-g-refine]). These breakdowns localize the sources of performance differences across platforms, instruction difficulty levels, and diagnostic capability dimensions that a single averaged score obscures. Specifically, ScreenSpot-V2 reports text and icon grounding across mobile, desktop, and web, while MMBench-GUI-L2 reports platform-specific accuracy under basic and advanced instructions. OSWorld-G together with OSWorld-G-Refine further characterize the model along text matching, element recognition, layout understanding, fine-grained manipulation, and refusal.

4pt

15 Case Study↩︎

To illustrate the behavior of Xiaomi-GUI-0 during real-device execution, we present two qualitative case studies on complementary aspects of the agent: end-to-end task completion and mid-trajectory error recovery.

The first case, shown in Figure 11, traces a complete execution process. Given a user instruction, the agent observes the current screen, decomposes the task into intermediate sub-goals, and emits a sequence of GUI actions until the goal is reached. The trajectory demonstrates that the unified action space supports the full pipeline from intent understanding to task completion on a physical device.

The second case, shown in Figure 12, illustrates the agent’s reflection and self-correction behavior. When the observed screen state deviates from the expected outcome of the preceding action, the agent records the discrepancy in its reflection field, revises its plan, and selects a corrective action rather than continuing the original trajectory. This behavior shows how explicit reflection enables the agent to recover from intermediate uncertainty during interaction.

References↩︎