:=IEEEBSTcontrol

simulation-based testing, mobile robotics, automated testing

1 Introduction↩︎

The responsible deployment of robotic systems—particularly mobile robots—requires rigorous validation prior to field operation. Validation determines whether a system satisfies requirements such as safety or functional correctness, typically framed as a pass/fail decision [1], but the reliability of these decisions depends on the diversity and scale of tested conditions. In practice, validation is commonly carried out using scenario-based testing, in which robotic system behavior is observed under concrete operating conditions [2]. A scenario is a series of interactions between a System Under Test (SUT) and its environment, capturing configurations of environmental conditions, tasks, system parameters, and contextual factors under which system behavior is evaluated [3]. Empirical studies show that developers deliberately vary such conditions to uncover corner cases and assess safety and reliability, yet scenario selection is often manual, experience-driven, and difficult to reproduce [4]. Simulation-based validation has therefore become central to scenario-based testing [5], [6], enabling controlled, repeatable exploration of operating conditions that would otherwise be costly, impractical, or hazardous [7]. If scenario variation and the number of runs are limited, conclusions may appear valid locally but fail to generalize globally.

In scenario testing, test coverage is defined as the percentage of executed scenarios out of the total number of scenarios [3], yet enumerating all possible scenarios is infeasible due to the combinatorial complexity of system configurations and environmental conditions. Scenario specifications often represent only partial descriptions of operating conditions. This is reflected in two common challenges in testing: How can meaningful test coverage be defined when the total scenario space is effectively unbounded? and how much testing is sufficient to draw reliable conclusions? Addressing both requires systematic exploration of the test space.

However, existing approaches often treat scenarios as monolithic test cases, rather than as compositions of environmental structure, task specification, system configuration, and contextual factors that characterize operating conditions. As a result, validation results remain tied to individual scenarios and cannot be reliably generalized. Without systematic variation across these dimensions and sufficient repetitions, scenario-based testing risks producing locally valid but globally misleading conclusions regarding requirement fulfillment and system performance. This motivates our central research question: How can scenarios be specified to account for these dimensions, their variations, and their systematic execution?

This paper makes the following contributions:

2 Related Work↩︎

Scenario-based testing is a fundamental practice for validating autonomous systems, particularly in autonomous driving systems (ADS) where explicit scenarios enable systematic exploration of operating conditions. Domain-specific languages such as Scenic and OpenSCENARIO DSL support probabilistic and parametrized scenario specifications [8], [9], and safety-focused studies highlight the need for large numbers of executions [10]. In robotics, scenario-based approaches have been applied to mobile robots [11], [12] and drones [13]. Recent work has addressed compositional scenario specification [2] and domain-specific languages for environment generation [14], [15]. Although parametrized scenarios are well established in autonomous driving, empirical studies show that testing robotic systems still faces substantial obstacles, including limited automation and difficulty in reproducing failures [5]. Our work aims to bridge this gap by describing how scenario specifications can be parametrized and composed through explicit variability dimensions and automated compositional producers, and demonstrates its implementation in a publicly available tool called RoboVAST.

Simulation-based testing is essential to modern robotics development [5], [6], yet empirical studies reveal persistent challenges including the simulation-to-reality gap, lack of reproducibility, and considerable resource costs [6]. While many robotics bugs can be detected in low-fidelity simulation [7], [11], existing test generation approaches [16], [17] and benchmarking initiatives [12], [18] typically address individual pipeline stages in isolation. Recent robot learning advances via visual-language-action models use GPU-accelerated environments like Nvidia Isaac Lab to produce large-scale execution datasets [19]–[21], demonstrating the value of scalable, systematic data collection pipelines that can exploit massive parallelism. In parallel, recent work shows that cloud infrastructure can be used to elastically scale robotics testing to thousands of parallel simulations without upfront hardware investment, amortizing costs across campaigns by paying only for used resources [22], yet existing solutions such as AWS RoboMaker remain predominantly commercial and closed-source [23].

Our work extends these efforts by supporting the full testing process: scenario specification, configuration, scalable execution, and evaluation, while introducing explicit variation points to systematically explore the test space.

3 Approach↩︎

To identify a scenario, scenario-based test design techniques often require an underlying model that describes how these interactions take place and/or what actions are performed by the SUT as a result of triggers from other systems [3]. However, because interactions are not discrete in robotics, and not all interactions are known in advance and some can only be discovered at runtime (e.g., due to noise, non-determinism), our approach specifies scenarios in terms of scenario elements – such as environment, tasks, system configurations – that can be varied and composed to cause a variety of interactions between the robot, the environment, and other agents or objects (e.g. dynamic obstacles). Our goal is to systematically validate these interactions by exposing the robot to a wide range of contexts resulting from these compositions. We refer to the associated planning, monitoring, and completion activities collectively as a test campaign. In the remainder of this section, we formalize the concepts in our approach and discuss their implementation in 4.

3.1 Scenario Templates and Variation Choices↩︎

Let us start by defining the scenario dimensions in the test space that we can vary and compose. Let \({\mathcal{D} = \{d_1{:}\tau_1, \dots, d_n{:}\tau_n\}}\) be a set of typed variability dimensions, where each variability dimension \(d_i{:}\tau_i\) has a domain \({\mathrm{Dom}(d_i) \subseteq \llbracket \tau_i \rrbracket}\). We consider the following dimensions: \(d_S{:}\Theta_S\) for the robot or SUT \(S\) with configuration space \(\Theta_S\), \(d_\mathcal{E}{:}\mathcal{E}\) for the set of environments \(\mathcal{E}\) in which it can operate, and \(d_T{:}T\) for the variations on the set of tasks \(T\). With \(d_S\) we account for the fact that robot systems are characterized by a huge amount of variability (e.g., varying sensor and actuator configurations, controller parameters, and software variants) [24]. A configured system instance is denoted as \(S(\theta)\) for \(\theta \in \Theta_S\). Environment variability in \(d_{\mathcal{E}}\) accounts for the distinction between the operating environment (sometimes referred to as world) \(w \in W\) and its environment representation \(e \in \mathcal{E}\), i.e., \(e\) is a model of \(w\). While the world \(w\) denotes the concrete physical environment, validation occurs on an environment representation \(e\) that captures exactly those features of an environment that are explicitly modeled, varied, and controlled. Possible representations for an environment \(e\) include structured descriptions of the environment geometry (e.g., maps or floor plans), static and dynamic entities (e.g., obstacles or agents), contextual parameters (e.g., dynamics, friction, and lighting), and semantic regions relevant for task specification. Finally, \(d_T\) captures task-related parameters (e.g., start poses, task specifications like goal poses or paths to be tracked, number of repetitions, etc.).

To enable the composable specification of scenarios that account for \(\mathcal{D}\), we use three levels of abstraction commonly used in testing terminology for test cases, as follows: An abstract scenario \(a\) describes the scenario in terms of abstract variables \(V\). At this level, the specification can be used to establish relations and constraints between scenario elements. A level down in the abstraction, a logical scenario \(b\) also describes the parameter ranges \({\mathcal{K} \rightarrow \mathrm{Dom}(\mathcal{D})}\) that can be assigned to \(V\). Finally, at the lowest level, a concrete scenario \(c\) represents a fully parametrized scenario instance with all possible values assigned by a function \(\mathcal{P}: \mathcal{K} \rightarrow V\). Note that we can define different \(\{\mathcal{K}_i \rightarrow \mathrm{Dom}(\mathcal{D}_i) | \mathcal{K}_i \subset \mathcal{K}, \mathcal{D}_i \subset \mathcal{D}\}\) to be assigned to \(V\).

Thus, we define a composable scenario specification as the tuple \({F = (a, \mathcal{K}, \Pi, \beta, \varphi, \gamma, \omega)}\), where \(\Pi\) denotes the ordered sequence of producers, \(\beta\) denotes the set of random seeds required by \(\Pi\), \(\gamma\) denotes the execution parameters, such as the number of runs \(\phi\), and \(\omega\) denotes the set of evaluation functions.

We refer to \(\Pi = \langle p_1, \dots, p_n \rangle\) as the ordered sequence of producers. This realization requires that \(\mathcal{P}\) only produces valid assignments \(\mathcal{P}(\mathcal{K}) \in \mathcal{V} (\mathcal{D})\), i.e., \(\mathcal{V}(\mathcal{D})\) represents the valid scenario space. Feasibility may require continuous dimensions to be discretized or sampled into a finite scenario space. We now define the set of admissible variation choices as \[\begin{gather} \label{eq:variation-choices} \mathcal{V}(\mathcal{D}) = \Bigl\{\, v : \mathcal{D} \rightarrow \prod_i \mathrm{Dom}(d_i) \;\Bigm|\; \\ \forall d_i \in \mathcal{D}; v(d_i) \in \mathrm{Dom}(d_i) \;\wedge\;\varphi(v) \Bigr\} \end{gather}\tag{1}\] where \(\varphi\) is a constraint predicate restricting admissible combinations of variability choices. Thus, \(\mathcal{V}(\mathcal{D})\) is a total assignment that fully specifies all variability dimensions, ensuring deterministic instantiation and enabling consistent comparison of execution outcomes across the explored scenario space. Constraints \(\varphi\) may relate dimensions in \(\mathcal{D}\), for example by requiring that the start-goal pair of a mobile robot navigation task (\(d_T\)) is reachable under the generated obstacle layout (\(d_{\mathcal{E}}\)) and compatible with the selected sensor configuration (\(d_S\)).

The full test space \(Q\) is the product of the dimensions in \(\mathcal{D}\), i.e., \(Q = d_S \times d_{\mathcal{E}} \times d_T\). \(|Q|\) is the number of all possible scenarios in the test space. Thus, we can define test coverage using the valid scenario space \({\mathcal{V}(\mathcal{D}) \subset Q}\) for \((\mathcal{K}, \mathcal{D})\) as \(\mathrm{coverage} =\left(\frac{|\mathcal{C}|}{|\mathcal{V}(\mathcal{D})|}\times 100\right)\%\), where \(\mathcal{C}\) is the set of concrete scenarios \(c \in \mathcal{C}\) composed from \(F\) and \({\mathcal{C} \subset \mathcal{V}(\mathcal{D})}\).

3.2 Instantiation Through Ordered Producers↩︎

Given a composable scenario specification \(F\), the concrete executable scenario instances \(c \in \mathcal{C}\) are obtained through a structured instantiation process by the function \(\mathcal{P}\). Each producer is a (possibly stochastic) function \({p_i : (\mathcal{K}, \beta) \;\rightarrow \mathcal{C}}\), where \(\mathcal{\beta}\) denotes the set of random seeds or stochastic parameters \(\eta\) controlling producer sampling procedures; making randomness explicit ensures reproducible scenario instantiation. Producers are applied in the fixed total order specified by \(\mathcal{P}i\), reflecting the dependencies between variability dimensions and ensuring their deterministic transformation and composition. The mapping is partial to capture constructive feasibility limits of \(\mathcal{V}(\mathcal{D})\): even when a variation choice satisfies the declarative constraints \(\varphi\), a producer may fail to generate a realizable instance under the previously produced conditions. For example, an obstacle-generation producer applied after map generation may fail if the requested number or size of obstacles cannot be placed without violating the clearance constraints. Such failures indicate unrealizable scenario instances and are treated as instantiation failures rather than as execution outcomes.

Instantiation is a deterministic loop over the ordered producers: starting from an empty configuration set \({\mathcal{C}^0=\emptyset}\), each producer \(p_i\) consumes the set of partial configurations \(\mathcal{C}^{i-1}\), and the seed \(\eta_i\), then applies its generation or variation function to all the partial configurations in \(\mathcal{C}^{i-1}\) for all the values in \(\mathcal{K}_i\), and returns an updated configuration set \(\mathcal{C}^{i}\). Formally, for \({F = (a, \mathcal{K}, \mathcal{P}, \beta, \varphi, \gamma, \omega)}\) with \({\mathcal{P}i = \langle p_1,\dots,p_n\rangle}\), this process is defined by

\[\label{eq:scenario-instantiation} \begin{align} \mathcal{C}^0 &= \{\emptyset\},\\ \mathcal{C}^i &= p_i(\mathcal{C}^{i-1}, \mathcal{K}_i, \eta_i) \quad \text{for } i=1,\dots,n,\\ \mathsf{Inst}(\mathcal{K},\beta) &= \mathcal{C}^n. \end{align}\tag{2}\] where \(n = |\mathcal{P}|\). 2 makes explicit how configuration fragments are accumulated across the producer order \(\mathcal{P}i\) in a combinatorial manner as a producer \(p_i\) generates a new partial configuration for each combination of \({\mathcal{C}^{i-1} \times \mathcal{K}^i}\). Hence, the instantiation process in 2 creates all the required test data for all the scenarios in \(F\).

3.3 Scenario Execution and Evaluation↩︎

Given an instantiated scenario \({c^{n} \in \mathcal{C}^n}\), the evaluation proceeds by executing the configured SUT. Execution is modeled as the relation \({\mathsf{Exec} : \bigl(S(\theta^n) \times \mathcal{E}^n \times T^n\bigr) \;\rightarrow\; \mathcal{R}}\) where \(\mathcal{R}\) denotes the space of raw execution outputs. Execution instantiates the environment representation \(\mathcal{E}^n\) as a simulation environment, configures the SUT \(S\) with parameters \(\theta^n\), establishes the required preconditions for \(T^n\) (e.g., the start pose), and produces execution outputs \(\rho \in \mathcal{R}\).

To decouple the execution backend from evaluation, the composable scenario specification \(F\) includes a set of evaluation functions that map the execution outputs to the oracle evaluation \({\omega:\mathcal{R}\rightarrow \mathcal{O}}\). This mapping is a two-step process. First, an observation function \(\mathsf{Obs} : \mathcal{R} \rightarrow \mathcal{T}\) maps the raw runs to execution traces. Then, the scenario traces are used by an oracle function \(\mathcal{O}\) to evaluate the scenario results. The definition and implementation of these functions are beyond the scope of this paper. In general, an oracle \(\mathcal{O}\) can map traces to pass/fail criteria \(\mathcal{O}: \mathcal{T} \rightarrow \{\mathsf{true}, \mathsf{false}\}\), or to metric functions \(m : \mathcal{T} \rightarrow \mathbb{R}\), among others.

Statistical evidence is obtained by evaluating these interpretations over trace sets for the number of runs \(\phi\): \[\label{eq:statistical-analysis} \bigcup_{i\in\phi}\mathcal{O}(\rho_i) \mid \rho_i \in \mathsf{Exec}(S(\theta^n), E^{n}, c^{n})\tag{3}\]

4 RoboVAST↩︎

To support developers in the validation activities throughout the development life-cycle, we present RoboVAST¹, a framework that implements our approach in 3. It can be used for planning (e.g., managing test coverage and/or test completion criteria in terms of \(\mathcal{V}(\mathcal{D})\) and the number of runs to be executed), monitoring and control (e.g., monitoring which test objectives have been met and by which \(c \in \mathcal{C}\) and for which \(p_i\)), and completion activities (e.g., making available test assets for later use). RoboVASTis a framework that implements scenario-based validation of robotic systems at scale. It addresses the central challenge of combining expressive scenario specifications with scalable execution to support robust, non-local conclusions about system behavior. Concretely, it realizes the methodology defined above through a workflow that combines declarative configuration and variation with Kubernetes-based parallel execution, spanning scenario specification, large-scale execution across many configurations and runs, and evidence interpretation in the form of validation verdicts and performance measurements (cf. 2). We first outline its core architectural roles and then detail the mechanisms that realize them in RoboVAST.

4.1 Conceptual Architecture↩︎

At the conceptual level, RoboVASTstructures testing into three complementary roles: (1) definition of test assets and evidence expectations, (2) declarative configuration and instantiation of scenario families through ordered producers and variability dimensions, and (3) parallel execution of configured system instances across many scenario variants using Kubernetes. This separation decouples what is evaluated (scenario space and criteria) from how it is realized (plugins, containers, and orchestration backends). RoboVASTtherefore provides concrete realizations of the composable scenario specification \(F\), the ordered producers \(\mathcal{P}i\), the instantiation function \(\mathsf{Inst}(\mathcal{K}, \beta)\), the execution relation \(\mathsf{Exec}\), and the observation and interpretation steps given by \(\mathsf{Obs}\) and \(\mathcal{O}\).

4.2 Declarative Specifications↩︎

RoboVASTemploys a YAML-based configuration file format as the primary artifact for the specification of \(F\). Concretely, the file declaratively specifies five distinct concerns: (1) metadata documenting name, agents and other information related to the test campaign such as description, creators, and licensing information; (2) configuration defining scenario families and their variability dimensions; (3) execution specifying containerization, resource allocation, and runtime environment; (4) results processing declaring postprocessing and publishing pipelines and (5) evaluation setting analysis-related parameters. This separation enables independent evolution of each aspect while maintaining a unified specification of the testing workflow.

The configuration section defines one or more scenario families, each with fixed parameters \(\mathcal{K}\) and/or declarative variations that induce systematic parameter sweeps as \(\mathcal{P}\) via plugin-provided variation types. Independent variation dimensions compose as a Cartesian product, automatically generating all admissible parameter combinations without manual enumeration. This declarative approach scales to high-dimensional parameter spaces while preserving explicit traceability from configuration to executed instances. An example vast file is shown in [lst:vast-file].

configuration:
- name: example
  variations:
  - FloorplanGeneration:
      floorplans:
      - environments/rooms/rooms.fpm
  - PathVariationRandom:
      num_goal_poses: 2
      path_length: 15.0
      seed: 20
  - ObstacleVariation:
      name: static_objects
      obstacle_configs:
      - amount_per_m: [0.4]
        max_distance: [0.0]
        args: width:=0.8, length:=0.8, height:=1.0
      seed: 42

4.3 Plugin-Based Variation System↩︎

RoboVASTimplements variability dimensions through an extensible plugin architecture. The plugin layer covers both general-purpose variation producers and domain-specific generators for mobile robot navigation. These plugins realize the producer functions \(p_i : (\mathcal{K}, \beta) \rightharpoonup \mathcal{C}\). General-purpose plugins support discrete value enumeration over finite lists and stochastic sampling from uniform or Gaussian distributions with configurable bounds. Each distribution-based plugin accepts an explicit random seed, ensuring reproducible sampling across campaign executions. For example, we model LiDAR uncertainty by adding zero-mean Gaussian noise with configurable standard deviation to the /scan laserscan topic and by simulating sensor failures via random dropout of individual range measurements, parameterized by a dropout probability.

Navigation-specific plugins realize these producers for concrete variability dimensions such as maps, tasks, and obstacle settings. FloorplanVariation and FloorplanGeneration use two user-authored artifacts: a Floorplan DSL definition and a corresponding Variation DSL to generate indoor environments with variation, for example by changing room or doorway dimensions via finite value sets or distribution-based sampling. The resulting artifacts, including 2D occupancy grid maps and 3D mesh representations, provide the successive environment states \(\mathcal{E}^{i}\) from the formal instantiation process. The PathVariation plugins generate start-to-goal paths that are guaranteed to be reachable in the generated map, either by stochastic pose sampling subject to path length and inter-pose distance constraints, or by exhaustive rasterization of the navigable space to enable systematic evaluation across the full environment topology. ObstacleVariation places obstacles along generated paths within a configurable proximity envelope. An example is provided in 3, as generated by the vast file presented in [lst:vast-file].

The combination of these plugins allows scenario specifications to cover critical corner cases and requirement-relevant conditions, and to validate performance metrics across a family of systematically varied scenarios. Moreover, the plugin system enables the extension to other test processes and activities, or other robotic use cases by developers of new producers.

4.4 Scenario Execution↩︎

Configuration generation and runtime parameter resolution realize the instantiation step in RoboVAST, implementing the formal instantiation function \(\mathsf{Inst}(\mathcal{K}, \eta)\). During configuration generation, RoboVASTapplies the producer plugins to produce the fully resolved OpenSCENARIO parameter values for every configuration variant. As certain producers are depending on others, e.g., the path generation can only succeed if a map was generated, plugins can share their data with others. As the configuration generation is time consuming but deterministic, given identical inputs and seeds, the results are cached internally. RoboVASTrealizes scalable execution through integration with Scenario Execution[25] and Kubernetes-based orchestration. This stage implements the execution relation \({\mathsf{Exec}(S(\theta^n), \mathcal{E}^{n}, T^{n}) \rightarrow \mathcal{R}}\). Scenario Execution is a middleware-agnostic framework that abstracts simulator-specific details [25]. It parses declarative scenario definitions from OpenSCENARIO DSL into behavior trees, which ensure that the necessary preconditions for each scenario are met, monitor and record the test run, and manage the post-execution test activities (e.g., ending a test due to time-outs, or terminating simulation and other processes). During execution, the instantiated environment representation \(\mathcal{E}^{n}\) is realized as a concrete simulation environment, and configured system instances \(S(\theta)\) interact with that environment over time.

To simplify the integration of external tools with RoboVAST  a multi-container approach is established. Besides the main container for handling data management, receiving the configuration and sending results, and executing the scenario, a developer can specify any custom container image. Depending on the requirements, only a minimal executable is needed that interacts through UNIX domain sockets with the main container. For each container, resource monitoring is automatically initiated to track metrics such as CPU usage and memory consumption. RoboVASTsupports both local execution for convenient development and debugging of test campaigns as well as distributed execution on clusters. Furthermore, the containerized architecture allows integration with arbitrary simulators, as long as they can be packaged in a compatible container interface.

To achieve scalability, RoboVASTdeploys execution via Kubernetes with Kueue for admission control and resource quota management. The execution workflow comprises four stages: (1) configuration upload to an S3 bucket; (2) Kubernetes Job creation for each configuration-run pair; (3) job queueing and admission by Kueue based on available cluster capacity and job resource requests; and (4) result collection via S3 download and optionally upload to other cloud storages. On startup, each job pulls its configuration files from S3, executes the scenario and finally pushes the results to S3 again. This scalability is essential to execute large-scale testing campaigns with many scenario instances, while the automated orchestration and data handling make managing such extensive campaigns feasible.

To handle substantial campaigns and subsequent analysis, execution produces a structured results directory with a well-defined layout. Each campaign execution creates a timestamped directory containing: (1) snapshot of all input files enabling campaign reproduction; (2) metadata about execution like execution time, duration, versions, and cluster information; (3) transient files that were created by producers; and (4) per-configuration directories with per-run subdirectories. Each run directory captures JUnit XML test results, resource usage time series, log files, and other files depending on the defined scenario (e.g. ROS bag files). This standardized layout decouples execution results from analysis tools, enabling backend-agnostic postprocessing and evaluation for both validation-oriented verdict computation and performance-oriented metric extraction.

4.5 Scenario Results↩︎

To keep extensive testing campaigns manageable from raw data to evaluation, RoboVASTincludes a dedicated results processing stage between execution and result interpretation. This stage bridges the raw run space \(\mathcal{R}\) produced by \(\mathsf{Exec}\) and the trace space \(\mathcal{T}\) consumed during interpretation. Its purpose is (1) to transform raw execution artifacts such as ROS bag files, logs, and test outputs into representations that are easier to inspect and analyze such as CSV tables; (2) generate combined metadata descriptions; (3) create or upload publication packages. This separation keeps execution backends independent from analysis logic and allows the same campaign results to be reused across multiple evaluations. In practice, this means that oracles can match a particular application, test objective, etc., by implementing the appropriate evaluation functions as postprocessing steps or notebooks, e.g., relying on derived event and oracle inputs, or on derived time series, aggregates, and summary statistics. For iterative analysis of large campaigns, results-processing artifacts are cached by input hash, allowing derived data to be reused when raw results and configurations remain unchanged. This enables iterative analysis without rerunning the full scenario campaign.

To make extensive campaign results accessible and interpretable, result interpretation in RoboVASTis centered on two approaches: A GUI tool that follows the structure of the output-data and provides developers and researchers the possibility to include custom Jupyter notebooks for visualization. These notebooks realize the observation and interpretation layer by mapping raw runs or processed artifacts to traces via \(\mathsf{Obs} : \mathcal{R} \rightarrow \mathcal{T}\), and implementing the oracles \(\mathcal{O}\). Examples include calculating variance from optimal paths or determining the minimum distance to obstacles over the course of a run. A second option is a Model Context Protocol server, which provides a comprehensive set of tools and resources to large language models to query information. Besides general search and query functions that allow processing of relevant campaign and run data, the navigation plugin provides specific navigation-related tools like map drawing and behavior evaluation.

5 Case Study Dataset↩︎

We evaluate our approach by creating an objective-oriented dataset for mobile robot navigation, reporting results across its scenarios. Following our methodology, each sub-dataset is defined as a set of scenarios with explicit variation dimensions and constraints, and each is aligned with a specific testing objective.

5.1 Structure and Objectives↩︎

We organize our test campaign in three vast files to demonstrate three distinct test sub-processes. As such, together with the osc file, they describe a specific set of configurations that we use to validate a test objective for the same SUT and same test basis – knowledge used to derive the tests – i.e., “test behavior, inputs, environment, pass/fail criteria”, etc. Our objective is to validate three software qualities: Performance, Robustness and Safety. Our SUT is a TurtleBot 4 using ROS2 executing navigation tasks. The datasets use five distinct maps, shown from left to right in 4. Unlike Nav2’s own system tests, which use a single arbitrary sandbox map, we employ multiple environments to expose failures that depend on environment geometry. The second map is derived from a real-world field-testing campaign in a 7000 m² university building [15], providing realistic indoor geometry, while the remaining four are synthetic: a hospital-like lobby, an office-inspired layout from the Floorplan DSL examples, and an artificial corridor network and nested-rooms environment created specifically to stress geometry-induced failure modes.

As a basis we use the default Nav2 configuration and the behaviors of the Nav2 actions NavToPose and FollowWaypoint for single-goal and multi-goal scenarios, respectively. We use the feedback from these actions as an initial criterion for success and failure, namely if the robot reaches within the default Nav2 goal tolerance of 0.15 m of the final goal pose. For each scenario instance, execution traces capture robot state, control behavior, task progress, and failure or safety events. Our oracles \(\mathcal{O}\) are implemented in .ipython notebooks that consume data from postprocessed rosbags, generating metrics dependent on the dataset.

Our three datasets are summarized in 1. Variations for the Nav2 parameters refer to the robot radius for the global costmap, with the default value of 0.22 m; we add an additional variation for 0.175 m. The message drop variations have the values 0.0, 0.05, 0.1, & 0.15, and noise 0.0, 0.05, 0.1, & 0.15 standard deviations.

The performance dataset validates path optimality by quantifying the deviation between executed and planned paths as the root mean squared error (RMSE) between the executed trajectory and an A* reference path sampled along arc length. Navigation tasks use 100 rasterized paths plus additional random paths of varying lengths and two global costmap robot-radius values; the environment is inflated by the robot radius before planning to obtain collision-free A* paths. For each scenario, we derive an RMSE threshold by multiplying the A* path length by the Nav2 local costmap robot radius; runs whose RMSE exceeds this path-length–scaled threshold are classified as performance failures, and the rest as successes.

The robustness dataset aims to validate the robustness of the SUT against sensor degradations by introducing message drop and noise to the laserscan. By using the FollowWaypoint action, we can vary the number of times a task is repeated, which increases the total execution time and thus the opportunities for navigation failures to accumulate; the default with no repetition is loops\(=0\), and we add one variation that repeats the goal sequence once (loops\(=1\)). Over longer runs under degraded sensing, even small errors in perception or control can build up, so repeating the same set of goal poses provides a direct probe of robustness over time. Our pass/fail criterion uses a fixed localization-error threshold based on the robot’s size: a run is classified as a robustness failure if its summed localization error (the accumulated difference between the robot’s estimated pose and its ground-truth pose over the run) exceeds 0.3 m, corresponding to the robot’s diameter; otherwise, it is counted as successful.

The safety dataset validates two safety properties of the SUT: collision and safety-response criteria. It further characterizes safety-related motion behavior by quantifying how close the robot comes to obstacles under safety violations, thereby assessing not only whether a collision occurs but also the margin by which unsafe behavior manifests. We focus on causing safety violations using the NavToPose action (with a single goal) while adding variations with static and dynamic obstacles in the SUT’s path. For static obstacles, we introduce variations based on density (0, 1, & 3 per 5 m) and maximum distance from the A* planned path (0 m, & 0.3 m), with all static obstacles modeled as cubes of size 0.8 m. For dynamic obstacles, we add a single cube of size 0.5 m directly on the SUT’s path (distance to path 0 m), spawning it at either 1 m or 2 m ahead of the robot along the A* planned path. We use a randomization factor of 3 for both types of obstacles, indicating the number of different obstacle position variations that are generated. Our pass/fail criterion is derived from the default TurtleBot 4 Nav2 configuration: with a maximum speed of 0.5 m s⁻¹ and a maximum deceleration of 2.5 m s⁻², we obtain a stopping distance of 0.05 m. We then define a minimum safe separation as the sum of the obstacle radius (computed as the maximum distance from the cube center to a corner), the robot radius of 0.15 m, and this stopping distance; a run is classified as a safety failure if the robot ever comes closer to any obstacle than this minimum distance during execution.

We executed all 3 sub-datasets on an auto-scaling Google Kubernetes Engine cluster with maximum 20 C4 worker nodes and a total of 1920 CPUs in approximately 7 h. We used 20 runs per configuration to obtain statistically meaningful estimates of configuration-specific success rates. Under a simple binomial proportion model with independent runs, 20 repetitions bound the standard error of the estimated success probability by approximately 0.11 in the worst case (around \(p = 0.5\)), yielding 95% confidence intervals of width about \(\pm 0.22\). This level of precision, obtained via the normal-approximation confidence interval for binomial proportions, is sufficient to distinguish configurations that fail systematically (e.g., success rates \(\le 0.2\)) from those that succeed reliably (e.g., \(\ge 0.8\)). Per run, two CPUs were allocated to both the simulation and navigation containers, and one CPU was allocated for scenario execution. The gathered data was limited to essential data, omitting large data sources such as laser scans or camera images. In total, approximately 737   of data simulating 1800 h and 1873 km of robot movement was captured.

5.2 Dataset Analysis↩︎

We analyze the outcomes of the validation campaigns based on dataset-specific success criteria and corresponding success rates, excluding startup failures as they do not relate to the navigation stack. The safety dataset yielded 15901 successful and 10065 failed runs (61.24% success); notably, 9463 of the failing runs (94.02%) belong to scenario variants in which all 20 runs fail, indicating that most safety violations are systematic (occurring at high obstacle counts or with dynamic obstacles) rather than random. The robustness dataset demonstrates 26041 successful and 5409 failed runs (82.80% success) with 32.80% of the failures coming from total-failure variants. This indicates that robustness suffered more from random than systematic failures, explainable by the added sensor dropout and noise and long task durations. For the performance dataset, results show 31714 successes and 609 failures (98.12% success), with 214 failing runs (35.14% of all failures) originating from systematically failing variants.

The heatmaps in 4 visualize the spatial distribution of failures by aggregating final poses of failed runs into failure densities with a blur radius of 0.2 m. While the success-rate analysis excludes startup failures, the heatmaps include them to show where execution failures occurred spatially. Regions with high failure density are not simply the most frequently traversed: many successful routes pass through areas with few failures, indicating that failures are tied to specific map features rather than visit frequency. Failures occur most often at corners, doorways, and near walls, whereas open areas and straight hallways rarely fail; dense spots inside rooms predominantly correspond to startup failures where the robot crashes or terminates before moving.

The complete postprocessed dataset is available at https://doi.org/10.5281/zenodo.19042250.

6 Conclusions↩︎

RoboVASTis an open-source framework for scalable scenario-based validation of robotic systems. It enables compositional scenario specifications with explicit variation dimensions, supporting large numbers of scenarios for broad test coverage and many runs per scenario to obtain statistical confidence in the resulting safety, robustness, and performance metrics under systematically varied operating conditions.

We demonstrated the approach on a mobile navigation system and released a dataset comprising 5480 distinct navigation scenarios and over 100000 runs, covering approximately 1800 h of simulated operation and 1873 km of virtual robot motion; the dataset and toolchain are publicly available to support further research on large-scale robotic system validation. Our current evaluation is limited to a single robot platform and navigation stack in quasi-static environments without rich interaction dynamics or multi-robot coordination. Future work includes addressing the sim-to-real gap via comparison of simulated results with real-world tests. We also plan to support additional simulators such as Isaac Sim and O3DE alongside our existing Gazebo integration, extend RoboVASTto other robotic use cases (e.g., manipulation and multi-robot fleets), and explore more systematic combinatorial strategies such as all-pairs testing for covering high-dimensional variation spaces.

References↩︎