1 Introduction↩︎

Teams of robots have a natural application in scenarios that benefit from strength in numbers [1]–[4]. When tasks involve robots supporting human operators, we anticipate the operators to provide abstract commands for the task (e.g., “explore this zone and find the missing people”), and supply semantically relevant hints (e.g., “there are likely ten people on the other side of the damaged bridge”). Given such information, human teams excel at collaborating and reasoning through language by decomposing, distributing and sequencing sub-tasks, which is a highly sought-after capability for teams of robots. The goal of this paper is to develop a single, light-weight model that enables decentralized coordination and reasoning in teams of robots commanded through natural language specifications (Fig. 1).

A fundamental challenge in realizing such multi-robot autonomy is the gap between the natural-language expression of a task and its representation as an algorithm that robots can execute [5]. Early work involved extensive manual design ofone-to-one correspondences with algorithmic primitives [6], [7], and thus had limited reasoning capabilities. In contrast, language models today can side-step such design by encapsulating vast amounts of human language data. Large multi-modal and language models (LMMs/LLMs) have recently shown excellent reasoning, disambiguation and even planning for single- and multi-robot tasks [8], [9]. However, their computational demands still make them inaccessible to practical, deployable robot platforms. While some recent work has developed methods to effectively extract capabilities from these models into more succinct representations [10]–[13], they depend on an infrastructure that can be hard to adapt to real-time, dynamic and unforeseen settings.

In this work, we bridge the gap between high-level reasoning and its real-time decentralized multi-robot implementation. We leverage the observation that robot tasks are algorithms: algorithms can be represented using automata, which can, in turn, be encoded by recurrent neural networks (RNNs). We train this RNN offline using task decompositions obtained from an LLM, distilling its reasoning capabilities and aligning its internal states with the tasks’ semantics. This leads to a single, tiny RNN that encapsulates a variety of multi-robot tasks in its internal state. This internal state is then used to condition a graph neural network-based policy trained using multi-agent reinforcement learning, thus connecting the reasoning process to a lower-level decentralized execution of the sub-tasks in a feedback loop. We evaluate this via simulations and real experiments on multi-robot problems that require reasoning and sequential decision-making. Our results demonstrate a first in achieving onboard, real-time, decentralized multi-robot reasoning and collaboration—all from a natural language command.

2 Background↩︎

Early work on interpretable natural-language commands [6], [7], [15]–[17] focused on developing ontologies with one-to-one correspondences with algorithmic primitives. However, this requires intense manual design and cannot accommodate ambiguous intent. LLMs [18]–[20] and LMMs [21]–[23] overcome this by encapsulating diverse human knowledge and style into one model. Hence, they exhibit effective ‘reasoning’ and flexibility to handle ambiguous yet expressive language prompts in both single- and multi-robot problems [8], [9], [24], [25]. For instance, LLMs can be used to enable algorithmic tasks grounded in semantics [26], or create formations [27] hard to specify in a metric space. Although effective, these works assume an infrastructure to connect the robots with the LLM/LMM in real-time.

To alleviate this, recent work has developed computationally efficient alternatives that rely on distillation, and distributed coordination [10]–[13], [28]. For instance,it is possible to distill an LLM into a smaller model that captures the relevant skills for robot task decomposition [10]; however, the decomposition cannot integrate feedback in real-time. Meanwhile, solutions using reinforcement learning allow onboard deployment but do not handle team-level prompts or credit assignment [11]. Yet another alternative is language translation: LLMs have shown surprising skills in translating human commands into linear [29] or signal temporal logic [30], or planning domain definition language [31] formulas. However, formal-based approaches need an initial time-consuming offline phase for each formula to be solved. Besides, they do not generate the low-level policies that actually solve the sub-tasks or feed back to the reasoning module to handle unexpected phenomena. Achieving decentralized ‘in-the-loop’ execution starting from natural, team-level commands is, thus, an open challenge.

Our solution overcomes these by distilling LLM-generated sub-task logic into a single tiny RNN, drawing on the connections between automata theory and recurrent models [32]. By employing a graph-based low-level RL policy for each sub-task, we enable both decentralization, and real-time feedback that effects transitions in the automata.

3 Preliminaries↩︎

In this work, a team of \(\mathsf{N}\) robots is given a task as a natural-language instruction \(\mathcal{W}\) (e.g., “Guide all people back to base”), which is typically a composition of some logically distinct ‘sub-tasks’, \(w\) (e.g., “search wide”, “navigate group to base”, …). The team has local connectivity, formalized as a time-varying, undirected graph \(\mathcal{G}_k = ({V}, {E}_k)\), where \({V} = \{1, \ldots, \mathsf{N}\}\) is the set of robots and \({E}_k \subseteq {V} \times {V}\) is the set of edges. We define the set of neighbors of robot \(r\) at time \(k\) as \({N}_{r,k}\). The objective is to train control policies that complete \(\mathcal{W}\), relying only on information available through \(\mathcal{G}_k\). Solving \(\mathcal{W}\) involves reasoning and solving sub-tasks sequentially, which we formalize as a Deterministic Finite Automata (DFAs). To learn to solve each sub-task effectively, we assume that only a reward signal is available, which we formalize as a Decentralized POMDP (Dec-POMDP).

Deterministic Finite Automata. A DFA is a tuple \(\mathcal{A} = (\mathcal{H}, \Sigma, \delta, h_0, \mathcal{F})\), where \(\mathcal{H}\) is a finite set of sub-tasks to be solved; \(\Sigma\) is a finite alphabet of inputs, with each proposition \(p \in \Sigma\) a boolean that describes an event that can trigger a transition between states; \(\delta: \mathcal{H} \times \Sigma \to \mathcal{H}\) is a transition function such that \(h_{k+1} = \delta(h_k, p_k)\), with \(h_k, h_{k+1} \in \mathcal{H}\) and input \(p_k \in \Sigma\); \(h_0\in \mathcal{H}\) is an initial state; and, \(\mathcal{F} \subseteq \mathcal{H}\) is the set of accepting states modelling the completion of the task. Note that individuals in a multi-robot team completing the same task may step through \(\mathcal{A}\) separately. We develop \(\mathcal{A}\) as a computational representation of \(\mathcal{W}\), and thus its states, \(h \in \mathcal{H}\), are representations of sub-tasks \(w\). The typical challenge is that \(\mathcal{A}\) is unknown, and constructing it from \(\mathcal{W}\) involves deduction and logic reasoning. While the sub-tasks may be describable as Markov processes, switching between them requires memory, captured by DFAs.

Decentralized POMDP. Given a DFA, the team must also be able to collaborate and complete the sub-tasks within it. We define each sub-task \(h \in \mathcal{H}\) as a Dec-POMDP \(\mathcal{M}_h \kern -0.1cm = ({V}, {S}, \{A_r\}_{r\in V}, T_h, R_h, \{\Omega_r\}_{r \in V}, O, b_h, \gamma_h)\), with a finite set of robot states \(S\), and actions \(\{A_r\}_{r\in V}\), such that \(A = \times_{r \in V} A_r\) is the joint action space. Functions \(T_h = P(s_{k+1}=s^{\prime}|h, s_k=s, a_k=a)\) and \(R_h: S\times A \rightarrow \mathbb{R}\) describe the dynamics and the reward of the sub-task \(h\). Observations are described by the finite set \(\{\Omega_r\}_{r\in V}\). Finally, the function \(O = P(o_{k+1}=o|s_{k+1}=s^{\prime}, a_k=a)\) describes the measurement model of the team, \(b_h\) defines an initial probability distribution of the sub-task, and \(\gamma_h\) is a discount factor. In a Dec-POMDP, the goal is to find a policy \(\pi_h\), \[\pi_h = \arg \max_{\pi} \;\; \mathbb{E}\left[\sum_{k=0}^{\infty} \gamma_h R_h | b_h, \pi\right],\] such that the sub-task \(h\) is solved.

Problem formulation. Our objective is to bridge the gap between high-level reasoning and low-level multi-robot execution by (i) obtaining a mapping from natural language instructions, \(W\), into a structured computational representation (i.e., a DFA), and then, (ii), learning a control policy that interprets the states of this representation (i.e., the sub-task sequence) to solve the overall mission. This solution must be decentralized, allowing each robot to utilize real-time feedback from its local environment and neighbors to progress through the automaton asynchronously.

We recognize (and indeed use) the fact that LLMs/LMMs today broadly possess the representational capacity to solve [prob:prob] by, for instance, autoregressively generating action sequences for each robot at each time-step, adapting them with new information. This requires persistent low-latency connection to servers, which is wasteful, uneconomical and often infeasible. Our objective instead is to produce a small neural network \(\pi\) that implements a solution to \(\mathcal{W}\) as a robust real-time feedback system easily distributed onto each robot.

4 Automata-based Reasoning and Collaboration for Robot Teams↩︎

To solve Problem [prob:prob], we propose a novel framework that works in three steps. First, an automated dataset generation phase queries an LMM with multi-robot tasks/problems, and compiles a dataset of the logic and sub-tasks needed to solve them. Second, a supervised learning phase distills the reasoning of the LMM by training an RNN that encapsulates the desired collection of automata in the dataset. Third, a reinforcement learning phase learns decentralized control policies that interpret the internal state of the RNN and solve the sub-tasks encoded in that internal state. Fig. 2 depicts a schematic of the proposed methodology, which we now elaborate in the following three subsections.

4.1 Automatic generation of reasoning data↩︎

Our first step is to generate a dataset \(\mathcal{D}\) that captures diverse tasks and their underlying logic. We will denote the set of tasks as \(\mathcal{T}\), with \(i \in \mathcal{T}\) a task. To generate \(\mathcal{D}\), we need a method that is conducive to learning the relationship between the language specification of a task and its representation as a set of sub-tasks and events in the automata.

For each task \(i\), we first prompt an LMM with a task description. The LMM then generates a formal decomposition for each of these tasks as scripts. For each task in \(\mathcal{T}\), we use its formal decomposition to perform rollouts as random walks and collect \(\mathsf{L}\) sequences of length \(\mathsf{K}\) comprised of sub-tasks and the events that triggered the transitions between sub-tasks. We represent sub-tasks as one-hot vectors \(h^l_{i,k}\), their corresponding sentence language embedding \(e^l_{i, k}\), and the atomic propositions \(p^l_{i, k}\) that represent the events as one-hot vectors. We also collect \(\mathcal{E}^l_i\), the sentence language embedding associated to the high-level natural-language command \(\mathcal{W}^l_i\). The dataset is a collection of successful steps that fulfill the tasks, \(\mathcal{D}= \{\{\mathcal{E}^l_i, \{h^l_{i, k}, e^l_{i, k}, p^l_{i, k}\}_{k=1}^{\mathsf{K}}\}_{l=1}^{\mathsf{L}}\}_{i\in\mathcal{T}}.\)

As shown in 2 (top), we automate the dataset generation by grounding the natural-language specifications \(\mathcal{W}^l_{i}\) in the actual environment that will be used to train the reinforcement learning policies. Instead of manually handcrafting each \(\mathcal{W}^l_{i}\), we procedurally generate \(\mathsf{L\cdot{}M}\) scenarios of the tasks to be solved. We then feed their features, including visual information, to the LMM with a prompt that requests natural-language specifications \(\mathcal{W}^l_{i}\) that match the scenarios.

Detailed examples of prompts and outputs of the LMM models can be found on the website. This process not only automates the data generation step, but also grounds it in the actual environment that the robots will operate in. The dataset generation process is conducted offline and only once. We now detail the process by which a neural network can be trained to distill the task-relevant logic and reasoning contained in \(\mathcal{D}\), so that it can interpolate and generalize across tasks and unseen scenarios.

4.2 Distilling reasoning with RNNs↩︎

Recent work has established a theoretical connection between automata and a second-order RNNs to prove that a network with sufficient capacity can represent any automaton [32]. This connection can also be applied to classical recurrent models like gated recurrent units, long short-term memory models or attention-based networks [13]. Consequently, we can train a sufficiently expressive recurrent model, in a supervised manner, to represent an automaton from \(\mathcal{D}\)—thus building a compact translation from a natural-language specification into an algorithm. This key insight underpins the model described in this section.

Given the dataset \(\mathcal{D}\), we train the parameters \(\psi\) of the RNN that operates according to \[x_{k} = f_{\psi}\big(x_{k-1}, p^l_{i, k} \; \big | \, \mathcal{E}^l_i),\] where \(x_{k}\) is the internal state of the model at time \(k\). The goal of training \(f_{\psi}\) is to encode the sub-tasks and transition logic of all automata \(\{\mathcal{A}_i\}_{i \in \mathcal{T}}\) contained in \(\mathcal{D}\). By explicitly conditioning on \(\mathcal{E}^l_i\) at every step, we ensure that incoming events and state transitions are learned in the context of \(\mathcal{E}^l_i\). This also ensures that latent states corresponding to different tasks remain separated in the latent space: even if two sequences from distinct tasks pass through identical event patterns, the task embedding \(\mathcal{E}^l_i\) shifts their trajectories in the latent space so that \(x_{k}\) for one task does not overlap with \(x^{\prime}_{k}\) from another.

To learn \(\psi\), a decoder projects \(x_{k}\) into a predicted one-hot state representation \(\hat{h}^{l}_{i,k} = d_{\phi}\left(x_{k}\right)\), which is trained to approximate the ground-truth label \(h^{l}_{i,k}\). The decoder \(d_{\phi}\) is implemented as a single-layer MLP, deliberately kept shallow to ensure that all temporal reasoning and semantic grounding remains encoded in the latent of \(f_{\theta}\). Finally, the loss \(\mathcal{L}({h}^{l}_{i,k}, \hat{h}^{l}_{i,k})\) is a multi-class cross entropy loss that penalizes incorrect state trajectories.

4.3 Decentralized collaboration with GNNs↩︎

Now that we have distilled the reasoning capabilities of LMMs into a single light-weight model, we must find policies that understand the latent representation of the automaton \(x_k\) and interpret it to generate low-level control actions that solve the sub-task dictated by \(x_k\). In our problem, we assume a Dec-POMDPs setting where we only have access to a reward signal. Therefore, to solve Problem [prob:prob] we propose a policy \(\pi\) that is a composition of the RNN \(f_{\psi}\) trained in Sec. 4.2 and a language-conditioned multi-task policy \(g_{\vartheta}\), \(\pi \equiv f_{\psi} \circ g_{\vartheta}\). We learn \(g_{\vartheta}\) using reinforcement learning. Specifically, at time-step \(k\), the action of robot \(r\) is \[\label{eq:control} a_{r,k} \sim g_{\vartheta}\big(a_{r, k} \,\big|\, \{o_{r^{\prime}, k}\}_{r^{\prime}\in N_{r,k}}, \; x_{r,k}\big),\tag{1}\] where \(x_{r,k}\) is the RNN’s latent state encoding the current sub-task, and \(\{o_{r^{\prime}, k}\}_{r^{\prime}\in N_{r,k}}\) is an observation vector from robot \(r\) and its neighbors at time \(k\). We denote the latent state with a subscript \(r\) because, for a decentralized implementation, each robot carries its own local copy of the RNN. This not only enables decentralized execution, but also lets robots use local information to step through their automata asynchronously when needed. On the other hand, the policy only depends on the local observation of robot \(r\) and information received from neighbors. Therefore, we parameterize \(g_{\vartheta}\) as a GNN that naturally handles time-varying communication graphs.

In learning \(\vartheta\), we take advantage of the dataset, where the RL scenarios are automatically paired with sentence embeddings. This lets us apply standard multi-agent reinforcement learning algorithms like Multi-Agent Proximity Policy Optimization [33] to train the multi-task policy, provided that every episode randomizes the sub-tasks and the reward signal \(R_{i,h}\) accordingly. Hence, unlike existing multi-task policies that operate over entire episodes, we train \(g_{\vartheta}\) to act over short temporal segments corresponding to individual sub-tasks. At execution time, an appropriate sub-task policy is selected by conditioning on the latent state of the RNN, which encodes the sub-task in the task automaton. With a sufficiently expressive model \(g_{\vartheta}\) we can capture the solution of all sub-tasks; by restricting training to individual sub-tasks, the computational load during training is reduced. High-level decision-making is delegated entirely to the RNN, allowing the multi-task policy to focus on low-level execution in the context of its current sub-task.

Lastly, \(o_r\) contains information from sub-task related measurements (e.g., position of obstacles) and event information related to the atomic propositions of the automata (e.g., a switch turns on). The latter can be shared among neighbors to improve the reactivity of the local RNN at each robot and the efficiency of the team overall. We therefore modify the input to the RNN from the observed \(p^l_{i,k}\) to an aggregated \(\Bar{p}^l_{i,k}\) that includes information from neighbors as well. The vector \(\Bar{p}^l_{i,k}\) is obtained from a general aggregation operation \(\doubleoplus\) (see Fig. 2 (middle and bottom)) that can be implemented in multiple ways. In the most general case, \(\doubleoplus\) can be learned, for instance, as another GNN. For some use-cases, it can be reduced to simple boolean logic such as AND or XOR; or it can even be extracted automatically from the language command, although we leave this option as future work.

5 Evaluations↩︎

Our pipeline accepts as input a task as a natural-language command, and leads a robot team to execute it. We encode numerous task automata into a single RNN that represents their logical decompositions, and train the decentralized policy conditioned on RNN states. The combination of the two makes for a powerful recipe—we bridge the gap between language-driven reasoning and its real-time distributed implementation as feedback control. Thus, we now evaluate our framework for (i) its correctness in representing the various task automata, and, (ii) its ability to complete a given multi-robot task, even when presented with disruptions, potentially long/repeated sub-task sequences, and real-world noise. We use the same RNN to encode all the tasks presented in this section³.

5.1 Automata Representation↩︎

Since the RNN encodes numerous automata, we first want that given a language input, \(\mathcal{W}_i\), the model to be initialized into the correct state, \(h_{i,0}\), of automata \(\mathcal{A}_i\). Then, subsequent state transitions must follow the logic of the task, and never produce a wrong state \(h_{i^{\prime},k}\) from a different task \(i \neq i^{\prime}\) even when presented with events from \(i^{\prime}\). The RNN is a Gated Recurrent Unit with a 1024 hidden layer and a projected language embedding with a \([512, 512, 64]\) MLP.

3 shows evaluation sequences for seven different tasks, \(\mathrm{\mathbf{T}\{0..6\}}\), as the model steps through the automata pool and the sub-tasks on the vertical axis when presented with the sequence of (ir-)relevant events on the horizontal axis. For instance, in T2: “Find the blue flag, spot the switch, and head for the target”, the first sub-task is ‘Find blue flag’, and the RNN remains in this state, rejecting irrelevant events until the observation ‘Blue’. It then transitions to ‘Navigate to the Switch’, and similarly, remains in this state until ‘Switch’. We observe that for each task, the initialization of the correct sub-task follows the task’s semantics, i.e., given ‘No Event’, the first sub-task always corresponds to the relevant one according to \(\mathcal{W}_i\). This is of key importance because it means that the RNN does not initialize an incorrect automaton.

We also observe that the rollout for each task follows a logically correct and semantically relevant sequence of sub-tasks. The RNN transitions occur only for relevant events, and all other irrelevant events from other tasks are rejected. For instance, ‘Found Base’ and ‘Spotted Enemy’ only produce self-loops in T2. Indeed, our supervised train and test accuracy is a 100 %, observed on 500 sequences per task with an 80-20 split. Note that the sequence of events (horizontal axis) has no particular order, and can continue indefinitely. We list the full set of tasks and events on the website. The representation capacity of the RNN indicates that it is possible to learn hundreds or even thousands of tasks in a single model [13], [34]. A procedural method that automatically “imagines” tasks for robot teams could potentially be used to extract numerous automata and distill them all into our RNN.

5.2 Ablation: Need for Recurrence and Language↩︎

Since the RNN can correctly step through complex, multi-step task automata, we now focus on evaluating its necessity in task completion, especially when using language inputs. We use the scenario shown in 4 (left), where a team of three robots is initialized at random locations in four rooms. Each of the first three rooms on the left has a switch that a robot can hit to gain passage into an adjacent room on the right, to ultimately reach its assigned goal (in the rightmost room). The team receives a natural language description of the overall task,for instance, “Robots, unlock the first, second, and third switches in sequence, then advance to the objective room and reach the target.” Any individual robot may be further initialized with a sub-task as: “Head to the second trigger in the second room from the left.”

To ablate recurrence, we train a ‘Vanilla RL’ baseline that only rewards the team when the robots navigate correctly to their goals, and has no reasoning or an explicit model for the sequence of sub-tasks. We separately ablate language and automata by training a ‘Tuned Rewards RL’ baseline with a hand-crafted multi-stage reward that explicitly decomposes the task. This causes overfitting to a sequence of sub-tasks: the team cannot exploit semantic hints from language and will require a new reward structure for each new configuration. All methods use a Graph-Attention GNN [35] with a single head of dimensions \([64,256,256,4]\).

In 4 (middle), we see a comparison (averaged over \(500\) runs) of our method against these two candidate baselines. ‘Vanilla RL’ performs well only in hitting the very first switch, after which, the successive rooms are never unlocked. The Tuned-Rewards RL method performs significantly better, since we already decompose rewards according to the stages in the task. This decomposition is manual, and thus laborious, error-prone, and infeasible to scale and apply to different scenarios. Our method sidesteps this problem by encoding the appropriate sub-tasks in the RNN (rooms, switches), and delegating the sub-task switching to it.

Our method also enables robots to be initialized non-uniformly (same/different rooms), both physically as well as in their initial RNN states through natural language, as seen in the three samples in 4 (left). We show quantitative analysis in 4 (right) by initializing robots in all six unique configurations for each method, trained for 1 × 10⁶ frames and evaluated over \(500\) runs. Clearly, when the team is initialized in the rightmost room, all methods perform equally well (\(=\SI{100}{\percent}\)), since the only sub-task then is to navigate to their respective goals. Similarly, initialization in ‘room-2’ is only slightly worse since it adds only one additional sub-task. For all other initializations, only the language-driven RNN method succeeds in completing the entire task reliably—especially in the hardest configuration where the robots are spread evenly (one in each room). Although the baseline, language-free RL methods eventually learn under predictable initializations, their catastrophic failures under “random” and “even” initializations persist, as these require asymmetric behaviors that are considerably harder to learn.

We note that our method does exhibit some susceptibility in the ‘random’ and ‘even’ configurations. We attribute this to highly divergent language initializations: a sufficiently meaningless expression can produce an incorrect mapping into an automaton’s state. As an instance from our website, “Exploration drones, activate the three room switches from left to right; then move into room four and finalize at the goal” with a sub-task “Units, make your way smoothly to the second release plate in the room after the left-most bay near the middle divider gate; signal ready when the ring glows”.

5.3 Comparison with Baselines in an Unstructured Task↩︎

We now test and benchmark the robustness of our method in handling more complex automata with parallel and heterogeneous sub-tasks for the team, partial language specification, and event-space disruptions. As baselines, we use (i) an online llm planner (Llama 3.3 70B [36]) to generate per-agent actions when called every 20 environment steps and provided with the language instruction, world state, and recent events; (ii) the GRU variant of latmos [13], and, (iii) a distilled Small Language Model (slm) based on the encoder of flan-t5-small [37]. The last two methods are fine-tuned in the same way our method trains the RNN, and matched to a low-level decentralized policy using MAPPO as ours.

We design a task for a heterogeneous team of four robots, two scouts and two carriers, that need to discover two targets in a 2D area, and transport them to their corresponding (assigned) medical bays. Only scouts can discover targets, and a target can be transported only when two carriers lift together. The resultant automaton necessitates parallelism, and exhibits several branches. We encode four different DFA families corresponding to four prompt groups: either target may be rescued first, target-1-first, target-2-first, and, mutex prohibiting scouts to search for target 2 before target 1 is rescued. Prompts can be as abstract as: “The orange casualty is more urgent.”

Table 1 reports success rates at two episode budgets. All trained methods fully solve the task when given the \(800\)-step budget, however, differences emerge with a tighter \(250\)-step budget. Our \(0.16\)M-parameter model remains competitive against the distilled slm and latmos (with \(168\times\) and \(10\times\) more parameters, respectively). For the strict constraint, the two language-conditioned methods complete \(60\%\) of the tasks, while latmos, which has no language input by construction, drops to \(40\%\). The gap is prominent here, since the constraint cannot be inferred from observations alone. The online llm planning fails this cooperative-carry task entirely: the model issues reasonable discovery commands, but cannot maintain synchronization between the carriers required for transport; the failure mode is coordination, not time-budget.

0pt 1pt

0pt 1pt

We evaluate robustness against two sources of errors: unseen language variations, and random event disruptions. We create 6 held-out paraphrases per DFA variant that are semantically equivalent to those in \(\mathcal{D}\), and substitute their corresponding \(\mathcal{E}\) randomly during rollout. For emulating event disruptions, at each environment step, we drop any event bit set to \(1\) with some probability \(p\). Both perturbations leave success rates within a few percentage points of the unperturbed baseline (Table 2). Notably, paraphrase substitution does not cause systematic degradation, indicating that the RNN encoder understands task semantics. The event-noise result is consistent with the policy treating events as soft signals that can be re-discovered.

5.4 Long Sequences, Disruptions and Zero-shot Deployment↩︎

Finally, we evaluate our framework for deployment on a team of robots in simulations and in real-world. Our objective is to qualify robustness against (i) disturbances in the task space (i.e., events that may naturally occur during the course of a mission), and, (ii) extrinsic noise in the physical world (such as what we expect in typical robot deployments).

1 shows four snapshots from a zero-shot physical deployment using a team of three RoboMaster robots[14] for the task, “Identify the purple flag, navigate to the switch and proceed to the goal”. Each robot receives positioning data from a motion-capture system, and runs its own feedback control onboard an Nvidia Jetson Orin. We use the ROS2 middleware to package robot policies as ‘nodes’ and use off-the-shelf WiFi modules to exchange messages. An offboard computer processes the input task and communicates with each robot; in general, we can distribute nodes freely across the network since our commands are not latency-critical.

In 5, we show the spatial (top-down) and temporal views from a different run of this task. This time, we emulate a task-space disturbance, ‘Flag Lost’, approx. 6 s after the flag was found, which causes the team to double back and recover the flag before heading to the switch. 5 (top) color-codes the trajectory of one robot to indicate the team’s current target (the other agents use light gray). We observe the behavior switch as two ‘loops’ between the purple flag and the yellow switch, corresponding to the two trips the team makes towards the flag. The temporal view in 5 (bottom) shows the process evolution. The robots first head towards the purple flag; upon obtaining it (\(\approx\SI{3}{s}\)), they change their target to the yellow switch. At the ‘Flag Lost’ event (\(\approx\SI{6}{s}\)), the RNN resets to heading to the flag. When the flag is found again (\(\approx\SI{9}{s}\)), the team continues to the switch and finally make their way to the goal. Similar to 3, we retain robustness against a chain of such events.

In 6, we show snapshots from six additional evaluations with variations on language instructions, randomly injected ‘Flag Lost’ events and unmodeled physical disruptions. For instance, 6 (b,c) show a person (masked for anonymity) manually displacing random robots in the team. Over 8 real-robot evaluations (6 with disruptions), the team successfully recovers from different combinations of such events, thanks to both the feedback control’s responsiveness and the automaton’s adherence to a global task sequence. However, under extreme physical disruptions, we observe poor tracking performance from the control policy as it also attempts to counteract the disturbance. This may be alleviated by adequate training and domain randomization. 6 (d–f) show evaluations on longer task sequences. Similar to the multi-room problem in 5.2, our method is successful due to the sub-task sequencing enforced by the automaton.

Finally, we evaluate scaling to larger teams. Because the policy uses a graph-based architecture, it generalizes naturally to variable team sizes. 7 shows snapshots from the same task in deployments with 6, 9, and 12 agents in simulation, using a policy that was trained with 3 agents. The teams complete the tasks successfully, while the inference time (per-loop) remains fairly manageable and well within realistic constraints for feedback control (under \(\SI{1}{ms}\)/loop/robot).

6 Conclusions↩︎

In this work, we presented a novel framework that enables teams of robots to perform complex, collaborative tasks from high-level natural language commands. Our approach successfully addresses the challenges of real-time reasoning, decentralized coordination, and the interpretation of ambiguous yet expressive semantic instructions in natural language. We effectively distill the reasoning capabilities of a large language model into a single, light-weight RNN. This model, paired with a graph neural network-based policy, allows robots to reason about task sequences and collaborate on sub-tasks in a fully distributed manner. Simulations and real-world experiments demonstrated that our method empowers robot teams to display interactive, onboard reasoning, leveraging semantic cues from human language to achieve their goals. We plan to extend this work to teams with a set of predefined robot capabilities, such that they can reason over and address all tasks solvable with those capabilities.

References↩︎