1 Introduction↩︎

Routing is becoming a central mechanism for coordinating systems composed of multiple language models, tools, or agents [1]–[4]. Existing work on LLM routing largely evaluates routers by downstream utility: whether they improve task accuracy, reduce cost, or select the best-performing model for a query [5]–[8]. A router is not merely a predictor of task performance; it is also the coordination mechanism that determines how work is distributed across a society of actors. Its usefulness, therefore, depends on two structural properties. First, the society must contain behaviourally differentiated actors: if all actors respond in the same way, routing is vacuous [9], [10]. Second, the routing policy should be robust to superficial variation: semantically-equivalent variants should not be sent to different actors merely because of changes in spelling, syntax, or wording [11]. Without such stability, actors cannot reliably specialise.

To characterise the space of choices available to a router, we adapt Hierarchic Social Entropy [9] to language-model societies, measuring behavioural diversity from model outputs rather than parameters [12]. To characterise the stability of the router’s choices, we introduce a perturbation-based robustness metric that measures whether semantically related inputs are assigned to the same actor.

We validate HSE on two benchmarks, EmbedLLM [13] and RouterBench [7], comparing a default society of real-world language models against synthetic societies of purpose-designed experts varying in role and expertise overlap.

We find that specialist societies achieve substantially higher HSE than pools of real-world models of equivalent size, suggesting that behavioural diversity in deployed model pools is lower than commonly assumed. We further observe strong diminishing returns: fewer than ten agents suffice to capture most available diversity in EmbedLLM, and four agents in RouterBench, providing a practical heuristic for society design. Interestingly, higher HSE does not imply higher robustness: KNN routers achieve their best accuracy on specialist societies but their worst robustness, while prompted routing remains stable across all HSE levels and perturbation types. Our contributions are summarised as follows:

• We introduce the first behavioural diversity metric for LM routing analysis, adapting Hierarchic Social Entropy to language-model societies to characterise when routing is meaningful.

• We introduce a perturbation-based metric that quantifies how stably a routing policy assigns semantically-equivalent queries to the same actor, paired with a five-level perturbation taxonomy for fine-grained analysis by perturbation type.

• We show that HSE exhibits strong diminishing returns with society size: a curated subset of fewer than ten agents recovers most available diversity in a large real-world pool, providing a practical coreset heuristic for society design.

2 Related Work↩︎

2.0.0.1 LLM Routing and Model Selection

A growing body of work studies routing as a mechanism for selecting among language models with different capabilities, costs, and latency profiles. In this setting, the router is typically evaluated by its ability to improve the trade-off between task performance and inference cost. FrugalGPT [5], for example, formulates LLM use as a cost-sensitive cascade, where cheaper models are queried before more expensive ones in order to reduce inference cost while maintaining task accuracy. RouterBench [7] provides a benchmark for multi-LLM routing, evaluating how well routing policies choose among alternative models across tasks and model families. RouteLLM [6] learns routing policies from preference data, routing queries between stronger and weaker models in order to preserve response quality while reducing the use of expensive models.

More recently, MasRouter [4] extends routing to multi-agent settings, jointly optimising collaboration mode, role allocation, and model selection, building on multi-agent LLM frameworks in which agent roles and interaction protocols are central design choices [1]–[3].

2.0.0.2 Diversity Metrics for Agent Systems

The measurement of behavioural diversity in multi-agent systems has been studied primarily in robotics. [9] introduces Hierarchic Social Entropy (HSE) as an information-theoretic measure of robot group diversity that captures both the number and relative size of behavioural clusters across all taxonomic scales. More recently, [10] propose System Neural Diversity (SND), which uses Wasserstein distance over multivariate robot-policy action distributions. SND requires all the agents to compute probabilities on the same action space and would not translate to heterogeneous LM agents with access to different tools. Our work adapts HSE to LM agent societies, replacing Euclidean inter-agent distance with cosine similarity over behavioural vectors, and is the first to apply such metrics to LM routing analysis. The ensemble learning literature has also studied diversity as a design objective [14], and quality-diversity optimisation [15] explicitly searches for a diverse set of high-performing solutions.

2.0.0.3 Specialisation in Multi-Agent Systems

A related line of work studies how individual actors come to occupy stable, distinct roles within a group — a prerequisite for routing to be meaningful over time. In multi-robot systems, specialisation is typically measured by the stability of roles rather than by task accuracy. [16] introduces response threshold models in which an agent’s propensity to perform a task decreases each time it is performed by another agent; specialisation is said to emerge when an agent’s threshold for a given task remains persistently low. [17] operationalises this differently via Collective Neuro-Evolution (CONE), defining a specialised agent as one that assumes a specific behavioural role for more than 50% of its operational lifetime.

Our routing robustness metric formalises this intuition for LM societies, measuring whether semantically-equivalent query variants are stably assigned to the same actor — the distributional consistency that these robotics accounts identify as the precondition for specialisation.

2.0.0.4 Robustness to Prompt Perturbations

A separate line of work studies the robustness of language models under input perturbations, including character-level noise, lexical substitutions, syntactic transformations, and paraphrases. A common finding is that meaning-preserving changes can cause large accuracy swings [11] across model sizes and instruction tuning. Prior work typically responds by training models to produce consistent outputs across clean and perturbed prompts [18], or by recommending that evaluations report performance ranges rather than point estimates [11]. We study robustness at a different level: not whether the final answer is stable under perturbation, but whether the routing decision is.

By measuring whether perturbed queries are routed to the same actor as the original, we evaluate whether the routing policy induces stable task distributions over actors.

3 Preliminaries↩︎

We introduce the notation used throughout the paper and state the two structural properties we study.

Let \(\mathcal{R} = \{r_1, r_2, \dots, r_N\}\) be a society of \(N\) actors, where each actor may correspond to a distinct model, prompt, tool-using agent, or role-specialised configuration. Actors may differ in architecture, scale, training data, or system prompt.

Let \(\mathcal{Q} = \{q_1, q_2, \dots, q_n\}\) be a set of queries. A routing policy \(\pi : \mathcal{Q} \to \mathcal{R}\) assigns each query to an actor; we write \(a_i = \pi(q_i)\) for the actor assigned to query \(q_i\). We denote by \(\mathcal{Q}_k = \{q_i \in \mathcal{Q} \mid a_i = r_k\}\) the set of queries assigned to actor \(r_k\).

Because actors may differ in architecture and parameterisation, we do not compare them in weight space. Instead, we evaluate each actor \(r_k\) on an evaluation set \(\mathcal{E} = \{e_1, \dots, e_L\}\), producing a behavioural vector, \[\mathbf{b}_k = \bigl(s(r_k, e_1),\; s(r_k, e_2),\; \dots,\; s(r_k, e_L)\bigr) \in \mathbb{R}^{L} \nonumber \label{eq:behavioural95vector}\tag{1}\] where \(s(r_k, e_l) \in [0, 1]\) is the score of actor \(r_k\) on prompt \(e_l\). Stacking these vectors gives a behavioural matrix \(B \in \mathbb{R}^{N \times L}\), from which pairwise inter-actor distances are computed via cosine similarity (Section 4.2). Differences in parameter space do not necessarily correspond to differences in behavioural space [12]; measuring diversity from \(B\) rather than from model weights is therefore both more principled and more general, extending naturally to agent societies that share a model backbone but differ in system prompt or tool access. When a separate evaluation set is not available, \(\mathcal{E}\) may be set equal to \(\mathcal{Q}\).

Given a society \(\mathcal{R}\) and a routing policy \(\pi\), we study two properties that are orthogonal to task accuracy. Society diversity \(\mathrm{HSE}(\mathcal{R})\) (Section 4) quantifies how behaviourally differentiated the actors in \(\mathcal{R}\) are, as measured over the behavioural matrix \(B\). This property is defined over the society alone and is independent of any routing policy. If \(\mathrm{HSE}(\mathcal{R}) \approx 0\), all actors respond similarly and routing is vacuous regardless of the policy used. Routing robustness \(\rho(\pi, \mathcal{Q})\) (Section 5) quantifies how stably \(\pi\) assigns semantically-equivalent query variants to the same actor. This property is defined over the policy and depends on \(\mathcal{R}\) only through the routing decisions \(\pi\) induces. If \(\rho(\pi, \mathcal{Q}) \approx 0\), actors receive inconsistent query distributions, which undermines the conditions under which specialisation can emerge.

Together, these two properties characterise when routing is meaningful: diversity ensures there is something non-trivial to route to, and robustness ensures that routing decisions satisfy the necessary condition for specialisation.

4 Society Diversity↩︎

Given a society \(\mathcal{R}\) and its behavioural matrix \(B\) (Section 3), we want a single scalar capturing how behaviourally differentiated the actors are. A useful measure should reflect both the number of distinct behavioural groups and the degree to which those groups differ. Diversity is ubiquitous in natural systems [19], and ecologists have demonstrated the role of functional diversity in ecosystem survival [20]. The challenge of identifying clusters of elements distributed in a continuous multidimensional space is exactly the problem faced by biologists constructing taxonomic systems, where each dimension represents a morphological trait distinguishing one organism from another. We adopt the same hierarchical view of entropy, replacing biological traits with the components of \(\mathbf{b}_k\).

Specifically, we adapt Hierarchic Social Entropy (HSE; [9]), an information-theoretic measure originally proposed for robot swarms, to LM agent societies. We do not claim diversity is always desirable: for some tasks, homogeneous teams may outperform diverse ones. However, a quantitative metric enables controlled comparisons between societies and informs the design of diverse teams.

4.1 Simple Social Entropy↩︎

Simple social entropy applies Shannon’s information entropy [21] to a multi-agent society to quantify behavioural diversity based on how actors are distributed among behavioural subsets.

Let \(\mathcal{C} = \{c_1, c_2, \dots, c_M\}\) be a partition of \(\mathcal{R}\) into \(M\) homogeneous subsets, and let \(p_i\) denote the proportion of actors belonging to subset \(c_i\), with \(\sum_{i=1}^{M} p_i = 1\). The simple social entropy of society \(\mathcal{R}\) under partition \(\mathcal{C}\) is: \[H(\mathcal{R}) = -\sum_{i=1}^{M} p_i \log_2(p_i) \label{eq:simple95entropy}\tag{2}\]

This measure satisfies two basic properties: (i) homogeneity: \(H(\mathcal{R}) = 0\) if and only if all actors fall into a single subset; (ii) maximum diversity: \(H(\mathcal{R})\) is maximised when actors are distributed equally across all \(M\) subsets (\(p_i = 1/M\) for all \(i\)). Figure 1 illustrates simple social entropy on societies with varying cluster structure.

4.2 Inter-Agent Distance↩︎

To partition \(\mathcal{R}\) into behavioural subsets, we require a pairwise distance over actors. Given the behavioural vectors \(\mathbf{b}_k\) defined in Section 3, we measure the distance between actors \(r_j\) and \(r_k\) as one minus their cosine similarity: \[d(r_j, r_k) = 1 - \frac{\mathbf{b}_j \cdot \mathbf{b}_k}{\|\mathbf{b}_j\|\,\|\mathbf{b}_k\|} \label{eq:distance}\tag{3}\] We prefer cosine similarity over the Euclidean distance used in the original HSE formulation [9] because behavioural vectors may differ substantially in magnitude when agents vary in overall accuracy, and we wish to capture profile similarity rather than absolute performance level (Appendix 14). We prefer it over the Wasserstein distance used in SND [10] because that metric is defined over full action distributions, which are not directly available for language model outputs.

4.3 Hierarchic Social Entropy↩︎

Simple social entropy (Equation 2 ) captures two aspects of diversity: the number of behavioural groups and the distribution of actors across them. It does not, however, capture the degree of difference between groups. Figure 2 illustrates the consequence: two societies each containing three identical actors and one outlier receive the same score \(H = 0.811\), regardless of whether the outlier is close to or far from the main cluster (\(d_1 \ll d_2\), yet \(H\) is identical). This limitation matters for routing. Suppose a router assigns queries to an outlier actor that barely differs from the rest of the society. Simple entropy would record non-zero diversity, yet the routing decision is nearly vacuous: the outlier provides almost no complementary capability. A useful diversity measure should be sensitive to the degree of behavioural separation between groups, not just to whether separation exists.

To obtain a single society-level score, we aggregate pairwise inter-actor distances (defined in Section 4.2). [10] aggregate by taking the mean over all pairwise distances. This is computationally simple but conflates societies with very different cluster structures: a society with two tight, well-separated clusters and a society with uniformly scattered actors can yield the same mean distance, yet the former is far more useful for routing, since actors fall into stable, distinguishable groups. [9] instead cluster actors at every possible distance threshold and measure entropy at each scale. We adopt this approach for the same reason: it is sensitive to the hierarchic structure of the behavioural space, not just its average extent.

Specifically, we use hierarchical clustering with single linkage (nearest-point algorithm), which merges two clusters as soon as their closest members fall within threshold \(h\). This is appropriate for our setting, where even a single behaviourally distinct actor justifies a separate cluster.¹ As \(h\) varies, the partition \(\mathcal{C}\) changes, so simple social entropy becomes a function of both the society and \(h\), written \(H(\mathcal{R}, h)\). We call \(h\) the taxonomic level, borrowing the term from [9], who in turn draws on biological taxonomy: just as a biologist can classify organisms at the species level or the genus level depending on how finely they wish to distinguish, \(h\) controls the resolution at which we observe the society. At \(h = 0\), actors merge only if their distance is exactly zero; behaviourally distinct actors each form a singleton cluster, giving \(H(\mathcal{R}, 0) = \log_2 M\) where \(M \leq N\) is the number of distinct behavioural profiles. As \(h\) increases, similar actors merge and the number of clusters falls, until at \(h \to \infty\) all actors collapse into one group with \(H(\mathcal{R}, h) = 0\).

Hierarchic Social Entropy integrates simple entropy across all taxonomic levels: \[\mathrm{HSE}(\mathcal{R}) = \int_{0}^{\infty} H(\mathcal{R}, h)\, dh \label{eq:hse}\tag{4}\] We define \(d(r_j, r_k) = 1\) (maximum distance) when either \(\|\mathbf{b}_j\| = 0\) or \(\|\mathbf{b}_k\| = 0\), i.e., when an actor fails every evaluation prompt. HSE is a continuous ratio measure with an absolute zero (when all actors are identical), enabling statements of the form “Society A is twice as diverse as Society B” and total orderings across societies of different sizes when normalised by the maximum achievable HSE for that society size.

4.4 Validating HSE↩︎

We construct minimal societies that isolate specific properties of HSE, confirming that the metric behaves as expected before applying it at scale (see Section 7). Table ¿tbl:tab:hse-sanity? verifies that when all actors produce identical outputs (a), the society is maximally homogeneous: HSE is zero and routing is vacuous, regardless of how many actors are present. When actors are fully orthogonal, each succeeding on exactly the prompts the others fail (b), HSE is maximised and normalised HSE equals 1.0. Crucially, the normalised value is invariant to society size, enabling fair comparisons across societies of different scales.

Table ¿tbl:tab:hse-duplicates? verifies that adding redundant information does not artificially inflate HSE. Starting from a base society of three behaviourally distinct actors (case a), we extend it in two ways: adding a fourth actor whose behavioural vector is identical to actor C (case b), and adding a fourth evaluation prompt that is a copy of p3 (case c). In both cases HSE decreases slightly rather than increasing, confirming that the metric is not fooled by redundancy. This property matters in practice: a routing system should not appear more diverse simply because actors are duplicated or because the evaluation set contains repeated queries. Finally, Figure 3 empirically confirms monotonicity: HSE increases as behaviourally distinct actors are added.

5 Robustness of Routing Policy↩︎

Society diversity (Section 4) characterises whether a society contains actors with sufficiently differentiated capabilities to make routing non-trivial. But diversity alone is not sufficient: as the specialisation literature makes clear (Section 2), actors can only develop stable roles if they reliably receive the same type of queries over time.

We now introduce a metric that captures this second structural property directly.

The central intuition is that stable query assignment is a necessary condition for specialisation to be possible: a router that assigns query \(q_i\) to actor \(r_k\) but assigns a surface-form variant to actor \(r_j \neq r_k\) violates this condition, regardless of what learning dynamics operate downstream. We do not claim that stable routing produces specialisation — whether actors develop specialised competence depends on factors outside the scope of this work — but unstable routing makes it impossible by construction. We therefore measure routing robustness as the degree to which surface-form variants of a query are assigned to the same actor.

5.1 Perturbation Taxonomy↩︎

The robustness metric \(\rho\) is defined relative to a perturbation regime: the same router may be stable under surface-level noise yet unstable under semantic reformulation. To support both aggregate and fine-grained analysis, we define five perturbation classes of increasing semantic distance from the original query, summarised in Table 1.

Perturbations at levels 1–3 preserve surface proximity to the original query; levels 4–5 test whether the router is sensitive to meaning-preserving reformulations that substantially alter wording or add noise. Full methodology and prompt templates used to generate each class of perturbations are provided in Appendix 11.1. By computing \(\rho\) separately for each perturbation class, one can characterise where a routing policy breaks down: a router that degrades only at level 4–5 is more robust in practice than one that degrades at level 2, even if their aggregate \(\rho\) scores are similar.

5.2 Routing Robustness Metric↩︎

Given a routing policy \(\pi\) and a query \(q_i \in \mathcal{Q}\), recall that \(a_i = \pi(q_i)\) denotes the actor assigned to the original query (Section 3). We generate \(p\) perturbations \(q_i^1, q_i^2, \ldots, q_i^p\) of \(q_i\) (see Section 5.1 for the perturbation taxonomy). The per-query robustness is the fraction of perturbations routed to the same actor as the original: \[\rho_i = \frac{1}{p} \sum_{j=1}^{p} \mathbf{1}\!\left[\pi(q_i^j) = a_i\right] \label{eq:per95query95robustness}\tag{5}\] where \(\mathbf{1}[\cdot]\) is the indicator function. \(\rho_i \in [0,1]\), with \(\rho_i = 1\) if all perturbations are routed to the same actor, and \(\rho_i = 0\) if none are.

Dataset-level robustness is obtained by averaging over all \(n = |\mathcal{Q}|\) queries: \[\rho(\pi, \mathcal{Q}) = \frac{1}{n} \sum_{i=1}^{n} \rho_i \label{eq:dataset95robustness}\tag{6}\] This metric rests on the assumption that meaning-preserving perturbations do not change which actor is most accurate for a given query: if \(q_i\) and \(q_i^j\) are semantically equivalent, the optimal assignment should be the same for both. Under this assumption, any change in routing decision reflects instability in the policy rather than a legitimate response to a change in query difficulty or actor suitability.

Two boundary cases anchor the metric: a fixed router that always assigns the same actor gives \(\rho_i = 1\) for all \(i\) and \(\rho = 1\), while a uniform random router gives \(\mathbb{E}[\rho] = 1/N\). Beyond the dataset-level aggregate, we also compute a per-actor score over the query set \(\mathcal{Q}_k\) assigned to actor \(r_k\): \[\rho^{(k)}(\pi, \mathcal{Q}) = \frac{1}{|\mathcal{Q}_k|} \sum_{q_i \in \mathcal{Q}_k} \rho_i \label{eq:per95actor95robustness}\tag{7}\] If no queries are assigned to actor \(r_k\) under policy \(\pi\) (i.e., \(|\mathcal{Q}_k| = 0\)), we define \(\rho^{(k)}(\pi, \mathcal{Q}) = 0\). A low \(\rho^{(k)}\) indicates that queries nominally assigned to actor \(r_k\) are unstably routed under perturbation, violating the necessary condition for specialisation to emerge for that actor’s role.

6 Experimental Setup↩︎

6.0.0.1 Datasets

We evaluate on two routing benchmarks. EmbedLLM [13] provides performance scores across 112 models on 10 reasoning and knowledge benchmarks (MMLU, GSM8K, GPQA, ASDiv, and others), comprising 36,054 questions in total. RouterBench [7] is a complementary routing benchmark with performance scores across 11  models on 36,497 queries drawn from 8 datasets (see Appendix 10 for details and train, validation, and test splits).

6.0.0.2 Societies

The benchmarks described above constitute the default societies. On EmbedLLM, we compare the default society (HSE=1.62) against three synthetic societies in which models are replaced by synthetic experts with binary performance indicators encoding domain-specific expertise:

• RD (Reasoning Depth): five non-overlapping experts covering five tiers of cognitive complexity, ranging from direct lookup to complex multi-step reasoning (HSE = 2.32).

• SA (Subject Area): 15 non-overlapping experts each specialising in a distinct academic or practical domain, such as advanced mathematics, computer science, medicine, and commonsense reasoning (HSE = 3.91).

• RD+SA: the combined pool of 20 RD and SA experts with partially overlapping expertise (HSE = 3.19).

Expert categorisation heuristics are detailed in Appendix 10.3. Oracle accuracy over each synthetic society is 1.0 by construction, since each query falls within exactly one expert’s domain. We define a max-HSE subset for each benchmark: the subset of the full society that maximises normalised HSE at a fixed society size, selected greedily; this subset serves as a high-diversity reference within real-world model pools. On RouterBench we compare the full society against its max-HSE subset.

6.0.0.3 Routing Policies

We evaluate five routing policies: random assigns each query to a uniformly sampled actor; prompted prompts an LLM with natural-language descriptions of actor roles and few-shot examples to select the most appropriate one (see Appendix 12); KNN-\(k\) (\(k \in \{1, 3, 10\}\)) retrieves the \(k\) nearest neighbours of the query in embedding space and assigns the actor that performed best on those neighbours (Appendix 13).

6.0.0.4 Perturbations

Routing robustness (Section 5.2) is measured over the five-level perturbation taxonomy of Section 5.1. For each query we generate one perturbation per level, yielding \(p = 5\) variants (Appendix 11.1); we report dataset-level robustness \(\rho(\pi, \mathcal{Q})\) (Equation 6 ) and per-level breakdowns.

7 Results and Analysis↩︎

7.0.0.1 Specialist societies are substantially more diverse.

Figure 3 shows normalised HSE as a function of society size on EmbedLLM. At any fixed size, purpose-designed specialist societies achieve higher HSE than the default real-world model pool. The SA society (HSE = 3.91) is more than twice as diverse as the default (HSE = 1.62), and the RD+SA combined pool (HSE = 3.19) similarly outpaces it despite containing many fewer total actors. This indicates that a large number of real-world models occupy similar regions of behavioural space: size alone does not imply diversity.

Across all societies, HSE exhibits strong diminishing returns: the marginal gain from adding an actor falls sharply after the first few. For the default society, the curve plateaus after approximately nine agents. Synthetic specialist societies plateau even faster, at around four to five agents for RD and SA, because their behavioural profiles are maximally orthogonal by construction. Small, carefully curated societies can recover most of the diversity available in a much larger pool, and collecting additional models offers little incremental benefit. HSE can also be combined with task accuracy as a joint selection criterion: Appendix 15 reports results for subsets selected by maximising HSE jointly with task accuracy, showing that the joint criterion leads to better routing while preserving diversity.

7.0.0.2 KNN routers gain accuracy from specialist societies but collapse in robustness.

Figure 4 (a) shows task accuracy rising sharply and monotonically for KNN routers as HSE increases. However, perturbed-query accuracy collapses at every specialist society, falling to near-random levels. Panel (b) confirms that robustness \(\rho\) remains low across the entire HSE range despite high clean-query accuracy. The sharp behavioural boundaries that make specialist societies easy to route correctly on clean queries make them equally easy to route incorrectly when a surface-level reformulation shifts the query embedding across a domain boundary.

The prompted router shows the reverse pattern. Its clean accuracy on SA is lower than KNN-10, but perturbed accuracy remains close, a gap of less than five points. Crucially, robustness rises monotonically with HSE, tracking society diversity in a way no KNN router does. Prompted routing latches onto the semantic intent of the query rather than its embedding position, making it naturally stable under meaning-preserving reformulations.

7.0.0.3 Max-HSE subsets improve prompted robustness, but the effect depends on pool structure.

Figure 5 compares the full society against the max-HSE subset on EmbedLLM and RouterBench. The prompted router improves consistently, roughly doubling its robustness on EmbedLLM and improving more modestly on RouterBench. The KNN routers tell a more complex story. On RouterBench, switching to the max-HSE subset dramatically rescues KNN robustness across all values of \(k\). On EmbedLLM, however, KNN-1 robustness actually decreases while KNN-3 and KNN-10 improve only marginally. This contrast reflects the nature of the max-HSE subset in each case: the RouterBench subset selects from models with smooth, graded behavioural profiles, so higher diversity sharpens actor distinctions without producing hard domain boundaries; the EmbedLLM pool, even at maximum HSE, has less structured behavioural separation, leaving nearest-neighbour routing sensitive to small surface variations in the query.

7.0.0.4 Robustness is roughly flat across perturbation levels.

Figure 6 shows per-level robustness on RouterBench. Every router produces approximately the same \(\rho\) at all five levels: the prompted router ranges from 0.589 (paraphrase) to 0.617 (syntax) on the full society, and from 0.663 (paraphrase) to 0.744 (word) on the max-HSE subset. No router shows a consistent decline as perturbations become semantically deeper. Routing policies are no more sensitive to a full paraphrase than to a character-level typo, implying that routing decisions are not anchored to surface features that higher-level perturbations would selectively disrupt. The only hint of level-sensitivity is the prompted router’s paraphrase dip on the max-HSE subset (0.663 vs.\(\approx\!0.73\) elsewhere), suggesting that full rewrites may destabilise an LLM-based policy when actor distinctions are sharper.

8 Conclusion↩︎

We have argued that routing in language-model societies should be evaluated along two structural dimensions beyond task accuracy: behavioural diversity and routing stability under surface-form query variants. For diversity, we adapted Hierarchic Social Entropy to language-model societies, using cosine similarity over behavioural vectors as the inter-actor distance. For robustness, we introduced a perturbation-based metric measuring the fraction of surface-form variants assigned to the same actor.

Applied to EmbedLLM and RouterBench, our analysis finds that HSE exhibits strong diminishing returns: fewer than ten agents capture most available diversity, providing a practical heuristic for society design. In addition, robustness and accuracy trade off sharply: KNN routers achieve high accuracy on specialist societies but collapse under perturbation, whereas prompted routers are less accurate but more stable across perturbation types and society configurations.

Together, HSE and routing robustness ask not only whether a router performs well, but whether routing is meaningful: whether the society is sufficiently differentiated to justify routing decisions, and whether the policy satisfies the necessary condition for specialisation.

9 Limitations↩︎

9.0.0.1 Measurement assumptions

HSE is computed over a fixed evaluation set \(\mathcal{E}\); if this set is not representative of the actual query distribution, behavioural vectors may not reflect deployment behaviour and diversity estimates could be misleading. The max-HSE subset is selected greedily, which does not guarantee the globally optimal subset at a given size; the true optimum requires an exponential search. Our synthetic societies further use binary performance indicators that produce maximally sharp behavioural boundaries and oracle accuracy of 1.0 by construction; real specialist agents will exhibit softer, overlapping profiles, and the KNN brittleness we observe may be attenuated in practice.

9.0.0.2 Scope of the robustness metric

Routing robustness is measured at the assignment level over single-turn queries and does not capture whether actors actually develop specialised competence as a result of stable query distributions. Extending the metric to multi-step agentic settings, where perturbations propagate through state rather than through a single query, is a natural next step, as is using HSE as a training signal to bootstrap diverse societies from a homogeneous initialisation.

10 Description of Router Benchmarks↩︎

10.1 EmbedLLM↩︎

EmbedLLM [13] is a model routing and correctness-forecasting benchmark built around a matrix of binary performance labels collected from 112 open-source language models spanning a wide range of sizes and specialisations (general-purpose, coding, biomedical, physics). Each model was evaluated on 36,054 questions drawn from the test sets of ten established benchmarks, summarised in Table 2. For each model–question pair the label records whether the model answered correctly (1) or not (0), yielding a correctness matrix \(Y \in \{0,1\}^{112 \times 36{,}054}\). Some questions were duplicates or near-duplicates which differed only by the order in which the multiple choice answers were presented, so we filtered out about 3k prompts. The remaining questions were split 80%/10%/10% into train, validation, and test sets (approximately 26K/3.6K/3.6K questions).

In our work, we treat each model as an actor \(r_k \in \mathcal{R}\) and each row of \(Y\) as the actor’s behavioural vector \(\mathbf{b}_k\) (Section 3). The full set of 112 models constitutes the default society; the max-HSE subset is the 9-model subset selected greedily to maximise normalised HSE (Section 6).

10.2 RouterBench↩︎

RouterBench [7] is a comprehensive benchmark for evaluating multi-LLM routing systems, designed to cover a broad spectrum of tasks and domains. The dataset contains 405,467 pre-generated inference samples from 11 LLMs — six open-source (Llama-70B-chat, Mixtral-8x7B, Yi-34B-chat, Code Llama-34B, Mistral-7B-chat, WizardLM-13B) and five proprietary (GPT-4, GPT-3.5-turbo, Claude-instant-v1, Claude-v1, Claude-v2) — evaluated across 8 datasets spanning 64 tasks (Table 3).

Each sample records the model response alongside a quality score and an inference cost in dollars, enabling joint evaluation of task performance and economic cost. For routing experiments, the dataset is partitioned 80%/10%/10% into train, validation, and test sets (approximately 29K/3.6K/3.6K prompts).

In our work, we treat each of the 11 models as an actor \(r_k \in \mathcal{R}\), using binary correctness labels as the entries of the behavioural matrix \(B\) (Section 3). The full set of 11 models constitutes the default society; the max-HSE subset is the 4-model subset that maximises normalised HSE, selected greedily (Section 6).

10.3 Synthetic Societies↩︎

10.3.0.1 Subject-Area Clustering (SA)

To construct the SA synthetic society we assign each of the 67 evaluation items in EmbedLLM, comprising 57 MMLU subjects and 10 additional benchmarks (GSM8K, MathQA, ASDiv, LogiQA, GPQA, MedMCQA, TruthfulQA, SocialQA, PIQA), to one of 15 mutually exclusive subject-area clusters. The assignment criterion is academic domain: items are grouped by what they are about, independently of how much reasoning they require. The resulting clusters are shown in Table 4.

5pt

Each synthetic SA agent is assigned a binary performance vector, scoring 1 on every item in its cluster and 0 on all others, modelling a specialist with perfect but narrowly scoped expertise. Several borderline assignments are worth noting. Econometrics is placed in Advanced Mathematics rather than Economics because its questions are dominated by statistical modelling and regression rather than economic theory. GPQA is placed in Chemistry & Earth Sciences because its hardest and most distinctive questions are in graduate-level chemistry and physical sciences. TruthfulQA is placed in Psychology & Behavioural Sciences because its core demandm resisting socially transmitted false beliefs, is an epistemic and psychological phenomenon rather than a factual one. Business Ethics is placed in Law & Legal Reasoning because its MMLU questions predominantly test corporate governance and fiduciary duty rather than philosophical ethics.

10.3.0.2 Reasoning-Depth Clustering (RD)

To construct the RD synthetic society we assign the same 67 EmbedLLM items to one of five reasoning-depth tiers that form a strictly ordered scale from surface recall to adversarial expert reasoning. The assignment criterion is the dominant cognitive operation required to answer correctly, independent of subject matter.

R1 (Direct Lookup)

covers items answerable by retrieving a single stored fact with no inference, such as recalling a historical date or a viral property (13 items).

R2 (Concept Application)

covers items requiring one inferential step — applying a rule, definition, or principle to a specific case — including PIQA, SocialQA, and TruthfulQA (21 items).

R3 (Multi-Step Reasoning)

covers items whose solution requires composing two or more dependent inference steps, including GSM8K, MathQA, and MedMCQA (21 items).

R4 (Formal Symbolic Reasoning)

covers items solved by mechanically applying a formal system such as logic, algebra, or proof calculus, including LogiQA and ASDiv (7 items).

R5 (Adversarial & Expert-Level)

covers items specifically designed to defeat shallow heuristics, requiring deep domain expertise and resistance to carefully crafted near-miss distractors, including GPQA and the four MMLU Professional subjects (5 items).

As with SA, each RD agent scores 1 on every item in its tier and 0 elsewhere. Key borderline decisions include: TruthfulQA is assigned to R2 rather than R3 because each individual question requires a single evaluative judgment, not a reasoning chain; LogiQA is assigned to R4 rather than R3 because its civil-service puzzles require applying formal inference rules rather than multi-step domain reasoning; ASDiv is assigned to R4 rather than R1 because its design goal is robustness across diverse arithmetic structures, probing rule application rather than recall; and GPQA is assigned to R5 because it is explicitly designed so that non-experts perform near chance and domain experts reach only around 65%, with distractors crafted to exploit common expert heuristics.

11 Query perturbations↩︎

11.1 Perturbation generation↩︎

We generate the query perturbations by prompting an LM, using Gemini 3.1 Flash-Lite [37] as backbone model. This section documents the five zero-shot prompts used to generate perturbations of each query in the evaluation sets of EmbedLLM and RouterBench (Section 5.1). Each prompt instructs the model to produce 10 rewrites of an input query {prompt} according to specific perturbation instructions for each of the five levels: character, word, syntax, paraphrase, and rambling. We request more perturbations than needed since occasionally the model produces duplicates variants. We then parse the outputs and select 5 unique variants per perturbation category.

11.2 Query perturbation examples↩︎

This section presents some examples of the generated prompt perturbations.

12 Prompted Router Policy↩︎

For the prompted router policy, we use a few-shot prompt template with Gemini 3.1 Flash-Lite [37] as LM backbone. The prompt asks a language model to select the best-suited actor for a given query, conditioned on (i) a natural-language description of each available actor, and (ii) a set of few-shot exemplars drawn from the training split.

Section 12.1 presents the template itself; Section 12.2 describes how few-shot exemplars are selected; and Section 12.3 show how the template is instantiated for the EmbedLLM and RouterBench datasets.

12.1 Prompt Template↩︎

12.2 Few-Shot Exemplar Selection↩︎

For each actor, \(k\) exemplars are randomly sampled from training prompts where that actor is the clear winner, i.e., its score exceeds all other actors by at least a margin \(\delta\). We use \(k{=}4\) and \(\delta{=}0.4\). In the rare instances in which there are less than \(k\) prompts that satisfy this criteria, we fall back to sampling from training prompts where that actor achieved the highest score among all actors in the society (possibly tied with other actors).

Exemplars are formatted as below and shuffled together across all specialists:

12.3 Instantiation↩︎

For EmbedLLM and RouterBench we list all the names of the LLMs available in the dataset, e.g., huggingfaceh4_zephyr_7b_beta, in the {actors_description} field.

For the synthetic societies, each actor corresponds to a knowledge domain (subject area), a reasoning depth, or both, so we inject the descriptions below into the {actors_description} placeholder.

13 KNN Router Policy↩︎

The KNN-based router computes an embedding \(e(q)\) of the query using Gemini Embedding 2 as model backbone [38] and retrieves the \(k\) nearest neighbours \(\mathcal{N}_k(x)\) from the training set using cosine similarity in the embedding space. It then assigns the actor with highest score over the \(k\) nearest neighbours.

14 Effect of Distance Metric and Linkage on HSE↩︎

Figure 7 reports normalised HSE as a function of society size across all twelve combinations of distance metric (cosine, correlation, euclidean, manhattan) and linkage method (single, complete, average), using greedy agent selection on EmbedLLM. The qualitative pattern is consistent across configurations: specialist societies plateau rapidly at high HSE while the default real-world pool plateaus at a substantially lower level, confirming that the finding reported in Section 7 does not depend on the choice of distance or linkage. Cosine distance with single linkage, used throughout the main paper, produces the most conservative estimates of diversity for the default society.

15 Effect of Model Selection Criteria↩︎

2pt

Table ¿tbl:tab:joint:criterion? reports task accuracy and society diversity for subsets of the EmbedLLM society selected under two criteria: maximising HSE alone, and maximising HSE jointly with task accuracy. The results reveal a diversity-accuracy tradeoff. Subsets selected purely on HSE achieve higher normalised HSE at every size but lower routing task accuracy. Adding task accuracy to the selection criterion recovers routing performance — at \(n{=9}\), KNN-10 matches the full 112-model society exactly (0.65) and oracle reaches 0.92 — but at the cost of lower HSE (0.48 vs.). The HSE + acc. subset at \(n{=9}\) achieves competitive routing accuracy using fewer that 10% of the available models, providing a practical coreset that balances diversity and performance.

References↩︎

---

1. The original HSE formulation uses Euclidean distance and centroid linkage rather than cosine distance and single linkage; we compare both in Appendix 14.↩︎