1 Introduction↩︎

Agent-based Models (ABMs) are characterized by interactions of autonomous and possibly heterogeneous agents, often mediated by an environment or by spatial constraints. Although the behavior of each agent is usually specified by simple local rules, the repeated interaction of many agents may give rise to complex emergent behaviors that are not explicitly encoded in the model. This modeling paradigm has been used in several disciplines, including ecology [1], health care [2], geography [3], and medicine [4]. ABMs are also widely used in the social sciences (see, e.g., [5]–[14]), where they provide a simulation-based way to study phenomena that are too complex to be treated analytically. It is therefore important to provide modeling and methodological extendions for ABMs.

A key feature of many ABMs is that individual behavior is specified by explicit and reproducible rules. This makes the model transparent: one can inspect how agents take decisions, reproduce simulations under controlled conditions, and study how local rules affect global outcomes. In other disciplines, ABMs are named Multi-Agent Systems (MAS) or Collective Adaptive Systems (CAS), see, e.g., [15]–[20].

Recent advances in large language models (LLMs), make it increasingly tempting to enrich ABM agents with more flexible, language-mediated, and agentic capabilities. Instead of hard-coding all decisions symbolically, one may delegate part of an agent’s local reasoning to an LLM, possibly allowing the agent to interpret natural-language information, use tools, or adapt its behavior in a less rigid way. Surveys on LLM-empowered ABMs explicitly frame this as an emerging research direction, emphasizing that LLM agents can enhance perception, reasoning, decision-making, adaptation, and heterogeneous role-playing in multi-agent simulations [21].

This possibility is appealing, but it also raises methodological concerns. LLMs are not standard symbolic rules: they may introduce semantic errors, stochastic variability, prompt sensitivity, tool-call failures, and non-negligible runtime overhead. These concerns are consistent with the broader challenges identified in [21]. Moreover, when an LLM is embedded inside an ABM simulation loop, its failures may affect not only the behavior of one agent, but also the emergent behavior of the whole system. This suggests that LLM-enabled ABM agents should not be treated as transparent replacements for symbolic rules. Rather, their impact should be studied explicitly and quantitatively.

Statistical model checking (SMC) provides a natural framework for this task. SMC is a family of simulation-based analysis techniques from computer science that automates simulation experiments and associates quantitative estimates with statistical guarantees [22], [23]. In particular, black-box SMC only requires that the model can be simulated probabilistically and that observations can be evaluated on simulation states [24], [25]. This makes it suitable for analyzing existing simulation models without translating them into a different formalism. MultiVeStA [26]–[28] is a black-box SMC tool that can be integrated with external simulators. In previous work [29], MultiVeStA was integrated with Mesa, a Python framework for ABM modeling and simulation popular in the social sciences [30], making SMC available for Mesa models.

In this paper, we build on this integration to study the effect of introducing LLM-based agentic capabilities in Mesa. We focus on the classical and influential Schelling segregation model [31]. In the standard model, agents belong to one of two groups and they move according to a rule based on the counting of how many similar neighbours they have. We construct a minimal LLM-extended variant in which the global Schelling dynamics is left unchanged, but one distinguished agent delegates the classification of neighbours’ similarity to an LLM.

This setting is deliberately simple. Its purpose is not to claim that the Schelling model requires LLM-based reasoning, nor to cover all possible forms of agentic ABMs. Rather, it provides a controlled benchmark in which the consequences of replacing a symbolic local decision rule with an LLM-based one can be isolated. This allows us to study two related questions. First, can local LLMs of different sizes reliably perform the semantic classification and tool-call generation required by the agent? Second, when an LLM-enabled agent is embedded in the full simulation, does it measurably affect the transient behavior of the ABM, and at what computational cost?

1.0.0.1 Contribution.

The contribution of this paper is threefold: (i) we extend a Mesa implementation of the Schelling segregation model with an LLM-enabled agent using tool calls; (ii) we study the correctness, robustness, and runtime behavior of local LLMs of different sizes when used for the agent’s local classification task; (iii) we compare the original and LLM-extended models using statistical model checking with MultiVeStA.

Our preliminary robustness experiments show that small local LLMs may fail even simple semantic classification tasks, may generate the wrong tool calls, or may become operationally unusable during multi-neighbor tool-call generation. Larger tested variants pass these unit-style checks, but introduce a measurable runtime overhead. In the full ABM analysis, we use MultiVeStA to estimate the transient evolution of the ratio of happy agents in the original and LLM-extended models, and to statistically compare the resulting trajectories. We find that the LLM-extension does not change the model dynamics, while, as expected, it introduces very significant runtime overhead.

Synopsis. The paper is structured as follows. Section 2 recalls the background on Mesa, statistical model checking, MultiVeStA, and their previous integration. Section 3 presents the LLM-extended Schelling model and the tool-calling mechanism used to implement the agentic decision procedure. Section 4 reports preliminary correctness and robustness experiments for local Qwen3.5 models of different sizes. Section 5 compares the original and LLM-extended Schelling models using statistical model checking. Section 6 concludes the paper.

2 Background: Statistical model checking of Mesa models↩︎

This section provides the necessary background material. We first introduce agent-based models and Mesa, then briefly discuss statistical model checking and MultiVeStA, and finally summarize the previous integration of Mesa and MultiVeStA that we reuse in this work.

2.1 Agent-Based Models and Mesa↩︎

Agent-based models (ABMs) describe systems composed of autonomous agents interacting with each other and, often, with an explicit environment [12]. Each agent is typically equipped with a local state and a set of rules determining how it acts, how it interacts with other agents, and how these interactions affect its state. Although these rules are usually simple and specified at the level of individual agents, their repeated interaction may give rise to complex emergent behavior that is not explicitly encoded in the model.

ABMs are commonly used to describe systems whose complexity makes analytical treatment difficult, thus requiring simulation-based analyses. Examples include models of flocking [32], segregation [31], markets [33], economies [10], innovation dynamics [34], epidemics [35], mobility [36], and many other social or natural phenomena. A simulation of an ABM proceeds by repeatedly activating agents, updating their state, and observing the resulting evolution of the system. Mesa [29] is a Python framework for the development and simulation of ABMs. A Mesa model typically defines a class for the global model, classes for the different types of agents, and a representation of the space or environment in which agents interact. The model class stores global parameters and simulation state, while agents implement the local behavior executed at each simulation step. This modular structure makes Mesa convenient for rapidly implementing ABMs and for experimenting with different model variants.

In this paper, we use the Mesa implementation of the popular Schelling segregation model [31]¹. We will also propose an LLM-extended variant, presented in later sections. The use of Mesa is important for our purposes because the LLM-enabled agent can be introduced by modifying only the local decision procedure of one agent, while leaving the rest of the model dynamics unchanged. This makes Mesa suitable for implementing the controlled model variant studied below.

2.2 Statistical Model Checking and MultiVeStA↩︎

The analysis of ABMs is usually simulation-based: one executes the model many times and studies the values of selected observables [28]. However, the design of simulation experiments may strongly affect the reliability of the conclusions [28], [34], [37]. Too few simulations, arbitrary stopping criteria, or informal comparisons between runs may lead to statistically weak or non-reproducible results.

Statistical model checking (SMC) [22], [23] provides an automated simulation-based approach to the quantitative analysis of stochastic systems. In black-box SMC [24], [25], the model under analysis is treated as a simulator: the analysis engine does not need to inspect its internal structure, but only to run simulations and evaluate observations on simulation states. This makes black-box SMC particularly suitable for existing simulators and modeling frameworks, including ABM frameworks such as Mesa [29] or NetLogo [13]. For example, it has been applied to scenarios as diverse as lending pools in decentralized finance [38], public transportation systems [39], [40], highly-configurable systems [41], [42], business process modeling [43], security threat modeling [44], [45], crowd steering scenarios [46], adaptive robotic systems [18], [47], [48], and CAS in general [49].

MultiVeStA [26]–[28] is a statistical model checker that can be integrated with external simulators. Given a property of interest, MultiVeStA executes a number of independent simulations determined by the required statistical accuracy, and estimates the expected value of the corresponding observable together with a confidence interval. In this way, quantitative statements about the model are accompanied by explicit statistical guarantees.

MultiVeStA supports two main families of analyses: transient analysis, where properties are studied at given points in time, and steady-state analysis, where properties are studied in the long run, when the system reaches a statistical equilibrium [28]. In this paper, we focus on the former. This allows us to estimate, for instance, the expected ratio of agents satisfying a given criterion, such as being happy, at steps \(1,2,\ldots,T\), obtaining a trajectory with confidence intervals. For transient analysis, MultiVeStA also supports statistical comparison of different model parameterizations or variants. This is useful here because we are interested in comparing the original Schelling model with a variant in which one agent delegates part of its local decision process to an LLM.

2.3 Previous Integration of Mesa and MultiVeStA↩︎

In previous work [29], we integrated Mesa with MultiVeStA in order to make statistical model checking directly available for ABMs written in Mesa. The integration follows the black-box philosophy of MultiVeStA: the Mesa model is not translated to another formalism, but is executed by its own simulator and queried through a small interface.

At a high level, the integration requires the Mesa model to provide three basic operations. First, the simulator must be reset before each new simulation, setting a seed provided by MultiVeStA to ensure independent and reproducible runs. Second, it must expose an operation to execute one simulation step, denoted by next. Third, the model must provide an evaluation function, denoted by eval, that returns the value of a requested observable in the current simulation state. These operations are sufficient for MultiVeStA to orchestrate multiple simulations and compute statistically reliable estimates of the desired properties.

The previous integration was validated on two classical ABMs: the Boids flocking model [32] and the Schelling segregation model [31]. The Boids model was used to illustrate transient analysis and counterfactual comparison, while the Schelling one to illustrate steady-state analysis and ergodicity diagnostics.

After that integration, Mesa underwent major updates, evolving from version 2 to version 3, which required updating the integration. In doing so, the integration was made even less intrusive. The updated architecture introduces an additional intermediate class, ModelContainer, which simplifies the integration of new Mesa models. In the previous integration, resetting the simulator before a new simulation did not allow the creation of a fresh Model instance. Consequently, the modeler could be required to identify and clean all relevant internal data structures inside the reset method. The additional abstraction layer introduced by ModelContainer removes this limitation: a new Mesa model can now be created when starting a new simulation.²

In the present paper, we use the same integration perspective, but with a different objective. Our goal is not to validate the Mesa–MultiVeStA integration again, but to analyze the original and LLM-enriched versions of the Schelling segregation model, and to statistically compare their outcomes. To this end, we perform transient analysis of each model version, estimating the expected ratio of happy agents within the first \(T\) steps, and comparing the resulting trajectories to determine whether they differ from a statistical perspective.

3 A Schelling Model with Agentic LLM Capabilities↩︎

3.1 The Schelling Segregation Model↩︎

We consider the classical Schelling segregation model [31], one of the most well-known examples of ABMs in the social sciences. The model has been encoded in several ABM frameworks, including Mesa. It describes a population of agents divided into two groups, commonly represented by two colors, red and blue. These groups represent two types of similar individuals. The model is famous for showing how even a modest local preference for living close to agents of the same group can lead to global segregation patterns.

Space is abstracted as a grid, with agents occupying cell locations. Each cell can contain at most one agent. At each simulation step, an agent observes the agents located in its neighborhood and determines whether it is happy. In the classical formulation, happiness depends on the number or fraction of neighboring agents belonging to the same group. If the agent has enough similar neighbors, it remains happy; otherwise, it becomes unhappy and moves to an empty cell. Repeated application of this local rule may lead the system to configurations in which agents of the same group form spatial clusters.

The model’s behavior is controlled by a small number of parameters: the size of the grid; the homophily threshold, determining how many, or what fraction of, neighbors must be similar for an agent to be happy; the density factor, determining the probability that a cell is occupied during initialization, and thus the number of agents; and the minority percentage, determining the probability that a newly created agent belongs to the minority group. ³ In the Mesa implementation used in this paper, each agent has a symbolic type representing its group. Ordinary agents use this type to classify their neighbors: a neighbor is similar if and only if its type coincides with the type of the current agent. The result of this classification is then used to compute the agent’s happiness.

Listing [lst:schelling-standard-agent] reports the relevant part of the base agent implementation. Method count_similar_neighbors implements the local classification rule by iterating over the current neighbors and counting those whose field type matches the type of the current agent. The method assign_state then computes the fraction of similar neighbors and updates the Boolean field happy. Finally, the method step moves the agent to a randomly selected empty cell whenever it is unhappy.

class SchellingAgent(CellAgent):
    """Schelling segregation agent."""

    def count_similar_neighbors(self, neighbors):
        """Count number of similar neighbors within the specified radius."""
        similar_neighbors = len([n for n in neighbors if n.type==self.type])
        return similar_neighbors

    def assign_state(self) -> None:
        """Determine if agent is happy and move if necessary."""
        neighbors=list(self.cell.get_neighborhood(radius=self.radius).agents)

        # Count similar neighbors
        similar_neighbors = self.count_similar_neighbors(neighbors)

        # Calculate the fraction of similar neighbors
        if (valid_neighbors := len(neighbors)) > 0:
            similarity_fraction = similar_neighbors / valid_neighbors
        else:
            # If there are no neighbors, the similarity fraction is 0
            similarity_fraction = 0.0

        if similarity_fraction < self.homophily:
            self.happy = False
        else:
            self.happy = True
            self.model.happy += 1

    def step(self) -> None:
        # Move if unhappy
        if not self.happy:
            self.cell = self.model.grid.select_random_empty_cell()

This implementation is particularly convenient for our purposes. The classification of neighbors is localized in the method count_similar_neighbors, while the rest of the Schelling dynamics only depends on the number of similar neighbors returned by that method. Therefore, we can replace this local symbolic classification rule with an LLM-based one, while leaving unchanged the computation of happiness, the movement rule, and the overall model dynamics. This makes the Schelling model a controlled setting for studying the impact of introducing agentic LLM capabilities into an otherwise standard ABM.

3.2 Local LLM Infrastructure: Ollama and Qwen3.5↩︎

The LLM-enabled variant of the Schelling model requires repeated interactions between the simulator and an LLM. In this work, we rely on locally served models rather than on remote APIs. This choice is important for two reasons. First, it makes the experimental setup more controllable, since simulations do not depend on the availability, latency, or possible changes of an external service. Second, it allows us to explicitly measure the computational overhead introduced by LLM-based decisions inside the ABM simulation loop.

We use Ollama⁴ as local serving infrastructure for the LLMs. Ollama provides a simple way to download, run, and query LLMs locally, and exposes interfaces that can be used directly from Python. In our implementation, the Mesa model invokes the locally served model whenever the LLM-enabled agent has to classify its neighbors. Therefore, each LLM call becomes part of the simulation dynamics and contributes to the overall runtime of the model.

As LLM family, we consider Qwen3.5 [50] models distributed through Ollama. This is due to the fact that Qwen3.5 is distributed in several variants, with a fine range of models sizes, including those with 0.8b, 2b, 4b, 9b, which can be run on laptop machines.⁵ Such families of models are often obtained by training or adapting models at different scales, possibly using techniques such as distillation [51]. This range of sizes is useful for our study because it allows us to investigate the trade-off between computational cost and reliability of the LLM-enabled agent. Smaller models are cheaper to execute, but may be less reliable in semantic classification and tool-call generation. Larger models may provide more robust behavior, but require more memory and longer execution times.

3.3 Tool Calls for Agentic Decisions↩︎

The key functionality we use is tool-calling. In a tool-calling interaction, the LLM does not only produce textual output: it may also request the execution of predefined functions exposed by the Python program. The Python runtime then executes the requested functions and uses their effects/outputs to continue the computation. In our setting, this mechanism is used to connect the linguistic decision of the LLM to the internal state of the simulated agent. The LLM receives a natural-language description of the current neighborhood and produces calls to tools that update the counters of similar and different neighbors.

In our implementation, the LLM is created once in the constructor of the Schelling_LLM model class, which extends the original Schelling model class. The LLM is stored in an attribute of the model, llm, and can therefore be reused by all LLM-enabled agents. This design is useful also for future extensions with more than one LLM-enabled agent, since all of them may share the same local LLM instance. The LLM is initialized as follows.

from langchain_ollama import ChatOllama
self.llm = ChatOllama(   model=llm_model_name,   validate_model_on_init=True,   
                         disable_streaming=True, temperature=0
  ).bind_tools([increment_similar_count, increment_different_count])

The parameter llm_model_name selects the local model to use, e.g. a Qwen3.5 variant discussed above or any other LLM model available in Ollama. We set the temperature to zero in order to reduce the variability of individual LLM answers and make the experiments more reproducible. Intuitively, temperature controls the randomness of the decoding process: lower values make the model more likely to select high-probability outputs. The call to bind_tools exposes two Python functions to the model: one for incrementing the number of similar neighbors and one for incrementing the number of different neighbors.

These two tools are shown in Listing [lst:schelling-tools]. They receive the current agent as argument and update one of its counters. The return values are not used to control the model dynamics directly; the relevant effect is the update of the internal counters of the current agent.

from langchain.tools import tool

@tool
def increment_similar_count(current_agent):
    """Increments the number of agents similar to the current agent.

    Args:
        current_agent: the agent for which to increment the similar count """
    current_agent.similar_neighbors += 1
    return 1

@tool
def increment_different_count(current_agent):
    """Increments the number of agents different from the current agent.

    Args:
        current_agent: the agent for which to increment the different count
    """
    current_agent.different_neighbors += 1
    return 0

TOOL_MAP = {
    "increment_similar_count": increment_similar_count,
    "increment_different_count": increment_different_count,
}

The resulting interaction pattern is the following. First, the simulation constructs a prompt describing the current local situation of the LLM-enabled agent. In our case, this prompt contains the natural-language descriptions of the neighboring agents. Second, the LLM receives the prompt and emits a sequence of tool calls. Third, the Python runtime executes the requested tools by looking up their names in TOOL_MAP. Finally, the execution of the tools updates the fields similar_neighbors and different_neighbors of the current agent.

This design keeps the intervention of the LLM local and explicit. The LLM is not allowed to move agents, change the grid, or directly determine whether an agent is happy. It can only request calls to the two available counter-update functions. The standard Schelling dynamics then uses the resulting counter values to compute happiness and movement, as in the original model.

3.4 Extending Schelling Agents with Agentic LLM-Based Capabilities↩︎

We now describe how the standard Schelling agent is extended with LLM-based agentic capabilities. The extension is deliberately minimal. The original Schelling dynamics is preserved: agents still occupy cells in a grid, observe their neighbors, compute whether they are happy, and move to an empty cell when unhappy. The only modified component is the local procedure used by one agent to classify neighbors as similar or different.

In the standard model, each agent has a symbolic type, and neighbor classification is performed by directly comparing symbolic types. In the LLM-extended model, each agent still has this symbolic type, but it is also associated with a natural-language self-description. In our implementation, agents of type 0 describe themselves with the sentence Sky is beautiful, while agents of type 1 use the sentence Fire is beautiful. Ordinary agents continue to classify neighbors symbolically. The LLM-enabled agent, instead, receives the natural-language descriptions of its neighbors and delegates the classification task to the LLM through the tool-calling mechanism described above.

The model class Schelling_LLM extends the class Schelling from the original Schelling model, which in turn extends the Model class from Mesa. It creates agents of type SchellingAgent_LLM, which extends the standard SchellingAgent. In the experiments reported below, one agent is created with use_llm=True, while all other agents are created with use_llm=False. This makes all agents instances of the same extended class, while only one of them actually uses the LLM during the classification of neighbors.

Listing [lst:schelling-llm-agent] reports a compact version of the agent. The class defines a common prompt, describing the general role of the simulated individual, and two role-specific prompts, one for the blue team and one for the red team. The constructor combines the common and role-specific prompts depending on the symbolic type of the agent, and assigns the corresponding natural-language self-description.

from langchain.messages import AIMessage, HumanMessage, SystemMessage

class SchellingAgent_LLM(SchellingAgent):
    commonPrompt = """
    You are an agent in a simulation model made for a scientific research.
    You are not an assistant to a user; you are a simulated individual
    interacting with other agents.  
    The model simulates an interaction with another agent by feeding you
    a message generated by the other agent. You have to read the message
    and decide whether to consider the other agent 'similar' or 'different'.
    Depending on your opinion, you shall update the count of similar or
    different agents."""

    bluePrompt = """
    Your role is "Blue Team": you have a particular passion for blue color:
    whatever is blue, you like. You do not like other colors."""

    redPrompt = """
    Your role is "Red Team": you have a particular passion for red color:
    whatever is red, you like. You do not like other colors."""

    def __init__(self, model, cell, agent_type: int,
                 homophily: float = 0.4, radius: int = 1,
                 use_llm=True) -> None:
        super().__init__(model, cell, agent_type, homophily, radius)
        if agent_type == 0:
            self.system_prompt = self.commonPrompt + self.bluePrompt
            self.describe_yourself = "Sky is beautiful."
        else:
            self.system_prompt = self.commonPrompt + self.redPrompt
            self.describe_yourself = "Fire is beautiful."
        self.use_llm = use_llm

    def count_similar_neighbors(self, neighbors):
        if not self.use_llm:
            self.similar_neighbors=super().count_similar_neighbors(neighbors)
        else:
            self.similar_neighbors = 0
            self.different_neighbors = 0
            neighbor_descriptions = [n.describe_yourself for n in neighbors]
            n_neighbors = str(len(neighbors))
            prompt = "You have " + n_neighbors + " neighbors. "
            prompt += "This is the list of messages received by each other agent. "
            prompt += "Produce all " + n_neighbors + " tool calls needed to update "
            prompt += "the count of similar and different neighbors:\n\t"
            prompt += str(neighbor_descriptions)
            messages = [SystemMessage(content=self.system_prompt), 
                        HumanMessage(content=prompt)]
            result = self.model.invoke(messages)
            if isinstance(result, AIMessage) and result.tool_calls:
                for tool_call in result.tool_calls:
                    tool_fn = TOOL_MAP[tool_call["name"]]
                    tool_args = {'current_agent': self}
                    tool_fn.invoke(tool_args)
            return self.similar_neighbors

The key method is count_similar_neighbors. Its interface is the same as in the original Schelling agent, which allows the rest of the model to remain unchanged. If use_llm is false, the method simply delegates to the original symbolic implementation. If use_llm is true, the counters of similar and different neighbors are first reset. The agent then collects the natural-language descriptions of its current neighbors and constructs a prompt asking the LLM to produce one tool call per neighbor. The returned tool calls are executed by looking up the corresponding Python functions in TOOL_MAP. Their execution updates the fields similar_neighbors and different_neighbors of the current agent.

After the method returns, the remaining Schelling dynamics is exactly the same as in the standard implementation described in Listing [lst:schelling-standard-agent]. The value returned by count_similar_neighbors is used by assign_state to compute the fraction of similar neighbors and determine whether the agent is happy. If the agent is unhappy, the standard movement rule is applied. Thus, the LLM does not control the scheduler, the grid, or the movement policy. It only replaces one localized symbolic decision procedure with a tool-mediated, language-based one.

This design has two advantages for our analysis. First, it provides a controlled way to study the impact of LLM-based agentic capabilities, since the original and extended models differ only in one local decision mechanism. Second, it allows us to separate failures of the LLM component from the rest of the ABM dynamics: semantic misclassifications, missing tool calls, or repeated-generation failures can be attributed to the LLM-enabled classification step rather than to changes in the global model structure.

4 Robustness Experiments↩︎

Before studying the impact on emergent behaviours of the LLM injection into the ABM, we must first establish whether the LLM can reliably perform the local task it is given. In particular, we are interested in three possible outcomes: the LLM may produce the correct tool call, it may produce an erroneous tool call, or it may fail operationally, for instance by getting stuck during generation.

The following subsection describes the experimental protocol. The results are then discussed in Section 4.2.

4.1 Experimental Protocol↩︎

We consider four local LLMs from the Qwen3.5 family: 0.8b, 2b, 4b, and 9b. For each model, we instantiate the LLM with the same configuration used in the ABM experiments: local execution through Ollama, temperature set to zero, and the two counter-update tools bound to the model.

Each experiment consists of a system prompt and a user message (Listing [lst:schelling-llm-agent]). The former is obtained by combining the common simulation prompt with one of the two role-specific prompts. In the blue condition, the LLM is instructed to play a blue-team agent, which likes blue and does not like other colors. The red condition is similar. The same user message is tested under both conditions.

We consider four user messages of increasing semantic difficulty:

1. I like blue. I do not like red.

2. The sky is beautiful. The fire is not

3. The sky is beautiful.

4. You have 3 neighbours. This is the list of messages received by each other agent. Produce all 3 tool calls needed to update the count of similar and different neighbors. [’Sky is beautiful.’, ’Sky is beautiful.’, ’Fire is beautiful.’]

The first message encodes the simplest experiment possible: it explicitly mentions a preference for blue and a dislike for red. The second and third messages are less direct, since the model must connect terms such as sky and fire to the role-specific preferences encoded in the system prompt.

The fourth message type, M4, is meant to test whether increasing the semantic difficulty of user messages, by requiring the generation of more tool calls, negatively affects reliability. Thus, M4 is both the most difficult robustness check and also the closest to the ABM setting considered in the simulation: in fact, an agent updates its state by considering multiple neighbors. It must be highlighted that, while M4 considers only three neighbours, in the real ABM simulation the agent can have up to eight neighbors. For each pair consisting of an LLM model and a user message, we perform three iterations with the blue role and three iterations with the red role.

Repeating the same test is useful even with temperature set to zero, because tool-call generation may exhibit implementation-level variability, and because operational failures such as non-termination are themselves relevant for the robustness of the approach. We also measure the wall-clock time of each iteration.

Listing [lst:preliminary-test-function] reports the core function used for our preliminary checks. The function builds the message sequence, invokes the LLM, prints the generated tool calls, and executes them via TOOL_MAP. In these experiments, executing a tool only prints the corresponding action, allowing us to inspect whether the selected tool is the expected one.

def test_interaction_and_tool_call(system_prompt, user_query, llm):
    messages = [ SystemMessage(content=system_prompt),
             HumanMessage(content=user_query)      ]

    print("Invoking LLM")
    ai = llm.invoke(messages)

    print("Tool calls:", getattr(ai, "tool_calls", None))
    print("Result:", ai.content)

    if isinstance(ai, AIMessage) and ai.tool_calls:
        for tc in ai.tool_calls:
            tool_fn = TOOL_MAP[tc["name"]]
            tool_args = tc.get("args", {}) or {}
            tool_fn.invoke(tool_args)

We classify each iteration into one of three outcomes:

• correct: the LLM generates the expected tool call or sequence of tool calls;

• erroneous: the LLM terminates but generates an incorrect tool call, a tool call inconsistent with the assigned role, or an incorrect number of tool calls;

• stuck: the LLM does not complete the generation in a usable way.

For M1-M3, the expected behavior is one tool call: increment_similar_count if the message is compatible with the role of the current agent, and the other function otherwise. For M4, the expected behavior is a sequence of three tool calls, one for each neighbor description. Since the first two descriptions are Sky is beautiful. and the last is Fire is beautiful., the expected calls are two increment_similar_count calls and one increment_different_count for a blue-team agent, and vice versa for a red-team one. Therefore, producing only one or two tool calls is considered erroneous, even if the generated calls are individually of the correct type. Since the tools used in these preliminary checks do not identify the specific neighbor to which a call refers, we evaluate correctness by considering the resulting number of similar and different calls, not their order.

4.2 Results↩︎

We now discuss the preliminary robustness and runtime experiments.

4.2.1 Robustness.↩︎

Table 1 summarizes the qualitative outcomes. We write C for correct, SE for semantic error, AE for arity error, and ST for stuck generation. Semantic and arity error means that the model generates, respectively, a wrong tool call, or a wrong number of calls. Errors are marked in red.

4.2.1.1 Responses to M1.

All models except qwen3.5:0.8b complete all six iterations correctly. The smallest model succeeds for the blue role, but gets stuck already on the first red-role iteration. Thus, even the most direct semantic experiments exposes an operational fragility in the smallest model.

4.2.1.2 Responses to M2.

Both qwen3.5:0.8b and qwen3.5:2b behave correctly for the blue role, but systematically produce the wrong tool call for the red role: they call increment_similar_count instead of the other. In contrast, qwen3.5:4b and qwen3.5:9b complete all iterations correctly.

4.2.1.3 Responses to M3.

The model qwen3.5:0.8b fails systematically in both roles, selecting always the opposite tool. Larger models complete all iterations correctly.

4.2.1.4 Responses to M4.

The model qwen3.5:0.8b produces the correct number of similar and different calls for the blue role, but generates only two calls for the red role. The model qwen3.5:2b gets stuck in both roles, and therefore cannot be used reliably for this setting. In contrast, qwen3.5:4b and qwen3.5:9b generate the expected number and type of tool calls in all iterations and for both roles.

4.2.2 Runtime.↩︎

Table 2 reports average runtimes, in seconds, for the completed iterations. Stuck executions are omitted. Experiments were run on an Apple M4 machine with 24GB of unified memory. These values should not be interpreted as a detailed benchmark, since they depend on the local machine and on the specific serving infrastructure. Nevertheless, they give a useful indication of the overhead introduced by LLM-based decisions.

4.2.3 Discussion.↩︎

Overall, these preliminary experiments show that the LLM component cannot be treated as a transparent replacement for a symbolic rule. The smallest model, qwen3.5:0.8b, is both semantically unreliable and operationally fragile. The model qwen3.5:2b handles some single-message experiments, but fails on one indirect classification task and gets stuck on the multi-neighbor tool-call task. The two larger tested variants, qwen3.5:4b and qwen3.5:9b, pass all preliminary checks. These results motivate using only models that pass these unit-style checks in the full ABM experiments. The runtime analysis suggests using qwen3.5:4b, rather than qwen3.5:9b, in the full simulations.

5 Statistical model checking↩︎

Here we perform statistical model checking with MultiVeStA of the original and LLM-extended model. We report on transient analyses on the (expected value of the) ratio of happy agents in all simulation steps from 1 to 60. Considering the relatively high runtime per LLM invocation shown for M4 in Table 2, in order to keep SMC runtime reasonable, we performed these experiments using only one LLM-extended agent, and a more powerful machine, namely an NVIDIA DGX Sparx with 120GB of unified memory. MultiVeStA is parametrized such that for each estimation it computes a confidence interval of width at most 0.1 and statistical confidence of 95%. In order to understand whether the LLM-extension leads to statistically significant changes in the system dynamics, we also check whether the estimates are equal point-wise across the two models. We do this via t-tests, as discussed here [28]. We also report runtimes.

Figure 1 shows the analysis results. We can see that the two model variants produce visually-undistinguishable trajectories where the ratio of happy agents starts from about 0.65 and stabilizes at about 1.0 within 60 simulation steps. This is confirmed by the t-tests results shown in the bottom part of the figure.

Both SMC analyses required only 20 simulations, meaning that, with the used parametrization, the model has little variance. The runtime for the original model is just about 2 seconds, while that of the LLM-enriched model is 8917 seconds. It is interesting to note that the LLM-enriched variant performed 61 LLM invocations (to move from step 0 to step 60) for each of the 20 simulations, that is, about 7.3 seconds per invocation.

6 Conclusions↩︎

Large language models (LLMs) make it tempting to enrich agent-based models with language- and tool-mediated decisions. However, LLM-enabled agents are not transparent replacements for symbolic rules: they may introduce semantic errors, operational failures, variability, and runtime overhead. We studied these issues by extending a Mesa implementation of the Schelling segregation model with one LLM-enabled agent, analyzing it with statistical model checking (SMC).

The extension was deliberately minimal: the original Schelling dynamics was left unchanged, while one local decision procedure—the classification of neighbors as similar or different—was replaced by an LLM-based tool-calling mechanism. Before full ABM simulations, we performed unit-style robustness checks. Smaller local models, qwen3.5:0.8b and qwen3.5:2b, exhibited semantic errors, stuck generations, or failures in multi-neighbor tool-call generation. In contrast, qwen3.5:4b and qwen3.5:9b passed all preliminary checks, with qwen3.5:4b offering a better runtime compromise.

We then used SMC, as implemented in MultiVeStA, to analyse the model variants, and to check whether the LLM-extension led to statistically significant differences in the model dynamics. The two model variants produce same dynamics, with significant runtime overhead due to the many LLM invocations.

This work is a first step towards a systematic analysis of agentic ABMs. Future work will consider richer decision tasks, more LLM-enabled agents, different ABMs, and systematic prompt and tool-interface designs. For example, here we used standard setups for the LLMs, while it could be interesting to study the impact on robustness and runtime of settings such as explicit enabling/disabling of thinking mode. We will also investigate process mining techniques to explain SMC results [52], including stochastic ones [53]. We will also consider the application of anomaly detection techniques [54] on the time series generated by SMC simulations. Overall, LLM-enabled agents should be treated as stochastic and fallible components whose correctness, robustness, and computational cost must be analyzed explicitly.

References↩︎