1 Introduction↩︎

Modern AI agents do more than generate text. They plan, call tools, inspect intermediate results, revise strategies, and sometimes collaborate with other agents. As these systems become more capable, evaluating them only by final success rate becomes increasingly incomplete. Two agents may both solve a task, but one may do so directly while another uses unnecessary tool calls, unstable reasoning paths, or excessive exploration. Two agents may both fail, but one may fail after gathering useful evidence while the other may behave randomly. Traditional metrics such as success, reward, cost, and latency capture important outcomes, but they do not fully describe the agent’s behavior [1], [2].

Entropy offers a simple and interpretable way to measure uncertainty and diversity in agent behavior. In information theory, entropy measures how unpredictable a distribution is. Applied to agents, entropy can describe how varied the agent’s actions are, how diverse its strategies are, how specialized its tool use is, and how much uncertainty it reduces during a task.

This paper proposes EEA as a practical evaluation layer for agent systems. The central idea is direct: collect traces from agent runs, normalize them into a common representation, and compute entropy-based behavioral metrics. EEA does not attempt to declare one universal definition of intelligence. Instead, it provides a set of measurable behavioral signals that help researchers and engineers compare agents more deeply.

2 Motivation↩︎

Most agent benchmarks focus on whether the agent completed the task. This is necessary, but not sufficient. Consider three agents that all answer a question correctly: one searches once and answers, another searches repeatedly before answering, and a third answers from memory without checking evidence. A success metric treats all three runs equally, even though their behavior differs in important ways.

Similarly, an agent with very low entropy may be overly deterministic. This can be useful for simple tasks, but risky in changing environments. An agent with very high entropy may be exploratory, but it may also be chaotic. A capable agent may show controlled entropy: broader exploration early in a task, followed by lower uncertainty as evidence is gathered.

EEA is motivated by five questions:

• Can we measure how diverse an agent’s behavior is?

• Can we tell whether an agent uses tools in a focused or scattered way?

• Can we observe whether uncertainty decreases over time?

• Can we distinguish robust variation from unstable outcomes?

• Can entropy-based metrics complement success rate, reward, latency, and cost?

3 Framework↩︎

EEA assumes that each agent run can be represented as a sequence of events. An event may be a tool call, model call, planning step, action, or final answer. A run may also include task success, outcome label, cost, latency, and optional probability distributions over hypotheses before and after evidence gathering.

3.1 Action Entropy↩︎

Action entropy measures how varied an agent’s actions are during one or more runs. If an agent repeatedly selects the same action, entropy is low. If it selects many different actions with similar frequency, entropy is higher. For actions \(a_i\) with probabilities \(p(a_i)\), action entropy is:

\[H_A = -\sum_i p(a_i)\log_2 p(a_i)\]

Low action entropy can indicate focus, but it can also indicate rigid behavior. High action entropy can indicate exploration, but it can also indicate instability. The value is most useful when interpreted alongside success and cost.

3.2 Trajectory Entropy↩︎

An agent trajectory is the ordered sequence of steps used to complete a task, such as:

search -> read -> answer
search -> code -> test -> answer

Trajectory entropy measures diversity across complete strategies. This is useful when the same task is run multiple times or when several agents are compared on the same benchmark. Low trajectory entropy means the agent tends to follow the same path. Higher trajectory entropy means it uses a wider range of strategies.

3.3 Tool Entropy↩︎

Tool entropy measures how an agent distributes its tool use. A tool-using agent may call search, code execution, memory, database retrieval, or external APIs. Tool entropy helps describe whether the agent relies heavily on one tool or spreads its behavior across many tools. A low value may show specialization. A high value may show broad exploration or inefficient tool selection.

3.4 Information Gain↩︎

Some tasks involve uncertainty over possible answers or hypotheses. If an agent starts uncertain and gathers evidence, a useful agent should reduce uncertainty. EEA measures this using information gain:

\[IG = H_{\text{before}} - H_{\text{after}}\]

Here, \(H_{\text{before}}\) is the entropy of the agent’s belief or hypothesis distribution before evidence is gathered, and \(H_{\text{after}}\) is the entropy after the agent has used evidence, tools, or intermediate reasoning. Positive information gain indicates that the agent reduced uncertainty. Negative information gain suggests that the agent became more uncertain. This metric is optional because not all systems expose probability distributions.

3.5 Exploration Efficiency↩︎

High entropy alone is not a sign of intelligence. An agent can be highly unpredictable and still fail. EEA therefore combines success and entropy through exploration efficiency:

\[EE = \frac{S}{H_A + \epsilon}\]

where \(S\) is success rate and \(\epsilon\) is a small value to avoid division by zero. Exploration efficiency rewards agents that succeed without excessive behavioral randomness, although difficult tasks may legitimately require more exploration.

In the implementation, these metrics can also be combined into an Entropic Agent Score (EAS). EAS is a configurable summary score that rewards useful uncertainty reduction and successful task completion while accounting for behavioral entropy and cost. It is not meant to be a universal score for all agents; it is a compact comparison signal for a chosen benchmark.

3.6 Robustness Entropy↩︎

Robustness can be studied by running the same task multiple times. A robust agent may show some variation in trajectory while still producing stable outcomes. EEA therefore compares trajectory entropy with outcome entropy. A common desirable pattern is moderate trajectory entropy, low outcome entropy, high success rate, and controlled cost.

4 Implementation↩︎

The accompanying implementation is a Python package called entropy-agent-eval. It is designed around a framework-neutral data contract. Agent frameworks do not need to adopt a new runtime. They only need to convert their traces into a normalized AgentRun representation.

{
  "task_id": "qa-001",
  "trajectory": ["search", "read", "answer"],
  "success": true,
  "cost": 0.08
}

Richer traces can include event types:

{
  "task_id": "coding-42",
  "events": [
    {"kind": "tool", "name": "search"},
    {"kind": "tool", "name": "python"},
    {"kind": "action", "name": "answer"}
  ],
  "success": true
}

The package includes core entropy metrics, an EntropyEvaluator, a configurable composite score, JSON and JSONL loading, a command-line interface, a generic event recorder, adapters for LangChain and Google ADK-style traces, a small benchmark harness, and optional plotting utilities. The implementation intentionally avoids making any agent framework a required dependency. This makes the evaluator usable with live callbacks, exported logs, database traces, observability systems, and custom research benchmarks.

The code for this work is available at https://github.com/olahsymbo/agent-eval. The repository includes the EEA library, integration adapters, benchmark scripts, experiment tasks, and the Learning Roadmap Agent experiment used in this paper.

5 Experiment↩︎

We evaluate EEA in two stages. First, we use a controlled benchmark with reference agent patterns. This checks that the metrics behave in a consistent way when the agent traces are known. Second, we run a real Learning Roadmap Agent through two different agent frameworks: LangChain and Google ADK. The goal is not to prove that one framework is better than the other. The goal is to show that EEA can record and compare different agent systems using the same trace format.

5.1 Controlled Benchmark↩︎

The controlled benchmark compares four reference agent patterns using synthetic traces whose behavior is known in advance: a direct LLM agent, a search-based ReAct agent, a search-and-code ReAct agent, and a planner-executor agent. The task set contains six tasks across factual question answering, multi-hop reasoning, and coding/debugging. Each agent is run three times on each task, producing 72 normalized runs in total.

Each run records the trajectory, success label, outcome label, latency estimate, cost, and before/after hypothesis distributions for information gain. This benchmark is useful because the behavior is easy to inspect. It gives a simple way to check that action entropy, tool entropy, trajectory entropy, and information gain respond to different agent patterns.

The controlled benchmark shows the kind of signal EEA is designed to expose. The direct-llm agent has the lowest trajectory entropy because it follows a fixed two-step path, but it also has the lowest success rate. The planner-executor agent has the highest success rate and information gain, but also higher cost, latency, and action entropy. The react-search and react-search-code agents show how tool use changes the behavioral signature. These differences are not visible from success rate alone.

5.2 Learning Roadmap Agent↩︎

We also built a Learning Roadmap Agent. The agent receives a learner profile, a learning goal, time constraints, and expected outcomes. It must produce a roadmap with an overview, prerequisites, weekly plan, projects, resources, and assessment checkpoints. This task is useful because a good answer requires planning, structure, adaptation to the learner, and clear deliverables.

The same task set was run through two implementations. The LangChain version used an OpenAI chat model through LangChain [3]. The Google ADK version used Gemini 2.5 Flash through Google’s Agent Development Kit [4]. Both implementations used the same roadmap planning helpers, the same task prompts, the same grading rubric, and the same EEA trace schema. The task set contained three roadmap requests: one for production LLM agents, one for Python data analysis dashboards, and one for AI-assisted frontend applications. Each framework ran each task once, producing six total runs.

5.3 Learning Roadmap Results↩︎

Both implementations completed all roadmap tasks successfully. They also had the same action entropy and tool entropy because both were normalized through the same planning trace: learner assessment, module selection, weekly schedule construction, assessment design, and model response. This is important because it shows that EEA can compare different frameworks through a shared behavioral representation.

The Google ADK run produced slightly higher information gain and Entropic Agent Score, while the LangChain run recorded a small nonzero token-derived cost. In this experiment, the Google ADK cost is shown as zero because the run did not return enough token-pricing information for the same cost estimate used by the LangChain path. It should therefore be read as missing or unestimated cost, not as evidence that the run was free. We do not treat this as evidence that Google ADK is generally better than LangChain. The experiment is too small for that kind of claim. The stronger point is that the same evaluator can record both systems, normalize their traces, and compare their behavior using the same metrics.

5.4 Limitations↩︎

This experiment is small. It uses three roadmap tasks and one repetition per framework. The automatic grading rubric checks for required sections and expected terms, but it does not replace human judgment. A stronger study should use more tasks, repeated runs, more models per framework, human review, and more careful cost accounting. Even with these limits, the experiment shows the core value of EEA: it makes agent behavior easier to compare across frameworks.

6 Discussion↩︎

EEA provides a practical way to inspect agent behavior without requiring access to model internals. It treats the agent as an observable system and studies the distribution of its actions, tools, trajectories, and outcomes. This makes the framework useful for both research and production evaluation.

The main advantage of entropy-based evaluation is that it captures behavioral shape. It can reveal when an agent is succeeding inefficiently, failing chaotically, overusing tools, or following overly rigid patterns. It also gives researchers a way to compare agent architectures using more than final answers.

However, entropy metrics require careful interpretation. High entropy is not always bad, and low entropy is not always good. The same value can mean different things depending on task difficulty, agent design, and environment. For this reason, EEA is best used alongside success rate, cost, latency, human judgment, and task-specific evaluation.

Another limitation is that some metrics depend on trace quality. If an agent framework does not expose intermediate steps, tool calls, or uncertainty states, the evaluator can only compute partial metrics. Standardized agent tracing would make entropy-based evaluation more powerful.

7 Conclusion↩︎

This paper introduced Entropy-Based Evaluation of AI Agents, a lightweight framework for measuring agent behavior through entropy. EEA complements traditional metrics by describing action diversity, strategy variation, tool-use patterns, uncertainty reduction, exploration efficiency, and robustness. The framework is designed to be practical: it works with normalized traces and can integrate with existing agent libraries, custom systems, and observability logs.

The main claim is not that entropy alone defines intelligence. Rather, entropy provides a useful behavioral lens. When combined with success, cost, latency, and outcome quality, it can help researchers and engineers better understand how agents explore, adapt, and fail.

References↩︎