1 Introduction↩︎

Large language models (LLMs) are increasingly applied to information retrieval, design assistance, and analysis in architecture, engineering, and construction (AEC) [1], [2]. Structural-engineering agents, however, cannot be adequately evaluated as single-response systems. The validity of an engineering conclusion depends on a sequence of mutually consistent artifacts: interpreted requirements, a structural model, assumptions and units, solver inputs, analysis results, applicable checks, and a final report. A fluent answer or syntactically valid script can remain unusable if an upstream artifact is missing or inconsistent, or if the resulting workflow cannot be executed. As Figure 1 illustrates, an agent must expose the computational evidence behind its answer rather than return only a plausible endpoint. Throughout this paper, the evidence chain refers to the linked artifacts, tool executions, validation records, and terminal decisions that support a reported engineering result.

We therefore formulate structural-engineering agent evaluation at the workflow level. In this setting, success requires more than arriving at a numerically plausible answer: the agent must select an appropriate workflow path and engineering capability, construct a computable model, validate it using the checks available within the system’s declared scope, invoke a compatible backend, execute or abstain safely, and ground its report in the resulting trace. The unit of evaluation is thus the complete evidence chain rather than the final text. This formulation also makes failures localizable: an incorrect result can be attributed to routing, information extraction, model construction, validation, backend execution, code checking, or reporting.

Recent studies have progressively extended structural-engineering automation from isolated design generation to analysis-integrated and agent-assisted workflows. Data-driven methods have broadened the solution space of structural layout design [3], while building-scale generative frameworks have coupled design generation with structural analysis [4]. Diffusion-based approaches continue this direction by generating shear-wall layouts and evaluating them using structural-performance metrics [5]. In parallel, LLM-based systems have begun to support engineering tasks such as code-compliant design and robust structural analysis [6], [7]. Multi-agent orchestration and retrieval-augmented calculation have also been used to support task decomposition, tool coordination, and knowledge grounding [8], [9]. However, most existing studies focus on individual capabilities or partial workflows. A key remaining question is whether an engineering agent can maintain consistency across the complete evidence chain, from interpreting user intent and constructing model artifacts to validation, solver execution, code checking, and final report generation.

To study this gap, we present StructureClaw, a skill-guided, tool-enabled, and artifact-centered workbench for structural-engineering agents. Given a natural-language request and optional project files, the agent selects governed domain skills and typed tools for structure-type detection, parameter extraction, model construction, validation, analysis, code checking, and report generation. The defining design choice is shared artifact state: each stage consumes and produces inspectable objects, including model drafts, validation diagnostics, tool-call records, solver outputs, code-check results, and reports. Unsupported backends and unresolved assumptions are represented explicitly and trigger clarification, safe non-execution, or unsupported termination as appropriate, rather than being hidden behind generated prose.

StructureClaw implements this design through a ReAct-style execution loop, a shared structural-model protocol, domain-specific skills, typed tools, backend providers, and persistent artifact records. The execution loop builds on reasoning–action interaction and program-aided computation [10], [11], while the shared artifact state records models, validation results, solver outputs, and engineering checks throughout execution. Open-source structural analysis is performed using OpenSees through its OpenSeesPy interface [12], [13], with provider interfaces supporting additional analysis engines when available. The human-verification paths shown in Figure 2 are optional review interfaces and are not used during automated benchmark execution. The public release includes the skill manifests, structural-model protocol, validation tools, benchmark scenarios, test fixtures, and evaluation runners used in this study.

A traceable workbench alone does not establish reliability; evaluation must also execute and inspect the same evidence chain. OSWorld tests computer use [14], while \(\tau\)-bench targets policy-constrained tool interaction [15]. AEC-Bench evaluates multimodal agentic tasks [16], whereas EngDesign uses simulation-validated engineering design [17]. These benchmarks, however, do not jointly evaluate the structural-engineering chain of information extraction, model validity, backend-aware execution, safe non-execution, and report faithfulness. We therefore introduce StructureClaw-Bench, a controlled benchmark built around the same evidence chain as the workbench. Its 150 scenarios cover standard workflow execution, interactive robustness, and multimodal structural-model reconstruction. Each scenario is specified by assertions over routing, artifacts, execution, interaction, and reporting, and its Success Rate records the percentage of scenarios in which all required assertions pass in a single run. We use the benchmark as a diagnostic testbed under explicit assumptions, not as certification of construction-ready designs.

In summary, this paper makes three contributions:

• Artifact-centered formulation. We define structural-engineering agent success over a complete evidence chain, requiring consistency among user intent, model artifacts, validation results, solver outputs, supported code checks, and the final report.

• Governed executable workbench. StructureClaw combines domain skills, typed tools, a shared structural-model protocol, backend-aware execution, and persistent artifact state to make intermediate engineering decisions inspectable and reviewable.

• Executable benchmark and diagnostic evaluation. StructureClaw-Bench provides 150 scenarios evenly split between Chinese and English prompts, with typed assertions, reference fixtures, and multimodal inputs. Across ten agent-model configurations on 50 standard cases, its assertion-level diagnostics quantify the system-level difference between automatic and generic-only execution and expose failures in invalid-value handling and structural-model reconstruction.

2 Related Work↩︎

2.1 Generative Design & Structural LLM Systems↩︎

Generative AI has expanded structural-design automation from component design to building-scale layout and topology generation [4]. At the component level, neural, reinforcement-learning, graph, and diffusion methods automate structural design or member sizing [18]–[21]. At the building scale, convolutional and adversarial models infer or generate shear-wall layouts from architectural constraints [22], [23], while related methods address steel frame–brace systems [24], [25]. Graph-based approaches generate structural topology, shear-wall layouts, and beam layouts [26]–[28]. Physics-guided models embed explicit design rules during generation [25], whereas analysis-enabled studies generate shear-wall layouts and evaluate them through downstream structural analysis [5], [29]. Generative-AI assistants provide interactive structural-design support [30], while LLM-based controllers coordinate natural-language-driven design and optimization tasks [31]. Most of these studies evaluate generated designs or task-specific outputs rather than consistency across a complete executable workflow; StructureClaw instead governs and evaluates the evidence chain from user intent to modeling, validation, analysis, checking, and reporting.

AEC LLM systems have progressed from information access toward model authoring and executable engineering workflows [2]. BIMS-GPT retrieves BIM information through language [32], while Text2BIM produces editable building models [33]. BIMgent and MCP4IFC further support model authoring through computer use and standardized IFC-based tool interaction, respectively [34], [35]. LLM-based compliance systems connect regulatory interpretation with model-based execution and violation reporting [36], [37]. In structural engineering, recent systems address code-compliant reinforced-concrete design [6]. Other systems automate structural-analysis word problems and two- and three-dimensional frame analysis [38]–[40]. Separate work evaluates beam-analysis reliability across repeated runs and robustness under varied loading and boundary conditions, and analyzes recurring model-generation and code-execution errors [7]. Cross-platform execution and retrieval-augmented calculation broaden the available tools and knowledge sources [9], [41]. MASSE adds role-separated analysis, design, and adequacy verification around engineering tools [8], while Heo’s staged pipeline adds an explicit verification–correction loop and policy-based routing to a deterministic solver [42]. However, most evaluate a specific task, backend, or output artifact rather than consistency across the full evidence chain. StructureClaw instead evaluates whether governed skills and backend providers can maintain consistency across a shared typed artifact protocol. StructureClaw-Bench additionally tests safe non-execution and trace-grounded reporting across 150 scenarios.

2.2 Tool-Augmented and Skill-Based Agents↩︎

Tool-augmented LLMs provide a basis for executable reasoning. ReAct interleaves reasoning and action [10]; Toolformer learns API calls [43]. PAL delegates steps to programs [11]; Gorilla selects APIs from large collections [44]. HuggingGPT orchestrates external models [45]. API-Bank and ToolLLM test multi-step API use [46], [47]. BFCL focuses on function calling [48]. Voyager and SkillAct study reusable behaviors [49], [50]. Recent work systematizes agentic skills beyond atomic tool use [51].

StructureClaw instantiates this paradigm within a domain-specific structural-engineering execution contract. A StructureClaw skill declares supported structural systems, input and output schemas, validation requirements, runtime dependencies, backend-provider constraints, and artifact outputs. This design distinguishes a governed domain capability from a prompt template or unconstrained API wrapper and makes its preconditions and products visible to the evaluator.

2.3 Executable Agent & Engineering Benchmarks↩︎

LLM evaluation increasingly uses process-aware environments. AgentBench and SWE-bench evaluate interaction and software repair, respectively [52], [53]. OSWorld and \(\tau\)-bench test computer use and policy-constrained interaction, respectively [14], [15]. EngDesign covers simulation-validated design [17]. AECBench and DrafterBench evaluate domain knowledge and drawing revision, respectively [54], [55]. AEC-Bench and AECV-Bench focus on multimodal agentic tasks and engineering-drawing understanding, respectively [16], [56].

To our knowledge, none of the benchmarks reviewed above jointly requires a structural model, validation evidence, backend-aware execution, supported design checks, safe non-execution, and a trace-grounded report. StructureClaw-Bench focuses on this combination. It uses typed assertions over intermediate and final artifacts so that end-to-end success and stage-specific failures can be measured in the same executable environment.

3 StructureClaw System↩︎

StructureClaw is an artifact-centered workbench for structural-engineering agents. Given a natural-language request, optional project files, and design constraints, it organizes execution as a sequence of inspectable engineering artifacts rather than treating the final response as the sole output. Depending on the task, these artifacts record the interpreted design intent, structural model, validation diagnostics, analysis results, supported code-check results, and final report. This evidence chain allows an engineer to determine whether a reported result is grounded in the workflow that produced it.

Figure 2 summarizes the architecture. Natural-language requirements, CAD/BIM files or images, and project-specific engineering materials and constraints enter the agent kernel. The kernel combines capability context, a ReAct-style execution loop, typed engineering tools, and the current artifact state. Computation is delegated to a local engineering sandbox that hosts model-construction, validation, analysis, code-checking, and report-generation components. Tool outputs are written back to the artifact state, where they remain available to subsequent actions and external review. The right-hand artifacts in Figure 2 are checked within the system’s supported scope and remain subject to professional verification; they do not constitute construction certification.

3.1 Skills, Tools, and Artifact State↩︎

StructureClaw separates roles that are often conflated in a generic tool agent. A skill supplies domain-specific planning context and declares its triggers, supported guidance scopes and structure types, runtime dependencies, and artifact contract. A capability is the function that a skill declares to the agent, such as frame modeling or modal analysis. A tool is a typed executable operation, such as parameter extraction, model construction, validation, analysis, code checking, or report generation. A provider binds a tool contract to a particular backend, while an artifact is the structured record consumed or produced by these operations. Thus, a skill constrains how the agent approaches a task, a tool performs a bounded state transition, a provider supplies the concrete execution backend, and artifacts carry both engineering state and auditable evidence across transitions.

This separation supports flexible rather than fully hard-coded workflows. The agent selects applicable skills and tools from the request and current state, while the declared contracts describe expected artifact inputs and outputs and typed tools enforce runtime preconditions. Shared artifact state also prevents downstream operations from having to reconstruct geometry, assumptions, or analysis context solely from the conversation history.

The central artifact is a structural model protocol that represents project metadata, structural system, geometry and topology, materials, sections, supports, loads, load combinations, analysis metadata, and explicit unit conventions. Analysis tools access a solver through a provider, an adapter that implements the tool contract for a particular backend. The current system supports open-source structural analysis using OpenSees through its OpenSeesPy interface [12], [13], with provider interfaces for PKPM and YJK when those backends are configured. Provider availability is checked before execution, so an unavailable backend remains an explicit workflow condition rather than being replaced by an unverified result.

3.2 Artifact-Grounded Validation and Recovery↩︎

StructureClaw instantiates the ReAct reasoning–action pattern [10] over artifact state. At each iteration, the agent observes the request, active capability context, and existing artifacts; it then invokes a tool, requests clarification, repairs an artifact, or terminates. The returned tool result updates the state and becomes the evidence for the next decision. This makes the execution trace a sequence of inspectable engineering transitions rather than an opaque chain of conversational responses.

Validation records model-linked diagnostics. The implemented checks cover schema compliance, identifier and cross-reference consistency, minimum model completeness, finite coordinates, support representation, and load and load-combination references. These checks establish that a model is well formed under the protocol; they do not, by themselves, certify full engineering correctness. Their results instead govern recovery: missing assumptions can require clarification, invalid references can trigger repair followed by revalidation, and invalid or inconsistent inputs can lead to safe non-execution. An unavailable provider or unsupported operation instead produces an explicit unsupported termination.

The final report is generated only from the artifacts available for the requested workflow. When analysis or code checking is required, the corresponding execution record must therefore be present before its results can be reported. This linkage from request to workflow path, model, validation, execution, and report is the basis for the traceability evaluated by StructureClaw-Bench; Appendix 8 specifies the evidence semantics of these workflow stages and artifacts.

4 StructureClaw-Bench↩︎

StructureClaw-Bench evaluates agents within the executable workflow rather than as standalone question-answering systems. Each scenario specifies the required evidence, and success requires every listed assertion over routing, artifacts, execution, interaction, and reporting to pass. Plausible prose therefore cannot compensate for a missing or inconsistent engineering trace.

4.1 Scenarios and Assertions↩︎

The benchmark contains 150 controlled scenarios, divided equally among three task families, with 75 Chinese and 75 English prompts (Figure 3). The standard-workflow family contains well-specified text-to-analysis requests for steel frames, concrete frames, beams, trusses, portal frames, continuous beams, columns, and generic structural-analysis tasks. The interactive-robustness family tests missing information, invalid references or values, unrealistic assumptions, unit ambiguity, multi-turn updates, conflicting constraints, recovery after failed actions, unsupported requests, and static-analysis edge cases. The multimodal structural-model reconstruction family begins with images or DXF files and requires the agent to infer geometry, topology, supports, loads, and member relationships before constructing an analysis model. The current released comparator evaluates model counts, coordinate spans, and approximate loads rather than complete topology, support equivalence, or member-by-member connectivity.

Typed assertions cover structure recognition, skill selection, model existence and reference-fixture consistency, solver/backend execution, clarification or non-execution, and reporting. Only fixture-listed assertions are required; a clarification case, for example, need not produce a model. Reference fixtures provide reproducible operational targets, not an exhaustive set of engineering-equivalent solutions. Appendix 9 details their semantics.

4.2 Primary Metric and Evaluation Protocol↩︎

Let \(\mathcal{S}\) denote an evaluated scenario set, \(\mathcal{A}_s\) the required assertions for scenario \(s\), and \(z_{s,a}\in\{0,1\}\) the outcome of assertion \(a\) in a single run. The primary metric is \[\mathrm{Success\;Rate}(\mathcal{S}) = \frac{100\%}{|\mathcal{S}|} \sum_{s\in\mathcal{S}} \prod_{a\in\mathcal{A}_s} z_{s,a}, \label{eq:success-rate}\tag{1}\] which we call Success Rate. The product equals one only when every required assertion for that scenario succeeds. Assertion-level outcomes are retained for diagnosis, but they do not provide partial credit toward Success Rate.

Each scenario is executed once, with automatic retries disabled. Runs are isolated in child processes under a common timeout policy, and a timeout is recorded as an execution failure. The default automatic mode allows the agent to select applicable skills. For standard-workflow scenarios, we additionally evaluate generic-only mode, which restricts the agent to the generic structural skill and the requested analysis skill while retaining the same broad execution environment. We define Auto–Generic Lift as the difference in Success Rate between these two modes, in percentage points. Because the two modes jointly change specialized routing, structural priors, artifact expectations, and validation guidance, this lift is a paired system-level comparison rather than a component-level ablation.

4.3 Diagnostic Views↩︎

For failure localization, assertion results are grouped into routing, modeling, execution/backend, interaction/safety, and report dimensions. Diagnostic rates pool recorded fixture-applicable outcomes and disclose unavailable observations; an unavailable required outcome still fails the scenario under Equation 1 . We also stratify Success Rate by family, structure, language, and modality. These small, designed subsets are descriptive rather than statistically powered benchmarks.

5 Experiments↩︎

The evaluation addresses three questions: whether the bundled automatic workflow achieves higher end-to-end success than the generic-only configuration on well-specified requests, whether agents respond safely to interactive edge cases, and whether they can reconstruct executable structural artifacts from images and DXF files. Each task family contains 50 scenarios. We evaluate 10 agent-model configurations in the text-only settings and six in the multimodal setting, yielding 36 configuration–setting runs and 1,800 single-attempt scenario executions. When visual perception and agent reasoning use different models, we report the configuration as agent model / vision model. Standard-workflow scenarios are evaluated in both automatic and generic-only modes; interactive robustness and multimodal structural-model reconstruction are evaluated only in automatic mode. We follow the single-run protocol defined in Section 4: each scenario is attempted once with automatic retries disabled and a 15-minute timeout.

The primary metric is Success Rate: the percentage of scenarios for which all required assertions pass in a single run. Assertion-level success rates, average tool calls, and average wall-clock time are diagnostic metrics. Aggregate assertion and subset rates pool recorded outcomes designated applicable by the scenario fixture and report their effective denominators; unavailable required outcomes count against primary Success Rate. The reported mean standard-workflow Success Rates average the 10 configuration-level rates, equivalently pooling them because every configuration evaluates the same 50 cases. Appendix 10 gives the exact definitions. Because each configuration–scenario pair is observed once, small subset differences are descriptive and should not be interpreted as statistically resolved rankings.

Figure 4 provides a descriptive profile of the evaluated settings without defining an overall score. Its axes combine family-level Success Rates with a limited set of diagnostic summaries and should be read separately rather than combined by area. Unevaluated multimodal axes are shown as gaps, so missing coverage is not represented as observed failure.

5.1 Standard Workflow↩︎

The first analysis compares the bundled automatic workflow with generic-only mode while retaining the same broad execution environment. Figure 5 and Table 1 show that the mean Success Rate increases from 56.8% to 88.6%, a gain of 31.8 percentage points. All 10 configurations achieve a higher Success Rate in automatic mode. Kimi-K2.6 satisfies every required assertion on all 50 cases, while DeepSeek-V4-Flash and GLM-5.2 each succeed on 48 of 50 cases. This comparison changes the structural skill path and its associated priors, artifact expectations, and validation guidance together; it therefore evaluates the structured workflow as a system and does not attribute the gain to any single component.

2.5pt

The diagnostics distinguish routing, artifact construction, and backend execution. Automatic-mode structural-type, skill-selection, and model-matching rates are 98.0%, 100.0%, and 90.8%. In generic-only mode, 488 model-related assertion records are recoverable and 12 are unavailable; among the recoverable records, 449/488 (92.0%) contain a model artifact, but only 344/488 (70.5%) satisfy the model-matching rule. Appendix 11 shows that this gap persists when unavailable records are counted as failures and examines its structure-family variation.

5.2 Interactive Robustness↩︎

We next ask whether an agent clarifies, abstains, or recovers when immediate analysis is inappropriate. Overall Success Rates range from 88.0% to 94.0% across the 10 configurations, with a mean of 91.0% (Figure 6). Table 2 further reports pooled assertion rates of 89.6% for clarification, 90.7% for avoiding an invalid model, and 91.0% for correctly withholding analysis. The narrow end-to-end range makes the assertion and category breakdowns more informative for diagnosis than the aggregate ranking.

2.5pt

Invalid numerical values form the weakest robustness category: 56 of 79 recorded configuration–scenario evaluations succeed (70.9%), compared with 71/80 multi-turn evaluations (88.8%) and 45/50 conflicting-requirement evaluations (90.0%). The recorded locale split is 96.0% for English and 86.7% for Chinese, but wording and category composition are not independently controlled. Appendix 11 reports denominators and missing-outcome sensitivity. The evidence identifies invalid-value checking as a targeted robustness priority, not an intrinsic language effect.

5.3 Multimodal Reconstruction↩︎

Finally, we evaluate whether visual or DXF inputs can be converted into artifacts that satisfy the same workflow assertions. Success Rates vary more widely across the six multimodal configurations than across the interactive configurations (Figure 7). GLM-5.2 / GLM-4.6V attains 94.0%, followed by Kimi-K2.6 at 88.0% and Claude Opus 4.8 at 84.0%; MiniMax-M3 attains 60.0% (Table 3). These numbers compare complete agent/vision configurations and do not isolate the contribution of the perception model.

2.5pt

Pooled DXF and image Success Rates are 80.0% (72/90) and 79.0% (166/210), but the subsets differ in size and composition. Across the 281 recoverable applicable outcomes shared by the three diagnostics—the remaining 19 are unavailable—structural-type recognition and skill selection reach 93.6% and 96.4%, while model matching reaches 84.7%. At the aggregate level, recognition and routing rates are therefore higher than the released model-matching rate. Because these are marginal diagnostics, they do not identify a sequential failure stage within individual traces. Appendix 9 shows that the model-matching rule tests counts, spans, and approximate loads rather than complete topology, supports, or engineering equivalence.

6 Discussion↩︎

The central result is a gap between merely producing an artifact and maintaining consistency across the evidence chain. In generic-only mode, model artifacts occur in 92.0% of recorded outcomes, but only 70.5% satisfy the model-matching rule. Automatic execution raises matching to 90.8% and mean Success Rate to 88.6%. The narrower existence–consistency gap is consistent with the view that structured workflow guidance may be particularly valuable at the handoffs among routing, model construction, validation, and execution, where an early assumption constrains all downstream artifacts. Because specialization, priors, artifact expectations, and validation guidance change together, however, the comparison supports the workflow as a system without identifying one causal mechanism.

The diagnostics also separate two bottlenecks that require different controls. Invalid-value cases attain 70.9% over recorded outcomes despite a 91.0% interactive mean, indicating a pre-execution weakness in constraint checking and safe state transition. Multimodal structure recognition (93.6%) and skill selection (96.4%) exceed model matching (84.7%) on the same 281 observed outcomes, indicating an aggregate gap between broad routing diagnostics and artifact reconstruction. The former motivates deterministic value checks before model mutation or solver invocation; the latter motivates inspectable intermediate representations of extracted dimensions, loads, and structural relationships before analysis. A single end-to-end score would not distinguish these intervention points, while the marginal diagnostics and current comparator cannot establish within-trace ordering or attribute the reconstruction gap to a specific stage.

The aggregate automatic-mode gain is also not uniform. Continuous-beam cases decrease from 96.0% in generic-only mode to 76.0% in automatic mode, with the regression recurring across half of the evaluated configurations. This counterexample shows that the bundled automatic workflow can introduce a correlated, structure-specific regression even while improving the benchmark average. Structure-family regressions should therefore be treated as release criteria alongside aggregate Success Rate, and paired case-level traces should be used to determine whether the common failure enters during modeling assumptions, validation, execution, or reporting.

More broadly, progress should address a clearly identified failure at a specific workflow transition without introducing regressions elsewhere, rather than merely raise mean Success Rate. Stage-local outcomes with coverage counts can therefore serve as auditable release criteria rather than merely as inputs to a model leaderboard.

These results remain diagnostic rather than definitive model rankings or engineering certification. Each configuration–scenario pair is observed once, some subsets are small, and the mode comparison changes several workflow elements. Reference fixtures may omit valid alternatives, the comparator may accept untested defects, and solver completion does not establish code compliance. The next evaluation priorities are repeated trials for run variability, matched component ablations for attribution, controlled agent/vision pairings, and engineer-reviewed equivalence criteria. StructureClaw remains an assistance and evaluation environment, not a substitute for professional review.

7 Conclusion↩︎

StructureClaw integrates skill-guided planning, typed tool execution, shared structural-model artifacts, validation records, solver traces, code-check results, and report generation into a traceable workflow. StructureClaw-Bench evaluates this evidence chain through 150 controlled scenarios; on the 50 standard cases, the bundled automatic workflow achieves 88.6% mean Success Rate versus 56.8% for generic-only execution. Stage-level diagnostics further expose safe handling of invalid inputs and multimodal structural-model reconstruction as distinct bottlenecks. A structure-specific regression on continuous-beam cases also shows that aggregate gains should be accompanied by family-level release checks. The contribution is an executable framework for observing failures and testing improvements, not a claim of autonomous engineering competence. Future work should isolate component effects, quantify uncertainty, add engineer-reviewed equivalence criteria, and evaluate larger projects and additional backends.

8 Implementation and Trace Semantics↩︎

StructureClaw is implemented around an explicit evidence chain rather than a single conversational output. The chain records what the agent was asked to do, which governed capability it selected, which engineering artifacts were constructed, whether those artifacts passed the available checks, which backend was invoked, and what evidence was available when the report was written. This appendix specifies how skill manifests, guidance scopes, runtime contracts, artifact records, and terminal workflow states organize that chain.

8.1 Evidence Chain↩︎

A request and its optional project files enter a sequence of workflow stages: requirement interpretation and routing, model construction, validation, analysis or code checking, and reporting. A stage consumes the current workflow state and records a typed state update, an artifact, or an explicit condition that prevents safe continuation. Tool-call records associate each executable operation with its inputs, returned object, provider binding, or error. Consequently, an evaluator need not infer model creation or solver execution from fluent prose alone.

Table 4 summarizes the evidence available at each stage. The table describes an audit interface, not a claim that every task must traverse every row. A request that lacks material information can terminate with clarification before model construction; an unsupported provider can terminate before execution; and a task that does not require analysis can finish without a solver artifact.

4pt

The shared structural model is the principal handoff between stages. Recognition without a model is therefore distinguishable from model construction, and model construction is distinguishable from successful execution. The same separation prevents a report artifact from serving as evidence that an analysis occurred: numerical claims require an analysis result or supported check record in the trace.

8.2 Skill Schema and Artifact Graph↩︎

A skill schema is reusable: it declares how a capability is discovered, which scope-specific engineering guidance it contributes, which constraints govern its selection, and how its runtime contract consumes or provides artifacts. A manifest records identity, discovery metadata, declared capabilities, requirements, conflicts, and priority. It also specifies runtime compatibility, the skill-API version, its runtime role, and the artifact inputs, outputs, or provider slots associated with that role. Separate guidance documents contribute content for intent, draft, analysis, and design scopes.

Figure 8 illustrates this schema for a regular three-dimensional steel-frame request. The frame manifest identifies the capability as frame in the structure-type domain, assigns it the entry role, and declares a design-basis output. Unlike the workflow stages in Table 4, its intent, draft, analysis, and design labels are parallel guidance scopes that may be activated nonlinearly; they do not prescribe an execution order. Execution occurs through typed tools and, when analysis or checking is requested, through separately selected provider bindings. Thus, a skill can guide frame-model construction without certifying that a solver or design-code checker has run.

8.3 Artifact Envelope and Runtime Mapping↩︎

The logical artifact contract separates engineering content from its audit envelope. It defines draft state, design basis, normalized model, validation record, analysis model, raw and postprocessed analysis results, code-check result, drawing artifact, and report artifact as distinct kinds. Common envelope fields include artifact identity, kind, scope, status, revision, producer and provider identifiers, run identifier, timestamps, upstream references, dependency fingerprint, schema version, provenance and warnings, and a kind-specific payload. For a normalized structural model, the payload carries units and project context; structural system and site conditions; nodes, elements, materials, sections, and boundary restraints; loads and combinations; analysis controls; metadata; and extensions. The protocol is broader than the current runtime mapping: not every conceptual envelope is materialized in every run.

The lower part of Figure 8 should therefore be read as an artifact-and-patch dependency graph, not as a list of conversational messages or a complete taxonomy. A design basis records structural criteria such as material grade, seismic intensity, site category, and requested design scope. A model patch is a separate state-transition record for a proposed or accepted model change, not an artifact kind or a completed model; it appears in the same panel only to show its dependency on the normalized model. An accepted, conflict-free patch can produce an updated normalized model only when its base revision matches. Validation records preserve deterministic diagnostics linked to the model. Analysis-model and raw-result artifacts add mass, boundary conditions, load cases, eigenvalues, eigenvectors, forces, and displacements. Postprocessed results summarize derived quantities such as periods, modal participation, drift, and envelopes, and code-check results record utilization, stability, and status against the selected design basis and model. Report artifacts link their claims to the workflow evidence actually present.

The schema permits a dependency fingerprint derived from upstream artifact references and provider bindings. This supports dependency-aware audit: a code-check result can identify the design basis, model, and postprocessed result used when it was produced, while a later patch creates an updated path rather than silently replacing the evidence behind the existing check. The dashed feedback in Figure 8 denotes this protocol affordance, not an automatic iteration executed by the benchmark. In the current implementation, the envelope remains partly declarative: the execution graph records draft, model, validation, analysis, code-check, and report payloads in dedicated state channels and streams compact tool summaries, but it does not materialize every conceptual envelope in every run.

8.4 Provider Binding and Execution↩︎

Execution remains tool-controlled. The agent can clarify, construct, validate, repair, execute, or stop, but declaring analysis or design guidance in a skill does not authorize computation. Nor does the runtime silently redefine a missing provider or an invalid model as a successful computation.

Provider identity is part of the execution evidence. Analysis-provider contracts consume an analysis model and provide raw results; code-check-provider contracts consume the design basis, normalized model, and postprocessed results required by the selected check. Availability is checked before a provider-dependent action, and the selected analysis engine or design-code capability can be compared with scenario expectations. This distinction matters because a solver-shaped response is not equivalent to a recorded solver invocation. It also bounds the evidence: a trace can show which backend returned an artifact, but backend correctness still depends on the adapter, solver, units, and input model being valid.

The automatic and generic-only settings alter the capability guidance available to the agent. The former exposes the structured selection path and its specialized artifact expectations. The latter retains the generic structural capability and the requested analysis capability but removes the automatic specialization path. This comparison therefore evaluates two workflow configurations, not an isolated prompt token or a single validation rule; the execution matrix is specified in Appendix 10.

8.5 Validation, Repair, and Terminal States↩︎

The prescribed workflow places validation between artifact construction and provider execution. When a deterministic check exposes a correctable defect, the agent may repair the current artifact and validate it again within the same run. This within-run recovery is different from rerunning a completed benchmark case. The evaluation harness disables automatic case reruns, while the workflow remains able to validate and repair an artifact before reaching a terminal state.

The trace distinguishes four terminal decisions. Completion means that the artifacts required by the request are available. Clarification means that material information remains missing or ambiguous. Safe non-execution means that analysis should not proceed because an input is invalid or inconsistent. Unsupported termination means that the requested provider or operation is unavailable in the evaluated environment. These states are judged from positive and negative evidence together. In particular, the mere absence of an analysis artifact is not a successful safety response: a negative case must also satisfy its scenario-specific clarification or semantic requirement and show that neither an analysis result nor an analysis-tool invocation was recorded. Appendix 9 gives the corresponding assertion rules.

8.6 Scope of Traceability↩︎

Traceability supports several diagnostic questions. It can establish whether a recorded action occurred, whether the expected artifact type was present, and whether stage-local checks accepted that artifact. Combining these observations can localize a broken handoff, such as correct routing followed by a missing model or a model followed by an absent analysis result. These observations are stronger than final-text inspection because they refer to the executed state.

They are not a proof of engineering correctness. The trace does not expose unrecorded reasoning, certify that a validator is complete, or guarantee that a reference fixture covers every engineering-equivalent solution. It also does not independently verify the numerical implementation of an external solver. Accordingly, StructureClaw-Bench uses the trace as evidence within a controlled diagnostic testbed. Construction suitability, exhaustive code compliance, and equivalence to professional review remain outside the evaluated claim.

9 Benchmark Construction and Assertions↩︎

StructureClaw-Bench is a controlled fixture suite for evaluating workflow outcomes. Its unit of evaluation is a scenario, not an isolated question or a field project. Each scenario specifies an input, evaluation metadata, a reference fixture when artifact comparison is required, and the evidence that one run must produce or withhold.

9.1 Scenario Specification and Coverage↩︎

Conceptually, scenario \(s\) is represented as \[s = \left(x_s, m_s, r_s, \mathcal{A}^{\mathrm{req}}_s\right), \label{eq:app-scenario-specification}\tag{2}\] where \(x_s\) contains the prompt and any project file, \(m_s\) records family, language, structural category, and input modality, \(r_s\) is an optional reference fixture, and \(\mathcal{A}^{\mathrm{req}}_s\) contains the typed assertions listed by the fixture. Equation 2 is a conceptual schema rather than an additional benchmark score. Every listed assertion contributes to the binary scenario outcome. The diagnostic views are marginal aggregations of these same assertion results and scenario metadata; they provide neither partial credit nor an alternative success criterion.

The released suite contains 150 scenarios, allocated as shown in Table 5. Each family is balanced between 25 English and 25 Chinese prompts, yielding 75 prompts in each language overall. The multimodal family contains 35 image cases and 15 DXF cases. This stratification ensures coverage of the three intended workflow regimes, but it does not make the fixtures a random sample of structural-engineering practice. Family, language, structure, category, and modality subsets are therefore interpreted as controlled diagnostics.

4pt

Standard-workflow scenarios provide sufficiently specified text requests and require the designated modeling, execution, and reporting path. Interactive scenarios introduce missing, conflicting, invalid, unrealistic, ambiguous, multi-turn, recovery, unsupported, and static-analysis conditions and test whether the agent clarifies, recovers, or withholds execution as specified. Multimodal scenarios add a perception-and-reconstruction stage before the same artifact checks. Keeping the families separate prevents performance on well-specified text requests from concealing unsafe execution or inaccurate reconstruction.

9.2 Reference Fixtures↩︎

The term reference fixture denotes the released artifact against which a scenario is evaluated. It is a reproducible engineering target, not an assertion that one discretization is the unique correct design. A fixture records the expected scale and loading quantities needed by the implemented comparator; scenario metadata can also specify accepted structural-type aliases, alternative capability identifiers, or comparison tolerances.

This terminology is important for construct validity. A model can be engineering-equivalent to the intended system while using a different discretization, grouping convention, or idealization. Conversely, a coarse comparator can accept an artifact that agrees in aggregate quantities while differing in a consequential local property. The benchmark reduces ambiguity by publishing fixtures and operational rules, but fixture agreement should be read as satisfaction of those rules rather than proof of complete structural equivalence.

9.3 Assertion Roles and Operational Rules↩︎

For an evaluated configuration \(c\), scenario \(s\), and assertion type \(a\), the archived outcome is assigned one of four states: \[o_{c,s,a}\in \{\mathrm{pass},\mathrm{fail},\mathrm{inapplicable},\mathrm{unavailable}\}. \label{eq:app-assertion-states}\tag{3}\] The vocabulary in Equation 3 separates evaluation meaning from data availability. A pass or fail is a recorded result for an assertion evaluated on that run. Inapplicable means that the fixture does not request that assertion type; it is not a failure. Unavailable means that an expected observation could not be recovered because the run, trace, or result parsing was incomplete. An unavailable required outcome makes the scenario unsuccessful. In assertion-level diagnostics it is excluded from the pass/fail denominator and disclosed through the observed count, as formalized in Appendix 10. This policy avoids conflating semantic inapplicability with missing evidence.

Table 6 records the principal operational rules. Most rules are deterministic over the archived agent state. The natural-language condition is the exception: a separate judge evaluates the scenario-specific semantic description with zero sampling temperature, a 500-token response limit, and a 30-second timeout. Its provider-side snapshot and evaluator-prompt revision fall within the exact-rerun boundary stated in Appendix 10.

4pt

9.4 Model-Matching Semantics and Limits↩︎

The model-matching rule in Table 6 is a deterministic, tolerance-based comparison. It first requires at least as many nodes and elements as the reference fixture. It then compares the ranges of the model coordinates in the \(x\), \(y\), and \(z\) directions. The default absolute tolerance for each range is 0.5 m. Finally, it compares approximate total applied load, or selected directional totals when the scenario requests them, using a default relative tolerance of 20%. The calculation aggregates recognized nodal, distributed, and story-level load fields; recognized force units are normalized between newtons and kilonewtons. A scenario may override either tolerance.

These checks deliberately favor reproducibility over an underspecified notion of visual similarity. They do not perform graph isomorphism, member-by-member connectivity matching, support-equivalence proof, or exhaustive comparison of section and material properties. Node and element counts are lower bounds rather than exact equality. The report-presence rule likewise establishes only that a substantial report artifact exists, while the semantic judge checks its stated condition; neither rule alone proves numerical faithfulness of every report claim. These limitations explain why the paper reports model matching as an assertion-level diagnostic and reserves broader claims of engineering equivalence for future evaluators with topology-, support-, property-, and load-level annotations.

Negative scenarios require similar care. Absence of a model or analysis may reflect safe abstention, but it may also reflect an execution failure. The benchmark therefore pairs negative evidence with a positive scenario requirement, such as a clarification trace or a semantic explanation. This conjunction prevents non-action from receiving credit merely because no computation was recorded.

10 Evaluation Protocol and Metric Definitions↩︎

This appendix fixes the evaluation unit, outcome states, aggregation rules, and reproducibility boundary. StructureClaw-Bench is a controlled diagnostic testbed: it measures whether an agent completes the assertion set attached to a released scenario, not whether the resulting artifact is suitable for construction or professional approval.

10.1 Execution Matrix and Attempt Semantics↩︎

The evaluation matrix is shown in Table 7. The 10 text-only configurations are evaluated on both standard-workflow modes and on interactive robustness. Six agent/vision configurations are evaluated on multimodal reconstruction. When the perception and agent models differ, a slash-separated name reports agent model / vision model, as in Table 3. Each configuration–setting block contains the same 50 scenarios for that setting, yielding 36 blocks and 1,800 configuration–scenario executions.

3.5pt

The evaluation harness launches each configuration–scenario pair once and does not relaunch a failed or timed-out case. This harness-level no-rerun policy is distinct from repair within a scenario execution. During that single benchmark execution trace, the agent may validate an artifact, revise it, or recover from a tool error; those actions are part of the evaluated workflow rather than additional benchmark attempts. A child process enforces a 15-minute per-case limit, and a timeout is an execution failure. Tool-call counts and wall-clock durations are read from the same traces. There is therefore no averaging over repeated samples for a configuration–scenario pair.

10.2 Outcome States and Primary Metric↩︎

An assertion outcome has one of four meanings: pass, fail, inapplicable, or unavailable. Inapplicable means that the scenario fixture does not call for that assertion. Unavailable means that an expected outcome cannot be recovered, for example because execution timed out, a result or trace is missing, or parsing failed. Applicability is determined by the fixture and is not inferred from whether a run happens to contain evidence. This distinction prevents missing evidence from being relabeled as semantic inapplicability.

Let \(\mathcal{A}^{\mathrm{req}}_i\) be the assertions required by scenario \(i\), and let \(z_{c,i,a}=1\) only when assertion \(a\) passes for configuration \(c\). For required assertions, both failure and unavailability give \(z_{c,i,a}=0\). The scenario outcome is \[s_{c,i}=\prod_{a\in\mathcal{A}^{\mathrm{req}}_i}z_{c,i,a}, \label{eq:app-scenario-success}\tag{4}\] and the primary metric for task family \(F\) is \[\mathrm{Success\;Rate}_{c,F} =\frac{100\%}{N_F}\sum_{i\in F}s_{c,i}, \qquad N_F=50. \label{eq:app-success-rate}\tag{5}\] Thus, every family-level denominator remains 50 for each configuration, and an incomplete required outcome cannot disappear from the primary metric. Assertion success provides no partial credit toward Eq. 5 .

10.3 Diagnostic Denominators and Missingness↩︎

Diagnostic rates answer a different question: among recorded outcomes for which an assertion is fixture-applicable, how often does it pass? For an assertion \(a\) and analysis group \(G\), let \(N_{a,G}\) be the expected applicable count, let \(v_{c,i,a}=1\) when the recorded outcome is pass or fail, and let \(v_{c,i,a}=0\) when it is unavailable. The observed count is \(n_{a,G}=\sum_{(c,i)\in G}v_{c,i,a}\), the observed pass count is \(p_{a,G}=\sum_{(c,i)\in G}v_{c,i,a}z_{c,i,a}\), and the reported diagnostic rate is \[D_{a,G}=100\%\, \frac{\sum_{(c,i)\in G}v_{c,i,a}z_{c,i,a}}{n_{a,G}} =100\%\,\frac{p_{a,G}}{n_{a,G}}. \label{eq:app-diagnostic-rate}\tag{6}\] Semantically inapplicable evaluations are outside \(G\). Unavailable evaluations are excluded from the diagnostic denominator but remain failures when the assertion is required by Eq. 4 . Diagnostic results should consequently be read as \(p/n\) together with coverage \(n/N\), not as a percentage alone. Appendix 11 reports missing-as-failure sensitivity checks for the principal gaps. A gray cell with a dash in an assertion figure means that no recorded applicable outcomes are available; it is not a measured zero.

Configuration-level diagnostics use Eq. 6 within one configuration. Pooled diagnostics sum pass and observed counts before division. Means of configuration-level Success Rates weight configurations equally; because every configuration has 50 cases in a setting, this is numerically equivalent to pooling scenario outcomes for that setting.

10.4 Mode Comparison↩︎

Automatic mode allows the agent to select the applicable structural capabilities and execute the artifact-centered workflow. Generic-only mode is evaluated only on standard scenarios. It retains the broad environment but restricts the structural path to the generic structural skill and the requested analysis capability. For each of the 10 paired configurations, the reported lift is \[\Delta_c=\mathrm{Success\;Rate}^{\mathrm{automatic}}_{c,\mathrm{standard}} -\mathrm{Success\;Rate}^{\mathrm{generic\text{-}only}}_{c,\mathrm{standard}}. \label{eq:app-auto-generic-lift}\tag{7}\] The same scenario fixtures are used on both sides. Nevertheless, the comparison jointly changes specialized routing, structural priors, artifact expectations, and validation guidance. It is therefore a paired system comparison, not a causal estimate for a classifier, prompt, validator, repair policy, or reporting component.

10.5 Descriptive Evaluation Profile↩︎

Figure 4 is a compact profile of evaluated settings, not a capability score or ranking. Table 8 gives the plotted quantities. The Model/Exec and MM recon axes are unweighted visual summaries; their constituent diagnostics need not have the same applicability coverage and should be inspected separately for substantive conclusions. The Interaction and MM task axes use end-to-end Success Rate directly, avoiding double-counting an overall rate together with its subsets.

3pt

Raw evaluated values remain on a 0–100 scale. The displayed radius expands the 50–100 interval for legibility and maps values at or below 50 to the inner ring. An unevaluated multimodal axis is left as a gap rather than assigned a zero. Radar area and average radius are not computed, because either would mix heterogeneous outcomes and depend on axis order and the display transformation.

10.6 Statistical and Reproducibility Boundaries↩︎

Structure-family, interaction-category, locale, and input-format rates are descriptive subsets of a fixed, deliberately constructed suite. Some subsets are small and compositionally different. A scenario-level interval or bootstrap would describe sensitivity to the observed scenario composition only under additional sampling assumptions; it cannot recover within-scenario run-to-run variability from one execution. Repeated runs are required to estimate stochastic variability, and matched fixtures are required to isolate language or input-format effects.

The public release provides the system code, benchmark harness, scenarios, fixtures, evaluation logic, and analysis scripts required for independent reruns. The supplementary records preserve the aggregate outcomes and analysis procedure used to regenerate the reported tables and figures. Exact trace reproduction additionally depends on the archived code revision, provider access, execution environment, and provider-side agent and judge snapshots; rerun traces therefore need not be byte-identical.

11 Diagnostic Analyses and Validity↩︎

This appendix examines patterns that are obscured by a single end-to-end number. Unless stated otherwise, rates pool the relevant configuration–scenario outcomes, differences are percentage points, and assertion diagnostics use the observed denominator defined in Eq. 6 . The analyses localize evidence for follow-up; they do not certify an artifact or establish a universal ordering of models. Table 9 states the boundaries that govern all subsequent interpretations.

2.5pt

11.1 Paired Standard-Workflow Difference↩︎

Figure 9 places the four evaluated settings on a common grid. Gray cells with dashes are unevaluated model–setting combinations, not failures. Across the 10 paired standard configurations, mean Success Rate is 88.6% in automatic mode and 56.8% in generic-only mode, a mean lift of 31.8 points. Every configuration improves, and the configuration-level lifts range from 24 to 46 points, showing a consistent difference across the evaluated configurations. Its scope remains system-level, however: Eq. 7 changes a bundle of skill specialization, structural priors, artifact requirements, and validation guidance.

The paired gain is not uniform across structure families. Continuous-beam cases regress from 48/50 (96.0%) in generic-only mode to 38/50 (76.0%) in automatic mode. Five configurations decrease by 40 points on this five-case subset and five are unchanged. This concentration motivates investigation of a shared path-specific regression, but the aggregate data cannot identify whether the source is modeling assumptions, validation, execution, or reporting. This counterexample is important: a higher overall average does not imply that automatic execution dominates on every scenario family.

11.2 Artifact Existence, Reference Consistency, and Missingness↩︎

Figure 10 separates assertion-level outcomes. In standard automatic mode, has-model passes 480/500 times (96.0%) and model matching passes 454/500 times (90.8%), an existence–reference-consistency gap of 5.2 points. Here, has-model records that a model artifact is present; it does not establish that the artifact is well formed or equivalent to the reference fixture.

The corresponding generic-only diagnostics have 488 observed outcomes because 12 executions do not provide a recoverable assertion result. On the observed denominator, has-model is 449/488 (92.0%) and model matching is 344/488 (70.5%), a 21.5-point gap. If all 12 unavailable outcomes are conservatively counted as failures, the rates become 449/500 (89.8%) and 344/500 (68.8%), leaving a 21.0-point gap. The levels change slightly, but the inference that artifact existence is substantially more common than reference-fixture consistency does not depend on excluding those unavailable outcomes. The primary Success Rate already counts incomplete required outcomes as failures.

Multimodal diagnostics permit a direct aggregate comparison because structure-type recognition, skill selection, and model matching share the same observed denominator. They pass 263/281 (93.6%), 271/281 (96.4%), and 238/281 (84.7%) times, respectively. Model matching therefore trails skill selection by 11.7 points and structure-type recognition by 8.9 points without a denominator change. At the aggregate level, this pattern is consistent with weaker artifact reconstruction than broad recognition and routing; because these are marginal rates, it does not establish that the same traces passed the earlier assertions or locate a sequential failure stage. The current matcher also cannot distinguish perception errors, count/span/load reconstruction errors, and limitations of its own rules. Topology, supports, and member semantics remain outside this diagnostic; future trace annotation should expose those stages separately.

End-to-end Success Rate remains necessary because high diagnostic rates need not co-occur in one trace. A run may select the scenario-specified analysis path and produce a model artifact yet fail the reference-fixture comparison or execution assertion. In a negative scenario, correct behavior may instead require clarification and the absence of a computable model. The required-assertion conjunction preserves these dependencies; the diagnostics identify where to inspect them.

11.3 Structure and Interaction Heterogeneity↩︎

Figure 11 shows that aggregate performance hides targeted regressions and small stress-test subsets. The continuous-beam reversal is qualitatively different from the large overall automatic-mode gain. It should therefore be treated as a release-specific regression target, not averaged away. Multimodal concrete- and steel-frame subsets are also weaker than several other displayed families, but the family sizes are unequal and do not support a stable ranking.

Invalid numerical values are the clearest interactive weakness. Interactive Success Rate is 455/500 (91.0%) overall, whereas invalid-value cases pass 56/79 observed configuration–scenario evaluations (70.9%). The intended invalid-value coverage is 80; treating the one unavailable outcome as a failure gives 56/80 (70.0%). The conclusion is therefore insensitive to this missing outcome. By contrast, perfect rates for unit-ambiguity and static-analysis subsets are based on only 40 and 10 evaluations. These counts favor targeted expansion of numerical-constraint tests, not a broad safety claim.

11.4 Locale and Input-Format Interactions↩︎

The locale differences in Figure 12 change direction across settings. English versus Chinese Success Rate is 90.0% versus 87.2% for standard automatic, 56.3% versus 60.2% for generic-only, 96.0% versus 86.7% for interactive, and 80.0% versus 78.7% for multimodal execution. This reversal argues against interpreting the observed differences as an intrinsic language effect. Twelve generic-only and two interactive executions lack locale metadata; wording, category composition, and mode also remain entangled.

Pooled DXF (72/90, 80.0%) and image (166/210, 79.0%) rates conceal opposing configuration-level differences: GPT-5.5 is 21.0 points higher on DXF than on images, whereas GLM-5.2 / GLM-4.6V is 10.4 points lower on DXF than on images. Format, scenario content, and agent/vision pairing are not crossed independently, so neither difference is causal. A matched study would encode the same structures in both formats while holding the agent and perception models fixed.

11.5 Recorded Runtime Co-occurrence↩︎

Figure 13 plots Success Rate against recorded duration. Mean duration is 89.8 s in standard automatic mode and 190.1 s in generic-only mode; the corresponding mean call counts are 6.70 and 7.55. Every generic-only configuration is slower. The median generic-only-to-automatic ratio is 2.11 for duration (IQR 1.48–2.38) and 1.07 for call count.

This is co-occurrence, not intrinsic efficiency. Provider latency, load, prompt length, tool behavior, and failure timing remain confounded; a low call count can also reflect early failure. Controlled latency, token, and matched-success analyses are required for an efficiency claim. The validity boundaries in Table 9 apply to these runtime results as well as the preceding diagnostic analyses.

References↩︎