1 Introduction↩︎

Software requirements are elicited primarily through stakeholder meetings [1], yet translating what has been agreed into formal, testable specifications remains a manual, time-consuming, and error-prone task [2], [3]. Behavior-Driven Development (BDD) addresses part of this by expressing acceptance criteria with a shared language. One such language is Gherkin, where senarios are described in Given/When/Then format making them both human-readable and executable by tools such as Cucumber [4]–[7]. In practice, however, Gherkin authoring remains manual: analysts write criteria after meetings from notes and memory, introducing three problems [3], [8]: (1) incompleteness: edge cases discussed verbally are not captured; (2) delay: criteria written days later lose tacit meeting context; (3) inconsistency: criteria diverge in style across sessions. A systematic mapping study found that automated tooling for Gherkin scenario authoring remains scarce, with teams consistently struggling to maintain well-formed scenarios as requirements evolve [9].

Large Language Models (LLMs) have made automated Gherkin generation from requirements feasible [10], [11]. However, an open question remains: does the representation strategy (whether Gherkin use is organized around epics or individual requirements) affect structural quality, requirement coverage, and practitioner perception? While Gherkin is designed for epic- or story-level granularity [4], [6], existing automated tools generate scenarios at per-requirement granularity [11]; comparing both strategies reveals the coverage–readability trade-off inherent to each approach.

We compare an epic-organized two-pass LLM pipeline (Timeless) against a requirement-aligned zero-shot baseline across four PURE (Public Requirements) dataset documents (107 requirements in total) using structural metrics, dual-metric requirement coverage, and blind expert evaluation.

This paper makes the following contributions:

1. An empirical comparison of epic-organized vs.requirement-aligned Gherkin generation (four PURE documents, 107 requirements) across structural quality, requirement coverage, and practitioner-perceived quality (four expert raters).

2. A two-pass LLM prompting pipeline within Timeless combining semantic epic synthesis with automated coverage-gap filling.

3. Evidence that lexical-only coverage metrics can underestimate epic-organized Gherkin, motivating the use of complementary lexical and semantic coverage measures in future evaluations of LLM-generated BDD artifacts.

Section 2 reviews related work; Section 3 presents the generation approach; Section 4 reports results; Section 5 discusses implications and limitations; Section 6 addresses threats to validity; Section 7 concludes.

2 Background and Related Work↩︎

2.1 Behavior-Driven Development, Gherkin, and the Authoring Challenge↩︎

Behavior-Driven Development (BDD) [4], [5] expresses software behavior through concrete scenarios in a shared language, bridging technical and non-technical stakeholders. Gherkin operationalizes this through a structured, human-readable syntax: a Feature block describes a product capability; each Scenario specifies behavior via Given (precondition), When (action), and Then (expected outcome) steps [4]–[6]. In BDD practice, related requirements are grouped into epics, broad, cross-cutting capabilities that a user wants the system to provide. Each epic maps to one Gherkin Feature block, and the individual requirements within that epic inform the concrete scenarios: Epic \(\to\) Feature block \(\to\) Requirements \(\to\) Scenarios [4], [6]. This hierarchy is central to the paper’s comparison: the epic-organized strategy follows this design intent; the requirement-aligned baseline generates one scenario per requirement, bypassing the epic grouping level.

The primary barrier to BDD adoption is authoring effort: producing correct, complete Gherkin requires both domain knowledge and structural discipline, a combination difficult to sustain at scale [8].

2.2 Large Language Models for Acceptance Criteria and Gherkin Generation↩︎

Recent LLMs have made automated Gherkin generation from raw requirements feasible [12], [13]: GPT-4 matches human-analyst performance on requirements classification, and ChatGPT evaluates user story quality comparably to human reviewers with a consensus-based prompting strategy. Closest to our work, Karpurapu et al. [11] conducted the most systematic evaluation to date, comparing GPT-4, GPT-3.5, and open-source models on BDD acceptance test generation from user stories: GPT-4 achieved the highest syntactic validity, but all models struggled with negative-path scenario coverage. Rathnayake et al. [14] confirmed high syntactic correctness alongside variable semantic completeness across three LLM families. Hassani et al. [15] showed that Gherkin quality is sensitive to input representation format, with iterative human review needed to recover quality lost to poorly structured inputs. Ferreira et al. [16] achieved a 95% acceptance rate in an industrial two-step pipeline; however, their approach assumes already-structured user stories rather than raw SRS prose.

To the best of our knowledge, prior work has not jointly examined epic-organized Gherkin generation, automated coverage-gap filling, and comparison with a requirement-aligned LLM baseline using both lexical and semantic coverage measures. This study addresses that gap using SRS-derived functional requirements from the PURE dataset.

3 Gherkin Generation Approach↩︎

3.1 Prompt Engineering for Gherkin Generation↩︎

The Gherkin generation step receives the current requirements list and the generated epics. The system prompt instructs gpt-4o-mini to produce a JSON array in which each element specifies a feature name, an In order to / As a / I want description, and a scenarios array of keyword-text step pairs, enforcing six structural constraints:

• One Feature block per epic (3–6 features total).

• Each Feature includes an In order to / As a / I want description.

• Each Feature contains 2–4 Scenarios: at least one happy path and one negative or edge-case scenario.

• Steps use only Given, When, Then, And, But.

• Steps are business-readable: no code, no technical jargon.

• Output is strictly valid JSON; no markdown fences or free prose.

These constraints address documented LLM failure modes [11]: one Feature per epic follows Gherkin’s design intent (Feature blocks describe capabilities at epic granularity [4], [6]); mandating a negative or edge-case scenario counters the consistent under-generation of failure paths observed across GPT-class models [11]; and business-readable steps follow the BDD principle that criteria should communicate intent, not implementation [5], [6].

The JSON schema helps enforce the required output structure, while a deterministic validation step checks whether each scenario follows the expected Given–When–Then order. Together, these constraints reduce the chance of malformed Gherkin and explain the high structural validity observed in this evaluation. Listing [lst:gherkin] shows a representative Gherkin scenario rendered from the JSON-constrained output.

Feature: Book Reservation
  In order to reserve books in advance,
  As a registered library member,
  I want to place holds on unavailable books.

  Scenario: Successful reservation of an available book
    Given I am logged in as a library member
    When I search for "The Pragmatic Programmer"
    And the book has available copies
    Then I can reserve a copy
    And I receive a confirmation notification

3.2 The Timeless Platform↩︎

Timeless [17] is a requirements engineering platform that transcribes meeting speech and extracts requirements in real time. Figure 1 shows the microservice architecture: a Transcription Service captures and transcribes audio via OpenAI Whisper [18]; a Requirements Service extracts and updates a requirements list from each transcription segment using an LLM; and a Manager Service coordinates both services, triggers downstream generation tasks (epics, mind map, Gherkin acceptance criteria), and streams state updates to the frontend via Server-Sent Events (SSE). The Manager triggers generation when it detects new requirements through LLM-based relevance classification. This paper evaluates the Gherkin generation component.

3.3 Pipeline Overview↩︎

Gherkin acceptance criteria are produced at the final stage of a four-step pipeline (Figure 2): (1) relevance classification (whether a transcription segment contains requirement-relevant content); (2) requirements update (adding or revising items in the requirements list); (3) epic and mind-map generation (grouping requirements into 3–6 epics); and (4) Gherkin generation (producing Feature blocks with Given-When-Then scenarios for each epic); Step 4 (Gherkin generation) runs after epics stabilize and is the focus of this evaluation.

4 Evaluation↩︎

4.1 Research Questions↩︎

We evaluate the approach through four Research Questions. The zero-shot LLM baseline generates one Gherkin scenario per requirement in a single prompt, without epic grouping, JSON schema constraints, or gap-fill passes, representing the current state of automated LLM practice in the literature [11], [14].

• RQ1: To what extent does an epic-organized, JSON-constrained generation pipeline produce syntactically valid and step-complete Gherkin scenarios compared with a zero-shot LLM baseline?

• RQ2: How do the requirement coverage characteristics (Requirement Coverage Rate [RCR] and Spurious Scenario Rate [SSR]) of epic-organized Gherkin compare to those of a zero-shot LLM baseline?

• RQ3: How does the epic-organized Gherkin representation strategy (grouping requirements into semantically coherent epics, each represented as a Feature block, rather than one scenario per requirement) affect scenario quality as perceived by practitioners?

• RQ4: What are the practical trade-offs between epic-organized and requirement-aligned Gherkin generation, and what are the implications for BDD adoption in agile teams?

The RQs form a layered evaluation: structural correctness (RQ1) is a prerequisite; coverage without sacrifice is the key comparison (RQ2, using dual lexical-semantic methods absent from prior BDD studies [11], [14], [15]); RQ3 adds practitioner-perceived quality; RQ4 synthesizes actionable tradeoffs.

4.2 Evaluation Design↩︎

4.2.1 Dataset.↩︎

We used the PURE dataset [19] (79 SRS documents, Zenodo CC-BY 4.0). PURE’s structured XML <req> elements allow consistent automated extraction of requirement statements, and the dataset has prior use as a Requirements Engineering (RE)–NLP benchmark [12], [13]. We selected four documents covering distinct domains (security, e-commerce, law enforcement, networking), each with 20–35 functional requirements, exceeding the single-document scope of comparable studies [11], [15]. Table 1 summarizes the selection.

4.2.2 Gherkin Generation Procedure.↩︎

For each selected document, requirements were extracted automatically using an XML parser targeting <req> elements. Duplicate sentences were removed. The resulting requirements list was passed to the Timeless Gherkin generation pipeline using the same LLM prompt and model (gpt-4o-mini, OpenAI) as the production system. Generation latency (LLM call time) was recorded for each document. No manual editing of the generated output was performed.

4.2.3 Baseline.↩︎

The zero-shot baseline uses the identical model (gpt-4o-mini) and requirements input as Timeless, but replaces the structured prompt with a minimal two-sentence instruction: “You are a software requirements expert. Generate Gherkin BDD acceptance criteria for the given requirements.” No epic context, JSON schema, or BDD structural constraints are provided, isolating the contribution of prompt engineering from the underlying LLM capability.

4.2.4 Metrics.↩︎

We use two fully automated metric categories.

4.2.4.1 Structural quality (RQ1).

Structural Validity (SV) measures whether Given precedes When precedes Then in each scenario; out-of-order keywords cause parse-time rejection by BDD frameworks. Step Completeness (SC) measures whether each scenario contains at least one step of each mandatory type. Negative Coverage (NC) is the proportion of Feature blocks containing at least one negative or edge-case scenario, identified by keyword matching (“fail”, “invalid”, “error”, “exceed”), a documented failure mode of LLM-generated Gherkin [11]. Scenario Density is the ratio of generated scenarios to input requirements.

4.2.4.2 Requirement coverage (RQ2).

Requirement coverage here measures the proportion of input requirements addressed by at least one generated scenario; it is distinct from requirements traceability as defined in IEEE 29148, which refers to bidirectional lifecycle artifact links. Prior automated BDD evaluation studies report only lexical or human-assessed metrics, without automated requirement coverage measurement [11], [14], [15]; this study introduces dual measurement to isolate vocabulary bias from genuine coverage gaps. We compute TF-IDF cosine similarity (\(\tau{=}0.20\), uni- and bigrams) for lexical coverage and OpenAI text-embedding-3-small cosine similarity (\(\tau{=}0.50\)) [20] for semantic coverage; dense embeddings provide a complementary view because they can capture some paraphrasing that TF-IDF may miss. Both methods compute RCR (proportion of requirements matched above threshold) and SSR (proportion of scenarios with no match above the selected threshold, indicating possible unsupported or weakly linked scenarios).

4.3 Results↩︎

4.3.1 RQ1 — Structural Quality.↩︎

Table 2 presents the automated structural metrics for Timeless and the naive baseline across all four documents.

The JSON schema constraint eliminates step-ordering errors by design: Timeless produces zero structurally invalid scenarios across all 119 generated, while the naive baseline records one mal-ordered scenario in the Gamma-J document (99% SV). The distinction between the two systems is not the one-point margin but its nature: Timeless’s perfect SV in this experiment reflects the effect of constrained generation and post-generation validation. Unlike the baseline, this approach explicitly checks the expected scenario structure before accepting the output. The baseline’s near-perfect SV is a statistical observation that holds for this model and dataset but provides no guarantee for unseen inputs or different models.

Step completeness is 100% in both systems (190 scenarios total), reflecting a GPT-class capability floor rather than a contribution of structured prompting.

The naive baseline achieves higher negative scenario coverage (81.2%) than Timeless (67.0%). The baseline groups requirements into broad features that each contain many scenarios, increasing the chance of including a negative case; Timeless’s gap-fill pass creates focused features with fewer scenarios per feature. Negative coverage correlates with scenario density, not prompt design: a feature with few scenarios rarely includes negative cases regardless of how it was generated.

4.3.2 RQ2 — Requirement Coverage.↩︎

We apply lexical (TF-IDF) and semantic (dense embedding) coverage analyses to isolate epic-level abstraction from genuine coverage gaps.

4.3.2.1 Two-pass coverage.

The two-pass design achieves 94.3% semantic RCR, which is comparable to the naive baseline at (92.9%), while TF-IDF RCR is only 72.0%. Three documents reach \(\geq\)​96% semantic RCR (100% on Internet Peering), driven by the explicit coverage mandate and second-pass gap-fill.

4.3.2.2 Lexical vs.semantic gap (TF-IDF bias).

The 22.3 pp lexical-semantic gap (72.0% vs.%) confirms TF-IDF underestimates abstracted Gherkin: Timeless scenarios paraphrase requirements at epic granularity, whereas the baseline echoes requirement wording directly (baseline gap: +16.9 pp). For this reason, lexical coverage should be interpreted alongside semantic coverage when evaluating structured Gherkin-generation pipelines.

4.3.2.3 Spurious Scenario Rate.

The TF-IDF SSR for Timeless averages 33.0%, particularly elevated for CCTNS (67.6%). This is expected: the second-pass gap-fill generates scenario clusters that address groups of uncovered requirements rather than individual ones, so no single scenario achieves lexical overlap above threshold with any individual requirement. Semantically, SSR is only 10.0%, confirming these scenarios are grounded in requirements; the elevated TF-IDF SSR is a measurement artifact.

4.3.3 Generation Latency.↩︎

The Timeless two-pass pipeline averages 58.9 seconds per document (gpt-4o-mini), compared to 21.5 seconds for the naive baseline. The additional latency reflects the second-pass gap-fill call; both pipelines complete in under 65 seconds, so acceptance criteria are available for practitioner review within the meeting in which requirements are elicited. This latency tradeoff, together with the NC and SSR tradeoffs identified above, informs the answer to RQ4 below. Figure 3 summarizes the key automated and expert metrics across both systems.

4.4 Human Expert Evaluation (RQ3)↩︎

4.4.1 Protocol.↩︎

To evaluate the representational quality of generated Gherkin from a practitioner perspective, we conducted a pre-registered blind expert evaluation [3] archived on OSF². Four SE researchers from four independent organizations served as evaluators (all PhD researchers; all with RE and/or BDD experience). Each received a blinded survey (Microsoft Forms) with four sections, one per document, presenting the full requirements list alongside System A and System B Gherkin outputs, randomly assigned to Timeless and the naive baseline (fixed seed; decode key archived on OSF post-collection). Each system was rated on three 1–5 Likert dimensions: Correctness (are Given/When/Then steps logically valid?), Executability (can a developer implement step definitions directly?), and Completeness (do scenarios collectively cover the requirements?). Correctness and Executability are per-scenario; Completeness is per-document. Inter-rater agreement is Fleiss’ \(\kappa\). Pre-registered hypotheses: H1: Timeless scores higher on Executability (behavior-level steps are more directly implementable); H2: Timeless scores higher on Correctness (JSON schema enforces structure); H3: the baseline scores higher on Completeness (higher requirement-aligned lexical coverage).

4.4.2 Results.↩︎

Timeless scores higher than the naive baseline on all three dimensions across all four documents (Table 4). Mean Correctness is 4.61 vs 4.14 and mean Executability is 4.61 vs 4.07, supporting H1 and H2. The largest margin appears in Internet Peering (Correctness \(\Delta{=}{+}0.79\), Executability \(\Delta{=}{+}0.92\)), where the baseline’s flat, requirement-echoing scenarios diverge most from executable behavior descriptions.

Contrary to H3, Timeless scores higher on Completeness (4.31 vs 3.50, \(\Delta{=}{+}0.81\)). Raters perceived epic-organized Gherkin as more complete than the requirement-aligned output, an unexpected result we attribute to Timeless’s higher scenario density (1.12 vs 0.60 per requirement) and the explicit second-pass coverage mandate. The baseline’s requirement-aligned structure, despite echoing requirement wording directly, does not translate into perceived coverage at the Feature-block level.

Fleiss’ \(\kappa\) is near zero across all documents (\(\kappa = -0.08\) to \(0.03\)), reflecting systematic rater scale-usage differences: one rater consistently used the lower end of the scale while three used the upper range. Despite this, the directional ranking (Timeless \(>\) Baseline) holds for 15 of 16 rater-document pairs on Correctness and Executability, and for all 16 on Completeness. All evaluation materials and analysis scripts are publicly available at https://osf.io/9apdf.

5 Discussion↩︎

5.0.0.1 For researchers.

The 22 pp gap between TF-IDF RCR (72.0%) and semantic RCR (94.3%) shows that lexical metrics undercount coverage of abstracted, epic-organized Gherkin. Studies that report only TF-IDF or BLEU-type coverage metrics for BDD generation [11], [14], [15] should be interpreted with this bias in mind: vocabulary mismatch, not missing coverage, drives the gap. More fundamentally, this is a construct validity concern: comparing a structured pipeline against a requirement-echoing baseline using only TF-IDF measures vocabulary retention, not semantic coverage, and systematically favors the less abstracted system. Future evaluations should report the lexical-semantic gap alongside both metrics; a large gap signals representation abstraction, not low coverage.

5.0.0.2 For practitioners.

A 59-second average generation time places scenario authoring within the requirements meeting itself; acceptance criteria are available for review before the meeting closes, not drafted separately the next day. JSON schema enforcement eliminates a concrete CI failure mode: malformed Gherkin breaks a build at feature-file loading, before a single test runs; schema-constrained output passes that check by construction, removing a manual validation step from the developer’s workflow. Epic-organized scenarios match the Feature/Scenario block structure that Cucumber and SpecFlow expect [4], [6], so output can be pasted directly into a .feature file and executed without restructuring; practitioners rated this output higher on Correctness (4.61 vs 4.14), Executability (4.61 vs 4.07), and Completeness (4.31 vs 3.50) [4], [6]. Teams with per-requirement traceability obligations for compliance or audit can run an additional single-pass generation to produce requirement-aligned output alongside the epic-organized structure, preserving both a readable BDD artifact and a per-requirement audit trail.

5.0.0.3 Limitations.

The evaluation uses four documents from the PURE dataset (107 requirements); while consistent with comparable studies [11], [15], this limits confidence in the observed trends. All documents contain 20–35 functional requirements; it is not known whether the epic-grouping strategy scales to larger SRS documents with 100+ requirements, where finer feature granularity may be needed.

6 Threats to Validity↩︎

6.0.0.1 Construct validity.

Provider coupling between gpt-4o-mini (generation) and text-embedding-3-small (embedding) means the semantic RCR score may be inflated if both models share similar internal representations; replication with Sentence-BERT [20] would isolate this effect. Structural metrics do not capture readability, domain accuracy, or test precision [3], which the expert evaluation addresses.

6.0.0.2 Internal validity.

The zero-shot baseline isolates structured prompting’s contribution; a chain-of-thought or few-shot baseline would narrow the NC and expert-rating gaps, though the architectural SV guarantee would remain. Results should be read as structured vs.unstructured prompting, not as the upper bound of LLM capability.

6.0.0.3 External validity.

Four documents yield four data points per metric, sufficient to show consistent trends but not enough for statistical generalization. Documents were selected to cover distinct domains (security, e-commerce, law enforcement, networking), following the sampling strategy of comparable LLM-based RE evaluations [12], [13].

6.0.0.4 Conclusion validity.

All results are from a single run per document with gpt-4o-mini (OpenAI, April 2026); LLM non-determinism means repeated runs may produce different scenario counts and content. Structural guarantees depend on the JSON-constrained prompting strategy and must be re-validated for each new model or version.

7 Conclusion and Future Work↩︎

Gherkin scenario authoring requires domain knowledge, structural discipline, and full requirement coverage, a combination difficult to sustain manually at scale. This paper evaluates a two-pass LLM prompting approach that organizes requirements into semantically coherent epics before generating JSON-constrained Gherkin output, compared against a zero-shot baseline across four PURE SRS documents (107 requirements).

JSON-constrained generation achieves 100% structural validity by architectural design. Semantic coverage reaches 94.3%, matching the zero-shot baseline (92.9%), while TF-IDF underestimates abstracted Gherkin by 22 pp, a systematic bias lexical-only studies cannot detect. Expert evaluation confirms that epic-organized output scores higher on all three quality dimensions, including Completeness where the pre-registered hypothesis predicted the reverse.

Three directions follow from these findings. First, extending the evaluation to \(n \geq 10\) PURE documents and live transcription inputs will test whether the coverage and quality trends hold across a wider requirement dataset (Section 6, external validity). Second, comparing the zero-shot baseline with chain-of-thought and few-shot prompts will separate the contribution of epic organization from the benefit of richer prompting alone (Section 6, internal validity). Third, replication with open-source embedding models such as Sentence BERT [20] will isolate the provider coupling concern in the semantic coverage measure.

The results suggest that epic-organized, schema-constrained generation is a promising direction for automated BDD support. Further evaluation with stronger baselines, more documents, repeated model runs, live meeting inputs, and manually validated traceability data is needed before the findings can be generalized.

Declaration of AI Assistance↩︎

During the preparation of this manuscript, the authors used ChatGPT to assist with grammar refinement, sentence restructuring, and formatting improvements. Following the use of this tool, the authors carefully reviewed and revised the content and assume full responsibility for the final version of the publication.

Acknowledgment↩︎

This work has been supported by FAST, the Finnish Software Engineering Doctoral Research Network, funded by the Ministry of Education and Culture, Finland and ANSE (AI Native Software Engineering) (1822/31/2025).

References↩︎