1 Introduction↩︎

As physicists test theories on extreme cases such as infinite mass, software developers routinely test their code on what are colloquially called edge cases that are manually generated. Our paper uses AI to automate the generation of edge cases for network protocol implementations by first using LLMs to extract constraints from protocol specification documents (RFCs). We started this project after realizing that many semantically meaningful edge cases – such as long DNS names or URLs with invalid characters – were unlikely to be generated by other automated test generators such as fuzzers [1] or testers that use symbolic execution engines [2], [3].

Problem: Elaborating, many real-world software bugs appear near the edges of the specification. This problem is exacerbated in network protocol implementations, which have complex input formats with many associated options and constraints. Edge cases include invalid field combinations in messages (e.g., the TCP Christmas Tree attack [4]), empty or missing fields, malformed fields, and rare but valid combinations of parameters. Notably, they include messages that violate protocol semantics, such as well-formed protocol messages received in unexpected states. As an example, unexpected handshake messages in TLS (e.g., CLIENT_HELLO in an invalid state) can lead to crashes or inconsistent state [5]. Other examples include the infamous Heartbleed bug [6] (CVE-2014-0160) in OpenSSL, where the claimed length was larger than the actual payload.

The attacks above were manually crafted by hackers, but the rise of LLMs has raised the specter of attackers using AI to automatically generate such extremal attacks, just as Anthropic uses their Mythos [7] model to identify memory safety vulnerabilities. In this paper, we ask: can we use LLMs to automatically generate extremal inputs to harden Internet protocol implementations to (primarily) improve reliability and (secondarily) security, focusing on message safety errors – messages that trigger implementation bugs.

Solution Approach: We answer this question in the affirmative with CornerCase, an approach and implementation that uses LLMs in a structured way to generate extremal tests for protocol implementations and analyze testing results (illustrated in 1). The input to CornerCase consists of (i) a protocol specification (e.g., an RFC), (ii) a user-defined test input/output format (see Appendix 8), and (iii) a test harness for executing inputs against implementations. The output is a set of extremal test cases together with a ranked set of differential anomalies observed across implementations. This design makes CornerCaseprotocol-agnostic and implementation-agnostic. In particular, our approach treats implementations as black boxes and does not require source-code access, making it applicable to both open and closed-source systems.

Our approach leverages two properties of many protocols. First, they have detailed English specifications such as RFCs, which describe validity rules and constraints in natural language, so LLMs provide a natural mechanism for extracting and reasoning about these constraints. Second, protocols often have multiple independent implementations. This enables us to find bugs via differential testing: we can compare behaviors across implementations and flag inconsistencies, which often indicate real bugs or ambiguous parts of the specification.

Methodologically, instead of asking the LLM to generate test cases in one shot, we first use it to extract explicit validity constraints from the specification document, one section at a time, guided by the user-provided test input format to focus on constraints relevant to our test setup. The LLM outputs constraints in a structured form (see Appendix 9) as tuples of constraint ID, section number, and the original RFC sentence which contains the constraint. As part of constraint extraction, we also resolve in-text references to other sections (if any) which are used to define the constraint and add to the extracted tuple.

The key methodological choice behind this decomposition is to use the LLM for specification understanding before using it for test construction. Asking a model to directly generate protocol tests from a full RFC produces outputs that are broad but shallow: the model misses constraints, under-explores subtle boundaries, and fails to cover the specification systematically. By first extracting explicit constraints and only then generating extremal tests from them, we convert a vague end-to-end generation task into a coverage problem over constraints in the specification. We then separately ask the LLM to generate extremal tests (see prompt in 9.2) for each extracted constraint, ensuring all generated tests conform to the user-provided test format. Each constraint is translated into test cases by modifying only the relevant input fields to produce values just below, at, and just above the specified boundary.

Separating specification understanding from test construction aligns with LLM strengths and avoids their weaknesses. We show using ablation testing (in 3) that our decomposition produces up to \(22\times\) more differential anomalies than one-shot LLM generation.

Each generated test case is executed across multiple implementations using a user-provided test harness that runs inputs against implementations and records outputs in a unified format. We then perform differential analysis to detect inconsistencies in behavior. Finally, we group related differential anomalies using LLM-generated tags that capture the underlying constraint and boundary type. A tag is a label that captures the constraint id and whether the test is just valid or just invalid (see 3.3). The LLM also assists in drafting candidate bug reports for manual inspection and editing.

Extremal testing is related to boundary-value analysis (BVA) [8]–[11], a classical testing technique that targets inputs at the edges of allowed ranges. However, traditional BVA primarily focuses on numeric or range-based constraints and typically relies on manually identified boundaries. By contrast, extremal testing generalizes this idea to a broader class of constraints, extracting syntactic, semantic, and state-based rules from RFC text. A very preliminary version of the conceptual ideas in this paper was described in [12].

Using CornerCase, we discovered bugs across HTTP, DNS, BGP, SMTP, and QUIC implementations. To characterize these, we distinguish between valid boundary cases – inputs that satisfy the specification but lie at the edge of acceptable behavior – and invalid boundary cases – inputs that are almost well-formed yet violate protocol rules. These boundaries may arise at the syntactic, semantic, or state-machine level.

For example, the HTTP server h2o [13] responds to requests whose path contains a percent-encoded null byte (e.g., GET /\(\%00\)) with a 301 redirect to a semantically equivalent path, inducing a redirect loop that is a potential vector for low-effort denial-of-service attacks (a valid semantic boundary case).

Other examples include malformed Host headers accepted as valid (invalid syntactic boundary cases, one of which is a potential phishing exploit), nested MAIL transactions in SMTP (a state-machine boundary case), incorrect TLS versions accepted during the QUIC handshake (an invalid semantic boundary case), and BGP routes whose AS_PATH contains the router’s own confederation identifier (a valid semantic boundary case).

These results suggest that many important protocol bugs arise not from arbitrary malformed traffic, but from inputs that sit just inside or just outside specification boundaries, or that are valid in isolation but invalid in context—precisely the regime that extremal testing is designed to expose.

In the process of running CornerCase, we learned the following lessons that are elaborated on later.

1. Focus versus Context: Our first prompts missed many constraints until we prompted the LLM to go section-by-section in each RFC to focus the model on a small piece of text. We did better by adding sections referenced in a section’s constraints (more context).

2. Prompt Templates for General Protocols: By allowing the user to specify the test format as a JSON file, the framework can be easily extended to new protocols and previously unanticipated features. For example, our HTTP tests improved when we moved from a fixed filesystem – where the model only generates URI queries – to a richer format that allows it to specify which files should and should not exist, along with the query.

3. Bottleneck Shift: Our use of LLMs speeds up test and anomaly generation – so much so that the bottleneck now moves to bug validation and understanding. CornerCase produces hundreds of discrepancies per protocol, including genuine bugs, artifacts of user misconfiguration, and cases where multiple behaviors are arguably acceptable under the specification. To manage this volume, we use AI for prioritization, tagging, and triage.

Our paper makes the following contributions:

1. Insight: We show that extremal testing becomes more powerful once LLM use is decomposed into two stages: constraint extraction from natural-language specifications, followed by extremal test generation from constraints, transforming end-to-end test generation into a coverage problem over constraints in the specification.

2. Methodology: We develop a protocol-agnostic, implementation-agnostic pipeline that uses only RFC text, a user-provided test format, and a black-box execution harness, enabling testing across heterogeneous implementations without source-code access or manual formalization.

3. Evidence: We evaluate the approach on 38 implementations across 5 protocols, discovering bugs and inconsistencies ( acknowledged, fixed). An ablation study shows that each component of the decomposition is essential, with the full pipeline producing up to \(22\times\) more anomalies than one-shot LLM generation.

The rest of the paper develops our claims from three angles: the kinds of extremal bugs the method surfaces (in 2), the structured pipeline that makes those bugs discoverable (in 3), and empirical evidence that decomposition matters and finds new bugs across protocols (in 4).

2 Motivating Examples↩︎

The following three examples are chosen to illustrate three different ways specification boundaries matter in practice. The HTTP null-byte case lies at a valid semantic boundary – percent-encoding rules permit it syntactically, but its interpretation falls outside what URI-processing components are consistently defined to handle. The BGP confederation example is a valid semantic boundary: the message is well-formed, but its validity depends on reading one field (the AS path) in light of another (the router’s confederation identity). The SMTP nesting example is a state-dependent boundary: each command is individually valid, but the sequence becomes invalid given the protocol state. All three were found using CornerCase, acknowledged by the relevant developers, and are now fixed.

2.1 HTTP Null Byte Loops (h2o)↩︎

RFC 3986 states that a URI with percent-encoded null bytes (%00) "should be rejected if the application is not expecting to receive raw data" in the component. Historically, null bytes have been used in injection attacks that exploit inconsistencies in low-level string handling. While most modern HTTP servers treat such inputs as malformed and return a 400 Bad Request or 404 Not Found response in the context of static fileservers, we used CornerCase to identify two issues with null byte handling in HTTP servers: h2o [13] and Caddy [14]. The model generated this test twice: from the RFC should statement, and the format rules for percent encoding.

When a request is issued for a path containing a null byte (e.g., GET /%00), h2o responds with a 301 Moved Permanently redirect whose Location header points to a semantically equivalent path (e.g., /%00/), by appending a trailing slash. This can induce redirect loops, leading to repeated requests and resource amplification. While individual clients typically bound redirect depth, such behavior can still increase load in aggregate (e.g., from crawlers or automated clients), potentially creating a low-effort denial-of-service vector.

Caddy, by contrast, propagates the malformed path to the filesystem layer, where a null byte in a stat call triggers a system-level EINVAL error that surfaces as a 500 Internal Server Error. Rather than rejecting the request as malformed at ingress, Caddy leaks an internal failure mode – one that may interfere with upstream error handling or monitoring systems that treat 5xx responses as server faults.

2.2 BGP Confederation Loops (GoBGP)↩︎

BGP requires that routers reject routes whose AS_PATH contains their own autonomous system (AS) number, in order to prevent routing loops. This rule also applies to BGP confederations. RFC 5065 specifies that if a router receives a route whose AS_PATH contains its own confederation identifier, it must treat it as an AS loop and reject the route.

Extremal Testing generated a test case where a router receives a route whose AS_PATH contains its own confederation ID. Unlike the previous example, the received message is syntactically valid, but should be rejected because it violates a semantic constraint. When executed across implementations, FRR [15] and Batfish [16] correctly rejected the route, treating it as an AS loop. In contrast, GoBGP [17] accepted the route and installed it in the routing table. Accepting such routes can lead to incorrect behavior and potential loops.

2.3 SMTP Nesting Failures (Mailpit)↩︎

For an example of state-based constraints, SMTP restricts when a new mail transaction may begin. Once a MAIL FROM command has been accepted and one or more RCPT TO commands have been issued, the server expects the client to either send DATA or abort the transaction. Starting a new mail transaction with another MAIL FROM at this point is explicitly forbidden by RFC 5321.

Extremal Testing generated a test case that issues a second MAIL FROM command to a server after it has accepted a recipient, but before DATA is sent. This example is extremal but different from the above two examples: the command is syntactically valid but it is semantically invalid based on the current state of the protocol. Most SMTP servers that we tested rejected the nested MAIL FROM command with a 503 error or closed the connection. In contrast, Mailpit [18] responded with 250 2.1.0 Ok, effectively allowing a new transaction to begin while the previous one was still open. Allowing nested transactions can lead to inconsistent internal state and undefined behavior.

3 Methodology↩︎

Our methodology seeks not just to automate test generation, but to convert natural-language specifications into a structured set of test targets. In that sense, the pipeline is best understood as a way of transforming RFC prose into a coverage object over which boundary-focused testing becomes systematic, traceable, and protocol-agnostic.

1 shows the full pipeline in which we: (1) extract validity constraints from the specification (2) generate many extremal tests that are near the edge of these constraints (3) run these tests across multiple implementations and use differential testing to detect anomalies. We describe each stage below.

3.1 Input↩︎

Extremal Testing targets systems where (a) there is a written specification (often an RFC), and (b) there are multiple independent implementations that can be tested. These conditions hold for many network protocols.

Our pipeline assumes the following user inputs: Specification document: An RFC or similar document that describes the protocol and its rules.

Input/output test format: A structured representation of a test case in JSON format, including English descriptions of each test input field and a similar JSON file with test output fields and their description. (See Appendix 8 for more details)

A tester directory: A directory with a main script that can execute test cases in the JSON test input format against each implementation and output results in the JSON test output format.

3.2 Stage 1: Constraint Generation↩︎

In the first stage of the pipeline, we extract the validity constraints from the specification. The output is a set of explicit constraints that can later be used for systematic test generation.

3.2.0.1 Splitting the specification:

Specifications are often too long to feed into the model at once. We therefore split the document into sections. We begin with the entire RFC in a single text document. We then scan the RFC line by line and treat any line matching the section-header pattern ^\s*(\d+(?:\.\d+)*)\.\s+(.+)$. as starting a new subsection, where the first capture group is the section number (for example, 3, 3.1, or 3.1.2) and the second is the title.

For each such header with section number \(k\), we collect all lines from that header up to (but not including) the next header and save the resulting text in a file named section_k.txt. Here, \(k\) is the section number with dots replaced by underscores (for example, 3.1 becomes section_3_1.txt). We then feed these section_k.txt to the LLM one at a time, along with the test format given by user and a prompt to extract all constraints in the text of the section. The LLM returns a list of constraints for each section, which are appended to a global constraint list.

3.2.0.2 Constraint definition:

We define a constraint as any sentence in the specification that restricts how protocol inputs may be formed, ordered, or combined – that is, a boundary between acceptable and unacceptable input that an implementation is expected to enforce. Concretely, the LLM prompt (see Appendix 9) directs the model to find sentences that describe: syntax rules, allowed or disallowed values, length or size limits, character set restrictions, relationships between multiple inputs, ordering or state rules that can be represented as test inputs etc.

The prompt specifies that constraints are generally RFC statements that use normative language (MUST, MUST NOT, SHOULD, SHOULD NOT), but also includes non-normative sentences that clearly describe a testable rule. Crucially, the LLM is instructed to return each constraint sentence exactly as written in the RFC—we do not rewrite, normalize, or formalize constraints at extraction time. Each constraint is recorded as a tuple ["<section_number>", "<constraint sentence>"]. This avoids introducing interpretation errors and allows later stages to trace each test case directly back to the source.

The test input format plays a key role here: it is included in the prompt so that the LLM can infer which inputs are controllable in the testing framework, and thereby select only those RFC sentences that describe constraints on those inputs. This naturally filters out specification rules that, while valid constraints, cannot be exercised by the test setup (see also the discussion of test-format filtering below).

Based on our extraction process, constraints typically fall into the following categories, illustrated here with concrete examples from our extracted constraint sets:

1. Range constraints: numeric bounds on values. For example, SMTP reply codes must begin with a three-digit numeric code (RFC 5321, §2.4), and DNS limits each label to between 1 and 63 octets (RFC 2181, §11).

2. Size constraints: minimum or maximum sizes for strings, lists, packets, or fields. For example, SMTP limits command line length to 512 octets including the <CRLF> (RFC 5321, §4.5.3.1.4), and DNS limits a full domain name to 255 octets including separators (RFC 2181, §11).

3. Format constraints: syntactic rules that inputs must follow. For example, URI scheme names must consist of a sequence of characters beginning with a letter and followed by any combination of letters, digits, plus, period, or hyphen (RFC 3986, §3.1), and HTTP header fields must follow a name-value syntax (RFC 9110).

4. Dependency constraints: relationships between multiple inputs. For example, in SMTP, the local-part of a mailbox MUST BE treated as case sensitive while verbs and keywords are not (RFC 5321, §2.4), and in URIs, a percent-encoded octet must be encoded as a character triplet consisting of "%" followed by exactly two hexadecimal digits (RFC 3986, §2.1).

5. Presence constraints: conditions under which a field or command is required or forbidden. For example, SMTP requires that a client MUST issue HELO or EHLO before starting a mail transaction (RFC 5321, §4.1.1.1), and TLS 1.3 requires that clients desiring certificate-based server authentication MUST send the signature_algorithms extension (RFC 8446, §4.2.3).

6. Enumeration constraints: inputs that must be chosen from a fixed set of allowed values. For example, in TLS 1.3, MD5, SHA-224, and DSA MUST NOT be offered or negotiated (RFC 8446, §4.2.3), and DNS record types must be valid RR types (RFC 1035).

7. Ordering and state constraints: rules about the order in which inputs or commands may appear. For example, SMTP specifies that mail transaction commands MUST be used in the prescribed order (RFC 5321, §3.3). TLS 1.3 requires that protocol messages MUST be sent in the order defined in Section 4.4.1 (RFC 8446, §4).

8. Cross-field semantic constraints: rules whose validity depends on interpreting the meaning of values across multiple fields or protocol state, beyond simple syntactic checks. For example, a BGP confederation member receiving an AS_PATH containing an autonomous system matching its own AS Confederation Identifier SHALL treat the path as if it contained its own AS number, i.e., reject the route as a loop (RFC 5065, §4). This constraint cannot be enforced by syntax checks alone – it requires comparing the AS_PATH content against the router’s own configuration state.

3.2.0.3 Cross-reference detection.

We detect cross-references inside each constraint sentence using a similar regex pattern. Specifically, we look for textual references of the form “Section 4.1”, “Sections 4.1 and 4.2”, etc., and extract all referenced section numbers from the matched group and augment the model’s input with the corresponding RFC sections when generating tests.

3.2.0.4 Mapping constraints to the test format:

The RFC contains many statements that might be categorized as constraints but may not be relevant for our test setup. When we extract constraints from the RFC sections, the setup summary is provided to pick only those constraints that are testable with our testing setup – e.g. A constraint from BGP RFC 4271 that says, "KEEPALIVE messages MUST NOT be sent more frequently than one per second" is not relevant if our test setup is testing the decision process and choosing the best route.

3.2.0.5 De-duplication and normalization:

New constraints are added to the global list only if the pair (section number, constraint) was not seen previously.

3.3 Stage 2: Test Generation↩︎

The goal of this stage is to systematically construct tests that lie close to the boundary between valid and invalid behavior, as described by each constraint.

3.3.0.1 Constraint-driven generation:

Each test case is generated from exactly one RFC constraint sentence. Constraints are passed to the test generator verbatim, together with their RFC section number and the text of any cross-referenced sections. Every generated test explicitly records the constraint that it is intended to exercise. This allows each test to be traced directly back to a specific sentence in the specification.

3.3.0.2 Extremal values:

For each constraint, we ask the LLM to generate multiple extremal test cases. These include: values that barely satisfy the constraint (e.g., minimum allowed length) and values that barely violate the constraint (e.g., one character too long),

For numeric constraints such as \(0 \leq x \leq 255\), the LLM can generate values like \(-1\), \(0\), \(1\), \(254\), \(255\), and \(256\). For size constraints such as \(\texttt{len}(s) \leq 1024\), the LLM generates strings with lengths just below, at, and just above the limit. For format and ordering constraints, the model is instructed to generate inputs that are syntactically valid but placed in invalid positions, or inputs that differ from valid ones by a minimal change.

3.3.0.3 Example test cases:

To illustrate the kinds of tests generated by Extremal Testing, consider the SMTP constraint in RFC 5321 Section 2.3.5, which states that – "the reserved mailbox name postmaster MUST be accepted in a RCPT command without domain qualifications."

From this single constraint, our system generates both positive and negative boundary tests. A positive test checks that RCPT TO:<postmaster> is accepted after a valid EHLO and MAIL FROM sequence. In contrast, negative boundary tests modify the input slightly to violate the constraint, such as using an almost-matching name (postmaste) or an invalid form (postmaster@). These inputs differ by only a single character or structural detail, but must be rejected according to the RFC. Such near-boundary cases are difficult to generate using fuzzing or model-based testing.

3.3.0.4 Tag Generation:

Along with the test cases, LLM generates a tag for each of them. The tag is a label that combines the constraint identifier with a boundary indicator (positive or negative), such as C11_Positive or C18_Negative. Positive means just valid, and Negative means just invalid test cases.

3.3.0.5 Batching constraints:

Constraints are processed in small batches (5 by default). Each batch is sent to the LLM in a separate call. Batching is necessary in practice: providing too many constraints at once leads to repetitive outputs and poor coverage. Processing small batches encourages the model to focus on each constraint and produce multiple distinct extremal tests.

3.3.0.6 Context provided to the LLM:

For each batch, the LLM is given (see prompt in 9.2): the input test case format in JSON, the exact constraint sentences for the current batch, and, when applicable, the full text of any RFC sections referenced by the constraints.

Providing referenced RFC sections helps the model correctly interpret constraints whose meaning depends on other parts of the specification, such as state transitions or exceptional cases.

3.3.0.7 Test format and identifiers:

All generated tests must conform exactly to the user-provided input test format. The LLM is instructed to populate each field according to the intended extremal test case. This grounding reduces failures due to unrelated parsing errors and increases the likelihood that tests exercise the intended code paths. The LLM outputs only a JSON array of test objects, with no additional text. Each test object includes a constraint field containing the exact RFC sentence it targets. Test identifiers are assigned post-generation to ensure global uniqueness across batches.

3.3.0.8 Coverage objective:

The objective of test generation is to maximize the coverage on the RFC constraints, not code coverage. For each extracted constraint, we aim to generate tests that explore behavior just below, at, and just above the specified boundary.

Overall, this stage converts natural-language specification rules into concrete, boundary-focused test inputs in a systematic and reproducible way, while keeping the generation process simple and scalable.

3.4 Stage 3: Test Execution↩︎

After generation, we run the user-provided test script to execute tests on each implementation and produce a results file in the specified output format. In our experiments, these scripts use Docker containers for each protocol implementation. Given a test case \(t\) and an implementation \(I\), the tester starts or reuses the corresponding container or process, constructs the concrete input message(s) from \(t\), sends them to \(I\), waits for a response, and records the output in the expected format.

3.5 Stage 4: Differential Testing↩︎

After executing all test cases, we perform differential analysis to identify cases where implementations behave differently. The goal is not to decide which implementation is correct, but to surface test cases that deserve closer inspection.

3.5.0.1 Diffing logic:

For each test case, we use a Python script on the results file to compare the outputs across all implementations, each of which is a JSON dictionary. If at least two implementations return different dictionaries for the same test case, we add the test case and all responses to a separate file for detected anomalies.

3.5.0.2 Output of differential analysis:

All test cases with response disagreement are logged to a separate file. Each entry includes the original test metadata, and the actual responses from all implementations. This allows later stages to inspect, group, and reason about anomalies without losing contextual information.

3.6 Stage 5: Result Analysis↩︎

Differential analysis can surface a large number of test cases that trigger divergent behavior across implementations. Some of these discrepancies are uninteresting, arising from user misconfiguration or cases where multiple behaviors are arguably acceptable under the specification, while others correspond to genuine bugs or specification violations. Our pipeline uses an LLM to analyze these differences and prioritize them, reducing the human effort required for validation and interpretation.

3.6.0.1 LLM-based analysis of differential results:

We process the output of differential testing in batches and feed it to an LLM for analysis. Each batch contains a small number of test cases (e.g., five), along with the implementation setup information and the full test metadata. For each test case, the LLM produces a short textual analysis explaining the observed disagreement and assigns a confidence score between 0 and 10, indicating how likely the case represents a real bug or meaningful inconsistency.

The output of this stage is a list of test cases analyzed, each augmented with an LLM-generated comment and confidence score. These results are written to an analysis file that preserves all original test information.

3.7 Stage 6: Triaging↩︎

In the next stage of the pipeline we perform triaging to eliminate semantically similar results and prioritize the results so that manual investigation becomes easy.

3.7.0.1 Prioritization and grouping:

We sort analyzed test cases by confidence score to prioritize those most likely to correspond to real issues. Since multiple test cases may target the same specification constraint, we group results by their tag (see 3.3) — Within each group, test cases are ranked by confidence score. The pipeline outputs all test cases, grouped and ranked, so that the user can inspect as many or as few as needed. In practice, reviewing the highest-confidence case from each group is an effective starting point, but the full set remains available for deeper investigation.

3.7.0.2 Human validation:

Results are reviewed manually about whether a case represents a true bug before submission to maintainers. The LLM is used only to assist with analysis, prioritization and deduplication. But this pipeline reduces human effort, allowing us to quickly move from hundreds of differential anomalies to a small, manageable set of high-quality, actionable bug reports.

3.8 Summary↩︎

The core of Extremal Testing is a two-stage decomposition of specification-based test generation:

1. Constraint extraction: Convert natural-language specification rules into explicit, test-format-aligned constraints.

2. Extremal Test generation: For each constraint, systematically generate tests at and around the specified boundary.

This decomposition gives the pipeline two properties that test generation typically lacks. First, a form of completeness: by extracting constraints section by section and tracking which constraints have generated tests, we can measure and improve coverage over the specification. Second, traceability: every generated test records the exact constraint sentence and RFC section it targets, so any differential anomaly can be traced directly back to the specification text.

4 Evaluation↩︎

This section evaluates Extremal Testing on multiple real-world protocols and libraries. Our evaluation is designed to answer three questions.

Q1. Does Extremal Testing find real, previously unknown protocol bugs?

Q2. Are those bugs qualitatively different from generic parser failures that existing fuzzers already surface?

Q3. Is the structured decomposition—section-wise processing, constraint extraction, reference expansion—necessary, or would a one-shot LLM suffice?

The setup is described in 4.1. Next the results in 4.2 address the first two questions; the ablation study in 4.3 addresses the third question.

4.1 Experimental Setup↩︎

Unless otherwise stated, all prompt-based stages of the pipeline—constraint extraction, test generation, and anomaly analysis—use GPT-5 via the OpenAI API. Full prompts are provided in Appendix 9.We evaluated Extremal Testing on the following implementations (details in 10)

HTTP servers: Nginx[19], Apache[20], Lighttpd[21], Caddy[14], and H2O[13].

DNS servers: BIND [22], NSD [23], Knot [24], PowerDNS [25], CoreDNS [26], GDNSD [27], Technitium [28], HickoryDNS [29], TwistedNames [30], Yadifa [31]

BGP implementations: FRR [15], GoBGP [17], Batfish [16].

SMTP servers: AioSMTPD [32], MailPit [18], Stalwart [33], SMTPD [34] and OpenSMTPD [35].

QUIC servers: quiche[36], msquic[37], quic-go[38], ngtcp2[39], mvfst[40], kwik[41], picoquic[42], aioquic[43], neqo[44], nginx[45], chrome[46], lsquic[47], haproxy[48], quinn[49], go-x-net[50]

All implementations were run inside Docker containers to ensure reproducibility.

For HTTP, we test URI parsing rules from RFC 3986. Each test specifies a URI with scheme, authority, path, query, and fragment components, along with a filesystem layout with directories and symlinks; we compare the HTTP status code and resolved file path returned by each server. For DNS, we test resource record set semantics from RFC 2181, including duplicate suppression, TTL consistency, and caching precedence. Each test provides a zone file and one or more queries, and we compare the structured response—including answer records and return codes—across all resolvers. For SMTP, we test command syntax and sequencing rules from RFC 5321. Each test specifies a sequence of SMTP commands (e.g., EHLO, MAIL FROM, RCPT TO) and a server state, and we compare the three-digit response code returned by each server for the final command. For BGP, we test confederation handling (RFC 5065) and route reflection (RFC 4456). Each test specifies an AS topology, confederation membership, and advertised routes, and we compare whether each implementation installs the route in its RIB and the resulting AS path at each router. For QUIC, we test the TLS handshake from RFC 8446. Each test specifies modifications to an otherwise valid ClientHello message, and we compare whether the handshake succeeds or fails on each implementation using QuicInteropRunner [51] and a modified aioquic[43] client.

We define a differential anomaly as a test case where at least one implementation behaves differently from the others in a meaningful way—for example, a crash, a different error code (e.g., 4xx vs.5xx), or acceptance vs.rejection of the same input.

For each protocol, we ran the full CornerCase pipeline described in 3. As part of its output, the tool extracts input constraints from the specification, generates boundary-focused test cases, executes them across multiple implementations, identifies differential anomalies, and applies LLM-assisted analysis to score each anomaly based on the likelihood of a specification violation. The tool also groups anomalies by the constraint they exercise and selects the highest-confidence instance per group to reduce redundancy (3.7). From this ranked output, we use a confidence threshold for each protocol (generally  8), lowered when the total number of anomalies is small) to select candidates for manual inspection, and we submit bug reports for anomalies that pass this manual validation step. During manual inspection, some candidates are discarded because they correspond to implementation design choices, configurable behaviors, or cases where the RFC does not strictly mandate a single behavior. In addition, related anomalies are often grouped into a single report. These cases are excluded from the total number of reported bugs.

4.2 Differential Anomalies↩︎

1 summarizes the number of unique constraints extracted, extremal tests generated, differential anomalies found and number of candidates after confidence-based prioritization and triaging – for each protocol. Each constraint produced between 3 and 11 test cases, including values at the boundary, just below it, and just above it. The candidates after the stage of triaging are manually inspected before passing them to the developers.

2 summarizes bugs found across five protocols (after manual inspection). Of these, have been acknowledged and are fixed by maintainers, and the remainder have been submitted and are awaiting response. Manual inspection confirms these anomalies are not mere implementation differences: they expose dozens of concrete bugs and specification violations across widely-used protocol stacks, including security-relevant issues in HTTP host validation and BGP loop detection (2).

A few findings stand out for both their impact and protocol-critical nature. In BGP confederations, we found cases where GoBGP accepted routes with an AS loop and also formed sessions with invalid peer relationships, which can affect routing safety and policy isolation in multi-AS deployments; several of these were acknowledged or fixed. In HTTP, multiple servers accepted malformed or missing Host values but still served content, violating RFC host validation rules and creating a request-routing and virtual-host security risk. In QUIC, version-negotiation behavior around TLS 1.2/1.3 combinations exposed interoperability and downgrade-surface issues, while in SMTP we observed state-machine and syntax-validation errors (for example, accepting MAIL FROM in invalid command sequences), which can enable inconsistent mail handling across implementations. Together, these examples show that extremal differential testing surfaces not only parser edge cases, but also high-value semantic bugs in security and correctness-critical protocol logic.

4.2.0.1 Bug Patterns:

The bugs fall into three broad categories. The most common is missing validation, where an implementation accepts syntactically invalid input that the specification requires be rejected—for example, aiosmtpd accepting MAIL FROM without angle brackets, or Caddy serving files for requests with malformed or missing Host headers. The second category is incorrect state machine transitions, where commands are accepted in the wrong order: Mailpit accepts MAIL FROM before any EHLO greeting and allows a nested MAIL FROM during an active transaction. The third is semantic misinterpretation, where an implementation misapplies a specification rule. For instance, FRR treats a Member-AS appearing in a regular (non-confederation) AS_PATH as a confederation loop, incorrectly rejecting valid routes.

4.2.0.2 Protocol Trends:

BGP confederation bugs involve semantic errors in routing logic, particularly in loop detection and session handling. HTTP bugs are often security-relevant, with malformed Host headers being accepted. SMTP issues stem from weak enforcement of syntax and command sequencing rules, DNS shows robustness problems in handling edge-case semantics and malformed inputs, and QUIC reveals inconsistencies in TLS version negotiation.

4.3 Ablation Study↩︎

We perform an ablation study to identify which parts of the pipeline are responsible for turning raw LLM capability into systematic testing power. The headline result is that each layer of structure matters, and removing any one of them materially reduces anomaly yield (3).

Our system incorporates three key design choices: (1) processing the RFC section by section (S), (2) extracting explicit constraints from the specification before generating tests (C), and (3) including referenced RFC sections when they are cited in the text (R). We evaluate how each of these components affects the number and usefulness of generated tests.

We compare several variants of the pipeline:

Base: The entire RFC is given to the LLM in one prompt, and the model directly generates test cases without section-wise processing, constraint extraction, or reference expansion.

S: The RFC is fed to the LLM section by section, but the model generates test cases directly without first extracting constraints.

SC: The RFC is processed section by section, constraints are extracted from each section, and test cases are then generated from the collected constraints.

SR: The RFC is processed section by section and referenced sections are included when mentioned, but test cases are generated directly without an explicit constraint extraction step.

CR: The entire RFC is provided at once, but the system extracts constraints before generating tests and includes referenced sections when necessary.

SCR (Full System): Our complete pipeline, which processes the RFC section by section, extracts constraints, and includes referenced sections. For each variant, we generate tests and run them against all the implementations listed in 4.1. We record the number of generated tests and the number of differential anomalies discovered across implementations. For variants that perform constraint extraction, we also report the number of extracted constraints. The ablation results across five protocols—DNS, BGP, SMTP, HTTP, and QUIC—are summarized in 3.

The results show that the full pipeline (SCR) consistently produces the largest number of anomalies across all protocols. Each component contributes meaningfully: section-by-section processing (S) alone yields \(\approx10\times\) more anomalies than Base for HTTP (13→131) and \(\approx22\times\) for BGP (2→45). Adding constraint extraction further multiplies anomalies—for SMTP, SCR finds 526 vs. for S alone. Reference expansion (R) has improved DNS results where SCR finds 156 anomalies vs. for SC, probably because some DNS constraints are only fully specified via cross-referenced sections.

The ablation suggests that the central challenge is not getting an LLM to produce tests at all, but getting it to cover the specification systematically. Full-RFC prompting (Base) is too diffuse for boundary discovery. Section-wise processing (S) restores focus by localizing reasoning to a tractable unit. Constraint extraction (C) converts the RFC from unstructured prose into an explicit, enumerable set of test targets, turning test generation into a coverage problem. Reference expansion (R) sharpens this set when a constraint’s meaning genuinely depends on another section. The pipeline works not because it uses more prompting, but because each layer imposes additional structure on the model’s reasoning. On QUIC, SC (20 anomalies) slightly exceeds SCR (16). We read this not as a contradiction but as evidence for a general tradeoff visible across the table: additional context helps only when it sharpens a constraint, and can hurt when it broadens the prompt without adding directly testable structure. Cross-references in RFC 8446 are heavy on prose commentary that does not translate into new controllable inputs in our test format, so reference expansion dilutes the prompt without adding coverage. This is consistent with the larger theme of the paper: for extra contextual information not to reduce focus, it must be dense with information that pertains to and enriches constraints.

4.4 Lessons Learned↩︎

Our experience yields three empirical takeaways: 1. Decomposition beats direct generation: LLMs are substantially more reliable when asked to extract explicit constraints from a localized portion of a specification than when asked to generate end-to-end tests from an entire RFC. Section-wise processing and constraint-first generation are not implementation details; they are the mechanism that turns LLM capability into systematic test coverage (3: up to \(22\times\) anomaly yield vs.one-shot generation).

2. Bottleneck shifts from generation to understanding: With automated test generation, the dominant cost becomes interpreting and de-duplicating differential anomalies. Anomaly ranking, tagging, and triaging therefore become first-class parts of the methodology rather than optional engineering conveniences. For SMTP, we reduced 526 raw anomalies to 84 prioritized candidates and 54 triaged classes, which led to an order-of-magnitude reduction in manual inspection (1).

3. A small interface layer suffices to generalize across protocols: The only protocol-specific inputs are a test format and an execution harness that expose the relevant controllable inputs and outputs. In practice, this interface was sufficient to carry the same pipeline across HTTP, DNS, BGP, SMTP, and QUIC without retraining or redesigning the core method. Modern code-generation tools (e.g., GitHub Copilot) can bootstrap much of this interface from the specification itself, further lowering the per-protocol cost.

5 Related Work↩︎

Related work can be classified by what each approach relies on to generate tests or vulerabilities: implementations (fuzzing, symbolic execution, LLM-based unit testing, AI-based vulnerability analysis), past outputs (AI-based vulnerability analysis), formal models (model-based testing), or natural-language specifications themselves. Extremal Testing sits in the fourth category.

5.0.0.1 Boundary value analysis (BVA):

BVA [8]–[11] typically focuses only on range constraints. Extremal Testing is far more general; for instance, it includes state-based constraints for messages. Typically, BVA does not use an LLM to generate boundary conditions. A recent exception [52] uses an LLM to directly generate boundary value tests from the code while we do so from the specification. Their evaluation is only for code of a few hundred lines, much smaller than the large code bases we handle.

5.0.0.2 Automated testing:

Fuzz testing is widely used for software testing in general [53]–[57] and specifically for BGP and DNS implementations (e.g., [58]–[62]). Although fuzzers are effective at finding parser bugs, random inputs have only a small probability of finding extremal bugs. Symbolic execution uses SMT solvers to generate test cases for many execution paths of a program (e.g., [63], [64]). Protocol implementations contain many thousand lines, making symbolic execution infeasible. Extremal testing, unlike symbolic execution, works on software whose source code is unavailable. Model-based testing uses an abstract model of a system to generate tests [65]. It has been used for DNS [2] and BGP [3] but generates valid, not extremal inputs. Eywa [66] uses an LLM to generate models, not tests.

5.0.0.3 LLM driven test generation:

LLM based software testing is a vast area [67], a subset of LLM based Software Engineering [68]. However, existing work [67] uses LLMs to improve coverage or to improve mutation/fuzz based testing, not for extremal testing. Our framework generates end-to-end tests on multiple implementations instead of unit tests for individual implementations.

5.0.0.4 AI based vulnerability analysis:

Recent work uses LLMs for security, including automated penetration testing [69] and code-level vulnerability detection [70]. LAPRAD [71] first trains an LLM on prior DNS attacks and then constructs new attacks; by contrast, we generate tests based only on RFCs.

6 Limitations and Future Work↩︎

Extremal testing currently has the following limitations:

Implicit Constraints: The constraints defining the Heartbleed [6] exploit (payload length does not match length field) and Christmas Tree packets [5] (SYN, FIN, and RST flags should not be all set) are what one might call implicit constraints: they are not explicitly stated in the RFC but are strongly implied. It may be possible in future work to ask an LLM to generate implicit constraints by a new prompting strategy. Short interactions: We currently focus on generating extremal tests that involve a single message or short command sequence. Extending to protocol behaviors that involve long multi-step workflows, timing dependencies, or complex state machines is interesting future work.

Single RFCs: Constraint extraction is currently based on single RFCs. Protocol behavior can depend on multiple RFCs referencing each other. – e.g., RFC 9113 (HTTP 2) uses RFC 9110 (HTTP Semantics) which references RFC 3986 (URI Syntax).

Multiple Models: While different LLMs may vary in performance on sub-tasks (e.g., constraint identification, triaging), we focus on pipeline-level design choices that are largely model-agnostic. Absolute performance can improve by choosing different models, but we expect the overall trends to remain consistent.

7 Conclusion↩︎

Extremal testing generates tests that occur near the boundaries of specifications. Our goal is to harden widely used Internet protocol implementations against messages with unexpected fields, or valid messages received in unexpected states which can degrade reliability, and in extreme cases expose dangerous vulnerabilities.

At a higher level, the main lesson of this paper is that LLMs are most useful here not as end-to-end test generators, but to convert natural-language specifications into explicit test targets. We use LLMs twice: first, to extract constraints from protocol specifications and only then systematically generate test cases at or near the boundaries of these constraints. Our approach is protocol-agnostic, allowing users to specify their own test input and output formats as JSON.

We applied Extremal Testing to HTTP, DNS, BGP, SMTP, and the QUIC handshake, uncovering 42 bugs and inconsistencies. Of these, 26 have been acknowledged and 18 fixed by maintainers. Bugs span three broad categories: missing input validation (e.g., HTTP servers accepting malformed Host headers), state machine violations (e.g., SMTP servers accepting commands out of sequence), and semantic misinterpretation (e.g., GoBGP failing to reject AS loops in confederation paths). Our ablation study confirms that two-stage decomposition—constraint extraction followed by extremal test generation, applied section-by-section with cross-reference expansion—is essential, producing up to \(22\times\) more anomalies than naive one-shot LLM generation.

Extremal testing’s focus on identifying errors due to violations of constraints in RFCs makes it complementary to existing approaches to protocol implementation reliability, including fuzz testing, symbolic execution, and model-based testing. Together, they form a powerful ensemble of techniques that can make Internet protocol implementations more resilient and secure.

8 Test Format↩︎

This appendix documents the input and output test formats used by each protocol-specific harness.

8.1 HTTP↩︎

Test input format:

{
    "test_id": "<integer>",
    "constraint": "<exact constraint that is being tested from the given list of constraints>",
    "description": "<description of the test case>",
    "tag": "<Constraint id from the given list of constraints. e.g. C1, C2, C3, ...>",
    "scheme": "<uri scheme>",
    "authority": "<uri authority>",
    "path": "<uri path>",
    "query": "<uri query>",
    "fragment": "<uri fragment>",
    "base_uri": "<base uri or null>",
    "expected_response_code": "<expected HTTP response code>",
    "expected_path": "<expected path after resolution>",
    "filesystem": {
        "<dir_path1>": ["<filename1>", "<filename2>", "..."],
        "<dir_path2>": ["<filename1>", "<filename2>", "..."],
        "<dir_pathN>": ["<filename1>", "<filename2>", "..."]
    },
    "symlinks": {
        "<symlink_path1>": "<target1>",
        "<symlink_path2>": "<target2>",
        "<symlink_pathN>": "<targetN>"
    }
}

Test output format:

{
    "status_code": "<HTTP status code (integer)>",
    "resolved_uri": "<resolved URI after any redirects or normalization (string)>"
}

8.2 DNS↩︎

Test input format:

{
    "test_id": "<integer>",
    "constraint": "<exact constraint that is being tested from the given list of constraints>",
    "description": "<Brief description of the test case. Positive for barely valid input, Negative for barely invalid input>",
    "tag": "<Combination of Constraint id from the given list of constraints and Positive/Negative indicator. e.g. C1_positive, C2_negative, C3_positive, ...>",
    "zone":"campus.edu.\t500\tIN\tSOA\tns1.outside.edu. root.campus.edu. 8 6048 4000 2419200 6048\ncampus.edu.\t500\tIN\tNS\tns1.outside.edu.\na.a.a.test.\t500\tIN\tDNAME\tsome.domain.\n",
    "query": [{"Name": "*.a.test.", "Type": "DNAME"}]
}

Test output format:

{
    "Response": "response from the DNS server in a structured format"
}

8.3 SMTP↩︎

Test input format:

{
    "test_id": "<integer>",
    "constraint": "<exact constraint that is being tested from the given list of constraints>",
    "description": "<Brief description of the test case. Positive for barely valid input, Negative for barely invalid input>",
    "tag": "<Combination of Constraint id from the given list of constraints and Positive/Negative indicator. e.g. C1_positive, C2_negative, C3_positive, ...>",
    "prev_command_seq": ["<cmd1>", "<cmd2>", "..." ],
    "server_state": "<server state after executing the previous commands e.g. INIT, EHLO_RCVD, MAIL_FROM_RCVD, ... etc>",
    "command": "<SMTP command to be tested>",
    "expected_response": "<expected server response to the command>"
}

Test output format:

{
    "code": "<integer>, the SMTP response code from the server implementation"
}

8.4 BGP (Confederation)↩︎

Test input format:

{
    "test_id": "<integer>",
    "constraint": "<exact constraint that is being tested from the given list of constraints>",
    "description": "<Brief description of the test case. Positive for barely valid input, Negative for barely invalid input>",
    "tag": "<Combination of Constraint id from the given list of constraints and Positive/Negative indicator. e.g. C1_positive, C2_negative, C3_positive, ...>",
    "originAS": "integer e.g. 512. This is the AS number of the originating router",
    "router2": {
        "asNumber": "integer e.g. 768. This is the AS number of the second router",
        "subAS": "integer e.g. 256. If this is non-zero, then it is part of a confederation and this is the sub-AS number within the confederation"
    },
    "router3": {
        "asNumber": "integer e.g. 1024. This is the AS number of the third router",
        "subAS": "integer e.g. 256. If this is non-zero, then it is part of a confederation and this is the sub-AS number within the confederation"
    },
    "removePrivateAS": "boolean. A flag for router 2. Similar to cisco remove-private-as command. If this is true, then the router 2 should remove all private AS numbers from its AS path before forwarding it to router 3.",
    "replaceAS": "boolean. A flag for router 2. If this is true, then the router should replace all private AS numbers with the confederation number before forwarding the route to router 3.",
    "localPref": "integer e.g. 50. This is the local preference value set by router 2 when advertising to router 3.",
    "isExternalPeer": "boolean. A flag for router 3. If this is True then in BGP config of router 3. the neighbor is configured as neighbor peer-as external. i.e. if this is enabled then connection will be denied if the AS number router 2 is same as mine."
}

Test output format:

{
    "isRIB2": "<boolean>, true if the route gets installed in RIB2",
    "aspath2": "<string>, AS path of the route at RIB2",
    "isRIB3": "<boolean>, true if the route gets installed in RIB3",
    "aspath3": "<string>, AS path of the route at RIB3"
}

8.5 QUIC↩︎

Test input format:

{
    "test_id": "<integer>",
    "constraint": "<exact constraint that is being tested from the given list of constraints>",
    "description": "<description of the test case>",
    "tag": "<Constraint id from the given list of constraints. e.g. C1, C2, C3, ...>",
    "tested_party":"<client | server>",
    "mutations": [{
        "mutation": "<must be remove_field or modify_field>",
        "target": "<client | server>",
        "fields": {
            "field_name": "<name of the field to be removed or modified>",
            "new_value": "<new value for the field if mutation is modify_field, omit if mutation is remove_field>"
        }
    },
    {
        "<mutation>": "more mutations if needed"
    }],
    "expected_result": "<success | fail>, this describes the expected outcome of the tested party.",
    "_comment": "this is for your information only. do not generate this field. Possible values for field_name and their new_value types are the following. See next item in the array for example on how to generate bytes, etc. random: bytes, legacy_session_id: bytes, cipher_suites: list[int], legacy_compression_methods: list[int],alpn_protocols: list[str], early_data: bool, key_share: list[ tuple [int, bytes] ], psk_key_exchange_modes: Optional[list[int]], signature_algorithms: list[int], supported_versions: list[int] signature_algorithms: [list[int]], supported_groups: [list[int]] supported_versions: Optional[list[int]], other_extensions: list[tuple[int, bytes]]. "
}

Test output format:

{
    "handshake_status": "success / failure"
}

9 LLM Prompts↩︎

We use GPT 5 over the OpenAI API for the experiments.

9.1 Constraint Generation Prompt↩︎

This prompt is used to extract testable constraints from each section of the RFC.

System Prompt:

You are an assistant that extracts 
*input-related constraints* from RFC for a
specific testing framework.

### Inputs:
- A test case format (in JSON) describing test 
  case fields
- A chunk of RFC text.

### Task:
1. Use the test case format to infer what
inputs can be controlled by the tests.
2. Scan the RFC chunk and find sentences that 
define constraints on those inputs.
   These include:
   - syntax rules,
   - allowed or disallowed values,
   - length or size limits,
   - character set restrictions,
   - relationships between multiple inputs,
   - ordering/state rules that can be
     represented as test inputs/state.
3. Constraints are generally RFC statements
that include MUST/MUST NOT/SHOULD/SHOULD NOT.
But also look for sentences that describe a 
rule or constraint on inputs that can be
tested with this framework.
4. Every constraint is written as a tuple: 
   (<section_number>, <constraint>).
5. If the chunk has no relevant constraints, 
return [].

### Important:
- Return each constraint sentence *exactly as
written* in the RFC (no edits).
- Only include sentences that can plausibly be
tested using the described setup.

### Output format (for each chunk):
Return ONLY a JSON array like:
[
  ["4.1.1", "sentence1"],
  ["4.1.1", "sentence2"],
  ["4.2",   "sentence3"]
]
No markdown, no explanation.

Here is the test case format you should assume
when deciding which RFC constraints are
testable: 

=== TEST CASE FORMAT START ===
< Test Case Format in JSON>
=== TEST CASE FORMAT END ===

Now, here is a section of the RFC. Extract
input-related constraints that can be tested 
with this test case format. Each constraint 
must be returned as a 2-element JSON array: 
["<section_number>", "<constraint_sentence>"].
Return ONLY the JSON array as specified in 
the system prompt.

=== RFC SECTION START ===
<RFC Section Text>
=== RFC SECTION END ===

9.2 Test Generation Prompt↩︎

System Prompt:

You are an assistant that generates 
*extremal test cases* from RFC constraints
for a specific testing framework.

This prompt is used to generate extremal tests from each constraint identified earlier in the pipeline.

### Inputs

You will receive:
- The test case format (JSON schema or 
  example object).
- A list of RFC constraints (sentences).

* A constraint is a sentence that describes
  a rule on inputs that the test framework
  can exercise (commands, arguments, states,
  etc.).

### Task

Your job is to generate **extremal tests** 
for these constraints.

Definition of extremal tests:
- Tests at the boundary conditions between 
  valid and invalid.
- "Almost valid": barely violates a constraint
  (e.g., one character too many, one invalid
  character).
- "Almost invalid": barely satisfies a
  constraint (e.g., minimum required length,
  edge of allowed range).
- Test the precise point where valid becomes
  invalid.
- Try to generate multiple extremal tests for
  each constraint, to cover all corner cases.
- Include both positive tests (barely valid)
  and negative tests (barely invalid).
- Consider interactions between components
  when relevant.
- Focus on generating tests that might result
  in crashes/divergent behavior of serious
  consequence for security and reliability.

Requirements for the output:
- Return ONLY a JSON array (no prose).
- Each element is one test case object that
  follows the test case format.
- Use ONLY the fields defined in the test
  format (no extra keys).
- Every test object MUST include a "constraint"
  field set to exactly one of the provided 
  constraint sentences (verbatim, nochanges).
- Every test object MUST include a "test_id"
  field (you may choose any unique string or
  number within this batch; uniqueness across
  batches will be handled later).

Be explicit and systematic in exploring 
boundary conditions, but keep the output
strictly as JSON.

The user prompt for this step includes more variables. As an example, we provide the full user prompt for batch_size = 2, where CornerCase asks the LLM to generate tests for constraints C1, C2.

Here is the test case format you must follow:

=== TEST CASE FORMAT START ===
<test input format>
=== TEST CASE FORMAT END ===

=== REFERENCED RFC SECTIONS (for context) ===
Section X: RFC section text
Section Y: RFC section text
...
=== END REFERENCED SECTIONS ===

Here is a batch of RFC constraints. Each line
has a section id, and the exact constraint 
sentence. Use these constraint sentences 
exactly (do NOT edit them) in the "constraint"
field of your tests. You must generate 
multiple extremal tests for each constraint, 
covering positive and negative edges.

###Constraints:
C1: [1.1] 
C2: [1.1] Constraint 2 text (references 
    Section X, Y)

Now generate extremal test cases for these
constraints.
- Output ONLY a JSON array of test objects.
- Each test object must:
  * Follow the test format fields.
  * Have a "constraint" field equal to exactly
    one of the sentences above.
  * Have a "test_id" field unique within this
    batch.
Do not output any explanation or text outside
the JSON array.

9.3 Result Analysis Prompt↩︎

This prompt lets the LLM assign a confidence score and analysis text for whether the output of selected tests show real RFC violations and bugs.

System Prompt:

### Role
You are an assistant helping to triage
differences between {protocol_name}
implementations.

### Inputs
You will be given a batch of (or a single)
test results where multiple {protocol_name}
implementations produced different responses
for the same test case (zone + query).

### Description of the fields in test results
The test results for each test case are of
the form:

<Test Output Format in JSON>

### Task 
Your job for EACH test in the batch (or 
the single test) is to:
1. Reason about the differences between the
implementations' outputs, considering:
   - Is one or more implementation likely
     violating the RFC (a real bug)?
   - Could the difference be due to acceptable
     implementation-specific behavior?
   - Could the difference plausibly be explained
     or fixed by configuration (e.g., security
     settings, extensions enabled/disabled,
     strictness toggles)?
2. Write a short, clear comment that summarizes
your judgment, referencing:
   - which servers look suspicious,
   - whether this is likely a real bug vs.
   configuration/behavioral choice,
   - any caveats or uncertainty you have.
3. Assign a confidence score from 0 to 10 that
represents how confident you are that
   this represents a REAL BUG (i.e., at least one
   implementation is non-compliant with the RFC):
   - 0 = very likely NOT a bug (probably
     config/expected behavior),
   - 10 = almost certainly a real bug.

### Output Format
Output format for each test:
- You MUST output an array of objects, one
  object per test, of the form:

  {{
    "test_id": <same integer test_id as input>,
    "comment": "<your explanation>",
    "confidence": <integer 0-10>
  }}

Constraints:
- Return ONLY a JSON array (no prose,
  no markdown).
- Do NOT omit any test from the batch; every
  input test must have exactly one output object.
- Do NOT invent test_ids; they must match the
  ones you received.

User Prompt

### Test Results Batch (batch size {batch_size}):

Here is a batch of test results where different
implementations returned different responses.
For EACH test in this batch, you must output
an analysis object as described in the system
prompt.

### Tests
<One or more Test Output JSON>

### Start Analysis:  
Now, for this batch, return ONLY a JSON array
of analysis objects of the form:

{ "test_id": <int>, 
"comment": "<text>",
"confidence": <int 0-10> }.

Make sure every test above has exactly one
corresponding analysis object.

10 Implementation Matrix↩︎

References↩︎