1 Introduction↩︎

In the era of distributed and agentic systems where cyber threats evolve at unprecedented speed, nidss serve as the first line of defense, continuously monitoring network traffic to detect malicious activities, thus being the cornerstone of modern network defense [1]–[4]. Open-source solutions like Snort¹ and Suricata² have established as industry standards, distinguished by their robust packet processing capabilities, extensive rule communities, and multi-threaded architecture. These systems rely on signature-based detection engines, where network traffic is analyzed against databases of predefined rules. Each rule comprises a header specifying protocol, IP addresses and ports, alongside an option section encoding detection logic. While this engine delivers high-fidelity detection of known threats, its efficacy remains contingent upon the quality, timeliness, and comprehensive coverage of deployed rulesets. These qualities, in turn, rely on significant human effort to create and maintain such rulesets [5], [6]. This is further exacerbated by the escalating complexity of contemporary threat landscapes, including polymorphic malware, zero-day exploits, and rapidly evolving attack infrastructure. This complexity has transformed ruleset maintenance into a critical operational bottleneck. As network traffic volumes continue their exponential growth, manual rule engineering emerges not merely as labor-intensive, but fundamentally unscalable [2], [7]. This challenge is well-documented as Security Operation Center (SOC) fatigue, wherein security analysts confront alert fatigue from high false positive rates, syntactic complexity of rule languages, and chronic personnel shortages [2], [8].

Traditionally, nids rule generation constituted a highly specialized task: it demanded deep protocol knowledge and attack semantics, as well as experience in rule syntax, since even trivial violations render rules non-functional. Limited expert bandwidth has created persistent vulnerability windows, as organizations struggle to maintain parity with threat evolution. To address this limitation, the research community has pursued two evolutionary trajectories. First, early efforts leveraged traditional ml techniques to extract signatures from labeled traffic datasets, framing rule generation as supervised pattern recognition [7], [9]–[12]. More recently, the advent of llms has catalyzed a paradigm shift, with studies exploring diverse applications including data fine-tuning for rule engineering [13], synthetic data generation for anomaly detection [14], enhanced rule explainability [15], and direct rule generation from vulnerability contexts [16]. Despite these technical advances, significant gaps remain in understanding how llms can be effectively integrated into operational nids workflows. Although automated systems exhibit syntactic correctness and promising quantitative performance on benchmark datasets, their practical utility as perceived and validated by domain experts has yet to be empirically assessed and, to the best of the authors’ knowledge, no prior work has examined security professionals’ trust in llm-generated rules or how they interact with such outputs.

To fill this gap in the literature, this paper proposes a human-centered investigation of llm-based nids rule generation. We systematically pursue the following research questions:

To answer these questions, we adopt a four-phase methodology. First, we conduct a comprehensive literature review of llm-nids intersections, formalizing prevalent system architectures and identifying common inputs. Second, we validate the system components and inputs through a pre-study with three domain experts. Thirdly, we generate a set of IDS rules using the formalized LLM-based agent system which components are grounded on the outcome of the second phase. Finally, we execute a rigorous two-fold evaluation of the generated rule, including a statistical analysis of the generated rules and a user study with security practitioners to investigate

This way, this paper contributes:

• a formalization of existing llm-based rule engineering functionalities grounded in literature and expert pre-study;

• a first empirical characterization of expert trust in llm-generated nids rules, revealing critical gaps between syntactic validity and semantic deployability;

• novel insights into expert-llm interaction patterns, cognitive load under network complexity, and usability barriers.

This paper is structured as follows: 2 details the methodology, including system formalization (3), user study (4), and evaluation (5), along with participant recruitment; related work is in 6; conclusions are in 7.

In this paper, we focus on nids, hereafter referred to as IDS for brevity. Additionally, we distinguish between rule generation – the specific task of producing signature syntax – and rule engineering—the holistic process of expert review, modification, and validation. In the remainder of this work, these terms are applied according to the specific technical task being discussed. Lastly, we refer to the involved individuals as participants in the context of our experimental results, though they represent professional security experts and rule engineers.

2 Methodology↩︎

2.1 Process Overview↩︎

2.1.1 System Formalization (3)↩︎

As a first step of the methodology, we formalize a llm-based nids rule generation system. Formalizing the system is necessary to ensure a consistent and reproducible basis for evaluation. It provides a well-defined interface, workflow, and behavior, enabling controlled assessment of expert judgments on rule correctness and deployability (RQ1), systematic analysis of interaction patterns (RQ2), and rigorous usability evaluation (RQ3). This formalization reduces confounding factors and strengthens the validity and interpretability of the study findings.

To achieve a rigorous system formalization, we conducted a comprehensive literature review and a preliminary survey to refine our understanding of the system and study design. Initially, we provide a deep understanding of the core components of an llm-based rule generation system (RQ4). Consequently, we validate the essential system inputs through the expertise of three professional participants via a pre-study survey. Such a validation encompasses the intricacies of rule anatomy, automation, human judgment, and the inherent challenges of the rule generation process. With both the literature review and the insights from the pre-study survey, we formalize the system to be tested via a human-centric study.

2.1.2 User Study (4)↩︎

We design a user study aimed at assessing both the functional impact of the proposed system and usability, the quality of the llm-generated rules, while also exploring the extent to which LLMs can support IDS rule engineering in practice. The study was structured to answer the research questions of this paper, mainly focusing on RQ1, RQ2 and RQ3. Participants are asked to complete a set of representative tasks involving IDS rule validation, creation and modification, including the evaluation of LLM-generated explanations to IDS rules. These tasks were designed to reflect realistic scenarios derived from the pre-study, and employed a combination of quantitative and qualitative measures, allowing to assess both performance improvements and the practical usefulness of LLM support. Additionally, participants provided open feedback through statement-based and open-ended questionnaires, enabling a comprehensive evaluation of both performance and user experience, as well as deeper insights into when and how LLM assistance is perceived as beneficial or limiting.

2.1.3 Evaluation (5)↩︎

Using the design choices elaborated previously, we contacted 137 participants to complete the user study via an online survey platform. We received 17 responses and used a subset of 10 participants given their elevated expertise for the results analysis. The evaluation is two-fold, including:

2.2 Participant Recruitment↩︎

To ensure the user study validity, we employed a multi-pronged recruitment strategy designed to engage high-level experts in IDS, SOC and rule engineering. First, we utilized purposive sampling by directly contacting established professionals within the authors’ professional networks and academic circles. This was supplemented by a targeted literature-based approach, where we reached out to the authors of works in the field of automated IDS generation (mentioned in 3.1 and elaborated in 6), ensuring that the study included individuals with a deep theoretical and practical understanding of the domain. To broaden our reach beyond immediate circles, we performed systematic discovery using two distinct methods:

Appendix 10 reports specific details of the recruitment phase. This hybrid approach allowed us to assemble a diverse cohort of experts (137), spanning both academia and industry SOC environments, thereby capturing a comprehensive range of perspectives.

3 System Formalization↩︎

3.1 Literature Review↩︎

Via the literature review we collect insights on how the current research integrate LLMs into the IDS rule generation process, with a particular focus on system definition, inputs, validation and inclusion of human-in-the-loop. We reviewed 11 primary works (detailed in 6 and compared in Appendix 11) tailored to automatic llm-based IDS rule generation systems. Our analysis revealed that the standard pipeline typically includes an input layer, a core llm agent featuring specialized modules for pipeline-specific tasks, and diverse prompting techniques. A subset of these works [16], [17] integrates post-generation components, such as syntax checkers or human-based feedback mechanisms.

The data inputs utilized across these systems vary significantly, reflecting different perspectives on what constitutes relevant information for rule generation. The most prevalent inputs are vulnerability specifications [13], [18], poc exploits [13], [19], and network traffic captures (PCAPs) [20], [21]. Specialized frameworks also leverage honeypot captures [22], malicious payloads [18], or Cyber Threat Intelligence (CTI) and SIEM logs [16], [23].

Regarding the llms landscape, the field is dominated by GPT-based models (GPT-3.5, GPT-4o, GPT-4o-mini) [24] and the Llama family [25], [26]. Other models utilized include Claude [27], Gemini [28], Falcon [29], Mistral [30], Qwen [31], Granite [32], Phi4 [33], and DeepSeek-V3 [34], [35], while one approach focuses on fine-tuning BERT-based models rather than using general-purpose LLMs [22].

Role Prompting is the most prevalent technique, used to define the LLM’s persona during rule generation. cot is frequently applied to handle complex reasoning tasks [18], [23]. Additional methods include Zero-shot/Few-shot prompting [17] and the use of instructional templates [13].

The majority of the analyzed systems do not include a dedicated syntax checker or corrector. Among those that do, Suricata parsing is the standard for validation [13], [17]. Correction is typically handled through customized correctors, LLM-based re-prompting for error fixes, or grammar checks specific to SIEM requirements.

Lastly, there is a significant lack of human-in-the-loop approaches, as ten out of the eleven papers report no human feedback mechanism. Only one system [16] explicitly incorporates human feedback into its autonomous mining and rule generation process.

3.2 Pre-study Survey↩︎

Inspired by the findings of the literature review, the pre-study survey is designed to gather human perspective on the rule generation, system inputs and rule engineering process. We recruited three participants for this study, selected specifically for their extensive professional experience in rule engineering.

1 details the demographics of the survey respondents. The participants represent diverse professional backgrounds, including the private SOCs, academia (P1 in 2), individuals formerly affiliated with government-based security authorities (P7 in 2) and industry-based experienced participant (P11). While the respondents’ primary expertise lies with the Suricata IDS engine, they also reported proficiency with Snort, Zeek, and various proprietary IDS solutions.

3.2.1 Rule Generation Considerations↩︎

When asked about the typical initialization of the IDS rule-writing process, all participants emphasized the need of characterizing the attack surface and define the specific detection objective. Furthermore, respondents noted that the strategic placement of the IDS within the network architecture is a critical factor. One participant specifically highlighted that the directionality of network traffic must be accurately reflected within the rule logic to ensure operational efficacy, which we later consider during the rule quality evaluation in 5.2.2.

3.2.2 System Inputs↩︎

When evaluating the information required to draft effective rules, there was a unanimous consensus on the importance of cve descriptions. The majority of participants (two out of three) identified network topology, device-specific cve inventories, and pocs of known vulnerabilities as essential inputs for the generation process.

3.2.3 Rule Engineering Process↩︎

A standard Snort or Suricata rule structure consists of a rule header and subsequent rule options. Through a combination of open-ended and multiple-choice questions, we gathered insights into header specificity (e.g., the explicit definition of IP addresses and ports), the importance of various option fields, and the potential for automation. Practitioners prioritize maximum specificity in rule headers to increase detection accuracy, typically opting for specific IPs, ports, or network variables (e.g., home_net) whenever the attack details allow. They transition to generic values (e.g., any) primarily when targeting broad behavioral patterns or when services run on non-standard, user-configured ports. Instead, when asked about the relevance of options fields, practitioners consistently identify content as the most critical rule option, valuing its flexibility in matching specific malicious patterns and customizing detection. Options that provide context and operational stability – such as msg for alert interpretation and flow for managing computational overhead – are also highly prioritized. Furthermore, technical precision options like payload positioning (e.g., offset, depth) and protocol-specific fields (e.g., http.uri, dns.query) are essential for refining detection and minimizing false positives. Defining when and how to use such fields represents a complex task, as we show in 5.1.2. Practitioners see automation as a tool to generate rule skeletons, compile metadata, and suggest headers from attack keywords.

3.3 Formalization↩︎

Based on the findings from our literature review and the pre-study survey, we formalized the system architecture illustrated in 1. The system comprises two primary components: an rule engineering agent, which leverages multi-source input data and specialized engines to generate syntactically valid IDS rules, and the human loop, which facilitates adaptive prompting to the LLM engine for iterative rule refinement.

The system ingest data from a network inventory, which provides critical context regarding device types, active services, and topological relationships. Furthermore, the rule engineering agent is integrated with a vulnerability inventory—comprising detailed vulnerability specifications mapped to specific network devices retrieved from existing benchmark [36] and the NIST catalogs (CVE³, CPE⁴). Finally, where available, the agent incorporates poc exploits associated with these vulnerabilities to ground the IDS rule generation process in realistic attack vectors.

The internal workflow of the rule engineering agent begins by preprocessing the aforementioned inputs into structured JSON templates to ensure consistent prompt engineering. A dedicated poc finder tool identifies and associates relevant exploit code with the identified vulnerabilities. This contextual data is then provided to the LLM engine using a combination of role-based and few-shot prompting techniques. The LLM engine itself is implemented via the Ollama framework [37], hosting a variety of open-source models for comparative benchmarking. To ensure technical viability, the raw IDS rules generated by the LLM undergo a multi-stage validation process:

To enrich existing functionalities of the system, the final stage incorporates a human-in-the-loop verification mechanism. This module allows domain experts to perform proactive modifications via adaptive prompting. By interacting directly with the LLM engine, the user can mitigate hallucinations and technical inaccuracies, ensuring that the outputted rules are not only syntactically sound but also semantically effective. The system ultimately produces deployment-ready IDS rules.

The formalized system is designed to encapsulate the common features of state-of-the-art frameworks and is not intended to supersede existing specialized tools. While a human-in-the-loop component is absent in the majority of contemporary literature, it constitutes the primary focus of this work. Given that the central objective of this paper is a human-centric user study, we explicitly include the human loop component enabling participants to interact with the formalized system and provide feedback on the generated rules.

As established in 3.1, system inputs vary significantly across existing frameworks. Our system formalization strategically prioritizes experts’ observations from the pre-study survey (see 3.2), including vulnerability specifications and poc exploits. This selection is designed to maximize the system’s utility in real-world environments where network traffic packet captures (PCAPs) may be unavailable or suboptimal, specifically favoring our approach for the following reasons:

To preserve essential network context without the weight of raw traffic, we utilize network topology information to map relationships between devices and services. This representation provides a more refined and compact alternative for llm processing while maintaining high informational density. Furthermore, the synthesis of poc data with topological inventories yields a more comprehensive technical understanding of an exploit’s mechanics, offering greater depth for rule engineering than the initial connection data typically extracted from traffic captures.

Finally, the selection of the Ollama framework for the LLM engine was driven by several strategic considerations. First, we prioritized a local implementation over API-based services to facilitate the benchmarking of open-source models, thereby ensuring research reproducibility and eliminating the financial overhead of proprietary models. Second, similar to other privacy-sensitive domains, IDS rule engineering is ideally performed using local or fine-tuned models [22], [40] to prevent the leakage of sensitive infrastructure data. Lastly, utilizing local models prevents the effect where provided prompts are used for subsequent model training, thus eliminating potential biases in our evaluation.

4 User Study↩︎

4.1 Study Design↩︎

The user study is structured into three primary phases:

4.1.1 Ethics, Consent, and Demographics↩︎

The study was conducted in strict accordance with the ethical guidelines of the lead authors’ institution and received formal approval from the Human Research Ethics Committee. All participants are provided with informed consent detailing the study objectives, data anonymization procedures, and their right to withdraw. To safeguard professional standing, all qualitative feedback and metadata were pseudonymized and stored on secure institutional servers. To contextualize the results, we collected demographic data and professional background information. This allowed for a stratified analysis based on expertise, work environment, and familiarity with IDS rule engineering. Key metrics included primary role, organization size, years of experience, and self-assessed proficiency with complex rule structures. Further information on participant demographics is included in 14.1.

4.1.2 Rule Evaluation and Cognitive Load Analysis↩︎

The second phase assesses the quality of LLM-generated rules. While a rule may be syntactically valid, the quality evaluation ensures it accurately encodes detection policies and effectively discriminates between malicious and benign traffic. This phase pursues the following objectives:

Candidate rules were sampled from the highest-performing models (see 5.1.1), filtered from hallucinations and bad explanations. To mitigate selection bias, participants performed evaluations blindly through a structured interface.

The evaluation interface presented the Snort rule, the corresponding network context (including topology, assigned CVEs, PoCs and network details), and a structured response form. The central evaluation metric is a binary adoption decision (Yes/No), reflecting whether the rule is considered correct by a user or not. Here, the concept of correctness refers to the possibility of adopting the rule in the presented scenario. Following each decision, participants reported their confidence on a 5-point Likert scale (1: uncertain, 5: certain) to measure the calibration between perceived and actual correctness. Additionally, the utility of LLM explanations was evaluated based on their contribution to rule comprehension by answering the question Does the explanation provided enhance the comprehension of the IDS rules? with answers from “not at all” to “extremely”. Finally, for rejected rules, participants estimated the corrective effort required on a 5-point scale.

As confirmed in the pre-study survey, analysts must not interpret rules in isolation, but rather in relation to network topology, asset roles, and the distribution of vulnerabilities. Such contextual richness introduces varying degrees of cognitive demand, which we model in our experimental design. To simulate real-world conditions, we modeled three network scenarios with increasing structural complexity. These scenarios—comprising 5, 10, and 18 nodes with corresponding cve densities—were derived from established benchmarks [36]. The smallest scenario models a simple, minimally segmented environment, while the intermediate and largest scenarios represent progressively more complex enterprise and multi-segment infrastructures with broader services and vulnerabilities. To maintain tractability, we limited each node to at most two CVEs, avoiding excessive combinatorial complexity. Appendix 14.4 depicts more details on the cognitive load analysis.

4.1.3 Interactive System Evaluation↩︎

The final phase evaluates the formalized system through direct user interaction. After a standardized tutorial, participants used a workflow-oriented interface (see 2) to perform iterative rule engineering. The system integrates selected llms and support network visualization, cve inspection, rule validation and chat-based interface. Such a setup enables assessment of system usability, user interaction with LLM agents, and the readiness of rules for real-world adoption.

The tasks enabled on the system mirror realistic SOC workflows. While the system supports free exploration for some tasks (e.g., syntax checking, rule writing, network exploration, PoC and CVE finder, and Snort runs), certain resource-intensive tasks (e.g., generating and modifying complex rules) were provided as curated examples. Specifically, we provided examples to generate new rules, explain generated rules, and modify the rules to update IP addresses and ports or add specific fields of the IDS rule. These tasks are indeed the most performed changes during rule engineering according to existing literature [2] and pre-study results. This approach ensures participants can evaluate high-parameter models (e.g., DeepSeek-R1 70B) without the latency associated with real-time generation. The free usage allowed participants to explore the interface using their own strategies, helping identify usability issues, interaction patterns, and perceived utility.

Following the interaction, participants completed the System Usability Scale (SUS) [41]. This quantitative measure was supplemented by structured 5-point scales assessing efficiency, effectiveness, and learnability [42], alongside open-ended questions to capture subjective user experiences and technical bottlenecks. The combination of quantitative and qualitative data provides a comprehensive evaluation of the system’s usability, highlighting strengths, weaknesses, and areas for improvement from the perspective of professional IDS rule engineers (see 5.2.3).

4.2 Online Platform↩︎

To accommodate the demands of this study while ensuring comfort for the participants, we developed a centralized web-based platform to host the study. This eliminated the need for local software installation or complex environment configurations (e.g., Docker). The platform was built using Python and Ngrok for backend functions, and a modern HTML/CSS/JavaScript frontend, ensuring a seamless transition between the survey components and the interactive system interface. We hosted the platform frontend using PythonAnywhere⁵, while the backend and system run on private server machines.

5 Evaluation↩︎

In this section we evaluate the LLM-based rule engineering agent over six different network scenarios and 27 different llms engines. More in detail, we consider the following network scenarios:

For the CIC-IDS network, we construct a total of four different combinations, including one or two CVEs for each of the devices in the network, and either if the CVEs match a corresponding PoC or not. The obtained four combinations are:

For the detailed description of the generation process used for the two synthetic networks, we refer the reader to 12. We consider these six network scenarios – two synthetics and four CIC-IDS – to account for the different levels of complexity that may arise when experts need to write IDS rules, thus mimicking the increase in cognitive load that they face in practice. Indeed, the combination of network size and vulnerabilities introduces interaction effects (e.g., exploit chains) that exponentially increase the analytical complexity beyond a simple additive effect. The assessment scenarios thus provide a controlled and representative environment for the proposed study. For each combination of network scenario and LLM engine, we run our agent 30 times for statistical significance⁶, aiming at generating as many IDS rules as possible. To keep the computational cost under control, the IDS rule generation process is terminated if the agent fails to produce any syntactically correct rule within an 30-hour time window. All rules were used for our statistical analysis in 5.1, while only the rules generated with the statistically best performing LLM engine – i.e., DeepSeek-R1 70B (see 13) – were used for our human-centered evaluation in 5.2.

5.1 Statistical Analysis↩︎

We first analyze the rules generated by the LLM-based agent from a statistical perspective, analyzing their syntax validity, semantic content and their proneness to hallucinations.

5.1.1 Syntax Evaluation↩︎

We first filter rules that violate basic Snort structure, yielding 2871 Snort-looking candidates. Out of the LLM engines considered, models with fewer than 4B parameters failed to generate any valid syntax (see ¿tbl:tab:generated95rules?), highlighting the complexity of automatic NIDS rule generation. We use a dashed line in ¿tbl:tab:generated95rules? to distinguish between models: those above successfully generated rules across topology, while those below failed.

 Finding : Small LLMs (\(\leq\)​4B parameters) are ineffective for IDS rule generation.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

Subsequently, we compile each rule in Snort, testing how many of them result in compilation issues. 3 (a) shows the fraction of syntactically valid rules generated by each LLM engine for CI\(_{1}\)—additional results across the other scenarios are reported in Appendix 15.5 and highlight similar findings. While smaller models like Gemma 2 9B and Llama 3.2 1B struggle, larger models achieve significant proficiency: Llama 3.1 70B reached 77% accuracy, while DeepSeek-R1 70B and Llama 3.3 70B reached 85% and 87% accuracy, respectively. Furthermore, a Pearson correlation of 0.5991 confirms a positive relation between model size and syntactic validity. Therefore, these results highlight how larger LLM engines have the necessary understanding of the NIDS grammar to generate usable rules.

 Finding : Large-scale models (\(\geq\)​70B) demonstrate high syntactic proficiency, with success rates up to 90%.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

To determine if this proficiency is LLM-intrinsic or correction-dependent – and thus connected to the agent’s tool usage –, we track rules that were generated immediately correct versus those requiring a feedback loop to be corrected. 3 (b) shows such fractions for each LLM engine for the CI\(_{1}\) scenario, while additional results across the other scenarios are reported in Appendix 15.5 with similar findings. 54.85% of valid rules require the corrector loop and this necessity persists regardless of model family. For example, only 21.74% of the syntactically best performing model (i.e., Llama-3.3 70B) rules were valid on the first attempt (see 3 (b)).

 Finding : Syntax-checking and correction modules are essential. Even the best LLMs struggle to produce compliant signatures without feedback.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

To further analyze the importance of the rule checking and correction feedback loop, we measure the number of correction loop iterations that each generated rule requires before being correctly output. As shown in 3 (c), even the best performing LLM engines require a non-negligible amount of rule corrections to generate valid rules, with results being similar across engines. On average, DeepSeek-R1 70B requires 0.76 corrections (increasing to 1.78 for rules requiring at least one fix). These results highlight a bimodal distribution where rules are either immediately valid or require significant effort to be fixed, with some rules requiring up to four corrections before being correctly output. Furthermore, we also note how this number represents a lower bound, as we stop attempting to correct a rule after five consecutive failed updates.

 Finding : Generated rules tend to be either immediately valid or cumbersome to fix.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

Finally, we analyze the impact that the inclusion of Proof-of-Concept (PoC) data has on rule generation. To this end, we compute the fraction of valid rules generated whenever the PoCs are (not) available across each LLM engine (see 3 (d)). The inclusion of poc information yields a moderate enhancement in output quality as the mean syntactic accuracy increases from 67.23% to 74.83% when PoC context is provided. Moreover, we observe a moderate positive correlation (Pearson = 0.6619, Spearman = 0.7160) indicating that engines performing well without PoCs also tend to perform well with PoCs. Lastly, we note that larger models, which can better handle long and complex contexts, benefit from PoC information, whereas models with limited context capacity are negatively affected by the substantial increase in input length, which rises on average from approximately 7,500 (without PoCs) to more than 58,000 (without PoCs) tokens per generated rule.

 Finding : PoCs improve rule generation quality only when the LLM engine can handle large context windows.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

To answer RQ4, DeepSeek-R1 70B and Llama3.1 70B emerge as superior engines (85-87% syntactic success), augmented by syntax checkers, PoC exploits, and vulnerability inventories; small models prove ineffective. As a complementary analysis, the fraction of hallucinated rules is in Appendix 15.1.

5.1.2 Rule Quality Analysis↩︎

We statistically analyze the distribution of header and body fields to asses the quality of generated rules, referencing the criteria established in our pre-study (see 3.2). 4 (a) presents the field distribution over the generated rule headers for the CI\(_{1}\) scenario aggregated over all engines. Most headers (87.87%) default to generic alerting actions and use the any keyword for source IPs and ports to map general threats for which the source is unknown. Conversely, the destination port is the most specific field, with 71.77% of rules correctly targeting ports 22, 80, 443, or 445. This specificity highlights the agent’s ability to reason about the most common network protocols – such as HTTP at port 80 or SSH at port 22 –, and the LLM engine’s tendency to generate rules focusing on where the traffic is landing. Therefore, the agent aims to match specific characteristics of an attack with the network protocol it expects to see—e.g., generating a rule against an ownCloud remote improper authentication attempt as defined in CVE-2023-49105, requires grounding the attack on HTTPS and thus on port 443. Meanwhile, it seems that the agent avoids enforcing specific constraints on the target destination – with 57% of destination IPs being unspecified/scenario-dependent – or blacklisting some potential source IP threats. These headers patterns mirror the ones in the Emerging Threats ruleset⁷ available to Snort and Suricata practitioners – where most headers are alert-driven and heavily rely on any and placeholders like external_net, thus suggesting that open-source LLMs are trained on this ruleset and back our intuition on memorization made in Appendix 15.1. Further analysis of the rule quality is depicted in Appendix 15.7.

 Finding : While 54.66% of the generated rules rely on general variables such as any, models demonstrate an inherent ability to reason about specific network protocol ports.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

While we will rely on the user study of 5.2 as the definitive test of rule quality, we here analyze the generated rule bodies to provide preliminary findings on their specificity and reliability. We analyze the average length of rule bodies across LLM engines in 4 (b) where results show stability across models, ranging from three to five fields. This distribution is quite far from the average length that characterize rule bodies of commonly available rule databases, where rules adopt several fields like payload positioning (e.g., offset, depth) and protocol-specific fields (e.g., http.uri, dns.query) to ensure rule specificity (as highlighted by the pre-study insights of 3.2.3). Therefore, the generated rules seem to lack the level of specificity that SOC experts require [44], hinting at LLM not being able to process the fine-grained information required to write very specific rules. Similarly, LLMs do not seem to posses the reasoning capability to compose multiple fields of the rule body together to improve the rule detection scope. These intuitions were backed during our user study, where human experts mentioned the lack of specificity of prompted rules and their worry towards possible false alerts (see 5.2).

Additionally, in 4, we visualize the distribution of the fields used in the bodies of rules generated using two of the best-performing LLM engines, DeepSeek-R1 70B (4 (c)) and Llama-3.3 70B (4 (d)). The most common parameters are the message and content fields, which are almost always used—the former is used to provide a description of the generated rule, while the latter provides the actual detection framework, aiming to match the specific traffic content. The results show that, while large models sometimes enable the agent to generate rules containing specific fields for detection, such as distance and within, the vast majority of generated rules rely on pure content matching. Once again, these intuitions were backed during our user study (see 5.2). These results highlight the intrinsic complexity of the IDS rule engineering task, where rule precision requires complex reasoning abilities about several aspects of the attack behavior across several parameters that current LLMs lack.

 Finding : Most generated rules rely on pure content matching, with specific fields such as distance and within – valuable fields for rule specificity – rarely used.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

To answer RQ1, these findings reveal that while large-scale LLMs achieve up to 90% syntactic validity, NIDS experts deem only 37.5% of generated rules semantically correct and deployable, primarily due to insufficient specificity, heavy reliance on content matching, and logic hallucinations.

5.2 Human Evaluation↩︎

Here we examine the outcomes of presenting a group of human expert SOC analysts with the rules produced by the top-performing agent – namely, DeepSeek-R1 70B – to evaluate their quality. We refer the reader to Appendix 13 for the detailed description of how DeepSeek-R1 70B was selected as the best performing engine across the used ones.

5.2.1 Participants Recruitment↩︎

The participant recruitment process was conducted in multiple phases to maximize the response rate. Initially, we reached out to 137 potential participants, establishing a 14-day window for survey completion. Following the initial outreach, a reminder was sent to all candidates after 11 days to encourage participation before the closing date. By the primary deadline, 14 responses had been recorded. Two days post-deadline, a final solicitation was directed toward the remaining non-respondents to capture any late submissions. Ultimately, we secured a total of 10 completed surveys, while 2 participants completed the survey only partially and 4 participants filled only demographics information. The subsequent analysis is based on the data provided by the respondents who partially or fully complete the study. 2 shows the participants demographic information and background.

5.2.2 Rule Quality Evaluation↩︎

To evaluate the technical fidelity of the generated rules, experts audited specific signatures across four distinct scenarios of varying complexity—i.e., topology size, number of cves per device, and number of pocs. The results, summarized in 5 (a), show that the experts maintained a high degree of critical rigor. When evaluating SS, despite the 70% of “Wrong” rate (indicating the expert identified a flaw or a need for modification), the average confidence was at its peak (\(\mu = 4.0\)). This suggests that the flaws in the llm-generated rules in simpler contexts are transparent and easily caught by human operators. For example, P6 highlights that “JumpServer runs on port 2222 and not 22”, while P4 identifies that a generated rule “detects SMB, while the message says it’s about SSH brute force”, thus highlighting contextual inconsistencies of the generated rules. This feedback also points to additional hallucinations [45], [46] that are harder to measure than what was done in Appendix 15.1, also highlighting the complexity of the rule validation task. Additional insights into the perceived quality of LLM-generated explanations, including participants’ generally positive feedback, are discussed in 15.2.

 Finding : Over small-scale scenarios, human users can easily pinpoint issues with generated rules independently of their level of experience.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

P1 also highlights the low specificity of the generated rule, mentioning how “the payload matching has low specificity” and that “there is no check to test that the traffic to be matched is indeed SMB v1, no check for the SMB message type and no check for offset/position at which the bytes to be tested should be at. The condition is so generic that it could trigger in many cases of file transfers.” Backing our intuition of Finding [finding:specificity95stats], rule specificity represents a fundamental source of complaint for the human experts across the considered scenarios, with P10 mentioning that “the rule looks for a specific byte sequence in the SMB traffic without containing an additional context or conditions that could help refine the detection of malicious activity”, P7 stating “the rule seems to be a bit too general” and questioning if “100 alerts within the span of 300 seconds are enough to cause a DoS attack?”, and P3 claiming that “the rule looks to be too generic to be useful”, thus backing our intuitive findings of 5.1.2.

 Finding : Human users stress the issue of generating specific rules that avoid triggering over normal traffic.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

As scenarios progressed to more complex topologies (Q3 and Q7 in 5 (a)), we observed a slight dip in both accuracy and confidence, which is consistent with the increased cognitive load required to verify multi-node cve mappings. For example, P8 mentions that they are “not confident if the rule will be useful”, while P5 states that “the rule appears correct, but may trigger false positives when web servers receive more requests from a single host for benign reasons”, thus showing reasonable doubts about their validation of the rule. This uncertainty is likely to be attributed to the complexity of dealing with a large source of information over an increasingly complex topology, which once again highlights the complexity of the task at hand and the usefulness that an automatic assistant system would have in practice. This intuition is backed by the increased correctness (60%) reached in the final scenario (Q8), where confidence stabilizes at \(\mu = 3.3\). Here, a few participants mention how the rules “seems to be correctly written” (P7), “correctly uses detection filter and the direction is also correct” (P6) and that the rule “should work fine” (P3). In this context, human struggle to elaborate the completeness of the provided information and rather focus on sub-context of the scenario or generated rule—e.g., “The content union select is valid” (P10), “I am not sure where the string used in the content match originates from” (P5) and “I don’t think the tracking will work” (P8).

 Finding : On average, human users tend to struggle to analyze in depth the generated rules when the context information grows.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

Interestingly, considering the set of highly expert human users as the participants with self-reported experience of at least 3 years with an IDS engine – i.e., P1, P4, P5, and P8 – the perception of the generated rule quality is harsher. Indeed, these highly expert participants evaluated negatively 11 out of 16 questions of the generated rules with a high average confidence score \(\mu = 4\). This is true for both small scenarios and large contexts. For example, in Q7, P1 states that “while the rule is trying to detect multiple SSH connections, the specific SSH banner is unlikely to be incorrect” and that “UDP port 445 is used for SMB over QUIC, which is encrypted, while SMBv1 traffic would happen over TCP”, thus pointing our an hallucination of the agent which is not capable of managing properly the protocol-level knowledge. Similarly, P4 identifies an issue with the written rule as “it should contain track by to make it work”, thus stressing once again the lack of specificity. Therefore, it seems that skilled human experts can pinpoint issues affecting the generated rules even within large contexts, possibly due to an acquired cognitive bias.

 Finding : Expert proficiency is a critical factor in rule validation: skilled participants successfully navigate large context volumes to identify issues, while less-skilled users struggle to maintain accuracy under the same conditions.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

In the second part of the study, the focus of questions Q2, Q4 and Q10 is quantifying how much the explanations provided by the llm actually helped experts understand the generated IDS rules. The data, depicted in 5 (b) (and 5), indicates a high level of consistency in the perceived value of the explanations. Across all three scenarios (A for the SS topology, B for the SM topology and C for CI), the average confidence/utility score remained steady at approximately 3.0. Specifically, in Q2, 50% of participants rated the explanation utility at a 4 or higher score, suggesting that the textual context significantly lowers the cognitive barrier. However, as the scenarios became more complex in Q4 and Q10, the distribution became even more concentrated in the middle-to-high range. A strong majority of participants (70%) clustered in the “Moderately” to “Very” helpful categories.

Having understood the general level of quality of rules that can be generated using llm engines, we here aim at understanding to what extent human experts would be willing to deploy the automatically engineered rules in practice. More in detail, for each of the generated rules tested in the user study, we ask the user if he would be confident deploying such a rule as is. Consequently, we ask to motivate their choice, aiming to gather some insight into the adjustments required to use the generated rules. Overall, the participants expressed a rather negative sentiment on the adoption of the automatically generated rules, with them answering “No” to the adoption question more than 70% of the times. The results are pretty equally distributed across network scenarios, highlighting how the gap between agentic AI definition and its practical adoption is still large. More in detail, most experts pinpoint some logical issues in the detection process of the rules designed by the LLM-agent which hinder their adoptions, such as “the content match would have to be adjusted to union select” and “the depth would have to be increased” (P5) or “the rule misses the detection logic” (P1), and “the targeted port being wrong” and “the traffic direction being incorrect” (P6). As such, most experts agree on the necessity of high human effort to adopt such rules with P5 mentioning how “essentially, a rule engineer would have to research the vulnerability from the ground up before he can start fixing the rule”, that “the most work would be to identify the correct sequence to match the malicious behavior in the payload” (P1) and that “how to use this rule to detect an attack in practice would require very high effort to figure out” (P4). These results are reflected also on the numerical level of effort expressed by the participants, with most of them voting from "Moderate" to "Very High" effort (\(\mu = 4.25\)). Furthermore, human experts tend to highlight once again the lack of specificity of the engineered rules, causing large amounts of false alerts. In particular, P3 mentions that some rules would “only give false positives” and how rules “would need to include many more checks to be useful”, while P7 highlights that a rule “seems to be way to general” and how it “would cause a lot of noise”.

Lastly, there exist a few instances of generated rules that were positively assessed by human experts, such as P9 mentioning that “it would take low to moderate effort to properly modify and make this rule production-ready, mostly tuning rather than rewriting from scratch”, thus suggesting that a few of the generated rules might be used effectively with some effort, thus providing some hope for future investigation. Appendix 15.6 provides a more detailed analysis of the effort required to correct the LLM-generated rules.

 Finding : Expert evaluation indicates that automatically engineered rules require substantial refinement before practical adoption; only 37.5% of the rules received positive assessments, and even these were met with neutral confidence (\(\mu = 3\)), suggesting significant skepticism regarding their immediate utility.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

5.2.3 SUS-based System Evaluation↩︎

To quantitatively assess the user experience of llm-based rule generation system, we employed the sus-based evaluation [41], a robust and industry-standard metric for perceived usability. Participants rated ten alternating positive and negative statements on a 5-point Likert scale, ranging from Strongly Disagree to Strongly Agree.

The system achieved a mean SUS score of 67 (standard deviation of 11.35) with a 95% confidence interval of [59.96, 74.04], calculated over the raw scores presented in 5 (c) and Appendix 15.3, thus satisfying the requirement to consider the system well usable (above 67) [41], [47]. To better understand the drivers of the system’s perceived quality, we analyzed the results across three primary dimensions: usability (Q1, Q2, Q3, Q6, Q7, Q8 and Q9), learnability (Q4 and Q10), and integration (Q5). The full set of questions is available in Appendix 14.2.

In terms of usability, the responses indicate high degree of perceived usability. Specifically, Q3 (“Ease of use”) received the strongest positive consensus, with 70% of participants providing a rating of 4. This is complemented by the results of Q9 (“Confidence”), where no participant reported a lack of confidence while interacting with the rule generation interface. The low agreement with Q2 (“Complexity”) – where 8 out of 10 participants disagreed that the system was unnecessarily complex – suggests that the underlying LLM-driven system is presented through a sufficiently abstracted and user-friendly interface, which professionals would easily use.

In terms of learnability, the system demonstrated strong learnability markers, which is critical for tools intended for expert security workflows. Q7 (“Learn quickly”) showed a significant positive skew, with 8 out of 10 participants “Agreeing” or “Strongly Agreeing” that most users would adapt to the system rapidly. This is further supported by Q10 (“Pre-learning”), where the majority of users disagreed with the notion that a significant amount of prior learning was required before they could effectively use the platform.

Lastly, the integration results for Q5 (“Well integrated”) were generally positive, while showing more variance than the usability metrics. However, the consistent disagreement with Q6 (“Inconsistency”) suggests that the participants found the system’s behavior predictable, a vital requirement of the reliability of automated IDS rule engineering.

 Finding : Most participants find the LLM-based rule generation agent intuitive and easy to adapt to: 70% rate its usability positively, and 80% agree it is not overly complex. Similarly, 80% believe most users would quickly adapt to it.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

To answer RQ3, the LLM-assisted system exhibits favorable usability, with experts rating it efficient and learnable for drafting/explaining rules, yet emphasizing the necessity of human oversight to mitigate limitations in semantic precision and false positives.

5.2.4 Statement-based Questions Evaluation↩︎

While the previous sus-based evaluation studies the usability of the system, this questionnaire assesses the technical viability and expert trust in an llm-based IDS rule generation system. The raw score distributions are depicted in 15.4. Participants perceptions are evaluated across three dimensions: trust and autonomy, tool utility and operational impact.

The results reveal a clear boundary regarding system autonomy. While participants expressed moderate confidence in the llm’s contextual awareness (Q1, \(\mu = 3.40\)), there is a near-unanimous consensus on the necessity of a human-in-the-loop (Q9, \(\mu = 4.30\)), with over 60% of participants providing the high ratings. This cautious stance is further evidenced by the low trust in a fully autonomous “suggest-and-deploy” mode (Q8, \(\mu = 2.00\)). Notably, 70% of participants rated their trust for full autonomy at 2 or lower.

Participants identified the internal verification and enrichment modules as the most valuable aspects of the system. In particular, the PoC finder (Q5, \(\mu = 3.70\)) and the Snort rule tester (Q6, \(\mu = 3.60\)) result as primary drivers of utility. The high valuation of the interactive generator (Q7, \(\mu = 3.40\)) indicate that experts value the verification loop, mainly for the ability to test and iteratively refine rules within the same interface. The inclusion of vulnerability context – i.e., poc and cve data – is confirmed to significantly improve confidence in the final rules (Q2, \(\mu = 3.40\)).

Regarding the deployment, experts are optimistic about the system’s ability to reduce time-to-deploy (Q3, \(\mu = 3.00\)), although responses remained distributed. Interestingly, when comparing rule quality to traditional scanners (Q10, \(\mu = 3.20\)), the majority of experts (60%) provided a neutral score, indicating that they view llm-based generation as a complementary technology that offers flexibility rather than a complete replacement for legacy logic. Finally, the relative ease of modifying llm-generated rules (Q11, \(\mu = 2.60\)) is rated highly, confirming that the system provides a manageable starting point for human operators.

 Finding : Experts perceive the system as a high-utility force multiplier for rule drafting and verification, but reject full autonomy in favor of a human-in-the-loop workflow grounded in verifiable vulnerability data.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

5.2.5 Open-ended Questions Evaluation↩︎

Lastly, we aimed at identifying the feelings of the human experts towards the usage of LLM-based agent for engineering IDS rules. To this end, we defined a set of open questions that focus on the users’ experience with rule writing, LLMs, and their desires for a rule engineering automated assistant. See Appendix 14.3 for the full list of questions. Here we analyze the feedback gathered from such questions. The participants experienced with rule engineering backed the results obtained during our preliminary study, with several experts mentioning that either

In this context, few of the prompted experts mentioned having used LLMs in the past in their rule-writing process mainly to check the syntax validity of a rule – e.g., P1: “deal with the task of checking the syntax” and P5: “understand syntax features” – or explaining complex rule templates—e.g., “I have used LLMs to explain existing rules” (P5) and “Explaining complex rules” (P8). However, no human expert mentioned being successful in leveraging LLMs to generate a valid and reliable rule from scratch. Thus, we highlight how human experts seem to be aware of the LLM limits when it comes to engineer IDS rules from scratch, as testified by P1 explicitly mentioning that he sometimes used LLMs in the past, but expressly stating how “the detection characteristics were provided by me”.

 Finding : Human experts seem to be aware of the limits of generative AI tools for IDS rule engineering, with most of them using LLMs only sporadically to check syntax validity or provide further insights on already existing rules.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

The above findings are backed by the overall feeling when answering the question “if this tool was integrated into your daily SOC workflow, how would it change your team’s priorities?”. Indeed, while some participants were positive about the system adoption, mentioning how such tool could “make creating rules easier” (P4) or how “the team could spend less time writing and tuning basic IDS rules” (P10), several participants were cautious about the adoption of the full system in an en-to-end rule generation fashion. For example, few experts focused on the fundamental principles that the system provides to understand the context and help the SOC team while not being interested in the complete automatic rule generation with P5 mentioning that “it could be valuable in understanding the network topology and identifying vulnerabilities” but also that they do not “think the LLM module would be used a lot” or how “it could be a nice tool to give priority to the team on what to focus on” (P4) or to “relieve from syntax rule checking” (P1). To this end, P3 response is emblematic “it would make it less fun, because the cognitive task will be mostly automated away”, as it highlights the adversity of SOC experts against completely automated pipelines. Finally, one participant expressed a fairly negative judgment, mentioning that “at the moment, [the system] wouldn’t [be helpful]” (P8).

 Finding : Experts see the possible benefits of introducing assistance tool, but are skeptical of automating end-to-end the rule engineering pipeline.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

To back the above, there is the feedback we received from the participants while answering to the questions

In this context, rather than focusing on tools or components that would help improve the end-to-end automatic engineering of rules, human experts focused on asking for components of the systems that would help the human analyst understanding the context at hand, the actual performance of the automated rules or the impact they might have. For example, to answer [q95missing95info], P5 mentions how “it would be nice to have some sort of CVE prioritization”, while P1 highlights the need to add a tool to measure the “detection performance with sample attack PCAPs”, and P10 asks for the addition of “suggestions of mitigation strategies and impact assessment”. Similarly, in answering [q95teach95llm] P1 mentions “what traffic it can expect to observe and what traffic is not being captured at all”, while P10 suggests “the operational context of devices and services within the network, for example: device criticality and role and the past incidents patterns”, once again highlighting the human perspective on sound and complete understanding of the given context rather than perfectionist rule engineering perspective. This behaviour is likely rooted in human experts having clearly in mind the risks that a completely automatic rule engineering system would bring as they deeply value the human-in-the-loop perspective. For example, when answering the question “who is ‘responsible’ for a false negative if an LLM-generated rule is accepted by an expert but fails in production: the model, the system designer, or the human reviewer?”, most participants agree on suggesting the human reviewer as the bearer of most responsibilities, with P3 stating “all of the above, but in practice the human reviewer” while P10 and P1 mention “the human reviewer bears the most direct responsibility” and “I think the human reviewer would ultimately be responsible”. These answers suggests how human analysts envision a human-in-the-loop partially-automated rule engineering system, where the human supervises the rule engineering process and suggests possible modifications or improvements rather than completely relying on AI tools—thus also backing our system vision of 3. In this context, it is interesting to note P1’s answer, as they also highlight how “over time, the system might influence the way human reviewers approach detection”, implicitly raising concerns about the practical usage of end-to-end automatic rule engineering. P1’s answer seem to suggest how over time the overly reliance on fully automated rule engineering pipelines may render the human analysts more prone to accept generated rules without thorough testing, and thus raising reliability issues.

 Finding : Participants tend to stress the need for carefully designing human-driven systems rather than end-to-end automatic rule engineering perspectives, as they have clearly in mind the importance of the rule engineering task and the human analyst’s responsibilities.

[]{#finding:\thefindingcounter label=“finding:\thefindingcounter”}

To answer RQ2, experts predominantly engage in iterative human-in-the-loop refinement via adaptive prompting and network visualization, correlating higher perceived rule quality with PoC integration and topology context, though cognitive load rises with network complexity, favoring verification over autonomous generation.

6 Related Work↩︎

The automatic generation of IDS rules has garnered significant attention in recent years and recent research suggests that it is a challenging process. Although not all of the works focus specifically on rule engineering, they provide insights on how socs operate in practice. Among them, Teuwen at al. [44] investigate the design of IDS rules within soc, identifying key design principles consolidated through interviews with experienced soc rule designers, while alert alchemy [2] examines the processes involved in the creation, management, and acquisition of IDS rules, successively deeply looking into the evolution of rulesets in SOC production environments by analyzing over 130 thousands of rules and 62 million alerts [48]. Differently, Jansen et al. [49] tackles the practical issues with the creation of graph-based IDS rules as opposed to log-based detection rules via a qualitative user study, while Alahmadi et al. [8] quantified the burden of alert fatigue – 99% of the generated alerts are false positives – via interviews and observations of practitioners. Finally, Singh et al. [50] studied real-world queries of LLMs from SOC analysts to discover that llms are integral part of the soc as cognitive aids.

While these studies investigate the operational realities of rule engineering and nascent llm integration, they leave critical questions unanswered about expert-llm collaboration in the domain of nids rule generation. Our study breaks new ground by revealing to what extent security professionals are willing to interact with, validate, and refine llm-generate rules in realistic scenarios.
With the advent of llms, their use for nids tasks has been increasing. Our work, builds upon these works and provides a better understanding of the practicality of llms for nids rule engineering via the user study. Some of these works introduce an llm-driven framework that automatically generates rules from code, like RuleMaster+ [19] – using poc –, RulePilot [23], and RuleXploit [13] – using exploit samples. In these cases, LLMs perform a deep code analysis and identify features from exploit payloads. Other works leverage LLMs as agents, like Hu et al. [18], who extract threat features from multiple input sources and transform them into actionable signatures, and GRIDAI [21], who focuses on generalizing existing signatures, and LLM4Rule [20], which is a framework for generalization of the rule. Differently, Moreno et al. [17] tackles the LLM-based rule generation in the Industrial Control Systems (ICS), via the integration of a domain-specific knowledge into the LLM prompting process, while Hex2Sign [22] focuses on the aspect of converting raw network telemetry into human-readable IDS signatures, with LI-NIDS [51] generating nids rules from numerical network flow features transforming network flow data into an abstract language that the LLM can process. FALCON [16] proposes the generation of rules from unstructured Cyber Threat Intelligence (CTI) while also providing a validation pipeline, finally RuleLLM [40] prompts the LLMs for characterization of benign and malicious behavior to extracts distinctive features for both classes.

7 Conclusion↩︎

This paper presents a human-centered investigation into the practical viability of llms for nids rule engineering. We formalize a state-of-the-art grounded system and rely on a user study to gather experts feedback on the semantic correctness of LLM-generated rules and their adoptability. Our findings reveal a critical syntax-semantics paradox: while LLMs construct syntactically valid rules, experts deem them only partially deployable due to their lack of specificity, over-reliance on content matching and susceptibility to hallucinations. Furthermore, while the system’s usability was rated favorably, a fundamental skepticism persists among practitioners. Experts currently view LLMs as auxiliary tools, useful for syntax validation and natural language explanation, rather than autonomous generators. This underscores a pivotal requirement for human-in-the-loop architectures that prioritize the analyst’s agency and decision-making workflow over automation. Ultimately, this work serves as a foundational assessment of LLMs in the rule engineering domain. By delineating LLMs limitations in this domain, we provide a roadmap to develop more robust, reliable, and human-aligned rule engineering workflows.

8 Open Science↩︎

The built framework leverages Ollama [37] for local LLM orchestration, following the three-phase workflow illustrated in Figure 1. Initially, the system provides the LLM with structured environmental context, including network topology, CVE identifiers, and node connectivity. Subsequently, the model synthesizes Snort-style IDS rules and corresponding technical justifications. Finally, each rule undergoes an automated verification loop where a syntax validator and the LLM iteratively identify and rectify errors, ensuring rule integrity through a traceable audit log of modifications.

Beyond enabling the replication of our results, this codebase serves as a modular foundation for future research in automated security policy generation. By releasing this framework, we provide a reusable and traceable environment for exploring the integration of LLMs into IDS rule generation automation.

9 Ethical Considerations↩︎

This study involves a human-centered evaluation of LLM-generated IDS rules, and we have taken several measures to ensure it adheres to the highest ethical standards. The recruitment process and the survey instrument were conducted in strict accordance with the ethical guidelines and internal protocols of the lead authors’ institution. Prior to participation, all individuals were provided with comprehensive instructions regarding the nature and content of the study. We obtained explicit informed consent from each participant before they commenced the survey, ensuring they were aware of their right to withdraw at any point. Furthermore, the evaluation was designed to protect participant privacy; the survey did not solicit, collect, or store any sensitive personal information or personally identifiable information (PII). All data used in our analysis is anonymized and treated in aggregate to ensure that no individual respondent can be identified.

10 Recruitment Details↩︎

To identify suitable participants for our study, we adopted a multi-stage recruitment strategy combining prompt-based discovery and structured academic search queries. This approach allowed us to target both academic and industry researchers working at the intersection of cybersecurity, intrusion detection systems, and LLMs.

Prompt 1: "Can you find list of researchers in academia and industry that on their Scholar or Scopus profiles have one of the following keywords or a combination of those: "Cybersecurity", "AI", "LLMs" and "IDS"?"
Prompt 2: "We will soon start a user study on the use of LLMs for Intrusion Detection Systems rule generation. For context, Snort and Suricata tools work based on rules that are usually written and generated by human analysts. In this study, we have tested different LLMs on the task of rule generation given several input informations such as the network topology, CVE list per each node in the topology, services for each node in the topology. Can you search for researchers (both academic and industry) that work in this domain?"

We first leveraged natural language prompts (Listing [lst:recruitment_prompt]) to identify researchers whose profiles explicitly mention relevant keywords such as Cybersecurity, AI, LLMs, and Intrusion Detection Systems. These prompts were designed to capture a broad yet relevant pool of candidates by querying publicly available academic and professional profiles. The second prompt further refined the search by focusing on researchers with demonstrated experience in applying LLMs to IDS rule generation, explicitly referencing tools such as Snort and Suricata and contextual inputs like CVE data and network topology.

Query 1: "intitle:("Large Language Model" OR LLM) AND (IDS OR "Intrusion Detection") AND ("Rule Generation" OR "Signature Generation")"
Query 2: "label:network_security label:LLM"

To complement the prompt-based approach, we conducted structured searches on Google Scholar using carefully designed queries (Listing [lst:recruitment_scholar_queries]). Query 1 targeted publications explicitly mentioning LLMs in the context of intrusion detection and rule or signature generation. Query 2 leveraged Google Scholar labels to identify researchers associated with both network security and LLM-related topics.

11 Rule Engineering Systems Comparison↩︎

3 provides a comparative overview of recent approaches that leverage llm for nids rule generation. Most approaches rely on prompt engineering techniques such as role prompting or chain-of-thought (CoT), while only a few explore zero-shot or few-shot configurations. The input data varies across studies, ranging from structured network traffic (e.g., PCAP files) and vulnerability reports to unstructured cyber threat intelligence (CTI) and exploit descriptions. Regarding model usage, both proprietary and open-source LLMs are widely adopted, including GPT-4 variants, LLaMA-based models, and domain-specific or fine-tuned architectures. However, explicit syntax validation and correction mechanisms remain limited, with only a subset of methods incorporating dedicated parsers or grammar checks tailored to IDS frameworks such as Suricata or SIEM systems. Human involvement is generally minimal, with most systems operating in an automated or semi-automated fashion. Only a few approaches incorporate human feedback loops or adaptive prompt refinement. Overall, the table highlights the growing diversity of LLM-based IDS rule generation methods while also revealing a lack of standardized evaluation pipelines and systematic human-in-the-loop integration.

12 Synthetic Network Generation↩︎

To ground our user study in realistic and controlled environments, we generated the synthetic vulnerable networks using the existing benchmark on vulnerable network generation [36]. It generates reachability graphs of any network topology and pairs them with vulnerability inventories derived from real-world CPE databases (e.g., Windows, MySQL, Apache). By benchmarking network size, running services, operating systems, and hardware, the generated scenarios emulate enterprise segments (small office, mid-sized firm, multi-segment infrastructure), and the realism of which is validated against real networks via statistical analysis.

13 Selecting Best LLM for User Study↩︎

To enable the human-based evaluation of the generated rules, we here address the issue of selecting which rules to prompt experts with. Indeed, in our experimental evaluation, we considered six different network scenarios and 18 LLM engines, generating a total of 1679 syntactically valid rules. However, it would be impossible to evaluate all such rules given the difficulty of reaching highly skilled experts working in the domain. Therefore, we here consider to identify one LLM engine as the best performing agent whose generated rules will be used for the human experts user study. Meanwhile, we integrate all six network scenarios in the user study, with the aim of capturing the human confidence towards the LLM-based generated rules whenever the task complexity varies.

We identify two parallel approaches to identify the best performing LLM agent either relying on

For the former, we consider as relevant information to identify the best performing LLM engines the following statistical parameters:

As such we compute a statistically-driven metric as the weighted sum over the normalized values of these statistical parameters—e.g., we aggregate the rules generated by each engine across all network scenarios, compute the statistics, normalize them and sum.

For the latter, we run a round-robin LLM-as-a-judge tournament in which each candidate engine is compared against every other one. For each pair of engines, we randomly sample 11 rules from each model and compare them pairwise. Each comparison is evaluated independently by a judge model – we provide the experimental results when using ChatGPT 4.5 nano as the judge, but similar results are obtained for ChatGPT 5.4 mini and ChatGPT 5.4 – which select the preferable rule. The winning rule – and thus the corresponding LLM engine – receives one point, while the losing rule is assigned zero points. After all matches are completed, engines are ranked by their cumulative score across the tournament, and the top-ranked engine is selected.

6 provides the comparison between the engine ranking computed using the LLM-as-a-judge paradigm and the statistical metric we defined. Both rankings prove how larger models are more effective engines for generating valid rules, with many of them performing comparably well. Surprisingly, in the LLM-as-a-judge setting, most LLM engines perform similarly which highlight the difficulty of relying on LLMs to test the quality of given Snort rules. Grounded on these results, we select DeepSeek-R1 70B as the LLM engine used to generate the rules to be tested in our human study. We select DeepSeek-R1 70B as it is the best performing model for both ranking, its relatively lighter nature – w.r.t. other large LLMs – and its capability of leveraging Chain-of-Thought [54]. Unsurprisingly, smaller engines provide to be less effective, as the generated rules are more vague when correct, being characterized by very few fields and general headers that are not usable in practice, characteristics that both the LLM judge and our metric grasp correctly.

14 User Study↩︎

14.1 Demographic Questions↩︎

The demographic information page of the online platform includes the following information:

• Current Primary Role: (e.g., SOC Analyst, Security Researcher, Threat Hunter, Network Engineer)

• Sector: (e.g., Academia, Finance, Critical Infrastructure/ICS, Government, Managed Security Service Provider).

• Organization size: (e.g., 1-10 employees, 10-50 employees, 50+ employees).

• Age: 18-24, 25-34, 35-44, 45-54, 55-64, 65+

• Country/Region: Relevant to check participation and interest in automation.

• Education Level: (e.g., MSc, PhD). This is standard for academic papers in this domain.

• Years of Experience in Rule Engineering: (e.g., <1, 1-3, 3-5, 5-10, 10+ years).

• Primary IDS Engine Used and familiarity: (Checkboxes for Snort, Suricata, Zeek/Bro, Other)

• Frequency of Rule Writing: (e.g., Daily, Weekly, Monthly, Rarely). This helps distinguish between people who manage the systems and those who actually craft the signatures.

• Self-Reported Proficiency: Ask them to rate their comfort level with writing complex signatures (e.g., using pcre, flowbits, or byte_jump keywords) on a scale of 1-5.

14.2 Usability Questions↩︎

To assess the perceived usability and practical value of the proposed system, we designed a set of Likert-scale questions targeting key dimensions of user experience, trust, and operational effectiveness. These questions aim to capture expert judgments on multiple aspects of the system, including confidence in LLM capabilities, the impact of contextual security information (e.g., CVEs and proof-of-concepts), and the perceived benefits in terms of efficiency, such as reduction in time-to-deploy. In addition, the questionnaire evaluates the usefulness of individual system components in supporting the rule writing workflow. Finally, the questions explore comparative and practical dimensions, such as how the quality of LLM-generated rules is perceived relative to traditional automated tools, and the difficulty of modifying such rules in practice. All items are measured on a five-point Likert scale, enabling a structured and quantifiable assessment of usability and perceived system effectiveness across participants. The questions are the following:

• How confident are you in an LLM’s ability to understand complex network topologies with several security contexts (e.g., CVEs and PoCs) without human oversight?

• How much does the inclusion of vulnerability data (e.g., PoCs) change your confidence in the generated rules?

• How much do you think the system would reduce the time-to-deploy?

• How valuable is the network topology visualizer for the rule automation process?

• How valuable is the proof-of-concept finder for the rule automation process?

• How valuable is the Snort rule tester for the rule automation process?

• How valuable is the interactive rule generator for the rule automation process?

• How much would you trust this system to run in an autonomous “suggest-and-deploy” mode?

• How necessary do you feel a human-in-the-loop is in this system?

• How does the rule quality of this system compare to rules generated by traditional automated vulnerability scanners?

• How would you rate the difficulty of modifying LLM-generated rules?

14.3 Open-ended Questions↩︎

The following questions are designed to gather insights into your experience with NIDS rule creation and the use of LLMs in rule engineering. Your responses will help evaluate usability, effectiveness, and potential areas for improvement in such systems.

• How do you currently approach NIDS rule creation? (e.g., manual writing, adapting templates, or using automated scripts?)

• Have you previously used any LLMs for rule engineering tasks? If so, how did you use them?

• Did you notice any instances where the LLM generated syntactically correct rules that were logically flawed for the specific topology?

• Did you leverage external tools/information when taking this user study that helped you writing the rules? If yes, please specify the source of information.

• What specific information was missing from the system’s output that would have helped you make a faster decision?

• What ‘logic style’ did you perceive in the rules generated? Did they feel overly aggressive (high false positives) or too conservative?

• If you could ‘teach’ the LLM one specific nuance about your unique network environment that isn’t in the IP/Service list, what would it be?

• Who is ‘responsible’ for a false negative if an LLM-generated rule is accepted by an expert but fails in production: the model, the system designer, or the human reviewer?

• Describe the ‘friction’ you felt while modifying the LLM’s output. Was it easier to fix a bad rule or would it have been faster to write it from scratch?

• If this tool were integrated into your daily SOC (Security Operations Center) workflow, how would it change your team’s priorities?

14.4 Cognitive load↩︎

To assess the impact of task complexity on participants’ performance, we designed three network scenarios with progressively increasing structural and informational complexity (4). These scenarios were specifically constructed to induce different levels of cognitive load while remaining representative of realistic operational environments in intrusion detection settings. The smallest scenario models a minimally segmented network, consisting of five nodes and a limited number of vulnerabilities. This configuration reflects environments such as small office or home office deployments, where the reduced diversity of assets and attack surface enables relatively straightforward reasoning about rule generation. The intermediate scenario increases both the number of nodes and the distribution of vulnerabilities, approximating a typical enterprise network segment. In this setting, participants must account for a broader range of services and potential attack paths, thereby requiring more complex reasoning and integration of multiple pieces of information. The largest scenario represents a multi-segment infrastructure with 18 nodes and a substantially expanded set of vulnerabilities. This configuration introduces layered complexity, where participants must simultaneously consider multiple hosts, services, and vulnerability interactions, significantly increasing the cognitive demands of the task. To ensure that the scenarios remained analytically tractable while still challenging, we constrained the number of vulnerabilities to at most two CVEs per node. This design choice prevents an exponential increase in possible rule combinations, while still preserving sufficient complexity to meaningfully differentiate cognitive load across scenarios. Overall, the three scenarios provide a controlled yet realistic progression of difficulty, enabling us to systematically evaluate how increasing cognitive load affects the rule generation process.

15 Complementary Analysis↩︎

15.1 Hallucinations↩︎

LLMs are known to be prone to hallucinations [45], [46], where the model confidently presents fabricated, factually incorrect, or nonsensical information. Therefore, we here evaluate practically this issue and identify two easy-to-detect hallucinations that might affect the generated rules. Specifically, we define as hallucinated rules, those rules:

The ratio of hallucinated rules is quite small whenever we consider network scenarios where the PoC information is not available (see 7), with almost no engine reaching more than 10% of hallucinated rules and several achieving no hallucinations at all. However, when PoCs are included, the hallucination fraction swiftly increases (as shown in 7 (a)).

The root cause of such an effect can be found in three concurrent phenomena:

Therefore, the inclusion of PoC information correlates with a 10.10% increase in hallucinations (rising from 2.81% to 12.91%), particularly in engines struggling with large context handling. Conversely, CVE-based prompts remain relatively stable with negligible hallucination rates.

We here also note that hallucinations may extend beyond the two modalities we considered in this subsection, including logical inconsistencies that are much more difficult to identify. Our hallucination analysis is designed to point out the easy-to-spot mistakes, while we refer to 5.2 for the identification of more “sophisticated” hallucinations that require human evaluation.

We further analyze the distribution of the two hallucination types – i.e., non-existing IPs and unrelated CVEs – in 7 (b). Hallucinations are frequently rooted in the generation of rules containing unrelated IP addresses rather than CVEs. We hypothesize that such a phenomenon is rooted on the memorization effect of LLMs [38], [39], [55], where some models may have seen Snort rules during training and memorized part of their content. LLMs subject to memorization may reuse the information from previously seen rules, including IPs, without adapting them to the context, thus introducing hallucinations. IPs are more common than CVEs in IDS rules available on the web, and thus, IP-based hallucinations are more common than CVE ones.

15.2 LLM Explanation↩︎

5 reports participants’ perceptions of the utility of the explanations provided across the three scenarios. Overall, the results indicate a moderately positive assessment, with mean scores ranging from 3.00 to 3.10 on a five-point Likert scale. For Scenario A (Q2), responses show a relatively broad distribution, with 50% of participants selecting ratings in the positive range (4–5), while 40% expressed lower to neutral perceptions (1-2). This scenario also exhibits the highest mean score (3.10), suggesting that explanations were perceived as slightly more useful in simpler contexts. For Scenario B (Q4) and Scenario C (Q10), the distributions are nearly identical, with 70% of participants selecting mid-to-high ratings (3–4) and no responses at the highest level (5). Both scenarios yield a mean of 3.00, indicating a stable but not strongly positive perception of explanation utility as scenario complexity increases. Across all scenarios, responses tend to cluster around the central and moderately positive values (3–4), with relatively few extreme ratings. Notably, the absence or limited presence of the highest rating (5) suggests that while explanations were generally considered helpful, they may not have fully met expert expectations in terms of clarity, completeness, or actionable insight. These results point to a consistent perception of moderate usefulness, with a slight decline in perceived utility as task complexity increases, potentially reflecting the growing cognitive demands associated with interpreting explanations in more complex scenarios.

15.3 SUS-based Results↩︎

6 presents the granular distribution of participant responses for each of the ten statements in the sus, while 5 (c) depicts the diverging Likert scale plot. A total of 10 experts participated in this evaluation following the system interaction phase.

The responses are recorded on a 5-point Likert scale, where 1 indicates "Strongly Disagree" and 5 indicates "Strongly Agree." Note that the sus consists of alternating positive and negative statements; for instance, a high score on Item 3 ("Ease of use") indicates a positive user experience, whereas a high score on Item 2 ("Complexity") would indicate a perceived usability barrier. These raw counts were used to calculate the final aggregate sus score reported in the main body of the paper.

15.4 Statement-based Results↩︎

7 reports the distribution of expert responses for the 11 statement-based questions used to assess the perceived professional viability of the proposed llm-based IDS rule generation system.

The statements cover both functional aspects of the system, such as the usefulness of the PoC Finder, Snort Rule Tester, and Interactive Generator, as well as trust-related aspects, including confidence in context-aware generation, willingness to rely on automated deployment, and the perceived need for human supervision.

Overall, the responses suggest a moderately positive perception of the system. The highest agreement is observed for the PoC Finder and Snort Rule Tester, suggesting that experts particularly valued components that provide supporting evidence and validation before deployment. In contrast, the lowest score is associated with the Suggest-and-Deploy mode, showing limited trust in unsupervised automatic deployment. This result is consistent with the strong perceived importance of maintaining a human-in-the-loop process, especially in operational security environments where incorrect IDS rules may lead to false positives, missed detections, or deployment risks. The relatively moderate scores for time-to-deploy reduction and comparison with traditional scanners further suggest that experts view the system as a useful aid rather than a complete replacement for established workflows.

15.5 Syntactic Validity of Generated Snort Rules Across Network Scenarios↩︎

For each scenario, we compile the Snort-looking rules produced by each llm engine and measure the fraction of rules that do not trigger compilation errors. These plots therefore capture whether each model is able to generate rules that are structurally compatible with Snort syntax, independently of whether the rule is semantically effective for detecting the intended attack.

8 complements the analysis in Section 5.1 by showing the percentage of syntactically valid and invalid rules for three additional CIC-IDS configurations. The first scenario considers one CVE per device with PoC information available, while the second and third scenarios consider two CVEs per device without and with PoC information, respectively. Across these settings, larger models generally achieve higher syntactic validity, although the ranking is not perfectly stable across scenarios.

Across the three scenarios, DeepSeek-R1 70B, Llama3.3 70B, Llama3.1 70B, and Qwen2.5 72B consistently emerge among the best-performing models, each maintaining syntactic validity above 80%. This suggests that higher-capacity models are generally more reliable at producing Snort rules that conform to the expected syntax. At the same time, the results show that scenario complexity affects models differently. Although the strongest models remain relatively robust, syntactic correctness is not guaranteed across all engines or network configurations, especially for smaller or less consistent models.

15.6 Rule Correction Effort Across Additional Configurations↩︎

9 complements the correction-effort analysis in Section 5.1 by reporting the results for the remaining CIC-IDS configurations. For each scenario, the figures show the average number of corrections required by each llm engine, considering only models that generated more than one rule. This metric quantifies the amount of manual or iterative refinement needed after the initial generation step. Lower values indicate that a model tends to produce syntactically valid Snort rules with limited correction effort, whereas higher values suggest that more substantial intervention is required before the rules can be compiled and deployed.

By applying the same metric across all additional scenarios, the plots enable a direct comparison of correction effort under different network configurations. They therefore capture the amount of refinement required to obtain a syntactically valid outcome.

Overall, these results provide insight into the practical usability of each model in an operational workflow. Models requiring fewer corrections are better suited to reducing analyst workload, whereas models requiring repeated corrections may still impose a significant manual burden, even when they eventually produce valid rules.

15.7 Complementary Rule-Header Analysis Across CIC-IDS Configurations↩︎

10 complements the semantic rule-quality analysis presented in Section 5.1.2 by reporting the corresponding header-field distributions for the remaining CIC-IDS configurations. For each scenario, the plots aggregate the fields used in the headers of the correctly generated Snort rules across all considered llm engines.

These distributions provide insight into the structural and semantic choices made by the models when defining rule headers, including the selected protocols, source and destination addresses, and ports. By applying the same aggregate metric across the additional scenarios, these plots enable a direct comparison of rule-header composition under different network configurations, including settings with one or two CVEs per device and with or without PoC information. They therefore complement the main semantic analysis by showing whether the models preserve consistent header-generation behavior across increasingly complex scenarios.

Overall, these additional results help assess whether the generated rules are not only syntactically valid, but also structurally meaningful from an IDS perspective. Consistent and appropriate header-field usage suggests that the models are better able to encode the relevant network context, whereas irregular or overly generic header patterns may indicate weaker semantic alignment with the target scenario.

References↩︎