1 Introduction↩︎

In recent years, task-based chatbots, i.e., software designed to deliver functionalities through conversations[1]–[3], have gained popularity due to their integration into real-world applications. Advancements in technology have made them ubiquitous in a multitude of domains, including e-commerce, healthcare, and customer help-desk [4]–[6] and a large number of design platforms have emerged, with Dialogflow, Amazon Lex, Rasa, Microsoft Bot, and IBM Watson being systematically acknowledged as the most popular platforms [7]–[10], in both open-source and commercial settings.

Given chatbot pervasiveness, tailored approaches to ensure their quality have recently emerged, yet raising novel challenges [11]–[14]. Unlike traditional testing approaches that validate the software, for instance, based on API calls or GUI actions, chatbots combine conventional software layers managing business logic with conversational interfaces that must interpret user requests through natural-language interactions, whose responses must also be interpreted to determine whether a task was correctly performed, raising issues in terms of oracles.

Initial efforts in chatbot quality assurance targeted the speech recognition module [15], [16] and non-functional aspects [17]. Later, testing chatbots as a whole was addressed by employing input mutation for robustness testing [18]–[20] and metamorphic testing [21] to address the oracle problem [22], [23]. However, these proposals demand extensive manual intervention and deep prior knowledge of the chatbot behavior.

Botium is the state-of-the-practice framework for automated end-to-end task-based chatbot testing [24], originally developed by Botium GmbH (now Cyara). Botium provides connectors to several widely used chatbot design platforms and frameworks. Subsequent work has exploited Botium to achieve advances in testing, particularly in test diversity [25], [26] and improved test coverage [27]. Still, all these tools present limitations in test effectiveness, as tests by construction are limited to single request-response interactions, produce only regression oracles, and are susceptible to flakiness² [26], [29]–[32]. Further, all the tools extending Botium only address Dialogflow and Rasa as chatbot development platforms, thus neglecting Amazon Lex despite being one of the most popular commercial solutions [7], [10].

To address such a need, in this paper, we present LexTester, an automated model-based testing approach for Amazon Lex task-based chatbots. We empirically compared LexTester with Botium, evaluating the complexity of the generated tests, conversation coverage capabilities, fault detection, and test generation efficiency. To the best of our knowledge, Botium is the only other tool specifically designed for Amazon Lex chatbots.

The paper provides the following contributions:

1. It proposes LexTester, the first model-based testing technique for Amazon Lex task-based chatbots. LexTester first derives a Dialog Graph representing all possible conversations of a chatbot, and then generates executable tests that exercise such conversations according to a coverage criterion;

2. It presents empirical results collected from prebuilt and third-party chatbots that show how LexTester outperforms Botium in terms of test complexity, conversation coverage, and fault detection, with comparable time efficiency;

3. It makes available a replication package with references to the LexTester source code, usage instructions, and results for experimental adaptation and replication, found at: https://gitlab.com/lextester1/lextester-exp.

The paper is organized as follows. Section 2 introduces the key concepts of task-based chatbots and testing with Botium. Section 3 describes LexTester. Section 4 presents the experiment, comparing LexTester with Botium. Section 5 discusses related work. Finally, Section 6 provides final remarks.

2 Background↩︎

This section introduces some background about the structure and functionality of task-based chatbots, characteristics of Amazon Lex chatbots, and core concepts of conversational testing in Botium.

2.1 Task-based Chatbots↩︎

Task-based chatbots are software systems designed to interact with users through conversations, structured as textual, vocal, or visual interactions, and integrated with backend logic and external services. Their primary goal is to efficiently fulfill user requests, prioritizing task completion over casual chat [2], [3]. Figure 1 captures the key concepts underlying task-based chatbots, as commonly supported by most popular development platforms [33], including Amazon Lex.

A task-based chatbot is designed to handle one or more intents, each one representing a specific user goal (e.g., Order Flower in the Figure corresponds to the user intention of buying some flowers at a given date).

Each intent includes a set of utterances (also called training phrases) and actions. Utterances are used to train the chatbot to recognize user requests associated with a given intent (e.g., “I want to order some roses” is an utterance for the Order Flower intent). Actions, on the other hand, define how the chatbot fulfills a request and generates a response. Actions may consist of simple plain-text responses (e.g., “Ok, I will place your order. Else?”) or more advanced operations involving business logic execution and interaction with external services (e.g., “Order confirmed!” for the Done intent that triggers a function to activate an external service).

Chatbots define slot types (also called entities), which represent the datatypes the chatbot recognizes (e.g., FlowerType). Slot types are instantiated through slots (also called parameters) extracted from user utterances (e.g., “roses” for the FlowerType slot type in the first utterance of the Order Flower intent) and can be referenced within actions (e.g., $flower in the first action of the Done intent). When necessary, a slot filling mechanism can be employed by the chatbot to prompt the user for missing required slots (e.g., “Which flowers?”).

Intents are connected to form flows, representing the possible conversational paths that interleave user requests and chatbot responses. Additionally, a fallback intent can be defined to handle cases where the chatbot fails to understand a user request, prompting the user to rephrase it.

A typical interaction with a chatbot begins with a user request. The chatbot leverages Natural Language Processing (NLP) models trained on utterances to map the request to an intent. Based on the identified intent, the chatbot generates a response and executes any associated actions, passing the extracted slots if required. If no intent is recognized, the fallback intent is triggered. This process continues until the user goal is fulfilled.

2.2 Amazon Lex Chatbots↩︎

Amazon Lex is the chatbot design platform developed by Amazon (current version V2³), integrated within Amazon Web Services (AWS), natively supporting connection to, among the others, Amazon cloud services, advanced Natural Language Understanding (NLU) technologies, and lambda functions (i.e., serverless computing services that enable the execution of backend logic, see Figure 1).

From a structural perspective, an Amazon Lex chatbot adheres to the typical task-based chatbot structure shown in Figure 1, hence being composed of intents, slot types, user utterances to train the NLU model, and so on. In addition, Amazon Lex provides a Visual Conversational Builder⁴ tool to design and visualize the user-bot conversational flows by means of interconnected blocks that represent the different units composing a conversation.

Examples of blocks are the Start and End blocks that initiate and terminate a conversation, Get slot value blocks aimed at eliciting slots, Confirmation blocks to ask for user confirmation before completing an action, Code hook blocks to invoke lambda functions, and Go to intent blocks redirecting the conversation flow toward another intent. Figure 2 shows a portion of a conversation modeled using the Visual Conversation Builder. Since the Visual Conversation Builder does not support direct export for model-based testing purposes, we took inspiration from it to design our Dialog Graph and integrate it within LexTester.

2.3 Botium Test Cases↩︎

Botium [24] is an automated quality assurance framework developed by Botium GmbH in 2018, widely used in both industrial and academic contexts [7], [13], [25]. It supports test case generation and execution for various chatbot design platforms, including Dialogflow [34], Amazon Lex [35] and Rasa[36], representing the state-of-the-practice approach for chatbot testing.

In Botium, a test case is a conversational scenario, composed of text-based steps between the user (annotated with #me) and the chatbot (annotated with #bot), derived from the files of the chatbot implementation that encode the possible requests and responses. Figure 3 shows an exemplified Botium test. When a test is executed, Botium connects to the chatbot on the specific platform and proceeds to simulate the user by sending the utterances established for the scenario to replicate, observing whether the chatbot responses are aligned with the expected ones. Botium supports multiple asserters that can be used to this end. For instance, the text asserter verifies if the expected response under the #bot tag matches the actual one. By default, Botium applies the intent asserter (keyword INTENT followed by the intent name that must be activated).

3 LexTester↩︎

LexTester is a model-based test generator developed in Java for Amazon Lex chatbots. The pipeline of the tool is shown in Figure 4.

LexTester takes as input the path to a Lex chatbot folder, whose structure is contained in JSON files incorporating the key conversational concepts introduced in Section 2, and a configuration file defining the coverage strategy for test generation and the data used to generate user input. The tool parses the content of the chatbot conversational data, then (i) generates a Dialog Graph, (ii) generates abstract test cases from the graph in the form of conversational paths according to the chosen coverage strategy, and (iii) generates concrete executable test cases in a Botium-like format from abstract test cases, instantiating them with actual input values. It is worth noting that LexTester relies on Botium infrastructure only for test execution.

3.1 Dialog Graph↩︎

Initially, LexTester constructs the Dialog Graph, that is, a directed multigraph inspired by the Amazon Lex Visual Conversation Builder, from the JSON files describing the chatbot structure. The Dialog Graph is formally defined as \(DG = (V, E, type, label)\), where \(V\) is the set of vertices representing conversational blocks (e.g., Start, Get slot value, Confirmation, Go to intent), \(E\) is the set of edges encoding the conversational transitions and outcomes (e.g., success, failure, alternative paths), \(type(V)\) identifies the block type associated with vertex \(V\), and \(label(V)\) stores conversational information associated with \(V\), such as utterances or expected slot values.

To generate the graph, LexTester iterates over all intents composing the chatbot and analyzes the exported JSON descriptors to infer the conversational block types to create. Depending on the identified type, LexTester extracts the relevant conversational information (e.g., slot filling messages, lambda invocations, conditional branches) and associates them with the corresponding vertices and edges. Finally, conversational flows are reconstructed by connecting the generated blocks according to the references defined in the chatbot configuration. The Dialog Graph is serialized as a DOT-format file employing GraphStream API⁵ for data construction and Graphviz⁶ for graphical visualization.

Figure 5 shows an example of a Dialog Graph generated by LexTester for a flower-ordering conversation. The flow starts from a Start block, followed by a Code Hook block invoking a lambda function. The block exposes three possible outcomes (success, failure, and timeout), whose associated values are undefined since they are not statically defined in the chatbot configuration but depend on the lambda implementation. In the success branch, multiple Get slot value blocks elicit the required slots (e.g., FlowerType), followed by a Confirmation block before reaching the End conversation block. Failure branches instead redirect the conversation toward the fallback intent, represented by a separate Start block.

3.2 Abstract Test Cases↩︎

Once the Dialog Graph is built, LexTester generates abstract test cases by traversing the graph from the Start block toward terminal vertices. LexTester is currently implemented to support the N-gram path coverage criterion, i.e., a sequence of up to N consecutive vertices traversed. For example, considering a sequence Start \(\mapsto\) Get slot value \(\mapsto\) Confirmation, a 3-gram covers the entire conversational subsequence, while a 2-gram covers each pairwise transition independently. During traversal, each visited vertex contributes conversational information used to incrementally build the test, including user utterances, chatbot responses, and the type of slots expected at that stage of the interaction. LexTester systematically generates conversational paths that include all distinct N-grams of the graph, thus exercising both short and long interaction patterns.

3.3 Concrete Test Cases↩︎

Abstract test cases are finally instantiated into executable Botium-like test cases, i.e., sequences of alternating #me and #bot interactions. LexTester relies on input values to fill slots required by the chatbot, which can be either selected from predefined datasets or randomly generated according to the required slot type. During instantiation, placeholders derived from path traversal are replaced with concrete user utterances consistent with the expected slot semantics, producing complete executable conversational tests, adhering to the test structure shown in Figure 3.

4 Empirical Evaluation↩︎

We evaluated LexTester against the state-of-the-practice tool Botium on five Amazon Lex task-based chatbots, considering the following research questions:

• RQ1-Test Coverage: How effective is LexTester in covering conversational elements? This research question studies the capability of LexTester in covering conversational elements compared to Botium;

• RQ2-Fault Detection: How effective is LexTester in detecting faults in conversations? This research question studies the capability of LexTester in detecting faults in conversations compared to Botium;

• RQ3-Test Efficiency: How does LexTester affect test generation efficiency? This research question studies the cost of applying LexTester compared to Botium.

To answer RQ1, we measured the percentages of intents, slot types, and slots covered by the executed tests.

To answer RQ2, we applied mutation testing on the five subjects, employing Mutabot state-of-the-art tool [29], [30], measuring mutation score (i.e., the rate of mutants detected over total mutants).

To answer RQ3, we measured the time required by LexTester and Botium to generate and execute a test suite, respectively. For LexTester, we also included the time required to generate the Dialog Graphs. We evaluated test efficiency in terms of test volume, measured as the number of total tests generated, and test complexity, measured as the minimum, maximum, and average number of user-bot interactions per test.

4.1 Subject Chatbots↩︎

To investigate the research questions, we selected five Lex chatbots: two prebuilt chatbots provided by the Amazon Lex platform and three third-party chatbots available on GitHub, selected considering conversational size (i.e., they present a high number of intents, slot types, and slots), popularity (i.e., they have at least one GitHub star), topic diversity (i.e., they are representative of different domains), and functional complexity (i.e., they implement actual backend logic).

Table ¿tbl:tab:chatbots? reports the name, topic, and conversational size (i.e., number of intents, slot types, and slots) of each selected chatbot.

4.2 Experimental Procedure↩︎

To conduct the experiment, we configured each third-party chatbot following the instructions provided in the repositories, while the prebuilt chatbots were generated directly from the platform.

For Botium, we used the Amazon Lex connector implementation to interact with chatbots on the platform and generate tests. For LexTester, we used the implemented N-gram coverage strategy with path length set up to 10.

Coverage was measured by inspecting the generated tests and execution outcomes. For fault injections, we employed the Mutabot tool [29], [30], originally designed for Dialogflow and equipped with 24 operators to mutate chatbot conversational elements, such as removing an intent, changing the name of a slot type, or modifying a user utterance. In this work, we adapted the operators to Amazon Lex, to simulate faults occurring on that platform. Each mutant was inspected, iteratively deployed on the platform, and run against each Botium and LexTester test suite, measuring mutation score. Finally, the time required for test generation and execution was measured using the timers implemented in each tool, while test complexity (i.e., minimum, maximum, and average number of user-bot interactions per test) was measured using an automated script.

To reduce randomness in the results, we repeated each test execution five times for both LexTester and Botium on each chatbot, cleaning chatbot states after each execution (e.g., created resources) to avoid any side effect.

4.3 Results for RQ1 - Test Coverage↩︎

Table ¿tbl:tab:rq1? shows the test volume and the coverage (%) of intents, slot types, and slots computed for each chatbot and technique.

In all cases except for the Cloud Assistant chatbot, LexTester was able to produce more tests, with test suites 1.9-175 times larger than those produced by Botium. Moreover, LexTester achieved the same or higher coverage than Botium in each chatbot. This is particularly evident for chatbots of high complexity (e.g., Cloud Assistant). On the other hand, due to its simplicity, the only chatbot in which both Botium and LexTester achieved the same coverage was Movie Recommender.

Concerning intents, none of the tools was able to cover fallback intents, as when no training data is provided for failing scenarios, the tools cannot exploit it for test generation, thus none of them ever achieved 100% of intent coverage. Overall, Botium and LexTester capabilities were very similar, achieving intent coverages of 79% and 83%, respectively, with LexTester covering one intent missed by Botium, which was unable to use proper data to generate tests.

The results highlighted better performance in LexTester for the coverage of the slot types and the slots. In fact, several chatbots defined slot types and instantiated slots in scenarios based on follow-up interactions (i.e., when a sentence does not provide all required slots, it activates a slot filling mechanism to elicit the missing ones), which Botium was unable to resolve, even in simple chatbots (e.g., in Order Flowers Bot the user is prompted to specify which flower to order, which is never covered by Botium). Overall, Botium achieved 73% slot type coverage and 60% slot coverage, whereas LexTester achieved 95% and 94%, respectively.

4.4 Results for RQ2 - Fault Detection↩︎

Table ¿tbl:tab:rq2? shows mutation score computed for faults injected with Mutabot in intents, slot types, and input/output elements pertaining to flows (e.g., a user request, a bot response, or a context variable representing data propagated between intents with a certain lifespan). We inspected the mutants generated to remove equivalent ones, which were not found. Further, we verified that the adapted mutations produced valid faults for Amazon Lex chatbots.

Considering intent-related mutants, LexTester and Botium produced the same results in all chatbots but Cloud Assistant, in which LexTester detected two more faults about an intent that Botium could not exercise (79% and 83% overall for Botium and LexTester, respectively). As expected, both tools missed mutations affecting the fallback mechanism.

On the other hand, faults in slot types and input/output elements pertaining to flows were more extensively detected by LexTester(overall, mutation scores of 58% and 65%, respectively), whereas Botium performed poorly (13% and 43%, respectively). The performance of Botium can be explained by the simplicity of its generated tests, as already observed for RQ1, since the tool could not thoroughly exercise slot filling mechanism; moreover, by default Botium is limited to the basic intent asserter, which does not specifically check for expected slot values or chatbot responses, hence missing any mutations affecting them.

Although promising, the outcomes of LexTester also suggest some limitations in fault detection. The tool was unable to detect faults affecting flow context data, as the generated tests could not determine their correct expiration timing after such data had been mutated from the original behavior. Further, some mutations affecting removal of prompts for slot filling were not detected, as LexTester did not produce tests exercising such conversational paths.

4.5 Results for RQ3 - Test Efficiency↩︎

Table ¿tbl:tab:rq3? shows the time required by LexTester and Botium to generate and execute the test suites, respectively, with the corresponding test volume and test complexity (minimum, maximum, and average number of user-bot interactions).

In all cases, the time required for test generation and execution was higher in LexTester than in Botium. The test generation time in Botium was very similar to LexTester, few seconds in most cases including graph generation. On the other hand, LexTester required more execution time than Botium. In general, test suites required 4 seconds to 45 minutes for Botium and 3 minutes to 53 minutes for LexTester, with a comparable time per test.

The longer time required by LexTester to complete the execution of the test suite was justified by the larger number of tests and the higher complexity of the tests. Regarding complexity, LexTester outperformed Botium in every chatbot, producing more sophisticated tests (between 2 and 6 user-bot interactions on average), whereas Botium was limited to generate just a user request and a chatbot response, missing more advanced scenarios. For example, considering the Cloud Assistant chatbot, the only subject for which LexTester generated fewer tests than Botium, LexTester still required more time to complete the test suite, as the higher complexity of its tests enabled conversational scenarios in which the user could create resources through interactions with the chatbot, functionality that could not be exercised by Botium.

4.6 Discussion↩︎

Results across all research questions showed that both LexTester and Botium are capable of generating large test suites with low time overhead and comparable execution times. LexTester can consistently generate tests of higher complexity, enabling the execution of more diverse and advanced conversational scenarios, leading to broader coverage of conversational and functional aspects than Botium, as also reflected by higher mutation scores.

While LexTester outperforms Botium in all the considered dimensions, it can show limitations in detecting faults affecting the lifespan of flow context data, which would require longer interaction sequences. As previously observed, both tools are limited by their reliance on available training data, which prevents systematic exploration of fallback intents. Although the Dialog Graph generated by LexTester explicitly models both successful and failure scenarios, some of the paths are not feasible during test generation, due to the absence of sufficient negative data to properly instantiate them. Finally, despite the high test volume, test redundancy remains an issue. Neither tool was able to fully achieve coverage of all conversational elements together. Nevertheless, LexTester reached 100% coverage of both slot types and slots in four out of five cases.

Overall, these findings are aligned with previous studies on task-based chatbot testing [25]–[27], [29], [30], [37]. At the same time, they highlight the advantages of LexTester in exercising complex conversational and functional behaviors, indicating the need for improved data generation when training data is scarce. The evaluation was conducted on a heterogeneous set of five Amazon Lex chatbots, including two with relatively high conversational complexity compared to other publicly available Lex chatbots, making them representative of non-trivial industrial-like scenarios. Moreover, test generation and execution times remained acceptable even for the more complex chatbots, suggesting that the approach scales well with increasing conversational complexity.

4.7 Threats to Validity↩︎

An internal threat to validity concerns the manual intervention required during the experiment. In particular, we adapted Mutabot tool to support Amazon Lex chatbots. Further, the assessment of coverage and fault detection required manual inspection of the generated tests and their outcomes, as well as inspection of the produced mutants. Third-party chatbots require manual configuration and deployment. To mitigate this threat, we carefully inspected each deployed chatbot to check that it worked as intended. Mutants produced by Mutabot were also inspected to ensure that no unintended deviations from the expected mutations occurred. Finally, we inspected test outcomes across multiple test executions, to collect consistent results.

An external threat to validity concerns the generalizability of the results. Although the findings cannot be considered definitive, they provide an initial empirical assessment of automated testing for Amazon Lex task-based chatbots. The evaluation included both prebuilt and third-party chatbots from different domains and complexities, and results were consistent with previous studies with other platforms [25]–[27], [29], [30], [37].

5 Related Work↩︎

As chatbots are increasingly integrated into real-world applications, the need for tailored and automated quality assurance techniques, capable of exercising conversational aspects and designing well-defined oracles has arisen. However, only limited approaches have addressed this problem so far [11], [13], [14], [31].

Botium [24] is a widely used automated quality assurance framework that supports test generation and execution for task-based chatbots, across a plethora of platforms, both open-source and commercial, including Dialogflow, Rasa, and Amazon Lex. Over the years, Botium has been used as a backbone for subsequent research works, specifically targeting Rasa and Dialogflow platforms [25]–[27]. CHARM [25] leverages Botium test cases for test augmentation, generating variants of user utterances aimed at evaluating chatbot robustness. ASYMOB [27] generates Botium-like test cases employing multiple coverage strategies. CTG [26] uses Botium output as seed tests to automatically augment test suites by capturing dynamic behaviors and improving conversational coverage.

As in traditional approaches, to assess testing effectiveness, mutation testing has been adapted to the chatbot domain, emulating actual faults affecting conversations [26], [29], [30], [37]. These studies have revealed a general low capability of existing tools in detecting injected faults.

Other approaches have explored black-box testing of task-based chatbots through user simulation [38], [39]. For instance, Bottester [38] simulates user interactions with chatbots, taking as input specification of conversational flows and time controlled events. Božić et al. [39] propose an automated testing approach based on AI planning, where a plan represents an abstract test case as actions to perform to achieve a given goal. More recently, TRACER [40] infers conversational models from chatbots using Large Language Models (LLMs) and synthesizes them into testing profiles to guide user simulation [41].

Similarly to CHARM [25], input mutation for task-based chatbots has been investigated by Guichard et al. [19], introducing paraphrases for evaluating chatbot robustness on the BoTest testing framework [18]. Input mutation has also been studied concerning the oracle problem, defining metamorphic relations [21], such as synonym transformations and word removals, and an ontology-based infrastructure to support test generation for Dialogflow chatbots [22], [23]. Dialtest [20] was proposed as a testing tool to generate input mutations for evaluating accuracy of NLU modules in recurrent neural network-based dialog systems.

Unlike previous approaches, LexTester is the first to enable model-based testing of Amazon Lex chatbots relying on Botium for test execution only.

6 Conclusion↩︎

Task-based chatbots are increasingly being adopted in real-world applications, yet tailored automated testing approaches still struggle to effectively exercise complex conversational and functional behaviors. In particular, research efforts have focused mainly on Dialogflow and Rasa platforms, mostly neglecting Amazon Lex despite being a popular chatbot design platform.

In this paper, we presented LexTester, an automated model-based testing approach for Amazon Lex task-based chatbots. LexTester was evaluated against the state-of-the-practice Botium tool. Experimental results showed that LexTester generates more complex and effective test suites than Botium, improving conversational coverage and fault detection while maintaining comparable execution costs. As future work, we aim at extending the study of LexTester to additional application contexts. At the same time, we plan to enhance the existing LexTester functionalities, in particular implementing more advanced coverage criteria, designing input mutations to enable robustness testing and manage scarce training data, studying dynamic and LLM-based oracles, and adapting test suite minimization techniques.

6.0.1 ↩︎

The authors have no competing interests to declare that are relevant to the content of this article.

References↩︎

---

1. Corresponding Author↩︎

2. “A flaky test is a test that passes and fails periodically without any code change” [28]↩︎

3. https://docs.aws.amazon.com/lexv2/latest/dg/what-is.html↩︎

4. https://docs.aws.amazon.com/lexv2/latest/dg/visual-conversation-builder.html↩︎

5. https://graphstream-project.org/↩︎

6. https://graphviz.org/↩︎