1 Introduction↩︎

Industrial decision-making is increasingly shaped by data. Under the broad umbrella of smart manufacturing and Industry 4.0, organizations collect and integrate growing volumes of heterogeneous information from production systems, sensors, enterprise platforms, quality records, and supply-chain processes [1]. Yet, despite this expansion in data availability, the dominant way of accessing and acting on such information remains largely unchanged: in manufacturing and operations, decision support is still commonly mediated through graphical, dashboard-based interfaces [2].

This predominance of dashboards is understandable. Visual interfaces can condense large amounts of information, support monitoring, and provide decision-makers with an overview of operational conditions through charts, tables, and other visual summaries [3], [4]. In industrial settings, they have become a practical interface layer through which data is turned into action. However, the same visual paradigm can also become demanding as data environments grow in scale and heterogeneity. Decision-makers may need to navigate across multiple views, compare distributed cues, and piece together relevant evidence before reaching a conclusion. In such contexts, the challenge is accessing and using data in a form that supports decision work efficiently and sustainably. As data assume a more central role in Industry 4.0 environments, this challenge also makes users’ proficiency in working with data and with data-driven systems increasingly consequential [5]. Put differently, the effectiveness of industrial decision support may depend not only on the interface itself, but also on the extent to which users are able to interpret, navigate, and act on the information it provides.

Recent advances in generative AI have made this issue more salient. Large Language Models (LLMs), in the form of Conversational Agents (CAs), are increasingly being proposed as a new access layer for data and analytical systems, including in manufacturing and related industrial domains [6]–[8]. Their appeal lies in the possibility of interacting with complex systems through Natural Language (NL): rather than navigating a dashboard, users can ask questions directly, request clarifications, and potentially obtain synthesized answers that reduce some of the effort involved in locating and combining information. More broadly, this has renewed interest in conversational user interfaces as a possible alternative, or complement, to traditional graphical interfaces for decision support [9].

At the same time, conversational access should not be assumed to be uniformly superior. In fact, dashboards and conversational interfaces embody different interaction logics. On the one hand, dashboards offer persistent overview, direct manipulation and simultaneous visibility of multiple information elements, and direct inspectability of the data representation. By contrast, LLM-based conversational interfaces mediate access to information through language providing stepwise dialogical exchange. The practical value of one or the other therefore depends on how well each interaction paradigm supports the kind of decision work users must perform. Or, in other words, on the fit between the interaction format and the cognitive demands of the task. This expectation is consistent with Cognitive Fit Theory, which argues that problem-solving performance is shaped by the match between the problem representation and the strategies required by the task [10].

This issue is especially relevant in industrial contexts, where decision tasks can vary substantially in the amount and type of information they require users to retrieve, compare, and integrate.

Despite growing interest in LLM-based systems, direct empirical evidence comparing conversational and dashboard-based interaction for industrial decision support remains limited. To address this gap, this paper presents a quantitative confirmatory study comparing an LLM-based conversational interface and a dashboard in industrial decision tasks based on the exploratory mixed-approach results obtained in a previous work. The study tests whether interface type is associated with differences in users’ task performance and perceived demands, and whether these effects vary as task demands increase. By doing so, it aims to provide more rigorous evidence on when conversational access to industrial data may constitute a meaningful alternative to conventional dashboard-based interaction.

2 Theoretical Background↩︎

2.1 Information Overload↩︎

Information Overload (IO) has been defined as the exceeding of processing capabilities given an amount of information, specifically occurring when information-processing requirements surpass an individual’s cognitive capacity [11]–[13]. Historically, this hiatus between cognitive limits and information volume was routinely lamented, but as technology progressed, these gaps were managed, reduced, or brought to higher levels of abstraction [14]. Such a concept becomes even more relevant in the context of the hyper-connected world of Industry 4.0, where huge data flows must be processed and visualized; here, IO acts as a primary stressor in the "information society". Managers are now called to pursue decision-making through a data-driven approach granted by the Internet of Things (IoT) and Cloud Computing, yet IO remains a pervasive threat at the interface level [15].

This phenomenon is commonly represented with the inverted U-curve inspired from the Yerkes-Dodson law, which shows that decision accuracy (and therefore its quality) declines after reaching a specific threshold of information load [12]. Beyond this critical point, the burden of a heavy information load confuses the individual, impairs the ability to set priorities, and makes prior information more difficult to recall. In managerial decision-making, [16] categorized IO as a "Decision Stressor" (together with time pressure, complexity, and uncertainty) that is specific to the problem at hand and significantly impacts the quality of outcomes.

Modern research emphasizes that IO is not merely a product of information quantity but is exacerbated by information complexity, low data quality, and the "technostress" induced by constant digital availability [11], [15]. Consequently, achieving objective and effective decision-making in modern industrial settings requires mitigating these cognitive pressures through structural preventions, such as visualization dashboards in Decision Support Systems (DSS), intelligent filtering agents, and the enhancement of organizational information management [11].

As IO cannot be measured directly with an objective instrument and is strongly context-dependent, studies typically assess it via either subjective measures, performance-based measures, or a combination of them [17]. Previous research also shows that IO could be related to MWL [12], [18], as it is reasonable to think that as information load increases, the need for cognitive processing capabilities grows accordingly. In this work, we induce higher vs. lower information load by varying the number of information cues required to reach a decision (i.e., manipulating task complexity), and we assess its impact through performance (decision accuracy and completion time) and subjective demand (MWL) as downstream indicators.

2.2 Mental Workload↩︎

Mental workload (MWL) is a core Human Factors and Ergonomics construct concerned with the demands imposed by a task and system, the effort required from the operator to meet those demands, and the potential consequences for performance [19]. It is often used to understand when task demands and system characteristics risk pushing operators toward degraded performance, either because demands exceed available resources (overload) [20]. Operationally, there is no single technique that can unambiguously indicate whether workload is “too high” or “too low,” and definitional debates have persisted for decades [20], [21].

Originally conceived for aviation contexts, MWL has been increasingly used in HCI to quantify cognitive demands imposed by interactive systems, but this import has been uneven. [22] note a “vague understanding” of cognitive workload in HCI and the absence of a standard definition, motivating more explicit construct specification in studies. Accordingly, in this study we adopt the integrative definition of MWL by [23], as the degree of activation of a finite pool of cognitive resources over time while cognitively processing the primary task, expressed through devoted effort and attention to cope with task demands, and mediated by stochastic environmental/situational factors as well as internal operator characteristics.

This definitional plurality has direct consequences for measurement choices. MWL (similarly to IO) has been assessed through subjective self-report questionnaires, such as NASA-TLX, the Bedford scale, and RSME; psychophysiological measures, such as eye tracking, pupil dilation, skin conductance, and electroencephalogram; performance measures, such as time, errors, and accuracy; and analytical approaches, each embedding different assumptions about what workload is [23].

Recent critiques argue that NASA-TLX’s dominance in HCI is not automatically justified [19], [24], as it was not designed for modern digital interaction contexts and its ease of use can encourage uncritical deployment without clarity on what it actually measures. Moreover, variations in instrument versions/modifications and analysis decisions (e.g., weighting, aggregate vs subscale interpretation) can further undermine validity if not explicitly reported and theoretically justified.

From a Cognitive Fit Theory (CFT) perspective—i.e., the idea that problem-solving performance results from the interaction between the problem representation and the task—MWL can also be treated as a cognitive cost arising when the system’s external representation does not align with the representation required by the task. When either representation is poorly suited to the task, users may need to invest additional effort in translating, integrating, or re-encoding information before they can reason with it [25]–[27]. Such misfit is expected to increase MWL and, in turn, slow decisions and/or reduce accuracy. Prior work links differences in information representation and interface design to both decision performance and subjective workload [26], [28]. Although CFT was originally developed around graphical and tabular representations, the same rationale can be extended to LLM-based CUIs: dashboards externalize information in a persistent visual space, whereas CUIs mediate access to information through sequential, language-based exchange. This distinction is consistent with the contrast between diagrammatic and sentential representations described by [29]. This motivates our comparison of a visualization-heavy dashboard and an LLM-based conversational interface, whose distinct representational forms may alter MWL even under comparable information demands.

Against this background, in this study we treat MWL as a construct capturing the cognitive costs associated with experimentally manipulated information load and interface type. For subjective MWL assessment in an unmoderated online scenario like ours, we use the NASA-TLX because it remains the de facto standard in HCI [22], enabling comparison with prior work while making known methodological pitfalls well documented. We use the unweighted “Raw TLX" aggregate score, as the weighting procedure adds participant burden and has limited evidence of added diagnostic value [30], [31].

2.3 Task Complexity↩︎

Task complexity has no single, universally accepted definition, consequently it is typically specified in terms of the properties that make a task complex. Reviews across information-systems and ergonomics traditions converge in treating task complexity as multidimensional rather than a single scalar attribute. For example, [32] synthesize prior work into structuralist, resource-requirement, and interaction perspectives, highlighting that complexity may stem from task structure (e.g., number and arrangement of elements), the cognitive resources required, and the dynamic interplay between task, environment, and operator.

[33] also identify discretion (i.e., multiple viable strategies) as one of the main elements supporting task complexity (alongside task and problem space). This is particularly important because different levels of discretion for a task can lead to very different complexities. In the case of high discretion a person can find different ways to cope with the task requirements. For example, when choosing between two similarly rated headphones on an e-commerce site, an individual might flip a coin instead of reading a detailed comparison. This illustrates why experiments that manipulate complexity must constrain discretion when feasible to ensure participants engage with the intended complexity.

Relatedly, [32] distinguish objective from subjective task complexity. While in objective computations task complexity is determined by a number of a priori quantifiable elements, subjective complexity takes a constructivist stance stating that the complexity of a task is not an inherent property of the task itself, rather it is contingent on the individual performing the task. This matters because task complexity is typically treated as a task-side antecedent of MWL [23].

However, for experimental work, an explicit objective formalization can be preferred as it supports reproducible manipulations and clearer separation from downstream psychological outcomes. A widely adopted operational framework is provided by [34], who decomposes task complexity into component complexity (e.g., number of cues/acts), coordinative complexity (e.g., interdependence and integration among cues/acts), and dynamic complexity (e.g., changes over time). This decomposition has been used in HCI decision-making contexts to standardize tasks and enable controlled comparisons across studies [35], or to define experimental complexity conditions [36], [37].

In this study we operationalize task complexity primarily in objective terms following Wood’s component and coordinative dimensions —dynamic complexity is held constant because tasks are static —, by manipulating the number of information cues required and the acts upon those cues needed to reach a decision. Therefore, subjective experience of difficulty/demand is treated as an outcome captured by MWL measures.

2.4 Data Literacy↩︎

Data literacy (DL) corresponds to the individual’s knowledge and skill set for understanding and using data effectively. Multiple definitions and frameworks emphasize that DL entails more than basic numeracy as it covers the ability to manage and interpret data, draw appropriate conclusions, and integrate data into decision-making [38], [39] . DL is often positioned as part of (or adjacent to) information literacy, with overlaps with statistical/quantitative literacy but a stronger emphasis on data practices and data-driven reasoning in applied contexts (e.g., finding, assessing, interpreting, and using data for decisions) [39], [40].

Beyond definitional work, information-systems research has examined DL as a capability that shapes how people experience and cope with data-intensive work. In particular, [5] connect DL to overload-related constructs (e.g., perceived data overload/technostress and related strain) as well as to performance-relevant outcomes in organizational settings.

In Industry 4.0, decision-makers face continuous, heterogeneous operational data streams (e.g., IoT/sensor, production and maintenance data) and are increasingly expected to act on analytics outputs under time constraints [41]. Reviews highlight the centrality of data-driven decision-making and near–real-time decision processes [42], [43]; precisely the kind of settings where limited data literacy can hinder effective use of analytics and decision-support outputs [44].

The rise of GenAI-enabled interfaces can reduce some “access” barriers (e.g., allowing natural-language querying), but it can also shift demands toward formulating good questions, checking assumptions, and validating outputs—making DL (and adjacent AI/data-critique competencies) more, not less, consequential in practice [45].

In this study, DL is treated as a pre-existing covariate that may shape task performance and subjective outcomes independently of the interface manipulation. We measure DL before the experimental tasks using the Brief Data Literacy Instrument (BDLI) by [46], a short self-report scale developed as a streamlined version of the 25-item Global Data Literacy Benchmark (GDLB) and psychometrically evaluated using confirmatory factor analysis and item-response theory across student and professional samples. We use BDLI scores both (i) as a covariate in analyses and (ii) to support balanced assignment across interface conditions.

2.5 Intended Reliance↩︎

Reliance is often understood as a behavioural consequence of trust in the reliability of an automated decision-support system [47]–[50]. In interactions with classifiers or detection systems, reliance is commonly operationalized as the rate at which users accept the model’s suggestions. The widespread adoption of complex AI models in decision-making, and their recognized influence on users’ reasoning processes, has therefore led the HCI research community to investigate appropriate reliance, under-reliance, and over-reliance [49], [51].

Since no unique acceptance rate is agreed upon as normative, these concepts are often defined in relation to the correctness of the AI’s suggestion and, in more recent studies, the correctness of the user’s own decision. For instance, reliance may be considered excessive when the acceptance rate: (i) exceeds the accuracy of the system; (ii) is higher for incorrect suggestions than for correct ones; or (iii) is higher for incorrect suggestions when the user would have been correct without support than for correct suggestions when the user would initially have been wrong.

Reliance is also influenced by several factors. On the system side, these include experienced reliability, the presence of explanations, confidence scores for suggestions, and the interaction pattern. On the user side, they include factors such as confidence in one’s own judgement. Reliance is also shaped by contextual and external factors, including time pressure, training interventions, and speed of interaction. In particular, the difficulty of the overall AI-assisted decision process can affect reliance: stronger over-reliance has been associated with more challenging tasks [37], [52]–[54] and with more elaborate explanations [52], [55].

In this study, however, reliance is not measured as behavioural acceptance of discrete AI recommendations. The comparison concerns two decision-support interfaces, a dashboard-based GUI and an LLM-based CUI, only the latter of which relies on AI. For this reason, the study focuses on an antecedent-oriented measure of reliance, namely intended reliance, drawing on the role of intention formation in models of trust and reliance in automation [49], and treats the GUI condition as the baseline. Intended reliance captures participants’ stated willingness to rely on the system for subsequent decision tasks after interacting with it. It is therefore treated as a post-use evaluative outcome that precedes actual behavioural reliance, distinct from objective task performance, perceived workload, usability, and confidence, but potentially shaped by all of them.

3 Research model and hypotheses↩︎

3.1 Primary hypotheses↩︎

Our primary hypotheses concern (i) the overall effect of interface type and (ii) whether this effect varies as a function of task demands (complexity) and individual capabilities, specifically data literacy (DL).

3.1.0.1 Interface effects

As argued above, an LLM-based CUI may reduce part of the cognitive effort involved in decision making by partially automating information access and supporting users through a stepwise interaction. Rather than requiring users to manually search across multiple views and integrate relevant cues on their own, the system can help narrow the information space and directly return task-relevant outputs [56]–[58]. In this sense, conversational interaction may reduce search, navigation, and intermediate integration costs.

At the same time, this benefit may come with countervailing costs. Because interaction is mediated through language, users have less direct manipulation of the underlying information space and may have to infer the system’s current interpretation from sequential conversational outputs rather than from visual structures and explicit controls [59]. This may reduce inspectability, make cross-checking harder, and constrain the simultaneous consideration of multiple cues.

By contrast, GUIs –and dashboards as such– offer persistent overview, simultaneous access to multiple information elements, and greater support for direct manipulation. These properties can be advantageous in decision-making contexts where users need to compare alternatives, inspect several indicators at once, or maintain awareness of the broader information space [56], [58]. However, these same properties can also increase cognitive demands when the display is dense, when many elements compete for attention, or when users must determine for themselves which information is relevant to the task at hand [60].

Overall, the two interaction styles are therefore expected to redistribute cognitive costs differently, thus shifting workload in different ways. As suggested in a previous exploratory study, the conversational interface may reduce workload by compressing access to relevant information and guiding the interaction, whereas the dashboard may better support visual inspection, overview, and explicit comparison. These competing mechanisms suggest that differences in subjective MWL should be more pronounced than differences in decision accuracy. In particular, decision accuracy may differ between interfaces, but the direction of this difference is difficult to predict a priori because each interface supports different aspects of the decision process. By contrast, completion time may favour the conversational interface overall, insofar as prior comparative work suggests that conversational systems can reduce upfront learning demands and limit the amount of information that must be processed at each step [58]. We therefore hypothesize that:

3.1.0.2 Interface \(\times\) task complexity

As objective task complexity increases, participants must process, retain, and coordinate a larger set of information cues. One possible expectation is that, if the LLM-based CUI functions as a cognitive assistant by partially automating information access and integration, its advantage should become more pronounced at higher complexity levels. However, prior work on Natural Language Interfaces (NLI) and CFT, together with findings from our first exploratory study, suggests that this expectation may be overly simplistic. In repeated use, initial advantages of conversational interaction may diminish relative to graphical interfaces [58]; likewise, earlier work comparing textual and visual interfaces found that performance advantages can shift as task complexity increases [26]. This is also consistent with later CFT accounts, which suggest that representation effects are contingent not only on task–format fit, but also on task complexity and the resulting cognitive processing demands [25], [61].

Consistently, our first exploratory study suggested that interface differences were task-dependent rather than uniform across levels of demand. As tasks become more demanding, the benefits of NL interaction may be offset by the need to revisit prior outputs, compare multiple cues across turns, retain intermediate information in memory, and verify answers whose basis is not directly visible. In such situations, the persistent overview, simultaneous visibility of information, and greater inspectability provided by GUIs may become increasingly advantageous. Taken together, these mechanisms suggest that, even if the LLM-based CUI offers advantages at lower levels of task demand, its benefits may diminish as complexity rises. We therefore predict that:

3.1.0.3 Interface \(\times\) data literacy

DL is expected to moderate interface effects because it shapes users’ ability to interpret, evaluate, and use data in decision-making contexts [39], [40]. Prior Information Systems research also links DL to how users experience data-intensive work, including perceived overload, and performance-relevant outcomes [5]. Because the dashboard requires more self-directed data interpretation and integration than the chatbot, lower DL should be penalized more strongly in the GUI condition, increasing the interface gap at low DL. Conversely, higher DL should enable individuals to better leverage the dashboard’s visualizations and data manageability, attenuating the interface gap and potentially reversing it for decision accuracy. We therefore predict that:

3.2 Secondary hypothesis: objective task complexity↩︎

We also test the direct effect of objective task complexity on subjective and objective indicators of information-processing demands. This secondary hypothesis functions as a manipulation-oriented check: it assesses whether the task complexity manipulation, designed according to Wood’s model, captures meaningful differences in perceived and behavioural task demands. This is important for interpreting the moderation hypotheses, because any \(interface \times complexity\) interaction is more informative if the complexity levels correspond to substantively different levels of task demand.

Objective task complexity theory suggests that more complex tasks require users to process more components and coordinate more relations among them [34]. Consistently, overload accounts predict that increasing information-processing demands should increase subjective workload and impair performance when they exceed available cognitive resources [17], [62]. We therefore expect higher task complexity to increase MWL, reduce decision accuracy, and extend completion time independently of the interface used:

3.3 Exploratory hypothesis: reliance↩︎

Finally, we examine intended reliance as an exploratory post-use outcome. Reliance is commonly treated as a behavioural consequence of trust in automation, but it is shaped by several factors, including perceived system reliability, task difficulty, confidence, and contextual demands [49], [51]–[53].

The direction of this effect is not specified a priori because the two interfaces expose different reliance trade-offs. A previous exploratory study conducted by us suggested that a LLM-based CA can reduce interactional effort, but may also make the evidential basis of its answers less inspectable. Conversely, a dashboard may impose greater interactional effort while preserving a more visible path between data, interpretation, and decision. Since intended reliance may be shaped by both perceived support and perceived verifiability, the expected direction of the interface effect remains open.

3.4 Research model↩︎

Figure 1 summarizes the research model. The study examines how determinants of information processing demands—interface type and objective task complexity—affect decision outcomes and perceived cognitive demands in a manufacturing decision-making scenario. Interface type is treated as the representational and interactional condition through which these demands are managed. In line with the IO and MWL perspectives outlined above, we treat objective task complexity as an experimental manipulation of information load: higher-complexity tasks require processing and coordinating more information cues to reach a decision. DL is measured prior to the tasks and treated as a pre-existing individual difference. Outcomes are captured through (i) subjective MWL and (ii) decision performance, operationalized as decision accuracy and completion time. Additionally, intended reliance is examined as an exploratory post-use outcome.

4 Method↩︎

4.1 Participants↩︎

The number of participants required to test the hypotheses was determined by conducting power analysis tests on the data collected during the exploratory study. Participants were recruited via Prolific using pre-screening criteria: (1) employment in manufacturing or closely related industrial sectors, (2) managerial role (junior, middle, or upper management), (3) decision-making responsibilities related to operations/production, supply chain/logistics, or business strategy, (4) fluency in English, and (5) residence in one of the targeted manufacturing economies¹. In total, \(N=313\) individuals started the study. Role seniority, industry, and decision responsibilities were re-assessed within the study to verify eligibility and to characterise the sample and participants whose in-study responses violated core eligibility criteria were excluded (\(N= 79\)), leaving \(N=234\) participants who entered the main study. Of these, \(N=43\) returned their submission prior to completion and \(N=6\) timed out. Among completed submissions, participants were excluded for non-compliance with study instructions (\(N=20\)), failing comprehension/attention checks (\(N=24\)), technical issues (\(N=5\)), or very low-effort responding evidenced by lack of verifiable task/interface engagement and repetitive questionnaire response patterns (\(N=2\)). The final analytic sample comprise \(N=134\) participants. Sample characteristics are reported in Table 1. Participants received £12 base compensation, with a performance-based bonus of up to £1.50 (maximum total £13.50). Participants excluded due to technical issues received partial compensation (£6) to account for time spent, while those who were screened out received £0.10. The median completion time was 26:22 minutes, corresponding to an average base reward rate of £27.37/hr.

The study protocol was approved by the Bioethical Committee of the University of Pisa (approval no. 39/2025). Participants provided informed consent, could withdraw at any time without penalty, and data were stored securely in pseudonymized form on University of Pisa servers, in accordance with European General Data Protection Regulation (GDPR).

4pt

4.2 Experiment Design↩︎

The study used a 2\(\times\)​3 mixed experimental design with a measured moderator:

Interface type is a between-subjects factor with two levels: a visualization-heavy dashboard and a conversational interface. Drawing on CFT, the two interfaces offer different external representations and interaction styles for accessing and integrating information, which should translate into differences in perceived workload and decision performance even under comparable objective information demands.

Task complexity is manipulated within subjects by varying objective complexity, primarily through Wood’s component and coordinative dimensions. Tasks are administered from lower to higher complexity for the methodological reasons detailed in § 4.4.

Data literacy is measured prior to the experimental tasks and treated as a pre-existing individual difference variable. DL is included in the model for two reasons. First, DL is theoretically relevant: the ability to interpret, validate, and act on data is expected to shape both performance and perceived cognitive demands in data-intensive decision tasks. Second, DL was used to support balanced assignment across interface conditions, improving comparability between groups on a theoretically meaningful capability.

4.2.1 Condition assignment↩︎

Participants were assigned at the individual level to one of two interface conditions with a target 1:1 allocation ratio. Because recruitment was sequential and fully online, we used a covariate-adaptive minimization procedure [63] to reduce the risk of chance imbalances on individual differences in DL and interface familiarity, that were expected to be prognostic for performance and perceived workload.

Assignment was performed after the pre-task questionnaires. DL was operationalized as the participant’s mean score on the BDLI and binned into three strata (low/mid/high) using two pre-specified cut-points decided after the pilot (5.0 and 6.0, respectively). Second, interface familiarity was derived from a brief technology-familiarity questionnaire administered separately for dashboard and chatbot. For each interface, familiarity was computed as the sum of (i) frequency of use and (ii) self-rated capability; these two interface-specific familiarity scores were then combined into a scale-free Relative Familiarity Index (RFI): \[\mathrm{RFI} = \frac{f_{\mathrm{dashboard}} - f_{\mathrm{chatbot}}}{f_{\mathrm{dashboard}} + f_{\mathrm{chatbot}}},\] which ranges from \(-1\) (leans chatbot) to \(+1\) (leans dashboard), with \(\mathrm{RFI}=0\) when both are equal. The combination of DL (3 levels) and RFI (3 levels) yielded 9 composite strata.

At assignment time, the server computed current arm counts within each composite stratum (eligible participants only) and compared the two candidate allocations for the incoming participant. The preferred arm was the one that minimized the absolute chatbot–dashboard imbalance across all strata and overall totals, while ties were broken at random. To retain allocation unpredictability, the preferred arm was selected with a biased coin (\(p=0.80\)), otherwise the alternative arm was assigned [63]. The assigned condition and covariate values were stored server-side and returned to the client.

4.3 Materials↩︎

The study materials consisted of a custom web application, two experimental interfaces, and a synthetic manufacturing dataset.

4.3.1 Study web application↩︎

The study was delivered through a custom web-based platform implementing the full experimental flow. The back end was implemented in Python using FastAPI and a PostgreSQL database, while the front end was implemented in React. The platform served the two experimental interfaces, namely the chatbot and the dashboard, as dedicated components while keeping the remaining study elements identical across conditions (e.g., task instructions, scenario framing, navigation structure, and timing/transition logic). The application also implemented centralized logging to support data quality checks and post-hoc auditing (e.g., page visits, timestamps, and interaction events). Both interfaces provided read-only access to the same underlying manufacturing dataset and time window, ensuring that differences between conditions reflected the interaction style rather than differences in available information. Before deployment, the complete experimental platform, including both the conversational and graphical interfaces, was assessed during pilot testing to identify technical failures and interaction obstacles that could interfere with task completion independently of the intended comparison between interaction styles.

4.3.2 Conversational interface↩︎

The conversational condition consisted of an LLM-based chatbot embedded in the study web application corresponding to the refined version of the CA architecture described in a previous study [64] and based on OpenAI GPT-4o model. The chatbot was implemented as a separate FastAPI service receiving user prompts from the front end. Responses were generated by combining the LLM with a constrained set of domain-specific retrieval functions. These functions provided read-only access to the study dataset, including machine signals, alarms, and maintenance records, allowing the chatbot to ground its responses in the same information space available in the dashboard condition. Finally, all chatbot interactions were stored in the central database at the level of conversations and individual messages. This logging supported later inspection of participant interaction traces and data-quality checks.

4.3.3 Graphical interface↩︎

The graphical condition consisted of a custom analytics dashboard implemented in React and composed of reusable widgets for data exploration and decision support (e.g., filtering, drill-down, and time-series inspection). Although developed specifically for the experiment, its layout and interaction patterns were informed by selected task-relevant components of the industrial partner’s analytics platform. In particular, the experimental dashboard adopted comparable conventions for organizing operational indicators, navigating among views, applying filters, and inspecting temporal trends. This design choice was made to improve ecological validity by grounding the graphical condition in interface patterns developed for industrial data exploration, while also establishing a credible graphical comparator rather than an ad hoc baseline that could introduce unnecessary interface-specific issues. Experimental control was retained by implementing only the data, views, and analytical operations required for the study tasks. Dashboard interaction was monitored through the platform’s event logger, which captured user actions for objective behavioural measures and for identifying non-engagement or technical failures.

4.3.4 Simulated scenario data↩︎

Real machine Key Performance Indicator (KPI) traces were used exclusively to characterize plausible baseline ranges, correlations among indicators, and temporal patterns associated with day and night shifts and maintenance events. Based on these characteristics, a one-year synthetic dataset was generated within the same feasible operating envelope. Controlled perturbations and noise were introduced to reproduce realistic variability and selected deviations relevant to the study tasks. The simulated decision scenario was anchored to a fixed reference point: 5 November 2025 at 11:00. For every participant, the simulated manufacturing data available through either interface extended only up to this timestamp, such that all participants examined the same operational state and the same information history. The synthetic dataset and scenario timestamp were held constant across interface conditions, providing a common ground truth for scoring task outcomes and ensuring that comparisons were not affected by differences in the underlying data or temporal context.

4.4 Experimental Tasks↩︎

4.4.0.1 Task design goals

The task set was designed to emulate data-driven operational decision-making in a manufacturing setting while remaining solvable by proxy participants. Tasks were grounded in the study scenario and the same underlying dataset across conditions, so that any performance differences could be attributed to interface/individual characteristics rather than information availability. Consistent with prior work treating task properties as a key determinant of information-processing demands, we explicitly specified and quantified objective task complexity for each task (see §2.3) [34], [35].

4.4.0.2 Task set and structure

The tasks and success criteria were identical across interface conditions and participants were instructed to base answers exclusively on information available in the interface and scenario. Task prompts were iteratively refined during pilot testing to ensure clarity, solvability with the available information, and alignment with the intended decision-making construct. Terminology and the plausibility of task constraints were further informed by stakeholder feedback from prior field work.

Participants completed a fixed sequence of three tasks in increasing objective complexity (7 reports full prompts, complexity computation, and reference solutions criteria). Task order was held constant because tasks were designed as successive stages of a single analytic workflow: later tasks presupposed familiarity with the scenario, terminology, and interface elements introduced in earlier tasks. In this setting, counterbalancing task order would systematically vary practice and carry-over across tasks, complicating interpretation of task-level differences. Following prior similar research that fixed lower complexity tasks first to establish a baseline and reduce order-driven inflation/deflation of completion times across complexity levels [26], we used a low-to-high sequence.

To mitigate fatigue and motivational decline in this fixed-order protocol, the study was designed to remain short (median completion time \(<30\) minutes) and performance-contingent bonuses increased across tasks.

Each task was administered under a task-specific time cap to bound session duration and to standardize exposure to the interface, reducing unbounded exploration that could introduce large between-participant variability [65]. The countdown was not displayed during task execution to avoid introducing an additional exogenous time-pressure cue via a visible clock/countdown, which can itself alter performance and perceived pressure [66]. Instead, prior to each task we informed participants of the typical time required (based on pilot data) and applied a generous cap set above the pilot median completion time for that task (7.3 reports pilot times and caps). Participants received a brief warning shortly before expiry; if the limit was reached, a short grace period allowed final submission before the study advanced. Upon grace expiry, the study advanced automatically and the task was recorded as timed out.

Nevertheless, because complexity could be partially confounded with task position, any complexity-related trends should be interpreted with this limitation in mind.

4.5 Procedure↩︎

Figure 2 summarizes the study flow. Data collection was conducted between 10 December 2025 and 2 February 2026. Participants first viewed an introduction page and provided informed consent (Step 1). They then completed a screening questionnaire (Step 2) that repeated the Prolific pre-screening items to verify eligibility²; ineligible respondents were screened out and the study ended. Eligible participants completed BDLI and interface familiarity pre-study questionnaires (Step 3). The system then assigned participants to an interface condition using covariate-adaptive minimization (Step 3→4, see § 4.2.1). Next, participants read the study scenario and completed an interface tutorial explaining how tasks would be carried out (Step 4). Participants then completed a brief instructional tutorial on the post-task questionnaires (NASA-TLX and intended reliance questionnaire, Step 5), followed by comprehension checks (Step 6); failing these checks ended the study. In addition, attention checks were embedded at multiple points during the study (e.g., within questionnaires and tutorials), and failing twice terminated the study . Participants who passed proceeded to the main study, completing three tasks presented in fixed order of increasing objective complexity (Step 7a). Each task was first shown in a reading phase, during which participants could inspect the task instructions but could not yet interact with the assigned interface. After reading the task, participants clicked the I read the task button to indicate that they were ready to begin. At that point, the assigned interface became available and the task timer started. Participants then completed the task and submitted their answer through the task panel by clicking Submit & Next. When participants approached the task time limit, a grace period was applied, as described in § 4.4. Each task was followed immediately by the post-task questionnaires (Step 7b). After the third task and its corresponding post-task questionnaires, participants completed a final open-ended compliance check asking whether they had used any additional tools beyond those provided in the study interface. Participants were then redirected to a closing page thanking them for their participation (End). Where applicable, the page displayed any bonus payment awarded during the study before participants were returned to Prolific for completion confirmation.

4.6 Measures↩︎

4.6.0.1 Subjective Mental Workload

Subjective mental workload was assessed using the NASA Task Load Index (NASA-TLX). Participants completed the NASA-TLX after each task. Following current recommendations, we used the "Raw TLX" scoring procedure (i.e., no pairwise weighting): subscale ratings were aggregated into a single overall workload score by averaging across the six subscales [30], [31].

4.6.0.2 Decision Accuracy

Decision accuracy was computed by comparison with pre-specified ground-truth criteria (reported in 7.4). Because tasks yielded raw scores on different ranges, accuracy was normalized using Percent of Maximum Possible (POMP) scoring [67]. For each task \(t\), a raw score \(S_t\) was transformed into a normalized Decision Accuracy Score (DAS) as follows:

\[DAS_t = \frac{S_t - S_{\min,t}}{S_{\max,t} - S_{\min,t}},\]

yielding \(DAS_t \in [0,1]\). This normalization places task-specific accuracy scores on a common scale, supporting comparison across tasks.

Skipped responses or non-answers were assigned the minimum possible score when participants had engaged with the assigned interface during the task window but did not provide a valid solution. In these cases, the normalized score was therefore \(DAS_t=0\). By contrast, submissions showing no task-relevant interaction with the assigned interface were treated as non-compliance and excluded from the analysis (see § 4.5).

4.6.0.3 Task completion time

This outcome was measured as time-on-task for each task, computed from client-side timestamps recorded by the study web application. Time-on-task was defined as the elapsed time between (a) the participant clicking the I read the task button and (b) their final answer submission.

4.6.0.4 Data literacy

Participants’ DL was measured using the BDLI [46], composed of 12 items covering core DL capabilities (e.g., analysis, acquisition/quality assessment, and wrangling). Responses were recorded on a 7-point Likert scale (1–7) and aggregated into a composite BDLI score by averaging across items (higher scores indicate higher DL).

4.6.0.5 Intended reliance

Reliance on the assigned interface’s outputs was assessed after each task using three items adapted from [68]. Items were rated on a 7-point scale from Unlikely (1) to Likely (7). The primary intended reliance measure for hypothesis testing was the sole reliance item (“I would take an action or make a subsequent decision based solely on the information I received/gathered from the chat assistant/the dashboard”). Two additional items probed intended reliance conditional on external verification (seeking a colleague’s second opinion or verifying using other analytical tools).

5 Statistical Analysis and Results↩︎

Statistical analyses and results are presented by outcome for convenience. A summary of the hypotheses and support decisions is shown in Table [tab:hypothesis-summary]. The primary analyses reported below use the model families specified for each outcome. Supplementary robustness analyses and industry-role confounding checks are reported in 8 and 9. These analyses did not alter the main interpretation of the confirmatory results.

3.5pt

5.1 Effects on Mental Workload↩︎

Table [tab:results-mental-workload] presents the results of each hypothesis with MWL as outcome. All MWL hypotheses were evaluated with probit cumulative link mixed models (CLMMs) fitted to tlx_mean rounded to 5-point ordered categories, with participant random intercepts to account for repeated observations; adjusted comparisons were summarized with estimated marginal means and contrasts [69]–[71]. Although this ordinalization aligns the endpoint with the CLMM framework, categorizing a continuous TLX score may discard some information [72].

3.5pt

5.1.0.1 H1a

The hypothesis was tested with the additive CLMM specified above. Interface condition was the focal predictor, with task complexity included as an additive adjustment covariate. Both factors were treatment-coded, so the estimated marginal contrast captured the adjusted difference between the dashboard and chatbot conditions [71]. The dashboard-minus-chatbot latent contrast was 1.096, 95% CI [0.708, 1.485], \(z = 5.533\), \(p < .001\), indicating higher mental workload in the dashboard condition and lower mental workload in the chatbot condition. Because the interval excluded 0 and the effect had the predicted sign, H1a was supported.

5.1.0.2 H2a

The hypothesis was tested by comparing additive and interface-by-task complexity CLMMs with an omnibus Likelihood-Ratio Test (LRT) and then probing the interaction with model-based simple effects and exact DiD-like contrasts; follow-up tests were Holm-adjusted [71], [73]–[75]. Accordingly, the primary quantities used for visualization and interpretation were latent-scale CLMM estimated marginal means and latent simple effects. This scale is appropriate because the hypothesis concerns moderation of the model-implied interface effect, rather than raw mean differences on an interval outcome. The figure therefore displays the latent Estimated Marginal Means (EMMs) and the simple effects expressed as Dashboard \(-\) Chatbot, so positive simple effects indicate higher latent mental workload for the dashboard than for the chatbot. The omnibus interaction was significant, LR = 16.460, df = 2, \(p < .001\). On the Dashboard \(-\) Chatbot scale, the primary DiD for T3 versus T1 was \(-0.962\), 95% CI [\(-1.463\), \(-0.461\)], \(z = -3.764\), raw \(p < .001\), Holm-adjusted \(p < .001\). The additional follow-ups showed the same attenuation from T2 versus T1, DiD = \(-0.824\), 95% CI [\(-1.322\), \(-0.325\)], Holm-adjusted \(p = .002\), whereas the T3-versus-T2 contrast was not reliable, DiD = \(-0.138\), 95% CI [\(-0.625\), 0.348], Holm-adjusted \(p = .577\). The plotted simple effects were consistent with this pattern: T1 = 1.737, 95% CI [1.226, 2.248]; T2 = 0.913, 95% CI [0.425, 1.402]; T3 = 0.775, 95% CI [0.291, 1.259]. The negative DiDs indicate that the positive Dashboard \(-\) Chatbot workload gap moved toward zero as complexity increased, meaning that the chatbot workload advantage attenuated with task complexity (Figure 3); H2a was therefore supported.

5.1.0.3 H3a

The hypothesis was tested with an additive-versus-interaction probit CLMM for the interface-by-data literacy effect. DL was operationalized as the mean of the 12 BDLI items and grouped as low [1,5], medium (5,6], and high (6,7], respecting the pre-specified cut-points decided after the pilot (§4.2.1); although this facilitated interpretation, categorizing a continuous score may reduce information [76]. The omnibus interaction was not significant, LR = 4.152, df = 2, p = .125, and the prespecified high-versus-low DiD was -0.576, 95% CI [-1.812, 0.660], z = -0.914, raw p = .361, Holm-adjusted p = .361. Thus, neither the omnibus interaction nor the primary follow-up showed reliable evidence that the interface slopes changed across data-literacy levels, so H3a was not supported.

5.1.0.4 H4a

The hypothesis was tested with a CLMM adding task complexity to a model that already controlled for interface condition; task effects were assessed with an omnibus LRT followed by estimated marginal contrasts [71], [73]. The omnibus task complexity effect was significant, LR = 127.302, df = 2, p < .001. Follow-up contrasts confirmed the expected monotonic pattern: T2 versus T1, beta = 0.415, 95% CI [0.111, 0.720], p = .001; T3 versus T1, beta = 1.536, 95% CI [1.195, 1.876], z = 10.797, p < .001; and T3 versus T2, beta = 1.121, 95% CI [0.800, 1.441], p < .001. Together with the reported monotonic increase from T1 to T3, these results supported H4a.

5.2 Decision Accuracy↩︎

Table [tab:results-decision-accuracy] presents the results of each hypothesis with decision accuracy as outcome. All accuracy hypotheses were evaluated with population-averaged fractional-logit Generalized Estimating Equations (GEEs) because accuracy_score is bounded in \([0,1]\); GEE accounted for repeated observations within participants, and response-scale adjusted means and confidence intervals were obtained with participant-level cluster bootstrapping when marginal effects or contrasts were summarized [77]–[80].

3.5pt

5.2.0.1 H1b

The hypothesis was tested with the additive fractional-logit GEE specified above, controlling for task. The primary effect was expressed on the response scale as the adjusted difference in predicted accuracy, equally weighted across tasks, with uncertainty summarized by participant-level cluster bootstrap intervals [80]. Adjusted accuracy was slightly higher in the chatbot condition (0.756) than in the dashboard condition (0.730), but the chatbot-minus-dashboard difference was 0.026, 95% CI [-0.012, 0.064], \(z = 1.318, p = .188\), corresponding to OR = 1.180. Because the confidence interval included 0, H1b was not supported.

5.2.0.2 H2b

The hypothesis was tested with a fractional-logit GEE including an interaction between interface condition and task, to assess moderation [74]. The omnibus interaction was assessed with a Wald test and followed by prespecified Holm-adjusted DiD contrasts on the predicted-accuracy scale [75], [80]. Because accuracy is a bounded response-scale outcome, the figure reports predicted probabilities and probability differences, which directly represent the hypothesized chatbot advantage in interpretable accuracy units; logit-scale contrasts were retained for model-based inference.

The omnibus interaction was significant, \(\chi^2 = 7.635\), df = 2, \(p = .022\). The primary DiD comparing T3 against T1 was \(-0.109\) on the predicted-accuracy scale, 95% CI [\(-0.188\), \(-0.028\)], with a logit contrast of \(-0.979\), raw \(p = .010\), Holm-adjusted \(p = .029\). The additional DiDs were not reliable for T2 versus T1, DiD = \(-0.038\), 95% CI [\(-0.129\), 0.048], Holm-adjusted \(p = .348\), or for T3 versus T2, DiD = \(-0.070\), 95% CI [\(-0.164\), 0.033], Holm-adjusted \(p = .348\). The plotted simple effects showed the same attenuation pattern: the chatbot advantage was 0.077, 95% CI [0.019, 0.134], for T1; 0.038, 95% CI [\(-0.035\), 0.111], for T2; and \(-0.032\), 95% CI [\(-0.092\), 0.031], for T3. Predicted accuracies were 0.933 versus 0.857 for T1, 0.866 versus 0.827 for T2, and 0.470 versus 0.502 for T3 for chatbot and dashboard, respectively. The negative T3-versus-T1 DiD therefore indicated convergence of the interface slopes across task difficulty, meaning that the chatbot accuracy advantage shrank and reversed descriptively in the hardest task (Figure 4); H2b was supported.

5.2.0.3 H3b

The hypothesis was tested with a fractional-logit GEE including an interaction between interface condition and the three-level data-literacy grouping, while controlling for task. DL used the same grouping described for H3a. The omnibus interaction reached significance, \(\chi^2 = 6.218\), df = 2, \(p = .045\), but the prespecified high-versus-low DiD was 0.087, 95% CI [-0.014, 0.198], with a logit contrast of 0.564, raw \(p = .106\), Holm-adjusted \(p = .106\) [74], [75], [80]. Additional simple effects suggested that the chatbot outperformed the dashboard at low DL by 0.124 points, 95% CI [0.035, 0.216], Holm-adjusted \(p = .009\), whereas no reliable interface difference emerged at medium or high data-literacy levels. Thus, although the omnibus interaction suggested some overall non-parallelism, the prespecified high-versus-low contrast did not support the hypothesized moderation pattern. H3b was therefore not supported.

5.2.0.4 H4b

The hypothesis was tested with a fractional-logit GEE including task as the focal predictor and interface as a control covariate. Task effects were assessed with an omnibus Wald test followed by Holm-adjusted pairwise contrasts on the predicted-accuracy scale, with uncertainty quantified by participant-level cluster bootstrap intervals [74], [75], [80]. The omnibus task effect was significant, \(\chi^2 = 229.910\), df = 2, \(p < .001\). Adjusted predicted accuracy declined monotonically from 0.894 for T1 to 0.846 for T2 and 0.488 for T3. The T2-versus-T1 contrast was \(-0.048\), 95% CI [\(-0.094\), \(-0.003\)], Holm-adjusted \(p = .037\); the T3-versus-T1 contrast was \(-0.406\), 95% CI [\(-0.447\), \(-0.360\)], Holm-adjusted \(p < .001\); and the T3-versus-T2 contrast was \(-0.357\), 95% CI [\(-0.405\), \(-0.308\)], Holm-adjusted \(p < .001\). This monotonic decline in accuracy from T1 to T3 supported H4b.

5.3 Task Completion Time↩︎

Table [tab:results-completion-time] presents the results of each hypothesis with completion time as outcome. All completion-time hypotheses were evaluated with Gamma GEEs with a log link because completion time was strictly positive and right-skewed; interface and task effects are therefore interpreted as time ratios, and response-scale summaries were accompanied by participant-level cluster-bootstrap intervals [79]–[82].

3.5pt

5.3.0.1 H1c

The hypothesis was tested with the additive Gamma GEE specified above, controlling for task complexity. The primary interface effect was interpreted as a chatbot-to-dashboard Time Ratio (TR), so values below 1 indicate faster performance in the chatbot condition. Completion time was somewhat shorter in the chatbot condition (adjusted mean = 213.606 s) than in the dashboard condition (adjusted mean = 221.673 s), but the chatbot-to-dashboard TR was 0.964, 95% CI [0.857, 1.083], \(z = -0.622\), \(p = .534\). Because the interval included 1, H1c was not supported.

5.3.0.2 H2c

The hypothesis was tested with a Gamma GEE including an interaction between interface condition and task complexity, to assess moderation [74]. The omnibus interaction was assessed with a Wald test and followed by Holm-adjusted ratio-of-time-ratios contrasts [75], [80]. Because completion time is positive and right-skewed, the Gamma log-link model makes multiplicative effects the primary inferential scale. For interpretability, however, the figure displays predicted completion time in seconds and the corresponding Chatbot \(-\) Dashboard time differences; negative values indicate that the chatbot is faster. The omnibus interaction was significant, \(\chi^2 = 42.797\), df = 2, \(p < .001\). The primary DiD comparing hard against easy tasks yielded a ratio-of-time-ratios of 1.633, 95% CI [1.348, 1.977], with a log contrast of 0.490, raw \(p < .001\), Holm-adjusted \(p < .001\). The additional follow-ups showed the same attenuation from T2 versus T1, ratio-of-time-ratios = 1.681, 95% CI [1.428, 1.979], raw \(p < .001\), Holm-adjusted \(p < .001\), whereas the T3-versus-T2 comparison was not reliable, ratio-of-time-ratios = 0.971, 95% CI [0.818, 1.153], raw \(p = .736\), Holm-adjusted \(p = .736\). The plotted simple effects were consistent with this pattern: for T1, the chatbot was faster than the dashboard, difference = \(-37.014\) seconds, 95% CI [\(-52.286\), \(-21.521\)], TR = 0.687, 95% CI [0.585, 0.801]; for T2, the difference was 27.954 seconds, 95% CI [\(-0.945\), 57.011], TR = 1.155, 95% CI [0.995, 1.343]; and for T3, the difference was 41.076 seconds, 95% CI [\(-14.733\), 98.251], TR = 1.121, 95% CI [0.959, 1.311]. Predicted times were 81.166 versus 118.180 seconds for T1, 208.533 versus 180.579 seconds for T2, and 379.967 versus 338.890 seconds for T3 for chatbot and dashboard, respectively. Because the significant ratio-of-time-ratios contrasts exceeded 1, the interface slopes converged: the chatbot time advantage weakened as complexity increased (Figure 5). H2c was therefore supported.

5.3.0.3 H3c

The hypothesis was tested with a Gamma GEE including an interaction between interface condition and categorized DL while controlling for task. DL was entered using the same low, medium, and high BDLI groups described above. The omnibus interaction was not significant, \(\chi^2 = 0.212\), df = 2, \(p = .899\), and the primary DiD comparing high versus low DL yielded a ratio-of-time-ratios of 1.028, 95% CI [0.672, 1.557], with a log contrast of 0.028, raw \(p = .890\), Holm-adjusted \(p = 1.000\) [74], [75], [80]. Thus, neither the omnibus interaction nor the primary follow-up showed reliable convergence or divergence of the interface slopes across data-literacy levels, and H3c was not supported.

5.3.0.4 H4c

The hypothesis was tested with a Gamma GEE including task complexity as the focal predictor and interface as a control covariate; follow-up contrasts were interpreted as time ratios on the response scale. The omnibus complexity effect was significant, \(\chi^2 = 562.997\), df = 2, \(p < .001\). Adjusted mean completion time increased from 100.11 s for T1 to 194.09 s for T2 and 358.71 s for T3. The T2-versus-T1 contrast yielded TR = 1.939, 95% CI [1.763, 2.120], Holm-adjusted \(p < .001\); the T3-versus-T1 contrast yielded TR = 3.583, 95% CI [3.219, 3.973], Holm-adjusted \(p < .001\); and the T3-versus-T2 contrast yielded TR = 1.848, 95% CI [1.687, 2.012], Holm-adjusted \(p < .001\). Together with the reported T1-to-T2-to-T3 increase in completion time, these results supported H4c.

5.4 Intended reliance↩︎

Table [tab:results-reliance-primary] presents the intended reliance results. Item r1 assessed willingness to rely on the interface output alone, whereas r2 and r3 assessed willingness to rely after obtaining a colleague’s second opinion and after verification with another analytical tool, respectively. Thus, higher responses on r2 and r3 indicate greater willingness to rely under an additional verification condition, rather than greater unqualified reliance on the interface.

5.4.0.1 H5

The hypothesis was tested separately for each intended reliance item (r1–r3) using item-wise probit CLMM, because the responses were discrete ordered categories rather than continuous scores [69], [83]. For each item, the interface condition was entered as the focal fixed effect, with CUI as the reference level, and task complexity was included as an adjustment factor, with T1 as the reference level and T2 and T3 treatment-coded. Participant ID was modelled with a random intercept to account for repeated within-participant responses [84]. The primary test was the LRT of the interface main effect in the additive model. Dashboard–chatbot contrasts were interpreted on the latent ordinal-model scale, with positive estimates indicating higher item ratings in the dashboard condition and negative estimates indicating higher item ratings in the chatbot condition.

For r1, the interface effect was not significant, LR = 0.601, df = 1, \(p = .438\), with a positive but imprecise dashboard–chatbot contrast, \(\beta = 0.158\), 95% CI [-0.241, 0.557], \(z = 0.776\). For r2, the effect was also not significant, LR = 0.776, df = 1, \(p = .378\), with a similarly positive but imprecise contrast, \(\beta = 0.144\), 95% CI [-0.177, 0.465], \(z = 0.881\). For r3, the interface effect was significant, LR = 6.847, df = 1, \(p = .009\), and the contrast was negative, \(\beta = -0.514\), 95% CI [-0.898, -0.131], \(z = -2.627\).

Taken together, H5 was not supported on its primary intended-reliance item: interface type was not reliably associated with willingness to rely on the interface output alone. No reliable interface difference was observed either for reliance following a colleague’s second opinion. However, participants reported greater willingness to rely on the chatbot after verification with another analytical tool. This item-specific finding may indicate a possible exploratory difference in intended reliance under verification conditions, but it does not support H5 as originally specified.

6 Discussion↩︎

Although LLM-based CUIs are often expected to improve decision support in data-rich settings, our findings do not support a simple superiority account relative to dashboards. Instead, the results reveal a differentiated pattern across subjective and objective outcomes. Subjectively, the CUI was associated with lower mental workload, as measured by the NASA-TLX, but this advantage attenuated and disappeared as task complexity increased. Objectively, completion time showed a similar pattern, whereas decision accuracy did not. In that case, the chatbot outperformed the dashboard only in the first task, and the overall evidence did not indicate a consistent accuracy advantage for either interface. Thus, subjective and objective outcomes showed only partial convergence: the conversational interface reduced perceived effort and, to some extent, task time, but these benefits did not generalize to decision accuracy. Overall, the findings point to a conditional rather than general advantage of the CUI, one that calls for theoretical explanation in relation to task demands and interface characteristics.

6.1 The Relative Performance of CUI and GUI in Industrial Decision Support↩︎

Our results suggest that LLM-based conversational systems represent a substantive advance over earlier conversational interfaces for decision support, although this advantage is conditional rather than uniform across outcomes and task demands. In our study, the conversational interface was not overall disadvantaged in decision accuracy, yielded lower reported MWL than the dashboard, and showed time advantages under lower complexity.

6.1.1 Effects on Subjective Workload↩︎

Regarding subjective MWL, the conversational interface appeared to reduce the perceived mental cost of task execution relative to the dashboard, suggesting that interaction through the chatbot simplified information access and reduced the cognitive effort participants felt they had to invest (H1a supported). This interpretation is consistent with the qualitative evidence from our previous exploratory study, where participants described the chatbot as reducing the need to search across the interface and manually assemble the answer. It also differs from the findings of [85], where participants reported higher MWL in the CUI condition than in the GUI condition. One plausible explanation is that the conversational system evaluated in that study differed substantially from the LLM-based interface used here, since the study predates the widespread adoption of current genAI chatbots.

However, the advantage of the CUI was not uniform across task-demand levels. When task complexity was considered, the interaction model provided better explanatory power than the additive model alone. The overall direction remained unchanged, but the pattern across tasks showed that the CUI’s MWL advantage attenuated as complexity increased (DiD \(\beta = -0.962\)), consistent with H2a. As shown in Figure 3, MWL increased more steeply across tasks in the CUI condition than in the GUI condition. One plausible explanation is that the CUI imposed a lower initial onboarding cost, in accordance with the previous study and [58]: users could begin with NL requests and progressively learn task criteria through the dialogue, whereas the GUI required earlier orientation to views, controls, and visual conventions. As task demands increased, however, the initial advantage became harder to sustain. More information had to be retained, integrated, and verified during the interaction, and these requirements may have been better supported by the persistent external representation provided by the dashboard.

6.1.2 Effects on Objective Performance: Decision Accuracy and Completion Time↩︎

The objective performance indicators showed a more limited advantage for the CUI than the subjective workload results. For decision accuracy, we did not find an overall difference between the two interface experimental conditions. Thus, the evidence does not indicate a general accuracy advantage for either the LLM-based CA or the dashboard. Likewise, the simple CUI-versus-GUI contrasts within each task were not significant, indicating that neither interface showed a reliable accuracy advantage at any single task level (see 8). This result should not be interpreted as evidence of equivalence, but it suggests that the LLM-based CUI did not reproduce the accuracy disadvantage reported in earlier work on non-LLM conversational or text-based interfaces [26], [86].

At the same time, the significant interaction for H2b indicates that the null overall effect masked a complexity-dependent shift. The interaction model accounted for the data better than the additive model, suggesting that the relative accuracy of the two interfaces changed as task complexity increased. Specifically, the significant T3-versus-T1 DiD (DiD \(=-0.109\)) indicates that the CUI–GUI accuracy gap was 10.9 percentage points smaller in the highest-complexity task than in the lowest-complexity task. In other words, the descriptively favourable chatbot gap observed at low complexity did not hold at high complexity. By contrast, the T2-versus-T1 and T3-versus-T2 contrasts were not significant, so the results do not support a clearly stepwise attenuation across adjacent task levels. Rather, they support a broader low-to-high complexity attenuation of the chatbot’s relative accuracy advantage.

A similar pattern emerged for task completion time. In the additive model, we did not find a significant overall difference between the two interfaces (H1c not supported). When task complexity was included as a moderator, however, the results indicated that the CUI’s time advantage was conditional on task demands (H2c supported), similar to MWL and decision accuracy. The conversational interface was significantly faster than the dashboard in the easier task, but this advantage disappeared as complexity increased. Both the T2-versus-T1 and T3-versus-T1 ratio-of-time-ratios were significant, indicating attenuation of the chatbot’s relative time advantage. However, the T3-versus-T2 contrast was not significant, suggesting that most of the attenuation occurred once the task moved beyond the lowest-complexity level, rather than progressively increasing from medium to high complexity.

In summary, the decision accuracy and completion time results suggest that the CUI’s objective performance advantage was fragile. The chatbot appeared beneficial when the task could be handled through a relatively direct request, but this advantage weakened when participants had to coordinate more information. This is consistent with [87], who showed that perceived cognitive effort, time, and accuracy do not necessarily move together across task-representation conditions.

Descriptively, the pattern also suggests that further increases in task complexity might eventually favour the dashboard, although this interpretation remains tentative. One possible explanation is that repeated use becomes progressively more beneficial in the GUI condition, as users overcome the dashboard’s higher initial orientation cost and become more efficient with its persistent visual structure, a pattern also reflected in the narrowing MWL gap as complexity increased. A similar pattern was observed by [58] for completion time. However, more evidence is needed to establish whether the observed attenuation reflects a genuine crossover tendency at higher levels of task complexity.

6.2 Information Overload and Cognitive Fit↩︎

One broader interpretation of these findings is that LLM-based CUIs may mitigate information burden more effectively than earlier conversational systems, while still retaining some scaling constraints of conversational interaction. A common expectation is that modern conversational systems can reduce IO by digesting information and offloading part of the user’s information-processing burden. Our results are consistent with this expectation at lower complexity, where the chatbot was associated with lower MWL and faster completion times. However, as task complexity increased through a larger number of information cues and greater coordination demands, this relative advantage attenuated.

This suggests that the limiting factor may not lie only in the underlying intelligence of the system, but also in the representational nature of the interaction style. When the solution space becomes larger, visual interfaces may better support filtering, comparison, and verification because of their higher representational bandwidth and their ability to keep multiple cues simultaneously visible [26], [29]. Similarly, [88] found that visualizations can attract attention, support information extraction, and improve comprehension more effectively than text. By contrast, the sequential and transient nature of NL interaction makes these operations harder relative to simpler tasks. Accordingly, the present findings therefore suggest that LLM-based CAs can lower the cost of accessing information, but conversational interaction may still be less suited as increasing task complexity raises the information load that users must manage.

CFT [10] helps explain why the same increase in informational demands may have affected the two interfaces differently. In general terms, cognitive fit depends on how well a representation supports the processing required by the task: better fit reduces the mental transformation needed to produce a usable internal representation. In the easier task (T1), participants had to identify the day with the lowest consumption by comparing values across a limited set. This type of task could in principle benefit from the dashboard’s visual representation. Yet participants reported lower MWL and completed the task faster with the chatbot. One possible explanation is that, at this low level of complexity, any fit-related advantage of the dashboard’s visual representation was too weak to dominate the broader interaction cost of using that interface. [89] argue that, in simple tasks, users may not consciously recognize or accurately report the cognitive-effort differences assumed by CFT, even when such differences exist. In our case, subjective MWL in T1 may therefore have been shaped more by the chatbot’s lower interaction threshold and easier task initiation than by any subtle representational advantage of the dashboard.

Interestingly, the remaining tasks were analytical, and therefore predominantly symbolic according to CFT, yet this did not translate into a stable advantage for the CUI. This suggests that representation type alone was not sufficient to determine cognitive fit in our setting, and that task complexity moderated the effect of interface type. This interpretation is consistent with [61], who found that task complexity can affect the relationship between information presentation format and decision performance. In line with Vessey’s extension of CFT [25], increasing complexity within an analytical task may alter the relative advantage of different representations by increasing the amount of analytical evaluation required. As these demands grow, users may cope more effectively by relying on a more perceptual representation. Our study did not allow participants to freely choose their representation or strategy, since interaction style was experimentally constrained. Nevertheless, the observed attenuation of the CUI advantage, together with the descriptive crossover in objective performance from the medium-complexity task onward, may signal a difference in coping mechanisms between the two interaction styles, consistent with Vessey’s prediction.

6.3 Data Literacy, Intended Reliance, and the Conditions of Effective Use↩︎

The DL analyses did not provide consistent evidence that participants benefited differently from the two interfaces. We expected lower DL to disadvantage participants more strongly in the dashboard condition, but this pattern was not reliably supported across outcomes (H3a–c). DL did not significantly moderate the interface effect on MWL or completion time. For accuracy, the omnibus \(Interface \times Data\;Literacy\) interaction was significant, indicating some non-parallelism across data-literacy groups; however, the prespecified high-versus-low contrast was not significant, and therefore the hypothesized moderation pattern was not supported. The result should thus be interpreted as an exploratory indication that DL may have shaped accuracy in some way, although the specific form of this effect remains unclear.

However, this does not imply that user capabilities are irrelevant, but it suggests that the present operationalization of DL may have been too broad. The BDLI offered a pragmatic self-report measure of general DL confidence, while the dashboard condition may depend more directly on narrower and objectively measurable abilities, such as visualization literacy: the ability to read, interpret, and use graphical representations to answer data-related questions [90], [91]. Future work should therefore use more task-specific and performance-based measures, distinguishing, for example, visualization literacy for dashboard use from linguistic and prompting abilities for CUI use.

A similar qualification applies to intended reliance. The primary item, which assessed whether participants would make a subsequent decision based solely on the information provided by the assigned interface, showed no reliable interface effect. Thus, the lower workload observed for the CUI did not translate into greater willingness to rely on it as a standalone basis for decision-making. The item concerning reliance after seeking a colleague’s second opinion also showed no reliable interface effect. The only significant interface effect emerged for the third item, where reliance was explicitly conditional on verification with another analytical tool. This contrast is informative: participants in the CUI condition did not report higher sole intended reliance than participants in the dashboard condition reported for their respective interface (r1), but they did report an higher intended reliance when the chatbot output could be checked against another analytical source (r3). This suggests that external verification may be more important for conversational decision support than for dashboard-based interaction, and that in decision making contexts a CA may not be perceived as sufficient when used as a sole source of evidence.

This interpretation aligns with work showing that users’ willingness to rely on algorithmic or AI-based support can depend on opportunities for control, adjustment, or independent checking [92], [93], and on users’ ability to evaluate the system’s output [53]. It is also compatible with recent evidence that efficient AI assistance can improve task performance while raising concerns about accountability and over-reliance when users receive direct answers without sufficient safeguards [94]. In the present study, however, reliance was measured as an antecedent-oriented intention, not as behavioural reliance or as evidence of appropriate reliance, as research literature usually do. The intended-reliance instrument was also newly adapted and has not yet been psychometrically validated. These results should therefore be treated as exploratory, while still offering useful indications for future research on reliance and related factors in LLM-based decision support.

6.4 Objective task complexity effects on experimental outcomes↩︎

The results also provide evidence that the task complexity manipulation behaved as intended. Following Wood’s model [34], task complexity was operationalized as an objective property of the task rather than as a subjective feeling of difficulty. The underlying assumption is that increasing objective demands should translate into greater subjective and behavioural difficulty [23]. The analysis therefore tested whether this objective manipulation predicted systematic changes in the main experimental outcomes after controlling for interface condition.

Across all three outcome families, the pattern was consistent. As expected, subjective MWL and completion time increased monotonically from T1 to T2 to T3, while decision accuracy showed the opposite pattern. Higher objective task complexity was therefore associated with higher MWL, lower accuracy, and longer completion time, supporting H4a–H4c. This pattern suggests that the task design captured behaviourally and perceptually meaningful differences in complexity. In this sense, Wood’s framework was useful both as a design rationale for constructing the experimental tasks and as a predictive framework for anticipating how task demands would affect user performance.

At the same time, the effects were not perfectly linear across outcomes. The largest accuracy loss occurred between T2 and T3, while completion time increased substantially at each step. This suggests that increasing complexity may initially be absorbed through additional time investment, but beyond a certain point it begins to affect decision accuracy more sharply. This interpretation is consistent with the intended manipulation of information load: the progressively increasing levels of objective complexity were specifically designed to move participants closer to the IO threshold. The observed pattern therefore supports the role of task complexity as a mechanism through which information load becomes visible in measurable performance outcomes.

6.5 Copy-Paste Use as a Task Strategy↩︎

A possible behavior that could have simplified tasks for participants assigned to the CUI condition is copy-pasting. Whereas dashboard users must reformulate the given problem through GUI interactions, chatbot users can copy-paste parts of the task request directly. In our data, this behavior was adopted (partly or fully in at least one task) by 29 CUI participants, while 35 did not use it. As reported in the 10, copy-paste use was not associated with significant differences in subjective workload (NASA-TLX) or accuracy. In contrast, it was associated with significantly faster completion times overall (about 59 seconds faster on average; moderate effect size, \(d \approx -0.71\)), with the strongest effect in Task 3 (Holm-corrected \(p=0.015\)). Overall, these results suggest that copy-paste primarily acts as a time-efficiency strategy rather than a mechanism for reducing workload or improving accuracy. Because this comparison was observational, it should not be interpreted causally. Nevertheless, the association indicates that copy-paste use may explain part of the CUI time advantage, while it does not appear to account for the MWL or accuracy patterns.

6.6 Implications↩︎

6.6.1 Theoretical Implications↩︎

A first theoretical implication of this study concerns CFT and its relevance in the era of genAI. The study incrementally extends CFT research in two ways. First, it applies the theory to LLM-based CUIs, understood as symbolic, language-based interfaces for decision support. In doing so, it shows that the theory remains analytically useful when information is represented and mediated through NL. Second, it connects this application to objective task complexity, adding evidence on the relationship between task demands and cognitive fit.

A related implication concerns the value of treating task complexity as an objective property of the task, rather than only as a post-hoc subjective judgement of difficulty [23], [34]. Defining task complexity a priori can make comparisons between heterogeneous interfaces more theoretically grounded, because observed differences in performance may otherwise reflect uncontrolled differences in what the task required. This does not make tasks interface-independent in an absolute sense, since the interface can still reshape how users act on the task. However, it helps distinguish between the demands built into the task and the additional costs or benefits introduced by a given representation or interaction design.

Objective complexity may also support stronger cross-study comparison in HCI. Systematic reviews and meta-analyses often classify tasks qualitatively, using broad labels such as simple and complex. These labels are useful, but they are often too coarse to determine whether different studies are testing comparable task demands. As discussed by [35], a quantitative or semi-quantitative task complexity specification could improve how studies are grouped, compared, and synthesized.

This implication has one caveat. Users may exercise discretion in how they complete a task, and high discretion can make the same nominal task produce different effective complexities because users may choose different resolution paths [33]. This is especially relevant when tasks contain dynamic components because changing task states can alter the number and sequence of operations required. In this study, this component was intentionally fixed to zero and excluded from the complexity computation (see Appendix 7.2). Future HCI studies using objective complexity should therefore report the procedure used to estimate it and the degree of discretion allowed during task execution.

6.6.2 Practical Implications↩︎

Our findings suggest that LLM-based CUIs should not be treated as straightforward replacements for dashboards. Instead, interface choice should be aligned with task demands. The CUI was particularly effective when tasks required bounded information access. In such cases, NL interaction can reduce navigation costs and allow users to obtain relevant information without manually searching across multiple views. However, the weakening of the CUI’s advantage as task complexity increased suggests that dashboards remain important for more demanding decision tasks. When users must compare several cues, the persistent visual representations offered by dashboards can still provide practical advantages. GUIs make information spatially available, support direct manipulation, and provide higher representational bandwidth than a pure CUI. Organizations should therefore be cautious about treating LLM-based assistants as general-purpose substitutes for graphical decision-support systems.

A more promising design direction is therefore a hybrid CUI–GUI interaction modality. In such systems, a conversational assistant could answer a user’s initial question, identify relevant indicators, and direct the user to the corresponding dashboard view or visualization. Conversely, the dashboard could provide the visual context from which users launch more specific NL queries. This would allow each modality to support the part of the task for which it is better suited. However, further research is needed to understand when such hybrids outperform individual modalities, and when they may instead introduce additional coordination costs.

6.7 Limitations↩︎

This study involved several methodological trade-offs, partly due to the nature of the constructs under investigation, which are theoretically and operationally contested, and partly due to the constraints of an unmoderated online experimental design.

First, DL was measured using a brief self-report questionnaire (BDLI) rather than a comprehensive performance-based assessment. This choice was pragmatic, given the lack of a single agreed definition of data literacy and the need to keep the study short. Moreover, the population under investigation (i.e., industrial managers) may have presented very similar levels of DL, which could have levelled off BDLI scores. Another possibility is related to the exaggerated perception that participants may have had of their own capabilities [95]. As a result, these factors may have contributed to the limited explanatory role of DL in the present results.

Second, MWL was measured through NASA-TLX. Although NASA-TLX remains widely used in HCI, mental workload is still conceptually contested [23], and recent work has questioned whether NASA-TLX fully captures the construct in contemporary interactive systems [22], [96]. Because the study was administered online, physiological measures such as eye tracking or pupillometry could not be included. The workload results should therefore be interpreted as perceived or subjective MWL, not as a complete measure of the cognitive processes involved during task execution.

Third, decision accuracy is easier to define for well-defined tasks than for more ill-defined decision tasks. For example, in Task 1, the request to identify the highest number in a set of numbers logically implies and allows for a definite solution. Conversely, Tasks 2 and 3 depend on a variety of context-dependent constraints and situations. While the logic of identifying the highest value is straightforward and easily communicated to participants, the more complex tasks did not have an equally determinate solution. For this reason, to make scoring possible, we applied a trade-off: the simplest and most explicit task wording was chosen and the solution set was defined using the simplest defensible scoring rules. However, other reasonable solutions may still have existed. Therefore, decision accuracy results should be read as alignment with our scoring model rather than as absolute correctness.

Finally, intended reliance was measured with ad hoc items developed for this study. Although these items were informed by the Theory of Planned Behavior [68], they were not psychometrically validated. For this reason, we analysed them separately and did not compute a composite reliance score. The reliance findings should therefore be considered exploratory, as stated in the related hypothesis.

Overall, these limitations delimit the interpretation of the interface comparison, and the results should be understood as evidence about the two tested interfaces under the specific task, measurement, and scoring choices adopted in this study.

6.8 Conclusions and Future Works↩︎

This study examined whether an LLM-based CAs can provide an effective alternative to dashboards for industrial decision support. The findings do not support a general replacement account. In fact, although the conversational interface reduced perceived mental workload overall and offered efficiency advantages in less demanding tasks, these benefits attenuated as task complexity increased. Moreover, it did not produce a consistent advantage in decision accuracy, and its completion-time advantages diminished as task complexity increased. The expected moderating role of general self-reported data literacy was also not supported, and the conversational interface was not preferred as a sole basis for subsequent decisions.

These results suggest that LLM-based conversational interfaces can improve access to industrial data by reducing interactional and navigational demands, but their value depends on the demands of the task. Therefore, rather than treating conversational systems as substitutes for graphical decision-support tools, future industrial interfaces should investigate hybrid designs that combine conversational access with visual overview and verifiable evidence. Future research should also examine whether the patterns observed in this study generalize beyond industrial settings by extending comparisons between conversational and graphical interfaces to other domains, such as healthcare, and to broader populations beyond professionals.

CRediT authorship contribution statement↩︎

Roberto Figliè: Writing – original draft, Writing – review & editing, Visualization, Validation, Software, Project administration, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Simone Caputo: Writing – original draft, Writing – review & editing, Methodology, Formal analysis, Data curation, Visualization, Conceptualization. Alan Serrano: Writing – review & editing, Conceptualization. Daria Mikhaylova: Writing – review & editing. Tommaso Turchi: Writing – review & editing, Supervision, Project administration. Daniele Mazzei: Supervision, Project administration, Funding acquisition.

Declaration of competing interest↩︎

One author holds a senior leadership role at Zerynth. All other authors declare no competing interests.

Funding↩︎

This work has been partially funded by Programme Erasmus+, Knowledge Alliances, application no. 621639-EPP-1-2020-1-IT-EPPKA2-KA, PLANET4: Practical Learning of Artificial iNtelligence on the Edge for indusTry 4.0. This work was also supported by the National PhD Programme in Artificial Intelligence funded under Italy’s PNRR, NextGenerationEU (CUP I51J22000590007). Zerynth provided cloud compute resources enabling access to GPT-4o via Microsoft Azure. Selected layout and interaction patterns of the graphical interface were informed by Zerynth’s industrial analytics platform. The sponsors had no role in the study design, collection, analysis, or interpretation of data, writing of the report, or the decision to submit the article for publication.

7 Task prompts, complexity and reference solutions↩︎

7.1 Task Prompts↩︎

7.1.0.1 Task 1 (low complexity)

On which day in May did the factory consume the most?

7.1.0.2 Task 2 (medium complexity)

From September to October, your factory has noticed excessive energy waste from machines sitting idle unnecessarily for too long. Management is considering installing an automatic power-down feature on one cutting machine to reduce energy costs. During a recent team meeting, it was decided that we should compare how long different machines sit idle and how much energy they draw while idle. Based on these observations, on which cutting machine would this feature have the greatest impact? To give the process crew some room for manoeuvre when planning operations, please also indicate your second-best choice.

7.1.0.3 Task 3 (high complexity)

Over the next 3 working days, we can deploy three specialist crews for one half-day each. Using data since the start of October, consider the period as a whole and pick three different machines, one for each crew to work on:

• Maintenance crew: choose the machine where reducing stops and downtime would most improve overall output and stability. Check: Availability, Alarm time and number of Alarms, Idle time, Offline time, Utilization rate.

• Process optimization crew: choose the machine where improving efficiency and flow would produce the biggest gain. Check: Performance, Average cycle time, Good cycles, Utilization rate, Consumption while working and Cost.

• Quality crew: choose the machine where improving first-pass yield would save the most good parts. Check: Quality, Bad cycles, Average cycle cost, Good cycles.

Assume safety is currently under control and comparable across machines. Constraint: to avoid disrupting operations and over-concentrating work on a single line, you must not select more than one machine from the same family (e.g., you can’t pick two Medium Cutting machines, or two Assembly machines).

7.2 Objective complexity↩︎

We operationalized objective task complexity using Wood’s task analytic model, which decomposes complexity into component complexity (acts and information cues), coordinative complexity (dependencies among acts), and dynamic complexity (changes in cue–product relationships over time) [34].

Component complexity (\(TC_1\)) reflects (a) the number of distinct acts required and (b) the number of non-redundant information cues, with repeated use of the same cue (or cues implied by others) counted once. Therefore for \(TC_1\), we summed the number of distinct cues required within each act across the task.

Coordinative complexity (\(TC_2\)) was operationalized as the number of required cross-act dependencies (e.g., integrating cues, applying conditional constraints, or comparing alternatives before selecting an answer) and was computed by counting directed dependencies between acts (e.g., one link per required dependency between acts \(i\rightarrow j\)).

Dynamic complexity was designed to be negligible by freezing the scenario time window and keeping the dataset static during each task. Accordingly, we set \(TC_3=0\) for all tasks.

Total complexity was computed as \[TC=\alpha TC_1+\beta TC_2.\] with \(\alpha=1\) and \(\beta=2\) as suggested by [34], assigning greater weight to coordinative than component complexity.

Table 2 reports the resulting complexity values for each task, computed from the finalized task scripts and interface instrumentation logs.

7.3 Tasks times and caps↩︎

Table 3 reports the per-task timing parameters implemented in the study web application well over the measured pilot median times. Therefore, for each task the system enforced a nominal time-cap. A single warning message was shown when the remaining time reached the pre-warning threshold (without showing a countdown). If the budget elapsed before submission, the task entered a grace period during which participants could still submit. Upon grace expiry, the study advanced automatically and the task was recorded as timed out.

3.5pt

7.4 Task decision reference solutions↩︎

7.4.0.0.1 Task 1.

For each day \(d\) in May, we computed daily consumption \(E(d)=\sum_{t\in d} c_t\). The task’s reference solutions are the full ordering of days obtained by sorting \(E(d)\) in descending order where the top-ranked day is \(d^*=\arg\max_d E(d)\).

7.4.0.0.2 Task 2.

Let \(\mathcal{M}\) be the set of eligible cutting machines in the September–October window. For each machine \(m\in\mathcal{M}\) we computed \[\begin{align} T_m=\text{total idle time (hours)}, \\ \qquad E_m=\text{total idle energy (kWh)}. \end{align}\] We then min–max normalized idle time and idle energy across machines: \[\begin{align} \tilde{T}_m&=\frac{T_m-\min_{k\in\mathcal{M}}T_k}{\max_{k\in\mathcal{M}}T_k-\min_{k\in\mathcal{M}}T_k},\\\\ \tilde{E}_m&=\frac{E_m-\min_{k\in\mathcal{M}}E_k}{\max_{k\in\mathcal{M}}E_k-\min_{k\in\mathcal{M}}E_k}. \end{align}\] and defined the single-machine impact score as \[I_m=\frac{\tilde{T}_m+\tilde{E}_m}{2}.\] The task’s reference solutions correspond to the ordering of ordered pairs \((m_1,m_2)\) (first and second choice, with \(m_1\neq m_2\)) by \[S(m_1,m_2)=0.6\cdot\frac{I_{m_1}}{\max_{k\in\mathcal{M}} I_k}\;+\;0.4\cdot\frac{I_{m_2}}{\max_{k\in\mathcal{M}} I_k}.\] Pairs were ranked by descending \(S(m_1,m_2)\); the top-ranked pair is the reference best+second-best response.

7.4.0.0.3 Task 3.

Let \(\mathcal{M}\) be the set of machines in the October–November analysis window, and let \(f(m)\) denote machine \(m\)’s family. We computed three crew-specific machine scores from window aggregates, using family-wise POMP scaling to map each indicator to \([0,1]\) (higher = stronger candidate within family): \[\mathrm{POMP}(x_m)=\frac{x_m-\min_{k:f(k)=f(m)}x_k}{\max_{k:f(k)=f(m)}x_k-\min_{k:f(k)=f(m)}x_k}.\]

Maintenance. Using availability loss \(1-\mathrm{Avail}_m\), downtime share \(\mathrm{Down}_m\), alarm rate \(\mathrm{AlarmRate}_m\), and utilisation rate \(\mathrm{UR}_m\): \[\begin{align} \mathrm{Maint}(m) ={}& \Big(0.40\,\operatorname{POMP}(1-\mathrm{Avail}_m) \\ &\quad + 0.40\,\operatorname{POMP}(\mathrm{Down}_m) \\ &\quad + 0.15\,\operatorname{POMP}(\mathrm{AlarmRate}_m) \\ &\quad + 0.05\,\operatorname{POMP}(\mathrm{UR}_m)\Big)^2 . \end{align}\]

Process. Using performance loss \(1-\mathrm{Perf}_m\), energy per good part \(\mathrm{EnergyGood}_m\), cost per good part \(\mathrm{CostGood}_m\), and utilisation rate \(\mathrm{UR}_m\): \[\begin{align} \mathrm{Proc}(m) ={}& \Big(0.35\,\operatorname{POMP}(1-\mathrm{Perf}_m) \\ &\quad + 0.35\,\operatorname{POMP}(\mathrm{EnergyGood}_m) \\ &\quad + 0.20\,\operatorname{POMP}(\mathrm{CostGood}_m) \\ &\quad + 0.10\,\operatorname{POMP}(\mathrm{UR}_m)\Big)^2 . \end{align}\]

Quality. Using quality loss \(1-\mathrm{Qual}_m\), scrap rate \(\mathrm{Scrap}_m\), cost per cycle \(\mathrm{CostCycle}_m\), and volume leverage \(\mathrm{Good}_m\): \[\begin{align} \mathrm{Qual}(m) ={}& \Big(0.30\,\operatorname{POMP}(1-\mathrm{Qual}_m) \\ &\quad + 0.40\,\operatorname{POMP}(\mathrm{Scrap}_m) \\ &\quad + 0.20\,\operatorname{POMP}(\mathrm{CostCycle}_m) \\ &\quad + 0.10\,\operatorname{POMP}(\mathrm{Good}_m)\Big)^2 . \end{align}\]

Machines were ranked globally for each crew by descending score (ranks \(r^{M}_m\), \(r^{P}_m\), \(r^{Q}_m\)), then mapped to utilities in \([0,1]\): \[u(r)=1-\frac{r-1}{N-1}, \qquad N=|\mathcal{M}|.\]

The task’s reference solutions correspond to the ordering of all admissible triples \((m,p,q)\) (maintenance, process, quality), subject to \(m,p,q\) being distinct machines and all from distinct families, by: \[\begin{gather} \mathrm{Score}(m,p,q)=\lambda\cdot\frac{u(r^M_m)+u(r^P_p)+u(r^Q_q)}{3}\\ +(1-\lambda)\cdot\min\{u(r^M_m),u(r^P_p),u(r^Q_q)\}, \end{gather}\] with \(\lambda=0.75\). Triples were ranked by descending \(\mathrm{Score}(m,p,q)\) and the top-ranked triple is the reference maintenance+process+quality assignment.

8 Robustness Analyses↩︎

This appendix reports robustness analyses for the tested hypotheses. The aim was to assess whether the conclusions reported in the Results section depended on the primary model families used there. Overall, the robustness checks led to the same substantive interpretation as the main analyses: the MWL, task complexity moderation, and task complexity main-effect hypotheses were supported, whereas the overall accuracy and completion time interface effects, the DL moderation hypotheses, and H5 were not robustly supported.

8.1 Mental Workload↩︎

The MWL robustness checks used participant-level non-parametric tests and ordinal-model alternatives to assess sensitivity to the primary CLMM analysis. The results were consistent with the main conclusions. Participant-level NASA-TLX ordinal-class sums were lower in the chatbot condition than in the dashboard condition, supporting H1a. The chatbot workload advantage was larger at T1 than at T3, and T3-minus-T1 workload increases were larger in the chatbot condition, supporting the attenuation pattern predicted in H2a. The robustness checks for H3a did not provide reliable support for the prespecified data-literacy moderation contrast. Finally, Friedman and paired Wilcoxon tests confirmed the expected monotonic increase in workload from easy to medium to hard tasks, supporting H4a. Table [tab:robustness-mwl-a] reports the detailed results.

4pt

8.2 Decision Accuracy↩︎

The decision accuracy robustness checks used participant-level and non-parametric alternatives to the primary fractional-logit analyses. The results were again consistent with the main conclusions. Participant-level mean accuracy was descriptively higher in the chatbot condition, but the uncertainty intervals included zero; H1b therefore remained unsupported. For H2b, the chatbot showed a clearer accuracy advantage at T1 than at T3, and the T1-to-T3 accuracy drop was larger in the chatbot condition, consistent with attenuation of the chatbot advantage as task complexity increased. The robustness checks did not support the prespecified data-literacy moderation contrast for H3b. For H4b, the Friedman test confirmed an overall task complexity effect, with accuracy declining most clearly in the hard task. Table [tab:robustness-accuracy-b] reports the detailed results.

4pt

8.3 Completion time↩︎

The completion time robustness checks used participant-level log-time analyses and non-parametric within-participant tests. The results matched the main interpretation. The chatbot was descriptively faster overall, but the uncertainty intervals did not support a reliable main interface effect; H1c therefore remained unsupported. For H2c, the chatbot was faster at T1, while this advantage disappeared at T3, and the T3-versus-T1 increase in log-time was larger for the chatbot than for the dashboard. This supports the attenuation pattern found in the main analysis. The robustness checks did not support the data-literacy moderation hypothesis H3c. Finally, Friedman and paired log-time contrasts confirmed a monotonic increase in completion time from easy to medium to hard tasks, supporting H4c. Table [tab:robustness-time-c] reports the detailed results.

4pt

8.4 Intended Reliance↩︎

As a secondary robustness analysis, we tested whether the interface effect on intended reliance varied by task complexity. For each item, an interaction model including interface and complexity was compared with the corresponding additive model using likelihood-ratio tests.

None of the omnibus interaction tests was statistically reliable: r1, LR \(= 4.590\), df \(= 2\), \(p = .101\); r2, LR \(= 2.918\), df \(= 2\), \(p = .232\); and r3, LR \(= 0.819\), df \(= 2\), \(p = .664\). Follow-up DiD -style contrasts were also imprecise and included zero. Thus, there was no evidence that the interface effect on intended reliance changed reliably across task complexity levels.

The interpretation of H5 therefore rests on the additive interface-effect models: no reliable interface difference was observed for r1 or r2, while r3 showed an item-specific difference indicating higher intended reliance in the chatbot condition. Taken together, the intended-reliance analyses do not support H5 as a general cross-item effect.

3.5pt

9 Industry-role confounding checks↩︎

This appendix reports checks in which participants industry role was included as a potential confounding variable. In all adjusted models, industry role was added as an additive covariate, with Junior Management as the reference category. It was not interacted with interface condition, task complexity, or DL. These analyses therefore test adjustment for possible confounding; they do not test whether the experimental effects differ across industry roles.

Preliminary diagnostics did not indicate substantial confounding. The sample included 21 Junior Management participants, 85 Middle Management participants, and 28 Upper Management participants. Role distribution did not differ reliably between interface conditions, \(\chi^2(2)=1.079\), \(p=.583\). Role was also not significantly associated with participant-level NASA–TLX, intended reliance, completion time, or accuracy in the diagnostic tests, with all Kruskal–Wallis tests non-significant and all ordinal role trends negligible.

Table 4 summarizes the adjusted models. Across outcomes, adding industry role did not change the substantive conclusions. The supported effects in the primary analyses remained supported after adjustment, including the mental workload interface effect, the interface-by-task complexity effects, and the task complexity main effects. The unsupported effects also remained unsupported, including the overall accuracy and completion time interface effects and the DL moderation hypotheses. The intended-reliance results also remained item-specific: r1 and r2 did not show reliable interface effects, while r3 showed higher intended reliance in the chatbot condition.

3.5pt

Overall, industry role did not show evidence of systematic confounding. It was not reliably imbalanced across interface conditions, was not clearly associated with the main participant-level outcomes, and did not alter the interpretation of the hypothesis tests after adjustment.

10 Supplementary copy–paste use checks↩︎

This appendix reports supplementary comparisons within the CUI condition between participants who used copy–paste and those who did not. Participants were classified as copy–paste users when their recorded interaction behaviour included at least one request classified as an exact or near copy–paste instance; all remaining CUI participants were classified as non-copy–paste users. The resulting groups comprised 29 copy–paste users and 35 non-copy–paste users.

Comparisons were conducted separately for NASA-TLX, accuracy, and completion time. For the overall comparisons in Table 5, outcomes were first averaged across available tasks for each participant, so that each participant contributed one value per outcome. Table 6 reports the corresponding comparisons separately for T1, T2, and T3. For each comparison, we report the mean difference between copy–paste and non-copy–paste users, a 95% percentile bootstrap confidence interval based on 10,000 resamples, Cohen’s \(d\) computed using the pooled standard deviation, and a two-sided permutation-test \(p\)-value based on 10,000 permutations. For the task-level analyses, permutation-test \(p\)-values were adjusted using the Holm procedure separately within each outcome across the three task comparisons.

5pt

4.3pt

References↩︎

---

1. The countries of residence were determined among the top 10 manufacturing economies, with the European Union considered as a single entity.↩︎

2. This procedure is suggested to ensure data quality by highlighting people who changed conditions (e.g., work, sector) since they took Prolific’s questionnaires, or to identify possible liars.↩︎