1 Introduction↩︎

Organizations perform business processes to deliver value-adding outcomes to their customers. Hence, Business Process Management (BPM) capabilities, such as process modeling, are a pivotal task in modern enterprises [1]. However, despite decades of efforts [2], process modeling still remains a costly activity due to, e.g., the difficulty of providing clear, up-to-date and easy-to-retrieve documentation [1] to those tasked to carry out such activities—the process modelers.

Inspired by recent developments in artificial intelligence (AI), such as large language models (LLM), researchers have proposed various techniques that can facilitate BPM-related tasks (e.g., [3]). Indeed, LLM can elaborate large collections of documents. Hence, by receiving an input from a given user, LLM can quickly produce an output that (i) accounts for existing documentation, while simultaneously (ii) answering the request of the user—i.e., a human. Yet, we found no evidence of practical applications of LLM for BPM in real contexts and, in particular, for process modeling. Hence, there is a need to investigate the effectiveness of such automation in industry [4]. Here, we tackle this challenge and showcase how a large enterprise, Hilti, can benefit from a LLM-powered chatbot—which we developed ad-hoc for Hilti—for BPM.

. We present a (the first) real-world case study showcasing the application of LLM for Process Modeling in operational contexts. Specifically, to advance the state of the art on BPM, we:

• describe the problems faced by the considered organization, Hilti, providing evidence of the necessities of modern enterprises (§2);

• carry out a requirement analysis by conducting interviews with Hilti’s employees, shedding light on the pain-points of professional process modelers (§3.1);

• use our interviews as a scaffold to develop an original LLM-based chatbot, (Fig. 1), designed to support Hilti’s process modelers (§3.2);

• evaluate the ability of to generate practical value by (i) having Hilti’s employees use and (ii) collecting and analysing their feedback (§3.3).

Our results (§4) show that is generally well-received, and identify room for improvement. We also derive lessons learned that future work can use to drive practical deployment of LLM-based technologies (§5).

2 Organizational Context and Problem Statement↩︎

Our case organization, Hilti Group, is a multinational company that was founded in 1941 in Schaan, Liechtenstein. It is a world market leader in fastening and demolition technology for construction professionals and provides tools, technologies, software and services to the global construction industry. In 2023, Hilti’s workforce consists of about 33.000 employees in more than 120 countries, making it a highly diverse, distributed organization that operates in complex and competitive markets all over the world. The size, complexity and business model of Hilti make it an ideal use case for testing the capabilities of LLM for process modeling: BPM is essential to ensure cooperation and consistent outcomes within Hilti’s ecosystem; furthermore, it is crucial for Hilti to optimize customer-facing processes. Hence, a smooth process modelling is pivotal for Hilti.

Challenge. Hilti has an extensive and heterogeneous documentation landscape which adds to the intrinsically complex nature of process modeling. Hilti’s employees spend abundant time searching through such documentation (our interviews revealed an average of \(\sim\)​40 minutes of search before modelling a process). Hence, to improve the productivity of their process modellers, and to actively explore innovative technologies, Hilti is interested in novel solutions that facilitate the routines of their employees.

Technological Gap. LLM-based solutions are common (in 2024). However, existing techniques cannot be applied to Hilti’s use case. This is because of the confidential nature of Hilti’s documents: publicly available models (e.g., ChatGPT) should not be able to access Hilti’s data to shape their responses; furthermore, even interacting with certain LLM (or their APIs) from within Hilti’s networks triggers warnings, preventing a reliable usage of these solutions—which are leveraged also by renown prior work, such as [3], [6]–[10].

3 Research and Methods↩︎

Inspired by Peffers et al. [11], we followed a Design Science Research (DSR) process consisting of four phases depicted in Fig. 2.¹ DSR is appropriate given our goal of examining LLMs for process modelling in organizations, as DSR emphasizes the creation of innovative solutions (in our case, ) while also considering the context in which these solutions will be applied.

3.1 Artifact Definition↩︎

As a preliminary step, we carried out a systematic literature review [13] which we used as a foundation to investigate the state of the art and define the scope of our project (see §2). Then, we carried out structured interviews [14] meant to identify pain-points and desiderata by professional process modelers² working for Hilti. We found an agreement with 10 employees, summarised in Table 1. The complete questionnaire is provided in our repository [15]. Among the most relevant questions, we ask: “what challenges do you experience when modelling processes?”, “how helpful is existing documentation when you model processes?” and “what would you like to see in a new AI artifact that supports process modelling?”; we also provide a list of functionalities for the AI artifact and ask to rate them on a 1–5 scale, as well as potential concerns. Finally, we inquire about the time spent looking for, and reviewing, existing documentation.

3.2 Artifact Implementation↩︎

We use the results of our interviews alongside those of our investigation of the state of the art to define the requirements of our technical artifact, i.e., the LLM-based chatbot . To develop , we rely on Botpress, a platform to build custom AI chatbots powered by GPT-based LLMs; for our prototype version of , we used GPT-3.5 Turbo, which we found provided satisfactory performance while also requiring less resources to generate an output. A crucial aspect of is its reliance on the BPMN Sketch Miner tool [5]. The syntax for this tool is entirely text-based, human-readable and light in terms of token consumption, making it appropriate for our case study. Therefore, we use few-shot prompting to teach to provide an output that matches the format expected by BPMN Sketch Miner. This output serves as the input for the model generation and transformation pipeline of BPMN Sketch Miner [16]. Such a design choice enables users of to directly paste the AI outputs into the online tool and get their model visualized (see Fig. 1).

Furthermore, we have leveraged retrieval-augmented generation (RAG) [17] to embed Hilti’s documentation into . Such documentation included: process descriptions from Hilti’s internal documentation repository (anonymised); and information about Hilti’s process management, and how to model processes at Hilti (taken verbatim from the learning platform for Hilti’s process modellers). These procedures enabled us to instill some knowledge about Hilti’s processes in —a functionality that was heavily endorsed by our interviewees.

3.3 Artifact Evaluation↩︎

We conducted a user study with our artifact and process modellers. Our aim was to answer evaluative questions on the quality of for Hilti.

First, the process modellers tested all functionalities of PRODIGY by creating custom prompts, with the intention of simulating their routine tasks. Their inputs and the corresponding AI-generated outputs are fully observable in our repository [15]. Then, we carried out semi-structured interviews during which the participants answered 28 questions. Among these, we ask to give an 1–5 rating to the statement “Using would make it easier for me to do process modeling tasks.” The complete questionnaire is provided in our repository [15].

3.4 Dissemination and Communication of the Results↩︎

To conclude our DSR process, we formalized our learnings and made them accessible to interested parties. We documented our observations, analyzed our findings, identified lessons learned, stated limitations, and recommended directions for future work. We shared our learnings within Hilti Group and the wider BPM community in academia and practice—some companies reached out to us and expressed their interest about the development process of .

4 Key Findings and Lessons Learned↩︎

We first summarise the major results of our user studies, and then outline our proposed “operating model” for our developed LLM-based chatbot, .

4.1 Preliminary Interviews: what do Hilti process modellers want?↩︎

These open interviews lasted for 60 minutes, and the results shed light on the pain-points and desiderata of our participants. We found that, before modelling a process, 60% spend between 15–60 minutes searching for documentation; and also 60% spend between 5–60 minutes to review such documentation. As a matter of fact, 90% state that it is “extremely important” that an LLM-based chatbot has access to Hilti’s documentation; however, we also found that, on a 1–10 rating (low to high) scale, the average usefullness of current Hilti’s documentation is 6.7—indicating helpfulness, but with huge margins for improvement. Nonetheless, with respect to AI-related concerns, some stated that “humans may misinterpret the AI’s outputs” or “AI may negatively impact collaboration with colleagues” or even about accountability (“the mindset that [the machine] does everything and we no longer have to worry about it is dangerous”).

4.2 Evaluation: what do Hilti process modellers say about ?↩︎

After letting our process modellers use , we collected their feedback via 90-minutes long semi-structured interviews; one participant to the preliminary interviews did not provide feedback since they were not available, so we obtained responses from nine employees. The general opinion was positive. Five of our participants asserted that they would use on a daily or weekly basis (i.e., whenever they have to carry out process-modeling duties). Moreover, six participants asserted that would speed-up their tasks (three remained neutral), and eight believe that makes their tasks easier (one remained neutral). Finally, we report in Fig. 3 the participants’ perception on the functionalities we integrated in , showing great appreciation.

4.3 Operating model: how should be used in practice?↩︎

During our implementation, we devised an operating model that describes how should be leveraged by real organizations; we have further refined our model (shown in Fig. 4) after receiving the feedback by our interviewees.

At a high-level, our model emphasizes continuous improvement through regular evaluations, feedback, and updates. To this end, our model delineates the interaction between a Governance Team (i.e., the set of employees within a company that oversee the development and maintenance of ) and a Process Modeller (i.e., the end-users of ). These two actors work collaboratively to ensure that the system performs well over time. For instance, the Process Modeller should be familiar with existing documentation and with the specific process, and scrutinize the response of accordingly; they should also be willing to provide feedback (collected in a dedicated repository) and receive guidance from the Governance Team—who must, in turn, define clear performance indicators for and periodically review the performance of prodigy (e.g., by analysing logs [18]) and apply updates if needed; as well as ensure that existing documentation is properly embedded in (in a timely manner).

5 Significance and Relevance in Research and Practice↩︎

Besides our key findings we underscore three orthogonal aspects of our research.

The perspective of process modellers in organizations. We coalesce the responses—not pertaining to AI—of our preliminary interviews, and derive an original framework representing dynamics of process modellers’ issues at Hilti. This is instructive because, during our literature analysis, we found some works mentioning pitfalls of process modeling (e.g., [19]) but without accounting for context. Our framework (displayed in Fig. 5, and described in the caption of Fig. 5) attempts to rectify this shortcoming, providing guidance for future work.

Evaluating LLM-/AI-based solutions. Upon further analysing the results of our evaluation interview, we have found that the reception of by our process modellers was highly dependant on their expectations and overall attitude towards AI and IT innovation. Indeed, some participants had “lower expectations” and provided prompts that were “more aligned” to the expected input of —and these participants rated more positively. In contrast, participants who were expecting that would “do their work for them” by issuing a single (and typically poorly phrased and/or unclear) prompt were more skeptical of ’s helpfulness. These results underscore the importance of (i) accounting for each end-user’s expectations while evaluating the performance of operational AI-based solutions; as well as (ii) educating end-users on the potential (and limitations) of AI-based tools.

The role played by higher education. With this paper, we (also) seek to bridge three domains: industrial practice, scholarly literature, and higher-education institutions [20]. This research has been predominantly carried by Clara Ziche for her MSc. thesis, during which she was working part-time at Hilti. The development of was driven by following the guidelines of prior academic literature, and the resulting artifact was appreciated by Hilti as well as by other companies that witnessed its capabilities. On this note, we find it instructive to trace the timeline of this research by outlining the path followed by Clara Ziche to bring our findings to light. In Sept–Dec 2023, after attending the BPM’23 conference, Clara investigated the state of the art and designed the interviews for the requirement analysis. In Jan 2024, Clara carried out the interviews, and began familiarizing with current LLM technologies. In Feb 2024, Clara developed and designed the questionnaire for its evaluation—which took place in March 2024. Disseminations occurred in Apr–May 2024.

6 Discussion: Scope and Limitations↩︎

We showcased an exemplary application of an LLM-based chatbot that can assist process modellers in a large organization, Hilti. In doing so, we have carried out a twofold user study with 10 employees of Hilti, and developed an original artifact, . Our research has a number of limitations. For instance, we do not claim that our findings can apply to other organizations—irrespective of their similarity to Hilti. Moreover, we cannot claim that even our own findings can apply to the entirety of Hilti: The participants of our user study are mostly based in Liechtenstein, and therefore cover the global headquarters perspective rather than regional and local perspectives. Furthermore, uses GPT-3.5 Turbo (which is not privacy-compliant), and it relies on BPMN Sketch Miner: if such a tool is taken down, the output of may lose its immediate usefulness. Finally, even our own participants have pointed out some shortcomings of , such as a poor “knowledge” of Hilti’s documentation. Such a result, however, was expected: the documents that has access to (with RAG) are just a drop in the deluge of files and logs included in Hilti’s databases (and we, as researchers, do not have complete access to such data).

7 Conclusions↩︎

We have presented the first case study showcasing how LLM can be used for process modeling in large enterprises—specifically, Hilti Group. We follow DSR guidelines and develop an original LLM-based chatbot, , which we test with professional process modellers from Hilti. Our findings revealed that end-users appreciate the functionalities of . However, concerns were raised about the poor alignment of ’s output with Hilti’s specifications. Such a shortcoming underscores the importance of integrating LLM-based solutions with the organization’s documentation—which is a task outside the responsibilities of process modellers. Hence, deployment of similar solutions in real contexts should be done with the support of the organization’s governance team: it is unrealistic to expect that “off the shelf” solutions work properly to drive the process modeling routines of complex and large organizations (see Fig. 6).

Acknowledgements. We thank Hilti for enabling and funding this research; and the participants to our user study for their contributions and feedback.

References↩︎