Keywords: quantum software testing, software infrastructure, empirical study

1 Introduction↩︎

With the advancement of quantum computing (QC), there is a growing need for high-quality quantum programs. Due to non-intuitive principles of quantum mechanics and increasing program scale, ensuring program quality and reliability remains challenging. As a solution in quantum software engineering (QSE) [@zhao2020quantum; @murillo2025quantum], quantum software testing (QST) [@10.1145/3377816.3381731] is a critical activity in the quantum software development life cycle (QSDLC), which assesses runtime behaviors of quantum programs and checks whether they satisfy given specifications.

Recently, empirical studies on QST have received growing attention. In such studies, benchmark quantum programs play a fundamental role in enabling fair and effective evaluation of testing approaches. Evaluating testing techniques requires benchmarks that go beyond small toy examples and cover programs with varying scales along with approximation to real-world scenarios. However, many existing QST studies still rely on fixed-size quantum circuits (i.e., low-level quantum programs in [@li2026methodological]) as programs under test (PUT). These programs are often written in hardware-oriented assembly languages such as OpenQASM or expressed in low-abstraction code, making them poorly scalable and rarely maintainable in a practical QSDLC. In contrast, scalable quantum programs (i.e., high-level quantum programs in [@li2026methodological]) employ richer abstractions and allow classical arguments to produce diverse quantum circuits. Their modular design more naturally follows software engineering (SE) practices, and such programs commonly appear in mainstream quantum software development kits (SDKs) such as Qiskit [@Qiskit-qiskit]. Therefore, the gap between current research benchmarks and practical programs may limit the applicability of existing QST techniques in practical program-level testing scenarios.

A recent study [@li2026methodological] underscores that prior QST studies rely on programs collected from diverse and fragmented sources. Even the two most frequently used benchmarks are only partially aligned with QST. Bugs4Q [@zhao2021bugs4q; @zhao2023bugs4q] includes many classical programs in quantum software stacks, while MQT Bench [@quetschlich2023mqt] was originally built for quantum hardware testing with quantum circuits as test inputs rather than PUTs. This purpose mismatch matters for software testing: when reused for QST experiments, such benchmarks provide limited support for functional specifications and standardized interfaces, making it difficult to design valid test cases and interpret test outcomes consistently. Therefore, without a benchmark tailored to QST, it remains difficult to conduct comprehensive, fair, and reproducible evaluations across studies.

Motivated by these limitations, we propose a benchmark-construction methodology for controlled QST evaluation on scalable quantum programs, and instantiate it as Qolumbina. It comprises 40 scalable Qiskit programs systematically filtered from 8 open-source repositories of programs with real-world provenance. To mitigate threats to testability and empirical validity arising from current quantum programming practices, including heterogeneous program interfaces and under-specified input constraints, the original programs are moderately refactored into standardized and executable benchmark subjects by improving interface compatibility, aligning language usage, and validating test inputs. To enhance the functional testability, we provide program specifications that document intended program behavior, together with unit tests as executable examples.

We further conduct a systematic empirical study to characterize the involved quantum programs and examine testing-relevant properties of scalable quantum programs. From the perspective of QST, we establish a set of criteria for analyzing program functionalities in terms of application domains and output characteristics. Empirical evidence reveals the functional diversity of Qolumbina based on the covered categories and indicates that the design of testing approaches should take such differences into account, although this issue has received limited attention in prior QST research. Then, we characterize program scale along both development and execution dimensions, using standard SE metrics overlooked in QST (e.g., lines of code (LOC)) and quantum-specific metrics (e.g., circuit width). The results show that, unlike fixed-size subjects, our benchmark programs instantiate circuits whose scale depends on classical inputs. Finally, we conduct two controlled experiments in both noiseless and noisy scenarios to examine the feasibility of using Qolumbina in QST experiments. By applying two recent QST approaches [@li2025preparation; @li2016dynamic] and aligning with their scopes, our experiments consider 23 programs with sufficient test suites for execution-cost analysis and 26 programs with 7 buggy variants each for fault-detection analysis. Experimental results demonstrate that Qolumbina can reproduce findings from prior studies and also provide insights into how fake backends affect QST result interpretation.

The main contributions of this paper are listed as follows:

• We present Qolumbina, a publicly available benchmark infrastructure for controlled QST research on scalable quantum programs. It contains 40 scalable Qiskit programs that are closer to practical quantum software development than the low-abstraction and circuit-like programs commonly used in prior studies.

• Instead of simply collecting programs, we propose a construction pipeline that turns open-source quantum programs into test-ready benchmark subjects through systematic selection, program refactoring, program specifications, unit tests, and standardized interfaces.

• We establish systematic QST-oriented taxonomies and criteria for characterizing program functionalities and scales, and also use them to validate benchmark diversity and scalability support of Qolumbina.

• We conduct two controlled experiments with recent QST approaches, demonstrating the feasibility of using Qolumbina and providing empirical findings for future QST research.

The remainder of this paper is organized as below. Section 2 introduces preliminaries of quantum programs and reviews related benchmarks. Section 3 presents the methodology for infrastructure construction. Section 4 reports the empirical study, and Section 5 discusses threats to validity. Section 6 outlines lessons for future research. Finally, Section 7 concludes the paper.

2 Background and Related Work↩︎

2.1 Quantum Programs and Their Design↩︎

Quantum programs are designed to implement quantum algorithms or subroutines. Especially for the mainstream gate-based quantum computation, the computational procedure of a quantum algorithm is realized by a sequence of quantum gates acting on qubits. The pure state of a qubit can be denoted as a state vector \(\ket{\psi}\), and a quantum gate can map the qubit state to \(U\ket{\psi}\), where \(U\) is mathematically a unitary operator. Regarding running quantum programs on physical hardware or their corresponding classical simulation, program outputs are not accessible until they are transformed into probabilistic outcomes through quantum measurement.

Today, quantum programming languages and frameworks, such as Qiskit, built on the host language Python, provide fruitful APIs for programmers to develop quantum applications. Following the practice of classical software engineering (CSE), these official APIs largely involve modularity and scalability, which requires appropriate code abstractions.

To compare code at different abstraction levels, Figure 1 illustrates two Qiskit examples, both of which realize the Greenberger-Horne-Zeilinger (GHZ) state preparation.

In the left code, each statement after circuit construction corresponds to a concrete quantum gate, but this hard-coded approach limits functionality to preparing a 6-qubit state only. By contrast, the right-hand example from Qolumbina incorporates higher abstraction for quantum gates and is scalable to prepare a general GHZ state, whose qubit number can be flexibly determined by the classical variable self._num_qubits.

Abstraction hides low-level implementation details behind high-level interfaces, supporting the modularity, reusability, and scalability needed as quantum systems grow in both size and complexity [@zhao2025when]. Nevertheless, prior QST studies have paid limited attention to scalable quantum programs with such abstractions, considering only a few exceptions including [@long2024testing; @long2025black; @li2025preparation; @li2016dynamic]. Hence, Qolumbina addresses this gap by enabling systematic evaluation of testing techniques on scalable quantum programs.

2.2 Benchmarks for Quantum Software↩︎

Recent studies have proposed benchmarks for quantum code, such as QuanBench [@10.1109/ASE63991.2025.00218] and QCircuitBench [@yang2026qcircuitbench], which are intended to evaluate large language models in code generation.

With a focus on underlying use for QST, Table ¿tbl:tab:32benchmark? lists more relevant open-source benchmarks covering QC and SE perspectives.

For the scope beyond assembly-level and fixed-size OpenQASM programs, Qiskit is the most popular framework, which motivates building Qolumbina on top of it.

In detail, Bugs4Q collects real-world bug-fix pairs in quantum programming, where many bugs do not result from unexpected behavior of quantum programs, but from misuse of quantum software stacks. Meanwhile, RevLib is an early repository of general reversible circuits for computer-aided design, rather than being dedicated to QC. The latest benchmark, QSimBench, is oriented toward QSE research on circuit-level objects and provides precomputed outcomes rather than program code. Since Table ¿tbl:tab:32benchmark? counts heterogeneous entities, benchmark sizes are not directly comparable, while Qolumbina still has the largest subject count among benchmarks whose primary components are quantum algorithms.

Several benchmarks [@quetschlich2023mqt; @lubinski2023application; @li2023qasmbench; @wille2008revlib] aim to evaluate performance of quantum hardware and its simulation. OpenQASM programs are common as they serve as an assembly-level intermediate representation that bridges high-level quantum algorithms and low-level hardware execution layers. Therefore, QC research uses these circuit-like programs as test cases to evaluate hardware performance characteristics such as fidelity under noise and entanglement capability [@li2023qasmbench]. Nevertheless, developers rarely construct applications at such a low level. QST focuses on program functionality and bug detection within the QSDLC . This distinction highlights the difference between testing quantum hardware and testing quantum programs.

Bugs4Q, MQT Bench, VeriQBench, QASMBench, and RevLib have been used for QST, but scalable quantum programs have been scarcely involved.

Benchmarks [@lubinski2023application; @tomesh2022supermarq] have not been explored for QST but include high-quality scalable programs, making them appropriate sources for Qolumbina to enhance program diversity. In addition, compared to many scalable programs included in existing benchmarks, which only consider input arguments directly mapped to qubit counts, Qolumbina also exposes the input ports of other classical arguments (e.g., the database to be searched in Grover Search), thereby enabling richer program configurations and input spaces for QST experiments.

3 Methodology of Benchmark Design↩︎

3.1 Pipeline of Benchmark Construction↩︎

Qolumbina is a benchmark infrastructure built on Qiskit 2.3.0 to support controlled and reproducible QST experiments. Figure 2 illustrates a pipeline for its construction and structure, with details provided in the following subsections. We first collect open-source quantum software repositories to identify scalable quantum programs. The selected programs are then moderately refactored to standardize interfaces and support valid test execution. Since many repositories lack clear descriptions of program functionality and usage, we consult accessible literature and tutorials to prepare QST-oriented documentation. Finally, we develop unit tests for each benchmark program and iteratively refine both the refactored programs and documentation based on test feedback.

3.2 Program Collection and Filtering↩︎

To promote program diversity while ensuring a reasonable selection scope, we consider three types of accessible sources for benchmark programs with real-world provenance:

• C1: Program sources or benchmarks mentioned or considered by QST studies

• C2: Artifacts of QST studies, which explicitly provide original PUTs

• C3: Relevant benchmarks cited by the publications that propose the benchmarks marked as C1

Specifically, we use real-world provenance to refer to traceable sources rather than deployment evidence. A program is considered to exhibit real-world provenance if it is claimed in the corresponding research paper, carries copyright information from a recognized quantum software project, or is hosted in an official quantum SDK repository like Qiskit. An existing survey [@li2026methodological] summarizes 15 C1 sources and 33 C2 sources. For C3, we newly extracted 2 sources, i.e., Lubinski et al.’s benchmark [@lubinski2023application] and SupermarQ [@tomesh2022supermarq], by revisiting the literature [@quetschlich2023mqt].

Then, we filtered out the candidate quantum programs that violated at least one of the following three criteria:

• F1: Programs scalable in both logic and structure, allowing classical arguments to flexibly determine quantum circuits

• F2: Programs accompanied by functional descriptions in their corresponding sources, or whose underlying algorithmic principles are elaborated on in accessible literature

• F3: Programs implementing a unitary operation of a quantum algorithm in a noise-free situation

F1 helps exclude programs that inherently represent fixed-size quantum circuits (e.g., preparing a Bell state with only two qubits), or heavily hard-coded logic (e.g., the left program in Figure 1). F2 is motivated by the assumption of testability, as test oracles would be infeasible to properly design or systematically derive in the complete absence of knowledge about expected behavior of the program. F3 follows the current focus on testing-related tasks, in which the evolution of the qubit within a quantum circuit is mathematically modeled as a unitary operation \(U\) [@li2024automatic; @oldfield2025faster; @miranskyy2025feasibility]. In fact, a general class of quantum algorithms involving mid-circuit measurements falls outside the scope described above. Nevertheless, the difficulty in accurately depicting the program specification, together with the scarcity of related QST studies [@paltenghi2024survey], makes it reasonable for Qolumbina to consider these cases as future extensions.

We completed program collection before February 15, 2026. Using F1–F3 as exclusion criteria, and after author discussion and consensus, we retained 40 deduplicated programs, treating them as distinct subjects when they exhibited observable structural or functional differences. To maintain source traceability, we preserved copyright information and repository links for each retained program. Especially, C1 provided 22 programs from sources previously used for testing, where three official Qiskit repositories [@Qiskit-qiskit; @qiskit_qiskit-textbook; @qiskit-community_qiskit-textbook] and MQT Bench [@quetschlich2023mqt] offered 17 and 5 programs, respectively. There were 12 directly included in artifacts of three prior studies [@abreu2022metamorphic; @li2025preparation; @long2025black] (C2), while Lubinski et al.’s benchmark [@lubinski2023application] contributed the remaining 6 (C3). To present a compact overview of benchmark programs, we manually organized them into 24 descriptive families according to their documented primary functionality, where all these families and included programs are listed in our artifact [@qolumbina_infrastructure]. For example, the family quantum_adder incorporates three types of adders executable in quantum circuits: Draper adder, full adder, and weighted adder.

3.3 Testability Refactoring↩︎

Following testability refactoring in SE practices [@reich2023testability], we moderately refactored the collected programs to unify heterogeneous interfaces and support unit testing. Overall, 95% of original programs underwent at least one of the following four testability-oriented adaptations with localized code changes intended to improve testability and maintainability while preserving output behavior for valid inputs. Modifications for refactoring are documented in code comments of the corresponding programs. In Section [sec: unit_testing], we will further introduce the use of unit tests to practically mitigate potential functional differences after refactoring.

• Structure reorganization: Program structures are reorganized to unify input-output ports across our infrastructure.

• Dependency decoupling: External dependencies that may affect program functionality or evolve over time are replaced with local equivalent implementations.

• Input validation: Assertions are inserted to prevent executing invalid test inputs that violate the program specification.

• Cross-language translation: Programs written in other frameworks or languages are translated into Qiskit/Python and validated against their program specifications.

Thirty-seven original programs were structurally reorganized, including unifying programs into classes that inherit from Qiskit’s QuantumCircuit and converting fixed classical variables into configurable input arguments.

Dependency decoupling was performed in 15 programs, following the insights of [@spadini2019mock]. For example, QFT subroutines that originally relied on the Qiskit package were replaced with equivalent implementations provided in Qolumbina. This improves the long-term usability and controllability of the benchmark, because the current QFT implementation is scheduled for deprecation in an upcoming version of Qiskit [@qiskit_qft-2026] (commit 3fe73d9).

Input validation was applied to 13 programs to support sound functional testing by rejecting invalid test inputs that would otherwise trigger undefined behavior. Specifically, constraints were enforced for both physical interpretability and logical rationality, such as the non-negativity of the evolution time in Hamiltonian simulation and the requirement that integers provided to comparator circuits be representable with the given number of qubits.

Finally, 10 programs originally written in Q# were manually translated into Qiskit counterparts conforming to the program specifications given by their source [@long2025black]. Since not all implementations have one-to-one counterparts in Qiskit, we constructed equivalent quantum operations when needed; for example, Qiskit’s Hadamard and multi-controlled \(X\) gates were used to implement Q#’s multi-controlled \(Z\) gate.

3.4 Testing-oriented Program Specifications↩︎

The program specifications provided on a webpage [@qolumbina_documentation_link] aim to facilitate controlled QST experiments by elaborating on the input arguments and expected behavior for each benchmark program. These documents are based on source descriptions and algorithm literature, such as the error-bound analysis of Quantum Monte Carlo [@woerner2019quantum], to support test oracle design and semantic checks after refactoring. To ensure specification readability, we consider the following five elements:

Formula-based specification. According to [@long2024testing; @li2026methodological], we utilized mathematical formulas to formally depict the expected output or structure of each benchmark program. For example, a mapping between state vectors, \(\ket{y}_n\ket{x}_n \mapsto \ket{(x+y)\text{ mod } 2^n}_n\ket{x}_n\), is introduced for draper_adder with two \(n\)-qubit input computational basis states \(\ket{x}_n\) and \(\ket{y}_n\) \((x,y\in\mathbb{N})\). Meanwhile, for programs with multiple subroutines, such as grover_search, we specified subroutine-level semantics, including the phase oracle, diffusion operator, and iteration-wise state updates. To mitigate confusion in specification interpretation, we underscored the little-endian ordering and qubit counts in the formal representation—details that are sometimes overlooked or insufficiently clarified even in parts of Qiskit’s official documentation. The little-endian ordering allows us to rigorously specify register indices, such as the above initial state \(\ket{y}_n\ket{x}_n\equiv \ket{y_{n-1} \cdots y_0x_{n-1} \cdots x_0}_{2n}\) with the bits \(y_{n-1}\) and \(x_0\) respectively occupying the most and least significant positions within the length-\(2n\) array. Our explicit emphasis on qubit counts is motivated by the two versions integer_comparator_greedy and integer_comparator_old sourced from Qiskit. Although both provide similar functionality, they employ inconsistent numbers of qubits (i.e., \(1\) and \(n\), respectively) to encode the comparison result, which may otherwise lead to ambiguity in formal specifications.

API documentation. The API documentation is generated via Sphinx [@sphinx] that automatically parses and formats the docstrings of program modules. The program variables are linked with the symbols denoted in the formula-based specifications. The potentially raised errors are listed to remind testers to avoid invalid inputs. In addition, we noticed that the official Qiskit documentation might omit functional descriptions of some arguments used in our benchmark programs, such as the basis variable in two programs in the pauli_rotations family. Thus, we carefully reviewed the source code and supplemented the specification for the corresponding variables.

Code example. We provide code examples to facilitate the construction of relatively complex test inputs. For example, counting requires an oracle operation as input, which may be represented either by a state vector or by a quantum circuit. Accordingly, we include two code examples to demonstrate both representations.

Documentation references. We provide references used to prepare our program specifications, which strengthen traceability and provide supporting materials for testers.

Technical debt. Given the growing attention to technical debt in quantum software [@openja2022technical; @ishimoto2024empirical], we explicitly marked two debts in the documentation to make known limitations and future maintenance costs visible. One is a development debt in multiplier_hrs, where some functional arguments remain unexposed due to redesign risks and validation costs. The other is a testing debt in monte_carlo, where simple and common QST oracles, such as those that compare individual outcomes or full output distributions [@ali2021assessing], are theoretically unreliable for this approximate algorithm.

3.5 Unit Testing↩︎

Qolumbina includes unit tests both as executable examples for follow-up QST research and as specification-based conformance checks. These tests exercise the standardized interfaces and check whether the refactored programs conform to the documented input constraints and expected outputs, thereby helping mitigate functional drift induced by refactoring. For a quasi-blind validation, two authors, independent of refactoring original programs and documenting program specifications, designed and executed 220 program-level unit tests across the 40 benchmark programs. They provided feedback based on the test results, further improving refactoring quality and program specifications. As a consequence, all provided test cases passed, achieving a statement coverage of 85%. This provides practical confidence that the refactored programs conform to the documented specifications on the exercised cases. Objectively, it is unrealistic to establish semantic equivalence to all original behaviors because the refactoring itself introduces reasonable changes to refine program testability.

In detail, all tests were executed through standardized interfaces provided by Qolumbina, as illustrated in Figure 3. These interfaces expose a subroutine under test, classical arguments, quantum inputs and measurements, and execution configurations, like shots, backends, random seeds, and output qubits. This higher-abstraction interface design simplifies test construction by hiding low-level Qiskit API details.

Both shot-based and shot-independent backends support test execution. The shot-based one, following the measurement-based execution commonly assumed in QST studies, supports one ideal simulator and 60 fake backends. The ideal simulator mainly targets theoretical analysis of computational procedure, while fake backends can better approximate characteristics of real quantum hardware. The shot-independent one, rather than simulating execution on physical quantum hardware, produces statevectors or unitary operators to enable direct checks of quantum states or transformations; this has recently gained attention in research [@miranskyy2025feasibility] and is also used in Qiskit testing practice [@Qiskit-qiskit].

4 Empirical Study↩︎

4.1 Overview of Research Questions↩︎

To characterize testing-related properties of scalable quantum programs and outline implications for controlled experiments, we propose the following research questions (RQs):

1. What are benchmark programs’ functional properties?

   RQ1.1 To what extent do the programs with real-world provenance go beyond purely pedagogical and artificial instances?

   RQ1.2 What are the output characteristics of programs?

2. What are the scales of benchmark programs?

   RQ2.1 What is the development complexity of the programs?

   RQ2.2 What is the execution complexity of the programs?

3. What is the performance of adopting benchmark programs for controlled QST experiments?

   RQ3.1 What is the time cost of program execution?

   RQ3.2 What is the performance of employing benchmark programs for fault detection?

Motivated by QST’s goal of checking the functional correctness of PUTs, RQ1 characterizes benchmark functionality through real-world provenance and output characteristics. Concerning that prior QST studies often used tutorial-style instances [@li2026methodological], mostly designed around artificial problems to demonstrate quantum advantages, RQ1.1 examines whether Qolumbina includes programs beyond pedagogical or artificial instances, thereby supporting empirical research on more practically relevant subjects. RQ1.2 investigates program outputs, since interpreting test outcomes requires knowing what expected output should be compared against observed behavior, which directly affects test oracle design.

Test scalability is one crucial requirement that expects the test approaches to still work for large-scale and sophisticated programs.

To this end, RQ2 evaluates whether Qolumbina provides suitable PUTs for studying test scalability. RQ2.1 assesses development complexity, as developers are generally more likely to introduce defects in programs with complex structures. RQ2.2 complements this view from the execution perspective by quantifying quantum circuit scale, a commonly considered factor in existing QST studies.

RQ3 conducts two controlled experiments using Qolumbina on shot-based simulators to evaluate whether it supports methodology of existing QST empirical studies. In terms of test cost and effectiveness, we reuse MSTC [@li2025preparation] and DOSS [@li2016dynamic] for feasibility analysis of Qolumbina because they are recent QST approaches with both available and functional artifacts, along with prior evaluations on scalable Qiskit programs. Across different backends, RQ3.1 follows MSTC by comparing execution time between two test-suite designs, while RQ3.2 uses DOSS as a comparably reliable test oracle to study reported failures on buggy benchmark variants.

4.2 RQ1: Functional Analysis↩︎

4.2.1 Data Annotation↩︎

Three authors with strong expertise in QSE performed the category annotation of each benchmark program in terms of their functionalities. Table ¿tbl:tab:32annotation? lists the four involved tasks, where T1, T2, and T3 are proposed to answer RQ1.1, while T4 is considered especially for RQ1.2.

For T3 and T4 involving predefined categories, since no existing taxonomy was found to match such analysis for QST, we carefully reviewed program specifications and source codes to determine the candidate categories along with their definitions. Then, three authors independently annotated all the 40 programs, following a clear guideline provided in our artifact [@qolumbina_empirical]. Fleiss’ \(\kappa\) [@fleiss1971measuring] is used to measure the agreement among authors beyond two. The agreement scores for T1–T4 are respectively 0.77, 0.70, 0.73, and 0.81. For the multi-label tasks T3 and T4, we compute Fleiss’ \(\kappa\) on each binary label and report the support-weighted macro average. According to [@landis1977measurement], all scores indicate substantial agreement. Furthermore, the authors organized an online discussion to resolve disagreements and then decide on the annotated metadata.

4.2.2 RQ1.1: Beyond Pedagogical and Artificial Instances↩︎

As shown in Figure 4 (a), the majority (i.e., 70%) of the benchmarked algorithms are not primarily pedagogical. Hence, moving beyond purely pedagogical and artificial benchmarks allows QST approaches to be evaluated on programs with clearer program-level functionality and real-world provenance.

For program instances, Figure 4 (b) exhibits that \(77.5\%\) of benchmark programs could be reused as a subroutine. This indicates that many programs capture modular components that can appear in larger quantum workflows, rather than only standalone demonstrations. Several reusable instances also implement non-tutorial algorithms. For example, oracle programs in the quantum_adder and quantum_multiplier families implement reversible arithmetic primitives, which can be integrated into larger quantum algorithms, including applications of quantum finance such as credit risk assessment [@egger2020quantum].

Figure 4 (c) shows that the benchmark incorporates a diverse spectrum of seven functional categories. The largest category contains 14 programs for algebraic or arithmetic operations on quantum registers, followed by 9 programs for classical-predicate evaluation. The remaining programs cover inference of hidden properties and structures, simulation of quantum dynamics, quantum-based search algorithms, state preparation subroutines, and sampling-related estimation, including representative cases such as Simon’s algorithm, Hamiltonian Simulation, Grover search, graph-state preparation, amplitude estimation, and quantum Monte Carlo. This distribution helps reduce the risk of evaluating QST techniques on a narrow set of algorithmic behaviors.

4.2.3 RQ1.2: Output Characteristics↩︎

Motivated by prior studies [@oldfield2025faster; @long2025black; @li2016dynamic] that test quantum programs with specific output properties, we summarize six output types to characterize the intended outputs of quantum programs. We assign each category based on the specified measurement on output qubits associated with program functionality, while we assume the default Pauli-\(Z\) basis when no measurement is predefined. Besides, one program may belong to multiple categories, such as QFT storing outputs in both amplitudes and phases, making it classifiable as “exact distribution output” and “phase-encoded output”.

In Table ¿tbl:tab:32output95category95table?, Qolumbina covers six categories, including dominant-output categories that have mentioned in only a few studies [@long2024testing; @mendiluze2025quantum] but have earned increasing attention, as well as approximation-related categories that have not been systematically investigated in QST so far. Deterministic outputs form the largest category, where testing such quantum programs is close to testing non-randomized classical programs, suggesting opportunities to adapt established CSE testing techniques to QST. For the widely studied “exact distribution output”, Qolumbina includes 11 programs and thus supports evaluation on a broader set of scalable programs. Eleven programs have phase-encoded outputs, posing a challenge for QST because validating their intended behavior may require phase information that is not directly reflected in output probabilities. This limitation can make existing test oracles based on hypothesis testing or statistical distance unreliable for phase-encoded programs like QFT [@li2016dynamic]. Thus, Qolumbina provides useful subjects for evaluating and improving QST techniques for phase-related output behavior.

Although fewer programs belong to the remaining categories, they remain valuable for QST research and may expose less-studied testing challenges. Both approximate-value and exact-value outputs require test oracle design to focus on functionality-relevant partial outcomes rather than the full output distribution. Approximate values and distributions further require checking whether errors exceed tolerable bounds.

4.3 RQ2: Scale Analysis↩︎

4.3.1 Complexity Metrics↩︎

Referring to Zhao [@zhao2021some], both complexity metrics general in SE and specific to QSE are considered to quantify the programs scale.

RQ2.1 estimates development effort using LOC and cyclomatic complexity, which respectively capture code size and control-flow complexity in a complementary manner. LOC counts only effective code lines, excluding blank lines and comments. We further exclude dunder methods and factory functions to focus on code relevant to functionality.

In the view of quantum circuits corresponding to PUTs, RQ2.2 estimates execution complexity using all the circuit width (a.k.a. number of qubits), size (a.k.a. number of quantum gates), and depth, following prior QST studies [@li2025preparation; @li2026methodological; @oldfield2025faster]. These metrics are computed on the circuits instantiated by classical inputs and transpiled for an ideal backend with full qubit connectivity and a sufficiently expressive basis-gate set, enabling fair comparison across abstracted and synthesized unitary operations.

4.3.2 RQ2.1: Development Complexity↩︎

Two boxplots in Figure 5 illustrate that Qolumbina spans a broad range of development complexity, with LOC ranging from 4 to 247 and cyclomatic complexity from 2 to 86. Although the median LOC is 32, such compact source codes do not necessarily indicate small testing subjects. Unlike low-level quantum programs prevailing in existing benchmarks and studies, whose LOC often reflects gate-level enumeration in one concrete circuit, programs in Qolumbina leverage higher abstractions and classical arguments to generate families of variable circuits. We will analyze instantiated circuit scale separately in RQ2.2.

Furthermore, five outliers representing large scales are identified, among which the Harrow–Hassidim–Lloyd algorithm (hhl) is the most complicated, possibly attributed to the combination of multiple quantum subroutines. In general, such outliers are valuable for QST research, as their larger source size and higher control-flow complexity create more implementation paths and subroutine interactions, providing PUTs for examining whether testing techniques scale to structurally complex quantum programs.

4.3.3 RQ2.2: Execution Complexity↩︎

In RQ2.2, we instantiate quantum circuits by providing each program interface with concrete classical arguments, many of which are reused from the unit tests described in Section 3.5. In Figure 6, we examine how a classical argument (denoted as \(n_{\mathrm{ind}}\)), which directly and independently determines the qubit count of certain quantum registers, affects instantiated circuit scale. Only 22 benchmark programs are filtered for investigation, because such an argument \(n_{\mathrm{ind}}\) is absent in the others. Across the three charts, circuit scale increases with \(n_{\mathrm{ind}}\), indicating that Qolumbina can support scalability evaluation by varying test inputs. This differs from many low-level quantum programs used before, whose fixed circuit structure makes circuit scale largely irrelevant to test inputs. More specifically, circuit width grows roughly linearly with \(n_{\mathrm{ind}}\) across the selected programs and is often larger than \(n_{\mathrm{ind}}\) due to automatic inclusion of ancilla qubits or quantum registers. By contrast, the maximum circuit size and depth grow much faster with \(n_{\mathrm{ind}}\), suggesting that testing larger instantiated circuits on classical simulators can become costly even for small increases in \(n_{\mathrm{ind}}\).

To explore other types of classical arguments that could impact the circuit scale, we conduct a case study on weighted_adder, whose formula-based specification is \(\ket{0}_{s}\ket{y}_{n_{\mathrm{ind}}} \mapsto \ket{\mathbf{w}^{\top}\mathbf{y}}_{s}\ket{y}_{n_{\mathrm{ind}}}\). Here, \(\mathbf{w}=[\lambda_0,\cdots, \lambda_{n_{\mathrm{ind}}-1}]^{\top}\) is an integer vector of \(n\) integer weights, the Boolean vector \(\mathbf{y}\) encodes integer \(y\), and the integer \(s = 1+\lfloor \log_2(\sum_{j=0}^{n_{\mathrm{ind}}-1}\lambda_j) \rfloor\) denotes the number of sum qubits. Thus, in theory, circuit complexity depends not only on \(n_{\mathrm{ind}}\) but also on the weight vector \(\mathbf{w}\). Table ¿tbl:tab:32exe95case95study? exhibits this effect under six test inputs, and we can see that the circuit width, size, and depth are largely governed by specific \(\mathbf{w}\). For instance, with \(n=5\), circuit size dramatically varies from 0 to more than 1,000 across different weight vectors. This case highlights that circuit complexity of scalable quantum programs depends not only on the argument with explicit relevance to the qubit count (like \(n_{\mathrm{ind}}\)), but also on other functionality-related arguments (like \(\mathbf{w}\)) that would substantially reshape the instantiated circuit, both of which should be carefully considered in test design.

4.4 RQ3: Experimental Analysis↩︎

4.4.1 Experiment Design↩︎

PUT selection follows the scopes of MSTC [@li2025preparation] and DOSS [@li2016dynamic]. RQ3.1 evaluates 23 programs that functionally support quantum state initialization with computational basis states, enabling comparison between test cases composed of mixed states and pure states with configurations suggested by MSTC. RQ3.2 runs 26 programs categorized with deterministic outputs or exact distribution outputs (referring to Table ¿tbl:tab:32output95category95table?), which matches DOSS’s investigation on individual outcomes or overall distribution. For feasibility analysis on fault detection, seven mutants are generated per program by manually applying mutation operators that correspondingly cover all existing QST mutation actions on quantum gates: adding, deleting, duplicating, modifying, moving, replacing, and switching [@li2026methodological]. The code line for mutation is randomly chosen, and human review is performed to validate that mutation operators are not equivalent and indeed alter original functionalities. In RQ3.2, reported failures are caused by incorrect functions instead of program crash.

Test cases mostly reuse examples from unit tests, along with moderate extensions for enhanced diversity. Following MSTC, RQ3.1 compares pure-state and mixed-state test suites with input-domain equivalence, associated with the number of qubits encoding test inputs (denoted as \(n_{\mathrm{test}}\)). Mixed-state test cases require extra qubits for state preparation, but reduce test-suite size while preserving input coverage by grouping multiple pure states into one mixed state. Overall, RQ3.1 generates 92 mixed-state, while RQ3.2 includes programs without the need of state initialization and executes 1,680 instantiated circuits under test. Also, test case design considers running large-scale circuits in the context of scalability, as the circuits in RQ3.1 and RQ3.2 respectively have widths up to 19 and 14 transpiled on the ideal simulator.

Shot-based backends are used throughout RQ3 following common QST practice [@li2026methodological]. Each test execution uses Qiskit’s default 1,024 shots and 20 independent repeats with different random seeds to balance statistical stability and execution cost. We evaluate two fake backends supported by Qolumbina, Algiers and Brooklyn, under both noiseless and noisy simulation settings, which allows us to study hardware-related impact on QST under affordable simulation costs. Both backends emulate IBM hardware with different native gates and coupling maps, supporting up to 27 and 64 qubits, respectively. Generally, RQ3 experiments are executed on Ubuntu x86_64 with an Intel(R) Core(TM) i9-10940X CPU @ 3.30GHz.

4.4.2 RQ3.1: Execution Time↩︎

Figure 7 reports the total execution time of all test cases for each \(n_{\mathrm{test}}\). In the measured range, execution time increases rapidly and appears approximately linear on the logarithmic scale. On the three noiseless backends, we reproduce MSTC’s finding that pure-state test suites take longer to execute than their mixed-state counterparts. Beyond MSTC’s setting of ideal simulation, we further observe the same trend on noisy simulation, where mixed-state test cases retain their advantage in better efficiency. Moreover, when running the same test case, noisy backends consistently require longer simulation time than noiseless backends.

Regardless of simulated noise, we observe clear differences in execution time among three backends. Across all cases, execution on the ideal backend is not always the fastest, and Brooklyn consistently incurs the longest time, indicating that test overhead can vary substantially across heterogeneous backends. One plausible explanation is that backend-specific properties, such as qubit connectivity and native gates, affect the transpiled circuit structure and thus simulation efficiency.

4.4.3 RQ3.2: Fault Detection↩︎

We use the test failure rate to measure fault-detection effectiveness, defined as the proportion of test executions reported as failed for each PUT. Figure [fig:32faults] demonstrates the distribution of test failure rate for 26 programs when DOSS is used as the test oracle on three backends. Upon ideal simulation, several programs like less_than_qubit still exhibit relatively low failure rates (e.g., below 0.25), indicating the presence of hard-to-trigger mutants and thus the need for systematic evaluation of testing techniques for such cases.

The comparison across backends reveals that the reported failure rates depend on backends even under the same test suite and oracle. In noiseless scenarios, DOSS has been shown to reduce both false positives and false negatives compared with common statistical oracles [@li2016dynamic], so its failure reports can provide a useful reference for program-induced faults before backend noise is introduced. For programs like parity_qubit, DOSS identifies similar failure rates across backends, suggesting that the oracle could tolerate certain noise effects. However, noisy backends mostly shift failure rates upward, indicating that some additional reports may be induced by backend noise rather than program faults alone.

Furthermore, Table ¿tbl:tab:32statistical95faults? compares test failure rates across backends using the Mann–Whitney U test and the Vargha–Delaney effect size \(\hat{A}_{12}\) [@vargha2000critique]. For each pairwise comparison, only 4 or 5 programs demonstrate negligible effects, while 21 or 22 exhibit large effects. The comparison between Algiers and Brooklyn yields large effects for 21 programs, showing that reported failures are shaped by the specific backend model rather than by noise presence alone. This result indicates that fake-backend choice is an experimental variable in fault-detection studies, rather than a peripheral execution detail.

5 Threats to Validity↩︎

Internal validity. Internal validity may be affected by human involvement in program filtering, testability refactoring, and data annotation. To reduce subjectivity, we defined explicit guidelines and resolved disagreements through consensus. Refactoring may introduce deviations from the original implementations, whose risk is mitigated by preserving source links, keeping refactoring moderate, documenting changes, and conducting quasi-blind unit tests. These checks support conformance to program specifications but do not prove full semantic equivalence to every original behavior.

External validity. External validity is limited by three factors. First, Qolumbina focuses on Qiskit programs without intermediate measurements, which aligns with the choice of current QST research. Second, our empirical study uses classical simulation, as is dominant in existing QST studies [@li2026methodological]. Experiments on physical hardware may differ due to time-varying noise and weaker reproducibility. Third, our fault-detection experiment adopts mutation operators rather than real-world bugs, which is common in QST [@li2026methodological]. Also, unlike Defects4J [@just2014defects4j] in CSE and Bugs4Q [@zhao2021bugs4q; @zhao2023bugs4q] for quantum software stacks with classical programs, Qolumbina does not include real-world bugs, since the development histories of benchmark programs are rarely available.

Construct validity. Construct validity may be impacted by concepts like real-world provenance, scalability, and testability, where they are determined through observable and traceable evidence rather than formal definitions and quantified metrics. We clarify that the notion of real-world provenance relies on claims in publications, documentation in source code, and inclusion in official SDKs, instead of industrial quantum applications. The scalability relies on the availability of classical inputs, while the testability is manually validated through explicit specifications, unit tests, and test-case examples.

Conclusion validity.

Our empirical study offers an initial assessment, but cannot cover all programs, test suites, backends, and QST techniques. Thus, the results should be interpreted as evidence of feasibility and testing-relevant phenomena rather than definitive conclusions about all QST settings. To support follow-up studies, we make the experimental settings and results explicit and reproducible.

6 Lessons Learned↩︎

Functionality-aware testing. Unlike low-level quantum programs, the programs in Qolumbina exhibit clear program-level functionalities. This makes it possible for QST studies to design testing approaches that take intended functionality into account. For example, draper_adder in Table ¿tbl:tab:32output95category95table? implements integer addition for \(x\) and \(y\), where both inputs are encoded in computational basis states. Testing it with qubits initialized in entangled states falls outside its intended input domain and therefore does not provide meaningful evidence about functional correctness. Our findings in RQ1 also highlight a class of quantum programs that produce dominant but inexact outcomes, for which functionally matched test oracles may be more appropriate than strict equivalence checks over the full output distribution. Although the output probability oracle [@ali2021assessing] is the most commonly used oracle in prior studies [@li2026methodological], its validity may be largely limited to programs with probability-distribution outputs, and it may fail to correctly identify phase outputs [@li2016dynamic]. These results suggest that test oracles should be selected according to the intended functionality and output characteristics of the target program.

Investigation of fake backends. Compared with the ideal simulator used in most existing QST studies [@li2026methodological], fake backends introduce hardware-related properties into test execution. RQ3 shows clear differences in test cost between the ideal simulator and fake backends, even in noiseless settings, which calls for further investigation. Based on RQ2, we also found that benchmark programs can be structurally complex, and that their execution complexity strongly depends on classical input arguments. This is likely related to their scalability and modular design. Under these conditions, testing large benchmark programs on noisy simulators may incur substantial overhead that classical hardware can hardly afford.

7 Conclusion and Future Work↩︎

In this paper, we systematically build Qolumbina, a benchmark infrastructure for QST empirical studies on scalable quantum programs. Qolumbina provides refactored benchmark programs, program specifications, test case examples, and standardized interfaces to improve benchmark usability and support functional testing. Our empirical study shows that Qolumbina covers diverse program functionalities and scales, and that many benchmark programs exhibit potential applicability and reusability. The study also highlights two issues that deserve more attention in future work: test case design should be aligned with program functionalities, and backend choice should be treated as part of the experimental setting. In the future, we plan to extend Qolumbina with more diverse quantum programs and SDKs, and also make improvements based on the state of research and industrial practice.

Data Availability↩︎

The infrastructure and empirical-study data for Qolumbina are fully available at [@qolumbina_infrastructure; @qolumbina_empirical], respectively. A webpage is further provided at [@qolumbina_documentation_link] generated from the repository [@qolumbina_documentation_repo].