PS  [1] unifies semantic and instance segmentation under a single metric, PQ. PQ is defined through a One-to-One matching of predicted to ground truth segments at a fixed \(\mathrm{IoU} > 0.5\) threshold. The threshold makes the matching trivial: above \(0.5\), at most one prediction can overlap a given ground truth segment, so the assignment reduces to a lookup. Below it, the original definition is undefined. Part-aware Panoptic Segmentation [2] later extended PQ to hierarchical targets, where a scene-level instance carries a meaningful part decomposition, but inherited the matching rule unchanged.
Two limitations follow. First, the One-to-One rule is brittle whenever predictions and references legitimately split or merge. An instance, correctly recovered as several adjacent fragments scores as one true positive and several false positives, even if their union matches the reference well. Several recent metrics tackle parts of this problem. Examples include proximity-based per-component evaluation [3], topology-aware connected-component Dice [4], and many-to-many Cluster Dice [5]. Each one targets a single failure mode. They do not share a common framework, and none extends to hierarchical evaluation. Second, the \(\mathrm{IoU} > 0.5\) threshold is an opinionated default. A prediction with \(\mathrm{IoU} = 0.49\) is treated identically to one with zero overlap, and the choice of threshold heavily shapes the resulting score. These issues bite hardest where instances are small, dense, or hierarchically organised: tiny-instance benchmarks, where a one-pixel shift dominates IoU [6], scene parsing with internal part annotations [2], and biomedical data, where lesions decompose into sub-regions and where expert disagreement induces ambiguous instance boundaries.
We address both limitations by recasting segment matching as a weighted bipartite assignment problem. Edges are formed between predicted and ground truth segments with \(\mathrm{IoU}\) above a configurable threshold \(\tau \in [0, 1)\), and TP, FP, and FN are anchored to the vertices of the resulting graph rather than to matched edges. This vertex-centric formulation preserves the invariant \(|\mathrm{TP}| + |\mathrm{FN}| = |G|\) across all matching strategies and generalises PQ to any \(\tau\). It admits four strategies: One-to-One, Many-to-One, One-to-Many, and Many-to-Many. The first three are well-defined within the PQ framework, and each one either tolerates or penalises a specific segmentation failure mode. Many-to-Many breaks key invariants and falls outside the framework. At \(\tau > 0.5\), the One-to-One case recovers the original PQ exactly. The same formalism defines the AUTC, the threshold-agnostic summary \(\int_0^1 \mathrm{PQ}(\tau)\,d\tau\), which inherits the same configurability across strategies and parts. AUTC decomposes into segmentation- and recognition-specific terms that isolate the source of a method’s threshold-averaged performance. For settings with dense, touching instances, we additionally describe a Voronoi-based region-wise variant. It complements One-to-Many matching by partitioning the image into per-instance cells, in which the same bipartite construction is applied locally.
As an add-on for hierarchical targets, we extend PartPQ to this framework and decouple it from the original One-to-One rule. The part-aware quality term then composes with any of our matching strategies. We release all of the above as a unified open-source package built on the Panoptica pipeline [7]. The package also exposes the prior approaches discussed above as configurable options, so practitioners can select the matcher and quality term that fits their task rather than adopting a tool’s default. We then assess the practical consequences of these choices across different case studies, including the previously unstudied interaction between matching strategy and part-level evaluation.

1 Preliminaries↩︎

Semantic segmentation assigns every pixel/voxel \(i\) a single class label \(l_i \in {\cal L}\) [8]. Pixels/voxels are grouped only by class, so distinct instances of the same class merge into one region. Instance segmentation requires a method to delineate every object instance in an image with a binary mask, typically accompanied by a class label and a confidence score [9], [10]. Unlike PS, it considers only thing classes and permits overlapping masks, since each instance is predicted independently rather than as a partition of the image.
PS, introduced by [1], unifies semantic and instance segmentation into a single coherent task. Given a set of \(L\) semantic classes \({\cal L}= {\cal L}\textsuperscript{St}\cup {\cal L}\textsuperscript{Th}\), a PS algorithm assigns every pixel/voxel \(i\) a pair \((l_i, z_i) \in {\cal L}\times \mathbb{N}\), where \(l_i\) denotes the semantic class and \(z_i\) the instance id. Stuff classes (e.g.grass, sky) are amorphous regions for which the instance id is irrelevant, while thing classes (e.g.sheep, car) are countable objects whose pixels/voxels must be partitioned into distinct instances. Unlike instance segmentation, PS forbids overlapping segments by construction, so each pixel/voxel receives exactly one label pair (see visual comparison of segmentation tasks in 8).

1.0.0.1 Panoptic Quality.

To evaluate PS on a single scale, [1] introduced the PQ metric. PQ proceeds in two steps: (i) matching predicted segments to ground truth segments, and (ii) aggregating the matches into a single score per class, which is then averaged over classes. A predicted segment \(p\) matches a ground truth segment \(g\) when their intersection-over-union \(\text{IoU}(p,g) = |p \cap g| / |p \cup g|\) satisfies \(\text{IoU}(p, g) > 0.5\). Together with the non-overlap property of PS, this threshold guarantees a unique matching: at most one predicted segment can match each ground truth segment, and vice versa, so no assignment problem needs to be solved [1]. The matched pairs form the TPs, while unmatched predictions and ground truth segments form the FPs and FNs, respectively. PQ is then defined as \[\label{eq:pq} \text{PQ} = \frac{\sum_{(p,g) \in \mathit{TP}} \text{IoU}(p,g)}{|\mathit{TP}| + \tfrac{1}{2}|\mathit{FP}| + \tfrac{1}{2}|\mathit{FN}|} = \underbrace{\frac{\sum_{(p,g) \in \mathit{TP}} \text{IoU}(p,g)}{|\mathit{TP}|}}_{\text{SQ}} \times \underbrace{\frac{|\mathit{TP}|}{|\mathit{TP}| + \tfrac{1}{2}|\mathit{FP}| + \tfrac{1}{2}|\mathit{FN}|}}_{\text{RQ}},\tag{1}\] which decomposes into a segmentation quality term \(\text{SQ}\), the average IOU of matched segments, and a recognition quality term \(\text{RQ}\), equivalent to the \(F_1\) score [11]. All segments contribute equally regardless of area.

1.0.0.2 Part-aware Panoptic Quality.

[2] extend PS with a third level of granularity: each thing instance may carry an internal decomposition into a fixed set of parts, e.g.a person into head, torso, arms, and legs. The corresponding metric, Part-aware Panoptic Quality (PartPQ), inherits the matching machinery of PQ and reweights only the per-pair quality so that parts contribute to the score only through a successfully matched parent instance. To our knowledge, PartPQ is the only existing panoptic-style metric that jointly evaluates instances and their internal parts, and it has not previously been applied to biomedical data. We revisit it formally in 3.7 and showcase its adoption within the medical image analysis framework.

1.0.0.3 The 0.5 threshold.

[1] argue that lower thresholds are unnecessary because matches with \(\text{IoU}\le 0.5\) are rare in practice. This assumption breaks down in domains where boundary localization is inherently ambiguous or where small, densely packed instances dominate the data. Biomedical imaging offers a plethora of such regimes, including multiple sclerosis lesion segmentation [12] and brain-metastasis segmentation in MR [13], liver and abdominal metastasis in CT [14]–[16], encapsulin segmentation in electron microscopy [17], and neuron segmentation in light-sheet microscopy [18]. In clinical scenarios such as liver metastasis or head and neck cancer, multiple independent lesions appear in close proximity within a restricted anatomical volume, yet must be identified as separate entities to monitor disease progression and treatment response. Under the One-to-One rule with \(\text{IoU}> 0.5\), otherwise correctly predicted segments in such regimes are systematically penalized as simultaneous \(\mathit{FP}\) and \(\mathit{FN}\), obscuring real model improvements.

2 Related Work↩︎

The Metrics Reloaded framework recommends PQ for the task of instance segmentation as it fuses both detection performance and segmentation quality of successfully matched segments into a single score [19]. While the authors discuss common pitfalls, such as class imbalance, they do not offer any means to address these difficulties for instance segmentation evaluation.
Defining a fixed matching threshold to classify predictions as either TPs or FPs results in a high sensitivity to the choice of the threshold. Predicted instances just slightly below the threshold are disregarded completely and penalized harshly. [20] introduced sortedAP in order to completely omit a fixed threshold by instead integrating over all existing intersection thresholds. To overcome the limitation of a threshold strictly greater than 0.5, the authors propose using the Hungarian algorithm for matching, which preserves a One-to-One relationship between segments and maximizes the accumulated IOUs of matches.
Motivated by biased evaluation of semantic metastases segmentation towards larger instances, [3] proposed Connected-Component Metrics incorporating per-component evaluation and a proximity-based matching criterion, giving each tumor the same weight irrespective of its size.
[21] reformulate the matching of predicted and ground truth segments as an assignment problem on a bipartite graph in the context of historical map evaluation. They define a matching as a One-to-One relation between the disjoint vertex sets of predictions and ground truth segments, with unmatched vertices allowed on either side. The same work introduces NPQ, which integrates the F-score over matching thresholds and is argued to be an equivalent rewriting of the standard PQ restricted to \(\tau \geq 0.5\). We adopt the bipartite formulation and extend it to sub-\(0.5\) thresholds and different matching strategies (3). The relation between NPQ, sortedAP, and AUTC is discussed in 3.5.

3 Rethinking Segment Matching for Panoptic Quality↩︎

We recast the matching of predicted to ground truth segments as a constrained bipartite assignment problem [21], preserving the behavior of standard PQ above \(\tau = 0.5\) and admitting well-defined extensions for \(\tau \leq 0.5\).
Let \(G = \{g_1, \ldots, g_m\}\) and \(P = \{p_1, \ldots, p_n\}\) denote the ground truth and predicted segments of a given class, and let \(\tau \in [0, 1)\) be a matching threshold. We construct a weighted bipartite graph \(\mathcal{B}_\tau = (G \cup P,\, E_\tau,\, w)\) with edges \[\label{eq:edges} E_\tau = \{(g_i, p_j) \in G \times P : \text{IoU}(g_i, p_j) > \tau\}\tag{2}\] and edge weights \(w(g_i, p_j) = \text{IoU}(g_i, p_j)\). A matching \(M \subseteq E_\tau\) is a subset of these edges. Let \(\deg_M(v)\) denote the number of edges in \(M\) incident to vertex \(v\). A matching strategy is defined by independently constraining \(\deg_M\) on each side of \(\mathcal{B}_\tau\) to at most one or leaving it unconstrained. The first word of a strategy’s name refers to the prediction side and the second to the ground truth side: “Many-to-One” therefore means \(\deg_M(g)\) is unconstrained while \(\deg_M(p) \leq 1\), allowing multiple predicted segments to cover a single ground truth segment (1 (b)). The four resulting strategies are summarized in 1; each is treated in detail in [sec:one-to-one] [sec:many-to-one] [sec:one-to-many] [sec:many-to-many]. The discussion of the optimization of \(M\) under each strategy is consolidated in 7.

3.0.0.1 Vertex-based TP, FP, and FN.

We anchor TP and FN to ground truth vertices and FP to predicted vertices: \[\label{eq:tpfnfp} \mathit{TP}= \{g \in G \!:\! \deg_M(g) \geq 1\}, \; \mathit{FN}= \{g \in G \!:\! \deg_M(g) = 0\}, \; \mathit{FP}= \{p \in P \!:\! \deg_M(p) = 0\}.\tag{3}\] This is an intentional deviation from the original PQ formulation, where the one-to-one constraint makes edges and matched vertices interchangeable: each edge in \(M\) corresponds to exactly one matched ground truth and one matched predicted segment, and TPs can be identified with \(M\) itself or with the matched ground truth set. Once we relax this constraint and admit Many-to-One or One-to-Many matchings, the equivalence breaks: a single ground truth or predicted segment may be incident to multiple matched edges, and \(|M|\) can exceed \(|G|\) or \(|P|\). We must therefore decide whether to count edges or vertices.
This vertex-centric formulation guarantees the invariant \(|\mathit{TP}|+|\mathit{FN}|=|G|\) across all matching strategies, so TP and FN partition the ground truth set: every ground truth segment is either recognized or missed, never counted multiple times. RQ therefore retains its interpretation as the recognition rate of ground truth segments rather than over-matching edges: a ground truth segment jointly described by several predicted segments in a Many-to-One matching contributes one TP, not several. SQ remains an average IoU per recognized object, with each ground truth segment weighted equally regardless of how many predicted segments are matched to it. The edge-based alternative \(|\mathit{TP}|=|M|\) would violate both properties under asymmetric matchings, where \(|M|\) can exceed \(|G|\) or \(|P|\).

3.0.0.2 Segmentation quality under multi-edge matchings.

For each matched ground truth segment \(g \in \mathit{TP}\), let \(P_M(g) = \bigcup \{\, p \in P : (g,p) \in M \,\}\) denote the union of predicted segments matched to \(g\). We then define \[\label{eq:sq} \text{SQ} = \frac{\sum_{g \in \mathit{TP}} \text{IoU}(P_M(g), g)}{|\mathit{TP}|},\tag{4}\] which preserves the \([0,1]\) bound on SQ and weights each recognized ground truth segment equally. In One-to-One and One-to-Many matchings, \(P_M(g)\) is always a single predicted segment and 4 reduces to the standard per-edge form. The construction is only nontrivial in the Many-to-One case (3.2).

3.1 One-to-One Matching↩︎

In One-to-One matching, every predicted segment matches at most one ground truth segment and vice versa. Formally, \(M \subseteq E_\tau\) is a One-to-One matching if \[\label{eq:1to1} \deg_M(v) \leq 1 \quad \text{for all } v \in G \cup P.\tag{5}\] One-to-One matching credits exactly one predicted segment per ground truth segment, as depicted in 1 (a). If two predicted segments both overlap the same object, only the higher-IOU one contributes to \(\mathit{TP}\) while the other is counted as \(\mathit{FP}\) (1 (b), left). Above \(\tau = 0.5\), One-to-One collapses to the original PQ definition of [1].
Like standard PQ, this penalizes over- and undersegmentation symmetrically, and is the only strategy that does so. If both failure modes carry equal cost in the downstream task, only One-to-One reflects that. This approach is well-suited for scenarios in which an exact assignment of predicted to ground truth segments is non-negotiable and the ground truth annotations exhibit low ambiguity and low noise.

3.2 Many-to-One Matching↩︎

In Many-to-One matching, multiple predicted segments may jointly describe a single ground truth segment, while each predicted segment is still matched to at most one ground truth segment. Formally, \[\label{eq:mto1} \deg_M(p) \leq 1 \quad \text{for all } p \in P.\tag{6}\]

Multiple predicted segments matched to a single ground truth segment are counted as a single TP, reducing the total FP count by the number of additional predicted segments absorbed into that match (1 (b)). This setting is appropriate when over-segmentation reflects a legitimate decomposition of a ground truth segment rather than a model failure, for example, when a long, thin instance is correctly recovered as several adjacent fragments.

3.3 One-to-Many Matching↩︎

In One-to-Many matching, a single predicted segment may describe multiple ground truth segments, while each ground truth segment is still matched by at most one predicted segment. Formally, \[\label{eq:1tom} \deg_M(g) \leq 1 \quad \text{for all } g \in G.\tag{7}\]

This setting is appropriate when adjacent ground truth segments of the same class are difficult to delineate and a merged predicted segment should still be credited rather than penalized as both FP and FN. For example, individual cells in a dense cluster may not be separated in the predicted segment, yet still convey the correct semantic and approximate spatial extent.

3.4 Many-to-Many Matching↩︎

A Many-to-Many strategy drops degree constraints on both the prediction and ground truth sides of the bipartite graph, allowing arbitrary subsets of \(P\) to match arbitrary subsets of \(G\). We argue that this strategy is not sensible within the Panoptic Quality framework. While our vertex-centric formulation (3 ) technically preserves the \(|\mathit{TP}| + |\mathit{FN}| = |G|\) invariant even here, the mathematical interpretation of these matches collapses. Specifically, a Many-to-Many assignment allows a single predicted segment to simultaneously contribute to the matched union \(P_M(g)\) of multiple distinct ground truth segments in the SQ calculation (4 ), double-counting predicted segments and violating the mutually exclusive nature of panoptic instances. In regimes where Many-to-Many matching is required to meaningfully evaluate a prediction, the problem is better framed as a clustering comparison rather than instance segmentation and thus operating entirely outside the bipartite assignment framework.

3.5 Area under Threshold Curve (AUTC)↩︎

Choosing a fixed threshold strongly influences the evaluation by making an opinionated assumption about the minimum IOU required for a TP. And predicted segments which overlap a ground truth segment just slightly below the threshold are harshly penalized in the final PQ computation. We bypass this choice by treating \(\tau\) as a free parameter and integrating PQ over all thresholds, yielding a threshold-agnostic summary. Let \(\mathrm{PQ}(\tau)\) denote the Panoptic Quality computed using the optimal matching \(M^\star_\tau\) derived from the thresholded bipartite graph \(\mathcal{B}_\tau\). We define the AUTC as \[\mathrm{AUTC} = \int_{0}^{1} \mathrm{PQ}(\tau) \, d\tau.\] Because Panoptic Quality itself takes values in [0,1] and we integrate it over the threshold parameter of unit length, AUTC also lies in [0,1]. Although defined as a continuous integral, AUTC reduces to an exact finite sum. Let \(E_0 = \{(g_i, p_j) \in G \times P : \text{IoU}(g_i, p_j) > 0\}\) denote the edges with strictly positive overlap, and let \(u_1 < u_2 < \dots < u_k\) be the unique edge weights \(\{w(e) : e \in E_0\}\) in increasing order. The edge set \(E_\tau\) changes only when \(\tau\) crosses an element of this set, so \(\mathrm{PQ}(\tau)\) is a right-continuous step function on \([0, 1]\). Setting \(u_0 = 0\), the integral becomes \[\label{eq:autc-discrete} \mathrm{AUTC} = \sum_{i=0}^{k-1} \mathrm{PQ}(u_i) \cdot (u_{i+1} - u_i),\tag{8}\] where the contribution from \(\tau \geq u_k\) vanishes: no edge satisfies \(\text{IoU}> u_k\), so every \(g \in G\) is an FN, every \(p \in P\) is an FP, and \(\mathrm{PQ}(\tau) = 0\) on \([u_k, 1]\).

3.5.0.1 Decomposing AUTC.

Both factors of PQ are themselves step functions in \(\tau\), so the same construction yields recognition- and segmentation-specific summaries: \[\mathrm{AUTC}_\mathrm{SQ} = \int_{0}^{1} \mathrm{SQ}(\tau) \, d\tau, \qquad \mathrm{AUTC}_\mathrm{RQ} = \int_{0}^{1} \mathrm{RQ}(\tau) \, d\tau,\] both bounded in \([0, 1]\) by the same argument and computable via the discrete form of 8 . Reporting them alongside AUTC isolates whether a method’s threshold-averaged advantage stems from better segmentation quality (\(\mathrm{AUTC}_\mathrm{SQ}\)) or from a higher recognition rate across thresholds (\(\mathrm{AUTC}_\mathrm{RQ}\)). We emphasize, however, that the multiplicative structure of PQ does not carry over to the integrals: in general \[\mathrm{AUTC} \;\neq\; \mathrm{AUTC}_\mathrm{SQ} \cdot \mathrm{AUTC}_\mathrm{RQ},\] because the integral of a product is not the product of integrals.

3.5.0.2 Relation to existing threshold-agnostic metrics.

Two prior metrics also integrate a panoptic-style score over matching thresholds. [21] introduce NPQ, the area under the F-score curve over \(\tau \in [0, 1]\) under One-to-One matching, and argue that the standard PQ is an equivalent rewriting of NPQ at \(\alpha \geq 0.5\). [20] propose sortedAP, which integrates the average precision score over the full threshold range using Hungarian One-to-One matching. Both metrics integrate a recognition-only score and are restricted to One-to-One matching. In contrast, AUTC additionally admits Many-to-One and One-to-Many matching strategies and decomposes into \(\mathrm{AUTC}_{\mathrm{SQ}}\) and \(\mathrm{AUTC}_{\mathrm{RQ}}\), which isolates the source of a method’s threshold-averaged performance.

3.6 Voronoi Matching↩︎

The Voronoi variant, introduced as part of the CC-Metrics framework [4], sidesteps global bipartite matching by partitioning the image domain into per-instance cells: every voxel is assigned to its nearest ground truth segment, and predictions are scored locally inside each cell. Since each cell contains exactly one ground truth segment, the bipartite construction of 3 reduces to \(|G|\) independent intra-cell matchings, into which any of the previous strategies can be plugged. The vertex-based TP/FN/FP accounting and PQ definition of 3 4 carry over unchanged. Predicted segments extending past a cell boundary are re-attributed to the nearest reference rather than counted as an additional FP, as depicted in 1 (c). The original CC-Metrics motivation is that local averaging weights every component equally, regardless of size, preventing a few large instances from dominating the score. Embedded in our framework, the same construction additionally resolves bipartite matching ambiguity when references are dense or touching. The formal construction is given in 6.

3.7 Part-Aware Panoptic Matching↩︎

Part-aware Panoptic Quality (PartPQ), introduced by [2], extends PQ to hierarchical targets: each scene-level class may carry an internal part decomposition, and a prediction is rewarded only if it both recovers the parent instance and agrees with its internal part labeling. This matches the biomedical setting almost directly: a tumor instance decomposes into edema, non-enhancing, and enhancing sub-regions; a vertebra into body, arch, and process; a cell into cytoplasm and nucleus (3).
PartPQ leaves the matching machinery of PQ entirely intact: instance matching is performed exactly as in any of the strategies above on the scene-level class \(l\), yielding the same TP, FP, and FN sets via 3 . The only modification is to the quality term: the instance-level \(\text{IoU}(P_M(g), g)\) in 4 is replaced by a part-aware quantity \(\text{IoU}_p(P_M(g), g)\). For a class \(l\) that carries parts (\(l \in \mathcal{L}^{\text{parts}}\)), \(\text{IoU}_p\) is the mean IOU taken over the part labels of \(l\) within the matched pair, treating voxels outside both \(g\) and \(p\) as background. For a class without parts (\(l \in \mathcal{L}^{\text{no-parts}}\)), it falls back to the instance IOU: \[\label{eq:iou-p} \text{IoU}_p(P_M(g), g) = \begin{cases} \mathrm{mean}\, \text{IoU}_{\text{part}}(P_M(g), g), & l \in \mathcal{L}^{\text{parts}} \\ \text{IoU}_{\text{inst}}(P_M(g), g), & l \in \mathcal{L}^{\text{no-parts}}. \end{cases}\tag{9}\] For \(l \in \mathcal{L}^{\text{parts}}\), the multi-class mean is computed across all part labels of \(l\) plus the background label assigned to voxels outside the matched segments. Predictions falling on the void part label inside \(g\) are not counted as part-level FPs but are still counted as FNs, mirroring the convention of scene-level PQ [1]. Substituting \(\text{IoU}_p\) into 4 , the per-class PartPQ is \[\label{eq:partpq} \mathrm{PartPQ}_l = \frac{\sum_{g \in \mathit{TP}} \text{IoU}_p(P_M(g), g)}{|\mathit{TP}| + \tfrac{1}{2}|\mathit{FP}| + \tfrac{1}{2}|\mathit{FN}|},\tag{10}\] which preserves the SQ\(\times\)RQ structure of standard PQ while embedding the part hierarchy into the quality numerator rather than into a separate matching pass.

Two consequences are worth highlighting. First, the hierarchy is strict: parts of an unmatched instance (\(g \in \mathit{FN}\) or \(p \in \mathit{FP}\)) contribute nothing to the numerator of 10 , and are not separately re-matched as their own instances. A necrotic core that is correctly delineated but whose parent tumor was missed at the instance level scores as a single FN on the parent, not as a part-level success. This is the property that distinguishes PartPQ from the common practice of independently reporting per-region PQ [23]. Second, PartPQ is orthogonal to the choice of instance matcher: any of the strategies in [sec:one-to-one] [sec:many-to-one] [sec:one-to-many] [sec:voronoi] can supply the parent matching, and \(\text{IoU}_p\) is computed downstream from it. The original formulation [2] fixes this matcher to the standard One-to-One rule of [1] with \(\tau = 0.5\). We relax that constraint, so the choice of matcher and the part hierarchy can be configured independently to reflect the failure modes of the underlying data.

4 Evaluation tutorials↩︎

We publish an interactive playground for exploring thresholds below 0.5 and different matching algorithms. The AUTC example (2) and toy example (4) may be extended and modified or completely new scenarios can be configured. Further, we provide two Jupyter notebooks showcasing the application of different strategies and part-awareness in the medical domain, including BraTS metastasis and part-aware liver lesion segmentation. All resources are accessible via an umbrella repository (https://anonymous.4open.science/r/pq-interactive/).

5 Discussion↩︎

We recast PQ as a constrained bipartite assignment and anchor TP/FP/FN counts to vertices rather than edges, which places four well-defined matching strategies on a single design axis and preserves \(|\mathit{TP}|+|\mathit{FN}|=|G|\) across all of them. On top of this, we build a threshold-agnostic summary (AUTC) and a part-aware extension that decouples PartPQ from the original \(\text{IoU}> 0.5\) rule so the matcher and the part hierarchy can be configured independently.
The vertex-based accounting is what lets these four strategies share a metric in the first place. Under any asymmetric matching the edge count \(|M|\) can exceed \(|G|\) or \(|P|\). Identifying TPs with edges would push RQ above \(1\) and dissolve its interpretation as a recognition rate. Anchoring TP and FN to ground-truth vertices closes this gap, so RQ stays directly comparable across One-to-One, Many-to-One, and One-to-Many matchings.
Across the toy example and case studies, the choice of strategy maps onto a tension between RQ and SQ. One-to-One enforces a strict per-object assignment and is the only strategy that penalizes over- and undersegmentation symmetrically. The max-weight greedy variant of [7] occasionally trades a marginal recognition for a higher accumulated overlap when two predictions cover the same reference, but in practice the two coincide in the vast majority of cases. Many-to-One greedy merge is the most lenient strategy under fragmentation, yet it still requires a single edge above \(\tau\) to register a detection: in the toy example, this produces a sharp drop above \(\tau \approx 0.2\) even when accumulated overlap across the reference remains high, so a heavily fragmented prediction can still be counted as a FN. Reporting AUTC bypasses the dependence on a single threshold, and its decomposition into \(\mathrm{AUTC}_{\mathrm{SQ}}\) and \(\mathrm{AUTC}_{\mathrm{RQ}}\) isolates whether a method’s threshold-averaged advantage comes from better localisation or higher recognition.
Choosing among the strategies is therefore less a question of correctness than of which failure modes the downstream task treats as recoverable. One-to-One is the safe default when annotations are clean, instances are well-separated, and a strict per-instance assignment is non-negotiable. Many-to-One fits regimes in which over-segmentation reflects a legitimate decomposition rather than a model failure. One-to-Many is the right tool when adjacent same-class instances are inherently hard to delineate and a merged prediction should still convey the correct semantic and approximate spatial extent. One-to-Many matching remains the most opinionated point in this design space. Crediting a single prediction for several reference segments tolerates undersegmentation in a way that fundamentally rebalances SQ against RQ, and the appropriate behavior depends on whether the downstream task treats a merged object as recoverable or as a hard failure.
The interaction between matcher and part hierarchy is most consequential under Many-to-One. When several adjacent fragments are credited as a single parent instance, their union defines the part-level \(\text{IoU}_p\), so a fragmented prediction whose parts are individually correct can recover full part-level credit, whereas the original One-to-One rule of [2] would reject the whole instance and discard its part labeling along with it.

Limitations, future work, and conclusion. Many-to-Many matching is intentionally outside our framework because the bipartite formulation breaks down once both degree constraints are dropped. Related approaches such as Cluster Dice [5] address the same failure mode through clustering, and a unified treatment that bridges assignment and clustering views remains open. The suggested greedy merge for Many-to-One is an approximation without an established optimality guarantee against \(M^\star_{\text{PQ}}\) (7). Tighter algorithmic bounds or an exact tractable reformulation would put the strategy on firmer ground. Our empirical study is limited in scope: we cover a small number of cases and a single model family per task, with PartPQ evaluated only on biomedical data and threshold-sensitivity analyses run on a subset of the benchmarks, so a larger-scale comparison spanning natural-image datasets, additional architectures, and the full strategy \(\times\) threshold grid is the most pressing direction for follow-up work.

We present a unified framework for PQ below the standard \(\text{IoU}> 0.5\) threshold, recasting segment matching as a constrained bipartite assignment in which vertex-based accounting keeps PQ consistent across One-to-One, Many-to-One, One-to-Many, and Voronoi matchings. The four strategies form a small, well-defined design space, and the right choice is set by which failure modes the downstream task treats as recoverable rather than by a single default. We release the framework as a unified open-source package built on Panoptica, so practitioners can configure the matcher, threshold, and part hierarchy to match their data.

Appendix↩︎

6 Voronoi Matching: Formal Construction↩︎

Let \(\Omega\) denote the image domain and \(d(x, g)\) the (Euclidean) distance from voxel \(x \in \Omega\) to the nearest voxel of ground truth segment \(g \in G\). The Voronoi cell of \(g\) is \[\label{eq:voronoi-cell} V(g) = \{ x \in \Omega : d(x, g) \leq d(x, g') \;\; \forall g' \in G \},\tag{11}\] with ties broken arbitrarily so that \(\{V(g)\}_{g \in G}\) partitions \(\Omega\). For each ground truth segment we define its local prediction as the restriction of all predicted segments to its cell, \[\label{eq:voronoi-local-pred} \tilde{p}(g) = \Big( \bigcup_{p \in P} p \Big) \cap V(g),\tag{12}\] and replace the bipartite edge set of 2 with the per-cell edges \[\label{eq:voronoi-edges} E^{\text{Vor}}_\tau = \{ (g, \tilde{p}(g)) : g \in G,\; \text{IoU}(g, \tilde{p}(g)) > \tau \}.\tag{13}\] The resulting matching \(M^{\text{Vor}} = E^{\text{Vor}}_\tau\) is one-to-one by construction—each ground truth segment is paired with the unique aggregated prediction inside its cell—so the TP/FN/FP accounting of 3 and the SQ/RQ/PQ definitions of 4 apply unchanged. Predicted mass that falls outside every cell never contributes to a TP but is still penalised through the global FP count.

The construction generalises directly: rather than aggregating all predictions inside a cell into a single \(\tilde{p}(g)\), the chosen instance matcher (One-to-One, Many-to-One, or One-to-Many) can be applied independently within each cell, and the per-cell results are aggregated into the global TP/FN/FP counts. This is the form realised by Panoptica’s region-wise pipeline, in which a Voronoi partition is computed once on the reference and the standard bipartite matching is run inside every cell.

7 Optimal Matching Across Strategies↩︎

A matching strategy constrains the structure of \(M\) but does not pin it down: multiple matchings may satisfy a strategy’s degree constraints. We must therefore specify an objective that selects an optimal matching \(M^\star\). Since PQ is the metric we ultimately report, the principled choice is to optimise it directly, \[\label{eq:max-pq} M^\star_{\text{PQ}} = \mathop{\mathrm{arg\,max}}_M \mathrm{PQ}(M).\tag{14}\] PQ couples edges through the union \(P_M(g)\) in the SQ numerator and through the global \(|\mathit{TP}|, |\mathit{FP}|, |\mathit{FN}|\) counts in the RQ denominator, so it does not decompose into a sum over edges and admits no obvious polynomial algorithm.

We therefore work with an edge-decomposable surrogate, the max-weight matching \[\label{eq:max-weight} M^\star_{\text{weight}} = \mathop{\mathrm{arg\,max}}_M \sum_{(g,p) \in M} \text{IoU}(g, p),\tag{15}\] and ask, for each strategy, whether \(M^\star_{\text{weight}}\) coincides with \(M^\star_{\text{PQ}}\) exactly, whether it diverges from \(M^\star_{\text{PQ}}\) and must be replaced by an approximation that lower-bounds \(\mathrm{PQ}(M^\star_{\text{PQ}})\), or whether the matching is uniquely determined by construction.

7.0.0.1 One-to-One.

Under the one-to-one constraint, \(M^\star_{\text{weight}} = M^\star_{\text{PQ}}\) exactly. Every matched ground truth has \(|P_M(g)| = 1\), so the SQ numerator equals the edge-weight sum, and \(|\mathit{TP}| = |M|\), \(|\mathit{FP}| = |P| - |M|\), \(|\mathit{FN}| = |G| - |M|\) make the PQ denominator \(\tfrac{1}{2}(|G| + |P|)\) independent of \(M\). Hence PQ is monotone in the edge-weight sum, and \[\label{eq:1to1-opt} M^\star = \mathop{\mathrm{arg\,max}}_{M \text{ one-to-one}} \sum_{(g,p) \in M} \text{IoU}(g, p)\tag{16}\] solves 14 exactly. \(M^\star\) is computable in \(O((m+n)^3)\) time by the Hungarian algorithm [24]. For \(\tau > 0.5\), the unique-matching property of [1] guarantees that any consistent assignment returns the same \(M^\star\), and our formulation reduces to the original PQ matching. For \(\tau \leq 0.5\) Hungarian assignment is required since the constraint is no longer automatically satisfied as multiple ground truths can compete for the same predicted segment.

7.0.0.2 Many-to-One.

Many-to-One is the only strategy in which the surrogate provably diverges from max-PQ: absorbing a low-overlap predicted segment into an already well-covered ground truth can increase the edge-weight sum while decreasing \(\text{IoU}(g, P_M(g))\), so \(M^\star_{\text{weight}}\) may dilute SQ relative to \(M^\star_{\text{PQ}}\). We therefore drop the proxy and target the SQ contribution per ground truth directly. Finding \(M^\star_{\text{PQ}}\) exactly under this constraint is, to our knowledge, intractable in general, so we adopt an approximation: algorithms that admit a candidate prediction only when it strictly improves the combined overlap, such as the greedy merge of [7], avoid the dilution failure mode and yield a valid matching whose PQ is a lower bound on \(\mathrm{PQ}(M^\star_{\text{PQ}})\).

7.0.0.3 One-to-Many.

Because \(\deg_M(g) \leq 1\), each \(g \in \mathit{TP}\) has \(|P_M(g)| = 1\), and the SQ numerator again equals the edge-weight sum. Moreover, ground truth segments do not compete for predicted segments: a single predicted segment may match arbitrarily many ground truth segments. The max-weight matching therefore reduces to selecting, independently for each ground truth segment, its highest-IoU predicted segment above \(\tau\) (or no edge at all): \[\label{eq:1tom-opt} M^\star = \bigcup_{g \in G} \{(g, \mathop{\mathrm{arg\,max}}_{p : \text{IoU}(g,p) > \tau} \text{IoU}(g,p))\},\tag{17}\] which is computable in \(O(mn)\) time. Max-weight remains our default objective: it coincides with max-PQ whenever \(\tau \geq 0.5\) and differs from it only for low-IoU edges that would barely satisfy the threshold while diluting the SQ average.

7.0.0.4 Voronoi.

No optimisation is required: the partition \(\{V(g)\}\) depends only on the ground truth, not on \(P\), so \(M^{\text{Vor}}\) is uniquely determined and trivially optimal under its own definition. Constructing the cells reduces to a single distance transform on \(G\), computable in \(O(|\Omega|)\) time, after which each per-cell IoU is evaluated in time linear in \(|V(g)|\).

7.0.0.5 Part-Aware.

PartPQ does not introduce a new optimisation problem. The parent matching \(M^\star\) is computed by the optimiser of the chosen instance strategy (16 17 , the greedy merge above for Many-to-One, or the deterministic Voronoi partition of 3.6); \(\text{IoU}_p\) is then evaluated per matched pair in time linear in the matched volume. Optimising the parent matching for \(\mathrm{PartPQ}\) rather than for the scene-level PQ would couple part- and instance-level overlaps and is, to our knowledge, not addressed in the literature; we treat the parent-then-parts decomposition as the operational definition.

8 Figures↩︎

References↩︎