1 Introduction↩︎

Surgical workflow analysis aims to automatically interpret intraoperative activities from video, with surgical phase recognition as a core task to model procedural progression and provide support for clinical decision-making and objective surgical assessment [1], [2]. Although supervised methods achieve strong performance [3], [4], they depend on dense annotations and generalize poorly across institutions and unseen procedures. Vision-language pretraining (VLP) [5]–[8] has emerged as a powerful paradigm for zero-/few-shot generalization, which allows models to transfer knowledge from the educational corpora to new tasks by aligning visual and textual representations.

Several important contributions in vision-language pre-training have been made in surgical workflow analysis [9]–[11]. SurgVLP [12] pioneers CLIP-style pretraining for surgical workflows with large-scale clip-caption datasets spanning multiple surgical types, while HecVL [13] and PeskaVLP [14] further introduce hierarchical supervision in euclidean embedding space and LLM-based language refinement to enrich structured supervision. Related efforts such as SurgLaVi [15] further constructs a hierarchical large-scale surgical video-language dataset to support domain-specific pretraining.

Surgical workflows exhibit a hierarchical tree structure [13]–[15], where phases decompose into ordered steps and fine-grained actions. Existing methods embed these relations in Euclidean space, which lacks the geometric inductive bias to preserve cross-level entailment, weakening parent-child dependencies and limiting structured reasoning and cross-procedure generalization. In contrast, hyperbolic space [16]–[18], with negative curvature and exponential capacity, naturally supports low-distortion embedding of hierarchical data and semantic containment. Moreover, current contrastive objectives [5], [12], [15] treat all non-matching video-text pairs as negatives, even when segments are semantically adjacent or temporally correlated. Such design ignores the inherent tree structure and pushes clinically related nodes apart, introducing structural false negatives and distorting workflow hierarchical tree structure.

To resolve the mismatch between tree-structured surgical knowledge and flat Euclidean embeddings, we propose HyperVLP, a hyperbolic video-language pre-training framework. We project video and text representations into a shared Lorentz hyperbolic manifold that naturally models hierarchical containment. To mitigate structural false negatives introduced by conventional contrastive learning, we design a geometry-aware hyperbolic contrastive loss that down-weights semantically related samples within the same procedural context. To further enforce clinically meaningful parent-child dependencies, we introduce a cone-based hyperbolic entailment objective that imposes geometric containment between parent and child embeddings across both inter-modal and intra-modal relations. Since hierarchical constraints in hyperbolic space can destabilize early alignment [19], [20], we adopt a progressive two-stage optimization strategy that first learns cross-modal alignment via geometry-aware contrastive learning and then enforces hierarchical containment through entailment-based refinement. We evaluate HyperVLP on Cholec80 [21], AutoLaparo [22], and MultiBypass140 [23], demonstrating consistent gains in phase recognition under both zero- and few-shot settings. Overall, our contributions are summarized as follows:

• We propose HyperVLP, a hierarchical surgical video-language pretraining framework in hyperbolic space that explicitly encodes the intrinsic tree structure of surgical workflows.

• We identify structural false negatives in surgical video-language contrastive learning and develop a geometry-aware hyperbolic contrastive strategy for hierarchy-aware alignment.

• We introduce a cone-based hyperbolic entailment objective that enforces geometric containment between parent and child embeddings across both inter- and intra-modal relations.

2 Method↩︎

2.1 Preliminaries: Hyperbolic Geometry↩︎

  Hyperbolic space naturally models hierarchical data due to its constant negative curvature and exponential volume growth. This geometry enables low-distortion embedding of tree-structured surgical workflows, where phases decompose into steps and fine-grained actions. We adopt the Lorentz model \(\mathbb{L}^n\), defined as the upper sheet of a two-sheeted hyperboloid \(\mathbb{L}^n = \{ \mathbf{x} \in \mathbb{R}^{n+1} : \langle \mathbf{x}, \mathbf{x} \rangle_{\mathbb{L}} = - \tfrac{1}{{c}}, x_0 = \sqrt{ \tfrac{1}{{c}} + \| \mathbf{x}_n \|^2 }, \, {c} > 0 \},\) where \({c} \in \mathbb{R}\) represents the curvature and \(\langle \mathbf{x}, \mathbf{y} \rangle_{\mathbb{L}} = \langle \mathbf{x}_n, \mathbf{y}_n\rangle_{\mathbb{E}} - x_0 y_0\) denotes the Lorentzian inner product with \(\langle \cdot, \cdot \rangle_{\mathbb{E}}\) indicating the Euclidean inner product. \(\mathbf{x}_n\) is the \(n\)-dimensional spatial component of \(\mathbf{x}\). The geodesic distance is \(d_{\mathbb{L}}(\mathbf{x}, \mathbf{y}) = \sqrt{\tfrac{1}{c}} \cosh^{-1}\left(-c\langle \mathbf{x}, \mathbf{y} \rangle_{\mathbb{L}}\right).\) Euclidean features are projected to \(\mathbb{L}^n\) by mapping tangent vectors at the origin \(\mathbf{o}=[\sqrt{1/c},\mathbf{0}]\) through the exponential map \(\exp_{\mathbf{o}}^{c}(\mathbf{x}) = \cosh \left( \sqrt{c}\, \| \mathbf{x}\|_{\mathbb{L}} \right) \mathbf{o} + \frac{\sinh \left( \sqrt{c}\, \| \mathbf{x}\|_{\mathbb{L}} \right)}{\sqrt{c}\, \| \mathbf{x}\|_{\mathbb{L}} } \mathbf{x}\).

2.2 Proposed Methodology: HyperVLP↩︎

Hierarchical Embedding in Hyperbolic Space. Our data contains three hierarchical levels (e.g., phase, step, action). During training, we sample any two adjacent levels to form a parent-child pair for hierarchical modeling. For each unit \((V,T,v,t)\), visual and textual features are extracted by encoders \(\mathcal{F}_v\) and \(\mathcal{F}_t\), linearly projected to the tangent space at the origin \(\mathbf{o}=[\sqrt{1/c},\mathbf{0}]\), and mapped to the Lorentz manifold \(\mathbb{L}^n\) via the exponential map: \(V_c = \exp_{\mathbf{o}}^{c}(W_v \mathcal{F}_v(V)), \quad T_c = \exp_{\mathbf{o}}^{c}(W_t \mathcal{F}_t(T)).\) The same transformation is applied to the step-level features \(v\) and \(t\) to obtain \(v_c\) and \(t_c\). Learnable scaling in the tangent space stabilizes optimization under negative curvature.

Geometry-Aware Hyperbolic Contrastive learning.  Standard contrastive objectives [5], [7], [24] always treat all non-matching samples as true negatives. This assumption breaks in hierarchical surgical workflows. We identify two sources of structural false negatives: (i) contextual proximity, where clips correspond to adjacent steps within the same phase; (ii) hierarchical entailment, where fine-grained clips are semantically subsumed by higher-level segments. Treating such samples as independent negatives enforces artificial separation and distorts the workflow topology.

For each anchor \(i\), we partition candidates into three sets (illustrated on the left side of Fig. 1): (i) \(\mathcal{S}(i)\): structurally related samples from the same procedure grouped by temporal containment (including contextual neighbors); (ii)\(\mathcal{P}(i)\): matched positive pairs (including hierarchical ancestors and descendants); (iii) \(\mathcal{N}(i)\): remaining cross-procedure samples. We define the geometry-aware hyperbolic contrastive loss between parent-level visual and textual embeddings as \[\mathcal{L}_{geohc}(\mathbf{V_c}, \mathbf{T_c})=-\frac{1}{B} \sum_{i=1}^{B}\log \frac{ \sum_{k \in \mathcal{P}(i)} {d_{ik}} }{ \sum_{k \in \mathcal{P}(i)} {d_{ik}} + \sum_{k \in \mathcal{S}(i)} {\alpha_{ik}d_{ik}} + \sum_{k \in \mathcal{N}(i)} {d_{ik}} }\] where \(d_{ik} = - d_{\mathbb{L}}(V_c^i, T_c^k)/\tau\) denotes the hyperbolic similarity between anchor \(i\) and sample \(k\). To mitigate structural false negatives, we introduce an adaptive weight \(\alpha_{ik} = \sigma\!\left( - \tfrac{d_{ik}} {\sum_{k \in \mathcal{P}(i)} d_{ik} + \epsilon} \right)\) where \(\sigma(\cdot)\) is the sigmoid function and \(\epsilon\) ensures numerical stability. This design suppresses structurally related samples that exhibit high similarity relative to true positives, thereby preventing over-separation of semantically correlated clips and preserving hierarchical topology. We further construct the symmetric formulation by anchoring textual embeddings against visual candidates, resulting in \(\mathcal{L}_{geohc}(\mathbf{T}_c,\mathbf{V}_c)\). The same geometry-aware mechanism is applied at the child level to enforce hierarchical generalization. The overall hyperbolic contrastive objective is defined as: \[\mathcal{L}_{GeoHCL} = \frac{1}{4} ( \mathcal{L}_{geohc}(\mathbf{V}_c,\mathbf{T}_c) + \mathcal{L}_{geohc}(\mathbf{T}_c,\mathbf{V}_c) + \mathcal{L}_{geohc}(\mathbf{v}_c,\mathbf{t}_c) + \mathcal{L}_{geohc}(\mathbf{t}_c,\mathbf{v}_c) ).\] Hierarchical Entailment Learning. Some prior studies [19] investigate hierarchical representation learning in hyperbolic space. In our work, we introduce a hierarchical-aware hyperbolic entailment mechanism that explicitly models multi-level semantic containment across modalities. Specifically, we adopt hyperbolic entailment cones in the Lorentz space, where each point \(\mathbf{x}\) defines a conical region \(\mathbb{R}_{\mathbf{x}}\) such that \(\mathbf{y} \in \mathbb{R}_{\mathbf{x}}\) is semantically subsumed by \(\mathbf{x}\). The half-aperture is given by \(\theta(\mathbf{x}) = \sin^{-1}\left(\tfrac{2K}{\sqrt{c}\| \mathbf{x}_{n} \|}\right)\), where \(c\) denotes curvature and \(K\) controls behavior near the origin. The aperture shrinks with increasing \(\| \mathbf{x}_{n} \|\), naturally enforcing tighter semantic specificity away from the origin. During training, we constrain a specific concept \(\mathbf{y}\) to lie within the aperture \(\theta(\mathbf{x})\) of its parent concept \(\mathbf{x}\) by penalizing the angular residual of an outward point \(\mathbf{y}\) with exterior angle denoted as \(\varphi(\mathbf{y},\mathbf{x}) =\tfrac{y_0 + c x_0 \langle \mathbf{y},\mathbf{x} \rangle_{\mathcal{L}}} {\|\tilde{x}\|\sqrt{c\langle \mathbf{y},\mathbf{x} \rangle_{\mathcal{L}}^2 - 1}}\), where \(y_0\) and \(x_0\) denoting the time dimension. Thus, the entailment loss is formulated as: \(L_{ent}(\mathbf{y},\mathbf{x}) = \max \left( 0, \varphi(\mathbf{y},\mathbf{x}) - \eta \theta(\mathbf{x}) \right)\), where \(\eta\) is a threshold to adjust the distance region.

To explicitly model semantic containment in surgical workflows, we introduce both inter-modal and intra-modal entailment supervision in hyperbolic space. Across modalities, we treat textual concepts as semantically containing their corresponding videos (e.g., “cut” entails all cutting clips), and enforce containment at both child and parent levels via \(L_{ent}(v_{c}, t_{c})\) and \(L_{ent}(V_{c}, T_{c})\), yielding the inter-modal loss \(\mathcal{L}_{inter} = L_{ent}(v_{c}, t_{c}) + L_{ent}(V_{c}, T_{c})\). Within each modality, we further impose parent-child structural constraints to preserve the inherent tree-like organization of surgical workflows, formulated as \(\mathcal{L}_{intra} = L_{ent}(t_{c}, T_{c}) + L_{ent}(v_{c}, V_{c})\). Together, these objectives enforce hierarchical consistency both across and within modalities.

Training Paradigm. We adopt a two-stage optimization strategy to progressively stabilize hierarchical alignment in hyperbolic space. In Stage I, all model parameters are optimized using only the proposed geometry-aware hyperbolic contrastive loss, aiming to establish robust cross-modal alignment while mitigating structural false negatives under hierarchical semantics. In Stage II, we freeze the visual and textual encoders and fine-tune only the hyperbolic mapping layers. In this stage, the model is jointly optimized with the geometry-aware hyperbolic contrastive loss and the cone-based hyperbolic entailment losses (both inter- and intra-modal), explicitly enforcing hierarchical containment constraints.

3 Experiments↩︎

3.1 Experimental Details↩︎

Training Dataset. Following prior work [12], we utilize the Surgical Video Lectures (SVL) dataset shared by the authors of  [12] and reorganize it into an explicit hierarchical video-text corpus for structured pretraining. SVL contains large-scale surgical lecture videos with ASR-generated transcripts and structured annotations such as keystep descriptions and video-level abstracts [13]. Instead of treating video-text pairs as flat clip-caption alignments, we explicitly construct parent-child units at three levels: fine-grained clips with local descriptions, phase-level segments with procedural summaries, and full videos with global abstracts. Parent segments are temporally composed of child clips, forming a tree-structured workflow hierarchy. This design enables supervision across modalities and across semantic levels through explicit containment relations.

Downstream Datasets and Evaluation. To validate the generalization capability of our method, we evaluate on three established surgical benchmarks: Cholec80 [21] (cholecystectomy), AutoLaparo [22] (hysterectomy), and Multibypass140 [23] (gastric bypass). Multibypass140 contains 140 laparoscopic Roux-en-Y gastric bypass surgeries annotated with 12 phases and 46 steps. Following prior work, we report results separately on its two center-specific subsets, StrasBypass70 (Stras70) and BernBypass70 (Bern70), each comprising 70 cases collected from the University Hospital of Strasbourg and Bern University Hospital, respectively, to assess cross-center generalization. All these datasets differ from our training data in both surgical type and medical center, introducing substantial domain shifts. Performance is evaluated on the official test splits. In all tables, results are reported as Top-1 accuracy and F1-score (Acc/F1).

Implementation Details. We adopt ResNet-50 [25] as the visual encoder \(\mathcal{F}_v\) and BioClinicalBERT [26] as the textual encoder \(\mathcal{F}_t\), both initialized from pretrained weights. In Stage I, training is performed with a global batch size of 256 on 8 NVIDIA A100 GPUs using the AdamW optimizer, with an initial learning rate of \(5\times10^{-5}\). In Stage II, training is conducted on a single GPU with batch size 128 and a reduced learning rate of \(1\times10^{-5}\). For fair comparison, all baselines follow the same SVL dataset and protocol as PeskaVLP [14].

3.2 Results↩︎

Zero-shot Phase Recognition. Table 1 reports zero-shot phase recognition across procedures (Cholec80, AutoLaparo) and centers (Stras70, Bern70). General-purpose models (MIL-NCE [28], CLIP [5]) show limited transferability, especially under cross-procedure and cross-center settings. Surgical VLMs pretrained on SVL substantially improve performance, confirming the benefit of domain-specific pretraining. Among all methods, HyperVLP consistently achieves the best results on all four benchmarks, with clear margins on both Accuracy and F1-score, particularly under cross-center evaluation (Stras70 and Bern70). Notably, HecVL and PeskaVLP are themselves Euclidean hierarchical surgical VLP methods built on the same SVL data and backbone. The consistent improvement of HyperVLP over them isolates the benefit of modeling the workflow hierarchy in hyperbolic rather than Euclidean space, beyond non-hierarchical CLIP/SurgVLP-style baselines.

Few-/Full shot Linear Probing. Following the protocol of PeskaVLP [14], we evaluate HyperVLP under 10% and 100% settings with frozen encoders and a linear classifier using V, L, and VL supervision. As shown in Table 2, general-purpose models show limited cross-procedure transfer, especially in the few-shot regime. Surgical pretraining improves performance, while HyperVLP consistently achieves the best results across datasets and supervision settings. The pronounced gains in the 10-shot setting indicate that our hyperbolic hierarchical modeling yields more linearly separable and procedure-agnostic representations.

Qualitative Analysis. To analyze how different semantic granularities are organized in hyperbolic space, we examine the distribution of embedding distances to the origin, where the radial coordinate reflects the level of hierarchical abstraction. Figure 2 presents the radial distance distributions of step- and phase-level video embeddings, along with their paired textual representations. The textual descriptions are generated with the assistance of a large language model [29] to ensure semantic consistency. We observe a clear separation between semantic levels as well as between modalities. Importantly, the relative ordering of step- and phase-level representations remains consistent across StrasBypass70 (a) and BernBypass70 (b), indicating that the learned hyperbolic space preserves a stable and procedure-invariant hierarchical structure.

Ablation Study. We evaluate the contribution of each loss component to phase recognition. As shown in Table 3, adding the geometry-aware contrastive loss (\(\mathcal{L}_{GeoHCL}\)), inter-modal entailment (\(\mathcal{L}_{inter}\)), and intra-modal entailment (\(\mathcal{L}_{intra}\)) progressively improves performance across datasets. \(\mathcal{L}_{GeoHCL}\) and \(\mathcal{L}_{inter}\) provide the primary gains, while \(\mathcal{L}_{intra}\) further enhances hierarchical consistency. The best results are achieved when all components are jointly optimized.

4 Conclusion↩︎

In this paper, we presented HyperVLP, a hyperbolic surgical video-language pretraining framework that explicitly models the intrinsic tree structure of surgical workflows. Unlike prior Euclidean approaches that treat workflow labels as flat categories, our method embeds visual and textual representations into a shared Lorentz manifold to preserve hierarchical containment. We introduced a geometry-aware hyperbolic contrastive objective to alleviate structural false negatives and a cone-based entailment mechanism to enforce parent-child consistency across and within modalities. A progressive two-stage optimization strategy further stabilizes hierarchical alignment in hyperbolic space. Extensive experiments on three publicly available benchmarks demonstrate consistent improvements in zero-shot and few-shot phase recognition, particularly under cross-procedure and cross-center domain shifts.

5 Acknowledgments↩︎

This research was partially supported by "Pioneer" and "Leading Goose" R&D Program of Zhejiang under Grant No. 2025C01132. This work benefited from French state funds managed by the National Research Agency via the IHU Strasbourg (ANR-10-IAHU-0002) and the ENACT AI Cluster (ANR-23- IACL-0004).

References↩︎