RelAfford6D: Relational 6D Affordance Graphs for Constraint-Driven Robotic Manipulation
June 25, 2026
Abstract
Bridging abstract semantics and precise physical control remains a fundamental challenge in open-world robotic manipulation. While recent data-driven policies show promise, their reliance on isolated contact points or latent affordance embeddings lacks the rigorous kinematic constraints necessary for complex articulated objects.To overcome the limitation, we introduce RelAfford6D, a novel training-free framework centered on a Relational 6D Affordance Graph. Given a free-form instruction, our system deduces a semantic topology linking a primary interacting part to its physical anchor. By elevating these topological nodes into precise metric \(SE(3)\) poses via vision foundation models, we analytically formulate downstream execution as a kinematic constraint satisfaction problem. The robot synthesizes continuous trajectories by tracking strictly defined physical manifolds (, revolute or prismatic orbits). Coupled with a closed-loop tracking mechanism for dynamic replanning against disturbances, our physically grounded approach achieves superior zero-shot success rates, cross-category generalization and execution robustness in both simulation and the real world environments, outperforming existing data-driven baselines.
1 Introduction↩︎
The goal of open-world robotic manipulation is to enable robots to understand natural language instructions and reliably interact with physically diverse, unseen objects [1]. This process is fundamentally challenged by the perception-action gap: the difficulty of transforming high-dimensional, multi-modal observations into precise, low-level physical control commands [2]. Recently, data-driven policies and foundation model-based paradigms [3]–[8] have attempted to bridge this gap using large-scale datasets and visual priors. However, a critical bottleneck remains in their underlying object representations, creating an irreconcilable tension between expressiveness and kinematic physical constraints. How objects are represented for manipulation therefore becomes a central issue. Particularly for complex articulated objects, existing representations often fall short. Traditional 2D pixel-level affordances [9] intrinsically lack 3D physical meaning. Conversely, 3D approaches based on sparse point clouds[10], [11] or keypoints [12], provide isolated contact coordinates, failing to encode the continuous relative geometry between interacting parts explicitly. Meanwhile, cutting-edge dense descriptor fields [13], [14] boast high expressiveness but typically formulate manipulation as navigating an implicit energy field. Ultimately, both 3D paradigms lack the explicit, hard kinematic constraints indispensable for structured manipulation. For instance, successfully opening a microwave does not merely depend on reaching its handle; it is strictly governed by the rigid-body relative geometry between the door and the main body.
To resolve this bottleneck, we introduce a novel intermediate representation: the Relational 6D Affordance Graph. Our key insight is that articulated manipulation is fundamentally governed by the relative rigid-body geometry between object parts rather than isolated contact points. Our representation explicitly models the manipulated object as a dynamic spatial relationship in \(SE(3)\), composed of a primary interacting part and a physical anchor part. This representation not only preserves the absolute precision of 3D metric spaces but, more importantly, naturally maps complex semantic operations into relative geometric constraint manifolds between the object parts, thereby establishing strict, mathematically grounded physical boundaries for downstream execution.
Built upon this representation, we present RelAfford6D, a novel training-free manipulation framework. We operationalize this framework through a principled pipeline that seamlessly bridges abstract semantic reasoning, visual grounding, and physical constraint solving. First, via Semantic Topology Generation, we leverage a structured kinematic knowledge base and retrieval-augmented reasoning to guide Large Language Models (LLMs). This module deduces the underlying kinematic chain of unseen objects directly from natural language, explicitly identifying the primary interacting part (\(T_P^*\)), its physical anchor (\(T_A^*\)), and the associated relational action primitive. Next, through Metric Visual Grounding, we integrate the robust zero-shot capabilities of Vision Foundation Models (VFMs) [15]–[17]. By elevating 2D part-level segmentation into 3D geometry estimation, we instantiate these abstract graph nodes with precise real-world \(SE(3)\) poses. Finally, we formulate action execution as a Closed-Loop Constraint-Driven Solving Process[18], [19]. Rather than relying on learned end-to-end latent features or open-loop heuristic planners, the downstream execution is mathematically redefined as a continuous \(SE(3)\) kinematic constraint satisfaction problem. By dynamically optimizing the relative 6D geometry between the primary and anchor parts, and continuously tracking the affordance nodes in real-time, our system synthesizes kinematically valid trajectories. This closed-loop formulation naturally detects spatial deviations and triggers online replanning, ensuring unprecedented robustness against dynamic external disturbances. Our main contributions are summarized as follows:
We propose the Relational 6D Affordance Graph, a novel intermediate representation that models manipulation as part-conditioned \(SE(3)\) relations, explicitly encoding rigid-body and articulation constraints.
We introduce a structured part-level kinematic ontology that encodes reusable relational priors for articulated objects, enabling zero-shot semantic topology inference directly from natural language without category-specific templates.
We reformulate manipulation as a closed-loop optimization on a relational \(SE(3)\) constraint manifold, replacing learned policies and heuristic primitives with physically grounded geometric feedback control.
Empirical results on diverse manipulation tasks demonstrate that our method achieves zero-shot performance and higher efficiency than training-based approaches, without requiring any task-specific training.
2 Related Works↩︎
Visual Affordance and Object Representations. Prior work on affordance learning and object representation can be broadly categorized by how they model the spatial and physical properties of the interaction space. A fundamental line of research focuses on 2D image planes, predicting pixel-wise affordance heatmaps or part-level masks to indicate action possibilities [2], [6], [9], [20]–[22]. While effectively grounding semantics, these 2D representations intrinsically lack metric depth, limiting their utility for precise 3D physical execution. Moving into 3D, recent zero-shot paradigms heavily leverage vision foundation models to extract spatial affordances. One prominent line of work represents affordances as sparse 3D point clouds[23]–[25] or keypoints[12], [26]. While these methods successfully bridge language and spatial locations without task-specific training, they formulate actions via isolated contact coordinates. Consequently, they struggle to explicitly encode the continuous relative geometry (, precise rotational axes) required for manipulating articulated objects. To enrich spatial expressiveness, another cutting-edge paradigm employs dense 3D descriptor fields or spatial value maps [13], [14]. They treat the workspace as an implicit energy field, guiding robotic trajectories by optimizing continuous feature gradients. Although highly effective for generic spatial reasoning or manipulating deformable items, they fundamentally lack explicit, hard kinematic constraints (such as rigid-body joint limits), which are indispensable for structured articulated manipulation. In contrast, 6D affordance representations [27], [28] explicitly encode both position and orientation. Yet, prior 6D methods still treat affordances as isolated actionable poses. Our RelAfford6D transcends this limitation by introducing a Relational 6D Affordance Graph. We explicitly capture the relative geometric structure, embedding mathematically rigorous \(SE(3)\) kinematic constraints directly into the zero-shot manipulation pipeline.A summarized comparison of representative affordance learning approaches is provided in Table 1.
Articulated Object Manipulation. As embodied AI technology develops, robots need to achieve precise interactions with common 3D articulated objects like laptops and drawers. Compared with grasping problems [29], [30], interacting with articulated objects requires a deeper understanding of kinematics, dynamics, and the ability to apply precise forces and torques over extended trajectories. To achieve this, datasets and simulation environments have been built [31]–[34]. Early approaches relied on geometric fitting or model-based estimation of articulation parameters such as joint types and axes [35], [36]. To better handle kinematic variability, primitive-based frameworks decompose manipulation into actions such as pulling and rotating, with parameters optimized via learning or planning [6], [37]. However, such methods often require task-specific training and remain brittle under distribution shift. Alongside affordance-based algorithms [27], [38], [39], flow-based [40], [41], RL-based [42], and diffusion-based [43], [44] algorithms have also shown remarkable achievements within simulated environments. However, these policies and algorithms are typically trained on relatively small real-world datasets that capture limited categories and tasks, restricting their generalization to open-world settings. In contrast to these paradigms that either rely on task-specific data or treat manipulation as isolated point-based interactions, our work fundamentally addresses the underlying physical constraints of articulated objects. By mathematically formulating action execution as an \(SE(3)\) kinematic constraint satisfaction problem, our framework naturally synthesizes kinematically valid trajectories without relying on task-specific training. This physically grounded approach enables robust, training-free manipulation generalization across both diverse simulated and real-world environments.
3 Method↩︎
In this section, we present RelAfford6D, a training-free robotic manipulation framework built upon the Relational 6D Affordance Graph. As illustrated in 2, our framework seamlessly bridges high-level semantic reasoning and low-level physical control through three core modules: Semantic Topology Generation, Metric Visual Grounding, and Constraint-Driven Kinematic Execution. Specifically, we address the following key questions: (1) How can we deduce the underlying kinematic chain and relational node topology of unseen objects from open-world language instructions? (3.1) (2) How do we elevate these topological nodes from 2D semantics into metric, geometry-aware \(SE(3)\) poses from raw visual observations? (3.2) (3) How can we mathematically formulate and solve the downstream manipulation as a robust, continuous \(SE(3)\) constraint satisfaction problem? (3.3)
3.1 Semantic Topology Generation↩︎
A central challenge in zero-shot manipulation is bridging the gap between unconstrained natural language and strict physical kinematic structures. While Large Language Models (LLMs) exhibit strong semantic reasoning in open-world scenarios, they lack the inherent physical priors to output kinematically valid part relations. To address this, we leverage PartNet-Mobility [32], a large-scale dataset with part-level semantics and mobility attributes, to construct a structured kinematic knowledge base. By integrating retrieval-augmented generation (RAG) with LLM reasoning, we perform zero-shot Semantic Topology Generation, effectively deducing the relational node topology of unseen objects without heuristic parsing.
Structured Kinematic Knowledge Base. Firstly, we construct a kinematic relation base \(\mathcal{K}\) from the PartNet-Mobility dataset. Every articulated object is abstracted into a set of interacting nodes \(s \in \mathcal{S}\) (, door, handle, knob), linked by kinematic constraint types \(a \in \mathcal{A}\) (, grasp, pull, rotate). Crucially, we model these relationships not as isolated interactions, but as relational pairs involving a primary interacting part \(s_p\) and a physical anchor part \(s_a\). Here, the token \(a\) serves as the topological edge defining the specific geometric constraint manifold (, revolute or prismatic) between the nodes. For example, the token rotate explicitly defines a revolute constraint edge between the pair (laptop screen, laptop base). Formally, the knowledge base is constructed as follows: \[\mathcal{K} = \{(s_{pi},\, s_{ai},\, a_i)\}_{i=1}^N\] Rather than merely providing a part-action mapping, this structured base encodes the fundamental kinematic chains essential for defining relative geometric constraints. Details are provided in Appendix A.
Zero-Shot Topology Retrieval. Prior to retrieval, raw user queries are processed by our LLM reasoning module to generate a concise semantic query \(q\). We embed \(q\)
using a language encoder \(\phi(\cdot)\) [45], and retrieve the top-\(k\) most relevant kinematic chain candidates from \(\mathcal{K}\): \[R(q) = \operatorname*{top-k}_{(s_i,a_i)\in\mathcal{K}} \;\mathrm{sim}\big(\phi(q),
\phi((s_{pi},s_{ai}), a_i)\big),\] where \(\mathrm{sim}(\cdot,\cdot)\) is the cosine similarity metric. For example, if \(q=\) “open the microwave door”, the retrieval does not just
return a single target, but returns the explicit relational pair \((\texttt{handle}, \texttt{microwave body})\) and the corresponding kinematic constraint type Revolute.
Relational Topology Inference. The retrieved candidates \(R(q)\) serve as a structured kinematic prior injected into the prompt provided to the LLM. Instead of unconstrained free-form reasoning, the LLM is guided to output a rigorous relational node topology: \[((s_p,s_a), a) = \mathrm{LLM}\big(q \mid R(q)\big),\] where \((s_p,s_a)\) define the exact semantic nodes for the primary and anchor parts, and \(a\) acts as the Kinematic Manifold Selector, dictating the specific topological edge (, a revolute or prismatic constraint) connecting them. This hybrid conditioning ensures that the system leverages the open-world generalization capabilities of the LLM while remaining strictly anchored to physically valid kinematic structures derived from PartNet-Mobility. The inferred topology \(((s_p,s_a), a)\) is then passed to the visual perception module to be instantiated in metric 3D space, and subsequently guides the constraint-driven execution module. Thus, this module transcends semantic filtering; it bridges open-world language queries with the mathematically rigorous topological graphs required for physical execution. An example of this process is illustrated in 3.
3.2 Metric Visual Grounding↩︎
Open-world robotic manipulation demands elevating abstract semantic reasoning into precise physical metric spaces. Existing methods often decouple semantic grounding from pose estimation or rely on sparse point clouds, resulting in a semantic–geometric gap: they may detect objects but fail to identify the precise 6D orientation of interactive parts required for kinematic control. We address this with a unified pipeline that instantiates the inferred topological nodes—the primary part \(s_p\) and the anchor part \(s_a\)—into explicit \(SE(3)\) coordinates.
Given an RGB-D observation \(\{I, D\}\) and the relational node pair \((s_p, s_a)\) derived from the high-level reasoning stage (3.1), the metric visual grounding module outputs a physically instantiated spatial graph \(\Psi = \{(M_p, \mathbf{T}_P^*), (M_a, \mathbf{T}_A^*)\}\). Here, \(M\) denotes the fine-grained part-level masks, and \(\mathbf{T}^* \in SE(3)\) represents the corresponding 6D poses. Formally, we define: \[\Psi = g_{\text{percept}}\big(I, D, (s_p, s_a)\big), \label{eq:percept}\tag{1}\] where the function \(g_{\text{percept}}\) is composed of two cascaded stages: \(f_{\text{geo}} \circ f_{\text{sem}}\).
The first stage, \(f_{\text{sem}}\), performs language-guided metric grounding using an open-world segmentation model (SAM3 [15]). Given the visual input \(I\) and the textual node query \((s_p, s_a)\), it generates independent binary
segmentation masks for both parts: \[\{M_p, M_a\} = f_{\text{sem}}\big(I, (s_p, s_a)\big).\] Unlike standard class-based segmentation, this step is explicitly guided by the structured kinematic intent rather than fixed
category labels. It converts the abstract semantic nodes (, handle and microwave body) into spatially structured, physically meaningful visual extents.
The second stage, \(f_{\text{geo}}\), performs rigid-body 6D pose reasoning on these extracted regions. The masked RGB-D regions, camera intrinsics \(K\), and an object-specific mesh \(\mathcal{M}_{obj}\) reconstructed from RGB-D reference views are fed into a category-agnostic pose estimator (FoundationPose [16], [17]) to obtain the part-level transformations: \[\{\mathbf{T}_P^*, \mathbf{T}_A^*\} = f_{\text{geo}}\big(I, D, \{M_p, M_a\}, K, \mathcal{M}_{obj}\big),\] where \(\mathbf{T}^* = [\mathbf{R} | \mathbf{t}]\) represents the 3D rotation and translation of the specific part in the world coordinate system. In contrast to conventional object-level pose estimators that reason over entire instances, our approach constrains pose inference within the specific topological masks. This yields independent, high-fidelity 6D coordinates for both the primary acting part and its physical anchor. Detailed visualizations of this metric grounding process are shown in 4. No category-level CAD model is assumed in our experiments; the reconstructed mesh is used in the model-free FoundationPose pipeline.
The key insight of this module lies in aligning semantic topology with geometric physics. By coupling semantic localization and geometry estimation in one pipeline, the resulting instantiated graph \(\Psi\) effectively bridges perception and control. Instead of treating parts in isolation, it provides the exact \(SE(3)\) boundary conditions required for the downstream constraint-driven execution (3.3).
3.3 Constraint-Driven Kinematic Execution↩︎
Given the instantiated spatial graph \(\Psi = \{(\mathbf{T}_P^*, \mathbf{T}_A^*)\}\) and the topological edge token \(a \in \mathcal{A}\) derived from the previous modules, the execution module converts this relational representation into a continuous 6-DoF robotic trajectory. We formulate manipulation as a \(SE(3)\) Kinematic Constraint Satisfaction Problem. Since the feasible motion space is analytically defined by joint geometry, no data-driven policy learning is required to infer valid motion directions.
Kinematic Constraint Manifolds. In our formulation, the semantic token \(a\) (, rotate, pull) serves as a Kinematic Manifold Selector. It defines the specific geometric
constraint manifold \(\mathcal{M}_a\) that governs the relative spatial relationship between the primary part \(\mathbf{T}_P^*\) and the anchor part \(\mathbf{T}_A^*\). The manifold reduces the feasible motion space from 6 DOF to a 1D subspace aligned with the physical joint. To ensure physical plausibility when interacting with articulated objects, the end-effector pose
\(\mathbf{T}_{ee} \in SE(3)\) must evolve along this manifold.
Firstly, the end-effector establishes a rigid grasp on the primary part, defined by an initial offset \(\mathbf{T}_{\text{offset}} = (\mathbf{T}_P^*)^{-1} \mathbf{T}_{ee}(0)\). During execution, the target state parameterized by a 1D variable \(q\) (, linear displacement or rotation angle) must strictly satisfy the relative geometry dictated by the anchor \(\mathbf{T}_A^*\):
Prismatic Constraint Manifold (\(a = \texttt{pull}\)): The motion is restricted to a 1D translation along an axis \(\mathbf{v}\) defined in the local coordinate frame of the anchor part. The translation axis is inferred from the relational topology between the primary and anchor parts. Specifically, given the instantiated spatial graph, we estimate a dominant screw axis \(\boldsymbol{\xi}_{\text{pris}} = (\mathbf{v}, \mathbf{0})\) that is geometrically consistent with the articulated structure and the semantic action prior. The feasible motion manifold is then defined as: \[\mathcal{M}_{\text{pris}}(q) = \mathbf{T}_A^* \cdot \mathbf{Trans}(\mathbf{v} q) \cdot (\mathbf{T}_A^*)^{-1} \mathbf{T}_P^* \cdot \mathbf{T}_{\text{offset}},\] where \(\mathbf{Trans}(\cdot)\) denotes pure translation in \(SE(3)\) and \(q \in \mathbb{R}\) is the scalar displacement.
Revolute Constraint Manifold (\(a = \texttt{rotate}\)): The motion follows a 1D rotational orbit around a hinge axis \(\mathbf{u}\) anchored at \(\mathbf{T}_A^*\). The rotation axis is inferred from the part-level spatial configuration and the semantic relation token, yielding a dominant screw axis \(\boldsymbol{\xi}_{\text{rev}} = (\mathbf{u}, \mathbf{p} \times \mathbf{u})\) that satisfies articulated joint consistency. The constraint manifold is defined as: \[\mathcal{M}_{\text{rev}}(q) = \mathbf{T}_A^* \cdot \mathbf{Rot}(\mathbf{u}, q) \cdot (\mathbf{T}_A^*)^{-1} \mathbf{T}_P^* \cdot \mathbf{T}_{\text{offset}},\] where \(\mathbf{Rot}(\mathbf{u}, q) = \exp(\hat{\boldsymbol{\xi}}_{\text{rev}} q) \in SE(3)\).
The joint axis \(\mathbf{v}\) and \(\mathbf{u}\) is analytically computed from the relative transformation between primary and anchor parts by aligning to the dominant geometric direction consistent with the semantic action token; detailed derivations are provided in the Appendix.
Trajectory Generation via Manifold Tracking. To balance collision safety and kinematic precision without introducing computationally expensive non-linear optimizations, we employ a decoupled two-stage trajectory generation strategy. First, for the global reaching phase, we utilize a sampling-based motion planner (RRT-Connect [46]) to safely navigate the end-effector to the initial grasp pose \(\mathbf{T}_{ee}(0)\) while naturally avoiding environmental obstacles. Second, for the local execution phase, the manipulation trajectory is analytically synthesized by directly tracking the instantiated constraint manifold. Given the task-specific target state \(q_{\text{target}}\) which can be inferred from the LLM’s semantic priors (, \(60^\circ\) for opening a microwave) or instantiated as a default kinematic exploration horizon, we uniformly discretize the 1D state interval \([0, q_{\text{target}}]\) into \(N\) steps to generate a sequence of strictly valid \(SE(3)\) waypoints: \(\tau^* = \{\mathcal{M}_a(q_k)\}_{k=1}^N\), where \(q_N = q_{\text{target}}\). This design ensures that global reaching remains collision-free, while local execution strictly follows the articulated joint geometry.
Closed-Loop Tracking and Replanning. To ensure robustness against dynamic external disturbances, we execute the trajectory in a closed-loop fashion. Leveraging the continuous tracking capability of FoundationPose at 5 Hz, we obtain live state updates of the active parts, denoted as \(\hat{\mathbf{T}}_P(t)\) and \(\hat{\mathbf{T}}_A(t)\).
Unlike open-loop affordance methods, our system continuously monitors the topological constraint error. If the deviation between the live tracked anchor pose and its initially planned state exceeds a geometric threshold \(\delta\) (set to 5 cm in our implementation), the constraint manifold is inherently shifted by the physical disturbance. In such events, the system immediately re-instantiates the specific manifold using the live coordinates and analytically re-synthesizes the valid \(SE(3)\) waypoints over the remaining horizon \(q \in [q_{\text{current}}, q_{\text{target}}]\): \[\tau_{t:N}^* = \Big\{ \mathcal{M}_a\big(q_k \mid \hat{\mathbf{T}}_A(t), \hat{\mathbf{T}}_P(t)\big) \Big\}_{k=t}^N,\] where the remaining sequence \(\{q_k\}_{k=t}^N\) interpolates towards the invariant target \(q_{\text{target}}\). This formulation ensures that replanning is not a simple heuristic reset, but a mathematically rigorous recalculation of the \(SE(3)\) trajectory based on the dynamically updated physical boundary conditions. Real-world experimental results demonstrating this dynamic robustness are shown in 5.
4 Experiments↩︎
4.1 Experiments Setup↩︎
Environment. Based on the SAPIEN physical simulator [33], we establish a comprehensive interactive environment for manipulation tasks. For the robotic actuator, we deploy a Franka Panda arm equipped with either a parallel-jaw gripper or a suction cup, depending on the specific task mechanics.
Task Suites. Since our framework is entirely training-free and fundamentally driven by geometric constraints, we evaluate its generalization under two complementary settings: (a) Articulated Object Manipulation. We define a benchmark comprising complex kinematic tasks governed by prismatic and revolute joints (, pulling a cabinet door, closing a laptop, turning a faucet handle). To ensure a strictly fair comparison, all learning-based baselines are trained on a unified dataset of 10,000 successful motion-planned trajectories spanning 20 articulated object categories. Detailed training protocols, observation spaces, and hyperparameter settings for all baselines are deferred to the Appendix. (b) Rigid Object Manipulation. To evaluate broad generalizability beyond articulated mechanisms, we additionally include classical rigid-object tasks (grasping, pushing, stacking). Baseline policies (, OpenVLA [3], Octo [47]) are trained using 5,000 expert demonstrations. Full implementation details are provided in the Appendix. In contrast, our method requires absolutely zero training or demonstrations, operating directly in closed-loop mode from live RGB-D observations.
Evaluation Metric. For articulated object manipulation, we adhere to the standard success metrics from prior literature [4], [6]. Let \(\theta_{\text{before}}\) and \(\theta_{\text{after}}\) denote the physical joint state (rotation angle in radians or prismatic displacement in meters) before and after the interaction. A trial is strictly marked successful if the state change exceeds a task-specific threshold: \(|\theta_{\text{after}} - \theta_{\text{before}}| \geq 0.1\). For rigid object manipulation, a trial is considered successful if the final spatial goal is achieved within a strict tolerance (, translation error \(< 3\) cm). All metrics are reported as the success rate over test trials.
| Method |  |
 |
 |
 |
 |
 |
 |
 |
 |
 |
AVG |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Where2act [39] | 0.15 | 0.23 | 0.15 | 0.40 | 0.26 | 0.68 | 0.40 | 0.07 | 0.18 | 0.27 | 0.27 |
| FlowBot3D [41] | 0.61 | 0.27 | 0.13 | 0.26 | 0.67 | 0.70 | 0.29 | 0.18 | 0.53 | 0.32 | 0.40 |
| ManipLLM [6] | 0.65 | 0.43 | 0.26 | 0.64 | 0.68 | 0.71 | 0.67 | 0.60 | 0.75 | 0.77 | 0.62 |
| RoboMamba [48] | 0.81 | 0.86 | 0.30 | 0.91 | 0.81 | 0.61 | 0.63 | 0.50 | 0.55 | 0.85 | 0.68 |
| Ours | 0.78 | 0.90 | 0.35 | 0.92 | 0.71 | 0.82 | 0.76 | 0.62 | 0.73 | 0.87 | 0.74 |
3pt
4.2 Quantitative Comparison↩︎
We evaluate our method against state-of-the-art baselines across the two complementary manipulation suites. While baselines rely heavily on domain-specific demonstrations, our approach remains fully zero-shot. Across both benchmarks, the results validate a core hypothesis: combining a relational affordance graph with rigorous kinematic constraint tracking provides stronger open-world generalization than end-to-end data-driven methods.
Articulated Manipulation. We compare our method with four representative baselines: Where2act [39], FlowBot3D [40], ManipLLM [6], and RoboMamba [4]. 2 reports the success rates across ten object categories. Despite being entirely training-free, our approach achieves an average success rate of 74%, outperforming all data-driven baselines. The performance gap fundamentally stems from how the action space is modeled. Methods like Where2Act and FlowBot3D primarily infer isolated contact points or 3D flow fields. Similarly, ManipLLM and RoboMamba regress end-effector poses from 2D or 3D visual features. These approaches lack awareness of the global joint mechanisms, often leading to kinematic violations (, pulling a revolute door in a straight line), which causes slippage or execution failure. By contrast, our method explicitly extracts the metric instantiated graph \(\Psi = \{(\mathbf{T}_P^*, \mathbf{T}_A^*)\}\) and enforces strict \(SE(3)\) manifold constraints (\(\mathcal{M}_{\text{pris}}\) and \(\mathcal{M}_{\text{rev}}\)). This physically grounded formulation guarantees the generated trajectories strictly adhere to the articulated joint geometry, yielding superior robustness on articulated objects.
Rigid Object Manipulation. 3 presents results on three classical rigid-object tasks. Every method is evaluated over 250 trials (10 random seeds with 25 trials per seed). Despite involving a different control objective (object-level 6D reasoning instead of articulation control), our framework generalizes without any task-specific re-training.
| Method | PickCube | StackCube | PushCube |
|---|---|---|---|
| RDT[49] | \(77.2 \pm 0.48\) | \(74.0 \pm 0.30\) | \(100.0 \pm 0.00\) |
| OpenVLA[3] | \(8.0 \pm 0.00\) | \(8.0 \pm 0.00\) | \(8.0 \pm 0.00\) |
| Octo[47] | \(0.0 \pm 0.00\) | \(0.0 \pm 0.00\) | \(0.0 \pm 0.00\) |
| Diffusion-Policy[50] | \(40.0 \pm 0.00\) | \(80.0 \pm 0.00\) | \(88.0 \pm 0.00\) |
| Ours | \(\mathbf{100.0 \pm 0.00}\) | \(\mathbf{100.0 \pm 0.00}\) | \(\mathbf{100.0 \pm 0.00}\) |
0pt
4.3 Ablation and Robustness Analysis↩︎
To quantify the contribution of our core components and evaluate the framework’s stability in unstructured environments, we conduct rigorous ablations and robustness tests. All variants are evaluated across ten manipulation tasks, averaged 30 trials per task. We report the ablation success rates in 4. Additionally, to provide a transparent understanding of the system’s operational boundaries, a comprehensive analysis of unsuccessful trials is detailed in the Appendix.
| Variant | Success (%) |
|---|---|
| No Kinematic Knowledge Retrieval | 65.0 |
| Replace visual foundation models | 68.5 |
| Anchor-Free (Track \(\mathbf{T}_P^*\) only) | 48.5 |
| No Dynamic Manifold Replanning | 35.0 |
| Ours (Full) | 74.4 |
0pt
Effect of Kinematic Knowledge Retrieval. In this variant, we disable the kinematic knowledge base, forcing the LLM to infer the interaction strategy purely from raw textual prompts without external physical priors. As shown in 4, this leads to a noticeable performance drop. Without the structural prior, the LLM frequently fails to construct a valid Relational Graph. It either outputs isolated parts without identifying the necessary physical anchor (, finding the laptop screen but ignoring the base), or hallucinates physically impossible kinematic edges (, predicting a "pull" edge for a revolute hinge).
Precision of Metric Visual Grounding. We evaluate the impact of segmentation quality by replacing SAM3 [15] with other detectors, including Grounding DINO[51] and T-Rex2[52]. As shown in 4, other detections lead to reduced success rates, particularly for thin articulated components (e.g., handles) whose boxes overlap with surrounding structures. Such imprecise initialization degrades the estimation of \(\mathbf{T}_P^*\) and \(\mathbf{T}_A^*\) in FoundationPose [16]. Nevertheless, the overall performance does not collapse, indicating that visual grounding is not the sole determinant of success. The relational \(SE(3)\) constraints compensate for moderate perception errors, demonstrating that accurate segmentation improves performance but is not the primary source of robustness in our system.
Necessity of the Relational Anchor Constraint. To validate our core hypothesis that tracking both the primary part \(\mathbf{T}_P^*\) and the anchor part \(\mathbf{T}_A^*\) is necessary, we evaluate an Anchor-Free variant. The robot only extracts and tracks the active part (, the handle) and attempts to manipulate it using a pre-defined vector. Without the anchor frame to dynamically establish the screw axis \(\boldsymbol{\xi}\), the generated trajectories deviate from the true joint kinematics (, pulling a revolute door in a straight line), triggering safety stops or causing slippage. This confirms that the relational spatial graph \(\Psi\) is indispensable for generating rigorous \(SE(3)\) waypoints.
Impact of Dynamic Manifold Replanning. To assess robustness against open-world dynamics, we disable the closed-loop tracking mechanism. The robot plans the constraint manifold once at \(t=0\) and executes the trajectory blind. Under external disturbances (, object is shifted or rotated during operation), the open-loop trajectory completely fails because the physical boundary conditions have changed. The significant drop in success rate confirms that continuous tracking of \(\hat{\mathbf{T}}_A(t)\) and dynamic recalculation of the manifold \(\tau_{t:N}^*\) are critical for maintaining physical consistency in unconstrained environments.
| Task | Clean | Distractors | Occlusion (30%) |
|---|---|---|---|
| Open Microwave (Revolute) | 78 | 75 | 72 |
| Pull Drawer (Prismatic) | 82 | 79 | 74 |
| Stack Cube (Rigid) | 100 | 100 | 85 |
0pt
Robustness under Visual Corruptions. Building upon the necessity of the previous parts, we further stress-test the full system against out-of-distribution visual clutter. End-to-end policies often suffer from catastrophic failure when faced with environmental noise. To demonstrate our advantage, we evaluate three representative tasks under the presence of physical distractors (random objects in the workspace) and partial occlusions (30% coverage) of the target parts. As summarized in 5, RelAfford6D experiences only graceful degradation under severe occlusion. As long as the perception module can partially recover the geometry to establish the spatial graph \(\Psi\), the downstream kinematic manifold tracking remains mathematically valid. This decoupled design provides a far more robust execution anchor than global visual embeddings.
| Tasks | |
|
|
|
 |
|
|
|---|---|---|---|---|---|---|---|
| FlowBot3D[40] | 3/10 | 3/10 | 4/10 | 2/10 | - | 5/10 | 2/10 |
| ManipLLM[6] | 6/10 | 7/10 | 6/10 | 7/10 | - | 7/10 | 5/10 |
| Ours | 9/10 | 8/10 | 8/10 | 8/10 | 10/10 | 8/10 | 9/10 |
0pt
4.4 Real-world Evaluation↩︎
To validate the effectiveness and robustness of our training-free framework in unconstrained physical environments, we deploy RelAfford6D on a Realman RM75 robotic arm equipped with a wrist-mounted Intel RealSense D435 RGB-D camera (shown in 6). We evaluate seven representative manipulation categories corresponding to the simulated tasks. A trial is considered successful if the intended physical state (, rotation angle for a revolute constraint, or displacement for a prismatic constraint) is achieved within a physical tolerance of 5 cm or \(5^\circ\). 6 summarizes the success rates. Despite operating under real-world sensor noise and uncalibrated visual backgrounds, our system maintains remarkably strong performance. It demonstrates robust zero-shot generalization from simulation to the physical world, proving that our mathematically rigorous manifold tracking is highly resilient to real-world domain shifts. Furthermore, by decoupling the computationally expensive semantic topology generation from the execution phase, the system spends about 20.7 s on one-time initialization while closed-loop tracking operates at 5 Hz, replanning costs less than 0.05 s, and low-level control costs less than 0.01 s. The detailed wall-clock latency breakdown is provided in the Appendix.
5 Conclusion↩︎
We present RelAfford6D, a novel, training-free framework that bridges the persistent perception-action gap in open-world robotic manipulation. Diverging from existing end-to-end or data-driven approaches, our core contribution is the formulation of a Relational 6D Affordance Graph. This representation serves as a physically grounded interface, cohesively linking the semantic topology generated by Large Language Models with the precise metric \(SE(3)\) visual grounding of vision foundation models. By explicitly extracting both the interacting part and its physical anchor, RelAfford6D formulates downstream execution as a continuous, closed-loop kinematic constraint satisfaction problem. Extensive experiments demonstrate that tracking strictly defined physical manifolds yields superior zero-shot success rates and open-world robustness across diverse articulated and rigid objects. Current experiments focus on revolute and prismatic joints; helical and Multi-DoF mechanisms can be represented as general screw motions or compositions of constraints, while deformable objects remain outside our rigid \(SE(3)\) formulation. While our structural constraints successfully mitigate moderate visual noise, severe object occlusions or hard perception failures (, incorrect initial part segmentation) can still perturb the topological extraction. Addressing these extreme visual corruptions and deformable manipulation through temporal 3D tracking, multi-view semantic fusion, or non-rigid relational modeling remains a promising avenue for future exploration.
Acknowledgements↩︎
This paper is supported by the Fundamental Research Funds for the Central Universities (project number YG2024ZD06) and NSFC (62176155).