FORGE: Towards Functional Tool-Use Generalization via Keypoint Trajectory Reasoning
July 07, 2026
1 Introduction↩︎
In open-ended real-world environments, the right tool is rarely at hand, yet humans adapt effortlessly, repurposing a book, a stone, or a shoe to drive a nail, because functionally equivalent tools share a common intent: a contact region to strike with and a motion to bring it onto the target. This capacity, which we term functional generalization, is a hallmark of human dexterity that robots have yet to achieve. Current manipulation policies overfit to the appearance and geometry of seen tools, failing entirely when handed a novel one that serves the same function [1]–[3].
Functional generalization is fundamentally harder than conventional generalization. As illustrated in Fig. 1, scene- and category-level generalization only require tolerating visual variations while the underlying motion stays the same [4]–[7]. Functional generalization, by contrast, demands that the motion itself change: striking a target with a novel tool requires locating its contact region, aligning it, and producing an appropriate motion, even when the tool’s shape and trajectory differ entirely from training. The core difficulty is a fundamental mismatch: functionally equivalent tools share recognizable structure in visual space, but this similarity does not transfer to action space, where each tool demands an entirely different motor pattern.
Bridging this perception-to-action gap requires an intermediate representation that carries functional intent across tools, satisfying two competing demands: expressive enough to capture where to make contact and how to move onto the target, yet grounded enough to be predicted from action-free observations without overfitting to tool-specific appearance. Dense representations like raw video entangle function-relevant cues with tool-specific appearance, while overly sparse ones discard the geometric and temporal structure needed for action. This trade-off raises the central question of our work: which representation best transfers functional intent from perception to action?
To answer this question, we systematically evaluate intermediate representations including affordance images, human video prompts, and 2D keypoint trajectories, finding that keypoint trajectories best balance functional expressiveness and action groundability: they capture function-relevant contact and motion structure while discarding tool-specific appearance that causes overfitting. Building on this, we propose FunctiOnal Reasoning and Grounded Execution (FORGE), a two-stage policy that decouples functional reasoning from action execution: it first predicts generalizable keypoint trajectories for unseen tools from large-scale action-free data, then grounds these functional plans into executable robot actions using a small set of action-labeled demonstrations. We further introduce a seven-tool hitting-function benchmark, where FORGE consistently outperforms state-of-the-art methods on unseen tools in both simulation and the real world, achieving over \(2\times\) improvement in average success rate. In summary, our contributions are threefold:
We formalize functional generalization in robotic tool-use and identify the perception-to-action gap as its core difficulty.
We find that 2D keypoint trajectories best balance functional expressiveness and action groundability, and propose FORGE, a two-stage policy that predicts generalizable keypoint trajectories from action-free data and grounds them into robot actions with limited demonstrations.
Through extensive simulation experiments across seven tools and real-world validation, we demonstrate that FORGE consistently outperforms state-of-the-art methods on unseen tools, achieving over \(2\times\) improvement in average success rate.
2 Related Works↩︎
2.0.0.1 Generalization Problems in Robotic Manipulation.
Generalization has been a central problem in robotic manipulation, and existing studies mainly evaluate it from three perspectives: scene-level, category-level, and cross-embodiment generalization. Scene-level generalization focuses on whether a policy can remain robust when the surrounding visual or physical contexts vary, such as changes in object layouts, lighting conditions, or camera viewpoints [4]–[7]. Furthermore, category-level generalization studies whether robots can transfer manipulation skills to unseen objects that belong to the same semantic category while differing in geometry, size, or texture [8]–[10]. More recently, cross-embodiment generalization has attracted increasing attention, which aims to train a unified policy that can transfer across different robot platforms, such as robotic arms and humanoids [11]–[13]. In this work, we go beyond generalization over visual appearance and robot embodiment and focus on functional generalization, in which an agent is required to accomplish the same function using diverse, unseen tools.
2.0.0.2 Improving Generalization in Robot Policy Learning.
To improve manipulation generalization, the most straightforward way is scaling up action-labeled robotic data and training policies end-to-end [11], [12], [14], [15]. However, collecting large-scale robotic data is time-consuming and labor-intensive, especially for complicated tool-use manipulations. Reducing the dependence on action labels, some works learn transferable motion priors from large-scale action-free video data [7], [16]–[21]. For example, MimicPlay [16] learns high-level latent plans from human play videos and grounds them into robot actions with a low-level visuomotor policy. Although videos contain rich motion and interaction cues, dense future frames do not explicitly reveal a compact abstraction of function-relevant intent. As a result, effectively learning functional priors from videos often requires extremely extensive pretraining. Another line of work improves action grounding with more structured intermediate representations, such as interaction heatmaps [9], [22]–[24], object-centric representations [1], [25], [26], or keypoint trajectories [2], [8], [27], [28]. For instance, ATM [28] predicts future trajectories of arbitrary points from video and uses them to guide a low-level policy for action generation. However, these methods mainly focus on standard manipulation tasks, such as object pick-up, while overlooking tool-use scenarios that require capturing more abstract functional intent, which are systematically investigated in our work.
3 Methods↩︎
3.1 Problem Formulation↩︎
We formulate functional generalization in tool-use manipulation as a Markov Decision Process (MDP), where the state \(\mathbf{s}_t\) consists of the visual observation \(\mathbf{o}_t\), the proprioceptive state \(\mathbf{q}_t\), and optionally a functional intermediate representation \(\mathbf{x}_t\) (e.g., affordance images, human video prompts, or keypoint trajectories) that carries function-relevant cues bridging perception and action, and the action \(\mathbf{a}_t\) is the motor command. A task function (e.g., hitting a target) is defined over seen tools \(\mathcal{T}_{\text{seen}}\) for training and disjoint unseen tools \(\mathcal{T}_{\text{unseen}}\) for evaluation, where the function stays fixed but the tool, its appearance, and the required contact positions and motions all change.
Two types of data are available for learning. The first is a large-scale action-free dataset \(\mathcal{D}_{U}=\{(\mathbf{O}^{(i)}, \mathbf{X}^{(i)})\}_{i=1}^{N_U}\), consisting of visual observations \(\mathbf{O}^{(i)}=\{\mathbf{o}^{(i)}_t\}_{t=1}^{T}\) and corresponding functional intermediate representations \(\mathbf{X}^{(i)}=\{\mathbf{x}^{(i)}_t\}_{t=1}^{T}\), which is abundant since it requires no robot labels. The second is a smaller action-labeled robotic dataset \(\mathcal{D}_{L}=\{(\mathbf{O}^{(i)}, \mathbf{X}^{(i)}, \mathbf{Q}^{(i)}, \mathbf{A}^{(i)})\}_{i=1}^{N_L}\), which additionally provides proprioceptive states \(\mathbf{Q}^{(i)}\) and ground-truth actions \(\mathbf{A}^{(i)}\) but is costly to collect, so \(|\mathcal{D}_L| \ll |\mathcal{D}_U|\). This data asymmetry, together with the perception-to-action gap discussed in Sec. 1, motivates a two-stage decomposition: \[\underbrace{ \pi_{\mathrm{FORGE}}( \mathbf{a}_{t:t+H} \mid \mathbf{o}_{t}, \mathbf{q}_{t}, \mathbf{x}_{t} ) }_{\text{FORGE}}= \underbrace{ \pi_{\mathrm{sys2}}( \hat{\mathbf{X}}_{t:t+H} \mid \mathbf{o}_{t}, \mathbf{x}_{t} ) }_{\text{functional reasoning}} \underbrace{ \pi_{\mathrm{sys1}}( \mathbf{a}_{t:t+H} \mid \mathbf{o}_{t}, \mathbf{q}_{t}, \hat{\mathbf{X}}_{t:t+H} ) }_{\text{grounded execution}} . \label{eq:forge95decomposition}\tag{1}\] FORGE learns a future predictor on \(\mathcal{D}_U\) for forecasting functional intermediate representations from observations, and an execution policy on \(\mathcal{D}_L\) for grounding predicted representations into actions.
3.2 Functional Intermediate Representations Selection↩︎
As discussed in Sec. 1, a desired functional intermediate representation \(\mathbf{X}_{t:t+H}\) should preserve the function-relevant structure shared across tools, while remaining predictable from action-free observations and groundable into precise robot actions. We hypothesize that 2D keypoint trajectories form a suitable choice of \(\mathbf{X}_{t:t+H}\). They discard tool-specific appearance that may cause overfitting, while retaining compact geometric information such as the hitting point, target position, and affordance-relevant tool structure. At the same time, they provide a structured temporal description of how function-relevant points should move, which makes them easy to predict from existing vision models [28] and to ground into actions. To validate this hypothesis, we compare keypoint trajectories with two alternative choices of \(\mathbf{X}_{t:t+H}\): affordance images and human video prompts.
Affordance Image. Given an affordance image \(I_{\mathrm{aff}}^{(i)}\), we encode it with an image encoder \(E_{\mathrm{aff}}\) [29] and use the resulting feature as a time-invariant condition: \(x_t^{(i)}=E_{\mathrm{aff}}(I_{\mathrm{aff}}^{(i)})\in\mathbb{R}^{d}\).
Human Video Prompt. Given a human demonstration video \(V_{\mathrm{hum}}^{(i)}=\{I_{\mathrm{hum},t}^{(i)}\}_{t=1}^{T}\), we encode it with a video encoder \(E_{\mathrm{vid}}\) [30] and apply temporal average pooling to obtain a sequence-level feature, which is used as a time-invariant condition: \(x_t^{(i)}=\mathrm{AvgPool}(E_{\mathrm{vid}}(V_{\mathrm{hum}}^{(i)}))\in\mathbb{R}^{d}\).
Keypoint Trajectory. We denote the keypoint trajectory as \(P^{(i)}=\{p_t^{(i)}\}_{t=1}^{T}\), where each frame \(p_t^{(i)}=\{p_t^{(i),k}\}_{k=1}^{K}\) contains \(K\) 2D keypoints. At time step \(t\), we flatten the keypoint coordinates and encode them with a keypoint encoder \(E_{\mathrm{kp}}\), yielding \(x_t^{(i)}=E_{\mathrm{kp}}(\mathrm{Flatten}(p_t^{(i)}))\in\mathbb{R}^{d}\).
To evaluate these candidates, we design a heuristic experiment in Fig. 2 (a), where each representation is pre-extracted for both seen and unseen tools and directly used as the condition for the execution policy \(\pi_{\mathrm{sys1}}\). The policy is trained on action-labeled data from seen tools in \(\mathcal{D}_{L}\) and evaluated on unseen tools, where success is determined in pixel space by checking whether the predicted hitting point overlaps with the target point within a tolerance threshold. Among the three candidates, 2D keypoint trajectories achieve the best performance, since they go beyond static region cues and dense visual motion by providing a compact, structured description of how function-relevant points move and align over time. We therefore adopt 2D keypoint trajectories as the functional intermediate representation for FORGE.
3.3 Functional Reasoning and Grounded Execution Policy↩︎
After selecting 2D keypoint trajectories as the functional intermediate representation for FORGE, we design a two-stage training strategy to decouple functional reasoning from action execution. This enables FORGE to learn transferable functional keypoint motion from action-free demonstrations, while grounding the predicted trajectories into robot actions with limited action-labeled data.
Stage 1: Action-free Functional Reasoning. As shown in Fig. 2 (b), we train the keypoint motion predictor \(\pi_{\mathrm{sys2}}\) on the action-free dataset \(\mathcal{D}_{U}\) to predict future functional keypoint trajectories. Given the visual observation \(\mathbf{o}_t\) and keypoint representation \(\mathbf{x}_t\) at time-step \(t\), \(\pi_{\mathrm{sys2}}\) generates a future keypoint trajectory \(\hat{\mathbf{X}}_{t:t+H}\) over horizon \(H\).
Formally, we instantiate \(\pi_{\mathrm{sys2}}\) as a conditional flow-matching model in the keypoint space. Given a ground-truth keypoint trajectory \(\mathbf{X}_{t:t+H}\), a Gaussian noise sample \(\epsilon\sim\mathcal{N}(0,\mathbf{I})\), and an interpolation time \(\tau\sim\mathcal{U}(0,1)\), we define the interpolated keypoint trajectory as \(\mathbf{X}_{t:t+H}^{\tau}=\tau \mathbf{X}_{t:t+H} + (1-\tau)\epsilon\). The keypoints motion predictor learns a conditional velocity field \(u_{\phi}\) by minimizing \[\mathcal{L}_{\mathrm{sys2}} = \mathbb{E}_{\tau,\epsilon,\mathcal{D}_{U}} \left[ \left\| u_{\phi} \left( \mathbf{X}_{t:t+H}^{\tau}, \tau \mid \mathbf{o}_t, \mathbf{x}_t \right) - \left( \mathbf{X}_{t:t+H}-\epsilon \right) \right\|_2^2 \right]. \label{eq:sys295flow95matching}\tag{2}\] During training, our task specifies the hitting point on the tool and the target point on the object. Given the hitting point, we use SAM2 [31] to segment the corresponding tool mask and sample \(N\) keypoints on the mask with Farthest Point Sampling (FPS). As shown in Fig. 2 (b), we then track these keypoints with CoTracker [32] to obtain keypoint trajectories for training. Since this stage only requires visual observations and keypoint trajectories, System-2 can leverage action-free data to capture transferable functional intent as keypoint-based motions, without relying on large-scale robot action labels.
Stage 2: Grounded Execution Policy. In the second stage, we freeze the pretrained keypoint motion predictor and train the execution policy \(\pi_{\mathrm{sys1}}\) on the action-labeled robotic dataset \(\mathcal{D}_{L}\). Given the observation \(\mathbf{o}_t\), robot state \(\mathbf{q}_t\), and 2D keypoint pixels \(\mathbf{x}_t\), the frozen \(\pi_{\mathrm{sys2}}\) first predicts future keypoint motions: \[\hat{\mathbf{X}}_{t:t+H} = \pi_{\mathrm{sys2}}(\mathbf{o}_t, \mathbf{x}_t).\] To improve robustness to prediction errors, we apply random pixel perturbations to the \(\hat{\mathbf{X}}_{t:t+H}\) during training, where each keypoint is shifted by 1–5 pixels with a probability of 0.5. The execution policy \(\pi_{\mathrm{sys1}}\) then grounds the perturbed keypoint trajectories \(\tilde{\mathbf{X}}_{t:t+H}\) into executable robot actions: \[\hat{\mathbf{A}}_{t:t+H} = \pi_{\mathrm{sys1}}(\mathbf{o}_t, \mathbf{q}_t, \tilde{\mathbf{X}}_{t:t+H}).\]
Similar to \(\pi_{\mathrm{sys2}}\), we instantiate \(\pi_{\mathrm{sys1}}\) as a conditional flow-matching model in the action space. Given a ground-truth action chunk \(\mathbf{A}_{t:t+H}\), a Gaussian noise sample \(\epsilon\sim\mathcal{N}(0,\mathbf{I})\), and an interpolation time \(\tau\sim\mathcal{U}(0,1)\), we define the interpolated action trajectory as \(\mathbf{A}_{t:t+H}^{\tau} = \tau \mathbf{A}_{t:t+H} + (1-\tau)\epsilon\). The action-grounded execution policy learns a conditional velocity field \(v_{\theta}\) by minimizing: \[\mathcal{L}_{\mathrm{sys1}} = \mathbb{E}_{\tau,\epsilon,\mathcal{D}_{L}} \left[ \left\| v_{\theta} \left( \mathbf{A}_{t:t+H}^{\tau}, \tau \mid \mathbf{o}_t, \mathbf{q}_t, \mathbf{\tilde{X}}_{t:t+H} \right) - \left( \mathbf{A}_{t:t+H}-\epsilon \right) \right\|_2^2 \right]. \label{eq:sys195flow95matching}\tag{3}\]
4 Experiments↩︎
Functional generalization remains a largely underexplored problem in robotic manipulation. In this work, we instantiate it with the hitting function and construct both simulation and real-world benchmarks to systematically evaluate whether a policy can transfer the same function to unseen tools. Specifically, we design experiments to test four hypotheses that together validate the central claims of this work:
H1: Functional generalization requires an intermediate representation, not end-to-end mapping.
H2: Among candidate intermediate representations, 2D keypoint trajectories best balance affordance expressiveness and action groundability.
H3: Keypoint trajectories must be function-aware; generic keypoint tracking is not enough.
H4: The two-stage design generalizes under moderate action-labeled tool diversity, making it a practical recipe for tool-use learning.
4.1 Experimental setup↩︎
Simulation Benchmark. As shown in Fig. 3, our simulation benchmark contains 7 tools, each with 3 different initial settings. For each tool-setting pair, we collect 30 demonstrations using a four-stage Model Predictive Path Integral (MPPI) planner, resulting in 630 demonstrations in total. The MPPI planner generates an action trajectory by optimizing stage-dependent objectives that sequentially encourage the robot to lift the tool, move it toward the target, align the tool-specific hitting point with the target cube, and execute the final strike. In each demonstration, we randomly initialize the target cube position and sample the tool-specific hitting point along the tool boundary, requiring the policy to adapt its motion to both the tool geometry and the target location. In the main setting, all methods are trained on 4 tools, including hammer, tobject, pickaxe, and mustard, and evaluated on the remaining 3 unseen tools. During evaluation, each unseen tool is tested under three initial settings with 30 rollout episodes. A rollout is considered successful only if the robot strikes the target cube using the designated hitting point.
Real-World Setting. We further construct a real-world benchmark on a Franka robot platform, as shown in Fig. 3. The real-world setting contains 3 tools: hammer, mustard, and book. Each tool has two hitting patterns, and each pattern contains 20 expert demonstrations, resulting in 120 real-world trajectories. We train the policy on hammer and mustard and evaluate its functional generalization ability on the unseen book tool. For each hitting pattern, we perform 20 real-world rollout trials. A trial is successful if the robot follows the specified hitting pattern and strikes the target.
Baselines. We compare FORGE against three baselines covering end-to-end and keypoint-based policies. Flow-Matching (FM) [33] and Diffusion Policy (DP) [34] are end-to-end visuomotor policies that map observations directly to actions, serving as references for the perception-to-action gap (Sec. 1). ATM [28] also conditions on keypoint trajectories, but obtains them from a generic tracker pretrained on LIBERO-90 rather than a function-aware predictor, allowing us to test whether the gain comes from using keypoints or from predicting function-aware keypoint motion. All baselines receive the same inputs as FORGE: visual observations, proprioceptive states, and the initial hitting and target point positions.
Evaluation Metrics. All experiments report the success rate (SR) as the main evaluation metric, which is defined as the number of successful trials divided by the total number of rollout trials.
| Method | Scoop | Shoe | Book | Overall | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2-5 (lr)6-9 (lr)10-13 | S01 | S02 | S03 | Avg. | S01 | S02 | S03 | Avg. | S01 | S02 | S03 | Avg. | |
| FM [33] | 0.00 | 0.03 | 0.17 | 0.07 | 0.00 | 0.07 | 0.03 | 0.03 | 0.13 | 0.17 | 0.17 | 0.16 | 0.09 |
| DP [34] | 0.13 | 0.13 | 0.27 | 0.18 | 0.00 | 0.13 | 0.10 | 0.08 | 0.33 | 0.23 | 0.23 | 0.26 | 0.17 |
| ATM [28] | 0.13 | 0.17 | 0.07 | 0.12 | 0.10 | 0.00 | 0.07 | 0.06 | 0.07 | 0.07 | 0.03 | 0.06 | 0.08 |
| (Ours) | 0.30 | 0.30 | 0.67 | 0.42 | 0.23 | 0.23 | 0.23 | 0.23 | 0.53 | 0.20 | 0.53 | 0.42 | 0.36 |
4.2 Main Results & Analysis↩︎
Simulation Results. As shown in Tab. 1, FM and DP achieve only \(0.09\) and \(0.17\) average SR on unseen tools, validating H1: without an intermediate representation exposing the contact region and its alignment with the target, end-to-end policies fall back on the motor patterns of the most visually similar seen tool. ATM also underperforms (\(0.08\)), validating H3: its generic tracker cannot anticipate function-relevant motion, so the keypoint trajectories it provides fail to describe how the contact region should approach the target. In contrast, FORGE achieves \(0.36\), outperforming DP by over \(2\times\) with consistent gains across all three unseen tools, and Fig. 4 (a) shows that its predicted keypoint trajectories explicitly bring the tool-specific contact region toward the target before impact. We further observe two failure modes (Fig. 4 (b)): imprecise hitting, where the 2D trajectories capture the overall motion but lack fine-grained spatial alignment, motivating 3D-aware functional representations; and tool dropping, where the execution policy produces jerky actions, motivating smoothness regularization or trajectory post-processing.
Real-World Results. We further evaluate whether FORGE can transfer the learned hitting function to the real world by comparing it with the FM base policy on the unseen book tool. As shown in Fig. 5, FORGE consistently outperforms FM under both hitting patterns. This further supports H1, showing that direct perception-to-action mapping is insufficient for functional generalization. As illustrated by the failure cases in Fig. 5, FM often fails to align the correct hitting region of the unseen book tool with the target, leading to missed hits. In contrast, FORGE predicts function-relevant keypoint trajectories that specify how the desired hitting region should approach and contact the target. These intermediate plans provide explicit guidance for the grounded execution policy, enabling more accurate alignment and successful real-world hitting.
4.3 Ablation Studies↩︎
Beyond comparing against baselines, we conduct two ablations to dissect the design choices behind FORGE. The first ablation varies the functional intermediate representation, conditioning the execution policy on affordance images, human video prompts, or 2D keypoint trajectories, to identify which representation best balances affordance expressiveness and action groundability. The second ablation varies the diversity of action-labeled tools, training FORGE on 1, 4, or 6 tools and testing on the unseen tools, to evaluate whether the two-stage pipeline provides a practical recipe for functional generalization using moderate robotic data.
4.3.1 Effect of Functional Intermediate Representations.↩︎
| Representation | Scoop | Shoe | Book | Avg. |
|---|---|---|---|---|
| Flow-Matching | 0.07 | 0.03 | 0.16 | 0.09 |
| + Aff. Images | 0.34 | 0.11 | 0.16 | 0.20 |
| + Hum. Vid. Prompts | 0.26 | 0.09 | 0.23 | 0.19 |
| + Key. Trajectories | 0.50 | 0.31 | 0.67 | 0.49 |
r0.55
As discussed in Sec. 3.2, we investigate the effect of different functional intermediate representations by directly training the execution policy \(\pi_{\mathrm{sys1}}\). Results in Tab. 2 show that introducing affordance images or human video prompts improves generalization, suggesting that tool-region cues and visual motion demonstrations of the hitting provide useful functional information. In comparison, 2D keypoint trajectories capture fine-grained hitting and target point information while explicitly describing structured functional motion over time. These results support H2: keypoint trajectories provide a more compact and action-groundable condition, leading to the best functional generalization.
| Method | Tools | |||||||
|---|---|---|---|---|---|---|---|---|
| 2-9 | Hammer | Tobject | Pickaxe | Mustard | Scoop | Shoe | Book | Avg. |
| (1-to-6) | – | 0.26 | 0.34 | 0.17 | 0.14 | 0.10 | 0.23 | 0.21 |
| (4-to-3) | – | – | – | – | 0.42 | 0.23 | 0.42 | 0.36 |
| (6-to-1) | – | – | – | – | – | – | 0.52 | 0.52 |
4.3.2 Effect of Action-Labeled Tool Diversity.↩︎
Beyond the standard 4-to-3 setting, we evaluate FORGE fine-tuned with robotic data from 1 or 6 tools and tested on the unseen ones. As shown in Tab. 3, the success rate grows monotonically with action-labeled tool diversity, from \(0.21\) (1-to-6) to \(0.36\) (4-to-3) to \(0.52\) (6-to-1), confirming that broader data coverage improves action grounding. This reveals a practical trade-off: too little robotic data limits grounding, while collecting demonstrations for many tools is costly. The non-trivial 4-to-3 performance supports H4: the two-stage design of FORGE provides a practical recipe, where scalable action-free data supports cross-tool functional reasoning and moderate action-labeled data suffices to ground predicted trajectories into executable actions.
5 Conclusion and Limitation↩︎
In this paper, we identify functional generalization as an underexplored problem, where an agent needs to accomplish the same function with unseen tools. To address this problem, we propose FORGE, a policy that decouples functional reasoning from action execution via an intermediate representation. In this way, FORGE learns functional intent from action-free data to predict keypoint-based plans, and then grounds them into robot actions by fine-tuning on limited action-labeled demonstrations. We introduce a seven-tool hitting-function benchmark to evaluate functional generalization in simulation and real-world settings. Extensive experiments show that FORGE improves generalization to unseen tools over state-of-the-art methods.
5.0.0.1 Limitations and Future Work.
First, the functional-relevant hitting and target points are specified in this study, while future work may leverage vision foundation models [35] or pretrain a keypoint proposal model to automatically identify function-relevant keypoints. Second, our pipeline does not consider the grasping process. We will extend FORGE to predict grasp styles from keypoint-based functional trajectories and study more dedicated end-effectors such as dexterous hands.
Appendix↩︎
6 Hitting Benchmark Construction and Implementation Details↩︎
This section provides additional details on the construction of our hitting benchmarks. We first describe the extraction of different functional intermediate representations in Appx. 6.1, then introduce MPPI-based action-label generation in Appx. 6.2. Finally, we summarize the training and implementation details of FORGE and all baselines in Appx. 6.3.
| Representation | # Tools | # Settings | # Initial Poses | # Samples / Pose | # Total |
|---|---|---|---|---|---|
| Affordance Image | 7 | 1 | 1 | 1 | 7 |
| Human Video Prompts | 7 | 3 | 5 | 5 | 525 |
| Keypoint Trajectories (Simulation) | 7 | 3 | 30 | 1 | 630 |
| Keypoint Trajectories (Real World) | 6 | 2 | 1 | 20 | 240 |
6.1 Extraction of Functional Intermediate Representations↩︎
As discussed in Sec. 3.2, we consider three types of functional intermediate representations: affordance images, human video prompts, and 2D keypoint trajectories. Specifically, we describe how each representation is extracted and incorporated as an additional condition for the grounded execution policy \(\pi_{\mathrm{sys1}}\). The statistics of these representations are summarized in Tab. 4.
| Stage | Objective | Main Reward Terms |
|---|---|---|
| Stage 1 | Lift the tool and align it with the target along the \(x\) and \(y\) axes. | \(\begin{aligned} r_{\mathrm{lift}} &= 18.0\min(z_{\mathrm{lift}}, z_1), \\ r_{\mathrm{lift\_prog}} &= 100.0\min(\Delta z_{\mathrm{lift}}, 0.02), \\ r_x &= 22.0\exp(-18.0 d_x), \\ r_{x\_\mathrm{prog}} &= 120.0\min(\Delta d_x, 0.02), \\ r_y &= 8.0\exp(-8.0 d_y), \\ r_{y\_\mathrm{prog}} &= 35.0\min(\Delta d_y, 0.02). \end{aligned}\) |
| Stage 2 | Move the tool toward the target while maintaining a suitable hovering height. | \(\begin{aligned} r_{xy} &= 14.0\exp(-10.0 d_{xy}), \\ r_x &= 12.0\exp(-18.0 d_x), \\ r_y &= 8.0\exp(-10.0 d_y), \\ r_z &= 7.0\exp(-18.0 e_z), \\ r_{xy\_\mathrm{prog}} &= 95.0\min(\Delta d_{xy}, 0.02), \\ r_{x\_\mathrm{prog}} &= 80.0\min(\Delta d_x, 0.02), \\ r_{y\_\mathrm{prog}} &= 40.0\min(\Delta d_y, 0.02). \end{aligned}\) |
| Stage 3 | Finely align the tool-specific hitting point above the target. | \(\begin{aligned} r_{xy} &= 14.0\exp(-14.0 d_{xy}), \\ r_x &= 14.0\exp(-24.0 d_x), \\ r_y &= 10.0\exp(-18.0 d_y), \\ r_{xy\_\mathrm{prog}} &= 145.0\min(\Delta d_{xy}, 0.018), \\ r_{x\_\mathrm{prog}} &= 135.0\min(\Delta d_x, 0.018), \\ r_{y\_\mathrm{prog}} &= 110.0\min(\Delta d_y, 0.018), \\ r_z &= 7.0\exp(-18.0 e_z). \end{aligned}\) |
| Stage 4 | Keep the hitting point close to the target and execute the downward strike. | \(\begin{aligned} r_{xy} &= 12.0\exp(-16.0 d_{xy}), \\ r_x &= 14.0\exp(-28.0 d_x), \\ r_y &= 10.0\exp(-18.0 d_y), \\ r_{xy\_\mathrm{prog}} &= 120.0\min(\Delta d_{xy}, 0.015), \\ r_{x\_\mathrm{prog}} &= 100.0\min(\Delta d_x, 0.015), \\ r_{y\_\mathrm{prog}} &= 80.0\min(\Delta d_y, 0.015), \\ r_{\mathrm{hit}} &= 150.0\min(-\Delta z_{\mathrm{head}}, 0.05), \end{aligned}\) |
| Contact / Success | Encourage valid downward contact with the target. | \(\begin{aligned} r_{\mathrm{contact}} &= 150.0 \;\text{if valid downward contact}. \end{aligned}\) |
Affordance Images. We manually annotate the grasp point and hitting region for each tool, producing 7 affordance images. When evaluating functional intermediate representations in Sec. 3.2, we use the affordance image of the corresponding tool as an additional condition for the grounded execution policy \(\pi_{\mathrm{sys1}}\). Since this representation is time-invariant, the same image feature is shared across all time steps of a trajectory.
Human Video Prompts. As shown in Fig. 6, we collect human video prompts for all 7 tools. Following the simulation benchmark, each tool has 3 settings. For each setting, we define 5 initial positions and collect 5 videos per position, resulting in 525 human tool-use videos. When evaluating functional intermediate representations, we use the video prompt corresponding to the tool, setting, and sample from Position 01 as an additional condition for the grounded execution policy. These videos provide visual demonstrations of how humans accomplish the same hitting function with different tools. We leave further exploitation of human video prompts, such as learning functional generalization from human tool-use demonstrations, to future work.
Keypoint Trajectories. We adopt 2D keypoint trajectories as the final functional intermediate representation and extract them for both simulation and real-world data. In simulation, the hitting and target points are directly obtained from the simulator. Given the initial hitting point on the tool, we use SAM2 [31] to segment the corresponding tool mask and sample \(N=5\) additional tool keypoints with Farthest Point Sampling (FPS), ensuring spatial coverage of the tool geometry. We then track these sampled tool keypoints with CoTracker [32] to obtain their 2D trajectories. Our simulation benchmark contains 7 tools with 3 settings each. For each setting, we randomly initialize 30 hitting and target positions, resulting in 630 simulation samples. For real-world data, we manually annotate the target and hitting points for each sample and obtain five additional tool keypoints using the same SAM2- and FPS-based procedure. Unlike simulation, where the hitting and target trajectories are available from the simulator, all 7 real-world keypoints are tracked by CoTracker. The real-world benchmark contains 6 tools, each with 2 hitting patterns and 1 initial position. We collect 20 samples for each initial position, resulting in 240 real-world samples.
6.2 MPPI-based Demonstration Generation↩︎
In the simulation benchmark, the action-labeled demonstrations are automatically generated rather than collected through teleoperation. We use a Model Predictive Path Integral (MPPI) planner with a four-stage rollout procedure, whose main reward terms are summarized in Tab. 5. The stages sequentially encourage the robot to lift the tool, move it toward the target, align the tool-specific hitting point with the target, and execute the final downward strike.
Besides the translational commands optimized by MPPI, we specify a target rotational pose at the end of each stage and generate the corresponding rotational commands through servo control. This helps the tool maintain an appropriate orientation during lifting, alignment, and hitting. Together, the stage-specific MPPI rewards and rotational servo control enable reliable automatic demonstration generation in simulation.
6.3 Training and Implementation Details↩︎
All models are trained on NVIDIA RTX 6000 Pro GPUs. For all methods, we use an action chunk size of 8, a history horizon of 8, and a batch size of 64. We build the simulation benchmark on Robomimic and conduct real-world experiments on a Franka robot platform.
For baseline methods, we train each policy for 120,000 steps using the corresponding action-labeled data. For FORGE, we follow the two-stage training procedure described in Sec. 3.3. We first pretrain the keypoint motion predictor \(\pi_{\mathrm{sys2}}\) on action-free data from all tools for 240,000 steps. We then freeze the pretrained predictor and fine-tune the grounded execution policy \(\pi_{\mathrm{sys1}}\) on the same action-labeled data used by the baselines for 120,000 steps.
| Method | Tobject | Book | Shoe | Overall | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 2-4 (lr)5-7 (lr)8-10 | P01 | P02 | Avg. | P01 | P02 | Avg. | P01 | P02 | Avg. | |
| FM [33] | 0.00 | 0.60 | 0.30 | 0.50 | 0.00 | 0.25 | 0.00 | 0.20 | 0.10 | 0.22 |
| (Ours) | 0.60 | 0.80 | 0.70 | 0.60 | 1.00 | 0.80 | 0.20 | 0.60 | 0.40 | 0.63 |
7 Additional Experimental Results↩︎
In this section, we first present additional real-world experiments with diverse unseen tools and hitting patterns in Appx. 7.1, and then provide more qualitative analysis in Appx. 7.2.
7.1 More Real-World Experiments↩︎
We conduct additional real-world experiments to evaluate FORGE under more diverse tool and hitting-pattern variations. Specifically, we train all methods on three seen tools, including hammer, pickaxe, and mustard, and evaluate them on three unseen tools: tobject, book, and shoe. Each unseen tool contains two hitting patterns, and each pattern is evaluated with 10 real-world rollout trials.
As shown in Tab. 6, FORGE outperforms the FM baseline across all three unseen tools, improving the overall success rate from 0.22 to 0.63. These results further confirm the main real-world finding: direct perception-to-action mapping struggles to adapt the hitting motion to novel tools, while keypoint-based functional plans provide explicit guidance for aligning the desired hitting region with the target. The FM baseline mainly fails due to ineffective downward strikes, missed hits, or incorrect hitting patterns. Please refer to our supplementary videos for qualitative demonstrations.
7.2 Additional Qualitative Comparisons↩︎
More Real-World Cases. Fig. 7 provides additional real-world comparisons between the FM baseline and FORGE across different unseen tools and hitting patterns. The FM baseline exhibits three typical failure modes: ineffective downward strikes, missed hits, and incorrect hitting patterns. These failures show that an end-to-end policy cannot reliably infer which tool region should be used or how the motion should change for a novel tool. By contrast, FORGE uses predicted keypoint-based functional plans to guide contact-region alignment and motion execution, leading to successful hits across these cases. This further supports that functional generalization benefits from an explicit intermediate representation between perception and action.
Comparison with ATM. Fig. 8 shows additional qualitative comparisons between ATM and FORGE on unseen tools. Although ATM also conditions the policy on keypoint trajectories, these trajectories are produced by a generic tracker and do not explicitly encode function-aware hitting motion. As a result, ATM often fails to guide the tool-specific contact region toward the target, leading to tool dropping or incomplete hits. In contrast, FORGE predicts function-aware keypoint trajectories that specify how the hitting point should approach the target before impact, enabling more reliable execution. These results further support our H3 in Sec. 4: keypoint trajectories must encode functional motion intent rather than only generic point tracking.
Please refer to our supplementary videos for demos of both simulation and real-world experiments.