Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale
June 13, 2025
Abstract
Affordance grounding aims to localize where to interact with an object, a fundamental capability for embodied agents. Yet progress is bottlenecked by data: manual annotation is prohibitively expensive and confines existing datasets to a narrow set of predefined object and affordance categories. We introduce Affogato, a framework for open-vocabulary affordance grounding centered on Affogato-750K, a large-scale dataset of 750K 3D affordance heatmaps paired with natural language queries. We build it with a fully automated pipeline that orchestrates foundation models to generate them at scale without human labeling. It covers significantly more diverse categories than any existing dataset. For reliable evaluation, we further provide 5K human-verified test pairs. We also present Espresso-3D and Espresso-2D, simple yet effective models with a unified architecture across both modalities. Pretraining on Affogato-750K improves both Espresso and prior methods and yields the largest gains on unseen object and affordance categories, showing that it provides broadly transferable supervision across architectures.
1 Introduction↩︎
![Figure 1: Overview of Affogato. At the heart of Affogato is Affogato-750K (dataset), validated through Espresso (models). Affogato-750K comprises 750K open-vocabulary affordance annotations across 150K 3D objects, built via an automated pipeline chaining Gemma3 [1] (query generation), Molmo [2] (point prediction), and MobileSAM [3] (mask generation) with multi-view voting for 3D aggregation. The test split is human-verified to ensure evaluation quality. Espresso-3D and Espresso-2D follow a unified architecture and demonstrate substantial pretraining gains for Espresso models and existing methods.](/article/2506.12009/figures/01_teaser/teaser_v6.png "fig:")
The theory of affordances addresses what an environment offers, provides, or furnishes animals [4]. For example, what does a cup afford to humans? The answer can be drinking, lifting, throwing, . While the concept of affordances has long been studied in psychology [5], [6] and robotics [7]–[9], here we focus on visual affordance grounding from a computer vision perspective [10]; given an object as a 3D point cloud or 2D image, the task is to localize the region where a described interaction takes place. Unlike other types of visual grounding/segmentation [11]–[13], affordance grounding is particularly challenging for two main reasons. First, an object may afford multiple functionalities across different regions, and these regions often have indistinct boundaries, making localization inherently ambiguous. Second, affordances are open-vocabulary by nature, and a fixed set of predefined terms cannot describe them adequately.
Addressing these challenges requires affordance supervision that is open-vocabulary, captures the soft and ambiguous extent of affordance regions, and can be generated at scale without exhaustive manual annotation. However, existing datasets fail to capture this open-ended nature; current 3D datasets [14]–[19] are constrained to a small set of predefined object and affordance categories, and 2D datasets [20]–[26] remain similarly limited in coverage (Table ¿tbl:tab:dataset95comparison?). This limitation stems from their reliance on manual annotation, which cannot scale to the diversity of real-world objects and interactions.
To overcome these limitations, we introduce Affogato (AFFOrdance Grounding All aT Once), a scalable framework for open-vocabulary affordance grounding (Figure 1). At its core is Affogato-750K, a large-scale dataset that directly addresses the scarcity of open-vocabulary affordance data. It is built fully automatically with an annotation pipeline that orchestrates the complementary strengths of multiple foundation models [1]–[3]. The pipeline generates open-vocabulary affordance queries, localizes and segments the corresponding interaction regions in each view, and aggregates these per-view predictions into robust 3D heatmaps.
Affogato-750K comprises 150K 3D objects from Objaverse [27], each annotated with five affordance queries and corresponding 3D heatmaps, totaling 750K annotations. We further derive a 2D split by projecting the heatmaps onto rendered views, and curate a test split of 5K human-verified pairs for reliable evaluation. As shown in Table ¿tbl:tab:dataset95comparison?, Affogato-750K significantly surpasses existing datasets in both scale and category diversity.
Alongside the dataset, we design Espresso-3D and Espresso-2D, minimal and symmetric models built from a vision encoder, a text encoder, and a heatmap decoder. Despite this simplicity, Espresso achieves strong affordance grounding performance in both modalities. Experiments demonstrate that pretraining on Affogato-750K yields substantial and consistent gains for both Espresso and existing methods, confirming the dataset’s effectiveness as a scalable supervision source across modalities.
To summarize, our contributions are as follows:
We introduce Affogato, a scalable framework for open-vocabulary affordance grounding, built on Affogato-750K, a large-scale dataset of 750K affordance annotations over 150K 3D instances that covers far more diverse object and affordance categories than existing datasets, with a 5K human-verified test split for reliable evaluation.
We develop a fully automated pipeline that orchestrates foundation models to generate reliable query-heatmap pairs at scale without manual labeling.
We present Espresso-3D and Espresso-2D, simple yet effective models with a unified design across 3D and 2D, and show that pretraining on Affogato-750K yields substantial gains for both Espresso and prior methods.
2 Related work↩︎
Visual affordance grounding. Visual affordance grounding aims to localize functional regions in visual data given affordance concepts. Early 2D approaches [20]–[26], [28], [29] focused on predefined affordance categories with limited object diversity. To overcome the scarcity of annotations, recent works explored weakly supervised learning [25], [30] and few-shot or zero-shot approaches [31], [32]. Other works leveraged external videos of human–object interactions [23], [24], [26], [33]–[35] to generate affordance heatmaps. However, supervision derived from such videos is often noisy and biased due to occlusions and limited viewpoints. Although recent advances in automation have enabled the construction of large-scale 2D datasets [35], [36], annotations based solely on single-view observations are inherently constrained compared to annotations aggregated from multi-view evidence.
In the 3D domain, 3D AffordanceNet [14] established the first closed-vocabulary 3D affordance grounding benchmark using PartNet [37] shapes. PartAfford [38] and OpenAD [15] expanded affordance vocabularies while maintaining categorical constraints. Recent advances toward open-vocabulary 3D grounding include LASO [17], which introduced free-form textual queries with 3D heatmap prediction, and 3D AffordanceLLM [18], which leveraged language models for query generation. SeqAfford [19] focused on textual diversity through GPT-4 generated questions, and LMAffordance3D [39] fused 2D and 3D features with language guidance. At the scene level, SceneFun3D [40] explored functionality segmentation, and Fun3DU [41] proposed a training-free framework leveraging VLMs for 3D scene affordance. More recently, methods that transfer 2D semantic knowledge to 3D affordance segmentation have also been explored [42], [43].
Despite progress, existing datasets remain severely constrained in scale and category diversity (Table ¿tbl:tab:dataset95comparison?). 3D datasets share the same 23 object classes derived from a single source [37], while 2D datasets cover at most 304 object and 73 affordance categories. All existing datasets rely on predefined closed-set labels, fundamentally limiting generalization to open-vocabulary queries. This data scarcity is the primary bottleneck for affordance grounding generalization.
Harnessing 2D foundation models for 3D supervision. Recent advances in 2D foundation models have opened new pathways for generating high-quality 3D supervision without manual annotation. Several works [44]–[46] leverage large language models (LLMs) and vision-language models (VLMs) to generate text captions for 3D objects using their multi-view rendered images, facilitating joint 3D-language learning. Others [47], [48] distill 2D visual knowledge from foundation models to train 3D geometric encoders, while scene-centric methods [49]–[51] extend this paradigm to large-scale environments by generating region-level 3D annotations through integration with 2D foundation models. Our work extends this paradigm to functional affordance understanding by leveraging 2D foundation models to generate 3D affordance supervision. A vision-language model generates affordance queries and interaction points, MobileSAM [3] produces 2D segmentation masks, and the resulting masks are lifted to 3D and aggregated via multi-view voting to obtain reliable 3D affordance heatmaps.
3 The Affogato-750K dataset↩︎
Advancing human-object interaction understanding in embodied AI systems demands comprehensive affordance grounding data—a critical resource currently lacking in the field. Existing datasets [14], [17] suffer from key limitations in scale, diversity, and annotation quality, creating a substantial barrier to progress in this domain. To address the issue, we propose the Affogato-750K dataset, a large-scale open-vocabulary affordance grounding dataset. In this section, we elaborate on the pipeline used to generate Affogato-750K. Our pipeline leverages large-scale 3D object repositories [27] and state-of-the-art foundation models [1]–[3] to automatically generate high-quality affordance annotations. This addresses the aforementioned limitations by enabling robust and generalizable affordance learning across diverse objects.
3.1 Source dataset↩︎
We build the Affogato-750K dataset upon Objaverse [27], one of the largest public 3D asset repositories. It contains more than 700K web-crawled 3D object meshes spanning diverse object categories and geometries. Since recent vision-language models (VLMs) take 2D images and text as input and cannot directly process 3D meshes, we use multi-view renderings of each 3D object as an intermediary representation that bridges 3D geometry and language. To this end, we utilize G-Objaverse [52], which already provides high-resolution multi-view renderings, along with depth maps and camera parameters, for over 280K Objaverse objects. Specifically, we select four subsets that have strong relevance to human-object interaction and functional affordances in daily lives: Daily-Used, Furnitures, Transportations, and Electronics, which results in 150K 3D objects.
3.2 Annotation pipeline↩︎
Our data annotation pipeline consists of three stages. Given a textured 3D object as input, it outputs a set of natural language affordance queries alongside spatially localized 3D affordance heatmaps. The entire process is automated and designed to scale to hundreds of thousands of objects, making it suitable for constructing large-scale datasets. The overall annotation process is illustrated in Figure 2. Stage-wise qualitative results are provided in the supplementary material.
![Figure 2: Overview of our data annotation pipeline. Given multi-view renderings of an object, Gemma3 [1] generates affordance queries, Molmo [2] points the affordance, and MobileSAM [3] converts the point into an affordance heatmap. The multi-view 2D affordance heatmaps are then aggregated on the 3D object surface to obtain a final 3D affordance heatmap.](/article/2506.12009/figures/02_data_pipeline/data_generation_final_2.png "fig:")
Stage 1: Open-vocabulary affordance query generation. Given multi-view images of a 3D object, we employ a vision-language model, Gemma3 [1], to produce a set of natural language queries (five per object) that describe how a human might interact with the object. These queries follow a constrained yet expressive format, allowing open-vocabulary interaction descriptions while maintaining spatial grounding. By conditioning on rendered views instead of object class labels [17]–[19], our approach leverages the rich knowledge embedded in VLMs to generalize to open-vocabulary understanding of affordances. This also enables the system to adapt to various intra-class variations (, chairs with and without armrests) while maintaining consistent affordance identification across diverse object geometries and functional categories.
Stage 2: Language-guided interaction point prediction. Once the affordance queries are generated, we utilize Molmo [2], a multimodal model capable of grounding natural language queries to spatial locations in images. Specifically, Molmo demonstrates remarkable precision in identifying exact pixel locations when provided with input images and natural language prompts that request localization of specific regions in the image. This capability is crucial for accurately mapping affordance queries to their corresponding spatial locations on the object. For each query and image pair, we instruct Molmo to predict pixel coordinates that represent the most likely interaction point in the given view. The predicted interaction points across views are then used to guide the next stage.
Stage 3: Affordance heatmap generation and aggregation. We convert discrete interaction points into continuous 3D heatmap representations. First, the predicted point from each multi-view image is used as a prompt for MobileSAM [3] to generate a 2D segmentation mask. To prioritize the precise interaction region over broader contextual areas, we adopt a smallest-mask selection strategy. The segmentation logits are transformed through a sigmoid function to create probabilistic heatmaps with values between 0 and 1, representing the likelihood of an affordance at each pixel location. These 2D heatmaps from multiple viewpoints are then projected onto the 3D object surface using the known camera parameters and depth information. We employ a voting-based aggregation process where each view contributes to the final 3D representation, with regions consistently identified across multiple views receiving higher confidence scores. The output of this stage serves as our final 3D affordance heatmap annotations.
Rendering 2D affordance heatmaps. To bridge our 3D dataset with 2D image domains, we project the aggregated 3D affordance heatmaps onto 2D image planes from multiple viewpoints. Since the multi-view aggregation in Stage 3 resolves the inconsistency of Molmo and MobileSAM predictions, these rendered 2D heatmaps exhibit high consistency across viewpoints with accurate affordance regions. For each object, we render heatmaps from 25 evenly distributed viewpoints and calculate affordance region visibility by summing the projected heatmap values. We select the most visible viewpoint for optimal representation.
Error mitigation. During the automatic dataset generation pipeline, we employ several strategies to mitigate errors. First, to enhance the fidelity of Gemma3’s responses, we adopt chain-of-thought (CoT) prompting [53], instructing the model to first predict the object’s semantic class and then generate affordance queries conditioned on that class and its functionalities. This approach prevents the model from relying solely on object shape, which often leads to affordance queries that fail to capture the object’s intended functions. Second, we apply a multi-view aggregation described in Stage 3 to reduce errors. Despite Molmo’s exceptional pointing capabilities, its predicted interaction points can still contain errors and vary across viewpoints. Although a few views produce incorrect heatmaps, the consensus aggregation reinforces the majority of correct predictions, resulting in robust, view-consistent outputs.
3.3 Statistics and analysis↩︎
Data statistics. As summarized in Table ¿tbl:tab:dataset95comparison?, Affogato-750K provides over 750K affordance annotations across 150K 3D object instances in four categories: Daily-Used includes the largest number of instances (121,799), followed by Transportations (11,609), Furnitures (8,759), and Electronics (7,937), each with 5 affordance query-heatmap pairs.
Semantic and spatial diversity. Unlike existing datasets constrained by predefined taxonomies, Affogato-750K spans a wide range of objects and affordances by leveraging VLM knowledge to generate diverse and context-aware affordance queries. Figures [fig:wordcloud95object] and [fig:wordcloud95affordance] demonstrate this semantic breadth across object classes and affordance types. Our annotations capture varied interaction patterns, from precise point interactions (button pressing) to extended surface interactions (holding), as shown in Figure [fig:characteristics95diversity]. Quantitatively, Table [tab:characteristics95heatmap] shows Affogato-750K achieves significantly higher coverage (ratio of points covered by annotation union) and diversity (average pairwise KL divergence between heatmaps) scores than LASO [17], indicating richer and more complementary affordance representations.
Human validation and test split construction. To construct a reliable test split, we randomly sample 5K affordance query-heatmap pairs and assign them in disjoint subsets to 10 hired annotators for verification. We instruct annotators to rate each pair on semantic relevance, spatial accuracy, and heatmap coverage, and to inspect all stage-wise intermediates to locate the originating stage of any error. This evaluation shows strong agreement with human judgment, achieving an 84.8% success rate, and reveals the common failure modes of our pipeline (Figure 4). We then build the test split from these verified pairs: incorrect queries are removed and incorrect interaction points are re-annotated through a dedicated annotation interface, yielding an object-instance-level test pool that is held out from training with no instance, view, or query overlap.
4 Espresso models↩︎
Our goal is to isolate the contribution of Affogato-750K, attributing improvements in affordance grounding to the dataset itself rather than to model design. However, prior 2D [25], [31] and 3D [14], [17], [18] affordance methods have evolved in separate research streams, each accumulating its own modality-specific design choices. Adopting such methods as cross-modal baselines would conflate architectural differences with dataset contribution, making it difficult to cleanly attribute performance gains to Affogato-750K. We therefore present Espresso, a unified baseline that is symmetric across 2D and 3D, isolating the dataset’s contribution from architectural differences.
Concretely, we present Espresso-3D for 3D point clouds and Espresso-2D for 2D images. Each model consists of a modality-specific visual encoder, a text encoder, and a text-conditioned heatmap decoder (Figure 5). The core of our design is a text-conditioned heatmap decoder that uses text embeddings as its queries instead of learnable ones. Although this follows the standard transformer-based mask decoder architecture [54]–[58], using text embeddings as queries naturally supports open-vocabulary affordance grounding without predefined categories. The following describes the modality-specific designs.
Espresso-3D. For the 3D vision encoder, we use the pretrained PartField model [48], which captures generic part concepts from large-scale 3D data. For the text encoder, we employ Recap-CLIP [59], which provides robust language understanding capabilities. Following PointRefer [17], we finetune both the pretrained 3D vision encoder and text encoder during training to adapt them to our downstream task while leveraging their strong pretrained representations.
Espresso-2D. For the 2D vision encoder, we use the pretrained DINOv2 model [60], and CLIP [61] as the text encoder. Following OOAL [31], we freeze both encoders during training to maintain their pretrained representations. Our architecture adapts the design of OOAL [31] but differs in that we remove the learnable prompt component and take a single affordance query as input, following the referring expression segmentation framework.
5 Experiments↩︎
5.1 Datasets and baselines↩︎
3D datasets. We primarily evaluate Espresso-3D on the LASO dataset [17], a benchmark for open-vocabulary 3D affordance grounding with free-form text queries. LASO provides both seen and unseen settings; the seen setting evaluates on object/affordance classes observed during training, while the unseen setting tests on held-out classes to assess zero-shot generalization. We further evaluate on our Affogato-750K test split, a held-out set of 5K human-verified pairs that covers substantially more diverse object categories and affordance concepts than LASO. It serves as a more challenging testbed and provides both in-domain and cross-domain settings to probe open-vocabulary generalization beyond the limited category coverage of existing benchmarks.
2D datasets. We evaluate Espresso-2D on the AGD20K [25], which consists of two object splits: seen and unseen. AGD20K-Weak, 1-shot, and Full represent weakly supervised, one-shot, and fully supervised versions of the AGD20K training dataset, respectively.
3D baselines. We select representative baselines for open-vocabulary 3D affordance grounding, which take language descriptions paired with 3D point clouds as input. Our evaluation includes multiple representative methods: Referring Transformer [62], 3D-SPS [44], ReLA [63], IAGNet [16], OpenAD [15], and PointRefer [17]. For OpenAD, since official results on LASO [17] were not available, we trained the model on the LASO dataset using their official code implementation. All other baseline results are taken directly from [17]. We compare these methods with our proposed Espresso-3D on both LASO seen and unseen splits to evaluate generalization performance.
2D baselines. For zero‑shot evaluation on AGD20K, we use Molmo+SAM2 [2], [64], LISA-7B [13] (a reasoning segmentation model), M\(^2\)SA-7B [65] (a part-level referring segmentation model), and AffordanceNet [36] (a reasoning-based affordance segmentation model). We format the open-vocabulary query for an affordance as “Point to the part that you should interact with to {affordance}”. For the affordance-specific model baselines in Table ¿tbl:tab:agd20k-all? (b), we adopt results from [66] and additionally include AffordanceNet [36].
Evaluation metrics. For 3D evaluation, following prior work [17], we use average Intersection over Union (aIoU), Area Under the ROC Curve (AUC), Similarity (SIM), and Mean Absolute Error (MAE). Since MAE is sensitive to annotation scale, we primarily report aIoU, AUC, and SIM. For 2D evaluation, we use Kullback-Leibler Divergence (KLD), Similarity (SIM), and Normalized Scanpath Saliency (NSS) [67]. Higher values are better for aIoU, AUC, SIM, and NSS while lower values are better for MAE and KLD.
| Method | Seen | Unseen | ||||||
| aIoU\(\uparrow\) | AUC\(\uparrow\) | SIM\(\uparrow\) | MAE\(\downarrow\) | aIoU\(\uparrow\) | AUC\(\uparrow\) | SIM\(\uparrow\) | MAE\(\downarrow\) | |
| Ref. Trans. [62] | 13.7 | 79.8 | 49.7 | 0.124 | 10.2 | 69.1 | 43.2 | 0.145 |
| 3D-SPS [44] | 11.4 | 76.2 | 43.3 | 0.138 | 7.9 | 68.8 | 40.2 | 0.158 |
| ReLA [63] | 15.2 | 78.9 | 53.2 | 0.118 | 10.7 | 69.7 | 42.9 | 0.144 |
| IAGNet [16] | 17.8 | 82.3 | 56.1 | 0.109 | 12.9 | 77.8 | 44.3 | 0.129 |
| OpenAD [15] | 14.2 | 85.1 | 53.3 | 0.103 | 14.6 | 80.7 | 51.8 | 0.109 |
| \(+\) pretrain | 16.1 | 86.8 | 53.9 | 0.100 | 15.5 | 81.8 | 53.4 | 0.103 |
| PointRefer [17] | 20.8 | 87.3 | 62.9 | 0.093 | 14.6 | 80.2 | 50.7 | 0.119 |
| \(+\) pretrain | 20.2 | 86.0 | 60.0 | 0.098 | 18.6 | 81.4 | 56.1 | 0.103 |
| -3D | 20.4 | 86.0 | 63.3 | 0.102 | 18.7 | 80.0 | 60.0 | 0.101 |
| 21.9 | 85.9 | 63.7 | 0.116 | 20.8 | 82.9 | 61.4 | 0.122 | |
![Figure 6: Qualitative comparison between OpenAD [15], PointRefer [17], and Espresso-3D on the LASO [17] test split.](/article/2506.12009/figures/06_qual3d/qual_3d_v2.png "fig:")
| Method | All \(\rightarrow\) All | Daily-used \(\rightarrow\) Furnitures | Furnitures \(\rightarrow\) Daily-used | |||||||||
| aIoU\(\uparrow\) | AUC\(\uparrow\) | SIM\(\uparrow\) | MAE\(\downarrow\) | aIoU\(\uparrow\) | AUC\(\uparrow\) | SIM\(\uparrow\) | MAE\(\downarrow\) | aIoU\(\uparrow\) | AUC\(\uparrow\) | SIM\(\uparrow\) | MAE\(\downarrow\) | |
| OpenAD | 3.1 | 64.8 | 32.9 | 0.150 | 7.6 | 67.8 | 36.8 | 0.177 | 1.6 | 54.3 | 30.8 | 0.201 |
| PointRefer | 10.5 | 76.1 | 40.5 | 0.120 | 13.5 | 78.9 | 44.3 | 0.157 | 2.8 | 60.6 | 26.5 | 0.122 |
| -3D | 13.6 | 79.0 | 42.9 | 0.111 | 18.2 | 80.4 | 47.5 | 0.125 | 4.6 | 62.4 | 30.4 | 0.130 |
5.2 3D affordance grounding↩︎
LASO [17] test split. Table 1 presents the 3D affordance grounding results on LASO. Without pretraining on Affogato-750K, all methods achieve reasonable performance on the seen setting but show limited generalization to unseen object-affordance combinations. Pretraining on Affogato-750K consistently and substantially improves unseen performance across all methods: OpenAD gains 0.9%p aIoU, PointRefer achieves the largest gain of 4.0%p aIoU (18.6 vs. 14.6), and Espresso-3D gains 2.1%p aIoU. While pretrained PointRefer shows a slight degradation on the seen setting (20.2 vs. 20.8 aIoU), it achieves the largest unseen improvement; Espresso-3D improves consistently on both seen (+1.5%p aIoU) and unseen (+2.1%p aIoU). These results confirm that Affogato-750K pretraining provides a strong and consistent benefit for open-vocabulary generalization, regardless of model architecture. Figure 6 provides a qualitative comparison on the LASO test split.
Affogato-750K test split. We evaluate open-vocabulary 3D affordance grounding on the Affogato-750K test split under both in-domain and cross-domain settings (Table 2), comparing Espresso-3D against OpenAD [15] and PointRefer [17] retrained on our dataset. Since benchmarks like LASO cover few categories, this experiment tests generalization under the broader and more diverse distribution of Affogato-750K. The in-domain split (All \(\rightarrow\) All) trains and evaluates on all objects, while the cross-domain splits (Daily-used \(\rightarrow\) Furnitures and Furnitures \(\rightarrow\) Daily-used) train on one object domain and test on the other to measure cross-domain transfer. Even the strongest methods leave substantial headroom on Affogato-750K, reflecting the broad object and affordance coverage over which it requires generalization. Across all three settings, Espresso-3D consistently outperforms OpenAD and PointRefer, mirroring their ranking on LASO and indicating that these trends are not benchmark-specific. The cross-domain results are strongly asymmetric: training on diverse daily-used objects and testing on furniture (Daily-used \(\rightarrow\) Furnitures) far outperforms the reverse (18.2 vs. aIoU for Espresso-3D), since the narrow furniture domain covers little of the broader daily-used categories.
5.3 2D affordance grounding↩︎
Table ¿tbl:tab:agd20k-all? (a) presents results on the test sets of AGD20K’s seen and unseen object splits, evaluated zero-shot without any AGD20K training. Each baseline is pretrained on its own respective dataset, while Espresso-2D is pretrained on Affogato-750K. Espresso-2D with Affogato-750K pretraining outperforms all baselines across both test sets, achieving 40.2 SIM on the seen split and 37.6 SIM on the unseen split versus 22 to 30 SIM for competing methods, demonstrating that Affogato-750K pretraining yields affordance representations that generalize well across diverse object categories. Table ¿tbl:tab:agd20k-all? (b) presents supervised learning results across different supervision levels. Espresso-2D with Affogato-750K pretraining, fine-tuned on AGD20K, achieves the best performance across all supervision settings. Notably, Affogato-750K pretraining provides consistent improvements over training solely on AGD20K—the Affogato-750K-pretrained Espresso-2D outperforms its from-scratch counterpart by 0.060 KLD and 1.6 SIM under full supervision, and AffordanceNet similarly benefits from Affogato-750K pretraining—validating that Affogato-750K’s diverse affordance concepts transfer across domain boundaries even when target-domain data is available. Figure 9 shows qualitative examples.
.47
| Method | KLD\(\downarrow\) | SIM\(\uparrow\) | NSS\(\uparrow\) |
|---|---|---|---|
| Seen split | |||
| Molmo+SAM2[2], [64] | 1.804 | 26.1 | 0.729 |
| LISA-7B [13] | 1.627 | 29.6 | 0.819 |
| M\(^2\)SA-7B [65] | 1.772 | 25.8 | 0.620 |
| AffordanceNet [36] | 1.729 | 26.6 | 0.755 |
| -2D | 1.426 | 40.2 | 0.985 |
| Unseen split | |||
| Molmo+SAM2 [2], [64] | 1.953 | 22.6 | 0.718 |
| LISA-7B [13] | 1.830 | 25.6 | 0.765 |
| M\(^2\)SA-7B [65] | 1.925 | 22.7 | 0.657 |
| AffordanceNet [36] | 1.932 | 22.6 | 0.700 |
| -2D | 1.571 | 37.6 | 1.016 |
.53
| Method | Sup. | KLD\(\downarrow\) | SIM\(\uparrow\) | NSS\(\uparrow\) |
|---|---|---|---|---|
| Cross-view-AG [25] | Weak | 1.787 | 28.5 | 0.829 |
| Cross-view-AG+ [68] | 1.765 | 27.9 | 0.882 | |
| AffCorrs [69] | 1.618 | 34.8 | 1.021 | |
| LOCATE [70] | 1.405 | 37.2 | 1.157 | |
| WSAG-PLSP [71] | 1.153 | 43.7 | 1.418 | |
| OOAL [31] | 1-shot | 1.070 | 46.1 | 1.503 |
| LOCATE-Sup [70] | Full | 1.907 | 23.6 | 0.641 |
| LOCATE-Sup-OWL [70], [72] | 1.927 | 23.4 | 0.624 | |
| AffordanceLLM [66] | 1.463 | 37.7 | 1.070 | |
| AffordanceNet [36] | 1.871 | 30.5 | 0.750 | |
| \(+\) pretrain | 1.721 | 33.7 | 0.852 | |
| -2D | 1.034 | 50.3 | 1.550 | |
| 0.974 | 51.9 | 1.645 |
5.4 Analysis↩︎
.46
| Pretraining | Filtering | aIoU\(\uparrow\) | AUC\(\uparrow\) | SIM\(\uparrow\) | MAE\(\downarrow\) |
|---|---|---|---|---|---|
| 18.7 | 80.0 | 60.0 | 0.101 | ||
| ✔ | 20.8 | 82.9 | 61.4 | 0.122 | |
| ✔ | ✔ | 20.2 | 82.4 | 61.1 | 0.119 |
.46
| Pretraining | Filtering | KLD\(\downarrow\) | SIM\(\uparrow\) | NSS\(\uparrow\) |
|---|---|---|---|---|
| 1.034 | 50.3 | 1.550 | ||
| ✔ | 0.974 | 51.9 | 1.645 | |
| ✔ | ✔ | 1.016 | 52.0 | 1.631 |
Effect of held-out category filtering. To check that the pretraining gains are not simply due to Affogato-750K covering categories that each benchmark holds out, we remove the pretraining samples that fall into those held-out classes and re-train. We query Gemma3 [1] to judge whether each sample’s class label falls into a benchmark’s held-out categories, and conservatively remove any potential match. For LASO, which holds out (object, affordance) pairs, we drop samples whose predicted pair appears in the evaluation set (5.46%); for AGD20K, which holds out object categories, we exclude samples assigned to those categories (14.0%). Since these predictions are imperfect, the filter likely over-removes, making the numbers a lower bound. Even so, filtered pretraining consistently surpasses training from scratch on both benchmarks (Table ¿tbl:tab:label95leakage?; on LASO, 20.2 vs. aIoU, within 0.6 of full pretraining), confirming that Affogato-750K’s benefits come from diverse supervision at scale rather than from covering the held-out categories.
![Figure 7: Zero-shot Espresso-2D on real-world robot scenes. Despite the visual domain gap from Objaverse renderings, Espresso-2D pretrained on Affogato-750K localizes open-vocabulary affordances correctly across diverse, cluttered Open X-Embodiment [73] robot workspace images.](/article/2506.12009/figures/07_oxe/oxe_qual_four.png "fig:")
Synthetic-to-real generalization. Although Affogato-750K is built from clean, isolated Objaverse renderings, the pretrained Espresso-2D transfers to realistic, in-the-wild inputs. To probe a large domain gap, we evaluate Espresso-2D zero-shot on cluttered robot workspace images from Open X-Embodiment [73]. As shown in Figure 7, despite never observing such scenes during training, Espresso-2D still localizes the queried affordances correctly across diverse robot-manipulation scenes, indicating that the affordance representations learned from Affogato-750K generalize beyond the synthetic source domain.
Figure 8: Analyses of our data annotation pipeline. (a) CoT prompting produces functionally grounded affordance queries rather than shape-only guesses. (b) Among the three mask hypotheses returned by SAM, the smallest isolates part-level structure at affordance granularity.. a — CoT prompting, b — Mask-size selection
Effect of chain-of-thought prompting. We qualitatively analyze the effect of CoT [53] prompting in our data annotation pipeline for affordance query generation. As shown in Figure 8 (a), without CoT prompting the model often fails to capture the object’s functional properties and instead relies primarily on shape when generating queries. For example, it produces “Point to the support where you would sit.” for a hat stand or “Point to the part you would hold to carry this.” for a door. In contrast, CoT prompting encourages functional reasoning, producing more appropriate queries such as “Point to the handle to open the door,” or “Point to the part you would place the hat on.”
Effect of mask-size selection. Another key design choice in our pipeline concerns how interaction points are turned into region masks. SAM [3], [64] returns three mask hypotheses for each interaction-point prompt, and we adopt the smallest one, as stated in Section 3.2. Figure 8 (b) compares the three choices: the larger masks tend to snap to whole-object silhouettes, whereas the smallest mask isolates part-level structures (, handle, lid, button) at affordance granularity. Although the smallest mask can occasionally miss parts in individual views, our multi-view aggregation closes such per-view gaps, making the smallest-mask choice the most reliable basis for precise affordance heatmaps.
6 Discussion and conclusion↩︎
![Figure 9: Qualitative comparison between Cross-view-AG [25], LOCATE [70], WSAG-PLSP [71], OOAL [31], and Espresso-2D, which is pretrained on the Affogato-750K train split and then finetuned on the AGD20K train split with full supervision.](/article/2506.12009/figures/09_qual2d/qual_2d_row4_v2.png "fig:")
We have presented Affogato, a unified framework for open-vocabulary affordance grounding across 3D and 2D domains. By chaining foundation models, our automated pipeline produces Affogato-750K, the largest affordance grounding dataset with 750K affordance annotations across 150K 3D assets, eliminating costly manual annotation while capturing diverse interactions beyond predefined categories. Our Espresso-3D and Espresso-2D models consistently benefit from pretraining on Affogato-750K, demonstrating improved generalization to unseen object-affordance combinations. Notably, our annotation pipeline can be readily applied to other 3D datasets, offering a practical path toward ever-growing affordance supervision. While our framework effectively localizes contact points for a given action, predicting finer-grained information such as temporal dynamics and force requirements [74] remains an open challenge. We believe Affogato and Affogato-750K provide a solid foundation for future research toward more comprehensive embodied AI systems.
Acknowledgements↩︎
This work was supported by RLWRLD Inc. and also by IITP grants (RS-2022-II220959: Few-Shot Learning of Causal Inference in Vision and Language for Decision Making (50%), RS-2022-II220290: Visual Intelligence for Space-Time Understanding & Generation (25%), RS-2024-00457882: National AI Research Lab Project (20%), RS-2019-II191906: AI Graduate School Program at POSTECH (5%)) funded by Ministry of Science and ICT, Korea.
7 Dataset link↩︎
https://huggingface.co/datasets/project-affogato/affogato
8 Implementation details↩︎
8.1 Data Annotation Pipeline↩︎
Use of pretrained models. Our data annotation pipeline uses the following pretrained models: (1) Gemma3 [1] (google/gemma-3-4b-it) for generating natural language affordance queries (2) Molmo [2] (allenai/Molmo-7B-D-0924) for predicting interaction points in 2D images, and (3) MobileSAM [3] for predicting 2D heatmap given interaction point prompts.
Computational resources. For Affogato-750K dataset generation, we use 8 NVIDIA H100 GPUs with 80GB of memory each. The data generation pipeline takes approximately 24 hours to process 150K Objaverse instances.
Image sampling. G-Objaverse [52] provides 38 views per object. For computational efficiency, we used the first 25 views that are captured with the same elevation but uniformly distributed azimuths around the object. For stage 1 of our pipeline, we sample 5 images at equal intervals from these 25 views to generate affordance queries. This sampling strategy ensures comprehensive coverage of the object from multiple perspectives while optimizing computational resources. For the remaining stages, we utilized all 25 views.
Human evaluation. As discussed in Section 3.3, we instruct human annotators to evaluate the quality of our automatically generated annotation and to refine annotations for those didn’t pass the quality check. We guide the annotators to rate the affordance query-heatmap pairs from three criteria: (1) semantic relevance between the query and object, (2) spatial accuracy of the predicted interaction points, and (3) coverage of the heatmap for the intended affordance. We provide the annotators with a web-based interactive viewer for screening the affordance query-heatmap pairs and assigning ratings based on the three criteria. The example of the web-based interactive UI is shown in Figure 10.
Figure 10: Web-based interactive viewer for (Left) quality evaluation and (Right) human refinement.
8.2 3D affordance grounding↩︎
Espresso-3D architecture. Our Espresso-3D model leverages the PartField [48] architecture for processing 3D visual inputs and incorporates Recap-CLIP [59] for encoding language queries. Specifically, we adopt a hybrid 3D encoder composed of PVCNN [75] and a triplane transformer, as introduced in the official PartField repository. We used the official pretrained checkpoint and froze the entire vision backbone during training to ensure consistent feature extraction.
For the text branch, we utilize the Recap-CLIP text encoder, which provides enhanced language grounding compared to standard CLIP variants. The resulting query embeddings are fed into a conditional heatmap decoder that predicts spatial affordance distributions over 3D points. The decoder augments the 3D point features from the vision encoder with Fourier-based positional encodings and uses them as keys and values in a cross-attention mechanism, where the language embeddings act as queries. The attended features are then refined through a residual feedforward network (an MLP with skip connections), which outputs the final heatmap over the point cloud.
Training detail. To ensure a fair comparison with prior methods, we adopt the same loss formulation as LASO [17]. The model is optimized using a combination of Binary Cross-Entropy (BCE) Loss to handle classification and Dice Loss [76] to improve region-level alignment. The two losses are summed with equal weights to form the final training objective. We use the same training setup for both the LASO and Affogato-750K datasets: 50 epochs, batch size of 64, and training on 8 NVIDIA RTX A6000 GPUs.
8.3 2D affordance grounding↩︎
| Method | Data | KLD \(\downarrow\) | SIM \(\uparrow\) | NSS \(\uparrow\) | |
| pretrain | augmentation | ||||
| Seen split | |||||
| -2D | ✔ | 1.493 | 35.5 | 0.920 | |
| -2D | ✔ | ✔ | 1.426 | 40.2 | 0.985 |
| Unseen split | |||||
| -2D | ✔ | 1.688 | 31.3 | 0.876 | |
| -2D | ✔ | ✔ | 1.571 | 37.6 | 1.016 |
| Method | Data | KLD\(\downarrow\) | SIM\(\uparrow\) | NSS\(\uparrow\) | |
| pretrain | Augmentation | ||||
| -2D | ✔ | 1.100 | 47.0 | 1.497 | |
| -2D | ✔ | ✔ | 0.974 | 51.9 | 1.645 |
AGD20K dataset. AGD20K-Weak refers to the original AGD20K dataset. The training set consists of 23,083 / 13,323 image-level labels for the Seen / Unseen splits, respectively, while the corresponding test sets contain 1,675 / 540 images. AGD20K-Oneshot refers to the AGD20K dataset for one-shot affordance learning. The training set consists of 50 / 33 images—one per object class—for the Seen / Unseen splits, respectively. The test set is identical to that of AGD20K-Weak. AGD20K-Full is constructed for fully supervised training, following the setup of [66]. The training set consists of 999 images including object classes from the training set of AGD20K’s unseen split, each annotated with dense pixel-level affordance masks. The test set contains 540 images from object classes in the test set of the unseen split.
Background augmentation in Espresso-2D pretraining stage. For the pretraining stage of Espresso-2D, we replace the null background in each rendered image with a randomly selected background from the Background dataset [77]. After background replacement, the image is resized to 256×256, randomly cropped to 224×224, and horizontally flipped with a random probability. Table ¿tbl:tab:augment? summarizes zero-shot performance on AGD20K with and without background augmentation during pre-training, as well as the fine-tuning results on AGD20K-Full. Empirically, we observe that background augmentation leads to better generalization compared to pretraining without it.
Espresso-2D architecture. Our Espresso-2D architecture is adapted from OOAL [31]. Multi-level features from different layers of DINOv2 are aggregated. To focus attention on foreground regions, cross-attention is restricted to the regions indicated by the mask derived from the CLS token. Unlike OOAL, which employs text prompt learning with fixed affordance labels as input, our model takes a single natural language query as the text input without using any text prompt learning.
Training detail. We used the CLIP ViT-B/16 as the text encoder and DINOv2 ViT-B/14 as the vision backbone. During pre-training on Affogato-750K, the model is optimized using Adam with a learning rate of 0.001. The training is conducted for 52,000 iterations with a per-GPU batch size of 512 on 7 NVIDIA RTX 3090 GPUs. For fine-tuning on AGD20K-Full, we use the Adam optimizer with a learning rate of 0.0001. Training is performed for 400 iterations with a batch size of 512 on a single NVIDIA RTX 3090 GPU. Binary cross-entropy loss is employed consistently in both the pretraining and fine-tuning stages.
9 Analyses↩︎
9.1 Analyses on Data Annotation Pipeline↩︎
Effect of data resolution.
| Method | Training Res. | Test Resolution (# Points) | |||
|---|---|---|---|---|---|
| 3-6 | (# Points) | 2048 | 4096 | 8192 | 16384 |
| -3D | 2048 | 0.237 | 0.245 | 0.249 | 0.251 |
| 16384 | 0.265 | 0.278 | 0.283 | 0.287 | |
Affogato-750K provides point clouds with a high resolution of 16,384 points, enabling the capture of fine-grained geometric details critical for affordance understanding. This represents a significant improvement over LASO [17], which provides only 2,048 points resolution. To assess the effect of data resolution, we trained Espresso-3D on Affogato-750K at two resolutions (2,048 vs. 16,384 points) and report the results in Table 3. The model trained with high-resolution data consistently outperforms its low-resolution counterpart across all test resolutions, indicating that learning fine-grained geometric details during training is crucial for accurate 3D affordance grounding at any scale.
Human vs. our annotation pipeline.
![Figure 11: Our annotation pipeline vs. Human. Comparison is conducted on 3D-AffordanceNet [29] meshes. First and third columns show affordance heatmaps predicted by Affogato-750K annotation pipeline, while second and fourth columns are human-annotated.](/article/2506.12009/figures/11_affogato_engine/affogato-engine.png "fig:")
We compare the affordance predictions from our annotation pipeline with human annotations on 3D-AffordanceNet [29] meshes. Figure 11 presents a qualitative comparison, where the first and third columns display affordance heatmaps generated by our annotation pipeline, while the second and fourth columns show human-annotated ground truth from 3D-AffordanceNet. The visual comparison demonstrates that our automated pipeline produces affordance heatmap predictions that closely align with human intuition about object affordances. This suggests that our annotation pipeline can serve as a reliable substitute for manual annotation, significantly reducing the time and effort required to create large-scale datasets.
Failure modes of annotation pipeline. Despite enabling automatic affordance annotation at scale, our annotation pipeline exhibits several failure modes in its pipeline. Across all evaluated samples, the dominant failure mode is wrong interaction point prediction in Stage 2, accounting for 34.9% of all failure cases. In these cases, Molmo struggles with fine-grained grounding, failing to localize the precise functional region and instead pointing to a semantically related but incorrect part of the object, and the error propagates to the subsequent mask generation stage. Stage 1 accounts for another 36.8% of failure cases in total, comprising incorrect affordance queries (27.6%) and object misidentification (9.2%). Despite our use of chain-of-thought prompting, Gemma3 [1] occasionally misidentifies the object category or generates affordance queries that are semantically misaligned with the object’s actual functionality, and such errors then propagate to later stages. Limited camera coverage accounts for 10.5% of failure cases across all stages, particularly when the queried object part is occluded or absent from the rendered views. For example, for the query “Point to the part where you would use to brake the car,” the brake pedal is often not visible because our multi-view images are captured along a circular trajectory around the object’s exterior. Finally, SAM-based mask generation [3], [64] contributes 5.2% of failure cases due to edge bias, which tends to over-emphasize object boundaries and can produce incomplete affordance masks or merge adjacent but functionally distinct parts. We further observe 12.5% miscellaneous failures.
9.2 Effect of pretraining-data scale↩︎
| Pretrain subset | 50K | 150K | 400K | 750K |
|---|---|---|---|---|
| LASO-unseen aIoU \(\uparrow\) | 18.7 | 19.4 | 19.6 | 20.8 |
| AGD20K-unseen KLD \(\downarrow\) | 1.064 | 1.007 | 0.974 | 0.974 |
To verify the “at scale” claim, we pretrain on Affogato-750K subsets of 50K, 150K, 400K, and 750K object-text pairs and measure downstream generalization (Table 4). Espresso-3D LASO-unseen aIoU improves monotonically at every step (\(18.7\to19.4\to19.6\to20.8\)), and Espresso-2D AGD20K-unseen KLD improves correspondingly while saturating early. The monotonic gains confirm that the improvement is driven by dataset scale rather than by the mere presence of pretraining. We attribute the earlier saturation in the 2D setting to the domain gap between Affogato-750K’s synthetic renderings and AGD20K’s real-world imagery, which we expect to bound the additional benefit that more synthetic pretraining data can provide.
9.3 Cross-domain asymmetry↩︎
| Train | #obj | \(\rightarrow\) Daily | \(\rightarrow\) Furn. |
|---|---|---|---|
| Daily-Used (full) | 121,179 | 10.4 | 18.2 |
| Daily-Used (sub.) | 8,759 | 6.0 | 10.7 |
| Furnitures | 8,759 | 4.6 | 20.3 |
As shown in the main-paper 3D affordance grounding results (Sec. 5.2, Table 3), the Affogato-750K cross-domain splits exhibit an asymmetry: Espresso-3D trained on Daily-Used and evaluated on Furnitures reaches 18.2 aIoU, whereas the reverse direction attains only 4.6. We attribute this asymmetry primarily to training volume and secondarily to category coverage. The two splits differ by \(\approx\)14\(\times\) in training volume (121K vs.8.8K objects) and \(\approx\)4\(\times\) in test-class diversity (515 vs. classes). As shown in Table 5, sub-sampling Daily-Used to match the Furnitures volume halves the cross-direction gap (\(13.6\to6.1\)), identifying training volume as the dominant factor, since geometry is invariant to sub-sampling. The residual gap reflects class coverage: Daily-Used-trained models cover the 122 Furnitures classes more readily than the reverse covers 515, so even volume-matched Daily-Used (sub.) scores higher on Furnitures (10.7) than on Daily-Used (6.0).
9.4 Failure cases of Espresso↩︎
Figure 12 illustrates failure cases from the zero-shot evaluation of the Espresso-2D model, pretrained on Affogato-750K, on the AGD20K dataset. Images in AGD20K sometimes contain multiple objects, which can result in several plausible regions corresponding to a single affordance. For example, in the left image, the "hold" affordance could refer either to the handle of the coffee machine or to the handle of the cup. Similarly, in the right image, "hold" could apply to either the chopsticks or the outer surface of the bowl. These cases highlight limitations in the ground truth annotations. To make more precise predictions, it may be necessary to include explicit object information in the prompt, such as “hold the cup” or “hold the bowl.” Our model is capable of accepting arbitrary natural language queries, making it well-suited for resolving such disambiguities effectively.
10 Limitations↩︎
As the Affogato-750K dataset is derived from the Objaverse 3D assets, our data do not contain the background information. Due to this limitation, we randomly synthesize background on the 2D images as shown in Table ¿tbl:tab:augment?, which is shown to be helpful when transferred to the real-world images. Note that our data engine can be extended to the indoor or outdoor scene data to tackle navigation environments, leaving them for future work.
11 Additional Qualitative Results↩︎
Qualitative comparison of zero-shot evaluation on AGD20K.
![Figure 13: Zero-shot evaluation comparison of Espresso-2D and others on AGD20K [25]](/article/2506.12009/figures/14_qual2d_suppl/2d_zero_shot_eval.png "fig:")
In Figure 13, we visualize qualitative results for 2D zero-shot affordance grounding. It illustrates common failure modes of existing methods [13], [65] that capture whole objects rather than precise part-level affordances. While Molmo+SAM provides rough part-level localization, our model trained on the multi-view aggregated Affogato-750K dataset achieves refined grounding capabilities.
Stage-wise qualitative results.
Figure 14 presents the intermediate qualitative results at each stage of our pipeline. Stage 1 takes multi-view images as input and predicts the object’s class name together with five candidate queries. Stage 2 leverages Molmo to point to the affordance region corresponding to each query. Stage 3 converts Molmo’s pointing locations into pixel-wise masks. These results highlight the robustness of our pipeline: even when Molmo produces incorrect pointings for some views or MobileSAM generates imprecise masks, the multi-view consensus voting effectively suppresses such errors, yielding an accurate final output.