1 Introduction↩︎

Affordance grounding [1] refers to finding potential “action possibilities” regions of an object, which plays a key role in bridging the gap between visual perception and robotic action. Recently, attempts made to endow models to have such grounding abilities can be broadly divided into two ways. The first type of methods is weakly supervised affordance grounding [2]–[4], which aims to identify affordance regions on objects using human-object interaction images and egocentric object images without dense labels. The affordance maps are obtained via class activation mapping (CAM) [5] or an auxiliary prior model [6]. However, these methods require a complex training framework design since there are two branches (exocentric and egocentric) that need to be balanced and optimized during training. Moreover, the generalization performance of these models is suboptimal, as their supported classes are fixed (e.g, LOCATE [2]) or very few parameters are optimized on the affordance task (e.g., PLSP [3]). To tackle this problem, the second type of methods uses affordance maps to directly supervise the models [7], [8]. However, some important affordance components trained from scratch (e.g., decoder) on only hundreds of pieces of manually labeled training data make it insufficient to obtain a highly generalized model. Thus, it is important for the fully supervised affordance grounding family to find a strong foundation model that is naturally suitable for the affordance grounding task and scale up the supervised data to obtain the ideal generalized model.

In this work, we try to decompose and address this problem in a step-by-step manner:

(i) Incorporating a suitable and generalized vision foundation model. With the recent advancements in large-scale pre-trained foundation models, many works have attempted to leverage the prior knowledge for downstream task transfer [9], [10]. We believe this paradigm is also effective for the affordance grounding task. Given that affordance grounding inherently requires pixel-level localization, we explore the possibility of incorporating SAM (Segment Anything Model) [6], [11] since SAM not only demonstrates remarkable generalization capabilities but also excels in precise object localization after pre-trained on large-scale masked-labeled data. Moreover, the model structure and decoder output format of segmentation and affordance grounding are highly similar. Considering these similarities between segmentation and affordance grounding, we hypothesize that incorporating SAM could be well-suited for our problem.

Although there are some similarities between segmentation and affordance grounding, task differences still exist. As shown in Figure 2, firstly, affordance grounding requires the recognition of functional parts of objects based on verb queries, while segmentation requires the separation of objects; the ground-truth of segmentation and affordance also differs slightly: the former uses a binary mask, while the latter uses a heatmap that encodes functional possibilities.

(ii) Effectively transferring the capabilities of SAM through data and model structure aspects. As mentioned above, there exist some differences of these two tasks, to fully unlock SAM’s capacity for affordance grounding, we first design an affordance-adaption module that utilizes learnable queries to interact with text and image features. These queries are then sent into the decoder to refine the original mask produced by SAM. We hope that this prompt learning-like module can effectively search for affordance-relevant parts in the comprehensive knowledge of SAM. We then scale up the fully supervised data in a coarse-to-fine annotated manner, which we named C2F-Aff. The dataset consists of three parts with each corresponding to a specific label format, and thus the training process is divided into three stages. In the first data part and training stage, we curate a mask-labeled dataset by merging and reclassifying existing datasets, focusing on object-affordance pairings. This enables the model to learn basic object-verb relationships. After that, we use a weakly supervised model to generate pseudo-labels for unannotated images in AGD20K [12] and post-process them to obtain annotations with higher quality to finetune the model. Finally, we use the high-quality, human-labeled data in AGD20K for further fine-tuning the model to boost the performance.

In our comprehensive experiments, we show that AffordanceSAM and C2F-Aff data can severally serve as an excellent base model and training data, which is evidenced by the leading performance of AffordanceSAM against other methods using the same amount of data and the performance enhancement of a wide range of fully supervised methods once trained on our C2F-Aff data.

In summary, our contributions are as follows:

• Showing that SAM is natural fit for affordance grounding with analysis and proposed AffordanceSAM.

• Introducing C2F-Aff dataset and a three-stage training scheme that boosts the performance of both AffordanceSAM and prior fully-supervised methods.

• Achieving SOTA performance on the AGD20K benchmark, as well as showing the evidence to generalize to novel actions and objects.

2 Related Work↩︎

We discuss the most relevant studies here and provide a more discussion in Appendix 1.

Affordance Grounding. Unlike detection [13] and segmentation [14] that yield bounding boxes or binary masks of the desired objects, affordance grounding [1] needs a model to output the functional possibilities in a object. Early works [15]–[17] have tried to equip the model with such capacity. But because of the data restriction at that time, these models can only recognize a few types of objects and actions. Next, Luo et al. collect the first large-scale affordance dataset called AGD20K [12] and annotated a few of the data with affordance maps (1675/23816). After that, many works engined by AGD20K have emerged. For example, LOCATE [2] localizes and extracts affordance-specific information in exocentric and transfers this knowledge to egocentric in a weakly supervised manner; AffordanceLLM [7] builds a vision-language learning framework based on LLaVA [18] to conduct the open affordance learning. PLSP [3] uses an additional semantic prior (e.g., SAM [6])³ to guide the traditional two-branch weakly supervised training process. However, none of the above works show satisfactory generalization ability to recognize regions for unseen objects and affordance actions (reasons are explained in the Introduction, and evidence is shown in our experiment results). In this study, we aim to find a foundation model that is naturally suitable for the task and scale up the annotated data to obtain such an ideal model.

Foundation Model for Downstream Tasks. Vision foundation models (e.g., SAM [6], CLIP [20], DINOv2 [21]) are large-sized models trained over vast amounts of data . After training, they are suitable as a starting point for a variety of downstream tasks under limited data conditions. Recently, many works have used different foundation models for different downstream tasks [9], [10], [22], [23]. For example, ActionCLIP [9] proposes a “pre-train, prompt and fine-tune” strategy to transfer CLIP to action recognition; FM-FSOD [10] uses DINOv2 to extract fine-grained features for few-shot object detection. Our work also shares similarities, while we focus on adapting SAM to the affordance grounding task with proposed affordance-adaption module and curated data.

3 Approach↩︎

The overall model architecture and training procedure of AffordanceSAM are shown in Figure 3 (a) and Figure 3 (b). As our model consists of two primary components: the EVF-SAM [19]⁴ baseline and the affordance-adaption module, we begin with describing each, and finaly, we provide a comprehensive elucidation of our coarse-to-fine affordance (C2F-Aff) dataset and training recipe.

3.1 Preliminary↩︎

Baseline EVF-SAM. As we need to provide text prompts that contain affordance queries to the model but the original SAM lacks language understanding abilities. We choose to start from EVF-SAM [19] which is the SOTA method among attempts [19], [24]–[26] that empowered SAM to follow text instructions. We seek to leverage its robust segmentation capabilities for affordance grounding.

The EVF-SAM original framework incorporates four key components: a multimodal encoder (BEIT-3 [27]) denoted as \(\mathcal{E}_M\), a prompt encoder \(\mathcal{E}_P\), a SAM image encoder \(\mathcal{E}_I\), and a SAM mask decoder \(\mathcal{D}\). The input image is firstly processed through two parallel paths: (i): conversion to SAM image tokens \(\mathbf{I}_s \in \mathbb{R}^{B \times N_s \times D_s}\), and (ii): transformation to multimodal image tokens \(\mathbf{I}_m \in \mathbb{R}^{B \times N_m \times D_m}\), where \(B\) represents batch size, \(N\) sequence length, and \(D\) feature dimension. Meanwhile, the text input is first tokenized to produce text tokens \(\mathbf{T} \in \mathbb{R}^{B \times N_t \times D_m}\). These text and multimodal image tokens are then combined with a learnable [CLS] token \(\in \mathbb{R}^{B \times 1 \times D_m}\) and processed by the multimodal encoder: \[\mathbf{F}_m = \mathcal{E}_M([\texttt{[CLS]}; \mathbf{I}_m; \mathbf{T}]) \in \mathbb{R}^{B \times (1 + N_m + N_t) \times D_m}.\] The [CLS] token output \(\mathbf{F}_c\) which is split from \(\mathbf{F}_m\) is transformed by the prompt encoder, then together with the SAM encoded image features to generate binary mask \(\mathbf{M}_b\): \[\mathbf{M}_b = \mathcal{D}(\mathcal{E}_I(\mathbf{I}_s), \mathcal{E}_P(\mathbf{F}_c)).\]

3.2 Affordance-Adaption Module↩︎

As shown in Figure 2, there exists a substantial disparity between the output of the segmentation task and the affordance grounding task. To efficiently adapt SAM to the dataset labeled with affordance maps instead of its original output masks, we introduce this module. Similar to BLIP-2 [28] and DETR [29], we introduce a set of learnable queries termed affordance queries. These queries, denoted as \(\mathbf{Q}_a\in {N_s \times D_m}\), first repeat over batch dimension and conduct cross-attention with the text features that interact with the image in the multimodal encoder. we expect these queries to extract information concerning the affordance action from these semantic features. What’s more, since different layers of SAM’s features often exhibit different levels of granularity and knowledge [30], and affordance may correspond to multiple parts of an object. We consider a diverse set of granularities can be beneficial. Therefore, the second cross-attention operation is performed between the processed affordance queries and fused visual features of SAM’s multiple layers formulated as: \[\mathbf{F}v = \sum_{i=1}^{j} \alpha_i \cdot Linear(\mathbf{G}_i), \quad \alpha_1 + \ldots + \alpha_j = 1,\] where \(\mathbf{G}_i\in \mathbb{R}^{B \times N_s \times D_s}\) denotes features from the global-attention blocks in SAM image encoder, \(\alpha_i\) is a learnable parameter that controls the fusion ratio of each feature. It is worth noting that we also add one self-attention block after each cross-attention block for enhanced feature learning. The final affordance queries \(\mathbf{Q}_{af}\) are first processed by two transposed convolution functions to adjust the dimension, then added with mask features from the original EVF-SAM to obtain the modified features for final process. Thus, we can obtain the affordance map output \(\mathbf{M}_a\) as: \[\mathbf{M}_a = \mathcal{D}(\mathcal{E}_I(\mathbf{I}_s), \mathcal{E}_P(\mathbf{F}_c), TransConv(\mathbf{Q}_{af})).\]

3.3 Coarse-to-Fine Affordance Dataset and Training↩︎

To fully leverage SAM’s generalization capacity for affordance grounding, we craft a coarse-to-fine training dataset (C2F-Aff) and training recipe shown in Figure 3 (b). The brief summary of C2F-Aff is displayed in Table 1, and more information on our data collection, division, organization and cleaning can be found in Appendix 2. We divide the data and training into three parts based on the form of the labels, while we always use a template of “<affordance action> <object_name>”, for instance, “wear hat”, as the text prompt for AffordanceSAM.

Dataset Part and Training Stage 1. In most training datasets for multimodal models and text-prompted segmentation models, such as LAION-400M [31], RefCLEF [32], and RefCOCO [33], there are very few objects with affordance properties and very little textual content that contains verbs. However, when transferring a foundation model to affordance grounding task in a fully supervised manner (e.g, DINOv2 in OOAL, LLaVA in AffordanceLLM, and SAM in our method), it is crucial for them to familiarize themselves with such objects and understand the affordance verb information for locating potential “action-possibility” regions. Thus, In the first part of our C2F-Aff dataset, we select three datasets, PADv2 [34], Handal [35], and RGB-D Part Affordance [36], which satisfy both objects contained with affordance actions and annotations of masks. We then combine them according to the categories of objects and affordance actions (details are in Appendix 2). And for We only use EVF-SAM baseline without the affordance-adaption module and fine-tune multimodal encoder and prompt encoder in this stage. The training loss function is given by \[\mathcal{L} = \lambda_{dice} \cdot \mathcal{L}_{dice} + \lambda_{bce} \cdot \mathcal{L}_{bce},\] where the overall loss \(\mathcal{L}\) is the weighted sum of DICE loss \(\mathcal{L}_{dice}\) [37] and BCE loss \(\mathcal{L}_{bce}\) [38], determined by \(\lambda_{dice}\) and \(\lambda_{bce}\).

Dataset Part and Training Stage 2. As mentioned in the Introduction, images with dense pixel labeling affordance maps are scarce and it can be costly for us to annotate. In practice, we find that training only on the hundreds of labeled images could not fully unlock SAM’s capacity. Hence, we utilize a weakly supervised model on AGD20K, LOCATE [2], to label the vast majority of the remaining unlabeled images in AGD20K. However, The raw output affordance heatmaps of LOCATE are usually suboptimal, which exist a large area of low thermal regions unrelated to the target. Hence, we design a post-processing program showed in Algorithm 4 to mitigate such issue. These processed pseudo labels help bridge the gap between binary segmentation masks from part 1 and the detailed affordance heatmaps needed for the final output, serving as an intermediate step in our C2F-Aff dataset. During training, we add affordance-adaption module and keep only SAM image encoder frozen. we follow AffordanceLLM [7] to use binary focal loss [39] (\(\mathcal{L}_{focal}\)) and set the weight of positive examples to be 0.9 and that of negative ones to be 0.1, as there are often more negatives than positives in an affordance map.

Dataset Part and Training Stage 3. To achieve the final high-quality supervised fine-tuning for boosting performance, we use the high-quality human-labeled samples in AGD20K as the third part of our C2F-Aff dataset and use them to supervise the model. We aim at regulating the model to generate the most precise and fine-grained affordance maps. We keep the loss function and trainable modules unchanged compared to stage 2.

4 Experiments↩︎

4.1 Experimental Setup↩︎

Implementation Details. In stage1, we initialize our AffordanceSAM with EVF-SAM [19] and train for 13 epochs with base learning as 0.00002. In stage2, we use the weights obtained from the first stage and randomly initialize affordance-adaption module, we train the model with the same number of epochs and base learning as stage1. In stage3, we further train our model for 26 epochs with larger base learning as 0.00004. In all training stages, we employ an AdamW optimizer [40] with a cosine learning rate scheduler, and the total batchsize per iteration is 32 using 4 NVIDIA A100 (80GB) GPUs.

Evaluation. We strictly follow the settings in AffordanceLLM to have two splits (easy split and hard split) of AGD20K to test our model. The easy split is the original unseen split of AGD20K, which has a lot of similarities between the objects in the train and test sets. The hard split ensures that there is no overlap between the object categories in the train and test set (both in our stage 2 and stage 3). Meanwhile, we follow the metrics provided in AGD20K to evaluate our model, which is Kullback-Leibler Divergence (KLD) [41], Similarity (SIM) [42] and Normalized Scanpath Saliency (NSS) [43].

Methods for Comparison. Affordance grounding methods can be mainly summarized into two main categories: weakly supervised and fully supervised methods. We compare our approach against representative weakly supervised baselines (Cross-View-AG [12], LOCATE [2], R-Mamba [4], and PLSP [3]), and fully supervised baselines (AffordanceLLM [7] and OOAL [8]).

We also provide more details about our implementation including training hyperparameters and model configuration, details of each metric, and detailed descriptions of each baseline method for comparison in Appendix  3\(\sim\)​5.

4.2 Main Results↩︎

We present both quantitative and qualitative experimental results and analyses of our proposed method. Below, we summarize the key findings:

Effectiveness of AffordanceSAM Model and C2F-Aff Dataset. As illustrated in Table 2, the AffordanceSAM model demonstrates remarkable performance, achieving competitive results even without leveraging a larger dataset (C2F-Aff). This is particularly notable when compared to weakly supervised methods. Trained only the AGD20K-Fully, AffordanceSAM outperforms AffordanceLLM and outperforms the previous SOTA OOAL model in terms of KLD and NSS, across both the easy and hard splits. From a data perspective, the introduction of the C2F-Aff dataset has significantly enhanced the performance of all fully supervised methods in the challenging hard splits, and the improvement is most prominent for our model. It is worth noting that all the fully supervised methods trained on C2F-Aff dataset outperformed other weakly supervised candidates. This illustrates the potential of the fully supervised method when scaling the supervised data.

Generalized Performance when Combining the Model and Dataset. By combining the model with the rich annotations of C2F-Aff, AffordanceSAM achieves state-of-the-art (SOTA) performance across most metrics. Specifically, AffordanceSAM attains the highest SIM and NSS scores on the easy and hard splits, respectively. Furthermore, AffordanceSAM demonstrates exceptional generalization to novel objects and actions, as evidenced by qualitative results on both AGD20K images and internet images (Figure 5 and Figure 6). For instance, when handling the object “cup", AffordanceSAM demonstrates a superior understanding of affordance actions compared to other methods and generates more accurate and precise affordance maps according to different actions”hold" and “drink". These results substantiate that our AffordanceSAM can benefit so greatly from the foundation model SAM, as well as the proposed C2F-Aff dataset, thus showcasing leading affordance grounding performance towards novel objects and affordance actions.

4.3 Ablation Study↩︎

In this section, we conduct an extensive ablation study to reveal the contribution and design-space of each component. Unless otherwise specified, we report the results on hard split of AffordanceSAM using our default settings. Please refer to Appendix 6 to see the detailed illustration of the evaluation setup of each table.

Contribution of Coarse-to-Fine Dataset and Training Recipe. As depicted in Table [tbl:subtab:training95reciep], our proposed coarse-to-fine dataset (C2F-Aff) and training recipe demonstrate a clear performance enhancement. It is noteworthy that directly employing EVF-SAM to assess results can be highly unsatisfactory (see the first row), which is consistent with the analysis in Introduction and Figure 2 that most vision foundation models like SAM lack the capability to recognize affordance verbs and thereby overly segments objects. Encouragingly, after trained on our three consecutive part of C2F-Aff in a stepwise manner, it can be found that the affordance performance has been progressively enhanced. Moreover, as shown in the last row, directly combining all the data can lead to worse result, this may because directly mixing different quality of data and ground-truth format causes the performance degeneration, which is consistent with the observations in tuning LLMs to MLLMs  [18], [44], [45].

Design Choices of Affordance-Adaption Module. We start from EVF-SAM without any structural changes and gradually integrate our methods to analyze the effect of each proposed design in affordance-adaption module. The results in Tab [tbl:subtab:affcon] reveal that each element consistently delivers notable improvements. In particular, we notice that adding learnable queries benefits a lot, which is in line with findings in other works [23], [46], [47] that use prompt learning when adapting a foundation model to downstream tasks.

Effect of Post-Processing Program. As we display in the first two columns of Figure 7, the quality of affordance maps directly output by LOCATE is sometimes terrible, which would result in corruption of the model capability supervised on these maps (see Table [tbl:subtab:postprocess] when \(\gamma = 0\)). By contrast, after processing with an optimal filtration intensity of \(\gamma=0.45\) and a number of 3 (see Table [tbl:subtab:fli95num]), most of the irrelevant areas are eliminated, while areas related to the functional part are preserved (see the second two columns of Figure 7). This optional design choice also leads to a significant performance improvement across all metrics.

5 Conclusion↩︎

In this paper, we introduce AffordanceSAM, a new state-of-the-art (SOTA) affordance grounding model. Built upon EVF-SAM with our proposed affordance-adaption module and trained with our systematically curated C2F-Aff dataset in a coarse-to-fine training manner, AffordanceSAM demonstrates superior performance in affordance grounding tasks. The zero-shot affordance grounding evaluation on internet images further demonstrates the generalization capacity of AffordanceSAM. We believe AffordanceSAM can become the new baseline, and the C2F-Aff dataset can become an effective data resource for subsequent studies and offer timely insights into how to leverage and extend SAM-like foundation models to benefit more relevant vision tasks.

6 More Discussion on Related Work↩︎

Affordance Learning for Robotics. Recently, Numerous methodologies have been developed by researchers to extract and interpret affordance information to enable the robots to grasp in a complex, dynamic environments [48], [49]. Specifically, several studies [50]–[52] leverage affordance to establish correlations between objects, tasks, and manipulation strategies for robotic grasping. Other studies [53]–[55] focus on deriving affordance knowledge from resources that can be deployed on real robotic systems. In this study, we only focus on the visual affordance grounding task, and our main insight is taking advantage of SAM’s generalization ability to compensate for the deficiencies of the previous methods. But we also believe our proposed AffordanceSAM can greatly improve the robot’s ability to recognize and grasp objects and become a default option for robot deployment by other researchers.

Segment anything model and its extension. The Segment Anything Model (SAM) [6], [11] is an interactive segmentation framework that produces binary masks in response to various prompt types (points, boxes, and coarse masks). Trained on a comprehensive dataset, SAM exhibits robust generalization capacity for segmenting diverse objects. Building upon SAM, SAM-HQ [30] addresses the segmentation quality of SAM by introducing a set of hq-tokens; LISA [24] combines SAM with LLaVA [18] to enable a segmentation model with reasoning ability. Our work also starts from SAM, but we focus on converting SAM to be a generalized affordance grounding model with our proposed affordance-adaption module and C2F-Aff dataset and training recipe.

7 More Details about Our Datasets↩︎

Here we provide the detailed dataset setup for C2F-Aff training and AGD20K testing of AffordanceSAM. The statistics of each dataset are provided in Table 3. Just as shown in this table and Table 1 of the main paper, our C2F-Aff dataset is divided into three parts according to the label format, and we will introduce each part in sequence parts.

Part 1. In this part, we use PADv2 [34], Handal [35], and RGB-D Part Affordance [36]. All three datasets are composed of different object and affordance categories, and images of the same affordance and category are treated as one class. The ground-truth of three datasets is binary mask, but a slight different is that Handal and RGB-D Part Affordance label the part of the object according to the affordance action, and PADv2 only gives the mask of the whole object. Moreover, as Handal and RGB-D Part Affordance are obtained from continuous frames of videos, which means many of the images of the same object category are very similar. We randomly sampled 5 images for those that belong to a video clip to avoid a potential over-fitting problem. And for PADv2, we filter duplicate images and add the remaining images to our training data. Noticing that we do not split any object when training, because the output form of this stage is completely different from the output used in the final evaluation.

Part 2 and Part 3. In these two data parts, we use the labeled and unlabeled parts of AGD20K [12] and use LOCATE [2] to annotate the unlabeled samples. We have already discussed this point in detail in the main paper. Moreover, in order to test the model’s performance accurately and fairly, we strictly follow AffordanceLLM [7] to split AGD20K for training and testing our model. The easy split is the original unseen split of AGD20K, which has a lot of similarities between the objects in the train and test sets. The hard split ensures that there is no overlap between the object categories in the training and testing sets. This setting can help to reflect the model’s generalization ability when the similarity between the training data and the test data changes. More details can be found in AffordanceLLM’s paper and project.

8 More Details about Implementation↩︎

In this section, we provide specific training hyperparameters in each stage and model configuration of AffordanceSAM in Table 4 and Table 5, respectively. We use the checkpoint after the whole training in first two stages as the initialization of the next stage.

9 Details of Each Metrics for Evaluation↩︎

In this section, we explain the metrics (KLD [41], SIM [42], and NSS [43]) to evaluate models.

\(\bullet\) Kullback-Leibler Divergence (KLD) measures distribution difference between the predicted affordance map (\(M\)) and the ground truth (\(M'\)), which is \[\mathrm{KLD}\left ( M,M' \right )=\sum_{i}M'_{i}\log\left ( \epsilon + \frac{M'_{i}}{\epsilon+M_{i}} \right ), \label{eq:no20}\tag{1}\]

\(\bullet\) Similiary (SIM) is also called histogram intersection, which measures the intersection between the predicted affordance map (\(M\)) and the ground truth (\(M'\)). The final range is from 0 to 1. It is given by

\[\mathrm{SIM}\left ( M, M' \right )=\sum_{i}\min\left ( M_{i},M'_{i}\right ),\\\] where \(\sum_{i}M_{i}=\sum_{i}M'_{i}=1\).

\(\bullet\) Normalized Scanpath Saliency (NSS) measures the correspondence between the prediction map (\(M\)) and the ground truth (\(M'\)). It is given by

\[\mathrm{NSS}\left ( M,M' \right )=\frac{1}{N}\sum_{i}\hat{M}\times M'_{i}, \label{eq:no22}\tag{2}\]

where \(N=\sum_{i}M'_{i}\), \(\hat{M}=\frac{M-\mu\left ( M \right )}{\sigma\left ( M \right )}\). \(\mu\left ( M \right )\) and \(\sigma\left ( M \right )\) are the mean and standard deviation, respectively.

10 Methods for Comparison↩︎

Affordance grounding methods can be mainly summarized into two main categories: weakly supervised and fully supervised methods. For weakly supervised models, they do not train on explicit labels of the affordance map. Instead, they are trained on two views (egocentric and exocentric) of the same object. As for fully supervised models, they are supervised under dense labels. In what follows, we explain the main idea of the baseline methods that we used for the evaluation and comparison.

Cross-View-AG [12] proposes a novel framework to extract invariant affordance from exocentric interactions and transfer it to egocentric views. This includes an Affordance Invariance Mining module to minimize intra-class differences in exocentric images and an Affordance Co-relation Preserving strategy to align correlation matrices between views for affordance perception and localization.

LOCATE [2] first localizes where the interaction happens and identifies interaction regions in exocentric views, then uses a PartSelect module to extract affordance-specific information, and transfers this knowledge to egocentric views for affordance grounding using only image-level labels.

R-Mamba [4] first extracts feature embeddings from exocentric and egocentric images to construct the hypergraphs. Then, a Hypergraph-guided State Space (HSS) block is introduced to reorganize these local relationships from a global perspective to locate the affordance regions.

PLSP [3] introduces a label refining stage, a fine-grained feature alignment process, and a lightweight reasoning module to boost the performance of weakly supervised affordance learning.

AffordanceLLM [7] leverages LLaVA [18] with a new mask token similar to LISA [24] and introduce depth maps as 3D information to build the main pipeline. A light-weight mask decoder trained from scratch is also introduced to produce an affordance map.

OOAL [8] proposes a vision-language framework that includes CLIP [20] and DINOv2 [21]. To boost the alignment between visual features of DINOv2 and affordance text embeddings extracted by CLIP, several light-weight modules like learnable text prompt tokens and a CLS-guided transformer decoder are introduced.

11 More Details about Ablation Setup↩︎

In this section, we have a detailed illustration of the evaluation setup of each table in our ablation study. The table index here refers to the index in the ablation part of the main paper.

Table 3. This table presents the effect of our proposed C2F-Aff dataset and training recipe. The first line of results is obtained by directly using EVF-SAM’s checkpoint to evaluate. Then we gradually add the training data, divided by the dataset part, and use the last checkpoint to evaluate. When combining all the data, we add the affordance-adaption module at the beginning of the training, which is slightly different from our default setting.

Table 4. This table presents the effect of each component in our proposed affordance-adaption module. The learnable queries in the second row do not conduct any cross-attention with the mentioned features but conduct self-attention twice, as we mentioned in Section 3.2 in the main paper.

Table 5 and Table 6. These two tables present the effect of the design choise of our proposed post-processing program. In the first row of Table 5, we do not use the 5\(\sim\)​6 steps in Algorithm 1. And so on for the second and third rows. Noticing that we only vary five values of \(\gamma\) results in Table 6, we believe more optional values of \(\gamma\) can be further searched; however, given the constraints of time and computational resources, as well as the diminishing returns on additional searches, we have decided not to pursue further searches. This decision does not affect the significance of our contribution.

12 Failure Cases and Feature Work↩︎

Although our AffordanceSAM achieves state-of-the-art (SOTA) performance under the AGD20K benchmark, and generalizes well beyond the training data, when faced with more complex scenarios (e.g., multi-object affordance), it sometimes cannot get accurate results. As shown in Figure 8, first, we find AffordanceSAM fails on completely separating two different functional regions of an object when prompted with multi-actions (see the first row of Figure 8). Next, we find that AffordanceSAM can not get an accurate affordance map when two different objects in a picture but are prompted with the same affordance action (see the second row of Figure 8). These limitations may be caused by the fact that every image we use to train AffordanceSAM is a single object format with a single affordance region involved.

Thus, endowing AffordanceSAM with the capability to deal with more complex scenarios like multiple objects or multiple affordance actions within a single image might be an exciting avenue for future research and development.

References↩︎