1 Background & Summary↩︎

Horticultural robotics is fast becoming an important part of society through sustainable farming. With the ever-increasing need for food production as the population grows, the reduction in arable farmland, and an aging workforce [1], [2], robotics is quickly becoming an accurate and efficient solution to address these issues [2], [3]. A stymying factor in the uptake of robotics and robotic vision in industry is the paucity of available data to train machine learning based algorithms for various tasks, such as monitoring [4], planning [5], or harvesting [6]. To address the issue of data paucity, we recently proposed an approach to obtain pseudo-labels for video data by using a neural radiance-field approach [7] and applied this to sweet pepper grown in glasshouses to produce the BUP-ST20 [8] dataset. In this manuscript, we expand beyond sweet pepper and include tomato which is a key agricultural crop. We do this by releasing two novel tomato datasets: BUTom21 and BUTom-ST21.

Tomatoes are an important crop in today’s broader social context. In Europe alone, the tomato harvest in 2024 was estimated to be 16.9 million tonnes of produce [9], which was a considerable increase of 5.7% from 2023. Currently, tomatoes make up almost 20% of all vegetable crops grown globally [10], making it one of the most commonly grown vegetables in the world. Due to a cultural shift in farming and the increase in farming costs Europe imports approximately €5 billion of tomatoes every year [10]. For farming these fruit, the labor expense can be prohibitive and robotic platforms are currently being developed to reduce costs and improve efficiency of farming procedures.

Robotics-based technological advances are evident in a number of fields, not just agriculture and horticulture. Data-driven agricultural robotics approaches rely heavily on the efficacy of their underlying models. In general, models and techniques are developed for a specific problem in a specific setting. In [11] it was shown that models can be adapted to both horticulture and agriculture with only a change in input data (arable farmland versus glasshouses). However, to be able to achieve accurate and sustainable robotic vision approaches extensive data is required. When we consider foundational models such as SAM3 [12] we see that, given enough generalized data, instance-based semantic segmentation is possible. The problem lies in the available data, particularly in agriculture and horticulture.

To date, most vision-based agricultural datasets can be considered small when compared to those such as ImageNet [13] or COCO [14]. This creates significant gap between supply and demand in the robotic vision and artificial intelligence fields in agriculture. In order to bridge this widening gap more accurate and robust data is required, particularly for instance-based semantic segmentation tasks. Instance-based semantic segmentation is an important component within agriculture as it provides high level information that enables a number of other important tasks. It not only infers where each object (instance) is in an image but also all the pixels associated with each object. This enables downstream tasks which would not be possible without object-level pixel-wise masks and the fruits or vegetables respective ripeness estimation.

Currently, in agriculture and horticulture, the datasets for instance-based semantic segmentation are small, containing as few as 82 images in the training set [15]. Recently, datasets such as BUP-ST20 [8] and APPLE MOTS [16] incorporate more frames in their training set, allowing more generalized learning; these two datasets were developed for spatial-temporal tasks and not specific to still image segmentation. However, most common horticultural datasets are aimed at crops such as sweet peppers [4], [8], oranges [17], berries and peaches [18], and apples [19]. This is in direct contrast to our earlier statistic that tomatoes are one of the most commonly produced vegetable crop in the world.

Recently, there has been a push to include more tomato-based datasets. In 2020 [20] released tomatoOD, a detection-based tomato dataset with 277 images in the training set and 55 images in the evaluation set. Unfortunately, from a robotics perspective, this dataset removed tomatoes that were occluded more than 50% of their appearance or if the fruit was too small or blurred. Also in 2020, Rob2Pheno [15] was released and captured in a glasshouse on RealSense D435 cameras. This is a small dataset, containing only 123 images in total (82 train/ 41 evaluation) and only the tomatoes in the foreground were annotated. The main benefit of this dataset is that they released instance-based semantic segmentation labels for all foreground tomatoes in the scene. With 804 images (643 train/ 161 evaluation) in its corpus Laboro Tomato [21] is one of the biggest tomato datasets aimed at segmentation and detection. It is captured on two different resolution cameras (\(3024\times{4032}\), \(3120\times{4160}\)) which adds further complexity to an already complex task. Their dataset also has the benefit of capturing multiple cultivar of tomatoes, both normal sizes and cherry tomatoes; and labels the subclasses into six different classes. In general, this dataset is not considerably complex in terms of the images, the tomatoes are assured to be the focus of the image and the full plant/vine/truss is not always captured in the image. Finally, in 2026 Zhang et al. [22] released both TomatoMAP-Det and TomatoMAP-Seg, as subsets of TomatoMAP. TomatoMAP-Det is a detection-based dataset with approximately 64k images. The segmentation dataset (TomatoMAP-Seg) contains 3,616 images, making it the largest known tomato dataset for segmentation purposes. This dataset also comes with 10 subclasses ranging from flower stage through to the fully ripe stage. In their approach, they leveraged SAM2 [23] to get the pseudo-labels of their dataset and then hand-corrected the ground truth, creating a known quantity to evaluate on. In general, tomato datasets are relatively sparse in the research community; we help to address this gap by introducing our two novel tomato datasets.

For sweet pepper, a well-represented fruit in research, and another highly produced fruit in agriculture, there are a number of well-known instance-based semantic segmentation datasets. Early datasets such as [4], [24] are small in stature but contain pixelwise labels for all fruit in a glasshouse. In [24] the images are captured on a hand-held camera scanning rows in a glasshouse, while [4] uses the robotic platform PATHoBot to capture all rows in a glasshouse cell. The illumination in these datasets, and many robotic vision datasets for agriculture is another concern. They are heavily impacted by the time of the day of capture (the position of the sun), the inclusion of the sun shield over the glasshouse, and the direction of the robot. To alleviate some of these concerns the WUR dataset [25] exploits artificial light to maintain more consistency within the dataset. However, unless the robotic platform of capture consistently run at night with artificial lighting this is less generalized to the horticultural domain. Moving away from sweet pepper, one of the largest instance-based semantic segmentation datasets for fruit is MegaFruits [18]. MegaFruits contains over 25k images of strawberries, blueberries, and peaches fully annotated with pixel-wise masks. In this dataset, the concentrate on the important task of occlusion and densely clustered fruit. Another large fruit-based dataset is MinneApple [19], which contains approximately 1k images with over 41k pixel-wise masks. In their dataset they again concentrate on occlusion of apples as they are generally densely clustered. Overall, these datasets show a recent push to include large amounts of fruit and vegetable data for the research community. However, these concentrate on still image information, meaning spatial-temporal models can not be trained as consistent tracklet IDs are not included.

For research on video instance segmentation (VIS) and multi-object tracking (MOT) very limited data is available in the agriculture and horticulture domains. We recently released BUP-ST20 [8] which is applied to sweet pepper and leverages the power of pseudo-labels generated through a neural radiance-field approach. As far as we are aware, this is still the largest dataset available that was captured on a robotic platform, contains instance-based semantic segmentation, robot odometry, depth images, and color (RGB) images in its corpus. Likewise, APPLE MOTS [16] was released to the public in 2022. Similarly to sweet peppers and tomatoes, it is based on the principle that common tracking datasets contain objects (such as pedestrians dressed in different clothing) which are somewhat easier to disambiguate. For horticulture, fruit is homogeneous in nature, generally speaking red tomatoes are harder to tell apart and re-identify at a later date. While, the different subclasses can be told apart from each other (orange versus green versus red) the same subclass can be much more difficult to distinguish.

To help alleviate these gaps, homogeneous VIS and MOT datasets and dataset paucity, we introduce the BUTom21 and BUTom-ST21 datasets. We follow the same protocols as our previous datasets, BUP20 and BUP-ST20, using similar techniques to deliver the final datasets. To download the data and see the minimal working code for running the dataloaders please see the github repostiory https://github.com/Agricultural-Robotics-Bonn/BUTom21-ST21. The datasets capture the tomato cultivar sweeterno from early stages of ripeness after full flowering, through to harvestable fruit. The two datasets are developed for different purposes and with different challenges and limitations; however, the main purpose revolves around instance-based semantic segmentation.

In more detail, the BUTom21 dataset is a fully hand-annotated still image dataset of tomatoes. It contains 123, 72, 98 images respectively in the training, validation, and evaluation subsets, and all annotations are supplied in a Coco format based json file. The dataset contains RGB and noisy depth images captured across 2 capture days and two different RealSense D435i cameras. In contrast to other datasets, and similar to BUP20, we annotate all tomatoes in the scene, not just those in the foreground. This creates a fully pixel-wise annotated dataset for tomato detection and segmentation in a real-world robotics scenario.

The BUTom-ST21 alleviates the data paucity issue within the horticulture and tomato domain. In total we have 7749, and 4536 pseudo-labeled images across the training and validation set, and 1386 images hand-labeled in the evaluation set. To create the dataset we follow a similar design paradigm to the BUP-ST20 dataset. The inclusion of tracklet IDs enables VIS and MOT evaluation in a highly dynamic horticultural scene with homogeneous fruits. A key difference from the BUP-ST20 dataset was the lack of robot odometry information. In BUTom-ST21 we did not have the benefit of a working odometry and therefore had to revert to traditional approaches to provide camera pose information. This paradigm shows that spatial-temporal datasets can be created when there is smooth movement of the camera(s).

Our two released novel datasets, BUTom21 and BUTom-ST21 are freely available for resarch purposes. We have separated these two datasets into two explicit locations to facilitate easier downloading and task specific usage. The BUTom21 dataset is a fully hand-labeled tomato dataset with subclass assignment. The training and validation subsets from BUTom-ST21 are pseudo-labeled using the BUTom21 dataset as the training information, and a fully hand-corrected evaluation subset. Overall, we created two complex novel datasets for various real-world agricultural robotic vision purposes. In this manuscript we:

1. Release a fully hand-annoated still image tomato dataset;

2. Release a spatial-temporal tomato dataset with pseudo-labeled training and validation subsets and hand-corrected evaluation subset; and

3. Provide extensive benchmarking of the two datasets for various tasks (outlining the challenges and limitations of each).

2 Methods↩︎

For our two novel datasets, BUTom21 and BUTom-ST21, the following section describes the data collection procedure and the various annotation tasks undertaken to create both the still image dataset (BUTom21) and the spatial-temporal version (BUTom-ST21). The data capture happened at the University of Bonn’s commercial-like glasshouse setup at campus Klein-Altendorf. Multiple crops are grown in the multi-cell glasshouse and of interest for this dataset was the sweeterno tomatoes grown in one such cell alone (no other fruit or vegetables are mixed with the tomatoes). The glasshouse cell itself is arranged such that there are six rows, where the two outside rows have sterile backgrounds and each row is about 36m in length. To capture the data we employed the PATHoBot [4] robotic platform which exploited the piperails installed parallel to the cropping rows. The piperails and the use of PATHoBot ensured we have consistent distance from the cameras to the heating rails in the rows (approximately 1.2m). PATHoBot is designed with four RealSense D435i cameras mounted evenly spaced along a vertical pole, this ensures that the entire tomato vine can be captured in one scan. For these datasets we utilized the two middle cameras, the topmost camera was deployed too high and thus did not capture any tomatoes, and the lowest camera captured superfluous information such as potting equipment and the ground. The RealSense D435i capture both RGB and registered depth information at a resolution of \(1280\times720\) and at approximately 15Hz; further details on PATHoBot and the camera setup can be found in [4].

The sweeterno is a medium to large sized vegetable that ripens in five distinct stages: green, mixed red, red, mixed orange, and orange. To ensure tomato subclass (ripeness) diversity, we captured all six rows on two separate occasions, approximately one month apart. By utilizing the two middle cameras we further added variety to the subclasses as more ripe tomatoes (red) appear at the the bottom of the vine, and more juvenile (green) appear towards the top. This is in contrast to the sweet pepper datasets BUP20 and BUP-ST20 which only uses a single camera.

While our two publicly released datasets, BUPTom21 and BUTom-ST21, are derived from same images they are processed differently and released separately. BUTom21 is a fully hand-labeled dataset for still image tasks such as detection and segmentation. BUTom-ST21 is a much larger spatial-temporal dataset with pseudo-labeled training and validation subsets and fully hand-corrected evaluation subset.

The still image-based BUTom21 dataset contains RGB and depth images from the two cameras captured over the two days. We uniformly select and then manually refine the selected images to be annotated. Our manual refinement was undertaken to ensure dataset diversity as well as equal camera and capture day representation. Image selection was also done to ensure the dataset was diverse enough to fully encapsulate the various challenges witnessed during capture. One of the key challenges witnessed during capture was the inclusion of “tiny” objects. These “tiny” objects occur due to juvenile tomatoes and largely occluded tomatoes; occlusion can occur because of other tomatoes, leaves, stems, trusses, and other objects. Other challenges witnessed during data capture were camera blurring due to vibration, illumination variation, scene complexity (inner versus outer rows), and subclass distribution; each of these variations was represented explicitly in the images annotated in the dataset. To ensure fairness of results and hyper-parameter tuning, we define three subsets of fully annotated data. There is a total of 293 images in the dataset, with 123, 72, and 98 images respectively for the training, validation, and evaluation subsets. The images for each subset were also extracted from specific rows in the glasshouse cell. The training subset is derived from rows 1 and 2 on the first day and rows 1, 2, and 3 on the second day of capture. While validation used row 3 and 4 on the first day and then only row 4 on the last day, and evaluation is composed of rows 5 and 6 on both days; ensuring a complete hold out and non-overlapping evaluation set.

The annotation process used a single person per image to annotate all tomatoes in the scene (foreground and background) with consistency checks by another operator. Tomatoes are annotated with individual pixel-wise masks and their associated subclass (green, mixed red, red, mixed orange, and orange) and stored in a json file in the Coco format using the Coco annotation software [26]. In https://github.com/Agricultural-Robotics-Bonn/BUTom21-ST21 we include a minimum working example of how to extract the images in a PyTorch style dataloader. Table 1 describes the full breakdown of the BUTom21 dataset, including the occurrence of each subclass and their associated labels in the json file. Overall, BUTom21 provides a new tomato dataset which can be used for any task that requires instance-based semantic segmentation labels, and examples of the dataset are shown in Figure 1.

The spatial-temporal BUTom-ST21 dataset is derived from the same images as BUTom21, and relies heavily on the original still image dataset. It contains 123, 72, and 22 sequences, respectively, for the training, validation, and evaluation subsets; Table 1 breaks down the dataset. Similar to BUP-ST20 we exploit pseudo-labels in the training and validation subsets, and hand-corrected annotations in the evaluation set. In total we have 7749 and 4536 images with pseudo-labels (training and validation) and 1386 fully annotated images. As a point of difference to BUP-ST20, which has a random number of frames per sequence, we ensure that each sequence has a consistent number of frames which is 63. As this is a spatial-temporal dataset, it is designed to be used for VIS and MOT purposes and so each pixel-wise annotation also includes a tracklet identifier. This ensures (or weakly ensures for the training and validation subsets) that each object in the scene is tracked through the sequence and obtains a unique ID per sequence.

To obtain our pseudo-labels we use PAg-NeRF [7] which is a neural radiance field technique (NeRF). PAg-NeRF takes as input RGB images, corresponding depth images, instance masks generated from depth filtered versions of BUTom21, and camera pose information for each frame. To generate each sequence, first we position an image from BUTom21 in the middle of a sequence and then we extract a total of 83 frames around it (approximately 41 each side), ensuring no overlap between sequences. We extract 83 frames as PAg-NeRF needs “warm up” frames to visualize the scene correctly at our first actual starting frame. The actual images annotated in sequences (63 images) start from the tenth image of the extracted images, and ends at the 73rd image. The instance-based semantic segmentation has been extracted by two well known techniques: Mask2Former [27] and Yolo26 [28] to compare the performance of both approaches. In our released data we have included the pseudo-labels from both approaches.

As we are only concerned with tomatoes in the current row we are scanning, the first step before training our two techniques is depth filtering. Due to the manner in which tomatoes grow and the noise in the depth images, we select a depth of 1.4m from the camera (compared to 1.2m in [8]). We then take the annotated pixel-wise mask from BUTom21, and if at least 50% of the mask is contained within \(0\rightarrow{1400}\) from our depth map we keep the mask, otherwise we remove it from the training procedure.

Once we have filtered out the masks based on their depth information we can now train our instance-segmentation models, both Mask2Former and Yolo26 are trained on the depth filtered BUTom21 dataset. Mask2Former is fine-tuned with the Swin-L [29] backbone and initialized with Coco pretrained weights [14]. For training, we maintain the default configuration and only modify the base learning rate to be 0.00005. We train the model for 510 epochs (with the best model selected based on the validation set), with a total batch size of 32 on 8 NVIDIA A100 GPUs. For Yolo26 we fine-tune using all their in-built configuration parameters, and the Ultralytics (https://github.com/ultralytics/ultralytics) internal model yolo26x-seg.pt. The batch size is set to 16 and evenly split across 8 NVIDIA RTX A6000 GPUs. Images are processed at their original \(1280\times{720}\) size, and both techniques use the full subclass (multi-class) information during training. We are now able to run inference on the BUTom-ST21 sequences and produce all associated instance-based semantic segmentation masks. It should be noted that at this point all masks are processed independently, and there is no consistent tracklet ID information.

The final step before employing PAg-NeRF for our consistency rendering, is to complete the camera pose information. After capture it was noted that the odometry information from PATHoBot was not functioning correctly, and for some rows there was no odometry information. To address this issue and show how new datasets can be achieved without noisy odometry information (such as that used in [8]) we exploit structure-from-motion (SfM) techniques to compute the poses. We also modified PAg-NeRF slightly to ensure it could handle this information directly. For each sequence, we first run COLMAP [30], configured with the PINHOLE camera model and initialized with our known camera intrinsics to enhance the accuracy of the recovered poses. This approach was successful in all but 13 sequences. In the unsuccessful sequences we employed hierarchical localisation [31] (HLoc). This approach used SuperPoint [32] for feature extraction and SuperPoint+LightGlue [33] for matching; and was able to successfully register the remaining sequences. For dataset completeness we also release both the HLoc and COLMAP poses for their respective sequences.

To render the consistent tracklet IDs we employ PAg-NeRF, with all of our now completed sequence information. PAg-NeRF is trained using the default configuration parameters, except for changing the scene scale to 0.02 and the deactivation of the pruning operation during training. We found that in both cases, by changing these values, we achieved better rendering results. This approach achieves our first pass at pseudo-labels for all three subsets where we now have: RGB images, registered depth, instance-based semantic segmentation masks, consistent tracklet IDs, and camera poses. For the training and validation set we do further post-processing to achieve more robust pseudo-labels due to four main issues with the original PAg-NeRF output: horizontally joining two (or more) objects, vertically joining two objects, tracklet ID switching, and finally reproducing the same tracklet ID after the first one has left the scene. We found this post-processing approach improved AP results by a minimum factor of 4. However, this still leaves the problem of missing small objects, and poor segmentation masks, making BUTom-ST21 a challenging dataset on which to achieve strong results. Overall, while we have removed a number of issues, as seen in Figure 2 there are still considerable limitations with the pseudo-labels.

Finally, for the evaluation set, we take the pseudo-labels and hand-correct them. This is completed by human operators removing joined bounding boxes, creating consistent tracklet IDs, adding missing tomatoes, removing false positives, and fixing segmentation masks. A final check is done by a single operator to maintain consistency. This creates the largest pseudo-labeled and hand-labeled spatial-temporal dataset for VIS and MOT purposes using tomatoes.

3 Data Records↩︎

Our two novel datasets released for research purposes can be found at the github repository https://github.com/Agricultural-Robotics-Bonn/BUTom21-ST21 with download instructions. Each dataset is self contained and can be downloaded without the other. It should be noted that the BUTom21 dataset contains only the annotated RGB and depth images in png format from two cameras. The BUTom-ST21 dataset contains RGB and depth images for the respective sequences in the dataset along with their associated annotations.

As a still image dataset, BUTom21 is separated in the manner described in Figure 3. The main root directory contains the CKA_tomato_2021.yaml file which breaks down a number of statistics as well as the subclass labels. Most importantly, it contains the breakdown of subset identifiers in the Coco based json file. The CKA_tomato_2021.json file contains the image information and the Coco format annotations, including bounding box locations, Coco mask information (polygons), and subclass classification. In the json, for the image information, the relative path is included, which means no extra search is required to find the desired image. The image_camera_row_identity.yaml supplies extra information about the images, including which camera they came from and which row they were captured in. Finally, the BUTom21_structure.md contains the full description of the yaml and json files, and in particular, how to use the information contained within.

From a file structure standpoint we have organized the dataset in a similar fashion to other still image-based vegetable datasets. The depth and RGB images are organized such that they are located based on their capture date and the row they were obtained from. An example of this breakdown is provided in Figure 3.

For consistency, the spatial-temporal dataset, BUTom-ST21 contains a similar file structure. The Mask2Former and Yolo26 annotation pickles are supplied in their own directory for the training and validation subsets. For the evaluation labels, we supply them in a separate directory as this ensures that researchers can select between the appropriate pseudo-labels but still utilize the hand-corrected evaluation subset. A visualization of the file structure is provided in Figure 4.

In the train_valid_eval_splits.yaml we supply the sequences that belong to each subsection. The camera_parameters.yaml file provides the intrinsic parameters for both cameras used in the dataset. The image_camera_row_identity.yaml is similar to that of BUTom21 and supplies information about what camera and on which date the sequences where captured. This YAML file also specifies which BUTom21 image is located at the center of its associated sequence. Finally, the BUTomST21_structure.md file provides indepth information about each file and its structure.

Each BUTom-ST21 sequence consists of 83 frames, where the center 63 have been annotated. All pickle annotations (Mask2Former/Yolo26 pseudo-labels and the hand-corrected evaluation labels) are supplied for these 63 central frames. We release the complete 83 frame RGB, depth, and pose data (the full input for PAg-NeRF). This enables researchers to re-run PAg-NeRF or apply an alternative method. The annotated_seq_frame_list.yaml file lists the 63 annotated frames for each sequence, while pagnerf_seq_list.yaml displays the full 83-frame input sequences fed to PAg-NeRF.

The RGB and depth images are separated based on their sequence number and the various pickle annotations are similarly arranged. The pseudo-labels are split into Mask2Former or Yolo26 based labels for comparison of the different approaches. Each of these pseudo-label directories are those trained on BUTom21 using the various image-based semantic segmentation approaches, refined with PAg-NeRF, and post-processed by us. The hand-corrected evaluation labels are stored in the eval_annotations directory. These are only supplied for the 22 evaluation sequences.

All annotations are supplied as Python-based pickle files (.pkl extensions). Each pickle file stores the annotation of a single frame as a dictionary keyed by the tracklet identifier (track_id) of every tomato instance in that frame. The track_id provides temporal consistency: the same physical tomato maintains the same track_id throughout a sequence, enabling instances to be tracked over time by matching keys between consecutive pickles. This makes BUTom-ST21 suitable for both VIS and MOT. These identifiers are guaranteed to be consistent in the hand-corrected evaluation subset and weakly consistent in the pseudo-labeled training and validation subsets. Each track_id maps to a dictionary containing four fields: the bounding box in Coco format ([x, y, w, h]), the area (mask pixel count), the semantic label (ripeness subclass), and the instance mask (full-resolution binary mask).

4 Technical Validation↩︎

To evaluate the viability of our two novel datasets we analyze their performance using state-of-the-art techniques for a variety of tasks. Our primary tasks for the still image dataset include: instance-based semantic segmentation, bounding box regression, and cross-domain anomaly segmentation. For BUTom-ST21 we evaluate the performance of our pseudo-labels for instance-based semantic segmentation, and bounding box regression; both evaluated on BUTom21 for completeness of analysis. Finally, we evaluate video instance segmentation (VIS) and multi-object tracking (MOT) as these are the main focus of the dataset.

Analysis on BUTom21↩︎

To analyze the performance of our novel still image BUTom21 tomato dataset, we first evaluate the performance for instance-based semantic segmentation. Table 2 outlines these results. Here we evaluate Mask2Former and Yolo26 models, and train four models per approach: single-class, multi-class, depth filtered single-class and depth filtered multi-class (where depth filtering is completed in a similar manner to that described above). For Mask2Former we train for 650 epochs maximum, using a total batch size of 32 across 8 NVIDIA A100 GPUs. Yolo26 is trained for 100 epochs maximum at all internal configuration parameters and we use a batch size of 16 split evenly over 8 NVIDIA RTX A6000 GPUs. For both approaches, the best model is selected based on the validation subset.

Table 2 outlines the instance-based semantic segmentation of our approaches. We evaluate for mean average precision (mAP), average precision at 50% and 75% and the mAP of the different object sizes. In our results we also provide an overall summary on the performance of “Tiny” and “Small” objects by grouping them together in “AllSmall”. For multi-class evaluations, we find that Yolo26 outperforms Mask2Former, showcasing its ability to better distinguish between subclasses. For both single-class evaluations we either see commensurate performance (not depth filtered) or an improvement when using Mask2Former for depth filtered. This indicates that for depth filtered approaches Mask2Former is generally able to perceive tomatoes in the foreground better than Yolo26. However, if you wish to do multi-class segmentation, then, on this evidence, Yolo26 is a better solution.

For bounding box regression, see Table 3, we see a shift in performance. Yolo26 is considerably better at detecting objects in the scene. We attribute this to the way Yolo26 functions, prioritizing detection over segmentation. These results show, supported by further experiments, that Mask2Former should be chosen for better segmented masks; conversely, Yolo26 is the obvious choice for better detections.

Our next evaluation is based on anomaly semantic segmentation. To analyze our results, we utilize a broad set of metrics similar to [34], and follow their evaluation protocols. For binary classification, we outline the area under the receiver operating characteristic curve (AUROC) metric. We also include two ranking metrics: area under the precision-recall curve (AUPR), false positive rate at 95% true positive rate (FPR95). Among localization metrics, we employ the (foreground) segment-wise intersection over union (sIoU), positive prediction value (PPV), and mean F1 score (meanF1).

For our results assessing the performance of uncertainty estimation methods to cross-crop (out of distribution) OoD objects, we employ various state-of-the-art techniques. These results are displayed in Table 4. Our two baselines, Mask2Former (ADE20k Weights) and Mask2Former (BUP20 Weights) are displayed at the top of the table. For the model trained on BUP20 weights, we fine-tune a separate Mask2Former model to generate single-class (tomatoes) semantic segmentation masks. In our work, we consider tomatoes to be the anomalous objects. We benchmark four uncertainty qualification approaches: Snapshot ensemble [35], Monte Carlo dropout [36], single-trajectory LoRA ensemble as representatives of Bayesian approaches, and deep deterministic uncertainty [37] as a representative single-shot deterministic approach. Then, inspired by [38] ¹, we benchmark six post-hoc calibration techniques: temperature scaling [39], logistic scaling [40], Dirichlet scaling [41], Meta-cal [42], local temperature scaling [43], and selective scaling methods drawn from [44]. All ten approaches use Mask2Former as their base segmentation approach.

From these results, we show that semantic segmentation for OoD can be improved over the baseline Mask2Former BUP20 weights approach, but not consistently. In fact, from these results we show that Mask2Former when trained on BUP20 and cross-evaluated out-of-domain on BUTom21 actually performs admirably. We believe that this work shows the potential of the dataset to be used for future anomaly detection.

Analysis on BUTom-ST21↩︎

Our first BUTom-ST21 analysis is completed to show the overall performance of the dataset as a still image instance-based semantic segmentation and then detection dataset. First, the sequences are converted to a still image format (only mask or bounding boxes are considered). We once again train Mask2Former and Yolo26 based on this considerably larger dataset, where we use the same hyper-parameter as the BUTom21 still image evaluation, however, we only train for 50 epochs for both approaches. The best model for all is chosen based on the spatial-temporal validation set, and final evaluations are performed on the BUTom21 (still image) evaluation subset. Due to the manner in which BUTom-ST21 is trained, we only consider the depth filtered BUTom21 evaluation set as we cannot train a BUTom-ST21 not depth filtered model. This allows us to directly compare BUTom21 to BUTom-ST21, consistent with our prior work [8].

Ultimately, the challenges associated with psuedo-labels are shown in Tables 5-6, when compared to Tables 2-3. There is between 5% and 24% drop in performance from BUTom21 to BUTom-ST21 for segmentation-based mAP. For detection (bounding boxes) there is a degradation in performance from 6% to 13%. These challenging results are caused by the poor segmentation outputs (see Figure 2) from both models. Overall, this makes BUTom-ST21 an interesting dataset for instance-based semantic segmenation, and perhaps uncertainty estimation could assist in assigning better segmentation masks.

Our next experiments use BUTom-ST21 for video instance segmentation (VIS). The VIS task requires evaluating both temporal consistency (tracklet IDs) and segmentation quality. To achieve this, we employ the state-of-the-art technique CTVIS [45] fine-tuning it with pretrained Coco weights on the BUTom-ST21 training set and validate it on the validation set. We use the Swin-L backbone with default CTVIS configuration, except for the learning rate: 0.0001 for multi-class and 0.00005 for single-class. The model is trained for a maximum of 350 epochs when using Mask2Former pseudo-labels and 600 epochs for the Yolo26 variant. We split a batch of 8 across 8 Nvidia A100 GPUs, with an image resolution downsampled to \(960\times{540}\). Results are provided in Table 7 where we report mAP, AP50 and AP75 across object sizes. Unlike the still-image AP variants, we use video-based average precision instead of image-based average precision.

In all cases, the Mask2Former labels provide better segmentation results across their respective sequences (video AP). This is consistent with the results in Table 5 where Mask2Former produced better segmentation quality. Additionally, consistent with expectations, the single-class approach performs better than the multi-class approach. We attribute this to the accuracy of the model in classifying the subclass information. Of particular interest to the research community, performance for “tiny” objects is very low, and we achieve a performance of 0.0 regardless of the model. This once again highlights the challenge and noise in the pseudo-labels, where “tiny” objects become unrecoverable.

Our final benchmarking approach on BUTom-ST21 is for multi-object tracking (MOT). As with all benchmarks we do not perform an exhaustive hyper-parameter sweep; instead, we use the results from BUP-ST21 as our starting point. We acknowledge that a more thorough hyper-parameter sweep could improve performance across all methods by a few absolute points. We compare learned models (CTVIS, PAg-NeRF) against methods without an inherent learning pipeline (ByteTrack [46], Halstead [47], and OCSort [48]). For the latter approaches, we use the depth filtered outputs from the BUTom21 model and run inference on all sequences in the evaluation subset.

To evaluate the performance of MOT on this dataset, we explore higher-order tracking accuracy (HOTA) for tracking association and object detection; multi-object tracking accuracy (MOTA) to evaluate false positives, false negatives, and ID switching; and IDF1 for frame-by-frame accuracy. We also show results for ID switching, false positives, and false negatives.

In Table 8, we see the best MOT performance in the learned approaches. These approaches benefit from learning all information in their training paradigm. CTVIS using Yolo26 labels is able to achieve the best HOTA, MOTA, IDF1, and false negatives, while PAg-NeRF using Mask2Former outputs achieves the best ID switches and false positives. Among standard (unlearned) approaches, we see a variation in best performance, except for the fact that in all cases Yolo26 performs the best. We performed a further analysis of this anomaly and found that Mask2Former produced considerable overlapping outputs creating additional challenges in an already difficult task. From our analysis, we found that by performing mask-based non-maximal supression (NMS) we were able to improve the ByteTrack Mask2Former performance to 76.6 (HOTA), 55.3 (MOTA), 73.8 (IDF1), 471 (ID switches), 7627 (false positives), and 11533 (false negatives). This is a considerable boost in performance and shows one of the limitations associated with instance-based semantic segmentation in Mask2Former.

For single-class evaluation (Table 9) we exclude PAg-NeRF as it was trained only on multi-class outputs. Once again, we see the learned re-identification strength of CTVIS outperforms all other approaches, though not by as considerable a margin as that witnessed in Table 8. The performance of Mask2Former outputs is much more inline with its Yolo26 counterpart, with Yolo26 matching or exceeding Mask2Former.

Overall, we have shown how our two BUTom* datasets can be used and established a number of benchmark values across multiple computer vision tasks in agriculture. The challenges of the two datasets are clearly evident, particularly for “tiny” objects.

5 Usage Notes (optional)↩︎

To aid with reproducibility, both BUTom* datasets have three uniquely defined subsets: training, validation, and evaluation. One of the key differences this paper has over the BUP-ST21 is the number of small and tiny objects. Table 1 has a summary of these statistics. To separate the size of objects we split them into the following groups: tiny: 0-100 (10\(\times\)​10) pixels; small: 101-3,025 (55\(\times\)​55) pixels; medium: 3,026-27,556 (166\(\times\)​166) pixels; and large: anything larger than 27,556 pixels. These object sizes are inspired by the Coco dataset [14] which we have refactored based on our image resolution. This conversion is required as Coco images are of the resolution \(640\times{480}\), while ours are \(1280\times{720}\).

To calculate the above object sizes we used the ratio of total pixels between the two datasets. We obtained a ratio of 1:3 as the image sizes were \(640\times{480}=307200\) pixels and ours \(1280\times{720}=921600\). Then we calculated object sizes based on this to again have similar total number of pixels \(\sqrt{32^2\times{3}} \sim 55^2\) pixels and \(\sqrt{96^2\times{3}} \sim 166^2\) pixels. Finally, we selected \(10^2\) as our lower limit to the tiny size because the Mask2Former approach downsamples the feature map 4 times, and at the lowest resolution 100 pixels becomes approximately 3 which we could consider as noise.

6 Data Availability↩︎

The two novel datasets are available for download by following the commands described in https://github.com/Agricultural-Robotics-Bonn/BUTom21-ST21. In this github repository contains the instructions for downloading both BUTom21 and BUTom-ST21. To ensure data storage viability, the two datasets will be hosted on BonnData (https://bonndata.uni-bonn.de/). These two datasets can be downloaded independently of each other, allowing researchers to decide which dataset best suits their needs or tasks. Figure 3-4 outlines the general structure of the datasets and what can be expected in each. The image format is supplied as png files, both for RGB and depth. For BUTom21 the annotations are supplied as a Coco format json, and for BUTom-ST21 we supply them as pickles per image. Each pickle contains the full information for each tracklet ID/object in the current scene, where tracklet IDs are consistent throughout the sequence. Also camera pose information per sequence is provided for BUTom-ST21.

7 Code Availability↩︎

We have supplied minimal working code. The code is available here: https://github.com/Agricultural-Robotics-Bonn/BUTom21-ST21 and includes minimalist PyTorch style dataloaders for each of the two BUTom* datasets. We also supply code to create Yolo26 style image directories and annotations from the Coco format and pickle format variants. No tracking or evaluation code has been supplied.

8 References↩︎

9 Author Contributions↩︎

Halstead contributed to the annotation, code and evaluation, and writing the manuscript. Guclu contributed to data preparation, annotation, code and evaluation, and writing the manuscript. Farag contributed to the experimental analysis and manuscript writing. Pallotta contributed to the annotation and manuscript. Hund contributed to annotation, and general organization. The remaining authors contributed to the writing of the manuscript and funding arrangements.

10 Competing Interests↩︎

None.

11 Acknowledgements (optional)↩︎

The authors gratefully acknowledge the access to the Marvin cluster of the University of Bonn. We the authors would also like to thank Yan Wang, Sina Raufi, Efe Incir, Patrick Zimmer, Nelson Pinheiro, Julian Rosbach, Rafay Aamir, Moein Taherkhani, Sicong Pan, and Gokul Krishna Gandhi Chenchani for their assistance in annotating the data.

12 Funding↩︎

This work has been funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) for KI-FOR 5351: MC 831/2-1 (459376902), RO 4839/6-1 (498564628), and GA 1927/9-1 (498577300).

13 Ethics statement*↩︎

References↩︎