1 Introduction↩︎

Bioacoustics is an interdisciplinary field that aims to understand the natural world through sound. Advanced machine learning (ML) techniques have been increasingly used in bioacoustics [1], particularly for species identification. ML and AI models help scale biodiversity monitoring by automating species identification and saving significant manual effort. Fine-grained differences in species sounds, limited data, and long-tailed distributions present unique challenges for AI research. Recent progress has been catalyzed by new datasets and evaluation benchmarks [2]–[4], giving rise to bioacoustic foundation models [5], [6].

Model development in this field is supported by large-scale global platforms such as Xeno-Canto [7] and iNaturalist [8]. By allowing citizen scientists to upload and annotate recordings from everyday devices like mobile phones, these platforms have facilitated the collection of sounds of diverse taxa from around the world. Citizen science recordings tend to be curated, focal recordings, often clustered around human population centers, and often featuring popular species. In contrast, biodiversity monitoring applications tend to use soundscapes that are collected from continuous, passive acoustic monitoring (PAM) devices and have much more background noise and species overlap. Such efforts also typically target ecologically important or endangered zones such as tropical rainforests or wetlands and give considerable importance to rare species. This disparity between the training data and deployment targets means that bioacoustics models must be robust to challenging domain shifts: acoustic, species presence/distribution, and geographic (1).

Despite the drawbacks, the sheer volume of annotated data provided by these platforms is an essential component of recent bioacoustic foundation models. Furthermore, citizen science websites include a wealth of auxiliary information such as recording location, date, and quality; individual call type, sex, and life stage; free-form notes left by the recordist as well as certain background species and recording device characteristics. Global taxonomic patterns in these metadata sources could be utilized to learn useful correlations that help bridge the domain gaps. While recent work has demonstrated the utility of supervised learning [6], it often uses species annotations as the only source of supervision. In this work, we employ a multi-task learning approach, jointly training our model with the different metadata prediction tasks alongside the primary species identification task, and quantify the effects of learning from metadata in this setting.

Multi-task learning with metadata presents several key challenges. First, complete coverage of all metadata is often not practical and models must appropriately handle missing metadata. Second, popular data augmentations such as mixup [9] require care when mixing recordings with varying metadata availabilities. Third, as we include more metadata, the number of training losses increases and the multi-objective optimization problem becomes more complex. Lastly, some metadata sources may encourage spurious correlations not grounded in ecological interpretation, potentially worsening the domain gap. In this work, we tackle these challenges to build metadata-aware species identification models.

We summarize our main contributions below:

1. We propose MetaPerch¹, a new bioacoustic foundation model that uses metadata as auxiliary targets.

2. We demonstrate that learning with metadata improves species identification with extensive experiments across a wide range of datasets and tasks covering diverse taxa and geographical regions.

3. We present a comprehensive empirical study of the disentangled importance of different metadata modalities and the effects of various loss formulations and other design choices in our training paradigm.

2 Related work↩︎

Conditioning on metadata. At test-time, metadata can provide additional context for species identification. Apps such as Merlin Sound ID use location to help disambiguate between acoustically similar species. [4] use species range estimation models [10] to construct geographic priors [11] and constrain the set of possible species at each location. [12] use location and week-of-year to train their own species range model for similar test-time species filtering. [13] also explore the benefits of location and time as inputs or priors. [14] explore the inclusion of location and recordist notes as free form text for zero-shot captioning. A key assumption here is the availability of metadata at test-time. In contrast, our work uses metadata solely during training, making it easier to handle missing metadata and facilitating broader downstream applicability.

Training with metadata. The use of metadata as auxiliary targets is largely underexplored in bioacoustics. [15] used sound descriptions, functional traits [16] and life-history [17] as metadata for zero-shot species ID. While these per-species metadata sources give additional context about the species itself, our work explores per-recording metadata that often provide complementary information. NatureLM-audio [14] and AnimalSpeak [18] use various recording metadata to construct templated text captions that are used to train audio-language models. In contrast, our work explores each metadata source with a dedicated prediction task, disentangling their effect and enabling more detailed analyses. [19] and [14] also explore the prediction of metadata such as location and vocalization call type as primary tasks for bioacoustic applications beyond species identification. In this work, we focus solely on species identification, comparing its performance with and without metadata.

Species-only learning. A large fraction of bioacoustics is devoted to species identification, and methods tend to rely on species labels as the only source of supervision. Prior work has focused on birds [12], [20]–[25] and broader terrestrial [4], [14], [26]–[32] and marine [33]–[35] taxonomic groups, leading to foundation models for bioacoustics. Recent work has shown the promise of increasing supervision and including general sound sources [6], [26]. We build on this progress and address the question of whether metadata can be useful sources to provide even more supervised signal and facilitate better transfer under challenging domain shifts.

General multitask and multimodal learning. Training with multiple tasks can be challenging because of differing loss and optimization semantics. Prior work has explored dynamic loss weighting [36], [37], gradient manipulations [38], [39], and multi-objective optimization [40] techniques. Strategies for handling missing modalities [41] are useful references when handling missing metadata. In this work, we address these challenges in the context of bioacoustics and present training recipes for metadata-aware species identification.

Metadata outside bioacoustics. The use of metadata, particularly location, has received significant interest in other fields as well. Soundscape mapping [42] aims to understand the acoustic signatures of different geographic locations. Remote sensing approaches also often rely on reconstruction of environmental metadata such as altitude and temperature to learn robust geospatial embeddings [43], [44]. Similar to bioacoustics, conditioning on location to form geographic priors has been explored in visual species identification [11]. Directly predicting metadata such as location has also received recent attention with images [45], [46] as well as audio [19]. While the use of metadata has seen some success in these fields, it remains under-explored in bioacoustics, and whether the successes transfer to this challenging domain is an open question that we aim to address in this work.

3 Methodology↩︎

Our primary goal with this work is to investigate the impact of a wide range of metadata sources on model performance when used to construct auxiliary supervision targets. We borrow elements of the base architecture design from the current SotA bioacoustics foundation model, Perch 2.0 [6], and modify key elements to enable faster iteration and better visibility of metadata influence, to establish a strong baseline for comparison which we refer to as BioBaseline. We build upon this baseline architecture and introduce auxiliary metadata prediction tasks. After careful model selection among models co-trained with metadata supervision signals, we compare the best-performing model, called MetaPerch, against BioBaseline and other recent bioacoustics foundation models on a wide range of evaluation tasks.

The following subsections describe BioBaseline and MetaPerch in more detail. We focus on the specific design choices behind MetaPerch in this section and present a large-scale study of these considerations and ablations in 4.4 and the Appendix.

3.1 Training datasets↩︎

Following [6], we train BioBaseline and MetaPerch on four labeled audio datasets, using the same class mappings and spectrogram configurations: Xeno-Canto [47], iNaturalist [48], the Tierstimmenarchiv [49] and FSD50K [50]. The first three predominantly feature animal vocalizations while the last contains a variety of anthropogenic and environmental sounds. In total, this collection of datasets constitutes over 18,000 hours of audio from 1.55M recordings, with 1.62M annotations across 14,597 animal species and 198 general sound event classes. Refer to ¿tbl:tab:data95summary? for summary datasets statistics and 2 for geographical distributions of the training and evaluation datasets. Additional details are included 6.1.

3.2 BioBaseline↩︎

In terms of architecture and training, we broadly follow [6] for BioBaseline. We use random window sampling with 5s windows and mixup augmentation [6], [9]. The 2D spectrogram of this mixed window is treated as a single-channel image and embedded using EfficientNet-B3 [51], a lightweight vision model with \(\sim\)​12M parameters. Activations of the penultimate layer, before pooling, are extracted as the spatial embeddings. The model includes a prototype learning classifier [52], [53] with these spatial embeddings and a linear classifier with spatially-aggregated embeddings to predict the class. Both heads are trained with a cross-entropy loss, but the embedding model receives gradients back-propagated from the linear head only.

Two components we do not include which differentiate BioBaseline and MetaPerch from Perch 2.0 are source prediction and self-distillation. Source prediction—predicting which recording a window was sampled from—requires the embeddings of each window to encode information to identify the original audio recording. We posit that this task may involve implicitly learning information that is encoded in various metadata sources, which can obfuscate the influences of different metadata on the resulting model. Additionally, the output head is the dimension of the number of recordings, which substantially increases the number of model parameters given our training corpus. Self-distillation involves a two-stage training regime—first training a “noisy student” model, then using that to learn a fine-tuned model—where hyperparameter tuning is required at each training stage. This process is computationally expensive and reduces scalability.

3.3 MetaPerch↩︎

What differentiates MetaPerch from other biacoustics foundation models is the addition of auxiliary losses during training. In conjunction with species detection, the model receives supervised signals by predicting corresponding metadata values from the input audio. We construct additional metadata prediction heads as small multilayer perceptrons (MLPs) each project the spatially-aggregated embedding into a dimension appropriate for the given metadata. The specific architecture details for each MLP—number of hidden layers, hidden dimension, and activation function—are treated as hyperparameters.

From our large-scale empirical study, we find that location, season, and background species as auxiliary prediction tasks yield a strong model. These metadata are also intuitively correlated with species identification. Predicting location incentivizes the model to also attend to ambient and background sounds for potential geographic cues, which should be helpful in soundscapes. As in [54], we group neighboring locations into S2 cells [55] and treat location prediction as a classification task. In conjunction with location, the season can add useful context for migratory species. A fraction of the recordings also include annotations for the background species, and attending to these could further facilitate better transfer to soundscapes. For an overview of the methodology, please refer to 3. The other metadata sources considered, but not used after model selection, such as recording characteristics (quality, sampling rate), and species individual attributes (vocalization call type, individual life stage) are also included in 3 4, and detailed in 6.4. Our model, and the general multi-task learning paradigm is formalized in 6.10.

3.3.1 Learning from metadata↩︎

Multi-loss training. Appropriate loss balancing is important for effective optimization in multi-task learning. We frame most metadata prediction tasks as classification problems and use the (softmax) cross-entropy loss—we find empirically that this provides useful training signal, and also yields similar loss landscapes and orders of magnitude across the tasks. For simplicity, we use a constant loss weighting strategy and include the weight of each metadata loss as a hyperparameter.

Missing metadata. While metadata such as location are available for most recordings, others such as background species or recording notes are missing for significant fractions of recordings, some for entire datasets (4). To fully utilize the metadata that is available, it becomes important to appropriately handle cases of missing metadata. Following prior work in multimodal learning with missing modalitites [41], we assign placeholder values for recordings that are missing a particular metadata and then zero-out their loss contributions.

Mixup with metadata. Metadata requires more care than species labels when mixup is involved. When mixing two recordings with distinct species, it is reasonable to define the target as the union of species present in both recordings. However, defining the location metadata targets as the average of the two recordings’ latitude and longitude may result in nonsensical targets, for example. In practice, the metadata prediction tasks used in MetaPerch are all framed as classification tasks, and we found that naively constructing a multihot target with all individual classes worked well.

Another edge case to consider is how to handle metadata that is only available in some but not all of the mixed recordings. In such cases, we still mix the audio from all recordings, but only mix metadata that is present. Although this design leads to potential mislabeling for cases with missing metadata, in practice, we observe that it works better than the other alternatives we explored.

Adversarial learning. Some species-metadata correlations may reflect training biases rather than ecologically grounded relationships. Using such metadata for auxiliary training may actually make the model more brittle to domain shift. In such cases, we can encourage embeddings to be agnostic to—rather than predictive of—such “spurious” metadata by leveraging an idea introduced in domain adversarial networks [56] and using a gradient reversal layer [57] between the backbone and the metadata prediction heads. By maintaining the forward pass while negating the backward pass, this layer ensures that the metadata head is trained with the usual gradients while incentivizing the model to learn embeddings that worsen metadata prediction. Our model selection procedure is allowed to explore both standard and adversarial training for each metadata source.

3.4 Validation datasets and model selection↩︎

In alignment with the model selection procedure presented in [6], we validate on tasks that mirror some aspects of the evaluation setup while ensuring that there is no test set leakage. We perform model selection based on three tasks categories:

1. observed-species classification (Powdermill [58] and Caples [58])

2. one-shot embedding retrieval (training sets of BEANS detection tasks, Weldy calltype [59], and Powdermill and Caples)

3. novel species linear probing (training sets of BEANS classification tasks, DCLDE [60], NOAA [61], ReefSet [33], and some other tasks from [20])

We average the ROC-AUCs in each validation task category and then compute a geometric average over the three categories, which we use for model selection. To tune hyperparameters, we use Vizier [62], a black-box optimization algorithm. Refer to 6.6 for an overview of our hyperparameter optimization search spaces and selected values.

4 Results and discussion↩︎

We investigate the impacts of metadata through the performance of MetaPerch on an extensive evaluation suite consisting of 17 species identification datasets. For each benchmark, we describe the evaluation setup, then discuss our performance on it. For hyperparameter tuning, we use Vizier [62], Python-based service for black-box optimization, with identical numbers of trials to perform model selection for both MetaPerch and BioBaseline. The selected hyperparameters are then used to retrain models 5 times with different random seeds to compute the mean and standard deviation on each metric. We report the mean performance in the tables below, and include additional results with standard deviations in 6.7.

4.1 BirdSet benchmark↩︎

We present our results on BirdSet [2] in ¿tbl:tab:birdset95results? and 5. The benchmark includes seven avian datasets from North and South America, the Hawai’ian islands, and Europe. Each PAM recording is accompanied with dense species annotations (bounding boxes), which we treat as a detection problem. Since all species in these datasets are present in our training corpus, we use the trained prototype learning head directly for species prediction and compute ROC-AUC scores.

Metadata improves transfer to soundscapes for known species. BirdSet captures model performance under the acoustic (focal \(\rightarrow\) soundscape) distribution shift. We see consistent improvements for MetaPerch over BioBaseline across datasets, with an increase of 0.015 in ROC-AUC on average. This improvement could be partly attributed to learning embeddings that attend to and are predictive of background species. MetaPerch also performs favorably in comparison to recent approaches: its ROC-AUC is 0.002 below the SotA, Perch 2.0, a more complex model.

Metadata helps more in underrepresented regions. The geographic distribution of our training datasets (2) is highly imbalanced: South America and Hawai’i, among others, are underrepresented compared to North America and Western Europe. In BirdSet, we observe strong performance for PER (Peru) [63], UHH (Hawaiʻi) [64], and NES (Colombia, Costa Rica), with an improvement of 0.025 ROC-AUC points on average. On the other hand, the North American and Western European datasets (HSN [65], NBP [66], SSW [67], SNE [68]) have relatively smaller improvements, averaging at 0.006 ROC-AUC points. These results suggest that metadata auxiliary training signal—particularly from location and season—encodes additional information that may help bridge the data gap for underrepresented regions.

4.2 BEANS benchmark↩︎

The BEnchmark of Animal Sounds, BEANS [3], includes 12 bioacoustic and miscellaneous audio datasets. For each dataset, a train split is included, which we use to train prototype learning probes on the model’s frozen embeddings. We treat datasets with focal recordings as classification tasks and report accuracy, and those consisting of PAM soundscapes (with dense labels) as detection tasks and report cmAP. Our results are presented in ¿tbl:tab:beans95results? and 5. Among the datasets presented in BEANS, we include only the bioacoustic tasks, and exclude CBI [69] because of potential overlap with our training set.

Metadata helps transfer to novel species for focal recordings. BEANS classification tasks include focal recordings with species and classes either completely unobserved or relatively rare in our training corpus. We observe consistent benefit on all classification tasks, with an improvement of 0.016 on average accuracy. Notably, on Watkins [70] and Bats [71] accuracy improves by 0.021 and 0.012, respectively, even though aquatic and very high frequency bat sounds differ greatly from the predominantly avian sounds in our training data. HumbugDB [72] contains mosquito data from different parts of the world, and encoded location information may explain MetaPerch’s better performance. MetaPerch also seems to better capture intra-species variations in the Dogs dataset [73].

Metadata helps transfer to soundscapes for novel species. Performance on the BEANS detection tasks captures model performance under both acoustic and species shift. MetaPerch also generally outperforms BioBaseline here, with an improvement of 0.006 on Mean cmAP. We observe strong improvements of 0.009 and 0.019 ROC-AUC on the avian datasets ENABirds [74] and DCASE [75], respectively. In contrast, metadata has limited benefit for the other datasets Hiceas [76], RFCX [77], and Hainan Gibbons [78], which feature marine mammals, anurans, and primates, respectively. We suspect that while MetaPerch improves transfer under species and acoustic shifts on their own, the benefits diminish when both shifts are present in conjunction.

Overall, MetaPerch achieves SotA performance on BEANS classification and detection tasks, outperforming the next best model by 0.008 accuracy and 0.006 cmAP points.

4.3 WABAD↩︎

WABAD [79] is a soundscape dataset consisting of 1,192 bird species from around the world. It is geographically diverse, particularly with recordings from Africa, Asia, and New Zealand—regions that are less prominent in the other evaluation datasets. Similar to BirdSet, we use the trained classifier heads (linear and prototype learning) directly to compute ROC-AUC. We also report a one-shot retrieval metric, similar to the validation tasks. Our results are presented in ¿tbl:tab:wabad95results?.

Metadata helps in globally distributed tasks. While the other benchmarks include datasets from different parts of the world, most tasks are geographically limited on their own, and the model is tasked with differentiating recordings within a task only. For tasks with species from different parts of the world, correlations with metadata such as location can be incredibly useful. In WABAD, MetaPerch improves over the baseline by 0.070 and 0.018 (ROC-AUC) for linear and prototype learning probes, respectively. ¿tbl:tab:beans95results? shows similar results with the other globally distributed tasks in BEANS: Watkins and HumbugDB.

Metadata benefits vary across biomes. We evaluate the performance of models on recordings from different biomes in WABAD. The Tropical and Sub-tropical biome has the highest improvements of 0.023, while for deserts, we see a marked drop of 0.024 ROC-AUC. These results suggest that the benefits for metadata are not uniform, and that recording environment may be correlated to the benefit of metadata for species identification.

Metadata allows embeddings to encode habitat information. In 6, we also visualize our and baseline embeddings, colored by the biomes. Our embeddings appear more separable by biome, albeit still fragmented, with distinct sub-clusters forming within each biome. This structure is also visible in the visualizations colored by species in 6.8. Since this is a multi-species detection dataset, we suspect that these sub-clusters correspond to species combinations that co-occur frequently.

4.4 Ablations↩︎

MetaPerch design choices. First, we study the effects of different design choices in MetaPerch. For the multi-loss training strategy (¿tbl:tab:weighting?), our validation performance suggests that treating the weights of each metadata loss as hyperparameters to be optimized works better than more involved strategies such as dynamic equalization or gradient surgery [39]. This is likely because all of our losses are softmax cross entropies, have similar gradient landscapes and are in similar orders of magnitudes. For mixup ([tbl:tab:mixup]), we observe that our mixing of any present metadata attains better validation performance than the stricter choice of mixing a metadata if and only if (iff) it is present for all. Using a weighted mean also leads to similar drops, and all 3 are better than no mixup at all.

In these experiments, we note a disconnect between the validation and WABAD performance: for example, after sorting the Vizier trials by their validation performance, the median trial actually performs better on WABAD than the best one. Since our design choices are driven by the validation performance, we may miss out on some test performance improvements, and these results highlight the challenges of choosing appropriate validation tasks.

BIRB. We analyze the performance of MetaPerch under controlled distribution shift experiments, similar to those proposed in BIRB [80]. Focusing on the Hawaiʻi [64] and Coffee Farms [81] datasets from BirdSet, we exclude all species present in these datasets from our training data and retrain BioBaseline and MetaPerch. The task is framed as few shot classification, where a few iNaturalist windows of each species are used to train a linear probe, which is then evaluated on 1) the Xeno-Canto subset containing held-out species and 2) the PAM datasets themselves. While the latter contains both species distribution and acoustic shifts, the former allows us to study species distribution shift in isolation.

These results are presented in 7. For lower shots, we observe that MetaPerch outperforms BioBaseline for both Xeno-Canto held-out and the soundscape datasets. This trend reverses for higher shots in the Xeno-Canto held-out set, suggesting potential shortcomings of MetaPerch. The Xeno-Canto held-out set reflects an artificial setting designed to disentangle the effects of acoustic and species shifts, whereas PAM soundscape experiments are more representative of natural deployment conditions. We report on both to provide a holistic picture of MetaPerch, but we expect PAM soundscape results to be more reflective of actual metadata benefits in practice.

Effects of missing metadata. We expect the benefits offered by a particular metadata to increase as more of this metadata is available. To better understand the sensitivity of model performance to metadata availability, we observe performance with location prediction when fractions of the metadata are artificially marked as missing. The results, presented in [fig:missing], show that while the performance decreases as lesser metadata is present, there are still noticeable improvements over baseline for as low as 1% availability. This further highlights the strong potential of metadata, and supports our choice of including metadata even if they are available for only a fraction of the training data.

Individual metadata performance. We observe improvements by encouraging features to be predictive of most metadata on their own, with boosts ranging from 0.008 for Day-Part to 0.019 ROC-AUC points for Background Species ([fig:individual]). Adversarial training is better for day-part and text embeddings. Interestingly, we observe similar performance for both standard and adversarial training with sampling rate and quality. Background species and location have the most improvement with standard training, and the most drops with adversarial training, highlighting their importance for robust bioacoustic features.

Location prediction formulations. Each metadata prediction task can be formulated in a number of ways, with corresponding prediction heads and losses. We explore some formulations of location prediction in [fig:location95losses] (additional details in 6.4). Strong WABAD performance across different formulations (regression, classification, and distillation) underscores the potential of location prediction. S2 cell classification at level 7 (average cell area is roughly 50,000km\(^2\)) performs best, followed closely by Cartesian regression. Finding a suitable loss formulation involves a tradeoff between learnability and specificity. For example, a coarser-grained S2 cell mapping may be easier to predict, but misses finer geographical context.

5 Conclusion and future work↩︎

Metadata offers a rich source of auxiliary supervision for bioacoustics foundation models: learning with metadata improves species identification by leveraging species–metadata correlation. Global patterns in readily available metadata such as location and time can help species identification and increase robustness to various domain shifts. Our study demonstrates the potential of using such metadata as auxiliary losses, highlights the most beneficial metadata sources, and explores the effects of different design choices in this paradigm. Strong performance benefits with our simple approach presents a lower bound for metadata benefits and lays the groundwork for future research in metadata-aware species identification.

An interesting future direction is in the design of alternative paradigms such as pre-training or test-time conditioning. Extending our method to other relevant metadata and base foundation models would make for meaningful follow-up work. Metadata could also guide data sampling in an effort to alleviate some spatiotemporal biases or construct training curricula. We hope the evidence presented in this work motivates researchers to consider incorporating diverse metadata and auxiliary supervision signals when constructing datasets and benchmarks.

Acknowledgements↩︎

We thank Bart van Merrienboer, Tom Denton and Lauren Harrell for their perspectives around this work. We thank Richard Song for guidance on deploying large-scale Vizier hyperparameter tuning.

Impact statement↩︎

Potential misuse: wildlife disruption or harm. An important goal of bioacoustics research in general is to aid conservation efforts and biodiversity monitoring by automating and scaling species identification. However, these models could also be used to harm species of interest. For example, species of interest could be identified using bioacoustic monitoring setups with species identification to locate species of interest, and this information could be used to harass, trap, or poach species.

For MetaPerch in particular, metadata such as location can be another cause for concern. While we do not release metadata prediction heads, our embeddings may still be used to train geolocalization models to locate species of interest. However, the S2 cell level used (L7) is coarse enough (average cell area of around 50,000 square kilometers) that predicting the S2 cell at that level would not be useful for poaching purposes.

Personally identifiable information. Human voice or other personally identifiable content may be present in the citizen science platforms such as iNaturalist and Xeno-Canto, and consequently, our training data. Since the recordings are openly available and licensed for research use, we do not obfuscate or modify the data for training.

6 Appendix↩︎

6.1 Additional dataset details↩︎

We enumerate all datasets used in our experiments in ¿tbl:tab:supp95data95stats?. The species in the different training datasets partially overlap. In total, we have more than 18,000 hours of audio from 1.55M recordings with 1.62M annotations across 14,597 species and 198 general sound event classes. Following [6], the Xeno-Canto data was obtained from the public API and the iNaturalist data was obtained by collecting the ‘research-grade’ audio examples reported to the Global Biodiversity Information Facility (GBIF). Since Xeno-Canto, iNaturalist and the Tierstimmenarchiv use different taxonomies (species names) the classes from Xeno-Canto and Tierstimmenarchiv were mapped into the iNaturalist classes. All bat recordings are removed from the datasets.

We visualize the geographical distributions of individual training datasets in 8, and their class frequency distributions in 9. These demonstrate both the geographical and ecological skew in the training datasets. The long-tailed distributions make species identification particularly challenging for underrepresented regions and rare species. We include addition statistics of metadata availability in 10.

6.2 Additional metric details↩︎

Let the labels for an evaluation task of \(N\) audio windows across \(C\) species be \(y \in \mathbb{R}^{N \times C}\) and the predictions made be \(\hat{y} \in \mathbb{R}^{N \times C}\).

ROC-AUC. This metric computes the probability of ranking a random positive above a random negative. It captures a model’s ability to distinguish between classes in a threshold-independent manner. ROC-AUC is computed for each class in a one-vs-rest fashion, and then averaged over all classes. \[\begin{align} \mathrm{ROC-AUC} = \frac{1}{C} \sum_{c=1}^{C} \mathbb{P}\big( \hat{y}_{i,c} > \hat{y}_{j,c} \mid y_{i,c}=1,\; y_{j,c}=0 \big) \end{align}\]

Accuracy. We report accuracy for BEANS classification tasks, which are single-label window-level prediction tasks on focal recordings. This is a simple top-1 accuracy, and may be brittle for class-imbalanced data in general. \[\begin{align} \mathrm{ACC} = \frac{1}{N} \sum_{i=1}^{N}\sum_{c=1}^{C} \mathbf{1}\big\{ \hat{y}_{i,c} = y_{i,c}\big\} \end{align}\]

cmAP. We report cmAP for BirdSet and for BEANS detection tasks. This metric measures the ranking ability of a model. Average precision is computed for each class in a one-vs-rest fashion, and then averaged over all classes. \[\begin{align} \mathrm{cmAP} &= \frac{1}{C} \sum_{c=1}^{C} \mathrm{AP}_c \end{align}\]

6.3 Additional baseline details↩︎

Preprocessing and augmentation. For each training datum, we sample a group of 5 second windows at random from distinct training recordings, also sampled at random. Since annotations do not include time bounds, we assign all labels of a recording to each of its windows. The audio windows are mixed [6], [9] with each other and converted into a spectrogram via STFT, visualizing the frequency (\(F\)) composition over time (\(T\)). The number of windows mixed in each datum and the weights used while mixing audio are themselves sampled from beta-binomial and symmetric Dirichlet distributions, respectively. An aggregate of the labels of each original window is used as the label for this mixed window.

Modeling. The 2D spectrogram (\(T \times F\)) is treated as an single-channel image and embedded using EfficientNet-B3 [51], a lightweight vision model with \(\sim\)​12M parameters. Activations of the penultimate layer, before pooling, are extracted as the spatial embeddings (\(T' \times F' \times dim\)). We use a prototype learning classifier [52], [53] with these spatial embeddings and a linear classifier with aggregate embeddings (\(dim\)) to predict the species.

Learning. Multihot class target vectors are scaled to sum to one and used as soft labels in a softmax cross-entropy loss to train the prototype learning and linear classifiers. For the prototype learning head, we include an orthogonality loss [53], [82] that maximizes prototype diversity. The embedding model receives gradients backpropagated from the linear classifier, but not the prototype learning classifier. This design promotes linear separability of learned features while retaining benefits of the stronger prototypical classifier. The model is trained with an Adam [83] optimizer for 500K training steps with a batch size of 256. Please refer to 6.6 for other hyperparameters.

6.4 Additional method details↩︎

Location. Species are inherently tied to their geographic ranges and knowing the location of a recording can be informative of the type of species one could expect in it. Predicting location may incentivize our models to also attend to ambient and background sounds for potential geographic cues, which may be helpful in soundscapes. As in [54], we group neighboring locations into S2 cells [55] and treat location prediction as a classification task. The S2 cells can be constructed at different resolutions and the number of classes changes accordingly. We also explore other regression and distillation based approaches in 4.4. We explore the following different loss formulations and prediction heads for location:

1. Regression. Perhaps the most intuitive approach is to regress the location coordinates directly. We use either a simple L2 loss between the spherical coordinates, their projections on a 3D Cartesian sphere, or the Haversine distance as the loss.

2. Classification. Similar to [54], we group neighboring locations into S2 cells² and predict the cell a particular recording would belong to. The S2 cells can be constructed at different resolutions (levels): lower resolution cells would likely make the learning task easier while higher resolution cells can allow learning finer geospatial patterns. Our training dataset has significant geographic skew and the resulting S2 class distribution is long-tailed. To remedy this, we also explore hierarchical S2 cells where higher resolution cells are used for regions with higher density of recordings (or species) and vice versa. We visualize the S2 cells for different levels and hierarchical S2 cells in 11 and 12, respectively.

3. Distillation. Following prior work in geo-localization [19], [45], we also explore the use of dense location embeddings as distillation targets. Approaches like GeoCLIP [45] and SatCLIP [44] learn location embeddings with the help of paired ground-level or satellite images. We distill that information into the model by learning to predict the corresponding location embeddings from the model’s embeddings (trained on bioacoustic data as inputs only). We use the Cosine similarity between predicted and true location embeddings as the training loss.

Season. Species’ vocal activity often varies according to the season. In conjunction with location, season can add useful context for migratory species. We use the date and location (specifically the hemisphere) of a recording to determine the season as one of Spring, Summer, Fall or Winter and frame this as a classification task.

Part of day. The local time of day could potentially provide a prior over the expected species. For example, audio recorded around dawn is likely to capture a chorus of birds [84], while another recorded after dusk may feature more insects. To account for global time zone differences, we use broad boundaries to determine the part of day as one of dawn, dusk, day, and night. This task is also treated as a classification problem.

We explore the following different loss formulations for learning with season or day part:

1. Classification. Each of the season or day-part categories are treated as independent classes and the model is trained with cross-entropy.

2. Unit circle. Both season and day-part have cyclic temporal relationships between the different categories. To better capture these, we represent the category as a point on the unit circle and task the model to predict the angle or cartesian coordinates. The full circle is divided into equal sectors for each category, so that the last category is followed by the first, capturing the cyclicity of season and day-part. The loss is now the angle between the predicted and true points.

Background species. Our training data consists of primarily focal recordings, each accompanied with annotations for the foreground species. A fraction of the recordings also include annotations for the background species, and attending to these could facilitate better transfer to soundscapes. Absence of background annotations does not imply absence of background species. Incorporating them directly into the main prediction task would incorrectly assume otherwise, so we treat background species prediction as metadata and handle missing cases accordingly. We use a loss similar to foreground species prediction.

Audio characteristics. Some recordings have additional fields where recordists can include audio quality as categorical grades (A-E or A-C). While this is somewhat subjective, it often captures some measure of the signal-to-noise ratio of the recording. The original sampling rate could also be a useful signal that could allow the model to better address any spectrogram artifacts introduced by resampling. We use the original sampling rate to assign a binary class of whether the recording was up or down-sampled during preprocessing (all resampled to 32kHz).

Miscellaneous. Some recordings have the stage of life (juvenile, adult etc) and gender of the species being recorded. FSD50K recordings have textual description and tags while a number of recordings on the bioacoustic platforms have notes left by the recordist. There are also more free-form metadata that hint at different characteristics of the audio such as the microphone type and recording method (field recording, in enclosure etc.). We combine these metadata sources together with the actual foreground label to construct textual prompts and use their text embeddings from the Gemini API as the supervisory targets. We use the negative cosine similarity for training.

Derived metadata. We also explore a few artificially constructed metadata. Based on the training species distribution, we categorize each species based on its abundance or rarity and use this as a classification target. We also explore the source of a recording (Xeno-Canto vs iNaturalist, for example) as another metadata.

6.5 Spectrogram construction↩︎

Each audio recording is resampled to 32kHz. We take in audio windows of length 5s (160,000 samples) and construct a log mel-spectrogram using STFT with a hop and window length of 10 and 20ms respectively, outputting a total of 500 frames with 128 mel-scaled frequency bins ranging from 60 Hz to 16 kHz. We use an uncentered Hann window with FFT size of 1024 samples. We calculate the energy (magnitude) spectrogram (so not a power spectrogram) and the output is scaled by the reciprocal of the sum of the window values. After the calculation of the mel-spectrogram we apply a logarithm with a floor of 10\(^{-5}\) and then multiply the output by 0.1.

6.6 Hyperparameters↩︎

We enumerate the hyperparameters for MetaPerch in 6.6. For each, we mention the search space, defined by the minimum and maximum values in case of continuous parameters and the set of possible values in the case of categorical parameters. For the former, we also include the type of steps taken within the search space (one of linear or logarithmic). Finally, we list the optimal value of each hyperparameter found by Vizier [62].

6.7 Additional results: BEANS, BirdSet, & WABAD↩︎

We include the mean and standard deviations for the 5 runs of MetaPerch and BioBaseline for BEANS (¿tbl:tab:supp95beans951? ¿tbl:tab:supp95beans952?), BirdSet (¿tbl:tab:supp95birdset?), and WABAD (¿tbl:tab:supp95wabad951? ¿tbl:tab:supp95wabad952?). For BEANS, we also include other datasets that we excluded from the main tables. These datasets were ESC-50, Speech Cmds, and CBI. We also see an improvement over baseline on the average of the full set of datasets, but the ones included in the main paper are more appropriate to our explorations.

6.8 WABAD additional embedding visualizations↩︎

We visualize the embeddings of WABAD, colored by location in 13 and by species in 15. In 16, we visualize the embeddings colored semantically (similar colors indicate similar features) on the world map. Each coordinate is colored by the mean embeddings of all recording windows from the location closest to that coordinate.

6.9 Additional ablation discussions↩︎

Multi-loss training. We explore the effects of the multi-loss training strategy in ¿tbl:tab:weighting?. Simply treating the weights of each metadata loss as a hyperparameter to be optimized works better than more involved strategies such as dynamic equalization or gradient surgery [39]. The likely reason is that since most of our losses are softmax cross entropies, they have similar gradient landscapes and are in similar orders of magnitudes, making the loss weighing problem easier. This is further supported by the relatively small difference in the best and median validation performances we observe during our vizier hyperparameter search. Interestingly, hyperparameters with the median validation score actually performs better on WABAD than the best validation score, highlighting a potential gap between our validation tasks and the evaluation setup.

Mixup variants. In MetaPerch, we mix any metadata present for a recording. We could be more strict, and mark a mixed metadata present if and only if (iff) it is present for all original recordings. This leads to a drop in validation performance ([tbl:tab:mixup]), suggesting that noisy-but-more metadata supervision may be better then clean-but-less. Using the sampled audio-mixup weights to get a weighted mean of the metadata instead of a simple mean leads to a small drop. We observe marginal differences in WABAD performance, suggesting it is less sensitive to this design choice. We see significant drops in performance if we remove mixup altogether.

Location loss formulations. Each metadata prediction task can be formulated in a number of ways. We explore some formulations of location prediction in [fig:location95losses]. Finding a suitable loss formulation involves a tradeoff between learnability and specificity. For example, coarser S2 cells may be easier to predict, but would likely miss fine-grained geographical context. However, we observe strong WABAD performance across different formulations: regression, classification, and distillation. This is likely an artifact of these formulations capturing the same underlying information, and further underscores the potential benefits of location prediction.

Following a similar procedure as our GeoCLIP and SatCLIP experiments, we use AlphaEarth [43] embeddings corresponding to the location and time (year) of our recordings as distillation targets and train our decoder with cosine loss. Results on WABAD are reported in 1. While the inclusion of AlphaEarth improves performance over the baseline, it falls short of classification with level 7, the formulation used in MetaPerch.

4pt

6.10 Multitask learning formalism↩︎

Let \(\mathcal{D} = \{(x_i, y_i, \{m_i^{k}\}_{k=1}^{K})\}_{i=1}^{N}\) denote the training dataset (post mixup), where \(x_i\) is an input audio spectrogram (window), \(y_i \in \mathcal{Y}\) is the primary target label corresponding to the annotated focal species, \(m_i^{k}\) is the target for metadata source \(k\) out of total \(K\) metadata sources. If a particular metadata \(k\) is absent for the datapoint \(i\), we set its target to a placeholder value \(m_i^{k} = \phi\).

We encode the audio spectrogram with a shared encoder \(f_{\theta}(\cdot)\). The species prediction head \(g_{\phi}\) then predicts the focal species as \[\begin{align} \hat{y}_i = g_{\psi}(f_\theta(x_i)), \end{align}\] where \(\psi\) are the species decoder parameters. Similarly, each metadata \(k \in \{1, \dots, K\}\) is associated with a decoder \[\begin{align} \hat{m}_i^{k} = h_{\psi_k}^{k}(f_\theta(x_i)), \end{align}\] with task-specific parameters \(\psi_k\).

The overall multi-task objective combines the species identification loss used in Perch 2.0 [6] along with metadata-specific loss formulations \(\{\mathcal{L}^{m}_k\}_{k=1}^K\) as: \[\begin{align} \mathcal{L}_{\text{total}} = \mathcal{L}_{\text{Perch}}(\hat{y}_i, y_i) + \sum_{k=1}^{K} \lambda_k \mathbb{1}[m_i^{k}\neq \phi]\mathcal{L}_k^m(\hat{m}_i^{k}, m_i^{k}), \end{align}\] where \(\{\lambda_k\}_{k=1}^K\) control the relative importance of the different metadata, and are treated as hyperparameters to be optimized by Vizier. Metadata specific preprocessing such as conversion to S2Cell [55] as well as other design choices such as the decision between standard or adversarial training is abstracted away in the loss functions \(\{\mathcal{L}^{m}_k\}_{k=1}^K\).

References↩︎

---

1. Model released at github:google-research/perch/metaperch.↩︎

2. https://code.google.com/archive/p/s2-geometry-library/↩︎