2

1 Introduction↩︎

Large Audio Language Models (LALMs) [1]–[5] have demonstrated remarkable capabilities in audio understanding and reasoning, bridging the acoustic and linguistic domains by aligning audio encoders with large language models. As these models grow increasingly proficient at describing, classifying, and reasoning about sound content, the ability to precisely localize sound events in time emerges as a critical yet underexplored frontier.

Classical Sound Event Detection (SED) systems achieve frame-level temporal precision yet remain constrained by closed-label vocabularies that cannot generalize to the richness of natural language, while LALMs handle arbitrary queries with remarkable flexibility but struggle to produce fine-grained temporal predictions. We study the task that sits between these paradigms, which we call Open-Vocabulary Audio Event Grounding: given an audio clip and a natural language query describing a target sound event, predict all onset/offset intervals where that event occurs, supporting multiple occurrences and an open event vocabulary.

Progress on this field is fundamentally limited by data scarcity. Existing temporal audio grounding datasets [6], [7] are small, narrow in label diversity, and the cost of manually annotating fine-grained boundaries for arbitrary sound events is prohibitive. The field therefore lacks a viable training resource even though the task is within reach of current LALMs. Our central claim is that this gap can be closed on the data side, without architectural change, by constructing supervision automatically and exploiting it with reinforcement learning.

2 We instantiate this with Auto-AEG (Figure 1), a scalable pipeline whose key insight is that the two kinds of data a grounding policy needs have complementary, annotation-free acquisition strategies. Synthetic clips, composed programmatically from real audio, carry exact ground-truth intervals and are ideal for an SFT cold-start; real-world clips, pseudo-labeled through multi-model collaboration, are imperfect but provide the informative reward signal exploited by Group Relative Policy Optimization (GRPO). Both stages scale without manual annotation: synthetic generation is unbounded, and pseudo-labeling extends to any public audio collection. We pair this with AEGBench, an independent, quality-filtered evaluation benchmark drawn from four diverse audio sources and stratified by hard-case difficulty for diagnostic analysis.

Training Qwen3-Omni-30B-A3B and Qwen2.5-Omni-7B on Auto-AEG data with a two-stage fine-tuning approach that contains interval-aware reward design substantially improves temporal localization on AEGBench, with mIoU gains of +73.9% and +23.1% over zero-shot baselines and subtle degradation of general audio understanding. These results confirm Auto-AEG as an effective data-side solution for scaling LALM temporal capability.

Our contributions can be summarized as follows:

(i) Auto-AEG, a scalable pipeline that automatically constructs open-vocabulary onset/offset supervision, providing both exact-GT synthetic data and in-the-wild pseudo-labels;

(ii) AEGBench, an independent quality-filtered benchmark with difficulty stratification;

(iii) a two-stage fine-tuning framework that couples an instruction-tuned cold start with interval-aware GRPO, which turns imperfect pseudo-labels into large, architecture-free gains in LALM temporal grounding.

2 Related Work↩︎

2.1 Video Temporal Grounding↩︎

Temporal sentence grounding in video (TSGV) has been studied extensively since [8] and [9]. Methods range from proposal-based approaches [10] to proposal-free span prediction [11] and transformer-based moment retrieval [12], [13]. Large video-language models have more recently addressed TSGV in a generative setting, with Time-R1 [14] showing that explicit temporal supervision can substantially improve timestamp reasoning in video LLMs—a result that directly motivates our data-construction approach for the audio modality.

2.2 Sound Event Detection↩︎

Sound event detection (SED) is the immediate predecessor of our task. SED has been studied extensively through the DCASE challenge series [15], with dominant architectures including CRNNs [16], PANNs [17], and self-supervised transformers such as AST [18] and BEATs [19]. Weakly supervised SED [20], [21] uses clip-level labels to avoid expensive frame-level annotation—a design philosophy aligned with our pipeline, which leverages pre-trained model outputs rather than manual temporal annotation.

2.3 Audio Temporal Grounding and Moment Retrieval↩︎

Text-to-Audio Grounding (TAG) [6] introduced the AudioGrounding dataset (4,662 clips), pairing onset/offset timestamps with phrase-level queries. [22] extend this to weakly supervised regimes. Audio Moment Retrieval (AMR) [7] scales temporal grounding to longer untrimmed audio. CASTELLA [23] provides the first real-world AMR benchmark with 1,862 recordings of 1–5 minutes. BLAB [24] evaluates long-form localization focusing on speech. TACOS [25] contributes 12,358 Freesound recordings with human-verified temporal boundaries and trains a frame-level contrastive model; it is the closest concurrent work but targets contrastive retrieval rather than LALM adaptation. TimeAudio [26] improves temporal reasoning through architectural modifications trained on a 260K-sample supervised dataset.

SpotSound [27] adapts LALMs to open-vocabulary temporal grounding and augments existing grounding corpora with synthetic clips formed by mixing foreground events into background ambience. The idea of synthesizing temporal supervision by composition is itself established—Clotho-Moment [7], AudioTime [28], and PicoAudio [29] all generate timestamp supervision automatically.

Our work innovates in two respects compared to the above concurrent works. First, in training paradigm: SpotSound relies on supervised fine-tuning with a negative-sampling objective, whereas we exploit the imperfect real-world pseudo-labels as a reward signal and show that GRPO is what converts automatically constructed data into large temporal-grounding gains. Second, in scope: we target large Omni models (up to 30B) and pair the pipeline with an independent, difficulty-stratified benchmark rather than reusing existing grounding test sets. To our knowledge, Auto-AEG is the first to couple annotation-free data construction with reinforcement learning for open-vocabulary audio event grounding.

These resources rely on human annotation or supervised training and remain at the scale of thousands to tens of thousands of clips. Auto-AEG instead fills the training data gap by automatically constructing onset/offset supervision for both LALM SFT and GRPO from existing public audio.

2.4 Audio Language Models↩︎

LALMs [3], [30]–[32] have achieved strong performance on auditory understanding tasks. Qwen2-Audio [31], Qwen2.5-Omni [33], and Qwen3-Omni [34] are Whisper-based models that process audio as a sequence of continuous encoder features and generate responses autoregressively. Despite their strong semantic competence, these models lack targeted temporal boundary supervision—the gap our data directly addresses.

2.5 Automated Data Construction and RL Fine-Tuning↩︎

Large-scale automated annotation has been used in vision and audio: pseudo-label pipelines using CLIP [35] generate bounding boxes for downstream training; WavCaps [36] used ChatGPT to clean captions at scale. Reinforcement learning via GRPO [37] has been successfully applied to improve reasoning in language models [14], including temporal understanding in video. We bring this paradigm to open-vocabulary temporal audio grounding, using automatically constructed pseudo-labels as the reward signal rather than curated supervision.

3 Task Definition↩︎

3.0.0.1 Formal Definition.

Let \(\mathbf{x} \in \mathbb{R}^T\) be an audio waveform of duration \(D\) seconds and \(q\) a natural language query describing a target sound event type. The model must predict a set of time intervals \(\hat{\mathcal{Y}} = \{[s_i, e_i]\}_{i=1}^{K}\), where each interval corresponds to an occurrence of the event described by \(q\) in \(\mathbf{x}\). Ground-truth segments are denoted \(\mathcal{Y}^* = \{[s^*_j, e^*_j]\}_{j=1}^{M}\). This formulation requires the model to: (a) determine how many occurrences exist without count supervision, (b) handle zero occurrences, and (c) generalize over open event vocabulary.

3.0.0.2 Response Format.

Models are prompted to list all matching intervals in JSON array format within <answer> tags, optionally preceded by a <think> reasoning chain:

<think> ... </think>
<answer> [[s1, e1], [s2, e2], ...] </answer>

An empty list [] indicates the target event is absent. All metrics are computed from the intervals extracted from the <answer> block.

3.0.0.3 Evaluation Metrics.

We use the following metrics:

mIoU: Mean IoU between each ground-truth interval and its best-matching prediction, averaged over present-event queries (\(|\mathcal{Y}^*|>0\)): \[\text{mIoU} = \frac{1}{|\mathcal{Y}^*|} \sum_{j} \max_i \operatorname{IoU}\!\left([s^*_j, e^*_j],\, [s_i, e_i]\right).\] This is a recall-oriented localization score: each ground-truth segment is credited for the prediction that best overlaps it, and extra or hallucinated intervals are not penalized. To control for over-prediction we additionally report Event F1 and onset Precision, which use symmetric matching and explicitly penalize spurious predictions; the AEGBench improvements reported in Section 7.1 hold under all three. Absent-event queries (\(|\mathcal{Y}^*|=0\), whose correct answer is the empty list []) are excluded from mIoU; their correct rejection is instead measured by Event F1 and the precision metrics, where any non-empty prediction counts as a false positive.

Recall-IoU@\(\theta\) (R@\(\theta\)): Fraction of ground-truth segments matched by at least one prediction at IoU \(\geq \theta\), at \(\theta \in \{0.3, 0.5, 0.7\}\).

Precision-IoU@\(\theta\): the dual quantity—the fraction of predicted segments matched by at least one ground-truth interval at IoU \(\geq \theta\). Their harmonic mean, F1-IoU@\(\theta\), is used as the localization component of the GRPO reward (Section 6.2).

Event F1 (ev_F1): Standard F1 computed by matching predicted and ground-truth intervals at IoU \(\geq 0.5\) under one-to-one assignment, yielding explicit true-/false-positive counts. F1-IoU and Event F1 track the same trend but differ in matching: F1-IoU permits many-to-one overlap matching and is smoother as a per-sample training reward, whereas Event F1 enforces one-to-one matching and is stricter as an evaluation metric.

Segment F1 (seg_F1) and Onset P/R: Frame-level and onset-level precision/recall following DCASE SED evaluation convention.

3.0.0.4 Difficulty Taxonomy.

We identify six hard-case categories in the audio modality, as shown in Table 1.

2

4 Auto-AEG: Scalable Data Construction↩︎

Constructing AEG supervision faces a fundamental tension: synthetic data can be assigned exact ground-truth intervals by construction, but it diverges from the acoustics of natural recordings; real-world audio better matches deployment conditions, but automatic temporal annotations are inherently noisy. Auto-AEG resolves this by matching each data type to the training objective it serves. Stage 1 generates synthetic data with exact ground truth—suitable for SFT, which optimizes directly against the target sequence and is therefore sensitive to label noise; Stage 2 annotates real-world audio with pseudo-labels suited to the noise-tolerant GRPO objective, which learns from a scalar reward rather than token-level supervision and can self-correct over multiple rollouts even when individual annotations contain errors.

4.1 Audio Pool Construction↩︎

Both stages draw from a pool of audio clips sourced from the FreeSound subset of WavCaps [36], chosen for its broad event-label diversity, permissive licensing, and community-supplied tags that serve as initial label seeds. Each clip is pre-screened by PE A-Frame [38] in frame-level scoring mode: clips where the stated tag produces no active frames above threshold are discarded before any further annotation. This early gate eliminates clips whose tag is acoustically absent or too quiet to localize reliably—retaining them would introduce systematically wrong pseudo-labels in Stage 2 with no downstream signal indicating the annotation is invalid. Each retained entry stores its audio, tag label, and detected intervals, and serves as the common source for both data stages.

4.2 Stage 1: Programmatic Synthesis for SFT Cold-Start↩︎

No large-scale dataset provides open-vocabulary onset/offset annotations, so Stage 1 generates 10,000 training clips by programmatically composing pool segments. For a randomly chosen target label, we lay down several non-overlapping occurrences, optionally add distractor events of other labels, and mix the result onto a Gaussian background at a controlled signal-to-noise ratio. The occurrence count is deliberately skewed toward multiple events per clip: a single-occurrence majority would bias the model to always emit exactly one interval and systematically under-detect multi-event queries at inference time. Because every interval is fixed by the composition script, the ground truth \(\mathcal{Y}^* = \{[s_j, e_j]\}\) is exact and noiseless.

This clean signal provides a stable cold-start that instills the response format and basic acoustic recognition before the model faces real-world variability. Each clip is paired with a List-All query asking for every interval of the target event as a JSON array. Detailed description of sampled recordings, sampling ranges and the exact prompt are given in Appendix 11.

4.3 Stage 2: Multi-Model Annotation for GRPO↩︎

Stage 2 annotates real FreeSound clips (10–30 s) to produce training data for GRPO. The annotation pipeline decouples what events are present from when they occur, because no single model handles both reliably: LALMs excel at semantic event identification but produce coarse or hallucinated temporal boundaries, while dedicated frame-level models locate boundaries precisely but require a text query as input.

Gemini identifies audible events from fixed-length 10-second chunks—chunking improves label focus on long clips where full-clip prompting dilutes attention across many events—and pools labels across chunks into a compact, semantically orthogonal inventory per clip. Each label is then classified as continuous (sustained events such as engine noise or rain) or discrete (transient events such as a bark or knock) before temporal localization. This distinction controls span-merging behavior: without it, a uniform rule would incorrectly fuse two consecutive bark events separated by a brief silence, or fail to bridge legitimate millisecond dropouts within a continuous rain shower—both are common labeling errors that degrade boundary quality. PE A-Frame [38] then localizes each label by thresholding per-frame audio–text similarity into onset/offset spans, merging adjacent spans for continuous-type labels only.

Because labels are coined independently per clip, near-synonyms accumulate across the dataset (e.g., car horn, vehicle horn, horn honking), fragmenting the training vocabulary and making the same physical event appear under multiple query strings. A final global pass addresses this: all unique labels are embedded with CLAP [39] and clustered by acoustic similarity, then a language model assigns keep, drop, or rename to each label within its cluster. Clustering groups acoustically similar candidates that embedding distance alone cannot distinguish in quality, and the LM applies specificity and redundancy judgements that a nearest-neighbor rule cannot. Prompts, thresholds, and merging rules are detailed in Appendix 12.

Table 2 summarizes both data stages.

2

5 AEGBench: Difficulty-Stratified Benchmark↩︎

AEGBench provides an independent, quality-controlled evaluation surface for open-vocabulary audio event grounding. It is constructed entirely separately from Auto-AEG training data to avoid circular evaluation.

5.1 Candidate Selection and Quality Filtering↩︎

Candidates are drawn from four complementary sources—AudioSet Strong Labels, the FSD50K eval split [40], the BBC Sound Effects Library, and short YouTube life-sound clips—spanning human-verified strong labels, community labels, and professional field recordings. Each candidate must pass an energy-contrast filter requiring the target event to be measurably louder than its background, together with active-ratio, duration, and per-category-cap constraints (Appendix 13). The energy-contrast criterion is the primary differentiator from prior benchmark construction, which relies on duration and silence-ratio filters: it directly quantifies whether the signal would yield a reliable, unambiguous temporal boundary, making it a more principled quality gate. Per-source counts after filtering are reported in Table 3.

5.2 Annotation and Hard-Case Tagging↩︎

Benchmark items are annotated by the same multi-model pipeline as Stage 2 of Auto-AEG—Gemini label identification, event-type classification, PE A-Frame localization, and CLAP-based global label cleaning—yielding clean onset/offset annotations over a canonical vocabulary. Each item is then tagged with the applicable difficulty categories from the taxonomy of Section 3.0.0.4 (polyphonic overlap, gradual onset/offset, repeated occurrence, low-contrast, long duration, and semantic ambiguity) to enable stratified diagnostic evaluation. The tagging rules and thresholds are given in Appendix 13.

5.3 Human Verification↩︎

After automated annotation, all 3,427 candidate items undergo human review. An annotator examines each item and:

This review results in timestamp corrections and deletions of invalid items, producing a final benchmark of 3,427 items. All benchmark items are kept strictly disjoint from the Auto-AEG training splits.

5.4 Benchmark Statistics↩︎

2 Table 3 shows the source distribution. Audio clips range from 10–120 s, with target event durations from 0.5–60 s. The six difficulty categories are not mutually exclusive; items belonging to the Polyphonic Overlap category form the largest difficulty subset.

5.5 Comparison with Existing Benchmarks↩︎

Table ¿tbl:tab:benchmark95comparison? positions AEGBench against existing temporal audio grounding resources across five key dimensions. Prior benchmarks such as AudioGrounding [6] and AMR [7] support natural-language queries but cover only short clips and provide no difficulty stratification. CASTELLA [23] extends to long-form recordings yet lacks both hard-case annotations and an associated training split. TACOS [25] is the only prior resource to pair an annotated test set with a large-scale training corpus, but its evaluation targets contrastive retrieval rather than open-vocabulary event localization, and its benchmark clips remain under 30 seconds. AEGBench is the only benchmark to simultaneously support natural-language queries, long-audio clips, difficulty-stratified hard cases, real-world recording sources, and a paired Auto-AEG training split covering the same open label vocabulary.

2

6 Two-Stage Fine-tuning Framework↩︎

We use the Auto-AEG data to fine-tune LALMs in two sequential stages. The goal is not to introduce a new training algorithm, but to validate data quality: if our automatically constructed data improves temporal localization on the independently-annotated AEGBench, the pipeline has produced useful training signal. Both stages share a single prompt template (Figure 2, Appendix 10) that elicits an explicit <think> reasoning trace before the final <answer>.

6.1 Stage 1: SFT Cold-Start↩︎

6.1.0.1 Backbones.

We train Qwen3-Omni-30B-A3B-Thinking (hereafter Q3-Omni) and Qwen2.5-Omni-7B [33] (hereafter Q2.5-Omni) using QLoRA 4-bit NF4, compute dtype bfloat16, LoRA rank \(r{=}16\), \(\alpha{=}32\), applied to \(\{q,k,v,o,\text{gate},\text{up},\text{down}\}\_\text{proj}\) in the language-model component. The audio tower is kept frozen and cast to bfloat16.

6.1.0.2 Data and Format.

SFT is performed on the 10,000 synthetic clips from Stage 1 of Auto-AEG, split 9:1 into train/val by random seed. Each example uses the List-All query format, with a <think> scaffold that states the queried event, the total clip duration, and enumerates each ground-truth occurrence by its onset/offset interval—exposing the temporal structure of each GT annotation in natural language and stabilizing training on the clean synthetic data.

6.1.0.3 Training Details.

3 epochs, learning rate \(2{\times}10^{-4}\), batch size 2, gradient accumulation to effective batch 16. Best checkpoint selected by validation loss.

6.2 Stage 2: GRPO Fine-Tuning↩︎

Starting from the Stage 1 SFT checkpoint, we apply GRPO [37] on the 5,244 real-world queries from Stage 2 of Auto-AEG.

6.2.0.1 Reward Function.

The total reward combines four components: \[r = 0.65\,r_{\text{iou}} + 0.15\,r_{\text{fmt}} + 0.05\,r_{\text{nem}} + 0.15\,r_{\text{prec}},\] where:

• \(r_{\text{iou}}\): F1-IoU@0.5—the harmonic mean of Recall-IoU@0.5 and Precision-IoU@0.5. Using F1 rather than pure recall simultaneously penalizes missed detections and hallucinated intervals, preventing the degenerate “dense-fill” strategy.

• \(r_{\text{fmt}}\): Format reward—presence of <think> (\(+0.3\)) + <answer> (\(+0.3\)) + valid JSON array (\(+0.4\)).

• \(r_{\text{nem}}\): Non-empty reward—\(+1\) if the prediction contains at least one interval, lightly penalizing over-rejection.

• \(r_{\text{prec}}\): Precision penalty—linearly decays when the ratio of predicted to ground-truth interval count exceeds 2; reaches zero at ratio \(= 4\). This explicitly suppresses the tendency to pad outputs with spurious short intervals to inflate recall.

Because Stage 2 retains only labels that PE A-Frame successfully localizes (Section 4.3), every GRPO training query has at least one ground-truth interval (\(|\mathcal{Y}^*|\geq 1\)). The count-based terms \(r_{\text{nem}}\) and \(r_{\text{prec}}\) are therefore always well-defined, and we use the no-rejection query variant (Appendix 10). The policy’s ability to reject absent events is consequently evaluated on AEGBench (via Event F1 and precision) rather than trained as an explicit reward objective. GRPO advantage is computed as \(\text{adv}_i = (r_i - \bar{r}) / (\text{std}(r) + \varepsilon)\), with KL-divergence regularization to the frozen SFT reference policy (coefficient \(\beta{=}0.04\)). Training runs for 3 epochs with DeepSpeed ZeRO-2 on 2 GPUs, learning rate \(5{\times}10^{-5}\), \(G{=}4\) rollouts per sample.

6.3 External Zero-Shot Baselines↩︎

We evaluate four external zero-shot baselines to anchor the trained models against the broader landscape: Gemini-3-Pro (closed-source, accessed via an OpenAI-compatible multimodal endpoint with base64-encoded audio), Kimi-Audio-7B-Instruct [41] (same-scale open-source audio LLM with native temporal-grounding support), Qwen2-Audio-7B-Instruct [31] (a same-family predecessor of our Qwen-Omni backbones), and Audio Flamingo Next [5] (the next-generation open-source audio LLM in the Audio Flamingo series). All baselines receive the same List-All prompt and are not fine-tuned. We note that Audio Flamingo Next rarely emits intervals under this prompt (onset recall \(0.046\)); its high onset precision (\(0.919\)) therefore reflects a small number of high-confidence predictions rather than strong coverage, so we report it for completeness and do not draw strong conclusions from its precision.

7 Experiments↩︎

7.1 Main Results on AEGBench↩︎

Table ¿tbl:tab:main95results? reports open-vocabulary AEG results on the full AEGBench (\(n{=}3{,}427\) items, 5,275 queries). All SFT+GRPO models use the Auto-AEG pipeline data only; no AEGBench items appear in training.

2

7.1.0.1 SFT Contribution.

Synthetic-data SFT alone already delivers substantial gains: \(+34.4\%\) relative mIoU for Q3-Omni (\(0.276 \to 0.371\)) and \(+30.9\%\) for Q2.5-Omni (\(0.324 \to 0.424\)). This confirms that the programmatic synthesis pipeline produces learnable temporal grounding signal even without any real-world annotation.

7.1.0.2 GRPO Contribution.

The effect of Stage 2 GRPO differs markedly between the two model families. For Q3-Omni, GRPO delivers consistent across-the-board improvements: mIoU rises from \(0.371\) to \(\mathbf{0.480}\) (\(+29.4\%\) relative), ev_F1 from \(0.343\) to \(0.524\), and onset precision from \(0.368\) to \(0.559\)—the model learns to commit to genuine boundaries while simultaneously improving recall. For Q2.5-Omni, the picture is more nuanced: overall mIoU slightly decreases (\(0.424 \to 0.399\)), yet event-level F1 climbs from \(0.416\) to \(0.474\) and onset precision rises substantially from \(0.411\) to \(0.594\). This divergence suggests that GRPO pushes the smaller 7B model toward a high-precision, lower-recall regime, trading overall interval coverage for sharper boundary estimates. The precision component of the reward is the likely driver: Q2.5-Omni responds by suppressing over-prediction more aggressively than Q3-Omni, at a cost to recall.

7.1.0.3 Cross-Generation Analysis.

Q3-Omni (30B) achieves the best overall result (\(\text{mIoU}=0.480\) at SFT+GRPO), while Q2.5-Omni (7B) peaks earlier at the SFT stage (\(\text{mIoU}=0.424\)). The fact that GRPO improves Q3-Omni on all metrics but produces a precision–recall trade-off for Q2.5-Omni suggests that the reward signal from our noisy real-world annotations is more readily exploited by the larger model, whose stronger priors allow it to improve boundary precision without sacrificing recall.

7.2 Hard-Case Analysis↩︎

Table ¿tbl:tab:hardcase? breaks down performance by the six difficulty categories defined in Section 5. Long Duration (LD) is the hardest category across all models: even Gemini-3-Pro scores only 0.194 on LD compared to 0.327 on GO, and the best trained model (Q3-Omni SFT+GRPO) reaches only 0.277—consistent with the 30 s encoder window limit imposing a hard ceiling on long-event localization. Gradual Onset/Offset (GO) and Low-Contrast (LC) are comparatively tractable after training, with Q3-Omni SFT+GRPO reaching 0.491 and 0.482 respectively, both surpassing Gemini-3-Pro on these categories. The precision–recall asymmetry observed in the main results recurs here: for Q2.5-Omni, GRPO improves LC and LD (where boundary sharpness matters most) but slightly hurts PO, GO, RO, and SA relative to SFT.

7.3 Evaluation on SED benchmark↩︎

Evaluating whether the grounding ability acquired through Auto-AEG training also improves performance on the conventional SED task it extends can further solidify our claims. We test this on DESED [20], a standard domestic- environment SED benchmark with a fixed 10-class vocabulary, evaluated under event-level F1 and precision at IoU\(=\)​0.5 together with mean IoU over matched events. All models receive the same List-All prompt; no SED-specific head or training signal is used, so any change reflects the grounding policy itself.

2 DESED confirms that Auto-AEG training improves performance on the standard SED task that grounding extends. On the primary event-level SED metrics, GRPO beats the zero-shot baseline for both models: event-F1 rises (Q3-Omni \(0.254 \to 0.287\); Q2.5-Omni \(0.228 \to 0.263\)) and event-Precision rises more sharply (Q3-Omni \(0.508 \to 0.607\); Q2.5-Omni \(0.298 \to 0.463\)). The gain is precision-driven—GRPO commits to fewer, better-localized events rather than denser predictions—which is exactly the behavior the interval-aware reward is designed to elicit. For the larger Q3-Omni the improvement extends to overall matched-IoU (\(0.509 \to 0.606\)); for the smaller Q2.5-Omni matched-IoU stays near the zero-shot level, consistent with the cross-generation finding that the 30B model exploits the reward signal more fully. Because DESED clips are only 10 s—well within the 30 s encoder window—these gains reflect boundary- localization improvement rather than any long-context advantage.

The stage-level breakdown shows that GRPO, not SFT, drives the gains. SFT alone regresses on the event metrics for both models (Q3-Omni event-F1 \(0.254 \to 0.245\), precision \(0.508 \to 0.409\); Q2.5-Omni event-F1 \(0.228 \to 0.189\)): the synthetic cold-start teaches the open-vocabulary List-All format, but its synthetic acoustics do not match DESED’s real domestic recordings. GRPO then recovers and surpasses zero-shot on event-F1 and event-Precision for both models. This recovery—turning imperfect real-audio pseudo-labels into a usable reward signal—is precisely the role GRPO plays in the two-stage design, and indicates that audio event grounding and sound event detection rest on a shared temporal-localization capability that the pipeline strengthens: the grounding policy trained on open-vocabulary data transfers back to the closed-set SED task it extends.

8 Conclusion↩︎

We presented Auto-AEG, a scalable pipeline for automatically constructing open-vocabulary onset/offset supervision, paired with AEGBench, an independently annotated, difficulty-stratified benchmark for evaluating LALM temporal grounding. Fine-tuning on Auto-AEG data with a two-stage SFT cold-start and interval-aware GRPO framework yields \(+73.9\%\) relative mIoU for Qwen3-Omni-30B and \(+23.1\%\) for Qwen2.5-Omni-7B over zero-shot baselines; GRPO consistently amplifies the larger model across all metrics, while on the smaller 7B model it trades recall for sharper boundary precision. AEGBench’s independence from the training pipeline—enforced through energy-contrast filtering, multi-phase label cleaning, and human correction—ensures that observed gains reflect genuine temporal grounding rather than annotation-pipeline imitation. The Auto-AEG pipeline can in principle be applied to any large public audio collection, establishing a viable data-side foundation for continued advances in LALM temporal capability.

9 Limitations↩︎

The Auto-AEG Stage 1 data is synthesized from FreeSound clips, which introduce a domain mismatch relative to naturally-recorded real-world audio. While GRPO partially corrects for this by exposing the model to real audio, the Stage 1 SFT checkpoint is initialized on synthetic distributions that differ from real recordings in room acoustics, source interaction, and background statistics.

Stage 2 pseudo-labels contain localization errors from PE A-Frame (40 ms frame resolution (\(\pm\)​20 ms boundary uncertainty)) and potential label errors from Gemini (hallucinated or imprecise descriptions). GRPO is noise-tolerant to these errors, but a more comprehensive noise analysis—e.g., measuring pseudo-label quality against held-out human timestamps—would provide stronger guarantees.

Audio recordings with long duration remain a challenge for most LALMs, which employ Whisper-based audio encoders that are limited to 30 s per encoding window.