1 Introduction↩︎

Automatic speech segmentation is the task of identifying where boundaries between speech units occur [1]. When these units are delimited by prosodic cues, they are commonly described as intonation units and play several roles in spoken communication, including structuring discourse and organizing information flow [2].

Earlier automatic approaches to prosodic segmentation relied on more traditional machine-learning pipelines. For instance, [3] combined acoustic information with syntactic post-processing and reported effective boundary detection for English and Russian. More recently, prosodic segmentation has also benefited from large pretrained speech models. [4] introduced PSST!, a Transformer-based approach that fine-tunes Whisper [5] to produce a transcription and mark intonation-unit boundaries within the same output sequence.

For Brazilian Portuguese (BP), [6] adapted heuristic methods originally proposed for English [7], relying mainly on speech rate and silent pauses. Such rules are simple and efficient, but their performance may degrade when parameters are not adapted to the target language or when the audio quality is low. [8] used Linear Discriminant Analysis to identify terminal and non-terminal breaks in BP, while [9] proposed a Random Forest classifier based on acoustic features. These studies show that acoustic cues such as fundamental-frequency movement, tessitura changes, and pauses are useful, but they also suggest that some BP intonation units remain difficult to detect with traditional feature-based methods [10].

This gap motivates the use of large pretrained speech models for BP. Because each language organizes intonation in specific ways, the segmentation strategy that works well for one language cannot be assumed to transfer directly to another [11]. We therefore propose SAMPA(Segmenter for Automatic Marking of Prosodic boundAries in Brazilian Portuguese), a deep learning-based segmenter adapted from PSST! [4]. The research question guiding this paper is: how accurately can a Whisper-based model identify terminal intonation-unit boundaries in Brazilian Portuguese speech? The main contributions of this study are as follows:

• We introduce SAMPA, a deep learning-based automatic prosodic segmenter for Brazilian Portuguese adapted from PSST! [4].

• We prepare training and evaluation data from manually segmented BP corpora and compare several audio-filtering configurations.

• We provide reproducible evaluation metrics for boundary detection and analyze model behavior using both quantitative scores and qualitative linguistic evidence.

The processed datasets, training and evaluation code, and trained models will be released on the Hugging Face Hub upon acceptance.

2 Data↩︎

We use two BP datasets with manually annotated prosodic segmentation, both representing the São Paulo variety of Brazilian Portuguese. The first dataset is the CORAA NURC-SP Minimal Corpus (MC) [12], which comprises data from the NURC (Norma Urbana Linguística Culta) project [13]. The original recordings were made on reel-to-reel tape in the 1970s and later digitized by the TaRSiLA project.¹ The dataset contains 21 audio files with transcriptions aligned to intonation units, along with metadata describing speakers and audio quality. Six annotators performed and reviewed the segmentation following the intonation-unit framework adopted in C-ORAL-BRASIL [2].²

The second dataset is CATNA (Corpus de Áudios e Transcrições Não-Alinhadas) [14], which is publicly maintained by the TaRSiLA project. CATNA has the same general configuration as the NURC-SP MC and contains 26 recordings. Five recordings, however, were incomplete, while the rest included an initial audio header that was not annotated in the corresponding Praat TextGrid files. TaRSiLA members provided corrected versions that allowed us to remove the headers and align the audio with the TextGrid annotations. Among those, 11 recordings that had been split into multiple files were reconstructed by merging their parts, and 10 other recordings were trimmed to remove the header. We refer to this processed dataset as CATNA-MT (Merged and Trimmed).

3 Model↩︎

SAMPA is obtained by fine-tuning the Whisper Large-v3 model [5] for the task of prosodic boundary prediction in Brazilian Portuguese, following the methodology proposed by [4] for English. This section describes the base model and the adaptation strategy adopted in this work.

Whisper [5] is an automatic speech recognition (ASR) system based on the Transformer encoder-decoder architecture [15], trained in a weakly supervised fashion on a large multilingual corpus of 680,000 hours of audio collected from the internet. This scale of training confers high robustness to different acoustic conditions and accents, as well as competence across dozens of languages, including BP. In this work, we use the large-v3 variant,³ which achieves the best performance for BP at the time of writing.

The key idea behind SAMPA is to cast prosodic segmentation as an extension of speech transcription. Instead of producing only lexical text, the model is trained to output a transcription that includes explicit boundary markers. Thus, transcription and prosodic segmentation are learned within a single sequence-to-sequence architecture. During fine-tuning, terminal boundary positions are represented by a special delimiter token (“!!!!!” in our case) inserted between transcribed words. For example, an audio clip containing two consecutive intonation units is represented as “first unit !!!!! second unit”. Importantly, this formulation requires no architectural changes to Whisper since the delimiter is added to the vocabulary and treated as a regular decoder output token. As a result, the model can use acoustic information from the encoder and linguistic context from the decoder when deciding where terminal boundaries occur. Figure 1 illustrates the resulting architecture.

4 Experimental Details↩︎

Our experiments were designed to transform the original datasets into a format compatible with the training procedure adopted in the PSST! framework. We first parsed the annotation files to extract the start time, end time, speaker, and text associated with each annotated segment. The long original recordings were then converted into shorter audio samples while preserving the alignment between speech and text. To generate the final training samples, consecutive segments belonging to the same speaker were concatenated whenever the resulting audio duration remained below 30 seconds. This constraint was adopted because Whisper processes audio inputs with a maximum duration of 30 seconds; longer recordings are automatically truncated by the model, which could result in the loss of prosodic boundary information located near the end of the audio segment. We describe the full preprocessing pipeline next.

4.1 Preprocessing pipeline↩︎

Our preprocessing pipeline grouped adjacent segments according to three conditions: (i) preservation of the original file ordering, (ii) speaker continuity, and (iii) maximum segment duration constraint. When all conditions were satisfied, the textual contents of the segments were concatenated using the delimiter token “!!!!!”, representing a prosodic boundary marker. Otherwise, a new segment was created. This procedure enabled the creation of a dataset that, when fine-tuning the ASR model, allowed the model to learn to apply the delimiter token in the transcribed text when there is a prosodic boundary in the original audio. After concatenation, each generated sample contained the following information: file identifier, speaker identification, initial timestamp, final timestamp, total duration, indices of the original concatenated segments, and the resulting text with embedded prosodic boundary markers. However, since the original audios were recorded in tape, a portion of the recordings were characterized by a significant amount of noise, making the samples derived from these recordings unsuitable for training. Therefore, a manual process was done to check which recordings were too noisy to be converted into training samples.

Four different recordings (\(\sim9\%\) of the original data, \(\sim3\) hours of speech) were held out to generate the test split. The recordings were chosen so that there would be two audios in which the main speaker was male and two in which it was female, with the aim to minimize gender bias. Furthermore, each gender has one clean recording and one with moderate noise, so that we could analyze the effect of noise in the inference process. The rest of the data (\(\sim29\) hours) was used to generate the train/validation split. The resulting dataset was converted into the format required for Whisper-based training. Audio signals were resampled to 16 kHz and paired with the corresponding transcriptions containing the prosodic boundary markers.

For out-of-distribution evaluation, we use MuPe-Diversidades [16], which is also prosodically segmented. This dataset contains 2.5 hours of audio and differs from the NURC-SP/CATNA data in two important ways: its recordings are substantially cleaner, and its speakers are more geographically diverse. Whereas most NURC-SP speakers were born in the state of São Paulo and therefore tend to represent the São Paulo variety, MuPe-Diversidades includes samples from 17 Brazilian states. Table 1 describes the characteristics of our datasets after preprocessing.

4.2 Training↩︎

Because several recordings in the training data contain noise, we also created alternative dataset versions by applying digital filters. Low-pass (LP) and high-pass (HP) filters were applied with different cutoff frequencies, yielding five training configurations:

1. original, unfiltered audio;

2. LP filter with a 3,200 Hz cutoff;

3. HP filter with a 400 Hz cutoff;

4. HP filter with a 600 Hz cutoff;

5. data augmentation with unfiltered, LP 3200 Hz, HP 400 Hz, and HP 600 Hz samples.

The cutoff frequencies were selected from preliminary experiments with a model trained on unfiltered data, where these settings produced the highest macro \(F_1\) values on filtered versions of the test audio. We trained five independent models, one for each configuration above. From the training data, 5% of the segments were randomly held out for validation. All models were trained for 4 epochs. Following PSST! [4], the learning rate was kept low during the initial warm-up steps (about 7% of the total steps) to reduce early overfitting and then increased linearly to \(10^{-5}\).

5 Results and Discussion↩︎

Prosodic boundary detection is evaluated as a binary classification problem: for each potential boundary position between corresponding words, the model must decide whether a terminal prosodic boundary is present. Following [10], we compare each generated token sequence with its reference by checking the presence or absence of the boundary marker between aligned words. We report Precision, Recall, binary F1 (\(F_{1}\)), and macro F1 (\(F_{1}\)). The binary F1 score is computed over the positive class, namely boundary positions. The macro F1 score averages the F1 scores of the boundary and non-boundary classes, penalizing both excessive insertion and omission of boundaries. We also report Word Error Rate (WER) to monitor whether segmentation fine-tuning harms transcription quality.

5.1 In-domain evaluation↩︎

Table 2 presents the best test-time filtering configuration for each training setup on the held-out test split. The strongest model by a small margin is the LP 3200 Hz training configuration, with a WER of \(0.103\), binary and macro \(F_{1}\) of \(0.731\) and \(0.858\), respectively. Overall, the boundary scores are very close across training configurations, with small variances (around 0.2%). These results indicate that, for the held-out data, training on filtered audio does not yield a large advantage over training on the original noisy data.

5.2 Out-of-domain evaluation↩︎

Table 3 reports the best configuration for each model on MuPe-Diversidades. Unlike the held-out test split, the out-of-distribution evaluation shows clearer variation across training setups. The HP 600 Hz model obtains the best binary \(F_{1}\) (0.796), while the HP 400 Hz and HP 600 Hz models tie for the best macro \(F_{1}\) (0.890). The LP 3200 Hz and data-augmentation models obtain lower boundary scores, suggesting weaker generalization to this cleaner and more geographically diverse dataset. At the same time, the absolute differences among the three strongest models remain small.

6 Low and high-pass filter analysis↩︎

Figure 2 (top) shows the performance of the LP 3200 Hz model under different test-time filters. The best filtered condition and the unfiltered baseline are nearly identical, differing by approximately 0.1 percentage points in \(F_{1}\). This pattern suggests that test-time filtering is not the main factor driving in-distribution boundary detection performance. Figure 2 (bottom) shows the HP 600 Hz model across test filters on MuPe-Diversidades. As in the held-out test split, the unfiltered condition remains close to the best condition, with a difference of about 0.3 percentage points in terms of binary \(F_{1}\).

6.1 N-gram analysis↩︎

To better understand model behavior, we performed an n-gram analysis of the linguistic contexts in which SAMPA marked boundaries. We compared the training configurations by examining correct boundary detections (true positives, TPs) and incorrectly inserted boundaries (false positives, FPs). The results are shown in Table 4.

The distributions were similar across the unfiltered, high-pass, low-pass, and data-augmentation configurations. For TPs, unigrams immediately before predicted boundaries included discourse markers, as né, and frequent nouns such as cinema, filme, and casa; unigrams immediately after TPs included conjunctions such as e, mas, and porque.

For unigrams, the frequent occurrence of discourse markers in utterance-final position suggests that SAMPA relies on prosodic information to mark boundaries, since the position of these elements affects their prosodic realization [17]. The presence of nouns before the TPs and connectives after them also points to the use of morphosyntactic cues, as connectives tend to be function words [18]. When analyzing trigrams, the presence of subject-verb constructions and the recurrence of speakers’ opinions or explanations suggest that syntactic and semantic information have influenced SAMPA’s decisions.

3pt

6.2 Prosodic analysis↩︎

Finally, we conducted an acoustic-visual analysis of correct and incorrect segmentations to qualitatively inspect the boundaries predicted by SAMPA. We examined test recordings in Praat [19] and analyzed the cases in which the model inserted a boundary between intonation units. Figure 3 illustrates a true positive example (top), where several prosodic cues support the reference boundary, along with a false positive example (bottom), where SAMPA inserts a boundary that is not present in the reference annotation.

In the true positive example, the F0 contour shows a continuous fall at the end of the first terminal unit (eu tenho ido bastante), a pattern typical of BP declaratives [20]–[22]. The pitch range also changes between the two units: the first ends in a lower range, and the following unit begins after a reset to a higher region. These cues are accompanied by a long silent pause. Together, they provide strong evidence for a terminal prosodic boundary.

In contrast, in the false positive example, the F0 curve falls on casa (marked by the arrow), but this local contour alone is not enough to characterize a terminal boundary. Differences in F0 before and after a candidate boundary can be relevant, but they are not sufficient to determine the boundary on their own [23]. Human annotation also involves perceptual judgment and the informational value of the unit [24]. More generally, intonation-unit identification is difficult because several phonetic-acoustic cues may contribute to boundary perception, and there is no full consensus on which cues distinguish different boundary types or how they interact [23].

A pause is also present in this example, but pauses may occur at both terminal and non-terminal boundaries [25]. They may create discourse effects such as expectation or emphasis and are neither sufficient nor always necessary for boundary marking [26]. Thus, pauses are important cues, but they can also lead a model to confuse boundary types [8]. In addition, the segment que já está dando uma dor de cabeça can be interpreted as an explanatory relative clause and could form an autonomous unit in some contexts. In this particular case, however, the annotators did not judge the prosodic and discourse evidence as sufficient to mark two distinct terminal units.

Overall, the false positive analysis suggests that SAMPA often generalizes from plausible acoustic and linguistic cues rather than inserting boundaries arbitrarily. Many FPs occur at positions that exhibit smaller boundary-like cues, even if they are not terminal intonation-unit boundaries in the reference annotation. This observation is consistent with the annotation framework, which considers not only acoustic evidence but also syntactic and pragmatic aspects of speech.

7 Conclusion↩︎

This paper presented SAMPA, a Whisper-based segmenter for terminal prosodic boundaries in Brazilian Portuguese. Following PSST!, by fine-tuning Whisper large-v3 to generate transcriptions with explicit boundary markers, SAMPA reformulates prosodic segmentation as a sequence-to-sequence speech transcription task. Across several filtering configurations, the best models reached \(F_{1}=0.731\) on the held-out NURC-SP/CATNA test split and \(F_{1}=0.796\) on the out-of-distribution MuPe-Diversidades dataset. Our experiments show that filtering choices have only a limited effect on the held-out test split, while high-pass training configurations generalize better to MuPe-Diversidades. The n-gram and acoustic-visual analyses further suggest that the model’s predictions are informed by morphosyntactic, semantic, and prosodic cues. At the same time, the false-positive cases reveal that boundary prediction remains sensitive to ambiguous cues such as pauses, local F0 movements, and syntactic structures that may signal weaker prosodic divisions. Future work should evaluate these error patterns more systematically and incorporate discourse-level information into automatic segmentation models.

Acknowledgments↩︎

We used AI tools, specifically ChatGPT 5.5 Pro, during the writing of this paper for English checking and polishing (grammar enhancement). We also used GitHub Copilot to assist with coding.

References↩︎