1 Introduction↩︎

Simultaneous interpretation (SI) is a task where an utterance is translated in real-time. Simultaneous machine translation (SiMT) systems should produce reasonably good translations with low latency [1], [2]. Due to the lack of a large-scale SI corpus, most SiMT systems are trained with offline machine translation (MT) corpora which are often abundant with easy access [3]. Yet, MT data is different from SI data because of the difference between offline and online translation in nature. Despite some efforts [4], it has always remained a question on how and to what extent SI data affects an SiMT system.

[4] recently made a call for constructing SI corpora to learn interpreters’ behaviours in modelling, but no corpus has been proposed as of now. This is mainly contributed by the fact that collecting and building parallel SI corpora is exceptionally costly and time-consuming [5]. Most existing corpora are in a small scale, mainly for testing purposes [4], [6]; see Table 1¹ for detailed statistics. In addition, aligning source and target sentences presents great challenges. [8] presented a large-scale document-level SI corpus, NAIST-SIC, and conducted analyses on a small manually-aligned subset. However, sentences are not aligned within each document pair for most of the corpus, so it cannot be used for model training due to the lack of alignment.

This work aims to fill the gap by building a large-scale parallel English-Japanese SI corpus named NAIST-SIC-Aligned. We start with the non-parallel NAIST-SIC and propose a two-stage pipeline alignment approach consisting of coarse and fine-grained alignment to make it parallel. The initial stage is coarse alignment which involves identifying minimal groups of source and target sentences that are considered translations of each other. The next stage is fine-grained alignment, where intra- and inter-sentence filtering techniques are applied over coarse-aligned pairs to improve data quality. We validate each step either manually or automatically, to ensure the quality of the data. Meanwhile, we compile a small-scale, manually curated SI test set for testing purposes. We additionally summarize alignment challenges and findings to guide future SI corpus construction for other language pairs. Lastly, we build SiMT systems based on our corpus and show significant improvement over baselines in both translation quality and latency.

2 Raw Corpus Construction↩︎

2.0.0.1 Raw SI data collection

We used a portion of SI data from NAIST-SIC in this study, referred to as SI\(^{RAW}\). This data comprises professional simultaneous interpreters’ real-time interpretations of TED talks, which span a variety of topics from technology to entertainment.² Interpreters with varying experience levels (S-rank: 15 years, A-rank: 4 years, B-rank: 1 year) participated in the interpretations. Note that SI\(^{RAW}\) underwent manual transcription. We specifically used interpretations by S-rank and A-rank interpreters, denoted as SI\(^{Srank\_RAW}\) and SI\(^{Arank\_RAW}\), respectively. Overall, the quality of SI\(^{Srank\_RAW}\) surpasses that of SI\(^{Arank\_RAW}\).

2.0.0.2 Manual subset alignment

[8] conducted sentence-level alignment on 14 talks (a subset of talks interpreted by all three rank interpreters) for analysis. This process involved manual alignment of source sentences with target sentences, referred to as true-align. We refer readers to the original paper for more details.

2.0.0.3 Error analysis

After a manual investigation on true-align, we categorise issues with the SI data into two groups: under-translation and mis-translation. Under-translation occurs when interpreters unintentionally omit content due to memory overload. Interpreters may also omit information intentionally, using tactics such as summarization, which is permissible. For instance, in the following interpretation, omitting "And if you drill into that" does not impede comprehension of the source sentence.

Mis-translation is the result of interpreters mistakenly rendering source sentences, often due to cognitive overload. The interpretation below clearly misrepresents the source sentence.

3 Parallel Corpus Construction↩︎

In this section, we present how we perform alignment on SI\(^{RAW}\) (§3.1). Then, we detail how we split data to train/dev/test (§3.2), followed by describing manual compilation of the test set (§[sec:test]).

3.1 Alignment↩︎

We propose a two-stage alignment method that the first stage involves coarse alignment, grouping minimal source and target sentences to establish translations, and the second stage refines these pairs through fine-grained alignment. Both stages are validated quantitatively and qualitatively, with insights and findings shared at each step.

3.1.1 Coarse Alignment↩︎

3.1.1.1 Alignment

Each talk consists of \(M\) source sequences, \(\left(e_{1}, e_{2}, \ldots, e_{M}\right)\), and \(N\) target sequences, \(\left(f_{1}, f_{2}, \ldots, f_{N}\right)\), where each \(e\) is at the sentence level, each \(f\) is at the chunk level, and \(N > M\). Note that there is no clear punctuation to group target sequences to sentence level, due to the nature of SI. The first step is to detect groups between these sequences that are translations of each other.³

Due to under-translation and mis-translation, some sequences lack corresponding translations, requiring deletion operations. We used the vecAlign [17] sentence aligner, which suits the above purposes. VecAlign creates source and target language sentences, compares sentence embeddings computed by LASER [18] using dynamic programming, and yields candidate pairs \(<E, F>\) with the lowest cost.⁴ Pairs with a cost higher than 1 or with empty \(E\) or \(F\) indicating no corresponding translation were removed.

3.1.1.2 Validation

The outputs of this stage are validated both qualitatively and quantitatively. Manual inspection reveals a relatively high alignment quality, particularly for SI\(^{Srank\_RAW}\). To perform quantitative evaluation, we use true-align to automatically generate coarse alignment from talks, denoted as auto-align, and then measure the extent to which true-align can be recovered. For a given source sentence, \(e\), we measure the similarity between the automatically aligned target sentence, \(F^a\), and the manually aligned sentence, \(F^m\), as follows: \[\setlength{\abovedisplayskip}{3pt} \setlength{\belowdisplayskip}{3pt} \setlength{\abovedisplayshortskip}{3pt} \setlength{\belowdisplayshortskip}{3pt} s =\frac{{LCS}(F^a, F^m)}{|F^m|}\] where \(LCS (A, B)\) stands for the longest common substring between strings \(A\) and \(B\).

We define \(f^a\) as correctly aligned to \(e\) when the similarity measure \(s\) exceeds a threshold \(\epsilon\). Higher \(\epsilon\) values result in lower accuracy. Notably, with \(\epsilon\) set to 0.8, the overall accuracy for the S-rank subset in true-align remains at 80%, underscoring vecAlign’s effectiveness in addressing the issue. We name the output at this stage Coarse, and we also prepare S-rank data as Coarse\(^{Srank}\).

3.1.1.3 Finding

After the manual check of the outputs, we summarise our main findings as follows.

• The quality of alignment results largely depends on the quality of interpretation data. SI\(^{Srank\_RAW}\) has a better quality of data than SI\(^{Arank\_RAW}\), and thus the aligned pairs of the former have higher quality.

• The quality of interpretation data is influenced by talk difficulty, which can be assessed from two perspectives: the speaker and the interpreter. A talk becomes challenging when the speaker speaks rapidly, employs jargon, has an accent, etc. It is also contingent on the interpreter’s skills and background knowledge, etc.

• Interpreters may exhibit varying performance levels for different talks. For easy talks, their performance remains consistent, contrasting the typical phenomenon of performance decline due to cognitive load over time. We attribute this to the interpreters working in a less stressful, simulated environment. In the case of difficult talks, the expected performance decline is observed.

3.1.1.4 Human Evaluation

We further analysed half of a talk from SI\(^{Srank\_RAW}\), denoted as sample (comprising of 78 pairs). Our findings indicate that 82% of the pairs are well-aligned, 11% are reasonably aligned, and the remainder are poorly aligned. Within the 11% subset, misalignment often occurs at the beginning/end of the target sentence, and removing it would enhance alignment quality.⁵ While these statistics may vary across talks, these observations guide our alignment design.

3.1.2 Fine-grained Alignment↩︎

The second stage is to perform fine-grained alignment consisting of two filtering stages. The purpose is to improve the accuracy of aligned pairs.

3.1.2.1 Intra-sentence filtering

Based on our observations, we decide to filter out the beginning and last chunk of a target sentence if it does not carry any substantial meaning. For example,

(meaning “then” in English) often appears as a filer in Japanese and should be removed. We consider Japanese words conveying important information (i.e., content words): “NOUN”, “PROPN”, “PRON”, “VERB” or “NUM”. Manual inspection of the output from this step on sample shows that this simple heuristic accurately detects and removes chunks that contain no content words in most of the time. We call the resulting data together the S-rank subset Intra and Intra\(^{Srank}\).

3.1.2.2 Inter-sentence filtering

To address potential quality disparities in Coarse alignment for talks other than Sample, we introduce rigorous surface-level and semantic-level filtering rules. i) For an aligned pair <\(E\), \(F\)>, we calculate \(\alpha\), representing the percentage of content words in \(E\) covered by \(F\). Content words include "NOUN," "PROPN," and "NUM,"; "VERB" is excluded due to its perceived lesser importance during interpretation [19]. This accounts for interpreters’ techniques aimed at managing cognitive load, allowing for certain word omissions. ii) We compute the length ratio \(\gamma\) between \(F\) and its corresponding offline translation \(T\), generated via Google Translate from \(E\). Large or small \(\gamma\) indicate potential mis-translation and under-translation issues. iii) We measure semantic similarity, \(\eta\), between \(F\) and \(T\) with BLEURT [20], a semantic-based MT metric. As \(T\) is the correct translation of \(E\), high \(\eta\) implies high coverage of \(E\) by \(F\). We denote the entire data and S-rank data as Inter and Inter\(^{Srank}\), respectively.

3.1.2.3 Validation

Each of the above steps has been manually validated by one of the authors. That said, the optimal values for \(\alpha\), \(\gamma\) and \(\eta\) vary by talk, and we leave automatically learning optimal values per talk for future work.

3.2 Data Split↩︎

Both our data and MuST-C [21], a common dataset in SiMT (both speech and text), contain TED Talks. To prevent potential data contamination, we ensured our train set (831 talks) is present in Must-C train data. We created training set variations for experimentation. For evaluation, we selected 10 and 15 talks from Intra as our dev and test sets, referred to as AUTO-DEV and AUTO-TEST. Importantly, these sets do not intersect with MuST-C training data. Additionally, we curated manual dev and test sets for testing.

To guarantee the quality of the dev and test sets, we randomly selected four and five talks interpreted by S-rank interpreters for annotation for the subsets.⁶ Data curation involves a two-step process.

3.3.0.1 Label aligning

We observed that a correctly aligned pair may not be suitable for evaluation due to issues such as under-translation or mis-translation. For instance, in the following pair, the target SI sentence aligns correctly with the source sentence but is not a correct translation.

• Source: They haven’t given up on government.

• Interpretation:

  UTF8min政府は、諦めていないのです。

  [The government hasn’t given up.]

Another instance involves words with similar pronunciations, especially for proper nouns. While hearing them in a speech may be appropriate, using them in written text could lead to errors. Hence, we instructed the annotators to assign two labels: good_align for well-aligned pairs, and good_mt for suitable for evaluation. Assigning the good_align label is generally straightforward and less ambiguous. However, assigning the good_mt label requires a judgement call; annotators collaborated on this task to reach a consensus and minimize bias. It is unrealistic to expect perfect interpretations; therefore, the principle for annotation dictates that omitting important content (e.g., numbers and proper nouns) is not allowed, but omitting less important content is permissible.

For the above example, the labels would be True and False, respectively.

3.3.0.2 Sentence editing

We further asked annotators to perform two sequential tasks on the target sentence. (1) Rectify alignment issues from automatic alignment. This is mainly done by introducing "true COARSE" for the test set if chunks are removed from or re-introduced to the target sentence as a result of auto alignment. We note that the need of merging sentences is rare; even in the case of merging, the resulting sentences do not lead to good pairs. This can be explained by human’s behavior that when interpreters have to merge sentences, they make mistakes more easily. (2) Perform minimal manual post-editing to ensure faithfulness and reliability without altering the data distribution significantly. Examples include, but not limited to, replacing pronouns with referred entities, removing self-correction to improve fluency, and correcting numbers.

In the example below, the annotator would add the text highlighted in red to the target sentence.

• Mod_Interpretation:

  UTF8min彼らは政府を諦めていないのです。

  [They haven’t given up on government.]

The entire process, consuming approximately 20 hours, resulted in SI\(^{TEST}\). Additionally, we manually annotated three talks for the dev set, creating SI\(^{DEV}\). Data statistics are provided in Table 2.

4 Experiments↩︎

4.1 Experimental settings↩︎

4.1.0.1 Datasets

In addition to our proposed data shown in Table 2, we also complemented it with the Must-C v2 En-Ja dataset, which is an offline dataset with 328,639 training instances.

4.1.0.2 baselines

We applied test-time wait-k [1] on offline MT models trained on offline (i.e., Must-C) and various online datasets, including Coarse, Coarse\(^{Srank}\), Intra and Intra\(^{Srank}\), as our baseline models.

4.1.0.3 Implementation details

For the wait-k systems, we primarily followed the fairseq toolkit [22] instructions with some distinctions. We used separate vocabularies for English and Japanese with SentencePiece [23] which was trained on Must-C training data. We set the vocabulary size to 8,000, as larger or smaller sizes yielded worse results. We used a batch size of 7,168 with the update frequency of 4, and set the dropout rate on 0.3. Early-stopping occurred after 16 epochs if validation loss stagnated. We selected the best checkpoint based on the loss on SI\(^{DEV}\) for all SI experiments. We evaluated model performance on SI\(^{TEST}\) with the SimulEval toolkit [24] on SI\(^{TEST}\) where translation quality is measured in BLEU⁷ and latency in average lagging (AL) [1]. We also measured translation quality with BLEURT. All models start from an offline MT system trained with MuST-C.

4.2 Results↩︎

Figure 1 shows BLEU and AL scores in a set of k values (i.e., 5, 7, 9, 11, 13). SiMT systems trained Inter and Inter\(^{Srank}\) outperform models trained on Must-C significantly, by an average of 6.43 and 5.74 BLEU scores, respectively, across all latency settings. This demonstrates the effectiveness of SI data despite Must-C En-Ja having significantly more data than our corpus. BLEURT scores confirm that both Inter and Inter\(^{Srank}\) offer the highest translation quality.⁸

5 Conclusion↩︎

The question of how simultaneous interpretation (SI) data impacts simultaneous machine translation (SiMT) remains unresolved. This is due to the lack of large-scale SI training data, constructing which imposes great challenges. In this work, we filled in the gap by introducing NAIST-SIC-Aligned, an automatically aligned English-Japanese SI corpus, together with a small-scale human annotated SI test set. We proposed a two-stage alignment approach to align source data with target SI data. Our results show that systems trained on our proposed data surpassed baselines by a large margin. We share findings and insights with SI corpus, hoping to offer guidance on future research on SI corpus construction.

Acknowledgements↩︎

Part of this work was supported by JSPS KAKENHI Grant Number JP21H05054.

6 Bibliographical References↩︎

References↩︎