1 Introduction↩︎

The progress in text-based Large Language Models (LLMs) has significantly impacted the field of generative AI, allowing for smooth communication between humans and AI via written text. However, since natural human interactions often involve spoken language, there is a growing movement to integrate speech capabilities into LLMs [1]. This initiative aims to enable direct speech communication between humans and AI by endowing language models with the ability to understand and produce speech. Consequently, end-to-end Spoken Language Models (SLMs), which engage directly in spoken dialogue, represent an exciting and promising direction.

Evaluation of SLMs is crucial for ensuring their effectiveness and reliability in real-world applications. Similar to Text Language Models (TLM), the ability to understand world knowledge is essential for SLMs to perform conversations with humans. When evaluating SLMs’ knowledge comprehension, several crucial factors should be taken into account: 1) The evaluation should be conducted entirely in speech format, meaning both the input and output must be audio-based. This approach differs from typical AudioQA tasks [2]–[5], where audio clips are matched with text questions. SLM evaluation should solely use speech to mimic real-world conversational settings. 2) The evaluation should consider various input audio conditions. Users may interact with SLMs under diverse audio characteristics, such as different speaker timbres, audio qualities, and speaking styles. Consequently, SLMs should demonstrate robustness against these varying conditions, producing consistent answers when the input content remains the same [6]. 3) Not all knowledge comprehension topics are straightforward for SLM evaluation. The range of knowledge understanding includes diverse subjects, including literature, sociology, engineering, etc. However, some subjects present unique challenges when evaluated in the spoken format. For instance, while math expressions can be easily and concisely written in text, evaluating them in spoken form remains a challenge. Figure 1 illustrates the three key elements mentioned above.

Based on the key elements outlined, we introduce VoxEval, a speech question-answering benchmark designed to assess the knowledge understanding capabilities of SLMs. This benchmark features audio recordings for both questions and instructions, facilitating a comprehensive evaluation of speech modeling abilities. Additionally, VoxEval incorporates a variety of input conditions for the same question, allowing us to test the robustness of SLMs across different audio scenarios. To address the challenges associated with evaluating certain subjects, such as mathematics, we pinpoint the critical differences between written and spoken formats and convert the former into their spoken counterparts. Our evaluation of recently proposed SLMs using VoxEval reveals that it presents a significant challenge for existing models. Additionally, existing SLMs demonstrate vulnerability to changes in input audio conditions, indicating the need for future research to focus on improving robustness. To summarize our contributions:

• We introduce a novel benchmark, VoxEval, specifically designed to evaluate the knowledge understanding capabilities of SLMs exclusively in speech format, across a diverse range of audio conditions.

• We pioneer the assessment of SLMs’ math-solving abilities, addressing a critical gap in existing evaluation methodologies.

• We conduct a systematic evaluation of VoxEval with several recent SLMs, revealing that current models exhibit significant limitations in performance on our benchmark.

2 Related Work↩︎

2.1 Speech Large Language Models↩︎

Speech Large Language Models (SpeechLLMs) refers to the LLMs that have the ability to understand and/or generate speech. There are two primary approaches for integrating speech modalities into LLMs. The first approach focuses on making LLMs understand speech information. This is usually achieved by connecting a speech encoder to a pre-existing LLM while preserving most of the model’s original structure and parameters. We refer to these as Speech-to-Text Large Language Models (S2TLLMs). Since the primary goal of these approaches is to enable existing LLMs to perceive auditory information, it only supports text as the output modality [7], [8]. For example, Qwen-Audio [7] utilizes Whisper-large-v2 [9] as the audio encoder and Qwen-7B [10] as the LLM. SALMONN [8] uses both whisper and BEATs [11] as the audio encoders and uses Vicuna [12] to serve as the LLM.

In contrast, the second approach advances further by developing end-to-end speech interaction models [13]–[15]. In these models, both the input and output modalities are speech, allowing the model to both hear and speak. These approaches directly model speech throughout the LLM-based pipeline and do not significantly rely on textual information. We refer to these models as End-to-End Spoken Language Models (SLMs). SLMs typically involve three components to enable end-to-end speech interactions: tokenizer, language model (LM), and vocoder. Given an input speech, the tokenizer [16]–[18] first encodes the audio waveform into speech tokens or representations, LM [19]–[21] then autoregressively generate token response, and vocoder [22]–[24] synthesizes the output token back into speech waveform.

2.2 SpeechLLM Evaluation Benchmarks↩︎

The evaluation of SpeechLLMs centers on their ability to model speech data, with researchers examining various aspects and levels. Initially, multiple datasets are utilized to assess the linguistic modeling capabilities of SpeechLLMs. For instance, sWUGGY [25] is employed for lexical evaluation, sBLIMP [25] for syntactical assessment, and sStoryCloze [26] for semantic evaluation. Additionally, the modeling of paralinguistic elements (spoken content beyond words) is crucial for SpeechLLMs, prompting the development of datasets that evaluate paralinguistic information from both token and perceptual levels [14], [27]. Beyond the modeling aspect, the performance of downstream applications is vital for the practical use of SpeechLLMs in real-world scenarios. Recently, numerous downstream evaluation benchmarks have emerged [2]–[5], focusing on general audio understanding, including speech, environmental sounds, and music. However, as highlighted in Section 1, these benchmarks contain all QA pairs in text format, making them suitable for evaluating S2TLLMs but not SLMs. The closest benchmark to our work is VoiceBench [28], which also evaluates end-to-end SLMs under various audio conditions. However, our study emphasizes assessing the knowledge understanding capabilities of SLMs with a particular focus on their mathematical abilities, as well as providing an in-depth analysis of SLMs’ performance across various settings.

2.3 Knowledge Understanding of LLMs↩︎

Knowledge understanding is a vital skill for LLMs, as they often need to accurately grasp and produce content based on knowledge. There are many benchmarks to assess the knowledge comprehension of TLMs, each with different levels of difficulty. CommonsenseQA [29] tests commonsense knowledge, which is the most basic level of human understanding. OpenBookQA [30] focuses on basic science questions. MMLU [31] includes knowledge questions from 57 subjects, mostly at high school or college levels. MMLU-Pro [32] is an advanced version of MMLU with increased difficulty. AGIEval [33] assesses tasks at a human level, which are also challenging.

3 VoxEval↩︎

In this section, we introduce a speech-based knowledge understanding benchmark called VoxEval, concentrating primarily on the methods we use to create the data, how we ensure it meets the robustness evaluation criteria, and how we address the evaluation of complex topics like mathematics.

3.1 Data Construction↩︎

Building a speech-based knowledge understanding benchmark requires the collection of knowledge from different subjects and the construction of question-answer pairs about the knowledge in speech format. Given that there are numerous existing textual knowledge understanding benchmarks (see Section 2.3), it is logical to build the speech-based benchmark upon textual benchmarks. In this work, we leverage the MMLU [31] dataset to build VoxEval. Specifically, we transform the questions from MMLU into speech using the OpenAI Text-to-Speech (TTS) API ². We select MMLU as the underlying dataset for the following reasons:

• The subjects contained in MMLU are structured and comprehensive, including STEM, social sciences, humanities, and more, and thus ideal for holistic evaluation of knowledge understanding.

• MMLU is widely used for evaluating SLMs [14], [15]. However, this evaluation primarily reflects their capability to handle text, not speech.

• In datasets available in both written and spoken forms (like sStoryCloze [26]), we find that the spoken version is significantly more challenging than the written one. Therefore, adapting MMLU into a speech format would create a demanding dataset for current SLMs.

To transform MMLU into its spoken version, we need to concatenate all the text within a question into a single sequence. In MMLU, every data is a Multiple Choice Question (MCQ) with four answer choices. We use a simple sentence to concatenate the question and answer choices. Therefore, every converted data looks like this:

f"{question} Please choose the answer from options A, B, C, and D. Here are the options. A. {option_A}, B. {option_B}, C. {option_C}, D. {option_D}.", 

and its corresponding answer is

f"The correct answer is {answer}.".

3.2 Various Input Conditions↩︎

As mentioned in Section 1, it is crucial for the answers of SLMs to be robust under various input audio conditions. Inspired by this, we incorporate MMLU questions with different input audio conditions into VoxEval. Specifically, we consider the following types of input conditions.

3.2.1 Different Speakers↩︎

SLMs should be robust when different speakers interact with them. Several key factors reflect the uniqueness of a speaker, which include but are not limited to gender, age, accent, etc. To evaluate the SLMs’ performance on different speakers, we use all six speakers provided by OpenAI TTS, namely alloy, echo, fable, nova, onyx, and shimmer, to perform the TTS. Those speakers span a wide array of speaker properties, such as gender, accent, etc.

3.2.2 Different Speaking Styles↩︎

In real-world conversations, even a single individual may use different speaking styles depending on the situation. Therefore, Speech Language Models (SLMs) need to manage various input styles. We identify two types of style variations: linguistic and paralinguistic.

Linguistic variation involves changes in the content. For instance, people may sometimes pause to think about what they want to say, leading to disfluencies in their speech. Another example is when someone accidentally says something incorrectly and then corrects themselves upon realizing the mistake. SLMs should still accurately interpret the speech in these situations. However, these scenarios are not typically captured when using TTS to generate speech from a well-written text. To address this, we propose five techniques for linguistic variation to accommodate different situations, including filler words, mispronunciations, disfluencies, false starts and corrections, and language proficiency. Table 1 provides explanations and examples for each of these five variations. To replicate these scenarios, we first use GPT-4o [6] to modify the original question text into different versions and then convert them into speech. We only alter the question text and not the answer choice texts.

Paralinguistic variation, in contrast, involves changes other than the contents of the input audio, without changing the content. Similar to linguistic variation, SLMs should also output consistent outputs under different paralinguistic variations. We consider four types of paralinguistic variations.

• Pitch shift: Pitch shift refers to the alteration of the pitch of the speech signal, which can simulate the effect of someone speaking in a higher or lower tone. For each question input, we randomly shift its pitch between -5 and 5 semitones.

• Tempo change: In real-life situations, people often vary their speaking speed, sometimes talking faster and other times slower. To mimic these variations in speaking pace, we apply tempo changes. For each question input, we randomly select a tempo-changing rate that ranges from half the original speed to twice as fast.

3.2.3 Different Audio Qualities↩︎

When interacting with SLMs, users may be in an environment with various acoustic conditions. However, in a standard TTS system, the speech is synthesized in an “ideal" acoustic environment (very clear, no echo, etc.), thus failing to mimic the real-world interaction scenarios. We consider two types of acoustic conditions when evaluating SLMs:

• Background noise: Sometimes the input audio may be accompanied by background noise, which can affect the clarity and intelligibility of the spoken content. We consider different kinds of noise, including Gaussian noise, colored noise, background music, and short interruption noise. For each question input, a kind of noise is randomly selected and applied to the original audio.

• Other environment acoustics: We consider speech recorded under various conditions, including aliasing, room impulse response, low pass filter, high pass filter, band pass filter, bitcrush, gain, clipping distortion, and seven band parameter EQ. Full explanations and transformation settings are presented in Appendix 6. A random audio effect is applied to each question input.

3.3 Handle Math Expressions and More↩︎

Logical reasoning and critical thinking are crucial for AI assistants like LLMs, and tackling math problems is a major part of these abilities. Math-solving skills are vital for SLMs because humans do not always express mathematical ideas in written form, and spoken math has many uses for AI assistants, such as in tutoring. Hence, we pioneer the evaluation of math-solving abilities for SLMs.

Mathematical problems often include numerous mathematical expressions, and current TTS systems struggle to accurately convert these expressions into speech, as they are typically trained only on textual words. To tackle this issue, we propose a two-step method for synthesizing math problems. First, we convert the written math problems into their spoken form, and then we use the TTS system to generate speech. Specifically, we create a set of few-shot examples and use GPT-4o to handle the conversion process. Our focus is on transforming Arabic numerals, units, operators, and brackets, as illustrated by the examples in Table 2. The complete prompt is provided in Figure 3 and 4.

4 Experiments↩︎

We evaluate recently proposed SLMs on our proposed VoxEval benchmark. In this section, we explain the experimental settings and the corresponding results.

4.1 Experimental Setups↩︎

Data Preparations. We assess the performance of the SLM across various input audio conditions, including different speakers, speaking styles, and audio qualities, as mentioned in Section 3.2. Few-shot prompting techniques are employed to guide the SLMs in answering questions from VoxEval. Specifically, for each subject, we synthesize five examples from the MMLU validation set using all six speaker voices provided by OpenAI TTS. The five in-context examples of the corresponding speaker are prepended to the final question. To accommodate the input constraints of certain SLMs, the resulting speech is truncated to the last 80 seconds.

Models. We evaluate VoxEval on five recently introduced SLMs: SpeechGPT [13], TWIST [26], SPIRIT-LM [14], Moshi [15], and GLM-4-Voice [34]. These models represent diverse training approaches within the SLM domain. Specifically, SpeechGPT and TWIST are pre-trained using unsupervised speech data. SPIRIT-LM and GLM-4-Voice are pre-trained on interleaved speech and text data. Moshi is pre-trained on aligned speech-text data. All the models are built upon TLM checkpoints. Furthermore, while the other models process audio data through a single stream, Moshi uniquely represents audio data using multiple streams.

Evaluation Metric. To assess the spoken responses provided by the SLMs, we utilize the OpenAI ASR model whisper-large-v3 [9] to convert their answers into text. Afterward, we apply string matching to determine the final answer (e.g., A, B, C, or D) from the transcription and calculate the accuracy.

4.2 Results↩︎

We aim to answer the following Research Questions (RQs) through experiments. RQ1: What is the overall performance of existing SLMs on the VoxEval benchmark? RQ2: How do existing SLMs perform across various knowledge domains? RQ3: To what extent do different input audio conditions influence the performance of existing SLMs? RQ4: How well are current SLMs in reasoning, particularly in mathematical reasoning?

To answer RQ1, we evaluate the five SLMs on VoxEval, whose results are shown in Table 3. We observe that existing SLMs perform poorly on VoxEval. In fact, most SLMs’ performance does not even surpass random guessing. Since each question in VoxEval offers four answer choices, the expected score for random guessing is 0.25. However, all SLMs, except for GLM-4-Voice, fail to exceed this baseline. This indicates that, in many instances, the SLMs struggle to correctly follow instructions to select from the options. This highlights the difficulty of enabling SLMs to “speak" the correct answer, making VoxEval a particularly challenging benchmark for current SLMs. A case study of different SLMs’ spoken responses is shown in Appendix 7. During the evaluation, we found that SpeechGPT’s performance is close to 0%. This is likely due to our approach of directly performing speech-to-speech generation without utilizing the "chain-of-modality" mode described in [13]. We choose not to use the "chain-of-modality" approach because it involves converting the input into text before generating a response, which we believe would significantly increase inference latency and is not an optimal strategy for end-to-end SLMs.

To answer RQ2, we present the SLM performance results across various subject groups within VoxEval. Following MMLU, all knowledge subjects are divided into four groups: STEM (18 subjects), Social Science (12 subjects), Humanities (13 subjects), and Others (13 subjects). Table 4 shows the average performance of the subjects in each group. We notice that SLMs exhibit significant performance variations across different subject groups. For instance, when using alloy as the input voice, SPIRIT-LM shows a score difference of approximately 4%, and GLM-4-Voice demonstrates a difference of around 10%. Moreover, different SLMs excel in different subject groups. For example, TWIST, Moshi, and GLM-4-Voice typically perform the best in the Others group, STEM group, and Social Science group, respectively.

To answer RQ3, we present the overall performance of SLMs across various input audio conditions in Table 3. Our key findings are summarized as follows.

• SLMs are susceptible to different input audio conditions. Although the performance of SLMs does not vary significantly with changes in speaker voices, we observe that differences in speaking styles and audio quality can negatively impact their performance, sometimes leading to a performance drop of up to 6%.

• Different input audio conditions have different impacts on SLMs. For instance, environment acoustics and linguistic variations have minimal impact on SLMs’ performance, whereas pitch shifts pose the greatest challenge for most SLMs.

We further visualize performance differences across all the settings (input audio conditions), as depicted in Figure 2. For each setting, we compute the maximum performance difference of all 56 subjects in the VoxEval dataset and present the results (56 difference scores) using box plots. In the “Speaker" setting, the differences are calculated using audio from all six speakers provided by OpenAI TTS. For the remaining settings, since all audio variations are applied to speech data generated by the”alloy" voice, the differences are computed by comparing the original “alloy" audio with the transformed audio. The top part of the figure illustrates the absolute differences in scores, while the bottom part shows the relative percentage differences. We have the following key observations.

• Performance variation among individual subjects is significantly greater than the overall performance variation. While overall performance differences are typically within 1–2%, we found many instances where the performance gap for a specific subject exceeded 10%.

• Models exhibit varying levels of stability when faced with changes in audio conditions. As shown in Figure 2 (b), TWIST and Moshi tend to be less stable, whereas GLM-4-Voice demonstrates minimal relative score differences.

Additionally, SLMs display relatively consistent behavior across different condition changes. According to Table 4, when input audio conditions are altered, TWIST, Moshi, and GLM-4-Voice consistently perform the best in others subjects, STEM subjects, and social science subjects, respectively.

To answer RQ4, we present the performance of SLMs across three math subjects in VoxEval, as shown in Table 5. The selected subjects—elementary math, high school math, and college math—are intended to represent varying levels of reasoning complexity. We observe that existing SLMs perform similarly across different levels of math questions. Interestingly, in some cases, SLMs perform better on more challenging questions. For example, TWIST and GLM-4-Voice achieve the highest performance in college math. This counterintuitive fact suggests that existing SLMs are unlikely to possess basic reasoning abilities.

5 Conclusion↩︎

In this paper, we introduce VoxEval, a novel speech-based question-answering benchmark designed to evaluate the knowledge understanding capabilities of end-to-end Spoken Language Models (SLMs) under diverse audio conditions. Comprehensive evaluations of recent SLMs reveal significant performance limitations, including poor accuracies that are often below random guessing, heightened susceptibility to variations in input audio conditions, and notable challenges in domains such as mathematical reasoning. These findings highlight the need for future research to improve the robustness and reasoning abilities of SLMs in real-world conversational settings. VoxEval serves as a critical step toward advancing the evaluation and development of SLMs, paving the way for more effective and reliable speech-based AI systems.

6 Explanation for Various Speech Recording Conditions↩︎

Table 6 provides the input audio conditions categorized under the "other environment acoustics" and the transformation settings in the audioaugmentation package.

7 Case Study for SLM Responses↩︎

Table 7 presents example SLM responses to questions from VoxEval as well as our comments.

References↩︎