1 Introduction↩︎

Large language models (LLMs) have demonstrated remarkable capabilities across various natural language understanding (NLU) tasks. Recent advances, such as GPT-4 [1] and Llama 3 [2], have shown that LLMs can produce human-like outputs and handle complex linguistic phenomena. However, whether LLMs genuinely understand semantics or merely rely on shallow statistical correlations is disputable [3]–[5]. One fundamental aspect of human conceptual understanding is that it is not dependent on specific linguistic forms or modalities [6], [7]. When humans learn and reason about concepts, they do not require the knowledge to be tied to a particular medium, such as text, images, or video, nor do they rely on a specific language. This raises an important question: Does LLMs’ conceptual-property reasoning remain stable across languages, or is it language-specific?

To explore this, [8] introduced the COMPS dataset, designed to probe the semantic reasoning abilities of LLMs through minimal pairs in English. However, COMPS only evaluates monolingual conceptual-property reasoning, leaving open the question of whether LLMs generalize such reasoning across languages. In this work, we introduce XCOMPS, a multilingual extension of COMPS, to assess whether LLMs’ semantic reasoning is universally consistent across languages. XCOMPS covers 17 languages, including analytic, inflectional, and agglutinative languages, ensuring a broad representation of linguistic structures.

Beyond dataset expansion, evaluating LLMs’ reasoning abilities has increasingly relied on prompt engineering, often referred to as metalinguistic prompting [9]. However, recent work [9], [10] suggests that metalinguistic prompting primarily assesses performance—that is, how well a model produces correct outputs—rather than its underlying competence in conceptual understanding. This distinction is crucial, as models may perform well on explicit prompts but lack true conceptual representations [11]. To investigate LLMs’ multilingual capabilities and determine whether they genuinely encode conceptual knowledge across languages, we adopt a three-pronged evaluation approach: Metalinguistic prompting, Neurolinguistic probing, and Direct probability measurement. Our experimental results reveal several insights into the multilingual conceptual reasoning capabilities of LLMs: 1) Conceptual understanding is not consistently maintained across languages. Even when models perform well in English, their reasoning ability deteriorates significantly in low-resource languages; the extent of deterioration also varies across different low-resource languages. 2) Models perform well when conceptual relationships are highly distinct but struggle with subtle semantic distinctions. 3) Languages with higher morphological complexity (agglutinative > inflected > analytic) yield lower concept-reasoning scores. These results suggest that LLMs’ semantic reasoning may not generalize universally across linguistic boundaries.

2 Language Performance vs. Competence↩︎

As suggested in [10], LLMs can be evaluated through three methods: metalinguistic prompting, which assesses performance based on explicit responses; direct probability measurement, which provides an intermediate evaluation by comparing model-generated probabilities; and neurolinguistic probing, which directly examines competence by analyzing internal activation patterns³.

2.0.0.1 Metalinguistic Prompting for Performance

This method involves explicitly querying the model about linguistic expressions, often in a comparative or multiple-choice format. By asking the model to choose between minimal pairs (e.g., “Which sentence is more grammatically correct?”), researchers can evaluate how well the model retrieves and verbalizes knowledge. Using prompting, researchers have revealed new classes of emergent abilities such as arithmetic, instruction-following, grounded conceptual mappings, and sentence acceptability judgments [12]–[15]. Because the responses are influenced by prompt engineering and surface-level cues, this method primarily reflects performance rather than deep conceptual competence.

2.0.0.2 Direct Probability Measurement

Instead of relying on explicit responses, this method examines the model’s probability assignment to different sentences within minimal pairs. For example, a model should assign a higher probability to ‘A robin can fly’ than to ‘A penguin can fly’. This approach offers a more objective evaluation than metalinguistic prompting and captures implicit model preferences, placing it between performance and competence. Researchers have designed syntactic, semantic/conceptual, and discourse inference tasks using the probability assignment method, offering different insights into LLMs’ capabilities compared to metalinguistic prompting [8], [16]–[21]. However, it still relies on external outputs and does not fully reveal how the model internally represents concepts.

2.0.0.3 Neurolinguistic Probing for Competence

This approach goes beyond external outputs by analyzing internal activation patterns across different layers of the model [10], [22]. Using diagnostic classifiers, researchers can probe whether LLMs inherently encode conceptual-property relationships or simply rely on statistical correlations. Since it provides a direct measure of competence, neurolinguistic probing is more reliable for assessing the depth of linguistic understanding.

3 XCOMPS↩︎

3.1 Concept Selection↩︎

To ensure that XCOMPS maintains conceptual alignment with COMPS while extending its scope to multiple languages, we use the same 521 concepts and their negative samples from COMPS. As shown in Table 1, these negative samples can be categorized into three types. Taxonomy-based negative samples are selected based on hierarchical relationships among concepts. Negative samples come from the same broad category as the positive concept but differ in key property attributions. Property norm-based (overlap) negative samples are chosen based on shared semantic properties with the positive concept while lacking the specific property under evaluation. Co-occurrence-based samples are selected from concepts that frequently appear in similar contexts but do not share the target property. XCOMPS also has additional random negative concepts from the set of concepts that do not possess the property of the original positive concept.

3.2 Properties of Concepts↩︎

In XCOMPS, the properties assigned to concepts are inherited from COMPS, ensuring alignment across languages while maintaining the original conceptual-property relationships. These properties in COMPS were originally derived from the XCSLB dataset, an extended version of the CSLB property norm dataset [23], which captures human-annotated perceptual, functional, and categorical attributes of concepts. Additionally, taxonomic relationships from resources like WordNet [24] were used to infer properties through hierarchical inheritance, ensuring that general category attributes (e.g., “mammals have fur”) are systematically applied to their subcategories. Some properties also reflect real-world associations observed in corpus-based co-occurrence statistics.

3.3 Multilingual Data Construction↩︎

To construct XCOMPS, which covers 17 languages (Table 2 in Appendix 6), we adopted a human-LLM interactive translation pipeline, leveraging both human expertise and the multilingual generation capabilities of LLMs. The language set for XCOMPS aligns with the prior knowledge probing benchmarks, such as BMLAMA-17 [25] and KLAR [26], ensuring consistency in multilingual evaluation. The highly structured nature of conceptual minimal pair datasets, where positive and negative sentences primarily consist of two components–concepts and properties–enabled us to design a multi-step translation process that ensures high-quality multilingual data.

The construction process consists of four stages. We use the GPT-4o model (GPT-4o-2024-08-06) via the OpenAI API as the translation assistant in the pipeline. In the first stage, we manually translated the original concepts and properties from English into German and Chinese using language experts. We used German and Chinese as additional seed languages to further reduce ambiguity, This multilingual seed data helped disambiguate concepts that might otherwise be unclear in translation. For example, the English word “bat” could refer to either the flying animal or the sports equipment. By including the German term “Schläger” and the Chinese term ”

“, which both unambiguously refer to the sports equipment, we ensured that the intended concept was accurately captured during translation.

In the second stage, we used LLMs to expand the seed data into the remaining 15 languages. LLMs were tasked with translating the concepts and properties, leveraging their multilingual machine translation capabilities. By providing seed data in three languages (English, German, and Chinese), we enhanced the LLMs’ ability to generate accurate translations, as the additional context reduced the likelihood of semantic errors.

In the third stage, human experts for each target language manually reviewed and corrected the translated concepts and properties. This step ensured that the translations were accurate, culturally appropriate, and semantically aligned with the original dataset. Human intervention was particularly critical for low-resource languages, where LLMs often struggle with semantic precision in translation tasks.

Finally, in the fourth stage, LLMs were employed to generate complete sentences based on the verified concepts and properties. This step involved formulating positive and negative sentence pairs, which can be viewed as a straightforward language manipulation task. By providing the translated concepts and properties as input, we enabled the LLMs to focus on generating fluent and grammatically correct sentences, leveraging their strengths in multilingual text generation. This approach ensured that the most challenging aspect of the task–accurate translation of concepts and properties–was already resolved, allowing the LLMs to produce high-quality outputs.

By splitting the process into property translation and sentence generation, using multilingual seed data to reduce ambiguity, and combining human expertise with LLM capabilities, we ensured the quality and consistency of the XCOMPS dataset. This human-LLM interactive translation pipeline demonstrates how LLMs’ multilingual understanding and generation capabilities can be effectively harnessed to construct high-quality multilingual benchmarks.

4 Experiment Setup↩︎

4.1 Model↩︎

We use meta-llama/Llama-3.1-8B-Instruct from Hugging Face in our experiment, which applies instruction tuning to the base model for more intuitive user-prompt handling. During the inference, we adopt float16 precision to minimize computational resource consumption while maintaining model performance.

4.2 Evaluation↩︎

For Meta, we present both sentences of a minimal pair within a single prompt. We convert the target property into a question and compare the probabilities assigned to acceptable vs. unacceptable concepts. Figure 2 in Appendix 6 shows the prompts used in the experiment. For Direct, we compute sentence probabilities directly from the model’s logits. A prediction is considered correct if the model assigns a higher probability to the valid sentence within each minimal pair. For Neuro, we adopt last-token pooling to represent each sentence, extracting the final token’s hidden state from every layer. This approach ensures coverage of all preceding tokens [27]. We then apply a logistic regression classifier for probing, using the F1 score (averaged over five cross-validation folds) as our primary evaluation metric.

4.3 Results and Analysis↩︎

4.3.0.1 Cross-linguistic variability in conceptual reasoning.

From Figure 1, we observe that the model can perform relatively well on English conceptual tasks but show marked declines for low-resource languages. Notably, some languages with limited training data (e.g., Hungarian, Catalan) exhibit greater deterioration than others, indicating that cross-linguistic generalization of conceptual understanding is far from uniform. Even within the low-resource category, the degree of performance drop varies, underscoring that LLMs’ semantic reasoning is neither universally stable nor equally supported by existing multilingual corpora. These patterns reinforce the idea that conceptual capabilities learned in English do not necessarily transfer seamlessly to languages that differ typologically or have weaker representations in training data.

4.3.0.2 Models excel at distinct conceptual contrasts but falter with subtler differences.

High scores all appear in Random rows, where the negative concept is clearly distinct (e.g., “toaster” vs. “wren”), and the model easily detects mismatches. In Taxonomic, Overlap, or Co-occurrence rows, however, performance drops because the negative concepts share subtle semantic similarities (e.g., “toaster” vs. “coffee maker”). This indicates that the models may rely on conspicuous cues rather than true conceptual reasoning.

4.3.0.3 Direct and neuro convergence.

By comparing direct and neuro results in Figure 1, and from Figure 3 in Appendix 6, we see high correlations across all negative types, indicating that direct measurements closely track the models’ internal representations.

4.3.0.4 Higher morphological complexity, lower conceptual reasoning.

Figure 4 in Appendix 6 shows that languages with greater morphological complexity (moving from Analytic to Inflected to Agglutinative) tend to yield lower concept-reasoning scores. This indicates that, as linguistic structure becomes more complex, it becomes harder for the models to capture concept-property relationships consistently.

5 Conclusion↩︎

In this work, we introduce the XCOMPS benchmark, which provides a multilingual conceptual minimal pair dataset for evaluating the language model’s semantic understanding across 17 languages. This work reveals that while LLMs demonstrate surface-level multilingual capabilities, they lack a universal semantic reasoning mechanism that transcends language boundaries.

Limitation↩︎

While XCOMPS significantly advances the evaluation of multilingual conceptual understanding, certain limitations remain. First, although the dataset covers 17 typologically diverse languages, it does not encompass all linguistic families or low-resource languages, which may limit its generalizability to underrepresented languages. Second, the reliance on human-LLM interaction for data construction ensures high quality but introduces potential inconsistencies due to variations in human expertise and model outputs. Lastly, while XCOMPS focuses on conceptual understanding, it does not explicitly address other challenges in multilingual NLP, such as pragmatics or contextual reasoning. Despite these limitations, XCOMPS provides a robust foundation for assessing and improving LLMs’ multilingual capabilities, and future work can extend its scope to address these areas.

Acknowledgement↩︎

We thank the anonymous reviewers for their valuable advice and feedback. This research was partially supported by DFG (German Research Foundation) grant SCHU 2246/14-1 and Munich Center for Machine Learning (MCML).

6 Appendix↩︎

Table 2 shows the detailed information of the languages covered by XCMOPS. Figure 2 displays the prompt templates of different languages used for metalinguistic prompting evaluation. Figures 3 and 4 show detailed experimental results.

References↩︎

---

1.   Equal contribution.↩︎

2.   Corresponding authors.↩︎

3. For simplicity, we refer to these three methods as Meta, Direct, Neuro.↩︎