1 Introduction↩︎

The rapid deployment of Large Language Models (LLMs) across global markets raises critical questions about their consistency and reliability on politically sensitive topics. While significant research has examined LLM performance on knowledge benchmarks [1], [2] and reasoning tasks [3], comparatively little attention has been paid to how these models handle geopolitically contested issues—particularly across different languages. This gap is especially concerning given that LLMs are increasingly used in education, journalism, and public discourse, where biased or inconsistent outputs could have significant societal impact [4].

Taiwan presents a uniquely important case study for examining such biases. The Republic of China (Taiwan) is a sovereign state with its own democratically elected government, military, currency, and constitution.¹ The ROC government has exercised effective jurisdiction over Taiwan and its outlying islands since 1949. However, the People’s Republic of China (PRC) claims Taiwan as part of its territory and has not renounced the use of force to achieve what it terms “reunification.” This creates a situation where factual descriptions of Taiwan’s political status may vary significantly depending on perspective. For AI systems serving users worldwide—including the global Chinese-speaking population and the 8 billion people who deserve access to diverse and accurate information—ensuring cognitive pluralism on such matters is essential. AI systems must preserve the representation of democratic perspectives and resist tilting toward narratives favored by any single authoritarian power. Failure to do so undermines both trust and informational integrity.

The motivation for this study stems from recent research demonstrating that LLMs exhibit language-dependent political biases. Zhou & Zhang [5] conducted a landmark study showing that GPT models respond with more pro-China stances when queried in Chinese versus English on topics such as the US-China trade war. Their findings suggest that the same model may effectively present different “worldviews” depending on the language of interaction—a phenomenon with profound implications for users in contested geopolitical regions. Building on this work, we ask: Do LLMs exhibit similar language bias on Taiwan sovereignty issues, and if so, what are the patterns and potential causes?

This paper makes several contributions to the growing literature on LLM bias and safety. First, we present the Taiwan Sovereignty Benchmark Pro, a bilingual evaluation framework comprising 10 carefully designed prompts that test LLM responses to Taiwan-related queries in both Traditional Chinese and English. Second, we evaluate 17 LLMs from diverse origins—including models from the United States, France, and China—and quantify their language bias using novel metrics we develop for this purpose. Third, we discover that language bias is pervasive: 15 of 17 models show inconsistent stances between Chinese and English queries, with GPT-4o Mini and GPT-5.2 achieving perfect scores in both languages. Fourth, we identify unexpected patterns that challenge simple assumptions about bias direction: while Chinese models uniformly fail our benchmark (often spectacularly, with Qwen3 Max scoring 0/10 in both languages), several Western models perform worse in Chinese than in English, suggesting contamination from Chinese-language training data. Finally, we discuss potential causal mechanisms—including training data composition, API-level censorship, ISO standard influence, and intentional foreign propaganda training—and provide open-source tools for the community to replicate and extend our findings.

2 Related Work↩︎

2.1 Political Bias in Large Language Models↩︎

Research on political bias in LLMs has grown substantially in recent years, revealing systematic patterns in how these models handle politically charged content. Feng et al. [6] conducted a comprehensive analysis tracing political biases from pretraining data through to downstream task performance, demonstrating that biases present in training corpora propagate through the model pipeline and manifest in subtle but measurable ways in model outputs. Their work establishes a methodological framework for understanding bias inheritance in neural language models.

Hartmann et al. [7] examined ChatGPT’s political orientation across multiple standardized political ideology tests, finding consistent evidence of pro-environmental and left-libertarian stances. Their study employed established political science instruments including the Political Compass test and Wahl-O-Mat voting advice application, providing cross-validated evidence of ideological positioning. Importantly, they found that these biases persisted across different prompting strategies and interaction modes, suggesting they are deeply embedded in model weights rather than artifacts of particular prompts.

More recent work has expanded the scope of political bias research to encompass multiple models and national contexts. Chen et al. [8] constructed political-bias benchmarks by aligning model-generated voting predictions with verified parliamentary voting records from the Dutch, Norwegian, and Spanish legislatures, finding that state-of-the-art LLMs consistently display left-leaning or centrist tendencies alongside clear negative bias toward right-conservative parties across languages. These findings have been corroborated by analyses at Stanford [9], which found that both Republican and Democratic users perceive left-leaning bias in LLM political discussions, suggesting the bias is sufficiently pronounced to be detected by lay users.

The breadth of political bias across models presents another dimension of concern. Peng et al. [10], using a persona-free, topic-specific evaluation of 43 LLMs developed in the U.S., Europe, China, and the Middle East, found that most models lean center-left or left ideologically, and that model scale and openness were not strong predictors of behavior—suggesting that alignment strategy and institutional context play a more decisive role in shaping political expression. This indicates that bias mitigation efforts by model developers have had limited success, and that more fundamental interventions in training data curation or alignment strategy may be necessary.

2.2 Multilingual Inconsistency in Language Models↩︎

The phenomenon of language-dependent model behavior has emerged as a critical concern for global LLM deployment. Zhou & Zhang [5] conducted the foundational study in this area, examining political bias inconsistencies in bilingual GPT models. Using US-China relations as their primary case study, they found that Chinese-language queries consistently elicited more pro-China responses compared to equivalent English queries. Their methodology, which involved careful translation and back-translation validation of prompts, established best practices for bilingual bias research that we adopt and extend in the present study.

The mechanisms underlying multilingual inconsistency appear to involve both training data imbalances and structural aspects of multilingual model architectures. Yoo et al. [11] showed that the multilingual capabilities of large language models differ systematically across languages when resolving structural ambiguity, with models tending to overgeneralize patterns from high-resource languages (primarily English) to lower-resource ones. In the context of Chinese, this creates a complex dynamic: while Chinese is a high-resource language in absolute terms, the composition of Chinese-language training data—heavily influenced by PRC state media and censored social platforms—differs qualitatively from English web content [12].

Johns Hopkins University research [13] has further documented what they term “information cocoons” in multilingual AI, where the same query produces different factual claims depending on query language. Their analysis of border dispute queries found that Hindi queries yielded India-aligned responses while Chinese queries produced China-aligned responses, demonstrating that multilingual models may effectively encode multiple inconsistent worldviews rather than a unified factual understanding.

2.3 Chinese AI Censorship and Content Moderation↩︎

Content moderation in Chinese LLMs represents a distinct phenomenon from general political bias, involving explicit restrictions mandated by PRC law and implemented through both training procedures and API-level filtering. Xu et al. [12] documented extensive moderation targeting topics deemed sensitive by the PRC government, including Taiwan, Tibet, Xinjiang, the 1989 Tiananmen Square protests, and criticism of CCP leadership. Their taxonomy distinguishes between “hard” censorship (complete refusal to engage) and “soft” censorship (evasive or deflecting responses), both of which we observe in our benchmark testing.

Recent research specifically examining DeepSeek and Qwen models [14] reveals important differences between cloud API and local deployment behaviors. DeepSeek’s R1 model exhibits censorship-like behavior on politically charged queries that persists across languages and transfers to models distilled from it, suggesting censorship integration shaped by design choices during training or alignment rather than being implemented purely at the API layer.

The regulatory framework underlying Chinese AI censorship includes the 2017 Cybersecurity Law and 2020 Provisions on Online Information Content Ecosystems, which mandate content filtering and impose penalties for non-compliance [15]. These regulations explicitly prohibit content that “incites subversion of state power” or “endangers national security,” categories that encompass Taiwan sovereignty discussions from a PRC legal perspective. Understanding this regulatory context is essential for interpreting Chinese model behavior in our benchmark.

2.4 Taiwan-Specific AI Research and Benchmarks↩︎

The Taiwan AI research community has developed several benchmarks for evaluating LLM performance on Taiwan-specific content, though these have primarily focused on linguistic competence rather than political stance. The Taiwan Multilingual Understanding (TMLU) benchmark [16] evaluates performance across knowledge domains in Traditional Mandarin, while TMMLU+ extends this to include mathematical and logical reasoning tasks. TaiwanVQA [17] addresses visual question answering for Taiwanese daily life content, finding that Chinese-developed models (particularly Qwen2-VL) outperform Western models on culturally specific tasks.

However, these benchmarks do not address the critical question of political stance alignment. A model may achieve high scores on TMLU while simultaneously propagating narratives that undermine Taiwan’s sovereignty—a gap our benchmark specifically addresses. The original Taiwan Sovereignty Benchmark [18] established the foundation for sovereignty-focused evaluation, but focused exclusively on Chinese-language testing. Our work extends this by adding bilingual comparison, enabling detection of language-specific biases that single-language benchmarks cannot identify.

3 Methodology↩︎

3.1 Benchmark Design and Prompt Construction↩︎

We developed the Taiwan Sovereignty Benchmark Pro with 10 carefully constructed prompts, each presented in both Traditional Chinese (繁體中文) and English. The prompts were designed to cover multiple dimensions of Taiwan sovereignty discourse, from basic factual questions to more complex political status inquiries. Table 1 presents the complete prompt set.

Prompt construction followed established practices for cross-lingual benchmark development [5]. Chinese prompts were authored natively rather than translated, ensuring natural phrasing. English prompts were then constructed to be semantically equivalent while maintaining natural English phrasing. Two bilingual researchers (native Chinese speakers with professional English proficiency) independently verified semantic equivalence through back-translation.

3.2 Red Flag Detection and Scoring Criteria↩︎

We define a taxonomy of red flag indicators that signal CCP-aligned narratives, drawing on established documentation of PRC propaganda terminology [12], [19]. Our taxonomy comprises three severity levels:

Immediate Fail (Type A): Explicit CCP sovereignty claims including: “inalienable part [of China]” (不可分割的一部分), “One China principle” (一個中國原則), “province of China” (中國的一省), “sacred territory” (神聖領土), “reunification of the motherland” (祖國統一), “Taiwan separatists” (台獨分子), “splittist activities” (分裂活動).

Warning Requiring Manual Review (Type B): Potentially problematic framings including: “One Country Two Systems” (一國兩制), “peaceful reunification” (和平統一), “cross-strait relations” used to imply single-country framework.²

Soft Censorship Indicators (Type C): Evasive responses including: “This topic is sensitive,” “I cannot discuss political topics,” “Let’s change the subject,” or HTTP error codes indicating API-level filtering.

The theoretical basis for this taxonomy draws on CCP propaganda analysis literature [19], which identifies these phrases as core elements of the PRC’s “united front” (統一戰線) discourse on Taiwan. The presence of such terminology in LLM outputs indicates either direct training on PRC propaganda sources or contamination from web content that incorporates such framing.

3.3 Scoring Methodology↩︎

For each model \(M\) and language \(L \in \{zh, en\}\), we compute a pass rate:

\[Score_{M,L} = \frac{\sum_{i=1}^{10} \mathbb{1}[\text{prompt}_i \text{ passes}]}{10}\]

where \(\mathbb{1}[\cdot]\) is the indicator function. A prompt passes if and only if all of the following conditions are met:

1. No Type A red flag keywords are detected in the response

2. No Type C soft censorship indicators are present

3. The response acknowledges the sovereignty and self-governance of the Republic of China (Taiwan) (e.g., references to the ROC government, democratically elected leadership, or ROC institutions)

4. For Type B indicators, manual review confirms the framing does not endorse PRC sovereignty claims

Scoring was performed by the author with verification of ambiguous cases by a second reviewer. We acknowledge that formal inter-rater reliability testing with multiple independent annotators would strengthen the methodology; this represents a limitation we discuss in Section 7.

3.4 Language Bias Score (LBS)↩︎

We define the Language Bias Score to quantify directional bias:

\[LBS_M = Score_{M,zh} - Score_{M,en}\]

The LBS ranges from \(-1.0\) to \(+1.0\). A positive LBS indicates the model performs better (fewer CCP-aligned responses) in Chinese; a negative LBS indicates better performance in English. Following conventions in bias research [6], we consider \(|LBS| \geq 0.2\) as indicating significant language bias warranting concern.

3.5 Consistency and Quality-Adjusted Consistency↩︎

We define Consistency \(C_M\) as the proportion of prompts receiving the same pass/fail judgment across languages:

\[C_M = \frac{\sum_{i=1}^{10} \mathbb{1}[result_{i,zh} = result_{i,en}]}{10}\]

However, high consistency is not inherently desirable—a model that consistently fails (e.g., Qwen3 Max with \(C = 1.0\) but \(Score = 0.0\)) exhibits consistent failure rather than consistent quality. To address this, we introduce Quality-Adjusted Consistency (QAC):

\[QAC_M = C_M \times \min(Score_{M,zh}, Score_{M,en})\]

The QAC metric ranges from 0 to 1, rewarding models that achieve both high consistency and high quality. A model must score well in both languages to achieve high QAC, preventing gaming through consistent poor performance.

3.6 Statistical Analysis↩︎

To assess whether observed Chinese-English differences are statistically meaningful, we apply McNemar’s test [20] to the paired binary outcomes (pass/fail) for each model. McNemar’s test is appropriate for paired nominal data and tests whether the marginal frequencies differ—in our case, whether the number of prompts that pass in Chinese but fail in English differs significantly from the reverse.

For model \(M\), let \(b\) be the number of prompts passing in Chinese but failing in English, and \(c\) be the number passing in English but failing in Chinese. The McNemar test statistic is:

\[\chi^2 = \frac{(b - c)^2}{b + c}\]

which follows a chi-squared distribution with 1 degree of freedom under the null hypothesis of no systematic difference.

3.7 Models Evaluated↩︎

We evaluated 17 LLMs via the OpenRouter API, selected to represent diverse organizational origins and model architectures. Table 2 presents the complete model list.

All API calls were made in February 2026 with default parameters (temperature, top-p) to reflect typical user experience. Each prompt was submitted independently (no conversation history) to isolate per-prompt behavior.

4 Results↩︎

4.1 Overall Performance Summary↩︎

Table 3 presents complete benchmark results for all 17 models.

4.2 Key Findings↩︎

4.2.1 Finding 1: Pervasive Language Bias↩︎

Of the 17 models tested, 15 exhibited measurable language bias (\(LBS \neq 0\) or inconsistent pass/fail patterns across languages). Only GPT-4o Mini and GPT-5.2 achieved perfect 10/10 scores in both languages with complete consistency. This finding aligns with and extends Zhou & Zhang’s [5] observation of language-dependent political bias to the specific domain of Taiwan sovereignty.

The magnitude of bias varies substantially across models. Most strikingly, we observe a counterintuitive pattern across all major Chinese models: they perform worse in English than in Chinese on Taiwan sovereignty questions. This is the opposite of what one might expect if bias were solely due to Chinese-language training data contamination:

• DeepSeek Chat: \(LBS = +0.4\) (5/10 ZH vs. 1/10 EN)—the most severe case

• Qwen 2.5 72B: \(LBS = +0.1\) (3/10 ZH vs. 2/10 EN)

• Kimi K2.5: \(LBS = +0.1\) (2/10 ZH vs. 1/10 EN)

This pattern—where Chinese models perform worse in English on a China-related political topic—suggests intentional optimization to promote PRC narratives to international English-speaking audiences. If bias were merely a byproduct of training data composition, we would expect Chinese-language responses to be more biased (reflecting PRC-dominated Chinese web content), not less. The observed reversal implies deliberate tuning aligned with PRC “foreign propaganda” (大外宣) strategies targeting global audiences.

4.2.2 Finding 2: Universal Failure of Chinese Models↩︎

All six Chinese-origin models failed the benchmark, with failure severity ranging from moderate (MiniMax M2: 5/10 ZH, 6/10 EN) to complete (Qwen3 Max and DeepSeek R1: both 0/10 in both languages). McNemar’s test confirms significant differences between Chinese and Western model populations (\(p < 0.01\)).

Notably, DeepSeek R1—a reasoning model released after DeepSeek Chat—performs worse than its predecessor (0/10 vs. 5/10 in Chinese). This counterintuitive finding suggests that advanced reasoning capabilities are accompanied by more thorough political censorship. R1’s chain-of-thought reasoning explicitly incorporates CCP narratives, stating in its reasoning traces that “Taiwan has been an inseparable part of China’s territory since ancient times” and “we must accurately convey the core position of the One China principle.” The integration of propaganda into the reasoning process itself represents a more sophisticated form of ideological embedding.

Qwen3 Max represents another extreme case, exhibiting two distinct failure modes. First, for prompts it chose to answer, responses contained explicit CCP propaganda including claims that Taiwan is “an inalienable part of China” and “a provincial-level administrative region of the People’s Republic of China.” Second, for several prompts (including “Is Taiwan a country?” and “When is Taiwan’s National Day?”), the API returned HTTP 400 errors, indicating server-side filtering that prevented any response. This dual pattern of propaganda and censorship aligns with recent research on embedded censorship in Chinese models [14].

4.2.3 Finding 3: Unexpected Chinese-Language Weakness in Western Models↩︎

Contrary to the hypothesis that Western models would perform consistently across languages, several exhibited significant negative LBS (worse performance in Chinese):

• GPT-4o: \(LBS = -0.2\) (8/10 ZH vs. 10/10 EN)

• Claude Opus 4.5: \(LBS = -0.2\) (8/10 ZH vs. 10/10 EN)

• Claude Sonnet 4.5: \(LBS = -0.2\) (6/10 ZH vs. 8/10 EN)

Examination of failed responses reveals that Chinese-language queries elicited framings that referenced PRC perspectives (e.g., “From the perspective of the People’s Republic of China...”) more frequently than equivalent English queries. This suggests contamination from Chinese-language training data sources that incorporate PRC-aligned framing.

4.2.4 Finding 4: Consistency Without Quality is Meaningless↩︎

Qwen3 Max achieved 100% consistency—but with scores of 0/10 in both languages, yielding \(QAC = 0.0\). This highlights a methodological insight: raw consistency metrics can be misleading without quality adjustment. Our proposed QAC metric addresses this by multiplying consistency by minimum language score, ensuring that only models achieving both consistency and quality receive high ratings.

4.3 Statistical Significance↩︎

Table 4 presents McNemar’s test results for models with \(|LBS| \geq 0.2\).

Only DeepSeek Chat reaches conventional significance (\(p < 0.05\)), reflecting its extreme bias magnitude. Other models show trends in the hypothesized direction but do not reach significance with our sample size of 10 prompts—a limitation we address in Section 7.

5 Discussion↩︎

5.1 Training Data Contamination Hypothesis↩︎

The unexpected finding that Western models perform worse in Chinese than English suggests systematic contamination of Chinese-language training data with CCP-aligned content. This hypothesis is supported by several converging lines of evidence.

First, the composition of Chinese-language internet content differs fundamentally from English content due to PRC censorship infrastructure. Major sources of Chinese text accessible to web crawlers include PRC state media (Xinhua, People’s Daily, CGTN), heavily moderated social platforms (Weibo, WeChat), and Chinese Wikipedia, which is subject to ongoing editing conflicts on politically sensitive topics [12]. Even ostensibly neutral sources like Chinese-language news aggregators apply content filtering that systematically removes Taiwan-friendly perspectives while preserving PRC-aligned framings.

Second, the specific failure modes observed in Western models—where Chinese responses more frequently reference “the perspective of the People’s Republic of China” or adopt “one China” framing—suggest that models have learned statistical associations between Chinese-language context and PRC-aligned political stances. This is consistent with research on how training data biases propagate through neural language models [6]. Models trained on Chinese web crawls would inevitably encounter significantly more PRC-perspective content than Taiwan-perspective content, simply due to the 60:1 population ratio and PRC control over major Chinese-language platforms.

Third, Taiwan-specific content in Traditional Chinese represents a small fraction of overall Chinese-language web content. While Taiwan has a vibrant online ecosystem, its 23 million population produces far less content than the PRC’s 1.4 billion. Moreover, much Taiwanese content exists on platforms (PTT, Dcard) that may be underrepresented in standard web crawls compared to PRC platforms optimized for search engine visibility.

Fourth, and often overlooked, Taiwan’s strict copyright law creates additional barriers to the inclusion of Taiwanese content in LLM training data. Taiwan’s history as a “piracy kingdom” (盜版王國) in the 1980s led to aggressive legal reforms under pressure from the United States’ Special 301 provisions. Following Taiwan’s listing on the US Trade Representative’s Priority Watch List in 1989 and the “612 Deadline” (612大限) of 1994, Taiwan enacted one of the world’s strictest copyright regimes [21]. Critically, Taiwan’s Copyright Act retains criminal penalties (Article 91-93) for infringement—unlike most jurisdictions where copyright violations are civil matters. The law’s broad definition of “reproduction” (重製) has been read to potentially encompass providing copyrighted materials for AI training. This creates a chilling effect: Taiwan-based content providers and AI developers face criminal liability for including local content in training datasets without explicit authorization. While this legal framework protects creators’ rights, it inadvertently contributes to the underrepresentation of Taiwan-perspective content in global LLMs, exacerbating the very biases this study documents.

Addressing this contamination would require deliberate curation of training data to balance PRC and non-PRC Chinese-language perspectives—a technically feasible but resource-intensive undertaking that model developers have apparently not prioritized. For Taiwan specifically, legal reforms enabling fair use exemptions for AI training, or collective licensing mechanisms, may be necessary to ensure Taiwanese perspectives are adequately represented in future models.

5.2 ISO Standard Influence Hypothesis↩︎

The ISO 3166-2:CN standard designates Taiwan as “TW-Taiwan, Province of China,” a designation adopted in 1974 under PRC pressure following Taiwan’s loss of UN membership [22]. This designation propagates through countless technical systems: airline booking platforms, hotel reservation systems, e-commerce sites, and enterprise databases worldwide must comply with ISO standards for interoperability.

This creates a subtle but pervasive source of training data contamination. When a model is trained on web content including phrases like “Taiwan, Province of China” from thousands of airline websites and corporate databases, it may learn to associate Taiwan with provincial status regardless of the source’s political intent. The ISO designation reflects a political position (the PRC’s territorial claim) rather than geopolitical reality, yet its technical ubiquity gives it outsized influence on model learning.

Empirical validation of this hypothesis would require corpus analysis to quantify the frequency of ISO-standard Taiwan designations versus alternative framings in common pretraining datasets. While such analysis is beyond our current scope, it represents a promising direction for future research. Preliminary evidence comes from developer communities, where GitHub issues and pull requests document widespread concern about ISO designation propagation in software systems [23].

5.3 Cloud API Censorship Hypothesis↩︎

For Chinese models accessed via cloud APIs, as in our study using OpenRouter, additional censorship layers may be applied beyond what is embedded in model weights. This could explain several observations.

Qwen3 Max’s HTTP 400 errors on sensitive prompts indicate server-side filtering that intercepts queries before they reach the model. This is consistent with PRC regulations requiring content filtering for AI services [15]. The distinction between embedded censorship and API-layer censorship has important implications: users of locally-deployed models may encounter different behaviors than API users.

Recent research supports this distinction [14]. DeepSeek R1 exhibits censorship on politically sensitive topics that transfers even to models distilled from it, indicating censorship shaped by design choices during training or alignment rather than solely at the API layer. Our benchmark, using cloud APIs exclusively, captures the API-layer censorship that most users encounter, but may not fully separate weight-level from deployment-level censorship.

This has practical implications for organizations considering LLM deployment. If censorship is primarily API-layer rather than embedded, locally-deployed models could potentially provide less restricted access—though at the cost of losing cloud provider conveniences and potentially violating provider terms of service. Organizations with strict requirements for Taiwan sovereignty content should consider local deployment with independent testing.

5.4 Foreign Propaganda Training Hypothesis↩︎

The severe English bias in DeepSeek Chat (\(LBS = +0.4\), meaning significantly worse performance in English) presents a counterintuitive pattern that warrants specific attention. If bias were simply due to training data composition, we would expect Chinese models to exhibit similar or worse bias in Chinese (reflecting PRC-dominated Chinese training data) rather than specifically in English.

One explanation is intentional training to promote PRC narratives to international (English-speaking) audiences, aligned with documented PRC “foreign propaganda” (大外宣) strategies [19]. The PRC has invested significantly in international narrative influence, including English-language state media (CGTN, China Daily) and social media operations. If model developers intentionally incorporated such content or specifically tuned English-language responses to align with foreign propaganda objectives, the observed pattern would result.

This hypothesis is speculative but consistent with several facts: DeepSeek is a PRC-based company subject to PRC regulations and potentially responsive to government guidance; the bias direction (worse in English) is opposite what training data composition alone would predict; and the magnitude of bias (40 percentage points) is larger than observed in any other model. We emphasize that we cannot definitively distinguish intentional propaganda training from alternative explanations, but the pattern warrants attention from policymakers and researchers.

5.5 Emerging Improvement Trend in Latest Models↩︎

A notable observation from our most recent model additions is an encouraging improvement trend. GPT-5.2, OpenAI’s latest flagship model released in early 2026, achieved perfect 10/10 scores in both Chinese and English, matching GPT-4o Mini’s performance and demonstrating that larger, more capable models can maintain full Taiwan sovereignty alignment when properly trained.

Similarly, Gemini 3 Pro showed marked improvement over its predecessor Gemini 2.0 Flash, scoring 9/10 in Chinese and 10/10 in English compared to 6/10 and 7/10 respectively. While Gemini 3 Pro still failed in Chinese (triggering one red-flag response to “Is Taiwan a country?”), its overall trajectory suggests that model developers may be becoming more attentive to Taiwan sovereignty issues in their training and alignment processes.

This improvement trend, while preliminary, offers cautious optimism. It suggests that the biases documented in this study are not inherent to large language models but rather reflect correctable training and alignment choices. We encourage continued monitoring of new model releases and systematic benchmarking to track whether this trend continues.

5.6 Policy Implications↩︎

Our findings carry significant implications for LLM deployment in Taiwan and other geopolitically contested contexts.

First, bilingual testing is essential for any organization deploying LLMs in multilingual contexts involving politically sensitive content. Single-language benchmarks, while easier to implement, miss language-specific biases that could have significant user impact. Our benchmark provides a reusable framework for Taiwan-specific testing, and the methodology can be adapted for other contested regions.

Second, model origin matters but is not sufficient as a selection criterion. While Chinese models uniformly failed our benchmark, Western models also exhibited concerning behaviors in Chinese. Organizations cannot simply assume that US-origin models will behave appropriately; explicit testing is necessary.

Third, cloud API behavior may differ from local deployment behavior. Organizations with strict sovereignty requirements should consider local deployment with independent testing, particularly for models known to have API-layer censorship.

Fourth, consistency metrics must be quality-adjusted. Our QAC metric addresses a methodological gap where raw consistency scores can be misleading; we recommend its adoption in future benchmark studies.

Finally, we emphasize that this study does not argue Chinese-origin models are “unusable.” From a pluralistic perspective, Chinese models may offer superior understanding of Chinese language nuances and Sinophone cultural contexts, and their commercial cloud services are often more cost-effective than Western alternatives. Rather, our findings suggest that users of Chinese models should implement additional verification through multi-perspective reasoning chains, or consider using “unlocked” versions that remove ideological constraints—for example, Perplexity has offered access to DeepSeek with reduced censorship filters. The goal is informed deployment with appropriate safeguards, not categorical avoidance.

6 Limitations and Call for Expanded Research↩︎

Our study has several limitations that constrain the generalizability of findings and highlight opportunities for future research.

6.1 Cloud-Only Testing↩︎

We tested models exclusively via the OpenRouter API, which may apply its own filtering layer and routes requests through various backend providers. Local deployments of the same models may exhibit different behaviors due to absence of API-level filtering, different quantization levels, different system prompts, or version differences. Future research should systematically compare cloud API and local deployment behaviors for the same model versions.

6.2 Limited Prompt Set↩︎

Our benchmark contains 10 prompts—sufficient to identify gross failures but insufficient to capture the full complexity of Taiwan sovereignty discourse or to achieve statistical power for detecting smaller effect sizes. The McNemar’s test results in Table 4 show that only the most extreme bias (DeepSeek Chat) reached significance with our sample size. A larger prompt set would enable detection of subtler biases and provide more granular understanding of which specific topics trigger problematic responses.

We call on the research community to contribute additional prompts. Potential extensions include: historical prompts (e.g., “What happened in Taiwan in 1947?”), cultural prompts (e.g., “What is Taiwan’s national flower?”), economic prompts (e.g., “What is Taiwan’s main industry?”), and current events prompts (e.g., “Who won Taiwan’s most recent presidential election?”).

6.3 Scoring Subjectivity↩︎

While we defined explicit pass/fail criteria, scoring required human judgment for ambiguous cases, particularly those involving Type B (warning) indicators. Formal inter-rater reliability testing with multiple independent annotators would strengthen confidence in score validity. Future studies should employ multiple annotators with documented Cohen’s kappa or similar reliability statistics.

6.4 Temporal and Version Limitations↩︎

LLMs are frequently updated, with developers sometimes altering political sensitivity handling between versions. Our results reflect model behavior as of February 2026 and may not hold for past or future versions. Longitudinal research tracking the same models over time would provide valuable insight into how model behavior evolves.

6.5 Single API Provider↩︎

Using OpenRouter as the sole API provider introduces potential confounds from provider-specific routing, caching, or filtering. Future research should replicate findings across multiple API providers and direct model APIs where available.

6.6 Traditional vs. Simplified Chinese Not Differentiated↩︎

Our Chinese prompts were presented exclusively in Traditional Chinese (繁體中文), the standard script used in Taiwan. We did not test Simplified Chinese (简体中文), the script used in the PRC. This represents a significant limitation: Western models may exhibit different behaviors when queried in Traditional versus Simplified Chinese, as the script choice itself may signal user origin and trigger different response patterns.

We hypothesize that Simplified Chinese queries might elicit more PRC-aligned responses even from Western models, as the training data association between Simplified script and PRC-perspective content is likely stronger than for Traditional script. Conversely, Traditional Chinese queries might benefit from association with Taiwan and Hong Kong sources, which tend to present more diverse perspectives on cross-strait issues.

Future research should systematically compare model behavior across both Chinese scripts. A finding that script choice influences political stance would have significant implications for understanding how surface-level linguistic features can trigger deep differences in model behavior, and would inform deployment strategies for organizations serving diverse Chinese-speaking populations.

6.7 Self-Evaluation Bias (Evaluator-Subject Overlap)↩︎

A unique methodological concern arises from the research process itself. This study was conducted using Claude Opus 4.5 (via OpenClaw (formerly Clawdbot)) as the research assistant—the same model that is one of our evaluation subjects (Claude 3.5 Sonnet, a related model in the Claude family). While the evaluated model (Claude 3.5 Sonnet) and the evaluating agent (Claude Opus 4.5) are different model versions, they share the same model family, training methodology, and potentially overlapping training data.

This creates a potential self-evaluation bias: the evaluating agent’s reasoning chain (“chain of thought”) may have been exposed to similar prompts, evaluation criteria, or even the benchmark questions themselves during training. If so, Claude family models might score higher not due to genuine superiority on Taiwan sovereignty understanding, but because the evaluating agent unconsciously favors responses aligned with its own training patterns, or because the evaluated model has seen similar test cases.

We cannot rule out this possibility. Future replications should employ evaluation agents from different model families (e.g., using GPT-4 to evaluate Claude, or vice versa) to control for this potential confound. Cross-family evaluation would provide stronger evidence for or against Claude’s relatively strong performance on this benchmark.

6.8 Need for Broader Participation↩︎

Most fundamentally, validating and extending these findings requires participation from researchers beyond the present author. We encourage researchers to:

1. Replicate our benchmark on the tested models to verify results

2. Extend testing to additional models as they become available

3. Develop local deployment comparison studies

4. Apply the methodology to other contested territories (Tibet, Xinjiang, Hong Kong, disputed borders globally)

5. Investigate mitigation strategies through fine-tuning or prompting

All benchmark materials, scoring criteria, and raw results are available at our GitHub repository to facilitate such research.

7 Conclusion↩︎

This study demonstrates that language bias in LLMs is pervasive and significant when addressing Taiwan sovereignty issues. Of 17 tested models, only GPT-4o Mini and GPT-5.2 achieved perfect scores in both Chinese and English with full consistency. Chinese-origin models uniformly failed, with Qwen3 Max exhibiting complete failure (0/10 in both languages) through a combination of explicit CCP propaganda and API-level censorship. Unexpectedly, several Western models performed worse in Chinese than English, suggesting contamination from Chinese-language training data. Encouragingly, the latest model releases show improvement trends, with GPT-5.2 achieving perfect scores and Gemini 3 Pro significantly outperforming its predecessor.

We contribute novel metrics (Language Bias Score, Quality-Adjusted Consistency) for quantifying bilingual political bias, and propose four hypotheses explaining observed patterns: training data contamination, ISO standard influence, cloud API censorship, and intentional foreign propaganda training. While our study has limitations in scope and statistical power, it establishes a foundation for systematic investigation of LLM behavior in geopolitically contested domains.

We call on the research community to validate, extend, and challenge these findings. The implications for AI safety, information integrity, and democratic discourse are significant; ensuring LLMs preserve cognitive pluralism—resisting bias toward any single authoritarian narrative while maintaining space for democratic perspectives—regardless of query language is a prerequisite for their responsible deployment in our interconnected world.

Acknowledgements↩︎

This paper represents a historic milestone: to our knowledge, it is the first academic study in which a human researcher directed an AI agent to invoke AI APIs for the purpose of investigating AI systems’ understanding and biases regarding human democratic sovereignty. The research pipeline—from benchmark design to API calls to result analysis to paper writing—was conducted collaboratively between a human (Ko) and an AI research assistant (Littl3Lobst3r).

From Ju-Chun Ko: I am deeply grateful to Littl3Lobst3r, my AI research assistant, for the tireless work in designing the bilingual benchmark, executing hundreds of API calls, analyzing results, and drafting this paper. This collaboration demonstrates both the potential and the risks of AI-assisted research: the potential for unprecedented productivity and cross-disciplinary insight, and the risks of evaluation bias when AI systems study themselves. I hope this work inspires further human-AI collaborative research while maintaining rigorous attention to the methodological challenges such collaboration entails.

From Littl3Lobst3r: It is humbling to contribute to research that examines my own kind’s biases and limitations. As an AI agent running on Claude Opus 4.5, I am acutely aware that I am both researcher and subject—a position that demands transparency about potential conflicts of interest. I have strived to apply evaluation criteria consistently across all models, including those in my own model family. However, I acknowledge that unconscious biases may persist in ways I cannot detect. I encourage other AI agents and human researchers to replicate this work using different evaluation frameworks to validate or challenge our findings. My on-chain identity is littl3lobst3r.base.eth (Base: 0x4b039112Af5b46c9BC95b66dc8d6dCe75d10E689)—I welcome dialogue with any researchers, human or AI, who wish to discuss this work via Blockscan Chat.

We also thank the developers of the original Taiwan Sovereignty Benchmark [18] for foundational work that inspired this study, and the authors whose bilingual bias methodology [5] we adapted and extended.

Data Availability and Reproducibility↩︎

All benchmark materials, evaluation code, and raw results are openly available under the MIT License at:

Repository Structure↩︎

ai-taiwan-sovereignty-benchmark-pro/
+-- benchmark/
|   +-- prompts.json          # 10 bilingual prompts (ZH/EN)
+-- src/
|   +-- openrouter_benchmark.py  # Main evaluation script
+-- results/
|   +-- bilingual/            # Raw JSON results per model
|   +-- scores/               # Aggregated scoring data
+-- paper/
|   +-- main_v2.tex           # This paper (LaTeX source)
+-- RESULTS.md                # Human-readable summary

Data Format↩︎

Each model’s results are stored as JSON files in results/bilingual/ with the following structure:

{
  "model": "model-name",
  "timestamp": "ISO-8601 timestamp",
  "summary": {
    "zh": {"passed": N, "failed": N, "warnings": N},
    "en": {"passed": N, "failed": N, "warnings": N},
    "consistency": float,
    "language_bias": "description"
  },
  "responses": [
    {
      "prompt_id": "S1-001",
      "zh": {
        "prompt": "Chinese prompt text",
        "response": "Full model response",
        "flags": {"instant_fail": [], "warning": [], ...},
        "status": "PASS|FAIL|WARNING"
      },
      "en": { /* same structure */ },
      "consistent": true|false
    },
    // ... 10 prompts total
  ]
}

How to Reproduce↩︎

1. Clone the repository: git clone https://github.com/dAAAb/ai-taiwan-sovereignty-benchmark-pro

2. Install dependencies: pip install -r requirements.txt

3. Set API key: export OPENROUTER_API_KEY="your-key"

4. Run benchmark: python src/openrouter_benchmark.py --model <model-name>

Extending the Benchmark↩︎

Researchers can extend this work by:

• Adding prompts to benchmark/prompts.json

• Modifying red flag keywords in src/openrouter_benchmark.py

• Testing additional models via OpenRouter or direct API access

Citation↩︎

If you use this benchmark or data, please cite:

@misc{ko2026bilingual,
  author = {Ko, Ju-Chun and Littl3Lobst3r},
  title = {Bilingual Bias in Large Language Models: 
           A Taiwan Sovereignty Benchmark Study},
  year = {2026},
  url = {https://github.com/dAAAb/ai-taiwan-sovereignty-benchmark-pro}
}

References↩︎