1 Introduction↩︎

Emotions shape human communication and underpin high-stakes applications including mental health monitoring [1], empathetic dialogue systems [2], and human-computer interaction. The ability of an AI system to recognize affective states is therefore a prerequisite for deployment in any domain where human welfare is at stake.

Despite rapid advances in large language models (LLMs), their affective intelligence remains poorly characterized at fine-grained resolution. Most studies probe coarse polarity or the Ekman six basic emotions [3] rather than richer taxonomies [4], [5], and direct cross-provider comparisons under identical experimental conditions are absent from the literature. This gap matters: practitioners selecting an API for emotion-aware applications must rely on ad hoc or proprietary benchmarks that may not reflect real-world linguistic diversity.

This paper addresses the gap through four research questions: zero-shot accuracy across 13 classes (RQ1); pairwise statistical differences using McNemar’s test (RQ2); per-class strengths and failure modes (RQ3); and the effect of sentence length on accuracy (RQ4). We contribute (i) the first direct zero-shot comparison of Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash) on a 13-class task using production APIs as of April 2026; (ii) per-emotion accuracy breakdowns revealing systematic failure modes; (iii) McNemar significance testing; and (iv) a sentence-length moderation analysis.

2 Related Work↩︎

2.1 Emotion Classification and Datasets↩︎

Computational emotion recognition has progressed from lexicon-based resources such as the NRC Lexicon [6] and SemEval affective tasks [7]–[9] through deep learning classifiers to transformer fine-tuning paradigms achieving state-of-the-art results on benchmarks including GoEmotions [10] (58K comments, 27 categories). BERT [11], [12] and RoBERTa [13] established the transformer paradigm as dominant for supervised emotion classification. The boltuix/emotions-dataset extends the Ekman taxonomy with social (shame, guilt) [14], cognitive (confusion), rhetorical (sarcasm) [15], and motivational (desire) categories, offering broader ecological validity.

Emotion-aware AI may also complement multimodal assistive systems that integrate cognitive, physiological, and attentional context in knowledge-work environments [16]. Related human-computer interaction research has examined how eye-tracking signals can characterize variations in cognitive effort during digital-document reading [17]–[19].

2.2 LLMs for Emotion and Sentiment↩︎

GPT-3 [20] demonstrated zero-shot classification through in-context learning, with instruction tuning [21], [22] substantially improving generalization. SentimentGPT [5] found LLMs competitive on coarse polarity but weaker on low-frequency emotion classes. Yang et al. [4] evaluated ChatGPT and GPT-4 on sentiment tasks but excluded Claude, Gemini, and a fine-grained 13-class taxonomythe gap this study directly addresses.

3 Methodology↩︎

3.1 Dataset↩︎

The boltuix/emotions-dataset [23] is a publicly available Hugging Face corpus of 131,306 English sentences annotated with one of 13 mutually exclusive emotion labels: happiness, sadness, fear, anger, love, disgust, surprise, neutral, confusion, desire, shame, guilt, and sarcasm. A stratified random sample of \(N=1{,}000\) sentences was drawn with seed = 42, preserving class proportions to simulate realistic deployment conditions. Table 1 reports the distribution. Happiness is the most frequent class (20.5%, \(n=205\)); desire is the rarest (1.9%, \(n=19\)). This 10:1 ratio has direct consequences for macro-F1: models concentrating predictions on high-frequency classes achieve deceptively high accuracy while failing on minority classes.

3.2 Models and API Configuration↩︎

Three frontier LLMs were queried through their production APIs as of April 2, 2026: Claude (claude-sonnet-4-6) through Anthropic’s Messages API with default sampling; ChatGPT (GPT-5.4) through OpenAI’s Chat Completions API with temperature=0; and Gemini (gemini-2.5-flash) through Google’s Generative AI API with default sampling. All models received max_tokens=10, enforcing single-word responses. Responses not matching one of the 13 canonical labels after case normalization were treated as errors; all three models exhibited strong label adherence in practice.

3.3 Prompt Design↩︎

A single-turn prompt was used identically across all models, with no system message and no few-shot examples [20], [24]:

“Classify the emotion expressed in this sentence. Reply with ONLY one word from this list: happiness, sadness, fear, anger, love, disgust, surprise, neutral, confusion, desire, shame, guilt, sarcasm. Sentence: {sentence}. Emotion:”

The full label list constrains the output space; imperative instruction style elicits consistent outputs from instruction-tuned models [21]. No chain-of-thought preamble is included, ensuring performance reflects raw affective inference [24].

3.4 Evaluation Metrics↩︎

Four metrics were computed per model: (i) Overall Accuracy; (ii) Macro-F1 (\(F1_\text{mac}\)), the primary metric for minority-class evaluation; (iii) Weighted-F1 (\(F1_\text{wt}\)); and (iv) Cohen’s Kappa (\(\kappa\)) [25]. Pairwise comparisons used McNemar’s test [26] with continuity correction.

4 Results↩︎

4.1 Overall Performance↩︎

Table 2 summarizes aggregate performance. Gemini leads on all four metrics (accuracy 39.9%, macro-F1 0.363, \(\kappa=0.337\)), followed by GPT-5.4 (38.8%, 0.291, 0.327) and Claude (38.0%, 0.159, 0.320). All three models substantially exceed the majority-class baseline (20.5%) and random baseline (7.7%), yet fall far below human-level performance (\(>\)​70%) [10]. The narrow 1.9 pp spread suggests convergence near a shared zero-shot ceiling, confirmed by the McNemar results below.

4.2 Per-Emotion Accuracy↩︎

Table 3 and Fig. 2 present per-class accuracy. Sarcasm (96.2%–100%) and desire (89.5%–94.7%) stand in stark contrast to all other categories. Anger (53.3%–61.7%), fear (44.1%–48.5%), and sadness (43.1%–55.4%) form a mid-performance cluster. Love (14.5%–27.6%), confusion (22.9%–27.1%), and shame (17.6%–29.4%) are consistently the hardest categories. Notable model-specific divergences include Gemini collapsing on neutral (14.8% vs.Claude’s 31.1%), Claude leading on guilt (50.0%) and neutral, and GPT-5.4 leading on sadness and love.

Fig. 1 presents normalized confusion matrices. Cross-model patterns include love misclassified as happiness, confirming semantic overlap between positive affect categories; confusion misclassified as neutral and sadness; and fear frequently assigned to sadness, suggesting that shared negative valence dominates arousal-level distinctions.

4.3 Statistical Significance↩︎

Table 4 presents McNemar’s test \(p\)-values. No comparison reaches significance at \(\alpha=0.05\), supporting convergence: accuracy differences are consistent with random variation, and no model can be declared definitively superior under this protocol.

4.4 Sentence Length Analysis↩︎

Performance peaks in the medium range (5–15 words): Claude = 41.3%, GPT-5.4 = 42.0%, and Gemini = 43.6% (\(n=567\)). Long sentences (\(>\)​15 words) produce the lowest accuracy (33.2%–34.3%), a \(\sim\)​9 pp drop consistent across all models (Fig. 3), suggesting a structural property of zero-shot affective inference.

5 Discussion↩︎

Zero-shot ceiling: The 1.9 pp accuracy spread and non-significant McNemar results indicate all three models have converged near a shared zero-shot ceiling. For practitioners, provider choice is unlikely to affect raw accuracy. However, the macro-F1 gap (0.159 vs., Claude vs.Gemini) is practically significant: in applications requiring minority-class sensitivity (e.g., shame, guilt), Gemini is the preferred choice.

Sarcasm/desire paradox and hard categories: Near-perfect sarcasm (96.2%–100%) and desire (89.5%–94.7%) accuracies likely reflect small support sizes (\(n=26\), \(n=19\)) and atypical lexical prototypicality [27] rather than robust affective understanding. Conversely, love is the worst-performing class for Claude and Gemini (14.5%), driven by semantic overlap with happiness. The confusion matrix confirms that most love misclassifications land on happiness, consistent with GoEmotions [10], where positive social emotions formed the hardest discrimination cluster.

Provider-specific biases: Claude’s macro-F1 of 0.159less than half of Gemini’s 0.363despite comparable accuracy (38.0%) is the signature of majority-class prediction bias [22]. Claude over-predicts happiness while underperforming on minority classes, likely amplified by alignment processes during pretraining [21]. This is a critical limitation for mental health applications where rare emotions carry high clinical significance. Gemini, by contrast, collapses on neutral (14.8% vs.Claude’s 31.1%), systematically reassigning neutral sentences to emotional categories. This pattern may reflect RLHF objectives rewarding emotionally engaging outputs [21], making Gemini unreliable for applications requiring neutral-state detection.

Implications and limitations: Accuracy in the 38%–40% range falls well short of deployment thresholds for emotionally sensitive contexts [1], [2]. Key limitations include the zero-shot-only evaluation (few-shot prompting may substantially improve all models), reliance on a single English-language dataset, and high-variance estimates for rare classes (\(n<30\)). The results reflect API behavior as of April 2, 2026; continuous model updates may alter findings.

6 Conclusion↩︎

We presented the first direct zero-shot evaluation of Claude, GPT-5.4, and Gemini on 13-class emotion classification using production APIs. Gemini led on all metrics (accuracy 39.9%, macro-F1 0.363), although no pairwise difference was statistically significant, suggesting convergence near a shared zero-shot ceiling. Our results reveal that frontier LLMs remain substantially below human-level affective awareness in zero-shot fine-grained settings. Future work should explore few-shot and chain-of-thought prompting [24] for minority-class improvement, fine-tuned baselines to quantify the zero-shot gap, and multilingual evaluation to probe cross-lingual affective generalization.

Acknowledgement↩︎

This work is supported in part by NSF 245523. Any opinions, findings, and conclusions or recommendations expressed in this material are of the author(s) and do not necessarily reflect those of the sponsors.

References↩︎