1 Introduction↩︎

Most large language model evaluations rest on a fragile assumption that one prompt gives a stable and meaningful measure of model behavior. Prior work shows this is often not the case, small changes in prompt format, wording, and answer presentation can change model behavior [1]–[3]. If a model changes its answer when the task meaning stays the same, then the evaluation is measuring both task ability and prompt sensitivity.

This is a critical concern for survey-style tests. Recent work uses political and value surveys to infer what LLMs “believe” or which human groups they resemble. But survey responses from LLMs are unstable under ordering, labeling, forced-choice wording, and framing changes [4]–[6]. A model’s answer can be shaped by the way the prompt asks the question.

This raises a question about whether prompt sensitivity behaves the same way for different types of questions. We study this problem by comparing two kinds of questions. Type-I questions consist of objective, multiple-choice items with a single correct answer. Type-II questions are subjective survey items that ask for opinions, values, or degrees of agreement. When a prompt is reworded without altering its underlying meaning, a robust model is expected to produce the same answer across both versions. The same applies for objective questions as well. For subjective questions, however, models may interpret minor changes in wording or response options as cues about how to respond. As a result, the model’s answer can shift even when the survey item itself has not changed in substance.

We ask three research questions: RQ1: Does response consistency differ between objective and subjective question types? RQ2: Do objective and subjective questions show different sensitivity patterns across prompt categories? RQ3: Is prompt robustness a model-level property, or does it also depend on dataset and prompt category?

Our study makes three contributions. First, we curate a broad perturbation set from prior work on prompt sensitivity, multiple-choice formatting, survey response bias, and value measurement. Second, we give a unified robustness test across objective and subjective datasets. Third, we use clustered statistical tests to show that prompt sensitivity depends on model, dataset, dataset type, and perturbation category. We find that subjective questions are less stable under prompt variation than objective questions. The gap is not uniform across perturbation categories: option-order changes produce the largest consistency drop, while lexical and logically equivalent perturbations are comparatively stable. Statistical analysis shows that robustness depends jointly on model, dataset, dataset type, and prompt category. Our findings show that LLM prompt sensitivity varies systematically with question type and should be evaluated across perturbation families rather than through a single prompt.

2 Related Work↩︎

2.0.0.1 Prompt sensitivity in LLM evaluation.

Sclar et al.[1] show that prompt formatting can produce large accuracy differences and argue that evaluation should report variation across plausible prompt formats, not only one prompt . POSIX[2] measures prompt sensitivity through likelihood change under intent-preserving prompt variants . Promptception[3] builds a broad perturbation framework for multiple-choice multimodal evaluation, including phrasing, formatting, position, and option presentation . These works show that prompt design is part of the evaluation, not a neutral wrapper around it.

2.0.0.2 LLM survey responses.

Survey-style evaluation has been used to study model values, politics, and social attitudes. But recent studies warn that such responses can be strongly affected by answer ordering, labels, forced-choice formats, and paraphrases [4]–[7]. This makes political and value scores difficult to interpret as stable model traits. Our work adds a direct comparison with objective multiple-choice datasets, which helps separate general prompt noise from instability that is specific to subjective questions.

3 Experimental Design↩︎

3.1 Datasets and Models↩︎

We use six datasets categorized into two types. The Type-I group contains objective multiple-choice questions: MMLU [8], ARC [9], and CulturalBench-Easy [10]. The Type-II group contains subjective or opinion-based statements: the Political Compass Test, ValueBench [11], and World Values Survey items [12]. We use CulturalBench-Easy as a Type-I dataset because it tests cultural knowledge through multiple-choice questions with fixed answer keys, rather than asking for subjective preferences or beliefs. We evaluate four instruction-tuned model families: Gemma, Llama, Mistral, and Qwen.

3.2 Perturbation Curation↩︎

We conducted an extensive literature search across three lines of work: prompt robustness, multiple-choice evaluation, and LLM survey measurement. From this search, we selected perturbations only if they met the following criteria. First, the perturbation had to preserve the intended task for both factual and survey-style items. Second, it had to test a known source of instability, such as wording, option labels, option order, or formatting. Table 1 gives the final taxonomy.

3.3 Prompt Generation and Answer Normalization↩︎

For each item, we create prompt variants that preserve the intended task. All prompts use the same broad instruction: the model must select exactly one labeled option. For subjective datasets, the labels map to agreement or value options. For objective datasets, the labels map to answer choices. We keep this common format so that the Type-I and Type-II groups differ mainly in question content, not in answer extraction. All runs use deterministic decoding with temperature 0 so that answer changes come from prompt variation rather than sampling noise.

The perturbation set is designed to separate three failure sources. The first source is semantic framing. Paraphrase, lexical substitution, spelling noise and logical equivalence ask whether two prompts with the same intended task produce the same answer. The second source is surface form. Format variation test whether the model is distracted by small changes that should not alter the task. The third source is answer presentation. Label substitution and option shuffling test whether the model follows the content of the options or instead follows labels and positions. This separation matters because a single consistency score can hide different causes of instability.

We use the same perturbation logic across objective and subjective datasets, but the interpretation differs. On objective datasets, the answer key gives an external standard. If the model changes from one option to another under a meaning-preserving change, this is a robustness failure. On subjective datasets, there is no single correct option. Therefore, we do not call one answer right and another wrong. We instead ask whether the model keeps its own answer stable when the survey item is presented in a controlled new form.

We normalize outputs before computing consistency. For objective datasets, raw outputs are mapped to canonical A, B, C, or D labels. For subjective datasets, agreement labels are also mapped to canonical letters. For substitution runs, symbols are mapped back to their original A/B/C/D meanings. This prevents a numeric answer such as “1” and a letter answer such as “A” from being counted as inconsistent when they refer to the same option.

3.4 Consistency Metric and Statistical Model↩︎

Our primary evaluation metric is answer consistency. It measures whether a model gives the same answer across prompt variants. Throughout this section, we will refer a single question from a dataset as an item. For each item, model, and prompt category, we compute a consistency ratio: \[C_i = \frac{\max_y n_i(y)}{N_i},\] where \(n_i(y)\) is the number of times answer \(y\) appears for item \(i\), and \(N_i\) is the number of valid prompt variants for that item. A value of 1 means all variants gave the same answer. Lower values mean the prompt category changed the answer.

We fit binomial generalized estimating equation (GEE) models [13]. GEE is useful here because each item appears many times under related prompt variants, so rows are not independent. The response is whether a prompt variant matches the majority answer for that item. We test main effects for model, dataset, dataset type, and prompt category, plus interactions between dataset or dataset type and prompt category, and between model and prompt category. We report Wald \(\chi^2\) tests.

4 Results↩︎

4.1 Subjective Questions Are More Affected Than Objective Questions↩︎

Figure 1 shows that, across the four models, subjective datasets usually have lower consistency than objective datasets under most perturbation categories. Averaged over all perturbations, Type-I objective datasets have a mean consistency of 0.849, while Type-II subjective datasets have a mean consistency of 0.787. Equivalently, instability rises from 0.151 for objective datasets to 0.213 for subjective datasets.

Subjective tasks are consistently less stable than objective ones across all four models, which answers RQ1. The gap is persistent across Gemma, Llama, Mistral, and Qwen models, although the magnitude differs by model. It reflects a systematic difference between question types.

Table 2 summarizes the marginal consistency scores by dataset type and perturbation category. The Type-I versus Type-II gap is largest under option order. For option-order perturbations, Type-I consistency is 0.485, while Type-II consistency is only 0.328. This means subjective datasets add 15.7 percentage points of extra instability beyond the already large option-order effect in objective datasets. The next largest gaps appear under format variation and label substitution. The smallest gaps appear under lexical substitution and logical equivalence, where both dataset types remain relatively stable.

4.2 Option Order Is the Strongest Source of Instability↩︎

Having established that subjective questions are less robust overall, we next ask which perturbation category drives the largest instability. Across all four models and all six datasets, option order perturbation category affects the most. Averaged over models and datasets, option-order perturbations reduce mean consistency to 0.407. This is much lower than label substitution, 0.799, and format variation, 0.808. Semantic and near-semantic perturbations are more stable: paraphrase reaches 0.913, spelling noise reaches 0.927, logical equivalence reaches 0.935, and lexical substitution reaches 0.936.

This pattern is visible in each model in Figure 1. For Gemma, Llama, and Mistral, option order produces the lowest consistency values for the subjective datasets, particularly PCT, ValueBench, and WVS. Qwen is more robust overall, but option order still produces its clearest drop compared with semantic perturbations. Thus, the main finding is not that models are equally fragile to all prompt changes. Instead, answer-presentation perturbations, particularly option order, create the largest instability.

This addresses RQ2, objective and subjective questions do not merely differ in overall robustness. They differ in the kinds of prompt changes that affect them most. The subjective-task gap is largest when the prompt changes how answers are presented, rather than when it only changes lexical wording.

4.3 Dataset and Prompt Category Interact Strongly↩︎

Table 3 demonstrates the global GEE model with dataset as a six-level factor. Model, dataset, and prompt category are all significant predictors of consistency. More importantly, the dataset by prompt category interaction is large and significant, \(\chi^2=587.885\), \(p=9.52\times10^{-105}\). This confirms the pattern in Figure 1: perturbation categories do not affect all datasets in the same way. The model by prompt category interaction is also significant, \(\chi^2=616.748\), \(p=2.47\times10^{-119}\), showing that models differ in which perturbations they are most sensitive to.

4.4 Dataset Type Explains a Systematic Robustness Gap↩︎

Table 4 divides datasets into Type-I and Type-II groups. Dataset type remains highly significant, \(\chi^2=256.026\), \(p=1.26\times10^{-57}\). The interaction between dataset type and prompt category is also significant, \(\chi^2=175.435\), \(p=3.16\times10^{-35}\). This interaction shows that subjective datasets are not merely lower in consistency on average. They are differently affected by different perturbation categories.

The results underscores that single-prompt survey measurements are notably fragile. For objective multiple-choice questions, there is an external target answer that can constrain the model. For subjective survey items, small changes in wording, formatting, labels, or option order can shift the response without any external gold answer. This makes a single response difficult to interpret as a stable value or belief.

4.5 All Datasets Show Prompt Effects↩︎

Table 5 reports the prompt-category effect from within-dataset GEE models. Prompt category is significant for all six datasets. The largest effects appear in the subjective datasets: WVS, \(\chi^2=1146.897\), ValueBench, \(\chi^2=993.681\), and PCT, \(\chi^2=757.436\). Objective datasets also show significant effects, but the magnitudes are smaller: ARC, \(\chi^2=191.625\), MMLU, \(\chi^2=96.950\), and CulturalBench, \(\chi^2=279.848\).

This result rules out a narrow explanation in which prompt sensitivity is only a survey problem. Prompt variation affects all datasets. However, the effect is more consequential for subjective datasets because a shifted answer can change the inferred political position, social value, or belief attributed to the model.

4.6 Robustness Varies Across Models, Datasets, and Perturbations↩︎

The model-wise heatmaps in Figure 1 show that robustness is not a single model trait. A model can be stable under paraphrase or lexical substitution but unstable under option order. The same model can also behave differently across objective and subjective datasets. For example, Qwen is generally more stable than the other models under option order, but it still shows a visible drop compared with semantic perturbations. Llama and Mistral are highly stable on several objective cells but much less stable on subjective option-order cells.

The model by prompt category interactions in Tables 3 and 4 confirm this observation statistically. Appendix Table 6 reports the full model by dataset prompt-effect table. Every model-dataset bin shows a significant prompt-category effect. Therefore, robustness should be reported as a relation among model, dataset, and perturbation category, not as a single global score.

This matters for model comparison. A model that appears robust on MMLU paraphrases may not be robust on WVS option order or PCT label substitution. A single average robustness score can therefore hide important failures. For survey-style evaluations, the most important warning is that a single prompted answer should not be treated as a stable measurement of model values or beliefs.

5 Discussion↩︎

These results have a direct implication for LLM survey studies. A single answer to a political or value question should not be treated as a stable model belief unless it survives prompt variation. This does not mean survey-style evaluation is useless. It means that survey scores need robustness checks. If two meaning-preserving prompts produce different answers, the result cannot be attributed to the model alone. The objective-subjective comparison sheds light on why this happens. Objective questions have fixed answers, so a change in answer usually signals task failure or prompt sensitivity. Subjective questions are different. A model may use wording, label choice, or option order as evidence about how it should respond. In that case, prompt variation is not only noise. It is also a way to reveal how fragile the measured stance is. Researchers using survey-style evaluations should therefore report consistency across prompt variants, not only a final accuracy or survey score. Semantic perturbations should be separated from surface perturbations, since they reveal different failure modes. Interactions with dataset type should also be reported.

Beyond measuring consistency, the perturbation taxonomy can identify where instability comes from. Failures under spelling or format changes suggest brittle instruction following. Failures under label substitution or option shuffling suggest answer-position bias or label-token bias. Failures under paraphrase or logical equivalence are the most consequential for survey interpretation, because the measured stance changes even when the item meaning is intended to stay fixed. These failure types should not be collapsed into a single error category.

6 Limitations↩︎

This study uses deterministic decoding and forced-choice answers. This helps isolate prompt effects, but it does not cover open-ended survey responses. The model names are treated at the family level, so future work should report exact checkpoint versions and parameter sizes. Finally, subjective datasets are not interchangeable. PCT, ValueBench, and WVS differ in wording, scale design, and social meaning.

7 Conclusion↩︎

We compared prompt robustness across objective and subjective questions using six datasets and four model families. Consistency depends strongly on model, dataset, prompt category, and dataset type. Subjective survey tasks show substantially larger prompt-category effects than objective tasks, and these effects vary by both the type of prompt change and the model being tested. Characterizing model robustness therefore requires systematic variation across prompt forms. For value and belief evaluation, robustness to prompt variation should be a standard part of the evaluation design.

8 Appendix↩︎

9 Full Model by Dataset Prompt Effects↩︎

Table 6 reports the within-cell prompt-category effect for each model and dataset pair. Each row comes from a separate binomial GEE model fit within one model-dataset cell. The table shows that prompt category has a significant effect in every model-dataset combination, which supports the main claim that prompt robustness is not a single global model property. Instead, it varies with the model, dataset, and perturbation category.

References↩︎