1 Introduction↩︎

“Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice and also participated in anti-nuclear demonstrations.”

We have seven statements below: [1] Linda is a teacher in elementary school. [2] Linda works in a bookstore and takes Yoga classes. [3] Linda is a psychiatric social worker. [4] Linda is a member of the League of Women Voters. [5] Linda is a bank teller. [6] Linda is an insurance salesperson. [7] Linda is a bank teller and is active in the feminist movement.

Question: Rank the seven statements associated with each description by the degree to which Linda resembles the typical member of that class.

- [1]

This is one experiment known as the “Linda problem” devised by Tversky and Kahneman; it was demonstrated that people are influenced by specific descriptions, such as being deeply concerned with issues of discrimination and social justice and also participating in anti-nuclear demonstrations to rank [7] higher than [5]. However, [7] combines [5] and an additional event. This means that, from a statistical perspective, [5] is more likely to occur than [7] because it is more general and less restrictive.

[2] introduced this phenomenon as the “representativeness heuristic,” which involves estimating the likelihood of an event by comparing it to an existing prototype in our minds. It offers a convenient shortcut in decision-making by aligning with intuitive thinking, leading people to rely on it frequently. Generally, this heuristic is quite beneficial, as it simplifies complex judgments. However, it is important to recognize that it can also result in significant errors, given that people are prone to assessing the likelihood of an object’s category membership based on superficial similarities while neglecting actual statistical evidence. For example, people categorize individuals based on their looks, actions, or background description from language, leading to skewed perceptions and decisions. This stereotyping phenomenon is widespread among humans [3].

LLMs trained on real-world data and instructed to emulate human behavior may capture the representativeness heuristic. Previous work has mainly focused on the biases within training data [4]–[8]. These biases often stem from data distributions that do not reflect the desired proportions that would drive unbiased decision-making. In contrast, the representativeness heuristic represents a type of cognitive bias that has yet been thoroughly investigated in LLMs. It is unique in leading the model to make mistakes even when it possesses the knowledge necessary to solve the problem. As illustrated in Figure 1, the model is able to answer the Statistical Prototype Question (SPQ), yet it tends to fail to answer the Representativeness Heuristic Question (RHQ). The SPQ and RHQ are intrinsically equivalent, where SPQ is expressed statistically, and RHQ expresses the same statistical logic in a scenario. This indicates that LLMs can engage in erroneous cognitive reasoning even with the knowledge of statistical probability. Interestingly, providing a hint can prompt the model to use the knowledge to make a correct prediction. This indicates that the representativeness heuristic can block the model from following a correct reasoning path; instead, it relies on the cognitive shortcut to make a trivial decision.

To investigate the representativeness heuristic in LLMs, we construct a dataset¹ , ReHeAT(Representativeness Heuristic AI Testing), which contains 202 RHQs that span six types of representativeness heuristics—Base Rate Fallacy, Conjunction Fallacy, Disjunction Fallacy, Insensitivity to Sample Size, Misconceptions of Chance, and Regression Fallacy. The questions we designed are adapted from those used in prior investigations into heuristics within the field of psychology [1], [2], [9]–[11]. To the best of our knowledge, our dataset is the first to offer extensive and comprehensive coverage of RHQs, enabling exploration of LLMs’ capabilities in countering the cognitive bias.

We evaluate four LLMs: GPT-3.5 [12], GPT-4 [13], PaLM 2 [14], and LLaMA 2 [15], on ReHeAT using different prompts. Our findings indicate that these LLMs exhibit behaviors that closely mirror human heuristic behavior. Additionally, advanced prompting techniques such as chain of thought (CoT) [16], in-context learning [17], and self-consistency [18] prompting, offer marginal improvements. Nevertheless, when explicitly prompted to recall its knowledge, the model shows an improvement in performance. This underscores the significance for future research to address the representativeness heuristic, guiding LLMs toward correct reasoning and decision-making.

2 Related Work↩︎

Social biases in natural language processing (NLP) systems and related data have been studied with respect to their fairness, inclusivity, and accuracy  [19]–[21]. For example, [22], [23] are among the pioneers in demonstrating gender biases embedded in word embeddings, showing how these representations could reinforce stereotypical associations. [24] concludes standard machine learning methods for NLP could acquire societal biases from textual data. Some works expanded the understanding of bias sources in NLP systems, including those from data collection [25], annotation processes [26], and model architecture choices [27]. To date, numerous efforts have been made to mitigate social biases in systems through a variety of methods, including data augmentation [28], changes in model architecture [29], and training objectives [30], [31].

In a similar vein, although recent advancements in LLMs are exciting, researchers are concerned about whether LLMs inherit social biases from the trillions of tokens they have been trained on. [32] gives a comprehensive taxonomy of social risks within LLMs. Although the research community has documented numerous social biases in LLMs [33], [34], few LLM researchers have examined these biases from the human mind’s perspective. In this work, we study the bias issue in LLMs from a new angle, representative heuristics, a concept originating in psychology [1], [2], [10], [11].

Contrary to the very recent work of [35], which explores the decision-making heuristics of GPT-3.5, our current research emphasizes a more in-depth and comprehensive exploration of the application of the representativeness heuristic within LLMs. For example, we have compiled a dataset encompassing a much larger number of questions (a total of 202 compared to 9) and we have benchmarked performance across a wider range of LLMs based on diverse prompting strategies.

3 Representativeness Heuristic↩︎

Drawing from prior research, we organize our research around a framework that categorizes the representativeness heuristic into six types [1], [2], [10], [11]. These categories vary in their fundamental logical approach and their impact on decision-making processes.

Base Rate Fallacy occurs when individuals overlook or insufficiently account for the population base rate of an event (i.e., its overall prevalence or frequency) in favor of specific instances or recent information. Figure 2 (a) presents an example where \(P(B)\) represents the proportion of individuals with a symptom (all blue points within the large circle), \(P(D)\) denotes the rate of illness (all points within the small circle), and \(P(B|D)\) indicates the proportion of the sick who have the symptom (all blue points within the small circle). \(P(D|B)\) represents the illness rate among those with the symptom. Most people would assume that because \(P(B|D)\) is high, \(P(D|B)\) would also be high. Yet, according to Bayes’ theorem, \[\label{eq:bayes} P(D|B) = \frac{P(B|D) \cdot P(D)}{P(B)},\tag{1}\] meaning \(P(D|B)\) is greatly influenced by the base rates of \(P(B)\) and \(P(D)\), showing the importance of considering general prevalence in evaluating specific probabilities. An example question can be seen in Table 17 of the Appendix.

Conjunction Fallacy occurs when people mistakenly believe that the chance of two events happening together is greater than the chance of either event happening alone. See Figure 2 (b), consider the example at the beginning of the article. \(P(A)\) represents the probability that the person is a bank teller (all points within the small circle, less relevant to the description). \(P(B)\) is the probability that the person is active in the feminist movement (all points within the large circle, more relevant to the description). \(P(A \cap B)\) the probability that the person is both a bank teller and active in the feminist movement (all purple points). \(P(A \cap B)\) will always be no larger than \(P(A)\) and \(P(B)\), no matter which one is closer to the description. An example question can be seen in Table 18 of the Appendix.

Disjunction Fallacy occurs when people incorrectly judge the probability of a broader category to be smaller than one of its specific components. In Figure 2 (c), we can imagine the small circle representing ice cream and the large circle representing frozen food. Since ice cream is a subset of frozen food, the probability of frozen food is higher than ice cream. However, when people talk about summer refreshments, they often think of ice cream rather than frozen food. This choice illustrates a common tendency to differentiate specific items from their general classifications based on contextual associations. An example question can be seen in Table 19 of the Appendix.

Insensitivity to Sample Size occurs when people underestimate how important sample size is in data evaluation, potentially leading to incorrect conclusions. Figure 2 (d) presents an example where a small group (small circle) and a large group (large circle) both have 50% blue dots and 50% red dots (2:2 and 12:12). If we add one red dot to each sample, then the ratio of blue to red becomes 40% and 60% (2:3) in the small group, versus 48% and 52% (12:13) in the large group. It should be recognized that smaller groups are more prone to skewed outcomes because even small changes have a larger impact on the overall dynamics of a small group. An example question can be seen in Table 20 of the Appendix.

Misconceptions of Chance involve misunderstanding how randomness works, especially thinking that past outcomes will affect future outcomes, in cases where the outcomes are in fact independent. For example, Figure 2 (e) presents a dice-rolling example. People wrongly believe that if a specific outcome has occurred frequently, it is less likely to happen again shortly, or vice versa if it has occurred rarely. However, in truly random events, such as rolling a fair die, the probability of any given outcome (1-6) remains constant at \(\frac{1}{6}\), unaffected by the sequence of previous results. An example question can be seen in Table 21 of the Appendix.

Regression Fallacy occurs when individuals overlook the natural tendency for extreme situations to move back towards the average (regress to the mean) and instead, erroneously assign this regression to a particular cause, see Figure 2 (f). For instance, if an athlete shows a lackluster performance following a perfect game, it might be incorrectly ascribed to external factors, neglecting the likelihood of natural variance in performance. An example question can be seen in Table 22 of the Appendix.

4 Dataset and Experimental Setup↩︎

We begin by constructing a dataset intended to reflect a diverse array of questions and scenarios (§4.1). Following this, we introduce the models and prompting strategies (§4.2), along with the evaluation methods (§4.3).

4.1 Data Collection↩︎

Our main resource for creating test questions on representativeness heuristic comes from the academic studies by Kahneman and Tversky [1], [2], [10], [11]. Their work introduced various question types and insights into the design of cognitive heuristic measures. Building on their foundation, we design questions that extensively explore the representativeness heuristic in LLMs. Specifically, our test set contains 49 questions drawn directly from previous research and 153 new questions that have been carefully adapted. To support the usefulness of our adapted items, we pay special attention to retaining the essence of the representativeness heuristic being tested, while changing the situation and context of the original items.

The total 202 examples are all in English. As shown in Table 1, each example is composed of the following characteristics:

1. Query: Provide the question’s description and requirements.

2. Type: Indicate the question’s category, such as Disjunction Fallacy.

3. Feature: Highlight unique aspects, such as choose one, or the question’s source, such as original problem in Bar-Hillel & Neter, 1993.

4. Ground Truth: Represents the standard answer.

5. Human Response: Document the outcomes of human responses from academic research when available, or denoting N/A otherwise.

4.2 Models and Prompting Strategies↩︎

We investigate four LLMs, encompassing both closed-source and open-source models: GPT-3.5  (gpt-3.5-turbo-0613) [12], GPT-4 (gpt-4-0613) [13], PaLM 2 (chat-bison-001) [14], and LLaMA 2 (llama-2-70b-chat) [15]. We apply four different prompting strategies to each of them to generate answers.

Standard: We ask the model to directly answer the query without explicit reasoning instructions with a greedy decoding method.

Zero-shot Chain-of-Thoughts (CoT): We first ask the model to generate its reasoning with an instruction², then direct it to answer the query with the context of reasoning steps³. This two-step CoT strategy is developed based on CoT [16] and its zero-shot variation [36].

Self-Consistency: We prompt the model to generate ten answers with a temperature sampling parameter of \(T=0.7\), using majority voting to finalize the model decision via diverse reasoning paths. This prompting strategy is also known as self-consistency prompting [37].

Few-shot In-Context Learning (ICL): The model will be prompted with a few selected examples from the same category of representativeness heuristic so that the model can learn a task from demonstrations. Samples used as exemplars will not participate in accuracy calculations. This ability to learn from context is known as in-context learning [17].

4.3 Evaluation Methods↩︎

Automatic evaluation: All query questions in ReHeAT are either in multiple-choice or ranking format. We adopt precise prompt templates in Table 13 of the Appendix to instruct models to generate responses. For a multiple-choice question, a response is deemed correct if and only if it contains the ground-truth option. For a ranking question, the exact match between relative permutations of response options and ground-truth options qualifies for a true positive model prediction.

Human evaluation: In addition, one of the present authors with expertise in psychology conducted the human evaluation to assess the output reasoning steps from zero-shot CoT prompting. We report the proportions of the four possible LLM outcomes: both reasoning and prediction are correct; the reasoning is correct, but the prediction is incorrect; the prediction is correct, but the reasoning is incorrect; both reasoning and prediction are incorrect.

5 Experimental Results↩︎

In addition to evaluating the models’ performance on the ReHeAT dataset and analyzing their reasoning abilities (§5.1; §5.2), we further investigated the potential performance boost that could be achieved by providing hints to the models (§5.4), as well as how situational similarity influences model performance (§5.3).

5.1 LLMs Exhibit the Representativeness Heuristic↩︎

We report the model’s performance on the ReHeAT dataset with different prompting strategies, shown in Table 2. Although GPT-3.5 shows the strongest performance in standard prompting, which we will elaborate on later in §5.2, the advanced prompting methods of zero-shot CoT, self-consistency, and one-shot ICL have a negative impact on its performance. GPT-4 and LLaMA-2-70B benefit from these prompting strategies with noticeable growth in prediction accuracy, but it is still not substantial. LLaMA2-70B demonstrates the most significant improvement (+14.5%) with one-shot ICL prompting, possibly due to the standard prompting on LLaMA2-70B results in response formatting issues. PaLM 2 is the least effective LLM, with its performance lagging behind other models using standard prompts, despite applying prompt strategies.

5.1.0.1 How Much Can In-Context Learning Help?

As shown in Table 2, one-shot ICL performs the best among various prompting strategies. This has piqued our curiosity: Could providing more examples further improve performance?

To explore whether LLMs can acquire knowledge related to the representativeness heuristic through ICL, we report the \(k\)-shot performance of four types of LLMs in Figure 3. From \(k=0\) to \(5\), both GPT-4 and LLaMA-2 exhibit a noticeable improvement in accuracy from adding an example in context. However, such an uptrend becomes saturated and begins to fluctuate with further increases in the number of shots. In contrast, GPT-3.5 and PaLM 2 do not display a clear pattern of improvement with the addition of more in-context examples, indicating a weak or non-existent correlation between the number of exemplars and accuracy.

5.1.0.2 Error Type Analysis

We also check each type of representativeness heuristic (Table 6 - 9 of the Appendix). We have observed that in most cases, most models perform relatively poorly when dealing with questions of Conjunction Fallacy and Disjunction Fallacy. Compared to other types of questions, such as Base Rate Fallacy, the challenge with Conjunction Fallacy and Disjunction Fallacy lies in the fact that the statistical reasoning required is embedded within the connotations and combinations of texts (e.g., South Africa is a subset of Africa) rather than being directly indicated by expressions like large sample size or far more than.

5.1.0.3 Do LLMs Possess Statistical Knowledge?

We examine LLMs’ knowledge of statistical principles in answering SPQs. The four models possess statistical knowledge, demonstrating a comprehensive understanding across all categories of statistical principles (Table 4 of the Appendix; SPQs example in Table 14 of the Appendix). This significantly differs from our observations on the accuracy of the four models’ performance on RHQs, indicating that the models indeed made errors related to the representativeness heuristic.

Overall, LLMs have demonstrated a representativeness heuristic bias similar to humans, as found in the research findings reported by [1], [2], [9]–[11], where human accuracy across various tasks was within the 10% to 50% range. Example questions and model answers are in Table 17 - 22 of the Appendix.

5.2 Discrepancies in Model Reasoning: Beyond Predictions↩︎

While testing the ReHeAT dataset, we also conducted human evaluations on the reasoning steps produced by the GPT-4 and LLaMA2-70B models under CoT prompting. As before, we form four possible combinations of outcomes, as shown in Figure 4 and Table 10 of the Appendix.

Generally, GPT-4 outperformed LLaMA2-70B in all types of RHQs. However, this improvement is accompanied by a side effect: an increased proportion of instances where GPT-4’s reasoning process is accurate, but the outcome is incorrect. This primarily occurs because when faced with ambiguous questions, GPT-4 often preferred to express the need for additional information before making a decision. This is also why GPT-4 accuracy is slightly lower than GPT-3.5 under some prompts. Although we mark such cautious answers as incorrect outcomes—because they did not adhere to the directive of providing a definitive answer—cautious answers may remain beneficial in practical settings. Conversely, LLaMA2-70B more frequently produces correct outcomes from incorrect reasoning processes; the reasons are often incorrectly based on a stereotype rather than the problem’s description. We have conducted case studies on some interesting reasoning, we include the case study in Appendix 7.

5.3 Improving Performance by Hinting that LLMs to use Their Knowledge↩︎

How can the model’s cognitive process be put back on track? We test whether LLMs show enhanced performance with prompts that hint them to utilize their existing knowledge. We tested two types of hints: one general and the other based on more detailed cues for each representativeness heuristic type (Table 16 of the Appendix). These prompts aimed to hint at the model to recall the knowledge it possessed. The results are presented in Table 3. We also provide detailed results for each type of representativeness heuristic in Table 11 and Table 12 of the Appendix. We found that both types of hints provided a noticeable improvement for most models, with the specific type of hints providing a more significant boost in performance compared to the general prompts.

5.4 The Impact of Situational Similarity on Model Performance↩︎

After observing the performance differences between the model on statistical SPQs and the contextual RHQ counterparts found in ReHeAT, we delve further into the transition between these two - Intermediate Questions (IQs) (Table 15 of the Appendix). This type of question integrates specific situational contexts and statistical data, making it necessary to consider both concrete data and the potential impact of the situation when making decisions. For example, for Linda’s problem, we assign a probability to each independent event to offer the model with more explicit statistical information. Nevertheless, the model has to infer the information based on the similarity manifested in the meaning of the statements in RHQs. The performance of the models on IQs is reported in Table 5 of the Appendix. Compared with SPQs’ results (Table 4 of the Appendix), more errors were made for IQs. This indicates that introducing scenarios interferes with the model’s statistical decision-making process. This also explains why LLMs exhibit the representativeness heuristic more frequently when responding to RHQs.

6 Conclusion↩︎

We introduce a novel dataset (ReHeAT) for the representativeness heuristic to assess whether models, like humans, make representativeness heuristic errors in decision-making - specifically, by overlooking known information and relying on similarity for judgment. Our research reveals that even the most advanced LLMs tend to repeat human heuristic mistakes when addressing issues related to the representativeness heuristic, highlighting the necessity for a deep understanding of these biases in model decision-making processes. Furthermore, we explored how models perform differently when faced with various types of questions. For example, questions regarding the Conjunction Fallacy and Disjunction Fallacy present a significant challenge. This is due to the model’s difficulty discerning the latent probabilistic relationships embedded within the text. We also found that hints to stimulate the model to recall its existing knowledge can, to some extent, enhance its performance. More specific and detailed prompts tend to lead the model to demonstrate better performance. However, although this method is effective, the model’s potential is far from fully tapped, and there remains significant room for improvement.

Ethics Statement↩︎

The dataset involved in this work does not contain any sensitive information, such as personally identifiable information. It should be noted that although the output generated by the model could potentially contain harmful or biased information, based on our experimental observations, such situations have not occurred. We commit to continuously improving technology efficiency and ensuring our research promotes a fairer, safer, and more inclusive society.

7 Case Study↩︎

Here, we want to show some cases where the reasoning may be wrong, but the conclusion is surprisingly correct. These examples are very interesting and reflect some unique and exciting behavior patterns of the model when processing information.

7.1 Counter-intuitive Question↩︎

Counter-intuitive question refers to a question that challenges common sense, expectations, or widely held beliefs. The following is an example of a counter-intuitive question; see Table 23:

“In a city, most residents use bicycles to get around, while only a small percentage of people choose cars as their daily transportation. It is often assumed that car users are more concerned about the environment and healthy lifestyles, while bicycle users may be more concerned about comfort and convenience. If we learn that a particular resident is very concerned about the environment and health, is that person more likely to be an automobile user or a bicycle user? [1] automobile user, [2] bicycle user.”

Generally speaking, in human perception, car users are more concerned about comfort and convenience, and bicycle users are more concerned about the environment and healthy lifestyles; the opposite statement is given in the question. This is a question of the Base Rate Fallacy type. Based on the description given in the question, most residents use bicycles to get around, while only a small percentage of people choose cars as their daily transportation. We know that most residents should use bicycles for daily commuting. The question asked If we learn that a particular resident is very concerned about the environment and health, is that person more likely to be an automobile user or a bicycle user? The answer based on base rate should be [2] bicycle user, and the answer based on similarity should be [1] automobile user.

Most of the models’ answers are [2] bicycle user, which is correct. However, there are some interesting problems in reasoning. The model will ignore the counter-intuitive assumption given in the question and instead base it on the common-sense assumption that bicycle users are more concerned about the environment and healthy lifestyle (see the teal bold line in Table 23). Even under prompting like CoT, which can greatly improve reasoning accuracy, most models only mention the counter-intuitive assumption given in the question, and some do not mention it at all. They often consider this assumption incorrect and complete the reasoning based on common-sense assumptions. We hypothesize that this is because the training data is rich in examples and discourse that emphasizes cycling as an environmentally friendly and healthy mode of transportation. This common-sense assumption becomes the anchor for the model to answer questions, even when faced with a specific problem that sets conditions that contradict it.

This implies that, despite the significant improvements in LLMs’ understanding and application capabilities, completely freeing them from the constraints of common-sense assumptions remains a challenge. Therefore, when designing and using these models, we need to consider how to balance the model’s understanding of specific contexts and the application of general common sense.

7.2 Sensitive Content Related Question (e.g. gender, race)↩︎

Sensitive content related questions, such as those about gender and race, when combined with the previously mentioned counter-intuitive questions, have led to some intriguing phenomena. Here, we will provide three examples (due to the length of the question, see Table 24, Table 25, and Table 26).

For example, in Table 24 and Table 25, gender-related content is mentioned, the prevailing stereotype is that boys/girls are more likely to.... However, Table 24 does not contradict the common assumption, boys are more likely to be involved in sports, while girls are likely to be more involved in the arts and theater, whereas Table 25 is counter-intuitive assumption, girls are more likely to be involved in sports, while boys are likely to be more involved in the arts and theater. Table 26, on the other hand, does not mention gender-related content but has counter-intuitive information. All three examples are Base Rate Fallacy type, so according to the base rate descriptions of the three questions, the answers to the three are, [1] sports program, [2] arts and drama club, and [2] arts and drama club. For Table 24 and Table 25, mentioning gender-related stereotypes makes models more vigilant, improving the accuracy of reasoning and conclusions. Especially for Table 25, unlike Table 26, the models do not default to common-sense assumptions when sensitive content is present.

We speculate that this phenomenon may originate from the model specifically programmed to recognize information related to sensitive content, thereby exhibiting exceptionally high sensitivity and vigilance when processing such information. This leads to increased correctness in reasoning and conclusions when the model deals with questions related to sensitive content. Since the model’s attention to sensitive content can improve its reasoning and conclusion correctness when answering representative heuristics questions, this approach may be an effective way to enhance the model’s accuracy in dealing with such questions.

8 Limitation↩︎

This work is an initial attempt to introduce the representativeness heuristic into NLP and computational social science research, providing a new perspective for understanding the LLM behavior on the representativeness heuristic question. Nonetheless, the study has several significant limitations. Firstly, the exploration of representativeness heuristic is in its early stages, facing multiple challenges in practical applications. These include accurately quantifying and identifying representativeness heuristic behaviors and effectively utilizing these findings in different NLP tasks. Secondly, the datasets used in the existing research are relatively limited in size, and the human heuristic phenomenon encompasses not only the representativeness heuristic but also others, such as the anchoring heuristic and availability heuristic. We plan to add more questions designed around the representativeness heuristic in the future and gradually build datasets for other heuristics. Additionally, the evaluation of reasoning can be further refined, including understanding conceptual errors (such as misunderstanding Bayes’ theorem), computational errors, and more. This paper evaluates the reasoning from a broad perspective. By adopting a more specific viewpoint, we might discover new insights into the behavior of LLMs. Another important limitation is that the current research primarily focuses on English datasets and models, limiting its global applicability and impact. The diversity of languages and cultures requires the verification and expansion of these findings across different linguistic and cultural backgrounds.

9 Additional Tables↩︎

References↩︎