When designing a visualization, one of the goals of a designer is to communicate the data effectively to the viewer. This typically means representing the data using visual encodings which are easy for viewers to decode. To identify the relative effectiveness of visual encodings for a particular task, visualization researchers turn to empirical studies where they compare how well participants perform on a task using two or more visual representations. The encoding which allows the average participant to perform better on a task is then considered more effective for that task. However, in the process of designing an experiment researchers need to make many, often subjective or ad hoc decisions that may have unknown effects. Here, we draw attention to the potential issue of researcher degrees of freedom in the empirical studies that visualization researchers use for determining the effectiveness of various visual encodings, and how this flexibility can impact our theoretical understanding.

This work was partly motivated by our prior experience designing an experiment to study the effect of a new visualization on a decision-making task [1]. Initially, we provided participants with a description of how the data was encoded in the visualization and what the information meant; yet we observed a lot of variance in how participants interpreted the visualization and used the information to make decisions. We speculated that some of this variance could be attributable to people misunderstanding how to read the chart [2]. Subsequently, when we included more explicit information on how to read the chart during onboarding, we observed less heterogeneity in participants’ responses. Since our goal was to evaluate the effectiveness of the visual representation, and not whether participants understood the representation correctly, we deployed the modified onboarding procedure.

For a typical experiment, researchers have to make several such experimental design decisions that can potentially have an impact the results. These include: how extensive the instructions should be, or whether participants should be trained on the task, given feedback, provided with performance-based monetary rewards, etc. However, these decisions are often made implicitly or even in an ad-hoc manner, and rarely articulated in a paper. 1 At the same time, there is insufficient clarity in the literature on the impact of these decisions, so the ad-hoc nature of these decisions and the lack of justification are understandable. We refer to these degrees of freedom in experimental design decisions as tacit factors. These decisions mirror the oft criticized undisclosed flexibility that researchers have in making decisions at various steps in the data analysis process  [3], [4], but occur instead during the experimental design stage.

In this work, we take a first step of investigating one such factor: the impact of performance-based financial incentives—rewarding correct or optimal decisions by participants through monetary bonuses—in crowdsourced visualization experiments. While some recent visualization studies, primarily on decision-making under uncertainty, have incentivized participants [1], [5]–[9], most empirical studies in visualization do not. However, we do not know how or if the decision to incentivize participants’ performance with money affects the results or the conclusions that we can draw from the studies. This can make it tricky to compare and interpret results across studies which differingly incentivize participants but are investigating performance on the same task and using the same set of visualizations. The theoretical argument for employing incentives, put forth primarily by economists, is that they induce more effort, which leads to improved performance on a task. However, we are not aware of any formal experiments on the effects of monetary incentives in crowdsourced visualization studies, a gap we attempt to close with this work.

We chose to study incentives over other tacit factors because we expected them to be fairly straight-forward to test in a controlled experiment, and to have a significant impact on participant performance. As visualization studies encompass a wide spectrum in terms of complexity, the types of cognitive processes that are involved, etc., incentives might impact different types of studies differently. We chose two prototypical visualization studies—perception of correlation [10]–[12], which involves a lower-level perceptual task; and decision-making under uncertainty [1], [9], [13], [14], which involves a higher-level reasoning task—to examine the effect of incentives on. We conducted two preregistered crowdsourced experiments where we varied incentives: participants were either paid a fixed amount independent of their performance, or were paid a smaller base amount, with the majority of their compensation tied to their performance.

In study 1 (perception of correlation), we partially replicated Harrison et al’s experiment [10]. We visualized pairs of positively correlated datasets using either scatterplots or parallel-coordinates plot and asked participants to choose the plot with the greater correlation of the two shown. We expected incentives to have no impact on performance (using just noticeable difference as the metric) in this scenario, as the difference between the plots either is or is not evident, and additional effort would likely not make a difference on task performance.

In study 2 (decision-making), we replicated the decision-making task from Joslyn et al. [13], [15], [16], using either an interval or a density plot to show the probability of temperatures falling below freezing, and tasked participants to make decision on whether to spend a budget to salt roads, or to preserve their budget and forgo salting. We expected incentives to have an effect on performance (using expected utility as the metric) in this scenario, as participants must make judgements about the optimal decision and estimate probabilities based on the visualized uncertainty distribution, where we expected additional effort to improve outcomes.

In both studies, our results showed no significant performance differences between the incentivized and non-incentivized conditions. While these are results of only two studies and therefore the findings need to be replicated before stronger conclusions are drawn, they seem to suggest that, at least in the context of crowdsourced studies on platforms such as Prolific, performance-based financial incentives may have fewer benefits than we presumed. 1Moreover, we do find some evidence that participants spend more time performing the task in the incentivized conditions, which raises some important ethical considerations for researchers regarding ensuring fair wages for crowdworkers.Moreover, we do find some evidence that participants spend more time performing the task in the incentivized conditions, which raises some important ethical considerations for researchers regarding ensuring fair wages for crowdworkers. We discuss the implications of our results, their possible causes, and the scope in which our results may generalize.

1 Background↩︎

1.1 Researcher Degrees of Freedom in Scientific Studies↩︎

During the course of designing, collecting and analyzing data, and reporting the results of empirical studies, researchers have to choose between several reasonable alternative choices [17], [18]. In the absence of theoretically or empirically grounded guidance, they often make these choices arbitrarily. Also referred to as researcher degrees of freedom, this flexibility, especially in the data analysis stage of a study, has received considerable attention in recent years as it can considerably increase the probability of a false-positive finding [3], [4], [19]–[22]. Our focus in this work is instead on the degrees of freedom that exist in the design stage of an empirical study (tacit factors). These tacit factors make it difficult to “establish the generalizability of an effect across contexts” [23], as, at least in the case of visualization and hci research, different study designs (e.g., whether or not participants are trained on how to read a chart) actually correspond to different contexts of usage [24].

1.2 The Effect of Financial Incentives↩︎

While the effect of incentives in experiments have been explored in prior work, primarily in the field of behavioral economics, the findings have been mixed. While some studies have found incentives to improve performance, most studies have found no effect or even a detrimental effect of incentives on performance [25]–[27]. Many of the studies which found that incentives did not improve performance also found that incentives reduced variance in responses. Based on their review of prior work investigating the effect of incentives, Camerer and Hogarth [25] propose the capital-labor-production theory to describe the potential effect of incentives in tasks. Here, capital refers to “cognitive capital” or knowledge that participants possess coming into the task. labor refers to effort that is required to perform the tasks. In most visualization and hci studies, participants primarily exert cognitive effort. Production refers to the kinds of capital that is necessary for performing a task.

The capital-labor-production theory [25] rests on a few key intuitions: (i) people dislike exerting effort, but will put in more effort if that means increased rewards; (ii) effort generally, monotonically, improves performance; (iii) capital can be a substitute for labor (for example, if one is familiar with the biases in area perception implied by Steven’s power law [28], they may be more accurate at comparing areas of two circles, but someone who does not know of these biases can be just as accurate by comparing the radii and computing the areas mathematically, which entails more effort); (iv) if a task’s production requirements (i.e., the effort it requires) are too low or too high, there would be little marginal gain for the participant from the extra effort that they would exert due to the presence of incentives.

One important caveat is that these studies were often conducted in physical lab spaces (not on crowdsourcing platforms), and involved very different types of tasks than what we typically employ in visualization studies. In a study conducted with crowdworkers on Mechanical Turk, Mason and Watts [29] found that participants performed a greater number of tasks as pay increased, but the accuracy of the performed tasks did not improve with increases in pay. However, the tasks that participants were asked to perform in these experiments were either quite menial (ordering images temporally) or were word games, both of which are tasks where incentives are unlikely to help [25]. 1Other studies have investigated factors that can influence performance through an interacting effect with incentives such as intrinsic motivation [30]. As our goal was to study the total effect of incentivesregardless of participants’ intrinsic motivation, we do not control for motivation in our study design or analysis [31].Other studies have investigated factors that can influence performance through an interacting effect with incentives such as intrinsic motivation [30]. As our goal was to study the total effect of incentivesregardless of participants’ intrinsic motivation, we do not control for motivation in our study design or analysis [31].

Based on the literature [25], we may expect the effect of incentives to manifest through improved performance on certain (but not all) tasks. Specifically, the tasks which are most likely to effected by incentives are those where most or all participants possess the required cognitive capital to perform the task, and the task does not require too little or too much effort. 1To investigate the effect of incentives, we chose two visualization tasks—(1) perception of correlation and (2) decision-making under uncertainty—which require varying degrees of cognitive capital and effort.To investigate the effect of incentives, we chose two visualization tasks—(1) perception of correlation and (2) decision-making under uncertainty—which require varying degrees of cognitive capital and effort. In addition, we may expect reduced variance in participants’ responses, and an increase in the amount of time participants spend on the task (which might be a proxy for increased effort).

1.3 Perception of Correlation↩︎

There has been a great deal of work in visualization and graphical perception on ranking different visual representations on their ability to help viewers accurately estimate correlation (and small differences in correlation). How accurately a viewer is able to perceive correlation using a specific representation is often determined by estimating just-noticeable differences (jnds)—the minimum difference in stimuli at which a viewer can reliably (75% of the time) detect a difference.

Prior work has found that viewers can perceive correlation fairly accurately when the data is represented using scatterplots [10], [11], and the jnds can be described using Weber’s law. Harrison et al. [10] evaluate perceptions of correlation using other visualizations such as parallel coordinate plots, stacked bars, ordered lines etc. The work of Harrison et al. [10], and a re-analysis by Kay and Heer [32] show that participants’ perceptions of negative correlation are almost as accurate when the data is represented using either scatterplots or parallel coordinates; however, participants’ perception is much less precise for positive correlation when the data is represented using parallel coordinates (compared to scatterplots).

Perception of correlation is often considered to be a more basic or lower-level cognitive function [33]. Prior work on perceptions of correlation have exclusively paid participants a flat amount for participating in the study. 1Our hypothesis was that if incentives were to have a meaningful effect, we might observe that a visualization such as the parallel coordinates plot, which is less accurate compared to scatterplots for positive correlation in the absence of incentives, is as accurate as scatterplots when incentivized. However, based on the findings of Camerer and Hogarth [25], we speculate that this task is unlikely to be impacted by incentives as it might not benefit from exerting additional effort. Thus, this study represents an important edge case which we seek to evaluateOur hypothesis was that if incentives were to have a meaningful effect, we might observe that a visualization such as the parallel coordinates plot, which is less accurate compared to scatterplots for positive correlation in the absence of incentives, is as accurate as scatterplots when incentivized. However, based on the findings of Camerer and Hogarth [25], we speculate that this task is unlikely to be impacted by incentives as it might not benefit from exerting additional effort. Thus, this study represents an important edge case which we seek to evaluate. In our study, we compare scatterplot and parallel coordinates plot for positive correlations. 1 as prior work has found, in the absence of incentives, scatterplots to be more perceptually accurate

1.4 Decision-making under Uncertainty↩︎

A growing number of studies have investigated the effectiveness of different visual representations in accurately extracting probability information from uncertainty representations [34]–[36], assessing risk [37]–[39], or making utility-optimal decisions [5]–[8], [15]. In this work, our focus is on the effect of visualizations in helping people make rational (i.e., decisions which are or close to utility-optimal). Prior work has found that uncertainty representations which are more expressive and present viewers with more complete distributional information such as density or violin plots, quantile dotplots, cumulative density plots lead to better decisions compared to interval plots which communicate a 95% (or similar) confidence or credible interval [5]–[7].

In these types of studies, researchers have often used financial incentives [5]–[7], [9] based on the notion that incentivising participants will increase both the internal and ecological validity of a study [7], [40], [41]. When making decisions, people, both in the real world and in experiments, are likely trading off some subjective costs against some subjective benefits. By explicitly stating what these costs and benefits are to a participant, incentives, in theory, provide benchmarks for defining optimal or good decisions, as well as enable the evaluation of participants’ decisions against the benchmark [6], [40]. As incentives reflect the costs and benefits associated with real-world decision-making, it also should make the findings more ecologically valid [7], assuming that the incentives in the experiment align well with the relative costs of real-world tasks.

These types of decision-making tasks require participants to perform relatively more complex (or higher-order) reasoning tasks, which require more effort. In our study, we examine the impact of incentives by comparing how participants perform in a decision-making task where the uncertainty distribution is represented using either an interval representation—specifically 66% and 95% intervals—which are typically considered sub-optimal, or density plots, which are have been found to be better [5], [7]. Based on the capital-labor-production theory [25], we expect incentives to impact the results, as participants who exert more effort would be more likely to spend time and more accurately estimate the probability values from the uncertainty representation which is necessary to perform the task. 1Specifically, in the presence of incentives, our hypothesis was that participants will perform meaningfully better when uncertainty information is presented using density plots compared to interval plots, while in the absence of incentives, we might expect this improvement in performance to be much smaller or even disappear.Specifically, in the presence of incentives, our hypothesis was that participants will perform meaningfully better when uncertainty information is presented using density plots compared to interval plots, while in the absence of incentives, we might expect this improvement in performance to be much smaller or even disappear.

2 Experiment Design Preliminaries↩︎

1We conducted preregistered experiments on the perception of correlation and decision-making tasks to study the effect of incentives. The study procedure for both studies was approved by the ethics board at Graz University of Technology (GZ EK-109/2026).We conducted preregistered experiments on the perception of correlation and decision-making tasks to study the effect of incentives. The study procedure for both studies was approved by the ethics board at Graz University of Technology (GZ EK-109/2026).

1For both studies, our goal was to ensure that participants’ average compensation would be approximately the same ($15/h) across both the baseline and incentivized conditions. We first calculated the average number of correct responses for both experiments based on our pilot studies. On average, participants correctly performed the task in study 1 in  40/65 trials. For study 2, prior work which used the same experimental setup [1] showed that the average participant had 4000 virtual dollars left at the end of the study. We then assigned dollar values for answering a question correctly in study 1 ($0.05) and for the amount of virtual dollars remaining in the bank in study 2 ($0.5 for every $1,000). Thus, we expected the average bonus that participants in the incentivized conditions would receive to be approximately $2 for both studies. The maximum bonus they could theoretically receive was $3.25 for experiments 1 and $9 for experiment 2. We anticipated both tasks to take approximately 9-12 minutes on average, which, at the desired compensation rate of $15/h, translates to a total compensation of $3. Thus, the compensation for the baseline (non-incentivised) conditions was set to $3 for both studies; the compensation for the incentivized conditions were $1.25 and $1.5 for Experiments 1 and 2 respectively. The slightly higher estimated average compensation in the incentivized conditions were set in order to comply with Prolific’s minimum wage of $8/h, which does not include bonuses.For both studies, our goal was to ensure that participants’ average compensation would be approximately the same ($15/h) across both the baseline and incentivized conditions. We first calculated the average number of correct responses for both experiments based on our pilot studies. On average, participants correctly performed the task in study 1 in  40/65 trials. For study 2, prior work which used the same experimental setup [1] showed that the average participant had 4000 virtual dollars left at the end of the study. We then assigned dollar values for answering a question correctly in study 1 ($0.05) and for the amount of virtual dollars remaining in the bank in study 2 ($0.5 for every $1,000). Thus, we expected the average bonus that participants in the incentivized conditions would receive to be approximately $2 for both studies. The maximum bonus they could theoretically receive was $3.25 for experiments 1 and $9 for experiment 2. We anticipated both tasks to take approximately 9-12 minutes on average, which, at the desired compensation rate of $15/h, translates to a total compensation of $3. Thus, the compensation for the baseline (non-incentivised) conditions was set to $3 for both studies; the compensation for the incentivized conditions were $1.25 and $1.5 for Experiments 1 and 2 respectively. The slightly higher estimated average compensation in the incentivized conditions were set in order to comply with Prolific’s minimum wage of $8/h, which does not include bonuses.

1While these amounts were the compensation advertised to participants before taking the study, as participants took significantly longer in the incentivized conditions, we adjusted the guaranteed compensation amount to both comply with Prolific’s standards and to match our desired target compensation of $15/h. The exact compensation amounts are reported separately for each study.While these amounts were the compensation advertised to participants before taking the study, as participants took significantly longer in the incentivized conditions, we adjusted the guaranteed compensation amount to both comply with Prolific’s standards and to match our desired target compensation of $15/h. The exact compensation amounts are reported separately for each study.

3 Experiment 1: Perception of Correlation↩︎

We conducted a preregistered study to partially replicate the perception of correlation study by Harrison et al. [10] using either scatterplots or parallel coordinates plot (see preregistration). As this task involves more lower-level cognitive functions [33], we test this as a possible edge-case of a visualization task where incentives will likely have little to no effect on performance.

3.1 Experimental Materials↩︎

Figure 1 shows a screenshot of the experimental interface. The study is available to browse here and the source code for the study is available on GitHub.

We adapted the perception of correlation task from Harrison et al. [10] for this experiment. Like Harrison et al. [10], in each trial, participants were shown two charts and were asked to select the one with the higher correlation in a Two-Alternative Forced Choice (2AFC) task. There were four experimental variables in this study: (1) the visualization (scatterplot or parallel coordinates plot); (2) the incentive scheme (advertised as a participation fee of $3 or a participation fee of $1.25 and a bonus of $0.05 for each correct response); (3) the base correlation (\(r_\mathrm{\small base}\)); and (4) the difference in correlation between the two charts (\(\Delta r\)). The visualization and incentive scheme were varied between-subjects whereas \(r_\mathrm{\small base}\) and \(\Delta r\) were varied within-subjects. To keep the experiment simple, we only considered positive correlations, and the approach from above—in other words the correlation of the second chart was always greater than the correlation of the first chart and is given by \(r_2 = r_\mathrm{\small base} + \Delta r\) (the order in which the charts appeared was randomized).

We test five levels of \(r_\mathrm{\small base} = \{0.3, 0.4, 0.5, 0.6, 0.7\}\) and thirteen levels of \(\Delta r = \{0.03, 0.04, \dots, 0.1, 0.12, 0.14, 0.18, \dots, 0.26\}\) in a fully crossed design resulting in 65 trials. In addition, we included five attention check questions where the correlations for the two charts are 0.01 and 0.99. The order of the trials were completely randomized. Unlike previous studies [10]–[12], we did not employ a staircase procedure due to the possibility of a confounding effect with the incentives—we felt that there might be a possibility of two participants ending up with the same payout in the incentivized condition even if they have different jnds due to the mechanisms of the staircase procedure. Unlike prior studies [10], [12] we provided immediate feedback to participants on whether they answered correctly, as seen in 1. At the end of the experiment, we informed participants about their total payout. After completing all trials, participants were asked to describe the strategy that they used in performing the task, and about their prior experience with incentivized studies.

Before participants began the study, we provided them with instructions on what correlations are with examples of what different levels of correlations look like, using the visualization that they will encounter subsequently. This is followed by participants completing nine training trials with feedback, using the same set of correlation stimuli that was used by Cutler et al [12].

We recruited all participants from Prolific. Our experiment was only eligible to participants who were fluent in English, and on desktop devices. We aimed to recruit 200 participants (50 participants in each condition). We excluded five participants who failed to meet our criteria, and recruited additional participants to meet our target. 1We received explicit consent from all participants to collect and share their responsesWe received explicit consent from all participants to collect and share their responses. The median completion time for participants in the baseline condition was approximately 11.5 mins, 1and they were compensated $3 (approximately $15.75/h)and they were compensated $3 (approximately $15.75/h); the median completion time for participants in the incentivized condition was approximately 13.5 mins, and they were received a guaranteed amount of $1.6 (adjusted up from $1.25) and the average bonus was $2.18 (corresponding to an average wage of $16.8/h).

3.2 Model specification↩︎

We adapted the approach used by Kale et al. [5], and used a Binomial logistic regression model to estimate the probability of a correct response. The logistic fits ( line 2) describing each participants’ responses are equivalent to psychometric functions [42], which can then be used to derive an estimate of jnd (which we discuss further below).

Whether a participant correctly chose the graph with the higher correlation in each trial is modeled as a Binomial distribution with probability \(p\), where 1 means the participant made the correct choice.

For each stimuli (which consists of two graphs), we assume that participants’ decisions would depend on the visualization, the incentive condition, the base correlation (\(r_\mathrm{\small base}\)), and the differential increase in stimuli i.e. correlation (\(\Delta r\)). This model incorporates the (directionless) assumption that the probability of a correct response (\(p\)) will change as \(r_\mathrm{\small base}\) changes, and the magnitude of this change in \(p\) will also vary with \(\Delta r\). Based on prior work, both of these effects are likely to be in the positive direction.

We expect the intercept (\(\alpha_{i}\)) parameter to vary between participants, as different participants will likely have different capital and perceptive abilities. \(\alpha\) is the slope parameters for the average participant (when \(\delta_{\alpha, i} = 0\)); \(\delta_{\alpha, i}\) captures differences between participants as random effects.

We use weakly-regularizing priors centered on zero (no effect) while permitting the possibility of significant distortion: \(\alpha \sim \mathrm{Normal}(0, 1)\). We use zero-centered priors with a standard deviation of 2 for the random effects parameters (\(\delta_{\alpha, i}\)).

We implemented these models in R 4.4.0 [43] and CmdStanR 0.8.0 [44]. The model ran for four chains with 5,000 warmup samples and 5,000 post-warmup samples each, thinned by 4 for a final total sample size of 5,000. We assessed convergence using the Gelman-Rubin diagnostic (\(\hat{R}\) = 1.00 for all population-level parameters, correlations and standard deviations) and the (bulk and tail) effective sample sizes (\(\text{ESS}_{min} \approx 2,000\)).

Based on the estimates of this model, we can calculate the JND at a specific level of \(r_\mathrm{\small base}\)for a particular visualization and incentive condition as:

3.3 Results↩︎

The results for Experiment 1 are shown in 4 as estimates of jnd (just noticeable differences). In the baseline (non-incentivized) condition, we find that the jnd for scatterplots is much lower at all levels of the baseline correlation, replicating the finding from prior work [10], [32].

As expected, we also do not find any evidence to suggest that incentives impact performance in the perception of correlation task—the jnds for participants in the incentivized conditions were comparable, if not slightly worse, than the jnds for participants in the baseline condition (4A). We also do not find any evidence to suggest that incentivizing participants led to reduced variance in their responses, as measured through the group-level standard deviation parameters in our multi-level logistic regression model (see supplement \(\blacktriangleright\) R \(\blacktriangleright\) dm-05-effect_variance.qmd). However, we did find that participants took slightly longer to complete the task in the incentivized conditions compared to the baseline conditions. 5A, visualizes the bootstrapped median and 95% quantile intervals for the median estimate. The difference in time spent is approximately 30s for parallel coordinates (, ), and approximately 17s for scatterplot (, ). The increase, which is approximately 6-10%, should likely be considered small.

Finally, as an aside, we want to highlight the ability of our experimental setup to replicate jnd estimates from prior work without using a staircase procedure. An additional advantage of this setup is that it allows us to model participants’ decisions directly using the Binomial logistic regression model described above.

4 Experiment 2: Decision-making under Uncertainty↩︎

In this preregistered study, we examined the impact of incentives on performance in a decision-making under uncertainty task adapted from prior work [15] using either interval or density plots as uncertainty representations (see preregistration). As performing this task involves relatively more complex cognitive functions, our hypothesis was that incentives would have an effect on performance.

4.1 Experimental Materials↩︎

6 shows a screenshot of the experimental interface. The study is available to browse here and the source code for the study is available on GitHub.

For this study, we adapted the scenario that was used by Joslyn and LeClerc [15], where participants were presented with the following hypothetical scenario:

Assume that you are an analyst of a road maintenance company contracted to treat the roads with salt brine to prevent icing in a U.S. town impacted by severe cold. Applying salt brine to the roads is costly for your company and is also detrimental to the environment, as it can pollute groundwater and kill roadside vegetation. However, not salting the roads can cause significant accidents during freezing temperatures, the costs of which are borne by your company. Your job is to salt the roads when temperatures drop below 0°C (32°F). You have a budget of $18,000 for 18 days. Salting all the roads in the town costs $1,000 (per night). If you fail to salt the roads and the temperature drops below 0°C (32°F), it will cost $5,000 from your budget. You will be shown a night-time temperature forecast distribution based on which, you will have to decide whether to salt the roads.

The payoff matrix for the decision problem described above can be represented using the following table:

Based on this incentive scheme, a decision-maker who wants to maximize expected utility should prefer the action \(a_1\) (to salt) if and only if the probability of temperature being below freezing, \(\mathrm{Pr(T \leq 0)} = p \geq 0.2\). Thus, \(p = 0.2\) is the optimal crossover point: the probability at which a decision maker should not have a preference between actions.

There were three experimental variables in this study: (1) the visualization (interval or density plot); (2) the incentive scheme (advertised as either a participation fee of $3 or a participation fee of $1.5 and a bonus of $0.5 for each $1,000 remaining in their account at the end of the experiment); (3) the actual forecasted probability of freezing (9 levels, each repeated twice). The experiment used a mixed-factorial design with visualization and incentive scheme varied between-subjects and the forecast varied within subjects.

The experiment consisted of 18 trials and two attention check questions which are interspersed among the 18 trials. In each trial participants are presented with a single forecast in the form of a Normal distribution (see 6). We varied the means and standard deviations of the Normal distribution to generate different forecasts with different probabilities of freezing: \(\mathrm{Pr}(T\leq0\text{\textdegree C}) =\) 1{0.595, 0.5, 0.405, 0.315, 0.235, 0.168, 0.115, 0.075, 0.046}. The attention check questions showed extreme forecasts (\(Normal(-11, 0.4)\) and \(Normal(11, 0.4)\)). The order of the trials were completely randomized. After completing all the trials, participants were asked open-ended questions similar to experiment 1.

We provided instructions to participants on how to correctly interpret the interval and density plots. This is followed by participants completing four training trials where participants were shown four different temperature forecasts and were asked to report the probability of the temperature falling below a certain marked temperature. Participants were then provided feedback on whether they answered the questions correctly.

We recruited all participants from Prolific. Our experiment was only eligible to participants who were fluent in English, and on desktop devices. As per our pre-registrations, we aimed to recruit 280 participants (70 participants in each condition). We experienced some data-logging issues which meant the data for 31 participants were not recorded. After excluding participants who failed to meet our pre-registered attention check criteria (six), we had 243 participants (60 in the , 62 in 59 in and 62 in . 1We received explicit consent from all participants to collect and share their responsesWe received explicit consent from all participants to collect and share their responses. The median completion time for participants in the baseline condition was approximately 16 mins, 1and they were compensated $3.75 (approximately $14/h)and they were compensated $3.75 (approximately $14/h); the median completion time for participants in the incentivized condition was approximately 24 mins, and they received a guaranteed amount of $4.2 (adjusted up from $1.5) and the average bonus was $1.6 (corresponding to an average wage of $14.5/h).

4.2 Model Specification↩︎

We used a linear-in-log-odds (llo) model [45], which is a good fit to account for the “distortion of judgement or misperception of probabilities” [8] in such decision-making under uncertainty tasks. This model, which has been used in prior work to model the same task [1], is derived directly from the utility-optimal decision criterion by first translating it into log-odds and then applying a linear transformation to derive a model for participants’ decisions:

The binary decision (where 1 represents the decision to salt and 0 represents the decision to not salt) made by a participant in each trial is modeled as a Binomial distribution with probability \(p_\mathrm{\small salt}\).

We assumed that participants’ decisions would be a function of the temperature forecast distribution shown, which can be represented using the corresponding probability of freezing value (\(p\)). The intercept (\(\alpha\)) parameter controls the fixed point of the function—how people map the optimal crossover point (\(p = 0.2\)) to the probability of salting—which shifts the crossover point. The further \(\alpha\) is from 0, the more bias there is, with negative values of \(\alpha\) suggesting that crossover point for the average participant is greater than the optimal, and positive values of \(\alpha\) suggesting that crossover point for the average participant is less than the optimal. The slope (\(\beta\)) parameter controls the degree of sensitivity. The further it is from 1, the more sensitive a participants is to their subjective crossover points.

We expect the intercept (\(\alpha_{i}\)) and the slope (\(\beta_{i}\)) parameters to vary between participants, as different participants will likely have different decision-making capacities. \(\alpha\) and \(\beta\) are the slope parameters for the average participant (\(\delta_{\alpha, i} = 0\); \(\delta_{\beta, i} = 0\)), whereas \(\delta_{\alpha, i}\) and \(\delta_{\beta, i}\) capture differences between each participants’ intercepts and slopes compared to the average participant, as random effects.

Perfectly unbiased responses would yield values of \(\alpha_{i} = 0\). \(\beta_{i} = 1\). We thus use priors centered on these values but which also permit the possibility of significant distortion: \(\alpha \sim \mathrm{Normal}(0, 1)\) and \(\beta \sim \mathrm{Normal}(1, 1)\). We use zero-centered priors for the random effects parameters (\(\delta_{\alpha, i}\) and \(\delta_{\beta, i}\)).

These steps were almost identical to study 1.

4.3 Results↩︎

The results for Experiment 2 are shown in 8. We found the value of \(\alpha\) to be negative across all conditions (8A). This suggests that the crossover point for the average participant is greater than 0.2, exhibiting a consistent bias towards risk-seeking behavior. The estimated value of \(\alpha\) was smaller for compared to the other three conditions. We also found the value of \(\beta\) to be significantly larger than one across all conditions (8B), indicating that participants are quite sensitive to the stimuli and their subjective crossover point. The similarity in the estimates of the \(\alpha\) and \(\beta\) parameters across all conditions suggest that the average participant in every condition likely exhibited very similar decision-making behavior.

We can directly compare decision quality by estimating the expected utility that the average participant in each of these conditions would achieve, which is a measure of how participants performed based on the information that was provided to them regarding how their performance will be evaluated; we report this measure in 8C. Contrary to our apriori expectations, we find no evidence to suggest that the average participant performs better in the incentivized conditions in terms of their predicted expected utility ( and ) compared to the baseline conditions ( and ).

We also found that the average participant in the interval condition performed slightly better than the average participant in the density condition, regardless of whether they are incentivized or not. This difference is relatively small—the difference in expected utility is 0.66 95% CI: [0.25, 1.1] when incentivised and 0.44 95%CI: [0, 0.87] when not incentivized (see 8C). 9 shows the distribution of utility attained by participants in the various conditions, as well as the posterior predictive intervals (see also 8 for more details). The actual average utility of participants in the interval condition under incentives was much larger than the other conditions; we suspect that this might have been an artifact of the rng in JavaScript—we implemented a series of checks but could not determine any systemic issues with the rng (supplement \(\blacktriangleright\) R \(\blacktriangleright\) dm-05-rng_check.qmd). Our finding—that intervals perform better than densities—is in contrast to prior work [5], [7], which have typically found that participants make better decisions when information is presented to them using density plots compared to interval plots.1 In our study, the interval plot contains more information—we show both the 66% and 95% intervals—and hence may not be a close replication of prior work. We discuss the implications of this finding in §6.2.

As in the first study, we found that participants took slightly longer to complete the task in the incentivized conditions compared to the baseline conditions. 5B, visualizes the bootstrapped median and 95% quantile intervals for the median estimate. 1The difference in time spent is approximately 80s for intervals (, ), and approximately 32s for density (, )The difference in time spent is approximately 80s for intervals (, ), and approximately 32s for density (, ). This increase of 20-40% is quite large, and might suggest that in tasks such as decision-making under uncertainty the use of incentives may lead to more time spent on the task, and perhaps more effort induced (depending on how one views the relationship between time spent on an online experiment and effort).

5 Crowdworkers’ Perspectives on Incentives↩︎

For both studies, we elicited qualitative responses from participants regarding (i) their prior experiences with performance-based incentives, and (ii) how they might have behaved if they were in the other condition (i.e., asked people in the incentivized condition how they would behave if they were given a flat amount and vice-versa). We conducted an exploratory qualitative analysis of participants’ responses using an inductive coding approach to identify a set of codes, which was then used to code all the responses. Below, we report the numbers combined across both studies (N = 444).

1Of the responses which we were able to code (224),Of the responses which we were able to code (224), significantly more participants (115) in our sample claimed to have positive attitudes towards incentives 1based on their past experiencebased on their past experience (e.g., “[...] they can make the task more engaging and encourage more careful decision-making” or “Surprised and happy. Its like a little gift you receive because you’re doing a good job”), compared to those who explicitly reported mixed or negative feelings (28) towards incentives (e.g., “Very mixed because most times there is little actual chance of receiving any bonus” or “They are often misleading, or sometimes held as a raffle, which is a scammy practice”). Among those who reported positive feelings, some say that it encourages them to do well (34) or makes the task more interesting and engaging (19). On the other hand, those who claim to have negative experience usually believe that it is because they are unfair or conflate bonuses with lotteries or raffles.

For our analysis on how participants self-reported they would behave in the other incentivized condition, we focused only those who were in the incentivized condition as it would be easier for them to imagine the counterfactual. 1Of the responses we were able to code (192/224)Of the responses we were able to code (192/224), a majority (119) claimed that would have behaved in the same way, there were several participants (35) who believed that the in the absence of incentives they might perform worse because they might be less motivated or focused (e.g., “If I had been guaranteed [...], my performance probably would have been slightly worse because the bonus made me pay closer attention to the tradeoff between salting costs and freezing risk.”). In the decision-making study, several (12) reported that they would be more risk seeking (e.g., “[...] would have been slightly less cautious and less incentive-driven.”). For these participants, incentives appear to be having the intended effect.

However, there were also a number of participants (13) who believed that they would have performed better in the non-incentivized condition perhaps because it would be less stressful (e.g., “[...] guaranteed payment offers greater financial security and could reduce performance anxiety.” and “I think it would be better, as [...] one would treat the study properly because he would get a proper compensation in return”). This might suggest that, because of how incentivized experiments are designed with a lower base pay, for a number of participants, incentives might be having the opposite of the intended effect. In the decision-making study, several participants (16) stated that they might have been less risk averse as they would not be concerned with saving money (e.g., “With a guaranteed reward, the incentive to save money would vanish. My performance would likely have become highly risk-averse”), again suggesting that incentives may not be having the intended effect for many participants.

6 Discussion↩︎

6.1 What is the Effect of Incentives?↩︎

Our work was motivated by several recent studies which have employed incentivized experiments to study decision-making under uncertainty [5]–[7], [9]. The rationale was that decision-making in the real-world entails consequences and incentives would bring that notion of realism to virtual survey world. Based on the capital-labor-production theory, our a priori hypotheses were that: (i) the effect of incentives would likely be small, if any, in the perception of correlation as the effort required (or production requirement) is low; and (ii) performance-based incentives would likely lead to a meaningful improvement in performance (in terms of effect size) in the decision-making task as more effort is required to properly perform the task (i.e., higher production requirements). The results of our two experiments (see f@4  4 @@ and 8, 8 @) suggests that, compared to a baseline of providing people a flat rate for participation, introducing performance-based financial incentives may not lead to improved performance in either the perception of correlation task or the decision-making task.

While expected for the perception of correlation task, this unexpected violation of our apriori expectations could potentially be explained by the capital-labor-production theory—it is possible that the the decision-making task may have been too difficult. We observed that participants in the incentivized conditions did spend longer in completing the task (see 5B), and this difference in time spent was particularly large for the decision-making task. If time spent on a task is considered a reasonable proxy for effort exerted by a participant, this result suggests that the average participant in the incentivized condition is likely putting in more effort then the average participant in the non-incentivized condition, even though they are not necessarily performing better on the task. We discuss potential implications for studying decision quality in §6.3.

Prior work [25] also suggests that incentives often reduce variance in responses, even if they do not have an impact on mean performance. Our model allows us to estimate this variance, and while the variance parameters estimated by our model, for both studies, are smaller for the incentivized condition compared to non-incentivized conditions, these differences appear to be very small (see the “Effect on Variance” subsection in supplement \(\blacktriangleright\) R \(\blacktriangleright\) dm-04-analysis.qmd).

6.2 Why Were Our Results Inconsistent With Prior Uncertainty Visualization Studies?↩︎

Perhaps the most surprising aspect of our experimental results was that the average participant making decisions using interval plots was expected to perform marginally better than with density plots. This is contrast to prior work [5]–[7] which has typically found that, in incentivized experiments, participants make better decisions when the uncertainty distribution is represented using densities rather than 95% interval plots, as densities provide complete distributional information. We expected the same result to hold for our study, even though we varied our design of interval plots. 1We initially suspected that it was becauseWe initially suspected that it was because, unlike previous experiments, we presented participants with both the 66% and 95% intervals. The two intervals provide additional information relative to prior studies, and supports a visual heuristic to make utility optimal decisions—if the 66% interval is slightly overlapping with the 0°C line, this indicates an approximately 20% probability that the temperature is going to be below freezing 10. In a “just 95% interval” condition, as in prior work, there is no such easy visual heuristic that a participant can rely on. In contrast, a participant in the density condition would have to compare areas to make the utility optimal decision, which might arguably have been more difficult, given how poor people are at area perception [28]. 1However, a follow-up study (see 9) comparing 95% intervals with density plots again showed no difference in participants’ decision quality between the two conditions. One possible cause could be the probability distributions used as stimuli in our study—the distributions were of varying standard deviations, which can interfere with the visual heuristics that participants use to estimate probabilities from densities [46], [47]. Our results paint a conflicting picture regarding the effectiveness of density plots for uncertainty communication, and warrants further research. However, a follow-up study (see 9) comparing 95% intervals with density plots again showed no difference in participants’ decision quality between the two conditions. One possible cause could be the probability distributions used as stimuli in our study—the distributions were of varying standard deviations, which can interfere with the visual heuristics that participants use to estimate probabilities from densities [46], [47]. Our results paint a conflicting picture regarding the effectiveness of density plots for uncertainty communication, and warrants further research.

1 At the same time, we also want to be wary of trying too hard to retrofit possible explanations. We are not aware of any other decision-making studies which evaluated multiple intervals. Hence, without replication studies, this should be viewed as one comparative study of the effectiveness of density plots and intervals, among several others, and urge readers not to draw any strong conclusions.

6.3 Ecological Validity of Decision-Making Studies↩︎

Given the results of our study, and how they were contrary to our initial hypotheses, we did consider shelving this paper. However, we realized that this would be a terrible idea for two reasons: (i) we would be contributing to the file drawer problem [48], [49] (self-selecting to not publish studies with negative results), and (ii) even though our study does not find evidence for the effect of incentives, it does raise some interesting questions about the use of incentivized experiments for studying decision quality and the ecological validity of studies run on crowd-sourcing platforms such as Prolific.

1 As this is just one study, it would be premature to conclude that incentives do not have any effect. An obvious first step would be to gather more evidence on whether incentives are useful—not just direct replications of this work, but also for other types of decision-making tasks that have been investigated in prior work in visualization 1  [5], [8], 1 and exploring other uncertainty representations. If we continue to find that incentives do not have any effect, we should rethink how we currently design decision-making experiments.

Early work in uncertainty visualization compared different representations based on subjective measures such as preference or intuitiveness (e.g., which representation do you prefer?) [50]–[52], ease of use [53], and subjective probability (e.g., on a scale of 1-7, how likely do you think the event is going to occur?) or confidence (e.g., on a scale of 1-7, how confident are you about your prediction?) [34], [36], [52], [54]. A critique of these types of studies is the lack of an objective benchmark: a user might perceive visualization A as being more intuitive than visualization B, or they might experience a greater or lesser degree of subjective uncertainty when viewing it. But how do we know that they are perceiving the correct level of uncertainty? Is perceiving less uncertainty or more uncertainty better?

Other studies have investigated participants’ ability to accurately estimate the probability of an event occuring using uncertainty visualizations (which sometimes reduces to visually estimating tail probabilities) [34], [35], [55]. These studies include an objective benchmark—accuracy of probability estimation—which could be used to evaluate participants’ performance. However, even if people are able to read and report numerical probabilities accurately, it is possible that their actual behavior diverges from what would be expected based on the reported probabilities, either due to bias or error, or because they associate different, subjective probabilities to uncertainty [56]. As such, measuring uncertainty through accuracy in reading numerical probability values may not necessarily tell us how good people are at using those probabilities to make good decisions.

Since uncertainty visualizations are typically used in the real world to make decisions and a person in the real-world is incentivized to make a correct decision because it has real consequences (e.g., we consult the weather forecast not because we care about knowing the probability of rain, but because we need to decide whether we should carry an umbrella or not), studying decision quality using uncertainty visualizations represented, arguably, a more ecological valid approach of evaluating different uncertainty communication approaches. While other metrics for measuring the effect of uncertainty visualizations could tell us that two visualizations are different, they do not necessarily tell us which one is better. Asking people to perform incentivized decision tasks, in theory, allowed us to measure decision quality against an objective baseline—the expected utility of a rational decision-maker—by comparing how far are people from that baseline.

Yet there are a couple of assumptions implicit in this line of reasoning: that all participants in incentivized decision-making tasks are attempting to maximize expected utility; and (assuming they are trying to maximize expected utility) people are able to translate the provided incentive structure or cost function into a decision rule based on probabilities (which is how the uncertainty information is typically presented to them). If these two assumptions hold, incentivized decision-making style experiments can be considered more ecologically valid than other approaches for evaluating uncertainty visualizations. Under these conditions, a decision-making study of uncertainty visualizations measures how people make decisions based on their subjective representation of uncertainty or subjective risk tolerance threshold, and an uncertainty visualization can be considered more effective if they allow a user to make decisions which are close to utility optimal. However, the qualitative analysis of participants responses to the open-ended questions in our study, and some introspective reflection, make us question these assumptions.

In their qualitative responses, a few participants stated to have adopted a risk-seeking strategy, as indicated by statements such as “gambling with [a] small amount is fun and there is no real concern of losing money” or “there is less personal ‘skin in the game,’ so I might have been more willing to gamble on borderline forecasts.” We also observed something similar in prior work [6], where some participants stated that they started taking risks after their initial decision-making strategy did not appear to have been paying off. While a few errant participants are to be expected, and can be easily accounted for in a mixed-effects regression model like the one we used, it is unclear how prevalent these attitudes are. In addition, our qualitative analysis suggests that (§5), due to the competing goals of crowdworkers on Prolific, incentives can have the opposite of the desired effect—as the guaranteed payment was low, instead of trying to maximize expected utility, participants simply behaved as if they were being paid the small base pay. If a significant proportion of participants are not trying to maximize expected utility and instead adopting a risk-seeking attitude when performing decision-making tasks, we might need to evaluate how incentive structures in experiments are designed so that they elicit the behavior we normatively want.

Even if participants were trying to maximize expected utility, it is possible that they were not able to translate this to a decision rule with the optimal crossover point (\(p = 0.2\)). Of the participants for whom we could deduce a crossover point (53), 25 participants stated that they decided to salt the roads when the chances of freezing were around 30% or higher 1(the crossover point for the remainder were at or around 20%)(the crossover point for the remainder were at or around 20%), suggesting an imperfect translation from the given cost function to a decision rule. This is reflected in the model estimates of \(\alpha\) (8A), and is consistent with findings from prior work on a similar task [1]. There is a qualitative difference between risk-seeking behavior when a participant decides to salt when \(p = 0.3\) knowing that the optimal crossover point is \(p = 0.2\), and “risk-seeking behavior” when a participant decides to salt when \(p = 0.3\) believing that the optimal crossover point is \(p = 0.3\). When evaluating decision quality, we want to make sure we are measuring the former and not the latter.

A different (mis)calculation can be observed among participants who stated that they would be more risk averse in the baseline conditions as they would not be incentivized to “save money.” If some participants did not know how to maximize expected utility, it might also mean that they are not sensitive to the incentive structure that is provided to them (i.e., could participants behave similarly if the penalty for not salting was $3,000 instead of $5,000?). From the current study, it is unclear whether some participants’ inability to identify a decision rule is simply an artifact of this specific task, the format of presenting incentive structures, difficulties in interpreting single event probabilities properly [57], not being sufficiently sensitive to incentives or something else entirely, and therefore warrants further research.

1In Experiment 2, we provided feedback to participants after every trial regarding what the simulated temperature was, and the costs incurred. It is possible that this immediate feedback could have led participants to adopt a sub-optimal strategy—according to Achtziger et al. [27], “the human tendency to repeat successful actions and avoid those which led to failure can impair performance by focusing attention on win/lose outcomes and away from the probabilities of the relevant uncertain events.” While the setup of the decision-making task used in our study does not allow feedback to be withheld, it is worth exploring whether participants are more rational decision-makers under uncertainty for tasks where immediate feedback is not provided. In Experiment 2, we provided feedback to participants after every trial regarding what the simulated temperature was, and the costs incurred. It is possible that this immediate feedback could have led participants to adopt a sub-optimal strategy—according to Achtziger et al. [27], “the human tendency to repeat successful actions and avoid those which led to failure can impair performance by focusing attention on win/lose outcomes and away from the probabilities of the relevant uncertain events.” While the setup of the decision-making task used in our study does not allow feedback to be withheld, it is worth exploring whether participants are more rational decision-makers under uncertainty for tasks where immediate feedback is not provided.

The use of incentives assumes that participants who are given a flat payment regardless of how they perform will put in less effort, and overlooks the fact that many participants in the non-incentivised conditions might have strong intrinsic motivation to provide high-quality data and perform well in the study [30]. According to Camerer and Hogarth [25], “some people like mental effort, and those people disproportionately volunteer for experiments!” If this attitude is widespread among crowdworkers on platforms such as Prolific, then the additional effort for implementing incentives might not be worthwhile for most researchers. An additional aspect to consider here are the policies of Prolific as a platform—the quality checks implemented by Prolific could serve as another source of motivation for crowdworkers to provide high quality responses in experiments, reducing the potential effect of performance-based incentives.

Finally, participants in our non-incentivized conditions were still informed on whether they answered a question correctly and how many they answered correctly so far (perception of correlation), or how much budget they have remaining (decision-making). In other words, while participants were not financially rewarded for their performance, they could interpret these as virtual rewards. This was a conscious choice in our experiment in order to minimize the discrepancies between the two experimental conditions. However, it is possible that, in crowdsourcing platforms such as Prolific, if participants are already highly intrinsically motivated to do well, virtual rewards could have a similar effect on effort as real rewards.

6.4 The Possible Role of Incentives in Other Studies↩︎

1Our work was an initial investigation into the role of incentives in two prototypical types of visualization studies. While our results may not generalize to all visualization tasks, we can speculate on how incentives might impact certain other types of visualization tasks, based on our findings and the capital-labor-production theory [25]. Our work was an initial investigation into the role of incentives in two prototypical types of visualization studies. While our results may not generalize to all visualization tasks, we can speculate on how incentives might impact certain other types of visualization tasks, based on our findings and the capital-labor-production theory [25]. Studies on graphical perception [58]–[60] or color perception [61]–[64], which require users to perform similarly low-level perceptual tasks as Study 1 are likely not going to benefit from the increased effort than incentives induce.

There are likely a class of visualization tasks such as some matrix-based tasks (e.g., Nobre et al. [65], performed a crowdsourced evaluation of node-link diagrams and adjacency matrices where participants were required to undergo lots of training and perform complex tasks), which either require a lot of a priori knowledge (cognitive capital) or a lot of effort (high production requirements), and thus may not benefit from incentives (and decision-making under uncertainty could possibly fall into this category as well!). In other words, these represent tasks where participants may perform poorly not because they do not want to put in more effort, but rather because they are not sure of how to perform the task well and putting in more effort may not necessarily help them perform the task better.

The type of tasks that we believe may benefit from incentives are ones which do not require a lot of cognitive capital but can feel a bit tedious; incentives might motivate participants to put in the required additional effort to complete the task accurately. An example could be the finding the shortest path between two nodes task using node-link or adjacency matrix representations—node-link diagrams have been found to be more accurate [66]–[68], but if incentivized, participants may be more willing to put in the effort to perform the tedious steps involved for adjacency matrices. A challenge here is ensuring that the incentive function is well designed and not arbitrary [40], [41]. We hope to explore such tasks in future work.

6.5 Ethical Considerations for Incentivized Experiments↩︎

1In both experiments, participants in the incentivized conditions took longer to complete the task compared to participants in the baseline condition. Therefore, (i) the initial median hourly wage for participants was lower than the advertised amount, and, after factoring in bonus, payment was below our target fair wage of $15/h; and (ii) the use of performance-based financial incentives increased the variance in wages that participants received—a participant who can perform the task quickly and well can receive a significantly higher wage than a participant who performs it slowly and poorly. In both experiments, participants in the incentivized conditions took longer to complete the task compared to participants in the baseline condition. Therefore, (i) the initial median hourly wage for participants was lower than the advertised amount, and, after factoring in bonus, payment was below our target fair wage of $15/h; and (ii) the use of performance-based financial incentives increased the variance in wages that participants received—a participant who can perform the task quickly and well can receive a significantly higher wage than a participant who performs it slowly and poorly.

1Ensuring a fair median wage for participants is still possible for incentivized experiments, as the researcher can increase the guaranteed amount so that the median wage, taking into account bonuses, reflects what is considered a fair wage. In order to ensure a wage of $15/h, and to comply with Prolific’s minimum guaranteed wage of $8/h, we increased the fixed participation fee (by $0.35 in Experiment 1, and by $2.7 in Experiment 2). Addressing the issue of variance in wages, however, is significantly more complicated, and to some degree not compatible with the goal of performance-based incentives. While there is always going to be some degree of variance in compensation in crowdsourced studies, as not every participant takes the same amount of time, using bonuses as a significant portion of the compensation exacerbates this issue. A solution here might be to increase the guaranteed amount to ensure that participants receive a fair hourly wage. However, as research teams typically operate under considerable budget constraints, this would likely mean that they would have to in turn reduce the bonus amounts that participants can earn, making them less “incentivizing.” Ensuring a fair median wage for participants is still possible for incentivized experiments, as the researcher can increase the guaranteed amount so that the median wage, taking into account bonuses, reflects what is considered a fair wage. In order to ensure a wage of $15/h, and to comply with Prolific’s minimum guaranteed wage of $8/h, we increased the fixed participation fee (by $0.35 in Experiment 1, and by $2.7 in Experiment 2). Addressing the issue of variance in wages, however, is significantly more complicated, and to some degree not compatible with the goal of performance-based incentives. While there is always going to be some degree of variance in compensation in crowdsourced studies, as not every participant takes the same amount of time, using bonuses as a significant portion of the compensation exacerbates this issue. A solution here might be to increase the guaranteed amount to ensure that participants receive a fair hourly wage. However, as research teams typically operate under considerable budget constraints, this would likely mean that they would have to in turn reduce the bonus amounts that participants can earn, making them less “incentivizing.”

1Running incentivized studies has real costs on the experimenter—increased implementation and design burden while setting up the experiment, the need to manually compensate participants on the crowdsourcing platform, the seemingly low advertised compensation on the crowdsourcing platform (which can have reputational costs for future studies). Taking all of this into account, there is a credible argument to be made against the use of performance-based financial incentives due to the high costs, seemingly low rewards, and the associated ethical questions regarding fair labour practices. Running incentivized studies has real costs on the experimenter—increased implementation and design burden while setting up the experiment, the need to manually compensate participants on the crowdsourcing platform, the seemingly low advertised compensation on the crowdsourcing platform (which can have reputational costs for future studies). Taking all of this into account, there is a credible argument to be made against the use of performance-based financial incentives due to the high costs, seemingly low rewards, and the associated ethical questions regarding fair labour practices.

7 Conclusion↩︎

We conducted two studies to test the value of performance-based monetary incentives for two different types analysis questions that commonly surface in visualization studies: a lower-level perceptual task, and a higher-level decision-making task. We did not find evidence to suggest that incentives affect performance on either task. To our knowledge, our work was the first formal study of monetary incentives in crowdsourced visualization studies. While we acknowledge that our results may not easily translate to other scenarios (e.g., lab studies, different types of tasks, significantly higher incentive pay, etc.), they seem to suggest that incentives, as currently deployed in many empirical studies, may matter less as a tacit factor than we expected. Given recent calls for incorporating decision theory, utility functions and incentives into visualization and hci experiments [40], [41], our results provide an empirical data point that putting them into practice may not be as straightforward as we might have initially presumed.

1 Running incentivized studies has real costs on the experimenter: it increases the implementation and design burden before the experiment, and participants need to be manually compensated on the crowdsourcing platform after the experiment. Other practical issues include a seemingly low average compensation on a crowdsourcing platform (that does not account for incentive payments), which can be a reputational issue for future studies.

1 Financial incentives are only one of the tacit factors that may influence experiments. We chose incentives as a first step because we expected it to be a fruitful aspect to investigate that can be easily studied. Another prominent tacit factor worth studying in the future is the impact of training, but training is multi-faceted: it could be just a textual introduction, an instructional video, or an interactive training session, complicating potential experimental designs and increasing the cost and difficulty of studying such factors. Nevertheless, we believe that clarifying and providing guidance to authors on such tacit factors of experiment design has the potential to lead to increase the quality of studies the visualization community conducts.

Acknowledgments↩︎

We would like to thank to Jack Wilburn and Zach Cutler for technical support on this project, and the anonymous reviewers for their feedback on the manuscript. This work was partially funded by the National Science Foundation awards 2213756 and 2403094.

8 Interpreting the Results of Experiment 2↩︎

As described in 4, there are two payoff relevant states (of nature), \(S = \{s_1 = Pr(T \leq 0), s_2 = Pr(T > 0)\}\) and two possible actions \(A = \{a_1, a_2\}\). Following prior studies in decision-making under uncertainty in visualization [5], [7], [9], to evaluate participant decisions for each trial, we simulate a draw from the visualized temperature distribution which determines the state of the world \(S\) (i.e., whether the temperature is below or above freezing). We then calculate the cost incurred based on the payoff matrix described in 4. To assess participants performance, we use three benchmarks—the expected utility of decision-maker responding at random, the expected utility of decision-maker with extreme risk-aversion (deciding to salt on every trial except the attention check ones), and the expected utility of completely rationally decision-maker.

\[\begin{align} & \mathbb{E}(U | \text{random}) \hskip1.25em = -1000 \cdot \left( \sum_k^n 0.5 \cdot 1 + 0.5 \cdot 5 \cdot p_k \right) \\ &\hskip6.8em = -3272 \\ & \mathbb{E}(U | \text{risk-averse}) = \sum_i^n -1000 \\ &\hskip6.9em = 0 \\ & \mathbb{E}(U | \text{rational}) \hskip1.29em = -1000 \cdot \left( \sum_k^n 1 \cdot \mathbb{I}(p_i \geq 0.2) + 5 \cdot \mathbb{I}(p_i < 0.2) \cdot p_k \right) \\ &\hskip6.8em = 3960 \\ \end{align}\]

where \(p_k\) represents the probability of freezing in a particular trial \(k\) and lies in \(\{0.595, 0.5, 0.405, 0.315, 0.235, 0.168, 0.115, 0.075, 0.046\}\) each repeated twice, and \(n = 18\).

In 9, we show the posterior predictive credible interval for expected utility in each condition based on the model estimated probability of salting for the average participant on a specific trial \(k\): \(p_{\mathrm{\small salt}} = \text{logit}^{-1}(\alpha_i + \beta_i \cdot [ \text{logit}(p_k) - \text{logit}(0.2) ])\). The expected utility is given by:

\[\mathbb{E}(U) = -1000 \cdot \left( \sum_k^n \mathbb{E}(p_{\mathrm{\small salt}}) + 5 \cdot (1 - \mathbb{E}(p_{\mathrm{\small salt}})) \cdot p_k \right) \\None\]

We sample draws from the expectation of the posterior predictive distribution using the add_epred_draws function from tidybayes 3.0.7 [69].

In 9, we find that approximately 30% of the participants performed better than the rational benchmark. While on the face of it, this might seem surprising, it is actually not that unlikely—the rational benchmark is an asymptotic guarantee given an infinite number of trials; for obvious purposes, we limit participants to 18 trials. To perform better than the rational benchmark (\(\mathbb{E}(U) > \mathbb{E}(U | \text{rational})\)), a participant simply has to get lucky and not encounter a freezing state of nature (i.e., \(s_1\)) on any of the trials where they decide to not salt. This is given by:

\[\prod_k^n (1 - p_i) \cdot \mathbb{I}(p_i \leq d)\]

where \(d\) is the decision boundary and \(d = 0.2\) for the rational decision-maker. The probability of outperforming the rational benchmark for a participant making decisions using \(d = 0.2\) is 0.42. If \(d = 0.3\), the probability of outperforming the rational benchmark is 0.25.

9 Experiment 3: Decision-Making under Uncertainty↩︎

We conducted a replication of Experiment 2, where we examined the impact of incentives on performance in the same decision-making task using either 95% interval (as opposed to 66% and 95% interval) or density plots as uncertainty representations. The rest of the experimental materials (task, procedure, tutorials and training) and data analysis model were the same as Experiment 2. As this experiment was conducted after the initial review cycle, we only include it as an appendix.

We recruited all participants from Prolific. Our experiment was only eligible to participants who were fluent in English, and on desktop devices. As per our pre-registrations, we aimed to recruit 240 participants (60 participants in each condition). After excluding participants who failed to meet our pre-registered attention check criteria (eleven), we had 229 participants (56 in the , 60 in 55 in and 58 in . We received explicit consent from all participants to collect and share their responses. The median completion time for participants in the baseline condition was approximately 13 mins, and they were compensated $3.5 (approximately $16/h); the median completion time for participants in the incentivized condition was approximately 13.5 mins, and they received a guaranteed amount of $1.9 (adjusted up from $1.5) and the average bonus was $1.5 (corresponding to an average wage of $15/h).

The results from this experiment are shown in 11. Similar to Experiment 2, we found the value of \(\alpha\) to be negative across all conditions (11A)—this means that the crossover point for the average participant is greater than 0.2, suggesting a consistent bias towards risk-seeking behavior. We also found the value of \(\beta\) to be significantly larger than one across all conditions (11B), and highest for the condition, indicating that participants are quite sensitive to the stimuli and their subjective crossover point. 11C allows to compare decision quality. We again do not find any meaningful effect of incentives. In addition, we found the expected utility to be greater for both interval conditions ( and ) compared to the density conditions ( and ). While this result is consistent with our previous experiment, it is contrary to the findings of prior work which have generally found that visualizing uncertainty using density plots lead to better decision quality.

As in the previous two studies, we again find that participants took slightly longer to complete the task in the incentivized conditions compared to the baseline conditions. 5B, visualizes the bootstrapped median and 95% quantile intervals for the median estimate. The difference in time spent is approximately 22s for intervals (, ), and approximately 30s for density (, ). This increase of 10-15% is smaller than what was observed for Experiment 2, and similar to the difference observed in Experiment 1.

References↩︎