An ordinal variable refers to variables where the categories have a natural order, but are not metric [1] (i.e., the differences between successive levels may not be treated as equal intervals). The use of ordinal measures (e.g., surveys which elicit participants’ responses on a Likert scale) are commonly used by researchers in visualization, as well as in other fields such as hci or psychology. These measures provide a valuable way of measuring metrics such as trust in a visualization, confidence in using a new interface, preference or attitudes etc., which we wouldn’t be able to capture otherwise.

Despite their widespread use, researchers tend to use a wide variety of statistical approaches for analysing Likert data, some of which makes strong and arguably unfounded assumptions about the data. For instance, the anova test, frequently used for analysing Likert responses, assumes that the data is metric [2]. Others use non-parametric approaches which, while not making assumptions on the data distribution, reduces the data to relative ordered ranks [3] which may also be problematic. Instead, Syiem and Velloso [3], echoing prior calls from other fields [2], have argued for the adoption of ordinal regression models instead, as these models make less stringent assumptions regarding the data, and more directly model the data generating process.

Like Syiem and Velloso [3], I also believe that ordinal cumulative-link models are the more appropriate approach for analysing Likert data—ordinal models make much more reasonable assumptions regarding the data-generating process and generally tends to fit the data better.² However, I posit that the a key reason for the lack of adoption of ordinal models is 1 perhaps not due to researchers not knowing how to implement such models, as there have been numerous blogposts [4]–[6], books [7] and scientific articles [3], [8]–[10] describing both frequentist and Bayesian approaches; instead, 1the fact that we currently do not have good approaches for visualising the results of ordinal models which are effective at communicating the takeaways that analysts wish to communicate regarding their analysis. the fact that we currently do not have good approaches for visualising the results of ordinal models which are effective at communicating the takeaways that analysts wish to communicate regarding their analysis. As prior work has used various approaches for communicating the results of Likert scale responses, I first conduct a preliminary review of reporting strategies used for ordinal data, based on a snowball sample of prior work; I summarise the key message that researchers are trying to communicate and identify key differences between when the data is treated as metric or ordinal.³

I find that an important benefit of a metric-based analysis of ordinal responses is the ability to report results and describe differences between conditions on the response scale (e.g., participants in condition [\(X\)], on average, reported 2 points higher trust on a 7-point Likert scale, compared to participants in condition [\(Y\)]); these differences can also be very intuitively visualized (see 2). Analogous statements are very difficult (although not impossible) to make if the data were analyzed using an ordinal model instead. Based on these findings, I propose the use of modified Complementary Cumulative Distribution Function (mCCDF) plots to visualize the results of ordinal regression models. Even though mCCDFs are less straightforward for interpreting the results (compared to the interval plots that researchers use when treating the results as metric), they do not violate the ordinal assumption by treating the data as metric. More importantly, I demonstrate how mCCDFs allow an analyst to communicate the analogous information using ordinal models that metric-based analyses of ordinal data affords, which can be further reinforced through annotations (3).

1 Preliminaries↩︎

To make the discussion throughout this paper more concrete, I implement and use the results of an ordinal regression model. I introduce the dataset and the the model below (see also 6)

1.1 The Dataset↩︎

To demonstrate the mCCDF visualization, we first need an ordinal model. I use a dataset on moral intuition [7], [11] which contains the results of an experiment where participants are presented with different scenarios involving the trolley problem. The scenarios manipulate three principles—action, intention and contact. In this experiment, participants are presented with a number of scenarios where one or more of these principles apply; thus whether a principle applies in a particular scenario is the experimental variable. Participants’ responses are recorded on a 7-point scale where higher values indicate greater moral permissibility. For a more detailed discussion on the experiment, I would point readers to [7], [11].

1.2 The Ordinal Regression Model↩︎

I fit a Bayesian⁴ cumulative-link model using the probit link function on the moral intuition dataset using action, intention and contact as predictors. The cumulative model assumes that an observed ordinal variable, \(Y\), originates from a categorization of a latent continuous variable, \(\tilde{Y}\) [3], [8], which follows the unit normal distribution (1A). To achieve this categorization, the model then assumes that there are \(k\) thresholds \(\tau_k\) which categorize the latent variable \(\tilde{Y}\) into \(k + 1\) ordered categories (1B). The thresholds \(\tau_k\), are on the scale of the latent variable, and are estimated from the data:

\[\Phi^{-1}(\mathbb{P}(y_i \leq k)) = \tau_k\]

where \(\Phi\) is the probit function⁵ corresponding to the unit normal distribution. We can estimate the probability of \(Y\) being equal to a category \(k\) as \(\mathbb{P}(Y = k) = \Phi^{-1}(\tau_k) - \Phi^{-1}(\tau_{k-1})\). This is extended to a regression model by assuming that the different conditions will have different underlying latent distributions \(\tilde{Y_j}\) (see 1).

2 Visualizing Ordinal Data↩︎

To get a better understanding of what researchers want to communicate from their analysis of ordinal data, I conducted an informal survey of prior work in visualization. I began with a few initial publications that I was familiar with which contained analyses of ordinal data [13]–[17] and 1conducted a search in the ieee digital library using either the keyword likert or ordinal regression for papers published between the years 2025–2026 at ieee vis, 1the premier publication venue in the field of visualizationthe premier publication venue in the field of visualization (similar to the methodology used by South et al [18]). This search returned 60 results. However, as this yielded only two examples of studies which used ordinal regression models, I expanded the years to 2020–2026 only for the keyword ordinal regression. This resulted in a sample of 72 papers, of which 55 used a visualization to communicate the results of a Likert data analysis. I supplement the findings from my survey with the results of South et al.’s [18] systematic review of the use of Likert scale in Visualization papers. The qualitative analysis can be found in supplement \(\blacktriangleright\) survey.conducted a search in the ieee digital library using either the keyword likert or ordinal regression for papers published between the years 2025–2026 at ieee vis, 1the premier publication venue in the field of visualizationthe premier publication venue in the field of visualization (similar to the methodology used by South et al [18]). This search returned 60 results. However, as this yielded only two examples of studies which used ordinal regression models, I expanded the years to 2020–2026 only for the keyword ordinal regression. This resulted in a sample of 72 papers, of which 55 used a visualization to communicate the results of a Likert data analysis. I supplement the findings from my survey with the results of South et al.’s [18] systematic review of the use of Likert scale in Visualization papers. The qualitative analysis can be found in supplement \(\blacktriangleright\) survey. 1 However, this returned only one example of a study which used ordinal regression models, suggesting that treating ordinal data as metric is likely the much more common approach; thus, I conducted a subsequent search using the keyword ordinal regression for papers published at ieee vis between the years 2020 – 2026. I chose this informal approach, as the objective of this survey was to get a sense of the diversity in analysis and reporting strategies that researchers have used 1and how researchers communicate the key takeaways from their Likert dataand how researchers communicate the key takeaways from their Likert data, and not to quantify how frequently each modelling approach has been adopted.

2.1 When the Data is Modeled as Metric↩︎

Several of the papers in my survey (25/55) reported means and an uncertainty estimate (most commonly 95% confidence or credible intervals, but also in some cases standard deviations; see 2A) for each item [16], [19], [23]. Some researchers report the means and 95% intervals aggregated across multiple items [14], [20], by taking the average for each participant, which is how Likert scales were original intended to be used [24], [25]. Other approaches that I observed were reporting the data distribution directly (2B) and comparing the means and the shapes of distribution. [17], or using boxplots and densities [26]. 1In South et al’s review [18], 30/65 papers with a visualization communicate some form of mean and uncertainty estimateIn South et al’s review [18], 30/65 papers with a visualization communicate some form of mean and uncertainty estimate.

Researchers primarily used ordinal data to make comparisons between conditions1 i.e., which condition is rated higher. by communicating mean point estimates and differences in means on the response scale. Even though a (hypothetical) statement such as “participants rated visualization [\(X\)] to be more trustworthy (M = 4.1, SD = 1.05) compared to visualization [\(Y\)] (M = 3.6, SD = 1.4)” may be technically flawed⁶ when applied to data which is not on an interval scale as it entails arithmetic operations [3], it provides a sense of the data distribution and the average response in a particular condition are (at least approximately), and crucially how large the differences between conditions are on the response scale. For instance, Nadib et al. [16] reporting higher trust in one of their conditions contextualize this by adding “the effect, albeit statistically significant, is not large: trust increases by roughly half of a Likert point [...].”

2.2 When the Data is Modeled as Ordinal↩︎

I observed four distinct approaches for communicating the results of an ordinal model: (i) visualizing the probability of each ordinal category for each condition [13], [21], [29]; (ii) visualizing the empirical cumulative distribution function (cdf) plots [22]; (iii) showing the data distribution and calculating the odds ratio between conditions [30]; and (iv) estimating mean and 95% credible intervals from posterior draws (for a single item or averaged across multiple items) [15], [31]. The difference in how difficult it is to comprehend the results of ordinal models compared to metric models is stark—the first two approaches (2C-D), while more appropriate for the analysis, can make it significantly more difficult for a viewer1 to quickly and intuitively make sense of what is going on 1to determine how much higher/better one condition is rated on average, compared to other conditions, in terms of points on the Likert scale (a straightforward task using the mean and interval plots).to determine how much higher/better one condition is rated on average, compared to other conditions, in terms of points on the Likert scale (a straightforward task using the mean and interval plots).

This might be perhaps why Yang et al. [15] and Dragicevic et al. [31] adopted the approach of first modelling the data as ordinal, and then treating it as metric by estimating the mean of posterior predicted probability⁷ across the ratings—it allowed them to communicate results on the response scale. While the approach by Rogha et al. [22] (see 2D) visualizes the empirical cumulative distribution function (i.e., the probability that a participant rated at least [\(k\)] on the 7-point Likert scale) for each condition on the response scale, it remains challenging and counter-intuitive to interpret1—to determine which condition is rated higher, the viewer has to identify the curve which is visually lower, which runs counter to the typical convention in visualization of encoding higher as better—to determine which condition is rated higher, the viewer has to identify the curve which is visually lower, which runs counter to the typical convention in visualization of encoding higher as better. This unintuitiveness of the ordinal model has also led to an arguably idiosyncratic decision for reporting results in my own prior work [13]—I reported the probability of participants rating 3 or higher on a 5-point Likert scale to compare across conditions, with the choice of 3 or higher being arbitrary (although hopefully justifiable). More conventionally, Bradley et al.’s [30] decision to report the odds ratio is similar to recommendations made by Syiem and Velloso [3]; they even augment this by converting the odd’s ratio to a Cohen’s d value. However, despite all this, the odds ratio between two variables on a latent (inverse probit) scale is significantly less intuitive than information communicated on the response scale.

1Researchers also often communicated their key takeaways based on the proportion of responses that fell into, or above, a specific category, without modelling the data as either metric or ordinal, and using visualizations of the distribution of Likert responses such as histograms, stacked bar charts or heatmaps (26/55 papers in my survey and 27/65 papers in [18]). This takeaway, which is analogous to estimating \(Pr(y_i \geq k)\), can be easily obtained using an ordinal model; however, this value can be quite difficult to estimate using either of the visualizations used in prior work (2C-D).Researchers also often communicated their key takeaways based on the proportion of responses that fell into, or above, a specific category, without modelling the data as either metric or ordinal, and using visualizations of the distribution of Likert responses such as histograms, stacked bar charts or heatmaps (26/55 papers in my survey and 27/65 papers in [18]). This takeaway, which is analogous to estimating \(Pr(y_i \geq k)\), can be easily obtained using an ordinal model; however, this value can be quite difficult to estimate using either of the visualizations used in prior work (2C-D).

3 Towards a Better Visualization of Ordinal Regression Models↩︎

3.1 Design Properties↩︎

Based on the discussion above, I outline a set of design properties for visualizing the results of ordinal regression models which will make it intuitive for a viewer to easily extract the key pieces of information they care about:

The visualization should communicate the central tendency on the response scale (e.g., what is the model estimated response for the median participant?

The visualization should communicate uncertainty in the central tendency on the response scale (e.g., what is the uncertainty in the median estimate?

The visualization should easily allow the reader to identify which condition is rated “better.” Consistent with most convention, I argue that the visual representation should encode conditions which are “better” higher (unlike the cdf plot in 2D).

The visualization should allow comparison between conditions on the response scale (e.g., how many points higher does the median participant in one condition rate than the median participant in another condition?).

1I propose the use of a modified Complementary Cumulative Distribution Function, which fulfils these design properties, as a better visualization of the results of ordinal modelsI propose the use of a modified Complementary Cumulative Distribution Function, which fulfils these design properties, as a better visualization of the results of ordinal models

3.2 The modified Complementary Cumulative Distribution Function (mCCDF) Plot↩︎

The Complementary Cumulative Distribution Function (CCDF) can be calculated from a cumulative probit-link regression model using a linear transformation: \(\mathrm{CCDF} = \Phi^{-1}(y_i > k) = 1 - \Phi^{-1}(y_i \leq k)\). 1Thus, the ccdf tells us the probability of an item being rated greater than category \(k\). However, for Likert data, we care about the probability of an item being rated at category \(k\) or higher, which can be obtained using the following modified CCDFThus, the ccdf tells us the probability of an item being rated greater than category \(k\). However, for Likert data, we care about the probability of an item being rated at category \(k\) or higher, which can be obtained using the following modified CCDF:

\[\mathrm{mCCDF} = \Phi^{-1}(y_i {\color{myblue}{\geq}} k) = 1 - \Phi^{-1}(y_i < k)\]

While seemingly trivial to estimate, the mCCDF has properties which make it more suitable to communicate the results of ordinal regression models, especially for single items. The rating corresponding to the intersection of the mCCDF draws with \(y = 0.5\) represents the median rating for a particular condition. In 3, the median rating is likely going to be a 5 or 6, even after taking uncertainty into account (D1, D2). The uncertainty in the model estimates is relatively low for this model and dataset due to the large number of responses collected.

In addition, the mCCDF more directly encodes one of the most relevant pieces of information for ordinal data—the probability that participants rated \(k\) or higher (\(\mathbb{P}(y_i \geq k)\)), unlike the conventional cdf plot which encodes \(\mathbb{P}(y_i \leq k)\). This makes it more intuitive to perform two types of comparisons: (i) how much more likely are participants in one condition to rate \(k\) or higher relative to another 1(D3), as shown in 4B2(D3), as shown in 4B2 (e.g., the probability of the action, intention, no contact condition being rated 6 or higher is 0.3 less than the baseline condition); and (ii) how much better one condition is relative to another 1(D4)(D4), based on the median participant (e.g., in 4B(3), the median participant in the action, intention, no contact condition is going to rate at least a four, and the median participant in the baseline condition is going to rate at least a 5). Note that while these comparisons can be performed by a viewer using cdf plots as well, they are far less intuitive (for an approximation of the challenge, compare the magnitude and direction of the difference between conditions in 4A).

An alternative approach for representing mCCDFs is visualizing them as step functions (a piecewise constant function). Representing mCCDFs as step functions has some trade-offs—reading the mean probabilities for each rating (and the associated uncertainty) for a particular condition becomes easier; however, determining the median rating for a condition, and thus comparing the median rating between conditions becomes more difficult. As such, the designer should choose which communication goal they wish to prioritise. ⁸

3.3 Scales with Multiple Correlated Items↩︎

In some of the studies in the survey [14], [15], [20] responses from multiple, potentially correlated, items were aggregated and reported using a single point estimate and uncertainty interval.1 The analogous approach which treats the data as ordinal would be to take the average, across the multiple scales, of the (posterior) probabilities for each rating 1An analyst could instead fit a single hierarchical model to the data, with the correlated items modelled as random effects with different latent means and variances (referred to as difficulty and discrimination parameters in item-response theory). As scales with potentially correlated items are often measuring the same underlying construct, partial pooling of information in a hierarchical model allows parameters to be more efficiently estimated. We can then create mCCDF plots for each item individually, an “average” item (i.e., when the latent mean is zero), or marginalized over all of the items. I demonstrate how such an analysis can be implemented, and how the corresponding mCCDF plots can be created for each item in supplement \(\blacktriangleright\) RScript \(\blacktriangleright\) 03-unequal-variance-ccdf.Rmd. An analyst could instead fit a single hierarchical model to the data, with the correlated items modelled as random effects with different latent means and variances (referred to as difficulty and discrimination parameters in item-response theory). As scales with potentially correlated items are often measuring the same underlying construct, partial pooling of information in a hierarchical model allows parameters to be more efficiently estimated. We can then create mCCDF plots for each item individually, an “average” item (i.e., when the latent mean is zero), or marginalized over all of the items. I demonstrate how such an analysis can be implemented, and how the corresponding mCCDF plots can be created for each item in supplement \(\blacktriangleright\) RScript \(\blacktriangleright\) 03-unequal-variance-ccdf.Rmd.

1An additional benefit of adopting such models is that they can be useful for evaluating the items themselves, by assessing the degree of similarity in responses to the different items. Analogous approaches are used in item-response theory to evaluate questions, based on the coefficients for the difficulty and discrimination parameters and using item characteristic curves (ICC), to create validated set of test questions (see Section 5-7 and Figure 11 in [32]). For ordinal models, mCCDFs can complement the information presented using ICCs (which show the modeled probability of responding at or above each category as a function of the latent trait) by visualizing how similar participants’ actual responses to each item actually are, in the response scale. An additional benefit of adopting such models is that they can be useful for evaluating the items themselves, by assessing the degree of similarity in responses to the different items. Analogous approaches are used in item-response theory to evaluate questions, based on the coefficients for the difficulty and discrimination parameters and using item characteristic curves (ICC), to create validated set of test questions (see Section 5-7 and Figure 11 in [32]). For ordinal models, mCCDFs can complement the information presented using ICCs (which show the modeled probability of responding at or above each category as a function of the latent trait) by visualizing how similar participants’ actual responses to each item actually are, in the response scale.

3.4 Should We All Be Using Ordinal Models?↩︎

While I personally prefer to use ordinal models for Likert data, I do not hold strong beliefs that this is objectively the best way of analysing ordinal data, as I am yet to find compelling evidence to suggest that the conclusions would be different regardless of the choice of models. Liddell and Kruschke [2] try to demonstrate that ordinal regression is better, and that treating ordinal data as metric can lead to inflated false positive and false negatives, using simulated data. However, for their simulation, they generate ordinal data by discretizing a latent variable; thus, what they actually demonstrated was that the model which is consistent with their assumed data-generating process was better than a model which made obviously incorrect assumptions in the small world of their simulation.⁹ In contrast, Dragicevic et al. [31] compare nine different data analysis approaches for the same dataset and found little difference in terms of takeaways between ordinal regression and metric-based approaches. 1 Thus, the objective of this work is not necessarily to urge researchers to adopt ordinal regression models, but rather demonstrate approaches for visualizing results of ordinal regression if researchers choose to adopt such models. 1My preference for using ordinal models is one based on statistical philosophy—I consider approaches which model the data directly, with as few assumptions as possible, to be better. The value of an ordinal model lies in the fact that it makes fewer assumptions regarding the data generating processes of Likert responses compared to metric-based approaches, which is why I would still recommend the use of ordinal regression model. However, the primary goal of this paper is not necessarily to urge researchers to adopt ordinal regression models but rather to demonstrate approaches for visualizing results of ordinal regression if researchers choose to adopt such models.My preference for using ordinal models is one based on statistical philosophy—I consider approaches which model the data directly, with as few assumptions as possible, to be better. The value of an ordinal model lies in the fact that it makes fewer assumptions regarding the data generating processes of Likert responses compared to metric-based approaches, which is why I would still recommend the use of ordinal regression model. However, the primary goal of this paper is not necessarily to urge researchers to adopt ordinal regression models but rather to demonstrate approaches for visualizing results of ordinal regression if researchers choose to adopt such models.

4 Conclusion↩︎

When analysing ordinal data, researchers appear to prefer to treat the data as metric and communicate the results—using point estimates, uncertainty, and differences between conditions—on the response scale. Complementary Cumulative Distribution Function (mCCDF) plots offer an approach for communicating the results of cumulative-link ordinal regression models on the response scale, allowing the reader to extract analogous pieces of information.

5 Supplementary Materials↩︎

All supplemental materials are available on OSF at https://osf.io/cdgby/, released under a CC BY 4.0 license. In particular, they include (1) Excel files containing the data for the literature survey, (2) full R scripts for recreating the analysis and figures in the paper, (3) figure images, and (4) a full version of this paper with all appendices.

6 Details on Dataset and Ordinal Model Used↩︎

6.1 Dataset↩︎

The dataset used in the analysis described in this paper was collected by Cushman et al. [11] in a series of experiments to collect empirical evidence on how people judge the “moral goodness or badness” of actions [7]. The experiments present participants with a scenario—variations of the trolley problem—and asks participants to rate the action (or inaction) on a scale from 1 to 7. The following is the traditional version of the trolley problem [7]:

“Standing by the railroad tracks, Dennis sees an empty, out-of-control trolley about to hit five people. Next to Dennis is a lever that can be pulled, sending the trolley down a side track and away from the five people. But pulling the lever will also lower the railing on a footbridge spanning the side track, causing one person to fall off the footbridge and onto the side track, where he will be hit by the trolley. If Dennis pulls the lever the trolley will switch tracks and not hit the five people, and the one person to fall and be hit by the trolley. If Dennis does not pull the lever the trolley will continue down the tracks and hit five people, and the one person will remain safe above the side track.”

The experiments by Cushman et al. [11] investigate the role of three principles from moral psychology and philosophy on moral judgements. The principles are:

1. The action principle: Harm caused by action is morally worse than equivalent harm caused by omission.

2. The contact principle: Using physical contact to cause harm to a victim is morally worse than causing equivalent harm to a victim without using physical contact.

3. The intention principle: Harm intended as the means to a goal is morally worse than equivalent harm foreseen as the side effect of a goal.

The above scenario applies the action principle as the actor has to perform an action to create the outcome (as opposed to being passive and letting events unfold). However, the other two principles do not apply as there is no contact (i.e., no contact), and the harm caused to the person on the footbridge is accidental and not intentional (i.e., no intention). The three principles can be varied to create different scenarios. In the study by Cushman et al. [11], participants “received 32 moral scenarios separated into two blocks of 16. Each block included 15 test scenarios and 1 control scenario.” The dataset contains responses from 331 participants, and includes details on participants’ age, gender, and level of education. For more details on the dataset, please refer to [7], [11].

6.2 Analysis↩︎

The mCCDF visualizations described in this paper are based on the results of the analysis outlined by McElreath [7], which applies a Bayesian ordinal cumulative-logit model. However, I instead use the probit link function (as opposed to the logit link function used by McElreath [7]), due to mathematical convenience. The model can be described using the Wilkinson-Rogers-Pinheiro-Bates syntax [37]–[39] as:

The rating the participants assigned is modeled as a Ordered categorical distribution.

The categorical distribution takes a vector of probabilities \(p = {p_1, p_2, p_3, p_4, p_5, p_6}\) of probabilities of each response value below the maximum response (\(p_7\)).

Each response value k in this vector is defined by its link, using the probit function to an intercept parameter, \(\tau_k\) and the linear model \(\phi_i\).

For each response, we assume that participants’ ratings depend on the condition that the principle (or the combination of principles) the scenario is based on.

We use weakly-regularizing priors centered on zero (no effect) while permitting the possibility of significant distortion: \(\beta, \tau_j \sim \mathrm{Normal}(0, 1)\).

We implemented these models in R 4.4.0 [40] and CmdStanR 0.8.0 [41]. The model ran for four chains with 5,000 warmup samples and 5,000 post-warmup samples each, thinned by 4 for a final total sample size of 5,000. We assessed convergence using the Gelman-Rubin diagnostic (\(\hat{R}\) = 1.00 for all population-level parameters, correlations and standard deviations) and the (bulk and tail) effective sample sizes (\(\text{ESS}_{min} \approx 2,000\)).

References↩︎