1 Introduction↩︎

With the proliferation of machine learning (ML) applications into workplaces and everyday life, more people are relying on the output and decisions from complex ML models. This complexity has made it increasingly difficult for non-expert audiences to interact with and understand the output of automated technology. There is a widespread notion that increased transparency and the use of techniques that inform audiences how systems work will lead to improved comprehension and trustworthiness (see [1] for a review). Yet, to date there is little empirical evidence to support that notion. Interpretability is especially recognized as an important feature of machine learning (ML) systems [2]. Predictions are becoming increasingly accurate through the use of “black box” models, such as deep neural networks, which are nevertheless too complex to easily comprehend. Furthermore, these algorithms are increasingly used to make consequential societal decisions.

Nevertheless, relatively little work has focused on how model output is actually processed and interpreted by human evaluators. In this paper, we draw upon Fuzzy Trace Theory (FTT) [3] – an empirically-validated theory of how humans interpret numerical stimuli – to make predictions regarding how model interpretability varies between different human subjects (see [4] for an example of this variation among NASA engineers). According to FTT, model output is encoded into human memory as several simultaneous mental representations that vary in precision. The most precise of these representations is referred to as the verbatim representation. For example, a model may generate an F1-score of 0.66. The verbatim representation of this error is simply “0.66”. However, this number can be difficult to interpret without context, such as the number of classes, and the balance between classes, in the model’s training set. Additionally, the historical difficulty of the task matters, as do the consequences of false positives and false negatives for decision makers. Finally, users may be unfamiliar with how to interpret output scores, impeding their ability to connect these scores to their initial question, or to understand the underlying uncertainty, accuracy, or validity of the associated metric. Beyond the verbatim representation, humans encode several gist representations of the model output which differ in terms of their levels of precision and reliance on context. For example, if the model is going to be used to make a prediction, the simplest categorical gist may distinguish between models that are “predictive” or “not predictive” relative to the state of the art. Even among two predictive models, a more precise ordinal gist may be used to distinguish between a “better” or “worse” model fit depending on the context. Thus, gist representations incorporate background knowledge regarding how the model is to be used in practice – its functional purpose. Gists may also be distinguished from one another by level of abstraction [5], [6]. For example, the gist of a model output – its functional requirements or why it is being used – is quite distinct from the gist of its implementation details – such as how a given result is derived (see also [7]). In short, a user’s ability to interpret, or make sense, of a model’s output, should be encoded distinctly from their ability to explain how the model achieved its result. This motivates our first hypothesis:

Hypothesis 1: Mental representations for interpretability and explainability are encoded distinctly and in parallel

Importantly, individuals differ from one another in the degree to which they rely on more or less precise mental representations when making sense of stimuli. According to FTT, specific skills and personality traits are expected to predict reliance on more precise representation for mathematical tasks. Specifically, individuals who are more numerate – i.e., possess more facility with numbers – have the ability to operate on a detailed representation. Beyond numeracy, individuals with a higher Need for Cognition [8], possess the willingness to rely on the verbatim representation when evaluating model output.

Hypothesis 2: Interpretability is significantly associated with data science expertise, numeracy, and need for cognition

2 Methods↩︎

2.1 Study Setting↩︎

We conducted this study during a design event in which teams of software developers designed systems to automatically combine machine learning models (e.g., different regression and classification systems) into integrated pipelines. Four automated systems were evaluated in this work, each created by an independent team as part of the design event. Each system enabled a user to import a dataset, explore or interact with it, build a model, and reach some final decision.

The purpose of these systems was to solve problems specified by two different groups of end users: 1) subject matter experts (SMEs), and 2) data scientists. Each team was tasked with developing a system front-end to elicit input from end users, and then use that input to automatically construct a pipeline of machine learning models to solve problems specified by those end users. End users then evaluated the output according to its utility along several dimensions. Specifically, end users were given the opportunity to provide feedback to developers of these automated systems by ranking models according to their perceived utility. Thus, the task for developers was to automatically generate a set of models and propose them to the end user for downstream selection who would, in turn, select their preferred model. End users were then given the opportunity to visualize the system generated models, more clearly specify their problems, and edit the system proposed models.

2.2 Subject Recruitment↩︎

The purpose of the study was to evaluate the functionality and usability of these prototype systems. We therefore recruited a convenience sample of subjects using an online sign-up page at http://parenthetic.io/studies. Subjects were recruited based on their stated interest and experience in specific domains. Specifically, we recruited subjects using snowball sampling from designated “gatekeepers” and supplemented this subject pool with a larger base of interested users that indicated relevant experience. Subjects from this user base received additional screening questionnaires to ensure they represented the targeted end user population.

2.3 Individual Traits Questionnaire↩︎

All subjects completed a questionnaire containing several validated psychometric instruments designed to measure skills and personality traits that, according to Fuzzy-Trace Theory, should be associated with gist extraction. These traits include two measures of preference for effortful thinking: Need for Cognition [8] and the Cognitive Reflection Test [9] and three measures of mathematical ability (numeracy) [10]–[12]. Additionally, we included a short form of the “Big Five” personality inventory [13] to assess the effects of other commonly measured individual differences. Finally, subjects assessed their own data science expertise on a 6-point Likert scale ranging from “I have no experience in data analysis” (coded as 0) to “I have extensive coursework and professional experience in data modeling and/or engineering” (coded as 6; see Supplementary Materials).

All subjects completed the composite questionnaire remotely using Qualtrics survey software, with each instrument presented in a random order and items within each instrument presented in random order. Responses for items in the Cognitive Reflection Test, Lipkus Numeracy Test, and Berlin Numeracy Test were marked correct or incorrect, and users were scored using the percent of items marked correct. The remaining inventories were scored according to their source.

Several of the personality traits that we measured are expected to be correlated since they index similar or related constructs (e.g., objective numeracy, subjective numeracy, cognitive reflection, and Need for Cognition should all be correlated somewhat [14]). We therefore conducted a second factor analysis on users’ personality trait scores. Specifically, we conducted an Exploratory Factor Analysis (EFA) with oblique (oblimin) factor rotation and maximum likelihood factor extraction. Factors were retained if their eigenvalues were greater than or equal to 1.0 (Kaiser’s criterion).

2.4 System Evaluation↩︎

After completing a “practice-run” with a simulated system to familiarize subjects with the task, each subject was then randomly assigned to evaluate two (of the four) separate systems. Users completed system-specific training developed by each developer team. Upon completion, subjects were instructed to access their assigned system. Users tested both of their assigned systems at their own convenience during a pre-scheduled week.

A survey was designed to evaluate users’ experience with their assigned systems. The items were originally written to capture three concepts of system utility: the usefulness, meaningfulness, and relevance of the systems’ output. Items were also included to evaluate the functionality of the system, i.e. whether the system crashed or failed to load.

The survey included 96 Likert-scale items and 12 free-response questions, including 24 items capturing usefulness, 28 items capturing meaningfulness, 24 items capturing relevance, and 20 items capturing functionality (see supplementary materials). Some items related to more than one of these concepts and were double-coded. To score the systems, protocol items were converted to numeric values. Negatively-framed questions (e.g., "I did NOT understand the model output") were reverse-coded, so that positive experiences always resulted in a positive value. Negatively- and positively-framed versions of the same item were then averaged. Items that pertained to the user’s assessment of the system, and that had both forward- and reverse-coded versions, were retained for the purposes of analysis, yielding 39 composite items. Finally, we analyzed these 39 items using an Exploratory Factor Analysis (EFA) with oblique (oblimin) factor rotation and maximum likelihood factor extraction. Factors were retained if their eigenvalues were greater than or equal to 1.0 (Kaiser’s criterion).

Subjects were paid $40/hour, for a maximum of 5 hours, for their participation unless they opted out of payment (e.g., due to employer restrictions). Subjects that did not complete their assigned tests were requested to fill out a feedback questionnaire. The study received IRB approval from IntegReview IRB.

2.5 Datasets↩︎

Subject matter expert participants came from five specialty areas: conflict studies, radicalization and extremism, botany, risk assessment, and counter-terrorism. Participants with subject matter expertise in any of these areas used specially selected datasets from their domain when evaluating their assigned systems. Participants with data science expertise, but who did not have subject matter expertise, used botany datasets to evaluate their assigned systems.

3 Results↩︎

3.1 Sample Characteristics↩︎

86 users were recruited to participate in testing across the four datasets. Of these, 43 (50%) users completed at least one test (32 users, 37%, completed two tests, but we only examined users’ first tests to eliminate learning effects). Of these 43 users, 38 (88%) also completed the full personality questionnaire. The distribution of the 38 users whose system evaluations were used in the final regressions is given in Table 2.

3.2 Explainability, Interpretability, Data Discovery, and Ease of Visualization are Discrete Aspects of System Utility↩︎

43 users completed at least one test, and therefore the complete evaluation survey. An EFA yielded 4 distinct factors, explaining 60% of the variance in the data (see Table 3).

These four factors measured the extent to which users were:

1. Able to discover new things (Discovery)

2. Able to make sense of system output (Interpretability)

3. Able to explain how system output was generated (Explainability)

4. Able to make use of Data Visualizations (Visualization)

As predicted, users’ abilities to explain how results were generated and to make sense of those results were statistically distinct.

3.3 Individual Traits Predict Subject Responses↩︎

38 users completed both the personality/individual traits questionnaire and the data-science self-report scale. These scores were used as input to a second EFA, explaining a total 53% of the variance in the subjects’ responses, yielding three distinct factors. These factors, described in Table 4, can be interpreted as positivity, metacognitive monitoring and editing, and self-efficacy (i.e., one’s self-assessment of one’s abilities). Results for both EFAs replicated when using factors derived by varimax rotation.

3.3.1 Interpretability is Associated with Metacognition, Discovery with Familiarity↩︎

For each dimension of system utility (Discovery, Interpretability, and Explainability, and Visualization), we conducted linear regressions using stepwise bidirectional elimination using the Akaike Information Criterion (AIC) as the selection criterion. The starting state for each regression procedure included all three personality factors as main effects, controlling for system used, dataset used, and whether the subject was a subject-matter expert, current used a formal method in their work, or currently used ML/AI tools in their work. The bidirectional stepwise algorithm progressively removed these main effects and control variables, or added statistical interactions between main effects, until a local minimum in AIC was reached(see Table 5). All results replicated when using factors derived by varimax rotation.

4 Discussion↩︎

Our results provide support for Fuzzy-Trace Theory’s predictions. Specifically, we find that interpretability and explainability are statistically distinct, as demonstrated by the fact that they loaded separately on a factor analysis. They are nevertheless somewhat correlated. This indicates that there are likely to be some subjects that are able to derive meaning from an understanding of how a system operates, but that the majority of subjects could not.

According to Fuzzy-Trace Theory, individual differences mediate the extent to which subjects rely on different mental representations. Specifically, factors associated with metacognitive monitoring and editing – such as numeracy and Need for Cognition – should predict reliance on increasingly precise mental representations. Our data show that a factor associated with these metacognitive traits is indeed associated with increased interpretability assessment, even after controlling for the specific system that generated model output.

Additionally, we found that a subject’s familiarity with formal methods was associated with discovery, controlling for dataset. In effect, these results suggest that the ability to generate a new result using an automated machine-learning system may be primarily driven by experience with that system. Notably, no significant predictors were associated with changes in explainability. Furthermore, we did not detect any changes in discovery or explainability between systems. Importantly, there was a difference in discovery between datasets, with the botany and counter-terrorism datasets both more difficult to use for discovery purposes.

Beyond these factors, our results suggest that in order to understand that result and apply it to a real-world problem, it must be interpreted. The ability to generate these interpretations, in turn, may depend on individual personality traits, such as those associated with mathematical ability and gist processing. Furthermore, the fact that interpretability is distinct from visualization indicates that communicating that meaning of a machine learning model’s results is not simply just a matter of improved visualization.

Notably, the strongest predictor of whether a subject was able to generate a meaningful interpretation was whether they used “System A”. This means that it is indeed possible to design systems that promote interpretability, and that these systems may not be the same as those that promote explainability.

Overall, our results suggest that interpretability depends on mental representation. Any algorithms that strives for interpretability must take mental representation into account, highlighting the need for design work to first extract these representations. Since users differ in both their background knowledge and in their individual traits, interpretable systems may have to tailor their outputs to the specific needs of their users. Importantly, these features are not entirely random; rather, our data seem to indicate that they depend on well-known and characterized predispositions to metacognitive monitoring and editing, and less so on personality traits that vary significantly between subjects (e.g., the “Big Five”).

4.1 Limitations↩︎

Due to the small size of our sample, it is likely that there are confounding factors at play. For example, the botany dataset contained the most subjects because all subjects without subject-matter expertise were assigned to it. Additionally, subjects were not evenly balanced between datasets, systems, and prior use of formal methods and ML/AI, in large part because not all recruited users completed testing. Future work must therefore more explicitly control for these factors. Finally, the small sample size leaves us unable to examine several theorized interactions, such as between subject-matter expertise and metacognition.

This work is also limited by the fact that data science expertise was self-reported. Indeed, our factor analysis results show that data science expertise loaded with subjective numeracy rather than with Cognitive Reflection and objective numeracy, meaning that it may be more of a measure of self-efficacy than of actual skill. Creating a reliable, theoretically motivated measure of data science expertise is an exciting opportunity for future work.

4.2 Conclusions↩︎

In conclusion, our analysis offers preliminary evidence for the distinct contributions of interpretability and explainability to system utility. Per Fuzzy-Trace Theory, interpretability should be associated with less precise, yet productive, gist processing, whereas explainability may be more associated with the ability to debug systems, but not necessarily the ability to apply their output to solve real-world problems. Both skill sets are necessary, motivating the need for more research into systems, and users, that can translate between these different mental representations when machine learning tools are used in high-stakes settings.

References↩︎