1 Introduction↩︎

The recent promise of multimodal models that integrate information from heterogeneous sources of data has led to their proliferation in numerous real-world settings such as multimedia [1], affective computing [2], robotics [3], and healthcare [4]. Subsequently, their impact towards real-world applications has inspired recent research in visualizing and understanding their internal mechanics [5]–[7] as a step towards accurately benchmarking their limitations for more reliable deployment [8], [9]. However, modern parameterizations of multimodal models are typically black-box neural networks, such as pretrained transformers [10], [11]. How can we visualize and understand the internal modeling of multimodal information and interactions in these models?

As a step in interpreting multimodal models, this paper introduces an analysis and visualization method called MultiViz(see Figure 1). To tackle the challenges of visualizing model behavior, we scaffold the problem of interpretability into \(4\) stages: (1) unimodal importance: identifying the contributions of each modality towards downstream modeling and prediction, (2) cross-modal interactions: uncovering the various ways in which different modalities can relate with each other and the types of new information possibly discovered as a result of these relationships, (3) multimodal representations: how unimodal and cross-modal interactions are represented in decision-level features, and (4) multimodal prediction: how decision-level features are composed to make a prediction for a given task. In addition to including current approaches for unimodal importance [6], [12], [13] and cross-modal interactions [14], [15], we additionally propose new methods for interpreting cross-modal interactions, multimodal representations, and prediction to complete these stages in MultiViz. By viewing multimodal interpretability through the lens of these \(4\) stages, MultiVizcontributes a modular and human-in-the-loop visualization toolkit for the community to visualize popular multimodal datasets and models as well as compare with other interpretation perspectives, and for stakeholders to understand multimodal models in their research domains.

MultiVizis designed to support many modality inputs while also operating on diverse modalities, models, tasks, and research areas. Through experiments on \(6\) real-world multimodal tasks (spanning fusion, retrieval, and question-answering), \(6\) modalities, and \(8\) models, we show that MultiVizhelps users gain a deeper understanding of model behavior as measured via a proxy task of model simulation. We further demonstrate that MultiVizhelps human users assign interpretable language concepts to previously uninterpretable features and perform error analysis on model misclassifications. Finally, using takeaways from error analysis, we present a case study of human-in-the-loop model debugging. Overall, MultiVizprovides a practical toolkit for interpreting multimodal models for human understanding and debugging. MultiVizdatasets, models, and code are at https://github.com/pliang279/MultiViz.

2 MultiViz: Visualizing and Understanding Multimodal Models↩︎

This section presents MultiViz, our proposed analysis framework for analyzing the behavior of multimodal models. As a general setup, we assume multimodal datasets take the form \(\mathcal{D} = \{ (\mathbf{x}_1, \mathbf{x}_2, y)_{i=1}^n \} = \{ ( x_1^{(1)}, x_1^{(2)}, ..., x_2^{(1)}, x_2^{(2)}, ..., y )_{i=1}^n \}\), with boldface \(\mathbf{x}\) denoting the entire modality, each \(x_1, x_2\) indicating modality atoms (i.e., fine-grained sub-parts of modalities that we would like to analyze, such as individual words in a sentence, object regions in an image, or time-steps in time-series data), and \(y\) denoting the label. These datasets enable us to train a multimodal model \(\hat{y} = f(\mathbf{x}_1, \mathbf{x}_2; \theta)\) which we are interested in visualizing.

Modern parameterizations of multimodal models \(f\) are typically black-box neural networks, such as multimodal transformers [16], [17] and pretrained models [10], [11]. How can we visualize and understand the internal modeling of multimodal information and interactions in these models? Having an accurate understanding of their decision-making process would enable us to benchmark their opportunities and limitations for more reliable real-world deployment. However, interpreting \(f\) is difficult. In many multimodal problems, it is useful to first scaffold the problem of interpreting \(f\) into several intermediate stages from low-level unimodal inputs to high-level predictions, spanning unimodal importance, cross-modal interactions, multimodal representations, and multimodal prediction. Each of these stages provides complementary information on the decision-making process (see Figure 1). We now describe each step in detail and propose methods to analyze each step.

2.1 Unimodal importance (U)↩︎

Unimodal importance aims to understand the contributions of each modality towards modeling and prediction. It builds upon ideas of gradients [18]–[20] and feature attributions (e.g., LIME [13], Shapley values [12]). We implement unimodal feature attribution methods as a module \(\mathrm{\small Uni}(f_\theta, y, \mathbf{x})\) taking in a trained model \(f_\theta\), an output/feature \(y\) which analysis is performed with respect to, and the modality of interest \(\mathbf{x}\). \(\mathrm{\small Uni}\) returns importance weights across atoms \(x\) of modality \(\mathbf{x}\).

2.2 Cross-modal interactions (C)↩︎

Cross-modal interactions describe various ways in which atoms from different modalities can relate with each other and the types of new information possibly discovered as a result of these relationships. Recent work [14], [15] has formalized a definition of cross-modal interactions by building upon literature in statistical non-additive interactions:

Definition 1 (Statistical Non-Additive Interaction [21]–[24]). A function \(f\) learns a feature interaction \(\mathcal{I}\) between \(2\) unimodal atoms \(x_1\) and \(x_2\) if and only if \(f\) cannot be decomposed into a sum of unimodal subfunctions \(g_1, g_2\) such that \(f(x_1, x_2) = g_1(x_1) + g_2(x_2)\).

This definition of non-additive interactions is general enough to include different ways that interactions can happen, including multiplicative interactions from complementary views of the data (i.e., an interaction term \(x_1 \mathbb{W} x_2\) [25]), or cooperative interactions from equivalent views (i.e., an interaction term \(\textrm{majority} (f(x_1), f(x_2))\) [26]). Using this definition, MultiVizfirst includes two recently proposed methods for understanding cross-modal interactions: EMAP [14] decomposes \(f(x_1,x_2) = g_1(x_1) + g_2(x_2) + g_{12}(x_1,x_2)\) into strictly unimodal representations \(g_1, g_2\), and cross-modal representation \(g_{12} = f - \mathbb{E}_{x_1} (f) - \mathbb{E}_{x_2} (f) + \mathbb{E}_{x_1,x_2} (f)\) to quantify the degree of global cross-modal interactions across an entire dataset. DIME [15] further extends EMAP using feature visualization on each disentangled representation locally (per datapoint). However, these approaches require approximating expectations over modality subsets, which may not scale beyond \(2\) modalities. To fill this gap, we propose an efficient approach for visualizing these cross-modal interactions by observing that the following gradient definition directly follows from Definition 1:

Definition 2 (Gradient definition of statistical non-additive interaction). A function \(f\) exhibits non-additive interactions among \(2\) unimodal atoms \(x_1\) and \(x_2\) if \(\mathbf{E}_{x_1,x_2} \left[ \frac{\partial^2 f(x_1,x_2)}{\partial x_1 \partial x_2} \right]^2> 0\).

Taking a second-order gradient of \(f\) zeros out the unimodal terms \(g_1(x_1)\) and \(g_2(x_2)\) and isolates the interaction \(g_{12}(x_1,x_2)\). Theoretically, second-order gradients are necessary and sufficient to recover cross-modal interactions: purely additive models will have strictly \(0\) second-order gradients so \(\mathbf{E}_{x_1,x_2} \left[ \frac{\partial^2 f(x_1,x_2)}{\partial x_1 \partial x_2} \right]^2 =0\), and any non-linear interaction term \(g_{12}(x_1,x_2)\) has non-zero second-order gradients since \(g\) cannot be a constant or unimodal function, so \(\mathbf{E}_{x_1,x_2} \left[ \frac{\partial^2 f(x_1,x_2)}{\partial x_1 \partial x_2} \right]^2> 0\).

Definition 2 inspires us to extend first-order gradient and perturbation-based approaches [13], [27], [28] to the second order. Our implementation first computes a gradient of \(f\) with respect to a modality atom which the user is interested in querying cross-modal interactions for (e.g., \(x_1 = \textit{birds}\)), which results in a vector \(\nabla_1 = \frac{\partial f}{\partial x_1}\) of the same dimension as \(x_1\) (i.e., token embedding dimension). We aggregate the vector components of \(\nabla_1\) via summation to produce a single scalar \(\lVert \nabla_1 \rVert\), before taking a second-order gradient with respect to all atoms of the second modality \(x_2 \in \mathbf{x}_2\) (e.g., all image pixels), which results in a vector \(\nabla_{12} = \left[ \frac{\partial^2 f}{\partial x_1 \partial x_2^{(1)}}, ..., \frac{\partial^2 f}{\partial x_1 \partial x_2^{(|\mathbf{x}_2|)}} \right]\) of the same dimension as \(\mathbf{x}_2\) (i.e., total number of pixels). Each scalar entry in \(\nabla_{12}\) highlights atoms \(x_2\) that have non-linear interactions with the original atom \(x_1\), and we choose the \(x_2\)’s with the largest magnitude of interactions with \(x_1\) (i.e., which highlights the birds in the image, see Figure 2 for examples on real datasets). We implement a general module \(\mathrm{\small CM}(f_\theta, y, x_1, \mathbf{x}_2)\) for cross-modal visualizations, taking in a trained model \(f_\theta\), an output/feature \(y\), the first modality’s atom of interest \(x_1\), and the entire second modality of interest \(\mathbf{x}_2\), before returning importance weights across atoms \(x_2\) of modality \(\mathbf{x}_2\) (see details in Appendix 8.2).

2.3 Multimodal representations↩︎

Given these highlighted unimodal and cross-modal interactions at the input level, the next stage aims to understand how these interactions are represented at the feature representation level. Specifically, given a trained multimodal model \(f\), define the matrix \(M_z \in \mathbb{R}^{N \times d}\) as the penultimate layer of \(f\) representing (uninterpretable) deep feature representations implicitly containing information from both unimodal and cross-modal interactions. For the \(i\)th datapoint, \(z = M_z (i)\) collects a set of individual feature representations \(z_{1}, z_{2}, ..., z_{d} \in \mathbb{R}\). We aim to interpret these feature representations through both local and global analysis (see Figure 1 (right) for an example):

Local representation analysis (\(\textrm{R}_\ell\)) informs the user on parts of the original datapoint that activate feature \(z_{j}\). To do so, we run unimodal and cross-modal visualization methods with respect to feature \(z_{j}\) (i.e., \(\mathrm{\small Uni}(f_\theta, z_{j}, \mathbf{x})\), \(\mathrm{\small CM}(f_\theta, z_{j}, x_1, \mathbf{x}_2)\)) in order to explain the input unimodal and cross-modal interactions represented in feature \(z_{j}\). Local analysis is useful in explaining model predictions on the original datapoint by studying the input regions activating feature \(z_{j}\).

Global representation analysis (\(\textrm{R}_g\)) provides the user with the top \(k\) datapoints \(\mathcal{D}_k(z_{j}) = \{ (\mathbf{x}_1, \mathbf{x}_2, y)_{i=(1)}^k \}\) that also maximally activate feature \(z_{j}\). By further unimodal and cross-modal visualizations on datapoints in \(\mathcal{D}_k(z_{j})\), global analysis is especially useful in helping humans assign interpretable language concepts to each feature by looking at similarly activated input regions across datapoints (e.g., the concept of color in Figure 1, right). Global analysis can also help to find related datapoints the model also struggles with for error analysis.

2.4 Multimodal prediction (P)↩︎

Finally, the prediction step takes the set of feature representations \(z_{1}, z_{2}, ..., z_{d}\) and composes them to form higher-level abstract concepts suitable for a task. We approximate the prediction process with a linear combination of penultimate layer features by integrating a sparse linear prediction model with neural network features [29]. Given the penultimate layer \(M_z \in \mathbb{R}^{N \times d}\), we fit a linear model \(\mathbb{E}\left( Y|X=x \right) = M_z^\top \beta\) (bias \(\beta_0\) omitted for simplicity) and solve for sparsity using: \[\label{sparse95eqn} \hat{\beta} = \mathop{\mathrm{arg\,min}}_{\beta} \frac{1}{2N} \| M_z^\top \beta - y \|_2^2 + \lambda_1 \| \beta \|_1 + \lambda_2 \| \beta \|_2^2.\tag{1}\] The resulting understanding starts from the set of learned weights with the highest non-zero coefficients \(\beta_{\textrm{top}} = \{ \beta_{(1)}, \beta_{(2)}, ... \}\) and corresponding ranked features \(z_{\textrm{top}} = \{z_{(1)}, z_{(2)}, ...\}\). \(\beta_{\textrm{top}}\) tells the user how features \(z_{\textrm{top}}\) are composed to make a prediction, and \(z_{\textrm{top}}\) can then be visualized with respect to unimodal and cross-modal interactions using the representation stage (Section 2.3).

2.5 Putting everything together↩︎

We summarize these proposed approaches for understanding each step of the multimodal process and show the overall MultiVizuser interface in Figure 3. This interactive API enables users to choose multimodal datasets and models and be presented with a set of visualizations at each stage, with an overview page for general unimodal importance, cross-modal interactions, and prediction weights, as well as a feature page for local and global analysis of user-selected features (see Appendix 9 for more algorithm and user interface details).

3 Experiments↩︎

Our experiments are designed to verify the usefulness and complementarity of the \(4\) MultiVizstages. We start with a model simulation experiment to test the utility of each stage towards overall model understanding (Section 3.1). We then dive deeper into the individual stages by testing how well MultiVizenables representation interpretation (Section 3.2) and error analysis (Section 3.3), before presenting a case study of model debugging from error analysis insights (Section 3.4). We showcase the following selected experiments and defer results on other datasets to Appendix 11.

Setup: We use a large suite of datasets from MultiBench [30] which span real-world fusion [31]–[33], retrieval [34], and QA [35], [36] tasks. For each dataset, we test a corresponding state-of-the-art model: MulT [17], LRTF [37], LF [38], ViLT [39], CLIP [40], CNN-LSTM-SA [35], MDETR [41], and LXMERT [42]. These cover models both pretrained and trained from scratch. We summarize all \(6\) datasets and \(8\) models tested in Table 1, and provide implementation details in Appendix 10 and user study details in Appendix 11.

3.1 Model simulation↩︎

We first design a model simulation experiment to determine if MultiVizhelps users of multimodal models gain a deeper understanding of model behavior. If MultiVizindeed generates human-understandable explanations, humans should be able to accurately simulate model predictions given these explanations only, as measured by correctness with respect to actual model predictions and annotator agreement (Krippendorff’s alpha [43]). To investigate the utility of each stage in MultiViz, we design a human study to see how accurately \(21\) humans users (\(3\) users for each of the following \(7\) local ablation settings) can simulate model predictions:

(1) U: Users are only shown the unimodal importance (U) of each modality towards label \(y\).

(2) U + C: Users are also shown cross-modal interactions (C) highlighted towards label \(y\).

(3) U + C + \(\textrm{R}_\ell\): Users are also shown local analysis (\(\textrm{R}_\ell\)) of unimodal and cross-modal interactions of top features \(z_{\textrm{top}} = \{z_{(1)}, z_{(2)}, ...\}\) maximally activating label \(y\).

(4) U + C + \(\textrm{R}_\ell\) + \(\textrm{R}_g\): Users are additionally shown global analysis (\(\textrm{R}_g\)) through similar datapoints that also maximally activate top features \(z_{\textrm{top}}\) for label \(y\).

(5) MultiViz(U + C + \(\textrm{R}_\ell\) + \(\textrm{R}_g\) + P): The entire MultiVizmethod by further including visualizations of the final prediction (P) stage: sorting top ranked feature neurons \(z_{\textrm{top}} = \{z_{(1)}, z_{(2)}, ...\}\) with respect to their coefficients \(\beta_{\textrm{top}} = \{ \beta_{(1)}, \beta_{(2)}, ... \}\) and showing these coefficients to the user.

Using \(20\) datapoints per setting, these experiments with \(15\) users on \(3\) datasets and \(3\) models involve \(35\) total hours of users interacting with MultiViz, which is a significantly larger-scale study of model simulation compared to prior work [15], [44], [45].

Quantitative results: We show these results in Table 2 and find that having access to all stages in MultiVizleads to significantly highest accuracy of model simulation on VQA 2.0, along with lowest variance and most consistent agreement between annotators. On fusion tasks with MM-IMDb and CMU-MOSEI, we also find that including each visualization stage consistently leads to higher correctness and agreement, despite the fact that fusion models may not require cross-modal interactions to solve the task [14]. More importantly, humans are able to simulate model predictions, regardless of whether the model made the correct prediction or not.

To test additional intermediate ablations, we conducted user studies on (6) \(\textrm{R}_\ell\) + P (local analysis on final-layer features along with their prediction weights) and (7) \(\textrm{R}_g\) + P (global analysis on final-layer features along with their prediction weights), to ablate the effect of overall analysis (U and C) and feature analysis (\(\textrm{R}_\ell\) or \(\textrm{R}_g\) in isolation). \(\textrm{R}_\ell\) + P results in an accuracy of \(51.7 \pm 12.6\) with \(0.40\) agreement, while \(\textrm{R}_g\) + P gives \(71.7 \pm 7.6\) with \(0.53\) agreement. Indeed, these underperform as compared to including overall analysis (U and C) and feature analysis (\(\textrm{R}_\ell\) + \(\textrm{R}_g\)).

Finally, we also scaled to \(100\) datapoints on VQA 2.0, representing upwards of \(10\) hours of user interaction (for the full MultiVizsetting), and obtain an overall correctness of \(80\%\), reliably within the range of model simulation using \(20\) points (\(81.7 \pm 2.9\)). Therefore, the sample size of \(20\) points that makes all experiments feasible is still a reliable sample.

We also conducted qualitative interviews to determine what users found useful in MultiViz:

(1) Users reported that they found local and global representation analysis particularly useful: global analysis with other datapoints that also maximally activate feature representations were important for identifying similar concepts and assigning them to multimodal features.

(2) Between Overview (U + C) and Feature (\(\textrm{R}_\ell\) + \(\textrm{R}_g\) + P) visualizations, users found Feature visualizations more useful in \(31.7\%\), \(61.7\%\), and \(80.0\%\) of the time under settings (3), (4), and (5) respectively, and found Overview more useful in the remaining points. This means that for each stage, there exists a significant fraction of data points where that stage is most needed.

(3) While it may be possible to determine the prediction of the model with a subset of stages, having more stages that confirm the same prediction makes them a lot more confident about their prediction, which is quantitatively substantiated by the higher accuracy, lower variance, and higher agreement in human predictions. We also include additional experiments in Appendix 11.1.

3.2 Representation interpretation↩︎

We now take a deeper look to check that MultiVizgenerates accurate explanations of multimodal representations. Using local and global representation visualizations, can humans consistently assign interpretable concepts in natural language to previously uninterpretable features? We study this question by tasking \(15\) human users (\(5\) users for each of the following \(3\) settings) to assign concepts to each feature \(z\) when given access to visualizations of (1) \(\textrm{R}_\ell\) (local analysis of unimodal and cross-modal interactions in \(z\)), (2) \(\textrm{R}_\ell\) + \(\textrm{R}_g\) (no viz) (including global analysis through similar datapoints that also maximally activate feature \(z\)), and (3) \(\textrm{R}_\ell\) + \(\textrm{R}_g\) (adding highlighted unimodal and cross-modal interactions of global datapoints). Using \(20\) datapoints per setting, these experiments with \(15\) users involve roughly \(10\) total hours of users interacting with MultiViz.

Quantitative results: Since there are no ground-truth labels for feature concepts, we rely on annotator confidence (\(1\)-\(5\) scale) and annotator agreement [43] as a proxy for accuracy. From Table ¿tbl:tab:local95rep? (left), we find that having access to both local and global visualizations are crucial towards interpreting multimodal features, as measured by higher confidence with low variance in confidence, as well as higher agreement among users.

Qualitative interviews: We show examples of human-assigned concepts in Figure 4 (more in Appendix 11.3). Note that the \(3\) images in each box of Figure 4 (even without feature highlighting) does constitute a visualization generated by MultiViz, as they belong to data instances that maximize the value of the feature neuron (i.e. \(\textrm{R}_g\) in stage \(3\) multimodal representations). Without MultiViz, it would not be possible to perform feature interpretation without combing through the entire dataset. Participants also noted that feature visualizations make the decision a lot more confident if its highlights match the concept. Taking as example Figure 4 top left, the visualizations serve to highlight what the model’s feature neuron is learning (i.e., highlighting the person holding sports equipment), rather than what category of datapoint it is. If the visualization was different, such as highlighting the ground, then users would have to conclude that the feature neuron is capturing ‘outdoor ground’ rather than ‘sports equipment’. Similarly, for text highlights (Figure 4 top right), without using MultiVizto highlight ‘counter’, ‘countertop’, and ‘wall’, along with the image crossmodal interactions corresponding to these entities, one would not be able to deduce that the feature asks about material - it could also represent ‘what’ questions, or ‘household objects’, and so on. Therefore, these conclusions can only be reliably deduced with all MultiViz stages.

3.3 Error analysis↩︎

We further examine a case study of error analysis on trained models. We task \(10\) human users (\(5\) users for each of the following \(2\) settings) to use MultiVizand highlight the errors that a multimodal model exhibits by categorizing these errors into one of \(3\) stages: failures in (1) unimodal perception, (2) capturing cross-modal interaction, and (3) prediction with perceived unimodal and cross-modal information. Again, we rely on annotator confidence (\(1\)-\(5\) scale) and agreement due to lack of ground-truth error categorization, and compare (1) MultiViz with (2) No viz, a baseline that does not provide any model visualizations to the user. Using \(20\) datapoints per setting, these experiments with \(10\) users on \(2\) datasets and \(2\) models involve roughly \(15\) total hours of users interacting with MultiViz. From Table ¿tbl:tab:local95rep? (right), we find that MultiVizenables humans to consistently categorize model errors into one of \(3\) stages. We show examples that human annotators classified into unimodal perception, cross-modal interaction, and prediction errors in Figure 5 (more in Appendix 11.4).

3.4 A case study in model debugging↩︎

Following error analysis, we take a deeper investigation into one of the errors on a pretrained LXMERT model fine-tuned on VQA 2.0. Specifically, we first found the top \(5\) penultimate-layer neurons that are most activated on erroneous datapoints. Inspecting these neurons carefully through MultiVizlocal and global representation analysis, human annotators found that \(2\) of the \(5\) neurons were consistently related to questions asking about color, which highlighted the model’s failure to identify color correctly (especially blue). The model has an accuracy of only \(5.5\%\) amongst all blue-related points (i.e., either have blue as correct answer or predicted answer), and these failures account for \(8.8\%\) of all model errors. We show examples of such datapoints and their MultiVizvisualizations in Figure 6. Observe that the model is often able to capture unimodal and cross-modal interactions perfectly, but fails to identify color at prediction.

Curious as to the source of this error, we looked deeper into the source code for the entire pipeline of LXMERT, including that of its image encoder, Faster R-CNN [46]¹. We in fact uncovered a bug in data preprocessing for Faster R-CNN in the popular Hugging Face repository that swapped the image data storage format from RGB to BGR formats responsible for these errors. This presents a concrete use case of MultiViz: through visualizing each stage, we were able to (1) isolate the source of the bug (at prediction and not unimodal perception or cross-modal interactions), and (2) use representation analysis to localize the bug to the specific color concept. In Appendix 11.5, we further detail our initial attempt at tackling this error by using MultiVizanalysis to select additional targeted datapoints in an active learning scenario, which proved to be much more effective (higher improvement with fewer data) as compared to baselines that add data randomly or via uncertainty sampling [47], which may be of independent interest.

3.5 Additional experiments and takeaways messages↩︎

New models: We included results on ViLT [39], CLIP [40], and MDETR [41] in Appendix 11.2, showing that MultiVizis a general approach that can be quickly applied to new models. We also study the correlation between performance and cross-modal interactions across several older and recent models, and find that the ability to capture cross-modal alignment, as judged by MultiViz, correlates strongly with final task performance.

Sanity checks: In Appendix 8.5, we show that MultiVizpasses the data randomization and model randomization sanity checks for interpretation approaches [48].

Intermediate-layer features: In Appendix 9.3, we show that MultiVizcan be extended to visualize any intermediate layer, not just the final layer of multimodal models. We showcase a few examples of \(\boldsymbol{R}_\ell\) and \(\boldsymbol{R}_g\) on intermediate-layer neurons and discuss several tradeoffs: while they reveal new visualization opportunities, they run the risk of overwhelming the user with the number of images they have to see multiplied by \(d^L\) (\(d\): dimension of each layer, \(L\): number of layers).

4 Related Work↩︎

Interpretable ML aims to further our understanding and trust of ML models, enable model debugging, and use these insights for joint decision-making between stakeholders and AI [49], [50]. Interpretable ML is a critical area of research straddling machine learning [48], language [51], vision [18], and HCI [52]. We categorize related work in interpreting multimodal models into:

Unimodal importance: Several approaches have focused on building interpretable components for unimodal importance through soft [7] and hard attention mechanisms [53]. When aiming to explain black-box multimodal models, related work rely primarily on gradient-based visualizations [18]–[20] and feature attributions (e.g., LIME [13], Shapley values [12]) to highlight regions of the image which the model attends to.

Cross-modal interactions: Recent work investigates the activation patterns of pretrained transformers [54], [55], performs diagnostic experiments through specially curated inputs [56]–[59], or trains auxiliary explanation modules [7], [60]. Particularly related to our work is EMAP [14] for disentangling the effects of unimodal (additive) contributions from cross-modal interactions in multimodal tasks, as well as M2Lens [45], an interactive visual analytics system to visualize multimodal models for sentiment analysis through both unimodal and cross-modal contributions.

Multimodal representation and prediction: Existing approaches have used language syntax (e.g., the question in VQA) for compositionality into higher-level features [61]–[63]. Similarly, logical statements have been integrated with neural networks for interpretable logical reasoning [64], [65]. However, these are typically restricted to certain modalities or tasks. Finally, visualizations have also uncovered several biases in models and datasets (e.g., unimodal biases in VQA questions [66], [67] or gender biases in image captioning [8]). We believe that MultiVizwill enable the identification of biases across a wider range of modalities and tasks.

5 Conclusion↩︎

This paper proposes MultiVizfor analyzing and visualizing multimodal models. MultiVizscaffolds the interpretation problem into unimodal importance, cross-modal interactions, multimodal representations, and multimodal prediction, before providing existing and newly proposed analysis tools in each stage. MultiVizis designed to be modular (encompassing existing analysis tools and encouraging research towards understudied stages), general (supporting diverse modalities, models, and tasks), and human-in-the-loop (providing a visualization tool for human model interpretation, error analysis, and debugging), qualities which we strive to upkeep by ensuring its public access and regular updates from community feedback.

6 Ethics Statement↩︎

Multimodal data and models are ubiquitous in a range of real-world applications. MultiVizis our attempt at a standardized and modular framework for visualizing these multimodal models. While we believe these tools can help stakeholders gain a deeper understanding and trust of multimodal models as a step towards reliable real-world deployment, we believe that special care must be taken in the following regard to ensure that these tools are reliably interpreted:

1. Reliability of visualizations: There has been recent work examining the reliability of model interpretability methods for real-world practitioners [68], [69]. [70] examines the motivations underlying interest in interpretability, finding them to be diverse and occasionally discordant. [71] find that state-of-the-art explanation methods may disagree in terms of the explanations they output. [72] further conclude that existing explanations on VQA model do not actually make its responses and failures more predictable to a human. We refer the reader to [49] for a critique on the disconnect between technical objectives targeted by interpretable ML research and the high-level goals stated as consumers’ use cases, as well as [73] for an analysis of how interpretable and explainable ML tools can be used in real-world deployment. Human-in-the-loop interpretation and evaluation could be a promising direction towards connecting technical solutions with real-world stakeholders, while also offering users an interactive medium to incorporate feedback in multimodal models.

2. Pitfalls of gradient-based interpretation: We are aware of the limitations underlying gradient-based interpretation of black-box models [69], [70] with issues surrounding their faithfulness and usefulness. Future work should examine the opportunities and risks of gradient-based approaches, particularly in the context of cross-modal interactions.

3. The role of cross-modal interactions: There has been work showing that certain multimodal tasks do not need models to pick up cross-modal interactions to achieve good performance [14]. Indeed, for tasks like cross-modal retrieval, simply learning one interaction between a word and its corresponding image region is enough for typical datasets. This makes interpretation of cross-modal interactions difficult, since even well-performing models may not need to pick up all cross-modal interactions.

4. User studies: Based on direct communication with our institution’s IRB office, this line of user-study research is aligned with similar annotation studies at our institution that are exempt from IRB. The information obtained during our study is recorded in such a manner that the identity of the human subjects cannot readily be ascertained, directly or through identifiers linked to the subjects. We do not collect any identifiable information from annotators.

5. Usability: While we tried to be comprehensive in providing visualizations to the user, more information beyond a certain point is probably not useful and may overwhelm the user. We plan to work closely with HCI researchers to rethink usability and design of our proposed interpretation tools through careful user studies. MultiVizwill also welcome feedback from the public to improve its usability.

6. Beyond MultiVizstages: While we believe that many multimodal problems can benefit from breaking them down into our proposed interpretation stages, we also acknowledge that certain problems may not benefit from this perspective. For example, problems in multimodal translation (mapping from one modality to another, such as image captioning) will not involve prediction layers and instead require new stages to interpret the generation process, and problems in cross-modal transfer will also require new stages to interpret knowledge transfer. In Appendix 12, we include more details on new datasets we plan to add to MultiVizto enable the study of new multimodal interpretability problems, and other interpretation tools we plan to add.

7. Evaluating interpretability: Progress towards interpretability is challenging to evaluate [69], [74]–[77]. Model interpretability (1) is highly subjective across different population subgroups [71], [78], (2) requires high-dimensional model outputs as opposed to low-dimensional prediction objectives [7], and (3) has desiderata that change across research fields, populations, and time [79]. We plan to continuously expand MultiVizthrough community inputs for new interpretation methods in each stage and metrics to evaluate interpretability methods (see Appendix 12 for details). Some metrics we have in mind include those for measuring faithfulness, as proposed in recent work [69], [74]–[77], [80], [81].

7 Reproducibility Statement↩︎

1. Our code, datasets, and documentation are released at https://github.com/pliang279/MultiViz. This link also includes human-in-the-loop evaluation scripts and instructions on running MultiVizfor new datasets and models.

2. Details on the MultiVizvisualization approaches are provided in Appendix 8.

3. Details on the MultiVizwebsite, sample webpages with visualizations, code structure, and sample tutorials are provided in Appendix 9.

4. Dataset collection and preprocessing details are provided in Appendix 10. We provide documentation for MultiVizin the form of datasheets for datasets [82].

5. Experimental details, including all details on user studies and evaluation, are provided in Appendix 11.

Acknowledgements↩︎

This material is based upon work partially supported by the National Science Foundation (Awards #1722822 and #1750439), National Institutes of Health (Awards #R01MH125740, #R01MH096951, and #U01MH116925), Meta, and BMW of North America. PPL is partially supported by a Facebook PhD Fellowship and a Carnegie Mellon University’s Center for Machine Learning and Health Fellowship. RS is supported in part by ONR award N000141812861 and DSTA. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, National Institutes of Health, Facebook, Carnegie Mellon University’s Center for Machine Learning and Health, Office of Naval Research, or DSTA, and no official endorsement should be inferred. We are extremely grateful to Ben Eysenbach, Martin Ma, Chaitanya Ahuja, Volkan Cirik, Peter Wu, Amir Zadeh, Alex Wilf, Victoria Lin, Dong Won Lee, Torsten Wortwein, and Tiffany Min for helpful discussions and feedback on initial versions of this paper. Finally, we would also like to acknowledge NVIDIA’s GPU support.

Appendix↩︎

8 Analysis Details↩︎

8.1 Unimodal importance↩︎

Unimodal importance aims to understand the contributions of each modality towards modeling and prediction. It builds upon ideas of gradient-based visualizations (e.g., Gradient [18]–[20]) and feature attributions (e.g., LIME [13], [27], [28], Shapley values [12], [83], [84]).

Taking LIME [13] for an example, given model \(f\), we would like to return weights over each of the \(x_1\) and \(x_2\)’s such that important modalities are accurately weighted. LIME perturbs the set of \(x_1\) and \(x_2\)’s, observes how model predictions change, and fits a local linear model with respect to that datapoint. The areas with the highest positive weights are presented as the important ones. Other feature attribution and visualization approaches, such as Gradient-based [72], [85] or Shapley values [12], [83], [84], work similarly [28].

We implement unimodal feature attribution methods as a module \(\mathrm{\small Uni}(f_\theta, y, \mathbf{x})\) taking in a trained model \(f_\theta\), an output/feature \(y\) which analysis is performed with respect to, and the modality of interest \(\mathbf{x}\). \(\mathrm{\small Uni}\) returns importance weights across atoms \(x\) of modality \(\mathbf{x}\).

8.2 Cross-modal interactions↩︎

Cross-modal interactions describe ways in which atoms from different modalities can relate with each other and the types of new information possibly discovered as a result of these relationships. MultiVizincludes two recent methods for understanding cross-modal interactions:

EMAP [14] decomposes \(f(x_1,x_2) = g_1(x_1) + g_2(x_2) + g_{12}(x_1,x_2)\) into strictly unimodal representations \(g_1, g_2\), and cross-modal representation \(g_{12} = f - \mathbb{E}_{x_1} (f) - \mathbb{E}_{x_2} (f) + \mathbb{E}_{x_1,x_2} (f)\) to quantify the degree of global (across an entire dataset) cross-modal interactions captured by a model.

DIME [15] further extends EMAP by designing an efficient method for feature visualization on each disentangled representation locally (per datapoint).

Higher-order Gradient is our proposed method for efficiently quantifying the presence of cross-modal interactions. Based on the gradient definition of statistical non-additive interaction [21], [24], a function \(f\) exhibits non-additive interactions among \(2\) unimodal atoms \(x_1\) and \(x_2\) if \(\left[ \frac{\partial^2 f(x_1,x_2)}{\partial x_1 \partial x_2} \right]^2 > 0\). Writing the multimodal model \(f\) as \(f(x_1, x_2) = g_1(x_1) + g_2(x_2) + g_{12}(x_1,x_2)\), we can isolate the effect of \(g_{12}(x_1,x_2)\) by taking a second-order gradient of \(f\) with respect to \(x_1\) and \(x_2\) so the \(g_1(x_1)\) and \(g_2(x_2)\) terms becomes zero. Theoretically, second-order gradients are necessary and sufficient to recover cross-modal interactions: purely additive models will have strictly \(0\) second-order gradient information, and any non-linear interaction term \(g_{12}(x_1,x_2)\) must have strictly non-zero second-order gradient information.

Definition 2 inspires us to extend first-order gradient and perturbation-based approaches [13], [27], [28] to the second order. Our implementation first computes a gradient of \(f\) with respect to one input modality atom (e.g., \(x_1 = \textit{birds}\)), which results in a vector \(\nabla_1 = \frac{\partial f}{\partial x_1}\) of the same dimension as \(x_1\) (i.e., token embedding dimension). We aggregate the vector components of \(\nabla_1\) via summation to produce a single scalar \(\lVert \nabla_1 \rVert\), before taking a second-order gradient with respect to all atoms of the second modality \(x_2 \in \mathbf{x}_2\) (e.g., all image pixels), which results in a vector \(\nabla_{12} = \left[ \frac{\partial^2 f}{\partial x_1 \partial x_2^{(1)}}, ..., \frac{\partial^2 f}{\partial x_1 \partial x_2^{(|\mathbf{x}_2|)}} \right]\) of the same dimension as \(|\mathbf{x}_2|\) (i.e., total number of pixels). Each scalar entry in \(\nabla_{12}\) highlights atoms \(x_2\) that have non-linear interactions with the original atom \(x_1\) (e.g., only the birds in the image, see Figure 2 for examples on real datasets). We implement a general module \(\mathrm{\small CM}(f_\theta, y, x_1, \mathbf{x}_2)\) for cross-modal visualizations, taking in a trained model \(f_\theta\), an output/feature \(y\), the first modality’s atom of interest \(x_1\), and the entire second modality of interest \(\mathbf{x}_2\). \(\mathrm{\small CM}\) returns importance weights across atoms \(x_2\) of modality \(\mathbf{x}_2\), and can build on top of any first-order unimodal attribution method (i.e., gradient visualization [20], LIME [13], or Shapley values [12], see Appendix 8.2).

We plan to make several approximations: only estimating single instances \((x_1, x_2)\) at a time which avoids the expectation, and computing the magnitude \(w(x_1, x_2) = \left( \frac{\partial f(x)}{\partial x_1 \partial x_2} \right)^2\) as a measure of cross-modal interaction strength. Specifically, given a model \(f\), we first take a gradient of \(f\) with respect to an input word (e.g., \(x_1 = \textit{dog}\)), before taking a second-order gradient with respect to all input image pixels \(\mathbf{x}_2\), which should result in only the dog in the image being highlighted (see Figure 2 for examples on real datasets).

We implement a general module \(\mathrm{\small CM}(f_\theta, y, x_1, \mathbf{x}_2)\) for cross-modal visualizations, taking in a trained model \(f_\theta\), an output/feature \(y\), the first modality’s atom of interest \(x_1\), and the entire second modality of interest \(\mathbf{x}_2\). \(\mathrm{\small CM}\) returns importance weights across atoms \(x_2\) of modality \(\mathbf{x}_2\), and can build on top of any first-order unimodal attribution method, such as gradient visualization [6], LIME [13], or Shapley values [12].

8.3 Multimodal representations↩︎

Given these highlighted unimodal and cross-modal interactions at the input level, the next stage aims to understand how these interactions are represented at the feature representation level. Specifically, given a trained multimodal model \(f\), define the matrix \(M_z \in \mathbb{R}^{N \times d}\) as the penultimate layer of \(f\) representing (uninterpretable) deep feature representations implicitly containing information from both unimodal and cross-modal interactions. For the \(i\)th datapoint, \(z = M_z (i)\) collects a set of individual feature representations \(z_{1}, z_{2}, ..., z_{d} \in \mathbb{R}\). We aim to interpret these feature representations through both local and global analysis (see Figure 1 for an example):

Local representation analysis (\(\textrm{R}_\ell\)) informs the user on parts of the original datapoint that activate feature \(z_{j}\). To do so, we run unimodal and cross-modal visualization methods with respect to feature \(z_{j}\) (i.e., \(\mathrm{\small Uni}(f_\theta, z_{j}, \mathbf{x})\), \(\mathrm{\small CM}(f_\theta, z_{j}, x_1, \mathbf{x}_2)\)) in order to explain the input unimodal and cross-modal interactions represented in feature \(z_{j}\). Local analysis is useful in explaining model predictions on the original datapoint by studying the input regions activating feature \(z_{j}\).

Global representation analysis (\(\textrm{R}_g\)) provides the user with the top \(k\) datapoints \(\mathcal{D}_k(z_{j}) = \{ (\mathbf{x}_1, \mathbf{x}_2, y)_{i=(1)}^k \}\) that also maximally activate feature \(z_{j}\). By further unimodal and cross-modal visualizations on datapoints in \(\mathcal{D}_k(z_{j})\), global analysis is especially useful in helping humans assign interpretable language concepts to each feature by looking at similarly activated input regions across datapoints (e.g., the concept of color in Figure 1). Global analysis can also help to find related datapoints the model also struggles with for error analysis.

8.4 Multimodal prediction↩︎

Finally, the prediction step takes the set of feature representations \(z_{1}, z_{2}, ..., z_{d}\) and composes them to form higher-level abstract concepts suitable for a task. We approximate the prediction process with a linear combination of penultimate layer features by integrating a sparse linear prediction model with neural network features [29]. Given the penultimate layer \(M_z \in \mathbb{R}^{N \times d}\), we fit a linear model \(\mathbb{E}\left( Y|X=x \right) = M_z^\top \beta\) (bias \(\beta_0\) omitted for simplicity) and solve for sparsity using: \[\label{pibojfah} \hat{\beta} = \mathop{\mathrm{arg\,min}}_{\beta} \frac{1}{2N} \| M_z^\top \beta - y \|_2^2 + \lambda_1 \| \beta \|_1 + \lambda_2 \| \beta \|_2^2.\tag{2}\] The resulting understanding starts from the set of learned weights with the highest non-zero coefficients \(\beta_{\textrm{top}} = \{ \beta_{(1)}, \beta_{(2)}, ... \}\) and corresponding ranked features \(z_{\textrm{top}} = \{z_{(1)}, z_{(2)}, ...\}\). \(\beta_{\textrm{top}}\) tells the user how features \(z_{\textrm{top}}\) are composed to make a prediction, and \(z_{\textrm{top}}\) can then be visualized with respect to unimodal and cross-modal interactions using the representation stage.

8.5 Sanity checks for saliency maps↩︎

According to [48], a visualization/interpretation method should be rejected if it admits invariance over either data or model, i.e. transformation of data or model does not change the output of the method. We perform a similar sanity check on MultiViz:

Data randomization test: MultiVizdoes not admit data invariance, as MultiViz visualizations on the same model varies between different data points and labels (see visualization examples in Figure 4, 5, and 6). The visualizations reliably capture unique input regions, related datapoints, feature concepts, and errors specific to each data point.

Model randomization test: In Appendix 11.2, we demonstrated that MultiVizproduces different results for two different models on the same data for both CLEVR question answering and Flickr-30K retrieval: MultiVizenables us to explain differences in performance across 2 models based on the accuracy of cross-modal interactions each model captures, so MultiVizpasses the model randomization test.

Therefore, our methods do not admit data or model invariance and passes the sanity checks from [48].

9 MultiVizVisualization Tool↩︎

We summarize these proposed approaches for understanding each step of the multimodal process in Table 3, and show the overall pipeline in Algorithm 7 and Figure 1. To enable human studies, MultiVizprovides an interactive API where users can choose multimodal datasets and models and be presented with a set of visualizations at each stage.

In this section, we will include both introductions to our code framework that enables easy application of analysis visualization methods to datasets and models, and also present the MultiVizwebsite that showcases some examples of visualizations generated for each stage on different datasets and models.

9.1 MultiVizcode framework↩︎

One additional major contribution of our works is that we designed a code framework in Python for easy analysis, interpretation and visualization of models on multimodal datasets with only a few lines of code. The framework is modularized and extendable to new datasets, models and visualization methods. Figure 8 is an illustration of the main modules of the code framework:

• Within the datasets module, we include scripts for retrieving information directly from the dataset, including getting specific data points from a split, getting the ground truth labels, label-id-to-answer and answer-to-label-id mappings, etc. Some dataset scripts also supports generating visualizations for data points (for example, the script for VQA supports generating pictures that contain both the image and the question).

• Within the models module, we write a wrapper for every supported model that inherits a common parent class called analysismodel, which defines a set of functionalities commonly used in various analysis methods. The functions in analysismodel include forward (just making a prediction on a specific data point), forwardbatch (forward but on multiple points in a batch), getgrad (compute gradient, if applicable), getprelinear (getting representation features), and many others. This design allows the same analysis script to work on vastly different models, as long as the models are wrapped by a class that shares these functionalities.

• Within the analysis module, we have scripts that can take in arbitrary data point and a model class (that inherits analysismodel) and perform various analysis methods such as LIME, DIME, EMAP, Sparse Linear Model, etc. These scripts generate the outputs in numerical format without visualizations, and users can choose to visualize them in arbitrary ways.

• Within the visualizations module, we have scripts that provide tools to visualize the analysis results from the analysis module.

In Algorithm 9, we showcase an example of running LIME, DIME, Sparse Linear Model and representation feature analysis (local and global), thus covering all stages of MultiViz. As you can see, the code is actually very short (without the comments) for running this many analysis and visualizations. Our code framework is also easily extendible to support new datasets, models and analysis/visualization methods, by writing and adding scripts to the datasets/models/analysis/visualizations modules respectively.

9.2 The MultiVizwebsite↩︎

We also created a visualization website accompanying MultiVizwhich organizes visualizations of all stages on a particular datapoint of specific dataset-model pairs. The URL link of the webpage is available at https://github.com/pliang279/MultiViz.

Figure 10 is one example webpage for a data point in VQA. On the left there is a control panel that allows users to switch between different datasets and instances (i.e., data points), and then below the two boxes shows all information about the data point (image and question in the case of VQA) and also the ground truth ("GT") label and the predicted ("Pred") label. On the right side, we have a graph showing a simplified version of the Sparse Linear Model: we only show the top 5 features with the highest weights for each label (the weights are shown as numbers on the lines). Note that we will show both correct and predicted labels in the graph (so if the model got the answer wrong, there will be two labels shown under "classes" as shown in Figure 12, and clicking on each label will navigate to a webpage that shows visualizations with respect to that specific label). In the middle tab titled Main View, we show the visualizations from U and C stages. In the case of VQA we present unimodal LIME as U stage visualization (first column under Main View) and DIME as C stage visualization (second and third column under Main View). We call this webpage the Overview webpage. For each of the top five representation features shown within the graph, the user can access \(\textrm{R}_\ell\) and \(\textrm{R}_g\) visualizations of each feature by clicking on the circle in the graph representing that feature and the user will see a feature webpage like Figure 11. Under Main View, we include local analysis visualizations (unimodal lime with respect to the feature in the case of VQA) on the top and then global analysis visualizations on the bottom. To return to the Overview page, the user can just press the label circle under "classes" in the graph on the right again.

We also show additional example webpages: MM-IMDb (Figure 13 and Figure 14, with first order gradient for U stage, second order gradient for C stage), CMU-MOSEI (Figure 15 and Figure 16, with first order gradient for U stage, second order gradient for C stage) and MIMIC (Figure 17, with first order gradient for U stage). Note that we only ran U stage for MIMIC LF model because its cross-modal interactions are negligible (second order gradients are all zero) and there are too few representation features to do sparse linear models.

We have also used modified versions of these webpages to conduct all our experiments with human annotators. See Appendix 11 for details.

9.3 Intermediate layer representation visualization↩︎

Our codebase is designed such that the user may specify any layer in a model as the representation and run \(\boldsymbol{R}_\ell\) and \(\boldsymbol{R}_g\) analysis on neurons in that layer. We showcase a few examples of \(\boldsymbol{R}_\ell\) and \(\boldsymbol{R}_g\) on neurons on the third-last layer on LXMERT model on the VQA dataset in Figure 18.

The reason we choose to use the second last layer in the models is mostly for ease of visualization in the P stage as it will just be a linear composition. If we use a different layer as our representation, the P stage will contain multiple layers with different weights and more complex interactions, making it more difficult for a human user to visualize how each neuron in the representation related to the final prediction.

10 Datasets and Models↩︎

All of our datasets build upon a diverse and standardized set of multimodal benchmarks in MultiBench [30]. We briefly describe the datasets and preprocessing here:

10.1 Datasets in MultiViz↩︎

10.1.1 Multimodal fusion↩︎

In multimodal fusion, the main challenge is to join information from two or more modalities to perform a prediction. Classic examples include audio-visual speech recognition, where visual lip motion is fused with speech signals to predict spoken words [86]. Information coming from different modalities have varying predictive power by themselves and also when complemented by each other (i.e., higher-order interactions). In order to capture higher-order interactions, there is also a need to identify the relations between granular units from two or more different modalities (i.e., alignment). When dealing with temporal data, it also requires capturing possible long-range dependencies across time (i.e., temporal alignment). MultiVizcontains the following datasets for multimodal fusion spanning:

(1) CMU-MOSEI is the largest dataset of sentence-level sentiment analysis and emotion recognition in real-world online videos [31], [87] with more than \(65\) hours of annotated video from more than \(1,000\) speakers and \(250\) topics. Each video is annotated for sentiment as well as the presence of 9 discrete emotions (angry, excited, fear, sad, surprised, frustrated, happy, disappointed, and neutral) as well as continuous emotions (valence, arousal, and dominance). The diversity of prediction tasks makes CMU-MOSEI a valuable dataset to test multimodal models across a range of real-world affective computing tasks. The dataset has been continuously used in workshops and competitions revolving around human multimodal language.

Dataset preprocessing: We follow current work [31], [88] and apply standard preliminary feature extraction for the CMU-MOSEI dataset.

Train, validation, and test splits: Each dataset contains several videos, and each video is further split into short segments (roughly \(10-20\) seconds) that are annotated. We split the data at the level of videos so that segments from the same video will not appear across train, valid, and test splits. This enables us to train user-independent models instead of having a model potentially memorizing the average affective state of a user. There are a total of \(16,265\), \(1,869\), and \(4,643\) segments in train, valid, and test datasets respectively for a total of \(22,777\) data points.

(2) MM-IMDb is the largest publicly available multimodal dataset for genre prediction on movies [32]. MM-IMDb starts from the movies of the MovieLens \(20\)M dataset and expands this dataset by collecting genre, poster, and plot information for each movie. The final dataset contains ratings for \(25,959\) movies. MM-IMDb is a realistic real-world multimodal dataset and is a popular benchmark for multimodal learning [32], [89], [90].

Dataset preprocessing: We used the same method as [32] to extract features from texts and images.

Train, validation, and test splits: The MM-IMDb dataset is split by genre into train, valid, and test datasets containing \(15552\), \(2608\), and \(7799\). The split was performed so that training, valid and test sets comprise \(60\%\), \(10\%\), \(30\%\) samples of each genre respectively.

(3) MIMIC-III (Medical Information Mart for Intensive Care III) [33] is a large, freely-available database comprising de-identified health-related data associated with over \(40,000\) patients who stayed in critical care units of the Beth Israel Deaconess Medical Center between \(2001\) and \(2012\). Following [91], we organized numerous patient data into two major modalities (using the \(17\) features in feature set A in [91]): time series modality, which is a set of medical measurements of the patient taken every \(1\) hour in a period of \(24\) hours. Each measurement is a vector of size \(12\) (\(12\) different measured numerical values); static modality, which is a set of medical information about the patient, represented in a vector of size \(5\). We use these modalities for \(3\) tasks: mortality prediction (\(6\)-class prediction on whether the patient dies in \(1\) day, \(2\) day, \(3\) day, \(1\) week, \(1\) year, or longer than \(1\) year), and \(2\) ICD-\(9\) code predictions (binary classification on whether the patient fits any ICD-\(9\) code in group 1 (\(140-239\)) and binary classification on whether the patient fits any ICD-\(9\) code in group 7 \(460-519\)). MIMIC poses unique challenges in integrating time-varying and static modalities, reinforcing the need of aligning multimodal information at correct granularity.

Dataset preprocessing: We followed the instructions on https://mimic.physionet.org/gettingstarted/access/ to download the dataset in the form of raw tables, then generated preprocessed data following the steps described in https://github.com/USC-Melady/Benchmarking_DL_MIMICIII (which takes \(1-2\) weeks running time) to get the data used for experiments. Specifically, we will use data in the file 24hrs/series/imputed-normed-ep_1_24-stdized.npz. When accessing this data from our code repo, set the imputed_path of the npz file above in the get_data.py and the script will generate the PyTorch data loader for the tasks (where we will normalize the data).

Train, validation, and test splits: We split the data into train/valid/test sets randomly (using a fixed random seed) in a \(80:10:10\) ratio (so \(28,970\) train, \(3,621\) valid, and \(3,621\) test data points) for a total of \(36,212\) data points.

10.1.2 Multimodal retrieval↩︎

Another area of great interest lies in cross-modal retrieval [92], [93], where the goal is to retrieve semantically similar data from a new modality using a modality as a query (e.g., given a phrase, retrieve the closest image describing that phrase). The core challenge is to perform alignment of representations across both modalities. MultiVizcontains the following datasets for multimodal retrieval and grounding:

(1) Flickr-30k [34] contains \(32,000\) images collected from Flickr, together with \(5\) reference sentences provided by human annotators enabling the tasks of text-to-image reference resolution, localizing textual entity mentions in an image, and bidirectional image-caption retrieval.

Train, validation, and test splits: The training items are generated from the captions of \(25,000\) images, and the test items are generated from a disjoint set of \(3,000\) images.

10.1.3 Multimodal question answering↩︎

Within the domain of language and vision, there has been growing interest in language-based question answering (i.e., “query” modality) of entities in the visual, video, or embodied domain (i.e., “queried” modality). Datasets such as Visual Question Answering [94], Social IQ [95], and Embodied Question Answering [96] have been proposed to benchmark the performance of multimodal models in these settings. A core challenge lies in aligning words asked in the question with entities in the queried modalities, which typically take the form of visual entities in images or videos (i.e., alignment). MultiVizcontains the following datasets for multimodal question answering spanning several research areas:

(1) CLEVR [35] is a diagnostic dataset for studying the ability of VQA systems to perform visual reasoning. It contains \(100,000\) rendered images and about \(853,000\) unique automatically generated questions that test visual reasoning abilities such as counting, comparing, logical reasoning, and storing information in memory.

Train, validation, and test splits: The complete dataset contains more than \(608\)K train, \(140\)K val and \(140\)K test (question, image) pairs.

(2) VQA 2.0 [36] is a balanced version of the popular VQA [94] dataset by collecting complementary images such that every question is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question. The reduces the occurrence of spurious correlations in the dataset and enables training of more robust models.

Train, validation, and test splits: The complete balanced dataset contains more than \(443\)K train, \(214\)K val, and \(453\)K test (question, image) pairs.

11 Additional Experiments and Details↩︎

In this section, we provide additional details on the experiments and additional results on several other multimodal datasets.

Computational resources: Preparations for all experiments (i.e. generating the necessary visualizations for the points for each dataset) are done on a private server with 2 GPUs.

The preparation time for model simulation experiment using 2 GPUs is about 12 hours for VQA, 1 hour for MM-IMDb and 2 hours for CMU-MOSEI. For the representation interpretation experiment, we generated all visualizations for the VQA data points in the experiment in about 3 hours on 1 GPU. For the error analysis, in addition to the visualizations already present on the MultiViz website, we also have 1 GPU available live during the human annotation (so human annotators can request second order gradient analysis on specific words, each second order gradient computation only takes 2-3 seconds).

In all of the above analysis, we re-use the sparse linear model we had already trained for each dataset when building the main MultiViz webpage (the initial training can take some time - scaling the Sparse Linear Model to the large VQA took over 72 hours with 1 GPU).

Note that VQA visualization generation is much slower than those for MM-IMDb and CMU-MOSEI. This is because in VQA we used DIME for cross-modal interaction interpretation, but in MM-IMDb and CMU-MOSEI we use second-order gradient. The newly proposed second order gradient is much faster compared to DIME since it only requires running the model once instead of up to 10,000 times in DIME.

Overall, the proposed MultiVizinterpretation stages are efficient and only add negligible time on top of existing trained models, especially for our newly-proposed second-order gradient method.

Participant risks and compensation: Participation in these human studies were fully voluntary and without compensation. There are no participant risks involved. We obtained consent from all participants prior to each short study. All annotations are fully anonymous and we do not store any information regarding the participants at all.

11.1 Model simulation↩︎

11.1.1 Setup↩︎

We design a large-scale use case of model simulation to determine if MultiVizhelps users of multimodal models gain a deeper understanding of model behavior, as shown in Figure 19. We design a human study to see what humans predict given MultiVizexplanations at each step (and across all steps). If MultiVizindeed generates human-understandable explanations, humans should be able to make a prediction on the task given these explanations only. Specifically, we compare the full version of MultiVizwith a set of local ablations, each consisting of only 1 additional stage:

1. U: Users are only shown the unimodal importance (U) of each modality towards the prediction.

2. U + C: Users are shown both unimodal importance (U) and cross-modal interactions (C) highlighted towards the final prediction.

3. U + C + \(\textrm{R}_\ell\): Users are shown unimodal importance (U) and cross-modal interactions (C) of the given datapoint highlighted towards the final prediction, as well as local analysis (\(\textrm{R}_\ell\)) of unimodal and cross-modal interactions of top ranked feature representations \(z_{\textrm{top}} = \{z_{(1)}, z_{(2)}, ...\}\) with respect to that local datapoint.

4. U + C + \(\textrm{R}_\ell\) + \(\textrm{R}_g\): Users are additionally shown global analysis (\(\textrm{R}_g\)) through similar datapoints that also maximally activate those same top ranked feature representations.

5. MultiViz(U + C + \(\textrm{R}_\ell\) + \(\textrm{R}_g\) + P): This constitutes the entire MultiVizframework by including visualizations of the final prediction (P) stage: sorting all top ranked feature neurons \(z_{\textrm{top}} = \{z_{(1)}, z_{(2)}, ...\}\) with respect to their coefficients \(\beta_{\textrm{top}} = \{ \beta_{(1)}, \beta_{(2)}, ... \}\) and showing these coefficients to the user.

We ask human annotators (who all have or are currently working towards a B.S. in a STEM field and have at least basic knowledge of machine learning models) to predict the output of a model analysis results and visualizations. In each of the following datasets (VQA 2.0, MM-IMDb, CMU-MOSEI), we divide 15 total human annotators into 5 groups of 3, each group getting one of the five settings above, and then we compute average accuracy and inter-rater agreement within each group. The full results are shown in Table 2.

11.1.2 VQA 2.0↩︎

In this experiment, we will perform model simulation on VQA 2.0 dataset with pretrained LXMERT (https://huggingface.co/unc-nlp/lxmert-vqa-uncased). We randomly selected 22 points from the validation split of the VQA dataset under the following criterion: (1) it is not a yes/no question and (2) the answer to the question is not infrequent (i.e. it occurs at least 220 times over 220K+ validation points). For each of the point, we run MultiVizanalysis and visualization: for U stage we run LIME on each modality; for C stage we run DIME; for R\(_\ell\) we run LIME with respect to the representation feature on this data point; and for \(\textrm{R}_g\) we run LIME on each modality with respect to the representation feature on 3 examples that maximally activates the feature; and for P we show the top 5 representation features with the highest weights with respect to the predicted class in a Sparse Linear Model trained on the training set of VQA. The webpage for each datapoint is organized into Overview page (containing U and C) as well as five Features page (\(\textrm{R}_\ell\) and \(\textrm{R}_g\) for each of the top 5 representation features) as well as a "graph" on the right showing P. An example Overview page is shown in Figure 20 and an example Features page is shown in Figure 21. In settings (1)-(4), we will use versions of the webpage with certain stages removed (for example, Figure 22 is the webpage for setting (2), only showing U and C).

Within each of the five groups, on each of the 22 points, human annotators are asked to predict what the model (LXMERT) predicts given a website containing some or all of the stages of analysis visualizations (depending on the group’s setting). In addition, they are given an answer sheet (see Figure 23) where they are given 4 answer choices for each data point to predict with, and they have to select one of the choices they think LXMERT most likely predicted as the answer to each data point. Before each annotator starts, they are taught how to interpret each analysis visualization, and then the instructor goes over 2 points together with the annotators as examples and the annotators need to finish the remaining 20 points on their own. Only the remaining 20 points counts towards the data collected in the experiment. We then compute average accuracy and inter-rater agreement score (Krippendorff’s alpha) within each group. In addition, groups under settings (3), (4) and (5) are asked whether they found the Overview or Features page more helpful.

As shown in Table 2, in general, human annotators were able to better predict the model’s predictions when they were given more information, as the groups that got more information almost always end up with both higher average accuracy and higher inter-rater agreement. Moreover, annotators in settings (3), (4), (5) reported that they found Features page most helpful compared to Overview page 31.7%, 61.7% and 80.0% of the time respectively, therefore showing that \(\textrm{R}_g\) and \(P\) helps make representation analysis a lot more useful.

11.1.3 MM-IMDb↩︎

In this experiment, we perform model simulation on MM-IMDb dataset with the LRTF model from MultiBench [30]. We randomly selected 21 points from the test split of MM-IMDb dataset. The original MM-IMDb dataset is designed for multi-label classification, but for simplicity, we only take the label with the highest prediction probability from LRTF as the predicted class, and effectively treat it as a single-label classification task during analysis, visualization and model simulation experiment. For each of the points, we run MultiVizanalysis and visualization: for U stage we show first order gradient analysis on image and text; for C stage we perform second order gradient analysis on the top ten words with maximum first order gradient; for \(\textrm{R}_\ell\) we show first order gradient on image and text with respect to each representation feature; for \(\textrm{R}_g\), on each representation feature we present 3 data points that maximally activates the feature, and also show first order gradient visualization for each; for \(P\) stage we show the "graph" on the right that ranks the top 5 representation features from Sparse Linear Model analysis as well as their respective weights. The webpage organization is the same as the webpage for VQA with the Overview page (Figure 24) and Features pages (Figure 25).

Within each of the five groups, on each of the 21 points, human annotators are asked to predict what the model (LRTF) predicts given a website containing some or all of the stages of analysis visualizations (depending on the group’s setting). In addition, we give human annotators 10 possible movie classes that the model could predict for these 21 points ("Drama/Romance", "Crime", "Sci-Fi", "Comedy", "Thriller", "Western", "Action", "War", "Documentary", "Horror"). Note that in reality, some of these categories are not mutually exclusive, but we intentionally designed our experiment this way to see if human annotators were able to determine the model’s prediction by looking at what specific properties within the movie’s poster or description the model focused on during the prediction process. Before each human annotator starts, they are taught how to interpret each analysis visualization, and then the instructor goes over the first point together with the annotator as example and the annotator need to finish the remaining 20 points on their own. Only the remaining 20 points counts towards the data collected in the experiment. We then compute average accuracy and inter-rater agreement score (Krippendorff’s alpha) within each group.

As shown in Table 2, in general, human annotators were able to better predict the model’s predictions when they were given more information, as the groups that got more information almost always end up with both higher average accuracy as well as higher inter-rater agreement. We were especially surprised to find that including \(\boldsymbol{C}\) stage actually helped, since MM-IMDb did not seem to be a task that relies much on cross-modal interaction.

11.1.4 CMU-MOSEI↩︎

In this experiment, we perform model simulation on CMU-MOSEI dataset with the MulT model from MultiBench [30]. We randomly selected 20 points from the test split of CMU-MOSEI dataset. The original CMU-MOSEI dataset is designed for a 7-way sentiment classification (-3 to +3), but we follow the preprocessing in MultiBench and convert it into a binary classification problem (where -1, -2, -3 are "Negative" and 0,1,2,3 are "Positive"). For each of the points, we run MultiVizanalysis and visualization: for U stage we show first order gradient analysis on image, audio and text (for image and audio, we compute gradient on each feature on each timestep, resulting in a 2d heatmap, while for text we just have a 1d heatmap), and we also show a processed video where we add bounding boxes around the visual features the model picked up (such as facial landmarks, facial expressions, lip movements, eye gaze, etc); for C stage we perform second order gradient analysis with selected words on image and audio; for \(\textrm{R}_\ell\) we show first order gradient on image, audio and text with respect to each representation feature; for \(\textrm{R}_g\), on each representation feature we present 3 data points that maximally activates the feature, and also show first order gradient visualization for each; for \(P\) stage we show the "graph" on the right that ranks the top 5 representation features from Sparse Linear Model analysis as well as their respective weights. The webpage organization is the same as the webpage for VQA with the Overview page (Figure 26) and Features pages (Figure 27).

Within each of the five groups, on each of the 20 points, human annotators are asked to predict what the model (MulT) predicts given a website containing some or all of the stages of analysis visualizations (depending on the group’s setting). Before each human annotator starts, they are taught how to interpret each analysis visualization, and the annotator needs to finish the 20 points on their own. We then compute average accuracy and inter-rater agreement score (Krippendorff’s alpha) within each group.

As shown in Table 2, in general, human annotators were able to better predict the model’s predictions when they were given more information, as the groups that got more information almost always end up with both higher average accuracy and higher inter-rater agreement. Moreover, human annotators were able to get perfect accuracy and agreement in settings (4) and (5), showing that including global analysis \(\textrm{R}_g\) provides enough information to simulate model predictions.

11.2 Cross-modal interactions↩︎

In order to verify local faithfulness of interpreting cross-modal interactions, we take a closer look at the qualitative and quantitative performance of our proposed second-order gradient method.

11.2.1 CLEVR↩︎

One gold standard for evaluating visualizations of cross-modal interactions involves using CLEVR [35] (for image question answering), because in this dataset we are given ground truth bounding boxes of each object and there are often cross-modal alignments that are obvious and without any controversy. We picked two representative models: MDETR [41], which is near-perfect (with 99.5% accuracy); and CNN+LSTM+SA [35], which was the best model amongst the baselines included in the paper that introduced CLEVR dataset [35]. We randomly selected 52 ground-truth alignment pairs, all of which aligns between a phrase in the question (1-4 words) and the one single object in the image. Then, for each pair, we compute the first-order gradient of each word with respect to the sum of all entries in the prediction logit vector, sum up the absolute gradients of the words in the phrase, before taking the gradient of each pixel with respect to the sum. We end up with a second-order gradient (SOG) on each pixel. Then, we then compute the average absolute SOG per pixel within bounding boxes of each object (given by CLEVR). We compute 2 metrics: alignment picked up by top 1 bounding box (how often does the aligned object match with the bounding box with the highest average SOG) and alignment picked up by top 2 bounding box (how often does the aligned object match with one of the bounding boxes with top 2 highest average SOG).

We show the results in Table 4. We found that under near-perfect setting (MDETR) where it is safe to assume that the model actually picks up all ground-truth alignments, our method was able to pick up over 80% of the alignments using top-2 bounding boxes, thus indicating that our method is quite faithful to the model’s actual prediction process. Moreover, we found that CNN+LSTM+SA, which is a relatively simple late fusion model with relatively poor performance, was much less likely to pick up the correct alignments according to our method, which makes sense. Below, we show examples of when the model picks up or is unable to pick up the ground-truth alignments in Figure 28.

11.2.2 Flickr-30k↩︎

In addition, we perform a similar experiment for Flickr-30k image-text retrieval by modifying the above approach slightly. We select 20 image-text pairs from the annotated dataset, and for each of them we take between 8-15 phrases to find the second-order-gradient (SOG) on each pixel. We take the ground-truth boxes from Flickr30k Entities [34] and calculate the average SOG for a given object per pixel across all the available boxes for the object. Additionally, we match the phrase against ground-truth phrase annotations to find relevant boxes. Finally, we calculate what percentage of the objects were recovered by double gradient from the ground-truth annotations, if any. For ViLT [39] model, we observe that second-order gradient is able to do so with 44% matching accuracy, as compared to 34% using random matching (see some examples of detected interactions in Figure 29). For CLIP, the matching performance is worse (35%) as the gradients are very scattered across examples, making it hard to localize one particular object (see some examples of detected interactions in Figure 30). Both these findings indicate potential future directions towards quantifying intermediate cross-modal interactions learned by a model beyond looking at final task performance.

11.3 Representation interpretation↩︎

We now take a deeper look to check that MultiVizgenerates accurate explanations of multimodal representations. Using local and global representation visualizations, can humans consistently assign interpretable concepts in natural language to previously uninterpretable features?

11.3.1 VQA 2.0↩︎

For VQA 2.0 dataset, we perform a representation interpretation experiment, where we give human annotators some visualizations on a particular representation feature and ask them to describe what concept they think that feature represents. We found 15 human annotators (with same qualifications as those in model simulation experiment), and divide them into 3 groups of 5. Each group is given a different setting (with different amounts of MultiVizvisualizations available):

1. \(\textrm{R}_\ell\): \(\textrm{R}_\ell\) only, i.e. one random example and Unimodal LIME explanation on the example with respect to this example. See Figure 31 for example.

2. \(\textrm{R}_\ell\) + \(\textrm{R}_g\) (no viz): In addition to \(\textrm{R}_\ell\) with LIME, we also provide \(\textrm{R}_g\) (top 3 examples that maximizes the feature’s value and top 3 examples that minimizes the feature’s value), but no LIME visualizations for \(\textrm{R}_g\). See Figure 32 for example.

3. \(\textrm{R}_\ell\) + \(\textrm{R}_g\): Same as setting 2, but we also provide Unimodal LIME visualizations for all examples in \(\textrm{R}_g\). See Figure 33 for example.

We gave the same 13 representation features to all 15 human annotators, where the first feature serves as an example and the other 12 are the ones we actually record for the experiment. The instructor first explains to each annotator what each visualization means, and then goes over the first feature together. Then, the annotator must write down a concept for the other 12 features on their own. We also ask each annotator to rate a confidence of 1-5 on how confident they are that this feature indeed represents this concept.

Once we have collected all 180 annotations (15 annotators each on 12 features), we manually cluster these into 29 distinct concepts that we show in Figure 34. For example, annotations like "things to wear", "t-shirts" and "clothes" all belong to "clothes" concept; all color-related annotations belong to "colors" concept; "material question", "made-of question" and "material of object" all belongs to "material" concept. We then compute inter-rater agreement score on each feature within each group of 5 annotators using Krippendorff’s alpha with 29 possible categories. We report both inter-rater agreement and average confidence in Table ¿tbl:tab:local95rep?.

As shown in Table ¿tbl:tab:local95rep?, as we give annotators more information, they were able to assign concepts more consistently (higher inter-rater agreement) and more confidently (higher average confidence score). Under setting 3 with full MultiVizvisualizations on feature representations, the 5 annotators completely agreed with each other on 7 out of 12 features, which is really impressive since there are so many possible concepts annotators could assign to each feature. Therefore, this shows that our visualizations, i.e. \(\textrm{R}_\ell\) and \(\textrm{R}_g\), really helps humans to better understand what concept (if any) that each feature in representation represents, and that \(\textrm{R}_g\) examples and visualizations are especially helpful.

A concrete example: In Figure 35, we show a concrete example of human annotators using MultiVizto assign concepts to feature representations in multimodal models trained on VQA 2.0. We show the information provided to users in each of the \(3\) ablation cases as part of the experiment, along with the actual user annotations from the user study:

1. In \(\textrm{R}_\ell\), we only provide the original seed datapoint and show visualizations of unimodal and cross-modal interactions with respect to a feature \(z\) for that datapoint. Using just local information, annotators struggle to identify the concept captured by the feature \(z\), with disagreement between ‘mirror’, ‘brushing teeth’, ‘bathroom’, ‘material’, and ‘none’, each with relatively lower confidence. Indeed, any of the concepts are present in the image and question, which makes it hard to choose a precise one.

2. In \(\textrm{R}_\ell\) + \(\textrm{R}_g\) (no viz), we provide both the original seed datapoint (local analysis), along with \(2\) similar datapoints that also maximally activate the feature \(z\) (global analysis), for \(3\) datapoints in total. Using both local and global information, users are better able to identify the commonalities between all \(3\) datapoints which all active feature \(z\), leading to \(3/5\) users identifying the concept as ‘asking about material’. However, the remaining \(2\) users answered ‘household objects/components’, which is another valid concept shared across those datapoints.

3. In \(\textrm{R}_\ell\) + \(\textrm{R}_g\), we show both local and global analysis (so \(3\) datapoints in total), in addition to the visualizations of unimodal and cross-modal interactions with respect to a feature \(z\) for all datapoints. With all pieces of information, all \(5/5\) users identified the concept as ‘asking about material’. Providing visualizations helps to resolve ambiguity in feature interpretation - the text importance identifies words like ‘counter’, ‘countertop’, and ‘wall’, along with the image crossmodal interactions highlighting these entities, which leads to high agreement and confidence among annotators in identifying the ‘material’ concept.

11.3.2 CLEVR↩︎

A few examples of interpreted representations are shown in Figure 36, in addition to the examples in Figure 4 of the main paper.

11.3.3 MM-IMDb↩︎

A few examples of interpreted representations are shown in Figure 37, in addition to the examples in Figure 4 of the main paper.

11.4 Error analysis↩︎

In this section, we conduct an experiment to see if human annotators will be able to categorize the reasons why the model fails to predict the correct answer.

11.4.1 Setup↩︎

We present three categories of errors:

1. Unimodal perception error: The model fails to recognize certain unimodal features or aspects. (For example, in Figure 5 top left example, the FRCNN object detector was unable to recognize the thin red streak as an object).

2. Cross-modal interaction error: The model fails to capture important cross-modal interactions such as aligning words in question with relevant parts or detected objects in image. (For example, in Figure 5 first one in middle column, the model is erroneously aligning "creamy" with the piece of carrot).

3. Prediction errors: The model is able to perceive correct unimodal features and their cross-modal interactions, but fails to reason through them to produce the correct prediction. (For example, in Figure 5 top right example, the model was able to both perfectly identify the chair with object detector and associate it with the word "chair" in the question (as shown by second-order-gradient analysis), but the model was still unable to reason with the given information correctly to predict the correct answer).

For each of the 2 datasets we used in this experiment (VQA and CLEVR), we found 10 human annotators and divide them into 2 groups of 5, one group for each setting: (1) under MultiVizsetting, for each data point, the human annotator is given access to full MultiVizwebpage as well as live Second-Order Gradient (i.e. the human annotator may request to compute second order gradient for a specific subset of words in the question, and he will be presented with the resulting second order gradient result); (2) under No Viz setting, the human annotator is given nothing but the original data point, the correct answer and the predicted answer. Each human annotator needs to classify each point into one of the three categories above, and they are also asked to rate their confidence in categorizing the error on a scale of 1-5.

11.4.2 VQA 2.0↩︎

In this experiment, we perform error analysis on VQA 2.0 with LXMERT. We first randomly selected 24 data points which the model got wrong, and then we ask 10 human annotators to categorize each point into one of the 3 categories above (5 annotators under MultiVizsetting and 5 annotators under No Viz setting). The webpage that the human annotators under MultiVizsetting sees is the same as the ones described in Appendix 9. In addition, since the LXMERT prediction pipeline is differentiable with respect to the detected objects by FRCNN object detector but not with respect to each pixel in the original image, the human annotators under MultiVizsetting will also be given all the bounding boxes of objects detected by FRCNN and also which ones have the highest second order gradient with respect to the specific words they picked. See Figure 38 for an example of all bounding boxes detected by FRCNN as well as second-order gradient analysis results for LXMERT.

During the experiment, the instructor first informs the annotators what each of the 3 categories of errors mean, and then explains each part of the visualizations they are given (if under MultiVizsetting). Then the instructor goes over the first data point together with the human annotators, and the human annotators must categorize the remaining 23 points on their own, and only those 23 points’ annotations will count towards the final result.

The result for VQA error analysis experiment is shown in Table ¿tbl:tab:local95rep?. As shown in the table, on average the human annotators are much more confident in categorizing each error, and also tend to agree with each other a lot more often when given MultiVizcompared to No Viz. This shows that MultiVizcan indeed help humans identify types of errors within a multimodal model. In addition, human annotators from the MultiVizsetting report that they can tell whether a model is able to perceive unimodal information correctly via U stage analysis as well as the bounding boxes produced by FRCNN, and they found second order gradient requested on specific words most helpful among all C stage visualizations (such as DIME) when determining if the model was able to find the correct cross-modal interactions. The data point presented in Figure 38 is one good example of this.

Error breakdown: Out of the 23 total errors, human annotators reported that on average 8.8 of them are category 1 (unimodal perception error), 6.8 of them are category 2 (cross modal interaction error), and 7.4 of them are category 3 (prediction error). This suggests that the majority of errors present in LXMERT is still caused by misunderstanding the basic unimodal concepts and cross-modal alignments rather than high-level reasoning of the perceived information, and that one possible future direction for improving the model pipeline is to use better unimodal encoders (than FRCNN) and find out some way to force the model to learn to align visual and text concepts correctly.

A concrete example: In Figure 39, we show a concrete example of human annotators using MultiVizto perform error analysis on incorrect predictions made by trained models, specifically into one of 3 stages: failures in (1) unimodal perception, (2) capturing cross-modal interaction, and (3) prediction with perceived unimodal and cross-modal information. We show the information provided to users in each of the 2 ablation cases, along with the actual user annotations from the user study:

1. No Viz does not provide the user with any information. Note that there are no intermediate stages we can ablate, since errors can occur at all stages, so removing any stage from MultiViz by definition cripples its ability to detect errors at that stage. However, users still use their intuition to make a most educated guess on which stage the model is likely to make an error in. For example, if some odd object seems hard to detect, users tend to guess unimodal error, and if the prediction involves complex reasoning that is hard even for humans, users tend to guess prediction error.

2. MultiViz provides the user with the unimodal importance and cross-modal interactions visualized for that incorrectly predicted datapoint. In the top example, users can tell that the unimodal importance on ‘cheese’ and ‘pizza’ are correct, along with the right image-text interaction highlighting the bounding pizza around pizza. Hence, it is a prediction error, which all users agree on. In the bottom example, users can see that while ‘man’ and ‘jeans’ are unimodally highlighted correctly, none of the image-text interactions highlight the bounding box around the man’s jeans, so they agree on a cross-modal interaction error.

11.4.3 CLEVR↩︎

In this experiment, we perform error analysis on CLEVR with CNN+LSTM+SA model. We first randomly selected 11 data points which the model got wrong, and then we ask 10 human annotators to categorize each point into one of the 3 categories above (5 annotators under MultiVizsetting and 5 annotators under No Viz setting). The webpage that the human annotators under MultiVizsetting sees is the same as the ones described in Appendix 9. In addition, the human annotators under MultiVizsetting can request the second-order gradient analysis result on specific words or phrases they pick, both the pixel-wise heatmap and top 2 bounding boxes with the highest average absolute gradient per pixel (same procedure as described in Appendix 11.2). See the bottom half of Figure 28 for an example of second-order gradient analysis result of CNN+LSTM+SA.

During the experiment, the instructor first informs the annotators what each of the 3 categories of errors mean, and then explains each part of the visualizations they are given (if under MultiVizsetting). Then the instructor goes over the first data point together with the human annotators, and the human annotators must categorize the remaining 10 points on their own, and only those 10 points’ annotations will count towards the final result.

The result for CLEVR error analysis experiment is shown in Table ¿tbl:tab:local95rep?. As shown in the table, on average the human annotators are much more confident in categorizing each error, and also tend to agree with each other a lot more often when given MultiVizcompared to No Viz. This shows that MultiVizcan indeed help humans identify types of errors within a multimodal model.

Error breakdown: Out of the 10 total errors, human annotators on average reported 6 of them belonging to category 2 (cross modal interaction error). This suggests that the major weakness of CNN+LSTM+SA is that it is not great at aligning phrases in text with the object the phrase refers to. This is expected because CNN+LSTM+SA is a late fusion model, which is known to be not great at capturing low-level cross-modal interactions.

11.5 Model debugging↩︎

11.5.1 VQA 2.0↩︎

Following error analysis, we take a deeper investigation into one of the errors on a pretrained LXMERT [42] model fine-tuned on VQA 2.0 [36].

We compute the penultimate features (the input to the last linear layer in the classification head) of the V set, and train a linear model that best maps the absolute values of these penultimate features to a binary label where 0 means the original LXMERT model got this point right and 1 means the original LXMERT model got this point wrong. Then, we pick the top 5 dimensions in the penultimate feature with the highest positive weight in the linear model, and task human annotators to inspect these neurons carefully through MultiVizlocal and global representation analysis. Human annotators found that \(2\) of the \(5\) neurons were consistently related to questions asking about color, which highlighted the model’s failure to identify color correctly (especially blue). The model has an accuracy of only \(5.5\%\) amongst all blue-related points (i.e., either have blue as correct answer or predicted answer), and these failures account for \(8.8\%\) of all model errors. We show examples of such datapoints and their MultiVizvisualizations in Figure 6. Observe that the model is often able to capture unimodal and cross-modal interactions perfectly, but fails to identify color at the prediction stage.

In this section, we describe our initial attempt at fixing this color-related bug by adding targeted data in an active learning scenario. If MultiVizindeed provides accurate insights for model debugging, we should be able to improve model performance using less data as compared to a control experiment that adds randomly sampled data (see Figure 40).

We first split the validation set (about 220K points) into 3 parts: the first 110K were called the V set (stands for "val"), the next 50K were called the U set (stands for "unlabeled"), and the last 60K were called the T test (stands for "test"). We are simulating a situation where in addition to the 450K training set, we have a labeled 110K validation set (V set), another 50K unlabeled points (U set), and 60K held-out test set (T set). Our goal is to debug or improve the given model (LXMERT) by selecting N points from U set to label and finetune the model with these N points.

We compare the following settings:

1. Random: We randomly sample \(N\) points from the U set.

2. Uncertainty: A common active learning baseline which selects the top \(N\) datapoints from the U set that the model is uncertain about based on the entropy of its predicted label distribution [47], [97], [98].

3. MultiViz color: For each of these 2 erroneous features, we picked \(\frac{N}{2}\) points from the U set that has the highest absolute values on the feature, and together these points form the N points related to color that we select from the U set. Note that we do not use label information about these additional datapoints.

4. MultiVizno color: Same as above, but we use 2 features that do not represent color.

5. MultiViz color: Same as above, but we use 1 feature that represents color and 1 that does not represent color.

Under each of these active learning settings, we finetune the last layer of LXMERT with the N selected points from U set for one epoch (batch size 32, learning rate tuned to the best performance), and the result is evaluated on the T set. In addition, since through MultiVizanalysis we found out that LXMERT is particularly bad on data points that either have ground truth correct answer "blue" or the original LXMERT predicts as "blue", we define a subset of T set we call "bluelist" that contains all 1729 points in the T set that either have ground truth correct answer "blue" or the original LXMERT predicts as "blue". The original LXMERT only has a 6% accuracy on bluelist. We try each setting 10 times (with different random seeds) and report average and standard deviation on improvement in accuracies on both the entire T set and bluelist over the original LXMERT.

We show these results in Table 5 and find that MultiVizsignificantly improves upon either random or uncertainty-based sampling as measured by performance on the overall VQA 2.0 test set. To obtain a deeper look at performance, we further evaluate performance on a targeted test set only containing questions asking about color (reflecting the main bug we found in the model). On this targeted test set, MultiVizsignificantly improves performance by \(30\%\) as compared to only \(1.4\%\) for random sampling. Using more features related to color also improved performance: \(27\%\) with 1 feature and \(30\%\) with both features. Surprisingly, we find uncertainty sampling had no effect (\(0.0\%\)) since the model predicted these incorrect answers on color-related questions with high certainty, so none of these color-related questions were additionally introduced to the model.

11.6 Summary of takeaway messages↩︎

From these results, we emphasize the main take-away messages:

1. From the model simulation experiment, we found that on all 3 settings of datasets and models, human annotators were able to get higher accuracy and better agreement when given strictly more stages of visualization from MultiViz. This suggests that each stage in MultiVizis complementary to helping humans better understand the models’ decision-making process.

2. Through a deeper inspection of cross-modal interaction visualization, we showed that second-order gradient is faithful to what the model internally aligns most of the time (over \(80\%\) using top 2 alignment accuracy on MDETR).

3. From the representation interpretation experiment, we found that having both local and global representation visualizations helps human annotators assign interpretable concepts in natural language to deep features with higher confidence and agreement.

4. From the error analysis experiment, we showed that MultiVizcan help users locate the stage of model that caused the error when the model makes a mistake, which provides insights for model error analysis and debugging.

5. Finally, we showcase a real-world model debugging case study: using each stage of MultiVizto localize the error, we were able to locate a real bug in the HuggingFace Transformers LXMERT library.

12 Limitations and Future Directions↩︎

12.1 Limitations↩︎

We are aware of some directions in which MultiVizcan still be improved and outline these for future work:

1. Large number of prediction classes: for complex tasks like VQA 2.0 where there are over three thousand prediction classes, a lot of rarely used answer choices will get "sparsed out" in the Sparse Linear Model analysis (since setting their weights to 0 barely affects overall accuracy), which makes it difficult to find related datapoints for local and global representation analysis. For example, Sparse Linear Model Analysis on LXMERT have zero weight from all representation neurons to the rare answer choice "abstract", so the five most important feature neurons are completely randomly selected.

2. Too few prediction classes: for VQA 2.0 subsets with ‘yes/no’ answer choices, we found that the final-layer activated features contain too much overlap to reliably visualize, and we have to extend MultiViz to rely on more intermediate-layer features. We added this experiment in Appendix 9.3. Overall, MultiViz(like general ML models), work best with a reasonable number of prediction classes, such as those in multimodal emotion recognition, standard multiple-choice multimodal question answering, and others.

3. Model requirements: Currently the two requirement of models is that they have categorical outputs (classification) and we can easily compute gradients via AutoGrad. The classification requirement is so that we can visualize given specific model outputs (e.g., word answers, emotion categories, video categories). For regression, we can extend MultiViz via discretizing the output space into categorical outputs. The second requirement enables us to perform first and higher-order gradient analysis, which means that we cannot currently support some neuro-symbolic multimodal architectures that have discrete steps (e.g., parsing and executing the question as a program [99]) in the middle of the model that prevents gradient flow. We plan to extend MultiViz via approximate gradients such as perturbation or policy gradients to handle these cases.

4. Visualization testing: We spent a lot of time into finding and training users. We carefully found users (who are not the authors and are not part of the same research groups) that have or are working towards a graduate degree in a STEM field and have knowledge of ML models. We showed them a training video describing how MultiViz can be used before each study session (see Appendix D for all experiment and user study details). Consequently, our user studies span over 60 hours of human testing on close to 100 total datapoints, which has enabled us to draw preliminary conclusions regarding the efficacy of multiple proposed stages towards model understanding and debugging. Future work can explore more standardized ways of human-in-the-loop interpretation and debugging of multimodal models, and we hope that MultiVizcan provide the initial data, models, tools, and evaluation as a step in this direction.

We plan to ensure the continual availability, maintenance, and expansion of MultiViz. Several immediate directions include new interpretation algorithms and holistic evaluation of interpretation methods.

12.2 New classes of interpretation tools↩︎

MultiVizis designed to be modular and support interpretation tools at each stage. While we have explored some directions, we plan to include the following methods in future work:

12.2.1 Cross-modal interactions↩︎

There have been several attempts at building multimodal models that are interpretable by design, with a particular focus on cross-modal interactions. Many of these involve parameterizing cross-modal interactions through attention models [11], [100] or graph-based models [31], [88]. As a result, there have been several approaches to study these specific types of cross-modal interactions, such as M2Lens [45], an interactive visual analytics system to visualize multimodal models for sentiment analysis through both unimodal and cross-modal contributions, and VL-InterpreT [44], an interactive visualization tool for interpreting vision-language transformers. We plan to include these in MultiVizto compare black-box post-hoc interpretation versus models interpretable by design.

12.2.2 Multimodal prediction↩︎

Beyond linear prediction, we also plan to investigate integrating neural networks with decision trees [101] to generalize linear reasoning into one based on compositionality defined by a decision tree, or other hierarchical prediction processes [62], [102]).

12.3 Evaluating interpretability↩︎

Progress towards interpretability is challenging to evaluate [69], [74]–[77]. Model interpretability (1) is highly subjective across different population subgroups [71], [78], (2) requires high-dimensional model outputs as opposed to low-dimensional prediction objectives [7], and (3) has desiderata that change across research fields, populations, and time [79]. We plan to continuously expand MultiVizthrough community inputs for new metrics to evaluate interpretability methods. Some metrics we have in mind include those for measuring faithfulness, as proposed in recent work [69], [74]–[77], [80].

12.4 Engagement with real-world stakeholders↩︎

Finally, we have plans for engagement with real-world stakeholders to evaluate the usefulness of these multimodal interpretation tools. We plan to engage these stakeholders in the healthcare domain to evaluate interpretability on the MIMIC dataset and those in the affective computing domain to evaluate interpretability on the CMU-MOSEI dataset. We also refer the reader to recent work examining the issues surrounding real-world deployment of interpretable machine learning [49], [71], [73].

References↩︎

---

1. we used the popular Hugging Face implementation at https://huggingface.co/unc-nlp/lxmert-vqa-uncased↩︎