1 Introduction↩︎

Affective Computing has emerged as a multidisciplinary field that integrates computer science, psychology, and machine learning to enable emotion-aware intelligent systems [1]. The integration of emotion recognition has become essential for modern applications, from personalized recommendation systems to digital healthcare [2], [3].

Among the existing approaches, continuous dimensional models have become the dominant paradigm. In particular, the two-dimensional valence–arousal space has emerged as a reliable and experimentally validated framework for the quantitative analysis and representation of affective states [4]. Its modality-independent nature makes it particularly well-suited to modeling emotions across heterogeneous media.

Despite these advances, existing approaches remain largely modality-specific, limiting their ability to compare or integrate emotional representations across different data types. This poses a fundamental challenge to building universal, emotion-aware systems capable of understanding and aligning affective signals in a unified way.

To address this limitation, we propose the Cross-Domain Multimodal Emotion Descriptor, CD-MED. As shown in Fig. 1, heterogeneous digital objects may contain different modalities and require different modality-specific emotion recognition models. The proposed CD-MED maps these outputs into a common emotional space, enabling emotion-based visual comparison across domains. CD-MED is an interpretable visual representation of multimodal emotional information, in which emotion category, intensity, affective position, and source modality are encoded by color, size, position, and shape, respectively. This representation allows users to understand complex multimodal emotional profiles without directly interpreting raw model outputs.

The main contributions of this study are as follows:

• We introduce the Cross-Domain Multimodal Emotion Descriptor, CD-MED, an interpretable visualization of multimodal emotional information, enabling users to understand emotion distributions across different channels through a unified visual descriptor.

• We propose a modality-agnostic comparison metric to measure the similarity between any two CD-MED profiles, even when they contain a different number of active modalities.

• We illustrate the effectiveness of CD-MED through a case study with various multimedia objects.

The structure of the paper is as follows. Section I is an introduction. Section II reviews related work on affective computing and emotion representation. Section III presents the proposed methodology. Section IV demonstrates the applicability of the proposed framework through a multimedia case study comparing movies and music. Finally, Section V concludes the paper and outlines directions for future research.

2 Related Work↩︎

Research in affective computing has evolved over the last few decades to enable machines to recognize, interpret, and respond to human emotions. The term Affective Computing was first introduced by Picard [1] to describe emotion-aware computational systems. Since then, emotion analysis has become an important research area in a number of domains including computer vision, natural language processing, speech analysis, and multimedia understanding.

Emotion representation is commonly based on two main paradigms: categorical and dimensional models [5]. Categorical models describe emotions as a finite set of classes, such as happiness, sadness, fear, anger, etc. [6]. These models are intuitive and easy to interpret. However, they may not capture mixed or gradual emotional states. Dimensional models, such as Russell’s circumplex model [4] and the valence–arousal space, represent emotions as continuous affective coordinates [7]. Further extensions added the third dimension of dominance to give the Pleasure-Arousal-Dominance (PAD) model [8]. Such models are more flexible for comparing emotions across text, speech, faces, music, video, brain signals, and multimodal input because they provide a shared space for affective representation [9], [10].

Emotion recognition has been widely studied in individual modalities, including text, speech, music, images, and video. Text-based approaches usually rely on lexicons or transformer-based language models [11], [12]. Audio-based methods extract acoustic or spectral features such as pitch, rhythm, energy, and Mel-spectrograms [13]. Image and video-based approaches analyze visual features, colors, objects, scenes, or facial expressions [14], [15]. These methods can achieve strong performance within a specific modality, but their outputs are usually not directly comparable because each modality uses different features, models, and emotion taxonomies.

Multimodal affective computing combines emotional signals from several sources, such as text, speech, images, and video [16]. Existing fusion strategies are commonly divided into feature-level (early fusion) and decision-level fusion (late fusion) [17], [18]. Feature-level fusion combines modality-specific features before classification, whereas decision-level fusion combines the outputs of independent models [19], [20]. These approaches improve emotion recognition accuracy when several modalities are available [21], but they are usually designed for a specific task, dataset, or object type [22].

The problem of comparing objects with different sets of modalities, such as a movie and a song or a book and an image, has received limited attention in the literature [23], [24]. Several studies have explored shared latent spaces for cross-modal understanding, in which heterogeneous data are projected into a common embedding space to improve semantic alignment [25]. However, most of these approaches focus on task-specific optimization rather than universal emotional representation. As a result, comparing emotional states across different media types remains a challenging problem.

3 Methodology↩︎

3.1 Proposed Approach↩︎

The proposed approach aims to represent heterogeneous multimedia artifacts in a common visual-emotional space. While text, music, images, and videos differ substantially in their low-level features, they all convey emotional information that can be represented in a common space.

Let \(E={e_1,e_2,\ldots,e_k}\) denote a predefined set of emotional categories, where each dimension corresponds to a specific emotion (e.g., happiness, sadness, fear, anger, love, trust, surprise, and others). Given an arbitrary artifact \(x\), its emotional representation is defined as

\[\mathbf{v}(x)=\left(v_1,v_2,\ldots,v_n\right),\]

where \(v_i \in [0,1]\) denotes the normalized value associated with emotion \(e_i\).

For multimodal artifacts, emotional information is extracted independently from each available modality.

The modality-specific emotional descriptors are arranged into a descriptor matrix

\[\mathcal{D} = \begin{bmatrix} v_{1,1} & v_{1,2} & \cdots & v_{1,n} \\ v_{2,1} & v_{2,2} & \cdots & v_{2,n} \\ \vdots & \vdots & \ddots & \vdots \\ v_{K,1} & v_{K,2} & \cdots & v_{K,n} \end{bmatrix}\]

where each row corresponds to a modality-specific emotional descriptor and each column corresponds to an emotional category.

The resulting descriptor matrix (\(\mathcal{D}\)) represents the artifact’s overall emotional profile in the universal emotional space. Consequently, heterogeneous objects, including images, musical compositions, textual documents, videos, and other multimodal content, can be directly compared using a common emotional descriptor regardless of their original modality.

To provide an interpretable visualization of the proposed universal emotional descriptor, each emotional category is mapped into a two-dimensional affective space using predefined valence-arousal coordinate areas. This representation allows emotional descriptors to be analyzed not only numerically but also geometrically. Each emotion \(E_i\) is associated with a pair of affective values: the valence dimension (negative to positive affect), and the arousal dimension (low to high activation). These values are predefined based on an established psychological emotion model and summarized in Table 1.

The angular positions were derived from the original Geneva Emotion Wheel (GEW) [26], [27] representation, which contains 20 emotions distributed across a \(360^\circ\) affective space. Accordingly, each emotion was assigned an angular sector of \(18^\circ\) (\(360^\circ / 20\)). The values reported in Table 1 correspond to the central angle (median) of each emotional sector. For example, the Happiness sector spans from \(18^\circ\) to \(36^\circ\), its representative angle is \(27^\circ\).

In addition to their affective coordinates, each emotional category is assigned a unique color for visual distinction. This color encoding provides an intuitive way to identify emotional regions in the shared affective space.

To preserve modality information during visualization, each artifact is represented using a modality-specific geometric marker. For example:

• Circle — text artifacts

• Square — image/poster artifacts

• Triangle — music/audio artifacts

• Diamond — video artifacts

Thus, while all artifacts share the same emotional coordinate system, their original modality remains distinguishable through shape encoding.

Fig. 2 illustrates the prototype of the proposed multimodal emotional descriptor. Thus, the proposed emotional representation integrates the following information:

1. Color - emotional category identity;

2. Shape - original artifact modality;

3. Position - affective location in valence-arousal space;

4. Color Opacity - the channel weight;

5. Size - the emotional intensity.

In the current implementation, channel weights are not used, and all modalities contribute equally to the final representation. Nevertheless, the weighting mechanism is retained as an optional, adaptable component for application-specific scenarios that require different contributions from different modalities.

Taken together, this multimodal visualization provides an interpretable framework for exploring emotional relationships among heterogeneous artifacts in a unified emotional space.

3.2 Comparison Strategy↩︎

To quantify the similarity between two emotion descriptors, three complementary similarity measures are employed: Earth Mover’s Distance (EMD), Jensen–Shannon (JS) similarity, and Weighted Jaccard similarity. Each metric captures a different aspect of descriptor similarity, and together they provide a comprehensive evaluation of emotional correspondence between digital objects.

3.2.1 Earth Mover’s Distance↩︎

Earth Mover’s Distance (EMD) [29] measures the minimum transportation cost required to transform one emotional distribution into another. In the proposed framework, the transportation cost between two emotions is determined by their relative positions in the valence–arousal emotion space. Consequently, transferring emotional mass between semantically close emotions incurs a lower cost than transferring it between emotionally distant categories. The transportation cost between emotions \(i\) and \(j\) is defined as

\[c(i,j)= \frac{ \min\left( |\theta_i-\theta_j|, 360^\circ-|\theta_i-\theta_j| \right)}{180^\circ},\]

where \(\theta_i\) and \(\theta_j\) denote the angular positions of the corresponding emotions. The similarity between two descriptors \(P\) and \(Q\) is then computed as

\[S_{\mathrm{EMD}}= 1-\mathrm{EMD}(P,Q).\]

3.2.2 Jensen–Shannon Similarity↩︎

Jensen–Shannon similarity evaluates the similarity between two normalized emotional distributions while remaining symmetric and numerically stable [30]. Let

\[M=\frac{P+Q}{2},\]

where \(P\) and \(Q\) denote the aggregated emotion descriptors.

The Jensen–Shannon divergence is defined as

\[JS(P,Q)= \frac{1}{2}D_{KL}(P\parallel M)+ \frac{1}{2}D_{KL}(Q\parallel M),\]

where \(D_{KL}\) denotes the Kullback–Leibler divergence.

The divergence is converted into a similarity measure as

\[S_{\mathrm{JS}}= 1-JS(P,Q).\]

3.2.3 Weighted Jaccard Similarity↩︎

To quantify the overlap between two emotional descriptors while preserving emotion intensities, the Weighted Jaccard similarity is employed [31], [32]:

\[S_{\mathrm{WJ}}(P,Q)= \frac{ \sum_i \min(P_i,Q_i)}{\sum_i \max(P_i,Q_i)}.\]

This metric measures the proportion of shared emotional content between two descriptors while accounting for differences in emotion intensity.

The complete workflow for comparing two universal multimodal emotion descriptors is presented in Algorithm 3.

A higher similarity value indicates a greater correspondence between the emotional profiles of the compared artifacts, regardless of whether they originate from the same or different modalities. Consequently, paintings, musical compositions, textual documents, videos, and other multimedia content can be analyzed within a common emotional framework.

4 Experiment and Results↩︎

4.1 Emotion Retrieval↩︎

4.1.1 Image Emotion Retrieval↩︎

Image emotion recognition is performed using the color-based approach proposed in [33]. The method represents images with a fuzzy color model in the HSI color space and compares the extracted color palette with emotion-specific palettes derived from the WikiArt Dataset.

For an input image, the similarity between its color palette and each emotion palette is computed using the Jaccard coefficient:

\[J(E,I)= \frac{|E \cap I|}{|E \cup I|},\]

where \(E\) is the palette of a target emotion and \(I\) is the palette extracted from the input image.

The obtained similarity values form the image emotional descriptor \(\mathcal{D}_{img}=[d_1,d_2,\ldots,d_{10}]\) corresponding to the ten target emotion categories (we use seven out of them).

4.1.2 Lyrics Emotion Retrieval↩︎

Lyrics are analyzed using the NRCLex library. Raw emotion scores are extracted from the input text and mapped onto a predefined emotion set used in our descriptor according to

\[v_i = S(M(e_i))\]

where \(M(e_i)\) maps the predefined emotion \(e_i\) to the corresponding NRC emotion category and \(S(\cdot)\) denotes the extracted emotion score. The resulting emotion scores are then normalized using max-normalization to obtain values in the range \([0,1]\), ensuring compatibility with descriptors extracted from other modalities.

4.1.3 Audio Emotion Retrieval↩︎

Music emotion retrieval is based on 30-second audio excerpts obtained via the Deezer API. Each excerpt is divided into 5 equal segments, converted into Mel-spectrograms, and processed by an emotion regression model trained on the PMEmo dataset [34]. For each segment, the model predicts valence and arousal values, which are subsequently mapped into the proposed emotional descriptor space:

The resulting \(\mathcal{D}_{audio}=\{d_1,d_2,\ldots,d_K\}\) preserves the temporal evolution of emotions and serves as the audio emotional descriptor.

4.2 Comparison↩︎

To provide an illustrative evaluation of the proposed universal multimodal emotion descriptor, three representative pairs of digital objects were selected. The first two pairs correspond to semantically related content originating from the same creative work but represented by different modalities: the Me Before You original soundtrack (lyrics and audio) compared with the corresponding book (poster and a textual summary of the book), and Leo Tolstoy’s novel Anna Karenina compared with its 2012 film adaptation, where the textual representation of the novel is based on its summary. As a contrasting example, a pair consisting of a happy song (Pharrell Williams - Happy) and a sad song (Tom Odell - Another Love) was included to evaluate the proposed descriptor’s ability to distinguish emotionally dissimilar content.

The similarity between the aggregated emotion descriptors was evaluated using Earth Mover’s Distance (EMD), Jensen–Shannon (JS) similarity, and Weighted Jaccard (WJ) similarity. The obtained similarity scores are summarized in Table 2.

As expected, both semantically related pairs achieved considerably higher similarity scores than the emotionally contrasting pair across all three evaluation metrics. In particular, the comparison between the Anna Karenina book and its film adaptation yielded the highest EMD similarity (\(0.8840\)), indicating a strong correspondence in their emotional representations despite being derived from different modalities. Similarly, the Me Before You soundtrack and the corresponding book exhibited consistently high similarity values. In contrast, the comparison between the happy and sad songs produced substantially lower similarity scores, with the Weighted Jaccard similarity equal to zero, indicating no overlap between their dominant emotional distributions. These observations demonstrate that the proposed descriptor preserves emotional similarity across heterogeneous modalities while effectively distinguishing emotionally contrasting digital objects.

5 Conclusion↩︎

This paper presented CD-MED, a Cross-Domain Multimodal Emotion Descriptor for representing heterogeneous digital objects within a unified visual emotional space. By transforming modality-specific emotion recognition outputs into a common emotional representation, the proposed approach enables direct visual comparison of digital objects regardless of their modality composition. The descriptor provides both an interpretable visualization in the valence–arousal space and a normalized emotional representation suitable for similarity analysis. Experimental results demonstrated that semantically related objects consistently achieved higher similarity scores than emotionally contrasting objects using Earth Mover’s Distance, Jensen–Shannon similarity, and Weighted Jaccard similarity.

The framework is flexible and easily extensible while preserving interpretability through a unified visual representation. At the same time, the current evaluation is intended as a proof of concept and is limited to a small number of representative examples. A more comprehensive quantitative evaluation is required to further validate the robustness and generalization capability of the proposed descriptor.

Acknowledgement↩︎

This research has been funded by the Science Committee of the Ministry of Science and Higher Education of the Republic of Kazakhstan (Grant No. AP22786412)

References↩︎