1 Introduction↩︎

Character image animation aims to generate video sequences of the characters from reference images and various control signals such as pose sequences and depth information, while maintaining consistent visual appearance throughout the generated video. This emerging field has garnered significant research attention due to its extensive practical applications across numerous domains, including animation production, cinematic content creation, and commercial advertising.

Early approaches to character image animation predominantly relied on GAN-based feature map warping techniques [1]–[3]. However, these methods were inherently limited by the fundamental constraints of GAN architectures, particularly their training instability. Recently, the emergence of diffusion models has catalyzed a paradigm shift in this field, leading to numerous innovative approaches [4]–[14] that achieve animation by incorporating information from the reference image into video diffusion models [15], [16]. These approaches typically employ convolutional networks [5], [6] to compress and encode control signals into the latent space of the diffusion model, enabling precise control of character animation during the denoising process. Leveraging the robust generative capabilities of diffusion models, these methods demonstrate exceptional video synthesis quality while maintaining both precise fidelity to input control signals and accurate preservation of reference image attributes.

Despite significant advances in these diffusion-based animation approaches, the challenge of animating multiple characters simultaneously has still not been well resolved. Specifically, when attempting to animate multiple characters with their respective motion sequences within a shared environment, existing approaches encounter significant challenges, including appearance inconsistencies, inadequate identity preservation, and imprecise character interaction dynamics.

Several recent approaches [17], [18] have partially addressed multiple character animation, however, these methods only work for some simple scenarios and may fail when encountering complex scenarios such as position interchange between characters. Moreover, a common limitation among these methods is their fundamental requirement that all characters should be presented within a single reference image, which makes it challenging for these methods to extend to multiple character scenarios due to the restricted selection of reference images.

In this paper, we present MultiAnimate, a comprehensive framework that addresses the aforementioned challenges. Our approach introduces novel framework capable of simultaneously animating multiple characters from distinct reference images and their corresponding pose sequences within a shared environment, while maintaining robust identity preservation throughout the generated video sequence. Several well-designed modules are proposed to achieve this, specifically: (1) Identity-specific reference net. We incorporate positional encoding mechanisms with reference net to accurately extract appearance features while preserving distinct identity information from multiple reference images, thereby preventing feature attribution errors and maintaining distinct character identities. (2) Identity-aware pose encoder. To solve the identity-pose binding problem of pose sequences, we utilize the cross-attention mechanism to inject identity information after extracting the features of pose sequences with convolutional networks to make the pose feature identity-aware. (3) Interaction Guider. To enhance the management of inter-character spatial relationships during animation, we incorporate an interaction guider module that assists the model’s generation process, which acts as an optional component when processing the pose sequences, enabling the occlusion awareness between pose sequences. In addition, we also introduce a background branch in the framework to achieve animation in the same scenario, which makes the whole framework more controllable.

Our framework has been evaluated through qualitative and quantitative experiments, demonstrating its superiority in terms of the quality of the synthesis images, preservation of identity throughout the video, and correct spatial relationships between characters, as shown in Figure 1. We also conduct some extensive ablation experiments to verify the effectiveness of the modules in our framework.

Generally, the key contributions of our method consist of the following:

• We present MultiAnimate, a novel framework for multiple character animation that enables concurrent animation of multiple characters using their respective reference images and pose sequences within a shared environment, while maintaining appearance consistency and ensuring naturalistic character interactions.

• We develop a sophisticated animation framework comprising several innovative components, including an identity-specific reference net that facilitates multiple reference image processing, an identity-aware pose encoder that establishes robust character-pose bindings, and an interaction guider module that orchestrates coherent character interactions.

• We validate our approach through comprehensive experimental evaluations and ablation studies, demonstrating both the effectiveness of our framework and the essential nature of each architectural component.

2 Related Work↩︎

2.1 Character Image Animation↩︎

Distinguished from GAN-based methods [1]–[3], [19]–[24], the emergence of diffusion models, particularly the Stable Diffusion family [15], [25], has catalyzed extensive research incorporating auxiliary modules to leverage their powerful generative capabilities for character image animation [4]–[14], [26]. LEO [27] models motion distributions in latent space to enable character animation from still images. LaMD [28] proposes a latent motion diffusion framework that factorizes video generation into motion synthesis and content reconstruction, leveraging a computationally efficient 2D UNet architecture. Animate Anyone [5] introduced a reference net architecture to capture and integrate reference features through self-attention mechanisms in the denoising net. DreamPose [11] implemented an adapter mechanism to incorporate reference image information into the original Stable Diffusion UNet. UniAnimate [14] achieved temporal aggregation of reference images through the Mamba [29] architecture, while MimicMotion [6] introduced a PoseNet with confidence-aware pose guidance to enhance motion realism. Further architectural innovations include Champ [10], which incorporated the 3D parametric human model SMPL [30] as a control signal, integrating additional multi-modal signals such as depth and normal maps throughout the generation process. MIMO [31] also proposed a 3D-aware approach, separately encoding and compositing human, background, and occlusion information.

Recent work on multiple character animation includes Xue et al. [17], which enables animation of multiple characters within a single reference image and pose sequence. Similarly, Wang et al. [18] developed a mask-aware identity encoder for multiple character animation. However, these approaches share a significant limitation: they require all characters to be present within a single reference image and their poses to be contained within a unified sequence, substantially constraining their practical applicability. In contrast, our methods proposed several modules to solve the above limitations, which allows multiple input of reference images and pose sequences.

2.2 Multiple Subject Generation↩︎

In the domain of image and video generation, the synthesis of content incorporating multiple subjects based on diverse reference images or concepts, commonly known as multi-concept personalization, represents a significant research direction. Several approaches have emerged to address this challenge. Some methods [32]–[35] achieve multi-concept personalization through model fine-tuning with multiple text embeddings corresponding to distinct reference images. Training-free methodologies [36]–[41] alternatively implement direct injection of multiple image embeddings during the inference phase. Mix-of-show [42] introduces an innovative approach utilizing multiple concept LoRAs and gradient fusion during the denoising process, enabling effective composition of customized concepts. Notably, Movie Weaver [43] presents a novel framework implementing anchored prompts and concept embeddings to establish direct associations between concepts and their corresponding reference images without architectural modifications, which has partially informed our framework.

3 Method↩︎

3.1 Preliminary↩︎

AnimateDiff [16] extends the conventional 2D UNet architecture [44] of the traditional Latent Diffusion Model (LDM) [25] to a 3D UNet incorporating a temporal motion module, enabling video generation capabilities. VAE encoders [45], [46] \(\mathcal{E}\) are employed to transform video frames \(x_{1:n}\) to latent representation as \(z_{1:n}=\mathcal{E}(x_{1:n})\). Subsequently, gaussian noise \(\epsilon \sim \mathcal{N}(0,1)\) will be added to \(z_{1:n}\) according to the time step \(t\), and the training objective of the framework could be formulated as:

\[\mathcal{L}=\mathbb{E}_{z_{1:n},c,t}(\Vert \epsilon-\epsilon_\theta(z^t_{1:n},c,t) \Vert_2^2)\] where \(\epsilon_\theta\) represents the denoising function of the denoising UNet, \(c\) represents the conditional inputs, and \(z^t_{1:n}\) represents the intermediate latent feature during denoising at timestep \(t\).

3.2 Overview↩︎

Conventional character image animation frameworks are limited to processing single reference images with corresponding control signals. Even recent approaches that support multiple character animation impose significant limitations: they require all subjects to be presented within a single reference image while simultaneously demanding comprehensive pose sequences containing poses of each character, substantially limiting their practical applications. This limitation motivates our fundamental research question: How can we enable independent reference images and pose sequences for multiple characters within a unified animation framework? To address this challenge systematically, we propose a comprehensive problem formulation.

Given a set of reference character images \(\{I^k\}_{k=1}^N\) and their corresponding pose sequences \(\{P_{1:n}^k\}_{k=1}^N\), where each pose sequence \(P_{1:n}^j\in\mathbb{R}^{n\times H\times W\times 3}\) comprises \(n\) frames of 2D human skeleton representations serving as the control signals for animating the respective reference image \(I^j\in\mathbb{R}^{H\times W\times 3}\). Our objective is to simultaneously animate these characters from distinct reference images within a shared scenario input \({B_{1:n}}\in\mathbb{R}^{n\times H\times W\times 3}\), synthesizing a video sequence \(x_{1:n}\in\mathbb{R}^{n\times H\times W\times 3}\). Additionally, we define optional interaction mask sequences \(\{M_{1:n}^k\}_{k=1}^N\), where \(M_{1:n}^j\in\mathbb{R}^{n\times H\times W}\) denotes the spatial occupancy region of character \(j\) in the animation result. The comprehensive conditional generation process \(g(\cdot)\) can be formulated as: \[g(\cdot):(\{I^k\}_{k=1}^N, \{P_{1:n}^k\}_{k=1}^N, B_{1:n}, [\{M_{1:n}^k\}_{k=1}^N])\rightarrow x_{1:n}\]

To achieve this, our framework incorporates fundamental modifications to both the reference net architecture and the integration of the pose image. Specifically, we introduce an identity-specific reference net that enables accurate feature extraction from multiple reference images. Additionally, we implement an identity-aware pose encoder that establishes explicit identity-pose bindings after feature extraction, thereby preventing identity ambiguity during the generation process. To facilitate naturalistic inter-character interactions, we also implement an interaction guider module as an optional component that orchestrates proper occlusion relationships and spatial interactions between characters during the animation process. Furthermore, inspired by MIMO [31], we integrate background information into the network architecture to enhance the framework’s practical utility. The overall pipeline of our framework is illustrated in Figure 2 (a).

3.3 Identity-Specific Reference Net↩︎

Since its introduction in Animate Anyone [5], the reference net architecture has gained widespread adoption in character image animation tasks, owing to its robust feature extraction capabilities from reference images. However, when applied to multiple character animation, traditional reference net imposes a significant constraint: it requires all characters to be present within a single reference image. This limitation substantially restricts both reference image selection and the practical applicability of the framework. When the number of characters to be animated increases, it is difficult to find a suitable reference image. To address this limitation, we propose an enhanced approach to reference net utilization, wherein character features are extracted independently from separate reference images and subsequently integrated during the denoising process. This design will enable a freer choice of reference images and enhance the scalability of the framework.

The conventional reference net architecture replicates and concatenates reference features with denoising features, a naive extension is directly concatenating features from multiple reference images. However, since the inherent spatial-agnostic nature of the reference net, a significant challenge arises: during the cross-attention operations, the denoising feature queries the entire concatenated reference feature without spatial discrimination. This lack of spatial awareness can lead to feature attribution ambiguities, as the denoising net cannot effectively determine which portions of the reference features should be utilized, thus leading to feature attribution errors, such as incorrect clothing assignments. To address this challenge, we propose an identity-specific reference net, as illustrated in Figure 3, which introduces spatial awareness into the feature processing pipeline. The whole process can be formulated as: \[f^{enc}_{i,l} = f_{i,l} +PE_{i,l}\] where \(f_{i,l}\) represents the feature representation \(i\) after the \(l\)-th layer of the reference net, and \(PE_{i,l}\) denotes the corresponding position encoding, which is learned during the training process. Each encoded feature \(f^{enc}_{i,l}\in\mathbb{R}^{h\times w\times c}\) is subsequently replicated along temporal dimension \(t\), and then concatenated with the denoising feature of the \(l\)-th layer \(f^{denoise}_{l}\in\mathbb{R}^{t\times h\times w\times c}\) along width dimension \(w\), yielding a composite feature representation \(f_l\in\mathbb{R}^{t\times h\times (n+1)w\times c}\), detailed visualization is presented in Figure 3. The denoising feature then queries the composite feature through a cross-attention mechanism to extract information from distinct reference features. Specifically, \[\begin{gather} f^{out}_{l} = softmax(\frac{QK^T}{\sqrt{d}})V \\ Q=q(f^{denoise}_{l}),\quad K=k(f_l),\quad V=v(f_l) \end{gather}\] the resulting feature with size \(\mathbb{R}^{t\times h\times w\times c}\) serves as input to the subsequent module, following the approach proposed in Animate Anyone [5].

3.4 Identity-Aware Pose Encoder↩︎

In character image animation, numerous approaches [5], [6], [14], [31] commonly employ convolutional networks to extract features from pose sequences, subsequently incorporating these features with Gaussian noise to initialize the denoising process. However, in our multi-character animation framework, where each reference image corresponds to a distinct pose sequence, the conventional approach of naively aggregating pose sequence features proves problematic. This direct feature aggregation often leads to identity ambiguity, as the individual pose sequences lack inherent distinguishing characteristics and become indiscernible when combined.

To establish distinguishable pose sequence features, we implement a cross-attention mechanism that binds identity information to the extracted features. Specifically, as shown in Figure 2 (b), we encode \(j\)-th reference images \(I^j\) through VAE encoder \(\mathcal{E}\) to obtain reference image feature \(z^j = \mathcal{E}(I^j)\), where \(z^j\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times4}\) and extract its pose feature \(z^j_{pose,1:n}=\mathcal{E}_{pose}(P^j_{1:n})\) from the corresponding pose sequences \(P^j_{1:n}\) with convolutional network \(\mathcal{E}_{pose}\). Notably, the pose feature of each frame shares the same size with the reference image feature, we consequently introduce a cross-attention layer for identity binding which could formulated as: \[\begin{gather} f^j_{pose,k} = Conv(z^j_{pose,k}+softmax(\frac{QK^T}{\sqrt{d}})V) \\ Q=q(z^j_{pose,k}),\quad K=k(z^j),\quad V=v(z^j) \end{gather}\] where \(f^j_{pose,k}\) denotes the identity-aware pose feature for the \(j\)-th pose sequence of the \(k\)-th frame, \(Conv\) denotes a \(1\times1\) convolution layer. The query is calculated from the pose feature \(z^j_{pose,k}\) and the key and value is calculated from the reference image feature \(z^j\).

3.5 Interaction Guider↩︎

While our framework successfully enables multiple character animation through the integration of identity-specific reference net and identity-aware pose encoder, the processing of independent pose sequences within a shared environment presents significant challenges, particularly regarding spatial conflicts and inter-character occlusions. Although 2D skeleton pose representations can be extracted using off-the-shelf networks such as DWPose [47], DensePose [48], and OpenPose [49], direct application of these networks fails to preserve crucial occlusion relationships in multi-character scenarios. To address this limitation, we implement an interaction guider module that leverages character-specific mask information produced by segmentation networks (i.e., SAM2 [50]) to process and refine the extracted 2D skeleton representations, thereby ensuring proper spatial relationships between characters.

We extend traditional 2D skeleton representation by introducing three distinct states: (i) visible skeletons with high prediction confidence, (ii) masked skeletons with high prediction confidence, and (iii) skeletons with low prediction confidence. The rendering protocol follows corresponding rules: visible skeletons are drawn conventionally, masked skeletons are rendered in white, and low-confidence skeletons are omitted. As shown in Figure 2 (c), we implement data augmentation through random mask modification and process skeleton representations accordingly, the occluded part of the 2D skeleton representation will be rendered in white. The interaction guider serves as an optional component that enhances the framework’s capability to handle complex inter-character interactions.

3.6 Long Video Inference↩︎

To process extended video sequences, we implement a sliding window approach by segmenting the input video into overlapping slices. Each slice consists of 14 frames with a 4 frame overlap between consecutive segments. The model inference is performed on individual slices, and the results in overlapping regions are averaged with their corresponding regions in adjacent slices to ensure temporal consistency. The algorithmic procedure is formalized in the following pseudo code:

4 Experiment↩︎

4.1 Dataset Construction↩︎

Our data collection process began with sourcing videos from public online platforms that feature multi-character interactions. Following data collection, we implement a filtering protocol to mitigate the impact of noise on the training process. Initially, we employ YOLO-X [51] for character detection and bounding box extraction from the video sequences in the dataset. Characters whose bounding box dimensions constitute less than 10% of the frame size are excluded from subsequent processing procedures, including segmentation and pose extraction, to maintain data quality and computational efficiency.

After detecting the bounding boxes of the characters, we utilize the extracted bounding boxes as prompts for SAM2 [50] to generate segmentation masks for all eligible characters within the video sequences. To maintain consistent character identification throughout sequences containing complex inter-character interactions, particularly position interchanges, we implement a position-based labeling protocol. This protocol establishes character indices based on their initial horizontal spatial distribution, where numerical identifiers (0,1,2,...) are assigned according to the sorted coordinates of bounding box centers in the first frame.

Furthermore, we implement an occlusion-based filtering criterion to exclude videos containing extensively occluded frames, as such occlusions can significantly compromise the quality of reference feature extraction and subsequent reconstruction. The filtering process evaluates the occlusion metrics for each character throughout the video sequence. Specifically, we compute the frame-wise occlusion ratio between each character’s bounding box and those of other characters. Videos are excluded from the dataset if any character’s bounding box exhibits occlusion rates exceeding 65% in more than half of the total frames, as such severe occlusions would impair the extraction of reliable reference features. Figure 5 illustrates the distribution of the number of characters per video in our dataset. The dataset comprises over 10,000 videos featuring multi-character interactions across diverse action categories. From this collection, we randomly select a 5% subset to serve as the test set.

4.2 Experimental Setups↩︎

4.2.1 Implementation Details.↩︎

Our training implements a self-supervised strategy with careful reference image selection criteria: characters are excluded if they occupy minimal image space or experience substantial occlusion throughout the video sequence. We employ state-of-the-art SAM2 [50] for character-specific mask segmentation and DWPose [47] for extracting corresponding 2D skeleton representations of eligible characters.

Our model training procedure consists of two stages on 8 NVIDIA A100 GPUs. The initial stage excludes the motion module and utilizes a batch size of 16. In the subsequent stage, we freeze all previously trained parameters and exclusively train the motion module using video sequences of 14 frames. During training, we implement center-cropping guided by pose sequences to optimize character centralization within each frame.

4.2.2 Evaluation Setups.↩︎

We perform the following evaluations on our multi-character benchmark. For all compared methods, we use the official pre-trained models. We note that comparisons with recent multiple character animation methods [17], [18] are not feasible due to their unavailability as open-source implementations. Given that the compared methods do not support multiple character animation inherently, we adapted their pipelines through a standardized extension: utilizing reference image that contains all target characters and employing the original methods’ pose extraction networks (e.g., DWPose [47] or DensePose [48]) to generate consolidated pose representations. These pose representations encompass all character poses within a single image for each frame of the video sequence. Moreover, to facilitate fair comparison with baseline methods, we select those images that contain all the characters to be animated as reference images.

4.3 Qualitative Comparison↩︎

Qualitative results of multiple character animation can be seen in Figure 6. While some existing methods maintain appearance consistency in initial frames, they exhibit significant degradation during complex scenarios, particularly when handling occlusions (as shown in the second row) and position interchange operations (demonstrated in the third row). In contrast, our method, benefiting from its architectural design, successfully maintains both appearance consistency and spatial relationships throughout these challenging scenarios. Moreover, in Figure 7, we present qualitative results of MultiAnimate for animating multiple characters, demonstrating that MultiAnimate can handle animation tasks involving more than two characters while preserving the identity and appearance of each individual character.

4.4 Quantitative Comparison↩︎

We evaluate performance across multiple methods using both image-level and video-level metrics. For image-level assessment, we employ SSIM [52], PSNR [53], LPIPS [54], MSE, and FID [55] metrics, while utilizing FVD [56] for video-level evaluation. As demonstrated in Table 1, our method achieves superior performance across all metrics compared to existing approaches.

4.5 User Study↩︎

To better show the superiority of our method, we conducted a comprehensive user study through a structured evaluation protocol. We recruited participants with backgrounds in computer vision research, offering compensation for their participation. The evaluation instrument comprised 20 questions addressing multiple quality metrics: visual fidelity, appearance and identity preservation relative to reference images, and temporal coherence. Participants were tasked with selecting optimal video outputs from randomized presentations of results from different methods, methodological sources were presented in randomized order to minimize selection bias. The results of the user study are shown in Table 2 and the results indicate that our method is significantly favored.

4.6 Ablation Studies↩︎

4.6.1 Identity-Specific Reference Net.↩︎

Figure 8 demonstrates the efficacy of our identity-specific reference net through comparative analysis of three experimental configurations: (a) a baseline reference net that simply concatenate reference features, (b) a reference net with static position encoding using constant values, and (c) our proposed identity-specific reference net with learnable position encodings.

The experimental results demonstrate that both settings (a) and (b) exhibit significant identity and appearance ambiguities. Setting (a) fails due to the inherent spatial-agnostic nature of conventional reference net. Moreover, given the high dimensionality and complexity of reference features, the static position encoding using constant values in setting (b) proves insufficient for effective feature differentiation. In contrast, our proposed identity-specific reference net addresses these limitations through learnable position encoding, successfully establishing distinct feature representations from multiple reference images and enabling accurate spatial attention mapping in subsequent processing stages.

In our approach, we add learnable position encoding to each layer of the UNet, drawing inspiration from state-of-the-art image generation models (e.g., Flux [57]) and video generation models (e.g., Wan [58]), which all incorporate PE into every layer of their models. Additionally, FreeFlux [59] demonstrates that existing SOTA models such as Flux use Rotary Position Encoding (RoPE) to achieve better performance than those that only inject PE in the first layer. We further illustrate this through experiments. When we only add this PE to the first layer of UNet, the performance of our framework degrades. The quantitative experimental results are shown in Table 3.

4.6.2 Identity-Aware Pose Encoder.↩︎

We present the ablation study on the identity-aware pose encoder in Figure 9. In experiments without the identity-aware pose encoder, we employ a conventional approach implemented in prior works, where pose sequences are directly aggregated with noise following feature extraction through convolutional networks. For comparative analysis, we conduct pose sequence interchange experiments, where we swap the corresponding pose sequences between reference images to generate paired results.

The experimental results demonstrate that without the identity-aware pose encoder, the generated outputs exhibit identity ambiguities, as the pose sequences share a common feature extraction network lacking the capability to differentiate between distinct pose features. Furthermore, in this configuration, interchanging pose sequences between reference images becomes ineffective, as these features are merely aggregated within the latent space. In contrast, our proposed identity-aware pose encoder efficiently processes multiple pose sequences while maintaining distinct identity associations, enabling accurate character animation even during pose sequence interchange operations. This demonstrates the successful establishment of robust pose-identity bindings.

As Xue et al. [17] utilizes pose sequences as images that contain poses of all characters without explicit identity differentiation. We substituted our identity-aware pose encoder with their approach, and the comparative results are presented in Figure 10. The experimental results demonstrate that the absence of explicit identity-pose binding leads to significant limitations, particularly in animating complex character interactions such as position interchange. Specifically, the inability to bind identities with their corresponding pose sequences results in animation artifacts and incorrect character behaviors. In contrast, our proposed identity-aware pose encoder successfully maintains identity consistency and accurately handles these complex interactive scenarios.

4.6.3 Interaction Guider.↩︎

Figure 11 demonstrates the impact of the interaction guider through ablation experiments. Without the interaction guider, the model exhibits deficiencies in handling complex inter-character interactions, particularly in resolving occlusion relationships. The incorporation of the interaction guider enables the model to effectively distinguish masked regions, resulting in accurate character interaction synthesis.

As Xue et al. [17] utilized a depth order guider to establish plausible inter-character spatial relationships. This mechanism initially generates masks through the dilation of skeleton representations for distinct characters, subsequently producing a depth-order image derived from a depth map that determines the relative positioning of these mask regions. Notably, this approach does not explicitly encode identity information within the individual mask regions. We substituted our interaction guider with their approach, and the comparative results are presented in Figure 12.

We can observe from the figure that while the depth order guider provides spatial relationship representation, its guidance remains insufficiently granular. Specifically, when processing intricate character interactions, such as the hugging action illustrated in the figure, the depth order guider fails to provide accurate spatial guidance. Furthermore, this mechanism introduces interference with background generation in our framework, presumably because its multi-colored compositional nature, spanning a broad value range, potentially affects the latent background representation when incorporated into the initial Gaussian noise. In contrast, our proposed interaction guider minimizes perturbations to both skeleton representations and overall frame composition, as well as enabling accurate synthesis of inter-character interactions.

4.7 Evaluation on Single Character Benchmark↩︎

To better assess the performance of our method, we conducted experiments on single-character animation using the TikTok dataset [60]. The quantitative results are presented in Table 4. We obtained the results for the comparison methods directly from their original paper, since MimicMotion and Moore-Animate did not report their performance on TikTok dataset, we use their offical pre-trained model and conduct evaluation on the TikTok dataset. Moreover, UniAnimate did not report their FID on TikTok dataset. These results demonstrate that although our method is designed for handling multi-character animation, it can be seamlessly applied to single-character tasks and achieves performance comparable to state-of-the-art methods.

5 Discussion.↩︎

Our framework occasionally introduces artifacts, particularly in highly detailed regions such as hands, due to inherent limitations in the base model’s generation capabilities. Recent video diffusion models [58], [61], [62] base on DiT [63] have demonstrated superior video generation capabilities compared to their UNet-based counterparts. We have been able to significantly mitigate these artifacts through simple post-processing methods, such as temporal smoothing. Future iterations of our work will explore the integration of our framework with these more advanced DiT-based architectures. Additionally, we acknowledge potential ethical concerns regarding the misuse of our framework for generating deceptive content. Therefore, we emphasize the importance of implementing appropriate usage controls and guidelines.

6 Conclusion.↩︎

Our paper presents MultiAnimate, a comprehensive framework that enables concurrent animation of multiple characters within a shared environment using distinct reference images and their corresponding pose sequences. The framework extends the conventional reference net through the introduction of position encoding, enabling precise feature extraction and differentiation from multiple reference images. Our identity-aware pose encoder establishes robust character-pose bindings, ensuring consistent character animation throughout the generated sequence. Furthermore, we incorporate an interaction guider module that enhances the framework’s capability to synthesize accurate inter-character interactions. Through extensive experimental evaluations and ablation studies, we demonstrate our framework’s ability to generate high-quality animation results while validating the effectiveness of each architectural component.

The authors would like to thank the members of our research laboratory for their helpful discussions and advices during this paper.

Declarations↩︎

Funding↩︎

This work was supported in part by the Natural Science Foundation of China under Grants 62372423,62121002,62072421, and was also supported by the Fundamental Research Funds for the Central Universities WK2100250070.

Conflict of Interest↩︎

The authors declare that they have no competing interests.

Data Availability↩︎

The multi-character dataset used in this paper was collected from publicly available online sources. We will release this dataset as open source upon acceptance of the paper to further benefit the research community. In addition, the TikTok dataset used in our experiments is publicly available and has been fully cited in the manuscript.

Code Availability↩︎

The code used in this study will be released upon publication to support reproducibility and further research.

Author Contributions↩︎

Zhongyi conceived the research idea, developed the proposed methodology, performed the experiments, and contributed to writing the manuscript. Guangyuan Wang contributed to the project through data collection and preprocessing. Li Hu, Tianyi Wei, Peng Zhang, Bang Zhang, Wenbo Zhou, Weiming Zhang and Nenghai Yu provided scientific guidance, supervised the research design and experimental validation, and assisted in revising the manuscript. All authors reviewed and approved the final version of the manuscript.

Reproducibility↩︎

We ensure reproducibility by providing clear distinctions between results and interpretations, along with detailed reporting of implementation components such as evaluation metrics, and computing resources.

Ethical Considerations in Character Image Animation↩︎

The method proposed in this paper can be applied to generating videos of human motions, with broad application prospects in areas such as cinematic content creation and entertainment. However, if misused, it may raise ethical concerns, for example through the creation of non-existent or falsified videos for malicious purposes. We advocate the responsible development and deployment of such technologies, including safeguards such as embedding watermarks in generated content.

References↩︎