VINECS: Video-based Neural Character Skinning
July 03, 2023
Abstract
Rigging and skinning clothed human avatars is a challenging task and traditionally requires a lot of manual work and expertise. Recent methods addressing it either generalize across different characters or focus on capturing the dynamics of a single character observed under different pose configurations. However, the former methods typically predict solely static skinning weights, which perform poorly for highly articulated poses, and the latter ones either require dense 3D character scans in different poses or cannot generate an explicit mesh with vertex correspondence over time. To address these challenges, we propose a fully automated approach for creating a fully rigged character with pose-dependent skinning weights, which can be solely learned from multi-view video. Therefore, we first acquire a rigged template, which is then statically skinned. Next, a coordinate-based MLP learns a skinning weights field parameterized over the position in a canonical pose space and the respective pose. Moreover, we introduce our pose- and view-dependent appearance field allowing us to differentiably render and supervise the posed mesh using multi-view imagery. We show that our approach outperforms state-of-the-art while not relying on dense 4D scans.
1 Introduction↩︎
Animating 3D characters is a long-standing and challenging problem in computer graphics and vision. Traditionally, rigging, the process of positioning a sparse skeletal structure inside a dense 3D mesh, and skinning, the process of associating vertices with the skeleton, require large amounts of manual effort by experienced artists. This is not only expensive and time-consuming but also inherently difficult since one set of skinning weights is typically not ideal for all sorts of articulated character poses.
There are also methods for automated rigging and skinning. Some recent works [1]–[4] propose a learning-based solution for obtaining skinning and rigging given a single 3D character mesh. However, they do not consider pose-dependent skinning correctives and lead to the typical appearance artifacts. Therefore, other techniques focus on pose-dependent skinning formulations accounting for pose-dependent variations and deformations [5]–[9]. Noteworthy, none of these methods tried to learn pose-dependent skinning weight correctives purely from 2D image data. Instead, they typically assume a 3D mesh is given, and most of them solely work on a fixed mesh resolution, which cannot be easily adjusted.
To overcome these limitations, we propose VINECS, i.e., VIdeo-based NEural Character Skinning. Given solely a multi-view video of a human actor, our method creates a fully rigged, skinned, and textured 3D template (including hands) along with pose-dependent skinning weights while no manual editing is required. Moreover, VINECS enables multi-resolution character skinning since skinning weights can be sampled over a continuous 3D canonical space, which is not limited to any specific mesh resolution.
For each character, we select one canonical pose frame, for which we extract a dense and textured 3D template mesh using implicit surface reconstruction techniques [10]. Given the template and the respective pose obtained from an off-the-shelf multi-view markerless capture system [11], we leverage an auto-rigging and skinning method, Pinocchio [12], for obtaining an initial set of skinning weights. Note that Pinocchio produces only a static set of skinning weights, which leads to the typical geometry artifacts when poses deviate too much from the canonical pose. Therefore, we propose a coordinate-based multi-layer perceptron (MLP)—which takes as input a 3D point on the surface of the template in canonical pose and the skeletal pose—and predicts the pose-dependent skinning weights at that point. Since we want to learn the skinning weights solely from multi-view imagery, we design dedicated losses consisting of a silhouette loss and a rendering loss. Specifically, we find that obtaining a well-behaved convergence is challenging only using a static texture for the rendering loss. Thus, we adopt the rendering formulation with an albedo network and a shadow network, which predict pose- and view-dependent appearance of our 3D character resulting in an improved convergence and skinning weights prediction. In summary, our technical contributions are as follows:
An end-to-end trainable method for animation-ready explicit character creation, involving static template generation, rigging, and pose-dependent skinning solely using 2D multi-view videos.
A coordinate-based and pose-dependent skinning formulation, which enables multi-resolution skinning.
A dedicated character appearance formulation and differentiable rendering enabling weakly supervised learning based on multi-view videos.
We compare VINECS to the state-of-the-art and our results confirm that our method is a clear step toward accurate and automated creation of animatable 3D characters.
Potential Impact. The focus of this work is not to model deformable geometry [13]–[16] or photorealistic appearance [17]–[20] of humans but on learning 3D characters with pose-dependent skinning from 2D observations only. We believe our method can greatly benefit many downstream tasks as many existing works leverage naked body models [21] for space canonicalization and feature assignment, fundamentally limiting them to clothing types of the same topology as the human body. As our rigging and skinning are accurate, easy-to-use, and robust to any type of clothing, our approach can stimulate research on the modeling of geometry and appearance of loose apparel.
2 Related Work↩︎
2.0.0.1 Static Skinning.
Linear Blend Skinning (LBS) [22], [23] linearly blends rigid transformations of each bone to obtain a posed geometry. However, this formulation can lead to a collapse of geometry near the skeleton joints, also called candy-wrappers effect. To overcome this, several follow-up works have been proposed using spherical representations [24], sweep surfaces [25], curve-based skeleton parameterizations [26], Dual Quaternions [27], or by optimizing the centers of rotation [28]. Other works focused more on computing the skinning weights itself by modeling the weight distribution as a heat diffusion process [12], as illumination [29], or by voxelizing the mesh to increase robustness [30], [31]. Later on, deformation editing was also modeled using a Laplacian energy formulation [32], and user-defined splines on the mesh enable intuitive editing of skinning-based deformation behaviors [33]. They all only assume a single geometry and a corresponding skeleton are given.
In contrast, data-driven methods assume multiple instances of the same character under different poses or a large collection of rigged and skinned characters. Earlier works [34], [35] require meshes under different poses and then automatically detect skeleton joints and compute the skinning weights. Recently, NeuroSkinning [1] was proposed as the first learning-based method predicting a static set of skinning weights given the mesh and a corresponding skeleton. RigNet [2] instead jointly predicts the skeleton as well as a set of skinning weights solely given a 3D mesh as input. Therefore, they collected a large dataset of 3D characters including artist-designed skeletons and skinning weights. SkinningNet [3] proposed a two-stream graph convolutional network architecture for improved skinning weight prediction given a mesh and the skeleton. Yang et al. [36] propose to predict the 3D shape together with the rigging from a single image. Liao et al. [4] proposed a learning-based method which jointly learns to predict skinning weights and transfer poses.
What all the above methods have in common is that they do not explicitly account for dynamic or pose-dependent effects, such as muscle bulging or coarse cloth deformations. In contrast, our proposed method learns pose-dependent skinning weights, which alleviate typical artifacts that arise when using a static set of skinning weights. Moreover, it can also represent coarse deformations induced by the skeletal pose, such as coarse cloth deformations.
Dynamic Methods. More closely related to our method are approaches that account for dynamic or pose-dependent effects, which are discussed next. There are works [37] that allow an artist to add pose-dependent effects by painting onto specialized textures or by adding so-called helper bones [38]. Alternatively, Kavan et al. [39] proposed to find the optimal skinning weights by minimizing an elastic energy and adding joint-based deformers. Interestingly, those methods do not assume any dynamic observations of the character while still modeling dynamic effects. However, they either require manual editing [37], [38] or can mostly model elasticity-based dynamics [39].
Similar to the static methods, several dynamic methods assume a (large) dataset of character meshes is given. Earlier works [6]–[8], [40] account for dynamic effects by capturing a set of range scans depicting the subject in different poses. Given a new pose, a set of pose-dependent blend weights is estimated, which linearly combines the set of example poses in the dataset or alternatively uses linear and radial basis functions [5]. However, since the complexity of computation typically linearly scales with the number of example poses in the dataset, those methods cannot scale up to a very large number of examples (e.g., 20000 pose frames), which is in stark contrast to our approach that can deal with such an amount of data. There are also parametric naked human body models [21], [41], [42] learned by applying PCA over a large set of human scans, which can account for different shape- and pose-dependent deformations. In contrast, we build a highly-detailed rigged and skinned character that can wear (loose) clothing in a completely automated fashion. Li et al. [9] predict static skinning weights and pose-dependent neural blend shapes. They assume a dataset of dense meshes is provided, whereas we purely learn pose-dependent skinning weights from videos. SCANimate [43] also requires dense point clouds, and learns a static set of skinning weights, while pose-dependent shapes are predicted by a pose-aware implicit field. SNARF [44] represents the character as a pose-dependent neural implicit surface in the canonical pose space while the learned skinning weights are pose-agnostic. Again, they require point cloud data while our method can be supervised directly on the video. In summary, none of the above methods that consider pose-dependent effects investigated learning solely from 2D imagery. Thus, we are the first method demonstrating that dynamic effects in the form of pose-dependent skinning weights can be learned only from videos.
Recently, some works were proposed to skin and rig implicit surface representations. ARAH [45] learns an implicit human model using neural rendering. However, as it relies on a SMPL template and an inverse root finding to convert from view space to canonical space, it easily fails when the garment deviates much from the body. Importantly, it also does not have the vertex correspondence over time. TAVA [46] avoids these problems by removing the template assumption, which, however, leads to poor geometry recovery mainly due to their density-based scene representation.
3 Method↩︎
![Figure 2: Method overview. Our method, VINECS, is a fully automated solution to template generation, rigging, and pose-dependent skinning solely using multi-view videos. We first recover the skeletal poses using markerless motion capture [11] and a static template mesh using implicit surface reconstruction [10]. An initial skinning is obtained by a heat diffusion process [12]. Since one fixed set of skinning weights can lead to artifacts when significantly changing the character pose, we learn pose-dependent skinning weights using a coordinate-based MLP. An auxiliary appearance field is applied to better supervise the learning of pose-dependent skinning.](/article/2307.00842/fig/method.png "fig:")
Given synchronized and calibrated multi-view RGB videos of a human character in motion, our goal is to learn an animatable 3D human model with dynamic skinning, without any manual process, such as meshing, rigging, and skinning. To this end, we first extract the static geometry of the actor from one canonical pose frame (typically depicting the actor in a T-pose) of the recording and automatically compute initial skinning weights (Sec. 3.1). However, the initial skinning weights are calculated only based on the static geometry of the human character, and, as studied in earlier works, one set of weights might be ideal for one pose while it can lead to strong artifacts in other poses. Thus, we propose a skinning network, called SkinNet, to predict the dynamic skinning weights as a function of the skeletal pose and 3D coordinates in the canonical space (Sec. 3.2). Given the skeletal pose, the static geometry, and the pose-dependent skinning weights, we leverage Linear Blend Skinning (LBS) to obtain the posed geometry. We then use a differentiable renderer to weakly supervise the skinning weights solely based on multi-view imagery. Thereafter, we propose a novel appearance field (Sec. 3.3), which is composed of an albedo component and a pose- and view-dependent shadow/shading component. In addition, we propose several regularization losses ensuring that the skinning weights are also robust to out-of-distribution poses and that the final animated geometry looks visually plausible (Sec. 3.4). An overview of our method is illustrated in Fig. 2. Before we explain our method in more detail, we first discuss our data assumptions and introduce notations.
Data Preparation. For a human subject, we record a multi-view video of the performance using \(C\) calibrated and synchronized cameras with a total length of \(F\) frames. For all frames \(\mathcal{I}_{c,f}\), we calculate the foreground segmentation using color keying or background matting [47] and compute respective distance transform images \(\mathcal{D}_{c,f}\). Here, \(c\) and \(f\) refer to the respective camera and frame index. We capture the size of the human skeleton and the motion using markerless motion capture [11]. The skeleton motion is parameterized by a root rotation \(\boldsymbol{\alpha}_{f} \in \mathbb{R}^3\), a root translation \(\mathbf{t}_{f} \in \mathbb{R}^3\), and the joint angles \(\boldsymbol{\rho}_{f} \in \mathbb{R}^D\). The full pose vector is defined as \(\boldsymbol{\theta}_{f} = [\boldsymbol{\alpha}_{f},\mathbf{t}_{f},\boldsymbol{\rho}_{f}]^T \in \mathbb{R}^{3+3+D}\).
Notation. We denote vectors by bold lower-case letters, matrices by bold upper-case letters, and scalars and functions by lower-case letters. To represent a variable in the canonical space, we use a bar over the letter, i.e., \(\mathbf{\bar{x}}\). For any point in canonical and global space, we use \(\mathbf{\bar{x}}\) and \(\mathbf{x}\), respectively, and mesh vertices are denoted as \(\bar{\mathbf{v}}_i\) and \(\mathbf{v}_i\) where \(i\) is the \(i\)th vertex. Without loss of generality, we assume the frame \(f\) is fixed in the following (if not explicitly mentioned otherwise) and omit the subscript for better readability.
3.1 Canonical Character Model↩︎
As the first step, we acquire a canonical model of the actor. Recently, implicit surface and coordinate-based representations [48] have gained considerable attention due to their flexibility and comparably simple integration into learning frameworks, and they were also leveraged in the context of pose-dependent human deformation modeling [44]. Moreover, we compared the reconstruction accuracy on the first frame \(f=0\) between classical multi-view stereo and implicit reconstruction method called NeuS [10] in the supplemental material. Interestingly, NeuS has a much higher level of detail and comparably less noise on the surface. Thus, we use NeuS [10] to reconstruct the mesh in the first frame, also called the canonical frame/space.
Implicit vs. Explicit Canonical Model. In stark contrast to Chen et al. [44], we run Marching Cubes [49] to convert the implicit surface into an explicit mesh and obtain per-vertex colors from NeuS by setting the viewing direction to the normal direction of the mesh. The reason for choosing an explicit mesh in canonical space is two-fold: 1.) It avoids running the rather slow Marching Cubes algorithm for every potentially posed mesh. 2.) We avoid the backward skinning [44], which boils down to a correspondence search that has to be solved using iterative optimization. 3.) We can maintain correspondence on the surface across poses, which is especially useful when texturing and other editing have to be applied. Thus, we aim for a hybrid strategy for template acquisition, i.e., using the state-of-the-art method for implicit surface reconstruction and then converting it to an explicit mesh.
Meshing and Initial Skinning. During training, we downsample the template mesh to \(N\) vertices where \(\bar{\mathbf{v}}_i \in \mathbb{R}^3\) denotes the \(i\)th vertex. Given the template mesh and the corresponding posed skeleton, i.e. \(\boldsymbol{\theta}_{f}\), we utilize Pinocchio [12] to obtain an initial set of skinning weights \(\mathbf{w}_{\mathrm{init},i} \in \mathbb{R}^J\) for each vertex \(i\) (see Fig. 4). Here, \(J\) denotes the number of skinning joints. Given these weights and a skeletal pose \(\boldsymbol{\theta}\), we can transform the vertex \(\bar{\mathbf{v}}_i\) from the canonical space to the posed space using LBS [23]: \[\label{eq:lbs} \mathbf{v}_i = LBS(\bar{\mathbf{v}}_i, \mathbf{w}_{\mathrm{init},i}, \boldsymbol{\theta}) : \mathbb{R}^3 \times \mathbb{R}^J \times \mathbb{R}^D \rightarrow \mathbb{R}^3.\tag{1}\]
Mesh Parsing. Since different body parts and materials (skin vs. clothing) have different deformation behaviors, we compute per-vertex human parsing labels. We render the 3D model into all camera views and apply a 2D human parsing approach [50] on each view. We then perform max-voting to obtain the per-vertex human parsing labels. More details can be found in the supplementary material. Note that so far, we obtained a segmented, rigged, and statically skinned character model without any manual user input.
3.2 Pose-dependent Skinning Field↩︎
So far, the initial skinning weights obtained in Sec. 3.1 have two main limitations: 1.) They are solely computed on a single static pose leading to artifacts such as candy wrappers and mesh distortions if the new pose significantly differs from the canonical one. 2.) The set of weights is computed per-vertex. Thus, the resolution of the mesh has to be fixed beforehand and any change in the mesh requires a new computation of skinning weights or some complicated weight transfer from one mesh to the other.
To resolve this, we propose SkinNet \(f_{\boldsymbol{\omega}}\), which is a coordinate-based MLP where \(\boldsymbol{\omega}\) denotes the trainable network weights. Given any 3D point \(\mathbf{\bar{x}}\) in the canonical space, \(f_{\boldsymbol{\omega}}\) predicts its skinning weight \(\mathbf{w}\) conditioned on the normalized human pose \(\tilde{\boldsymbol{\theta}}\), which can be formulated as: \[\label{eq:skin95net} \mathbf{w}_{\tilde{\boldsymbol{\theta}}} = f_{\boldsymbol{\omega}}(\mathbf{\bar{x}}, \tilde{\boldsymbol{\theta}}) : \mathbb{R}^3 \times \mathbb{R}^D \rightarrow \mathbb{R}^J,\tag{2}\] where \(\mathbf{\bar{x}}\) and \(\tilde{\boldsymbol{\theta}}\) are concatenated before being fed into the MLP. Here, the normalized pose \(\tilde{\boldsymbol{\theta}}\) is obtained by neglecting the global translation as well as the yaw angle, i.e., the rotation around the spine. The intuition behind is that the skinning weights do not depend on where in global 3D space the person performs a respective pose, and, similarly, the skinning weight prediction should be agnostic to which direction the person is facing. Thus, we mask those degrees of freedom before feeding the pose vector into the network. To obtain the transformed point \(\mathbf{x}\) in posed space, one can insert Eq. 2 into Eq. 1 resulting in: \[\label{eq:final95skin} \mathbf{x} = LBS(\mathbf{\bar{x}}, f_{\boldsymbol{\omega}}(\mathbf{\bar{x}}, \tilde{\boldsymbol{\theta}}), \boldsymbol{\theta}).\tag{3}\] Revisiting the drawbacks of initial skinning discussed above, the SkinNet now supports pose-dependent skinning weights such that the artifacts of static skinning can be reduced. Moreover, since SkinNet is a coordinate-based MLP, it supports arbitrary mesh resolution since each vertex is queried independently, which is in stark contrast to an architecture that outputs all skinning weights at once and where the final tensor shape is fixed to a specific mesh resolution (see the supplemental document, which evaluates this design choice). Details about the architecture are provided in the supplemental document as well. Next, we explain how we learn a neural dynamic appearance model of the actor, which can then be leveraged in our image-based losses.
3.3 Auxiliary Appearance Field↩︎
We can now pose the geometry with our pose-dependent skinning weight formulation. However, since we want to supervise SkinNet solely on multi-view images, we also have to model the appearance of the actor. A naive choice would be to just leverage the static vertex colors acquired from NeuS. However, we found that this texture contains baked-in shadows and wrinkles, and cannot account for the pose- and view-dependent appearance changes harming the training of the SkinNet. We also refer to our ablation studies in Sec. 4.3. To address this issue, we propose an auxiliary appearance field, which consists of an albedo field predicting a pose- and view-agnostic color for a given point \(\mathbf{\bar{x}}\) in the canonical space and a shadow field predicting the pose- and view-dependent shadow/shading properties of \(\mathbf{\bar{x}}\). Important to note here is that we are not interested in creating a highly photorealistic appearance model of the human, but we mainly use it as an auxiliary tool to better supervise the learning of the skinning weight network.
More specifically, the albedo field, referred to as AlbedoNet, is an MLP that predicts a static albedo value \(a_{\boldsymbol{\omega'}}(\mathbf{\bar{x}})=\mathbf{a} \in \mathbb{R}^3\) for a given 3D sample point in the canonical space. Here, \(\boldsymbol{\omega'}\) are the trainable weights of the albedo network. Moreover, the shadow field is also parameterized by an MLP \(r_{\boldsymbol{\omega''}}(\mathbf{\bar{x}}, \boldsymbol{\theta}, \mathbf{n}, \mathbf{d}) = s \in \mathbb{R}^+\), called ShadowNet, which predicts a scalar multiplier accounting for color changes due to shadows and shading. Note that we assume that such effects happen uniformly across all color channels and, thus, we are only predicting a single scalar instead of a scaling factor per color channel. In addition to taking the point in canonical space, the shadow field also takes the skeletal pose \(\boldsymbol{\theta}\) as input as well as the surface normal \(\mathbf{n}\) of \(\mathbf{x}\) in global space and the viewing direction \(\mathbf{d}\), which allows to potentially model pose- and view-dependent lighting effects such as shadows and shading. \(\boldsymbol{\omega''}\) denotes the respective network weights. Details about the architectures are provided in the supplemental document. The final color \(\mathbf{c}\in \mathbb{R}^3\) for point \(\mathbf{x}\) is computed as \[\label{eq:final95color} \mathbf{c}= a_{\boldsymbol{\omega'}}(\mathbf{\bar{x}})r_{\boldsymbol{\omega''}}(\mathbf{\bar{x}}, \boldsymbol{\theta}, \mathbf{n}, \mathbf{d}).\tag{4}\]
3.4 Multi-view Video-based Supervision↩︎
Next, we describe our dedicated supervision strategy that solely requires multi-view imagery. We first introduce the individual loss terms and then describe our training strategy. Importantly, we can obtain a posed template mesh \(\mathbf{V} \in \mathbb{R}^{N \times 3}\) and the pose- and view-dependent per-vertex colors \(\mathbf{C}_c \in \mathbb{R}^{N \times 3}\) by evaluating Eq. 3 and Eq. 4 for all vertices \(\bar{\mathbf{v}}_i\) and views \(c\). Moreover, the following losses that are applied to the deformed geometry and its appearance can directly backpropagate into the respective networks since our formulation is fully differentiable.
Silhouette Loss. This loss [20] ensures that the projected posed geometry projects onto the foreground masks for all views (see supp. material). Since our pose-dependent skinning field Eq. 3 is differentiable with respect to the SkinNet weights \(\boldsymbol{\omega}\), the silhouette loss can provide supervision and learn pose-dependent skinning by matching the posed model silhouette and the image silhouettes.
Rendering Loss. Although the silhouette loss ensures that the posed geometry matches the multi-view silhouettes, it cannot supervise drifts on the visual hull. Supervision directly from the RGB images can help with resolving this issue since appearance features can constrain drifts in the image plane. Thus, we introduce the rendering loss \[\mathcal{L}_\mathrm{rend} = \sum_{c=1}^C \lVert \Pi_c (\mathbf{V}, \mathbf{C}_c) - \mathcal{I}_c \rVert_1,\] which leverages a differentiable renderer \(\Pi_c\) based on the one of Habermann et al. [20] to render the posed and colored model into all camera views \(c\) that is then compared to the ground truth image \(\mathcal{I}_c\). Importantly, this loss can supervise both the pose-dependent skinning field as well as the pose- and view-dependent appearance field.
Laplacian Loss. Since the data terms alone can still lead to degenerate results, we also introduce some regularization on the geometry and the skinning weights. First, we regularize the posed geometry with a Laplacian loss \[\mathcal{L}_\mathrm{lap} = \sum_{i=1}^N \parallel \mathbf{v}_i - \frac{1}{|\mathcal{N}_i|} \sum_{k \in \mathcal{N}_i} \mathbf{v}_k \parallel_2^2 ,\] where \(\mathcal{N}_i\) is the indices of one-ring neighborhood of vertex \(i\). This regularizer ensures locally smooth geometry.
Skinning Regularization. We found that SkinNet can overfit the training poses and respective deformations by predicting non-local skinning weights, e.g., the foot joint has a non-zero skinning weight for vertices on the shoulder. This, however, harms generalization to novel poses at test time. To prevent this, we regularize the skinning weights by constraining their values based on the geodesic distance \(d_\mathrm{geo}\) between the vertex and the initial body part it belongs to. This can be formalized with the following loss \[\mathcal{L}_\mathrm{skin} = \sum_{i=1}^N \sum_{j=1}^J \mathbf{w}_{i,j} \left( \frac{ \min_{k \in \mathcal{A}_j}d_\mathrm{geo}(\bar{\mathbf{v}}_i, \bar{\mathbf{v}}_k)}{d_\mathrm{geomax}} \right) ^r ,\] \[\mathcal{A}_j = \{ k | j = \mathop{\mathrm{argmax}}_{t} \mathbf{w}_{\mathrm{init},k,t} \},\] where \(\mathcal{A}_j\) is the set of vertices, which have the highest skinning weight with respect to joint \(j\). \(r=3\) is a hyper-parameter which controls how penalization increases as the geodesic distance increases. \(d_\mathrm{geomax}\) is the maximum geodesic distance between any of the vertex pairs.
Part-wise Regularization. In contrast to the clothed areas, the skin part has rather static skinning. Thus, we use the initial skinning to regularize the predicted skinning weights of the skin part, i.e., the set \(\mathcal{G}\) of vertices, which have the parsing label skin. However, even for skin vertices, we found that the initial skinning weights around the joints are not accurate while the rigid parts (i.e., the vertices away from joints) usually have correct initial skinning. We identify the set of rigid vertices as \(\mathcal{R} = \{ k | \max (\mathbf{w}_{\mathrm{init},k}) > u \}\) where \(u=0.95\). Thus, our part loss \[\begin{align} \mathcal{L}_\mathrm{part} = \sum_{i \in \mathcal{R} \cap \mathcal{G} } \parallel f_{\boldsymbol{\omega}}(\bar{\mathbf{v}}_i, \tilde{\boldsymbol{\theta}}) - \mathbf{w}_{\mathrm{init},i} \parallel_2^2 , \end{align}\] enforces that the predicted skinning weights are close to the initial ones iff the human parsing label of a vertex is the skin (\(\mathcal{G}\)) and the vertex belongs to the rigid part (\(\mathcal{R}\)).
Training Strategy. Our training proceeds in four stages. First, we train SkinNet without using the rendering loss \(\mathcal{L}_\mathrm{rend}\). This step is required to ensure that the posed mesh roughly overlaps with the foreground masks. Next, the AlbedoNet is trained using only the rendering loss \(\mathcal{L}_\mathrm{rend}\) while the SkinNet weights \(\boldsymbol{\omega}\) are fixed. Importantly, the AlbedoNet is not pose-dependent, but we optimize the weights \(\boldsymbol{\omega}'\) across all frames. Afterward, the AlbedoNet and the SkinNet are fixed and we train the ShadowNet using the rendering loss \(\mathcal{L}_\mathrm{rend}\). Again we train on all training frames. Lastly, since we now have a pose- and view-dependent appearance model available, we refine the SkinNet weights including the rendering loss. We found this gives the best performance, as validated by our results.
4 Results↩︎
We now qualitatively and quantitatively evaluate our approach, compare it to previous works, and perform ablation studies evaluating our proposed components in more detail.
Dataset. We evaluate our method on three subjects wearing tight and loose clothing (pants or skirts). Two of them are from the DynaCap dataset [20] and they are referred as D2 and D5. D2 is wearing short pants and a T-shirt while D5 is in a skirt. The number of cameras varies between \(94\) and \(101\) for those two sequences and it consists of around 19000 training frames per subject depicting the actor in various articulated poses. For evaluation, there is a separate testing set of around 7000 frames per subject. For convenience, we keep 1 out of 10 frames for training and evaluation. Additionally, we captured a new subject, V6, wearing a T-shirt and trousers using 116 cameras. We captured a separate training (17730 frames) and testing sequence (5000 frames). Importantly, for this sequence, we recover hand gestures and a respective 3D model including hands.
Metrics. For evaluation, we reconstruct the dense 3D geometry for the testing frames using multi-view stereo reconstruction [51]. We do not use NeuS here as it is very time-consuming. Those dense meshes serve as ground-truth scans and we compare the posed meshes recovered by our method as well as competing methods in terms of the Chamfer distance and the Hausdorff distance going from the recovered mesh to the ground-truth scan (M2S) and vice versa (S2M). We report the average results over all testing frames. All metrics are reported on centimeter scale.
Competing Methods. We compare to SCANimate [43] and SNARF [44], which, in stark contrast to our work, require dense point clouds for training. They learn a static set of skinning weights and a pose-aware implicit shape. Since our approach mostly focuses on skinning, we compare to SCANimate with and without pose-aware shape, referred to as SCANimate and SCANimate* in the following. Finally, we compare to RigNet [2], which can jointly predict a rigging skeleton and static skinning weights solely given the template. Unfortunately, SkinningNet [3] and NeuroSkinning [1] do not provide their code and, thus, we cannot compare to them. For more details and comparisons to other (slightly less) related works [45], [46], we refer to the supplemental document.
4.1 Qualitative Results↩︎
In Fig. 3, we qualitatively evaluate our approach. In the first column, we show the reconstructed static geometry for different subjects. In the following columns, we show the characters in novel poses from the test set overlayed onto the respective images. Note that our posed character nicely matches the reference image, which confirms that our pose-dependent skinning indeed generalizes to novel poses and results in accurately posed meshes.
In Fig. 4, we further visualize the effect of the pose-dependent skinning weights obtained by our approach. The initial skinning weights are fixed across poses leading to the typical skinning artifacts and wrong deformation behavior. In contrast, our proposed pose-dependent skinning weights adapt according to the pose and as a result the deformations look more natural.
4.2 Comparison↩︎
| Quantitative Geometry Comparison | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Subject | D2 | D5 | V6 | ||||||
| Method | Chamfer\(\downarrow\) | M2S\(\downarrow\) | S2M\(\downarrow\) | Chamfer\(\downarrow\) | M2S\(\downarrow\) | S2M\(\downarrow\) | Chamfer\(\downarrow\) | M2S\(\downarrow\) | S2M\(\downarrow\) |
| Pinnochio [12] | 3.760 | 2.162 | 1.599 | 5.077 | 2.811 | 2.267 | 3.358 | 1.871 | 1.487 |
| SCANimate\(^*\) [43] | 3.750 | 2.099 | 1.650 | 5.453 | 2.965 | 2.488 | 3.502 | 1.890 | 1.612 |
| RigNet [2] | 3.599 | 2.078 | 1.521 | 4.989 | 2.836 | 2.153 | 3.369 | 1.894 | 1.475 |
| Ours | 3.034 | 1.746 | 1.288 | 4.512 | 2.442 | 2.070 | 2.993 | 1.719 | 1.274 |
| SCANimate [43] | 2.842 | 1.646 | 1.196 | 4.982 | 2.284 | 2.698 | 3.154 | 1.714 | 1.440 |
| SNARF [44] | 3.306 | 1.981 | 1.325 | 4.560 | 2.405 | 2.154 | 3.489 | 1.952 | 1.537 |
In Tab. ¿tbl:tab:quantitative?, we compare our results to the previous works. Note that we achieve the most accurate results when compared to works only focusing on character skinning. This validates the necessity of pose-dependent skinning weights since other works rely on static ones. In this table, SCANimate and SNARF predict a pose-aware shape while assuming dense point clouds are given. Since it is not the scope of this work to predict pose-dependent non-rigid deformations and our method solely learns from video data, we placed these works separately. Nonetheless, it is worth noting that our approach performs better than SNARF for all subjects, and is close to SCANimate in terms of accuracy for subject D2 and V6 even though our method does not require point clouds for training. Moreover, we show improved accuracy compared to SCANimate for subject D5, who wears loose clothing. We found that SCANimate performs worse on subjects with loose clothing as they rely on SMPL while our method does not have such a restriction.
We also visually compare to those works in Fig. 5. We can see that also qualitatively our method achieves a lower error especially in the regions of the joints where typically most of the skinning artifacts become visible. This can be nicely seen in the error maps, which visualize the per-vertex error. Further, the posed meshes clearly show fewer artifacts for our approach compared to previous work. Note that the entire character including the geometry, rigged skeleton, as well as the pose-dependent skinning can be obtained solely from multi-view video without any manual user input.
4.3 Ablation↩︎
| Ablation Study on D2 | |||
|---|---|---|---|
| Method | Chamfer\(\downarrow\) | M2S\(\downarrow\) | S2M\(\downarrow\) |
| Initial weights | 3.761 | 2.162 | 1.599 |
| Static weights | 3.354 | 1.935 | 1.418 |
| w/o \(\mathcal{L}_\mathrm{rend}\) | 3.116 | 1.783 | 1.333 |
| w/ albedo-only | 3.047 | 1.750 | 1.297 |
| w/ NeuS appearance | 3.152 | 1.786 | 1.365 |
| w/ NeuS albedo | 3.137 | 1.795 | 1.342 |
| w/ displacements | 3.191 | 1.805 | 1.386 |
| Ours (10 views) | 3.129 | 1.791 | 1.337 |
| Ours | 3.034 | 1.746 | 1.288 |
Next, we evaluate our design choices in Tab. ¿tbl:tab:ablation?. First, we evaluate the effect of pose-dependent skinning weights. Therefore, we compare our final result to the initial skinning, which we obtain by leveraging Pinocchio [12] and a version of our method where we use our supervision scheme, but optimize a static set of skinning weights, i.e., the weights are fixed across all poses. From the results, we can see that pose-dependent weights significantly improve the accuracy over the two baselines proving the effectiveness of our formulation.
This is also visualized in Fig. 6.
Another key aspect when learning those weights solely from video data is our proposed supervision scheme. When we do not employ the dense rendering loss or soley use the albedo without shadow estimation, the performance drops. Moreover, we also evaluate the accuracy when using a dense rendering loss with the static appearance from NeuS and using NeuS appearance as albedo while the shadow network is learned. We found that our proposed rendering scheme performs best. To validate the effectiveness of training pose-dependent skinning, we replace the SkinNet by a network predicting per-vertex displacement, which generalizes worse than SkinNet. We train our model with sparse cameras (10 views), and the performance is still satisfying.
5 Discussion and Conclusion↩︎
Limitations. Although we propose the first method for learning pose-dependent skinning solely from multi-view video, our method still has some limitations, which opens up interesting directions for future work. Currently, we query the SkinNet for every vertex in canonical space, which may be inefficient when many points are sampled. Thus, future work could involve exploring more efficient architectures like hashgrids [52]. While hand and body can be faithfully animated, there is no control over the facial expression. Thus, extending our automated character creation pipeline with some parametric model for facial expressions might be a promising direction. Also, a joint consideration of rigging, skinning, and pose tracking may further improve the quality of the 3D animation-ready characters.
Conclusion. We proposed the first method for creating a fully rigged and skinned character solely from multi-view video without any manual processing. To this end, we first acquire a canonical template using an implicit surface reconstruction, automated initial skinning and mesh parsing. Most importantly, we learn a pose-dependent skinning field from a multi-view video, which greatly reduces artifacts. We show how such a skinning field can be learned in a weakly supervised manner by an appearance model and differentiable rendering. Our work demonstrates that animatable character creation can be fully automated while maintaining high-quality geometry, rigging, and skinning.
6 Appendix↩︎
In the following, we show quantitative and qualitative comparisons against ARAH [45] and TAVA [46] (Sec. 7). Then, we provide more details concerning our method (Sec. 8-11). We also provide additional information about how the competing approaches are trained and evaluated (Sec. 12). Moreover, we provide more qualitative results that compare our design choice of using NeuS [10] for template generation with classical multi-view stereo reconstruction [51] (Sec. 13). Lastly, we demonstrate that our method can pose template meshes of varying resolution without re-training (Sec. 14).
7 Additional Comparisons↩︎
| Quantitative Geometry Comparison | |||
|---|---|---|---|
| Subject | D2 | ||
| Method | Chamfer\(\downarrow\) | M2S\(\downarrow\) | S2M\(\downarrow\) |
| Initial weights [12] | 3.760 | 2.162 | 1.599 |
| ARAH [45] | 12.985 | 7.625 | 5.360 |
| TAVA [46] | 5.369 | 3.017 | 2.351 |
| Ours | 3.034 | 1.746 | 1.288 |
We compare VINECS against ARAH [45] and TAVA [46], which are animatable human volume rendering methods, on the subject D2. We provide the quantitative results in Tab. ¿tbl:tab:supp95comparison? and the qualitative results in Fig. 7. The evaluation protocol is the same one as described in the main manuscript. Their reconstruction error is significantly higher than the one of VINECS. Interestingly, their error is also higher compared to using initial skinning weights. We found two major reasons for this: 1) The DynaCap dataset [20] is significantly more challenging than the datasets used for evaluation in their works, and their methods seem to not scale well to more complex conditions (motions, lighting). 2) For TAVA, we found that they use a density-based representation for which it is hard to extract accurate geometry. In contrast, our approach can scale well and achieves a higher accuracy.
We would also like to highlight that their setting and goal is different from ours. Their goal is to obtain an implicit animatable human model for volume rendering, whilst ours is to obtain pose-dependent skinning for explicit human animation.
8 Human Parsing Labels↩︎
To obtain the per-vertex human parsing labels, we first apply a 2D human parsing method [50] pre-trained on the LIP dataset on renderings from multiple views. More specifically, we render the template mesh for one frame, animated by the initial skinning weights colored by the texture obtained by NeuS [10]. We apply ambient lighting for the rendering. We found for certain views, especially views from the top, the human parsing method often failed. Thus, we discarded such views. Then, we run the human parsing method for the rendering of the remaining views. For each view, we can obtain a label for each vertex by finding the label of the nearest 2D pixel from its projection, and we perform max-voting to obtain the final label. Note that if the label with the most votes is the background, we select the label with the second most votes instead. After that, we iteratively run a mode filter within one-ring neighborhood until no vertex is labeled as the background. Originally, the method of Li et al. [50] predicts \(20\) classes. We merge these classes and only keep two classes for our training, i.e., the skin and the clothes.
9 Network Architectures↩︎
All the neural networks used in this paper, namely SkinNet, AlbedoNet, and ReflectanceNet, are based on the coordinate-based multi-layer perceptrons (MLP) sharing the same set of hyper-parameters. The network contains five layers with \(256, 256, 128, 256\) and \(256\) neurons in them, respectively. There is a skip connection from the input to the third layer. Inspired by [53], we use SoftPlus as the activation function. When the query point is fed into the network, we compute its positional encoding [54] and concatenate it with the 3D coordinate. In addition, we re-parameterize the network weights using the weight normalization [55].
For SkinNet, a SoftMax layer is applied on the output of the MLP, along the dimension of joints, to ensure that the output skinning weights satisfy the partition of unity. For ReflectanceNet and for AlbedoNet, there is no processing of the output during training, while during inference, we clip the values so that they are in the range \([0; 1]\). As for the output of ReflectanceNet, we compute its exponential as the final scalar multiplier.
10 Silhouette Loss↩︎
It is a bidirectional loss, which aims to align the projected boundary of the mesh to the foreground mask \(\mathbf{M}\): \[\begin{align} \mathcal{L}_{silh} = \sum_{c=1}^{C} ( & \sum_{i \in B_c} \parallel d_c(\pi_c(\mathbf{x}_i)) \parallel^2 + \nonumber \\ + & \sum_{\mathbf{p}\in \{ \mathbf{u} \in \mathbb{R}^2 \| d_c(\mathbf{u}) = 0 \}} \parallel \pi_c(\mathbf{x}_\mathbf{p}) - \mathbf{p}\parallel^2 ) . \end{align}\] Specifically, the first term pushes boundary vertices \(B_c\) to the boundary of the foreground mask, where the distance transform value \(d_c\) equals zero. In the second term, we minimize the distance between every pixel \(\mathbf{p}\) on the zero contour of the distance transform map and its closest projected vertex \(\mathbf{x}_\mathbf{p}\).
11 Training Details↩︎
We implement our method using TensorFlow [56]. All experiments are performed on a single NVIDIA A40 GPU (48GB). Our training consists of four stages. During the first stage, we train SkinNet alone without the rendering loss for \(50000\) iterations. Next, we train AlbedoNet for \(5000\) iterations and ReflectanceNet for \(5000\) iterations. Lastly, SkinNet is refined with the pre-trained appearance field for \(20000\) iterations. The whole training takes around \(18\) hours.
For all training stages, the network weights are optimized using Adam [57]. We clip the gradient values to \([-1, 1]\). The learning rate is \(0.001\) and the batch size is \(4\).
12 Competing Methods↩︎
Pinocchio [12]. We re-implement Pinocchio using Python. Since in our work, the skeleton is obtained from [11], we only use its skin attachment part to compute the skinning weights based on the skeleton. We use the library SciPy [58] to solve the sparse linear system for the heat equilibrium equation in Pinocchio.
SCANimate [43]. SCANimate requires the registered SMPL [21] pose and shape parameters for all training scans. As we already had the pose tracking of the training sequence, we can animate our template mesh using the initial skinning weights, and the animated meshes can roughly align the scans. Thus, we manually label \(30\) correspondence points between our template mesh and SMPL template and optimize SMPL parameters by fitting to the correspondence points on our animated template mesh. With the paired scan and SMPL parameter, we train SCANimate for each of our characters.
SCANimate* [43]. Since the focus of our paper is to learn the skinning weights, instead of the pose-aware shape, we test SCANimate without the pose-aware shape. More specifically, during inference, we input the canonical pose parameter to the pose-dependent geometry module to obtain a canonical shape, which we keep constant for all poses. We use this canonical shape as the query for the forward skinning network to obtain the skinning weights, which then animates the canonical shape by LBS.
RigNet [2]. We ran RigNet pre-trained on “ModelsResource-RigNetv1” dataset on our template mesh to obtain the skinning weights. Similar to Pinocchio, we only use the skinning prediction module, which takes a mesh and the aligned skeleton and predicts the skinning weights.
ARAH [45]. We train ARAH using the same hyper-parameter settings as in their public codes. It is trained for 1.5 days on 4 NVIDIA A40 GPUs, which is much more expensive than the training of VINECS. We extract the mesh from the SDF of ARAH using Marching Cube [49] with the resolution of \(256^3\).
TAVA [46]. We train TAVA following the same setting as in their public code. The whole training takes 30 hours on a single NVIDIA A40 GPU. The mesh is extracted from the density field using Marching Cube with the resolution of \(256^3\).
13 MVS vs. NeuS↩︎
![Figure 8: NeuS [10] (top) vs. Multi-view stereo reconstruction [51] (bottom). In most cases, NeuS generates more high-frequency details (cloth wrinkles) while introducing less noise (e.g., around the calf).](/article/2307.00842/supp_fig/mvs.png "fig:")
We choose NeuS instead of classical multi-view stereo reconstruction to extract the template mesh because we found in most cases NeuS generates more high-frequency details while introducing less noise (see Fig. 8).
14 Multi-resolution Results↩︎
Our method supports multi-resolution character skinning because SkinNet is an implicit function and can take arbitrary 3D positions as input. During training, we only input the vertices of the template mesh, which has a fixed resolution of around 10K vertices, to SkinNet, because our supervision requires an explicit mesh. However, even though only trained on a fixed resolution, our method generalizes well to different resolutions. We re-sample the original mesh generated by NeuS to different numbers of vertices (1K, 5K, 10K, 50K). Then, they are fed into the same pre-trained model and animated. All meshes deform naturally and have low 3D errors (Fig. 9).