1 Introduction↩︎

Patients with genetic disorders often go through protracted diagnostic odysseys before receiving a diagnosis, which is partly due to limited physician specialist availability and non-specialists’ lack of familiarity with these conditions [1], [2]. Due to the heterogeneous presentations and individual rarity of genetic disorders, diagnosis can be challenging. Facial dysmorphisms manifest in many genetic disorders and play a key role in clinical genetics diagnosis [3]. The availability of digital facial images in clinical workflows and registries enables next-generation phenotyping tools that leverage computer vision and deep learning to detect syndromes from 2D facial photographs [4]–[7]. These tools offer new possibilities for supporting clinicians in the diagnostic process, potentially detecting subtle details and identifying conditions that may be difficult to recognize. Most existing methods, however, are designed to directly predict disorders [4]–[7]. While effective in well-represented settings, such as more common and easily recognized pediatric conditions in European ancestry populations, this paradigm introduces important limitations [3]. Performance typically depends on large, well-annotated datasets and may vary across ancestry and age groups, while limited interpretability constrains clinical trust and adoption [8]. More fundamentally, disorder-level classification ties predictions to a predefined set of known syndromes, restricting generalization to previously unseen conditions and offering limited insight into the phenotypic traits that drive predictions.

In clinical practice, since a single obvious syndrome is often challenging to identify, the differential diagnosis process involves systematically evaluating phenotypic traits. This process is commonly structured using the Human Phenotype Ontology (HPO) [9], a standardized, graph-based vocabulary that captures abnormalities across physiological and psychological traits, including detailed facial traits corresponding to variations in facial morphology [10]. This suggests that facial analysis can be more naturally framed as the prediction of phenotypic traits rather than direct syndrome classification. Modeling facial dysmorphisms at the level of phenotypic traits enables more granular and reusable representations, supports interpretability by linking predictions to clinically meaningful traits, and allows models to capture patterns shared across disorders. Importantly, this approach aligns machine learning outputs with clinical reasoning and remains informative even when no exact syndrome match is available, prioritizing generalization and clinical utility over benchmark performance. Despite these advantages, HPO-based facial modeling poses practical challenges. Available annotations are often sparse, presence-only, and incomplete, leading to ambiguous negatives, while hierarchical dependencies among HPO terms introduce additional complexity. To address these issues, we leverage the HPO hierarchy while simplifying it by removing cross-links and explicitly accounting for label incompleteness at the phenotype level.

To date, the GestaltMatcher DataBase (GMDB) [4], [11] represents the largest curated, FAIR-compliant dataset of facial photographs, cytogenomic and molecular diagnoses, and clinical annotations for rare genetic disorders. It serves as both a reference resource and a foundation for developing phenotyping tools. While GMDB includes HPO annotations, these are typically sparse and derived from publications or clinical reports, resulting in weakly labeled data. To systematically evaluate facial phenotype prediction, we focus on a subset of disorders with sufficient representation and well-characterized facial dysmorphisms, complemented by targeted, image-centric HPO curation to improve the quality of labels for facial traits.

To better reflect phenotypic structure, we move from pixel-based images to geometric 3D face meshes, which are more robust to pose and illumination, align naturally with anatomical regions, and facilitate interpretability at the level of facial regions and points. With our approach, the focus is on the hierarchical HPO tree structure without cross-links, where each HPO term is represented as an individual model. Given the specific focus of the HPO leafs, we further propose including information reduction into the training cycle, since not all mesh points are relevant. Therefore, we introduce a cascading training strategy in which each HPO term model uses data from its descendants and passes point masks to its children. After each model’s training, a point importance analysis identifies unimportant points and eliminates them for the children - also called feature elimination.

Together, this work introduces a hierarchical, phenotype-centered framework for predicting facial HPO terms from 3D facial meshes, along with a large, clinician-curated benchmark with explicit positive and negative annotations for facially observable HPO terms. This combination of methodological and data contributions complements existing disorder classification systems by emphasizing interpretability and clinically aligned generalization.

2 Related Work↩︎

In this section, we contrast syndrome (genetic condition) and phenotypic descriptors classification. We briefly discuss key concepts and how our approach differs from direct syndrome classification.

2.1 Syndrome Classification↩︎

The classification of syndromes using 2D images is a major research area in clinical genetics. Works like GestaltMatcher [4] provide a useful classification tool for clinicians; this approach utilizes a pretrained deep Convolutional Neural Network (CNN) and finetunes it for syndrome classification. With their improved version with GestaltMatcher-Arc [5] they used a model ensemble of iResNet and ArcFace with cross entropy loss. In addition, Islam et al. [12] investigated smaller CNN architectures (e.g., ResNet50, VGG16, VGG19, AlexNet) and added an attention layer to these networks to improve performance. They showed that the VGG16 model performed best despite their smaller architectural build. However, these approaches focus solely on 2D images and on the direct classification of syndromes with (deep) CNNs, which makes it difficult to explain how they achieve their results and, as a result, does not help physicians understand how specific facial traits may contribute to syndrome classification.

Another promising approach leverages 3D images, where the third dimension provides information gain. Hallgrimsson et al. [13] investigated faces of people with genetic disorders as well as healthy individuals based on facial shape differences in order to classify syndromes using parametric and machine learning methods. Another approach using 3D images was introduced by Banniser et al. [14], in which faces of people with genetic disorders were generated alongside their healthy counterparts (so-called counterfactuals), and the syndrome class was determined using Bayes’ theorem. Mahdi et al. [15] classified syndromes with 3D images but extracted facial regions (e.g., nose, mouth) from the 3D face and classified the facial region and the rest of the face separately before merging the results for the final outcome. Therefore, they used geometric encoding to reduce the complexity of the 3D data and linear discrimination analysis for classification. Overall, these methods show promising results, but the third dimension increases data complexity and, consequently, usually requires more data for models to achieve higher performance. In our approach, we use less dense 3D face images with lower complexity, thereby reducing the data required for high performance.

In a combined version of 2D images and geometric information in the form of a face mesh, Dingemans et al. [16] used a CNN (VGGFace2) pipeline to process 2D images and, in parallel, compute edges from the face mesh extracted from the 2D images to feed their results into a Bayesian classifier for syndrome classification. However, they used the distance between face mesh points rather than the points themselves and, more importantly, predicted disorders rather than HPO-based phenotypes, which differs from our approach. We aim to classify facial phenotype traits, not syndromes. Dingemans et al. [7] developed PhenoScore, where they also used HPO terms as a second input, in addition to 2D images, to enhance information gain when classifying syndromes. However, it is possible that the provided HPO terms can already classify a syndrome without the need for analysis of the 2D image. In addition, clinicians would have to identify HPO terms themselves before obtaining a result about the possible underlying disorder. Therefore, we propose a direct approach to classify HPO terms, potentially improving disorder classification results.

2.2 Phenotypic Descriptor Classification↩︎

An example of the kind of work our method could support is that of Robinson et al. [17], which uses their LIRICAL framework to compute the probabilities of disorders based on underlying HPO terms via likelihood ratios. Clinicians can already benefit from the LIRICAL framework by identifying HPO terms themselves and providing them to the framework. Finally, the HPO term classification can also be addressed from the genotypic side, as Ullah et al. [18] demonstrated using a bipartite network, since there are lists of known genes connected to one or more HPO terms. However, genetic and genomic testing is not universally accessible, and can be slow and expensive, meaning that it is not always feasible for physicians to rely on such approaches.

3 Dataset↩︎

The main dataset used in this work is the GestaltMatcher DataBase (GMDB) [11]. The GMDB is a curated, FAIR-compliant dataset of clinical and other phenotypic data from individuals with genetic disorders, with a particular focus on facial dysmorphism. It contains annotated patient cases, including frontal face photographs and, when available, molecular or cytogenomic diagnoses. The dataset further includes clinical descriptive traits (HPO terms), sex, age, ancestry, pedigree information, and other relevant medical images and data, with varied availability and completeness across cases. GMDB serves both as a reference resource for clinicians and medical researchers and as a benchmark and training dataset for next-generation phenotyping tools and methods, such as GestaltMatcher(-Arc) [4], [5].

3.1 GMDB-HPO↩︎

For this work, we focused on a subset of disorder groups that have both relatively high representation in GMDB and notable facial dysmorphism. Disorder groups include all subtypes of a disorder; hereafter, these are collectively referred to as disorders. For instance, Kabuki syndrome has subtypes 1 and 2 (caused by variants in two different genes), which are in this work both considered as Kabuki syndrome. Although subtypes may differ, they share key dysmorphisms. Based on these criteria and reflecting a practical trade-off between dataset diversity and manual annotation effort, we selected 10 disorders for downstream analyses in this initial study. After all rounds of quality control, this subset contains 1,230 images of 1,001 individuals, with a total of 16,569 present and 34,355 absent HPO-term annotations across 72 HPO terms. However, in the training process only 70 HPO terms were used, as two HPO terms (HP:0410030 - Cleft lip and HP:0004493 - Craniofacial hyperostosis) had insufficient data. Furthermore, all respective parent HPO terms were added, yielding a final set of 107 HPO terms (as described in Section 4.1). Table 1 shows the disorders and their patient and image distributions in GMDB-HPO.

3.1.1 Annotation Effort↩︎

As GMDB is weakly labeled with respect to HPO terms (terms annotated as present are known to be present, but unannotated terms are not necessarily absent), we designed an annotation protocol and annotation tool to address this incompleteness. Initially, the OMIM and ORPHA resources for each disorder were reviewed via the HPO website, and all frequent HPO terms within the “Head and neck”, “Ear”, and “Eye” categories were collected [9], [19]–[21]. With the assistance of medical genetics experts, this list was further pruned to prioritize terms that are typically observable in frontal face photographs. In addition, mutually exclusive terms (e.g., “Narrow mouth” vs “Wide mouth”) were identified to streamline the annotation process. The annotation protocol prompts annotators to label each pre-selected term as “present”, “absent”, or “uncertain” (when visibility or interpretation is unclear), and also allows annotators to add additional relevant HPO terms not included in the pre-selected list of HPO terms. To minimize annotation bias, the patient’s disorder label is initially hidden during annotation. The annotation tool is shown in Figure 1.

In total, 124 annotators contributed to the annotation effort. Annotators were recruited in multiple rounds through the GMDB platform. The participants were GMDB users with medical backgrounds who expressed interest in the study. Participation was voluntary, and contributors with the highest annotation activity were eligible for co-authorship, mentions, or acknowledgments in accordance with standard authorship guidelines. All annotations were performed in accordance with GMDB’s data access and governance policies. The annotation effort began in January 2025; however, for the purpose of this study, a data freeze was applied in December 2025, defining the dataset used for all analyses reported here.

3.1.2 Quality Control↩︎

Throughout the annotation effort, a total of 1,676 frontal face photographs were assessed and annotated. A subset of photographs exhibited annotation inconsistencies, such as the absence of any term annotated as present or the absence of any term annotated as absent. We therefore applied an initial round of quality control (QC) to remove inconsistent or incomplete annotations, resulting in 1,301 images. Following QC, the annotations generated in this study were combined with existing HPO annotations from GMDB for the same images, incorporating previously labeled terms. A final round of QC was then applied to resolve conflicts between newly added annotations and existing ones (e.g., a term marked as present in GMDB but annotated as absent in this study). These overlaps between existing HPO terms and new HPO annotations were resolved by following the two-step merging procedure: 1) Present traits (PT) and absent traits (AT) from the GMDB were merged with their new manual present (MPT) and absent (MAT) counterparts; 2) MPTs that existed within the AT were not added to the PT, with the same applying to MAT and AT. After this final integration and QC, the resulting dataset comprised 1,230 photographs, with a total of 72 HPO terms annotated. A flowchart with the stages of QC is shown in appendix Fig. 8, and a full list of HPO term frequencies and their corresponding mean Inter-Rater-Agreement (IRA) with standard deviation and 95% confidence intervals across all samples were computed using Fleiss’ Kappa [22] are shown in appendix Table 1.

3.2 Independent Test Set↩︎

In addition to GMDB-HPO, we evaluated model performance on an independent test set annotated by multiple experts. This dataset was constructed prior to the start of the large-scale annotation effort described in Section 3.1, originally with the aim of exploring the feasibility of HPO-based classification from facial images. While its size was insufficient for training a meaningful model, the dataset provides a valuable resource for evaluation.

After similar QC, the test set comprises 17 disorders, each with up to 5 patients. For each image, three medical experts independently annotated HPO terms. In contrast to the main dataset, where each image is annotated by a single expert selected from a large pool of annotators, this setup ensures consistent multi-expert annotation across all samples, thereby reducing inter-rater variability and increasing annotation reliability. After quality control, we used a total of 75 test images, whose distribution is shown in Table 2.

The annotation protocol was designed specifically for HPO classification. For each disorder, HPO terms classified as “frequent” or “very frequent” in the Human Phenotype Ontology and Online Mendelian Inheritance in Man (OMIM) [19] databases were preselected. Annotators were asked to assess the presence of each term in every patient image. A term was considered present if at least two out of three annotators agreed on its presence. In addition, annotators were instructed to label any non-preselected terms that were not categorized as “frequent” or “very frequent” but were deemed clearly present; such terms were included without requiring majority agreement. Though annotators were not asked to explicitly label the absence of terms, due to the majority voting, we feel that it is reasonable to assume terms were not present if they were labeled by no more than a single annotator.

Importantly, this test set includes both disorders present in the training data and disorders not observed during training, enabling evaluation in both seen and unseen disease settings. As such, it serves as a complementary evaluation resource, providing insight into model performance under reduced annotation noise, improved label consistency, and generalizability to other disorders.

4 Method↩︎

This section describes our methodology for cascaded hierarchical classification using feature elimination as displayed in Figure 2. The framework includes a preprocessing step (4.1), optional metadata parameters (age, sex, ethnicity), face-region masks, and a cascading HPO model tree.

The HPO is not a tree structure with single connections between nodes. Cross-links between multiple nodes are possible, making it more like a graph than a tree. However, we had to cut cross-links to simplify the graph into a tree with a clear hierarchical structure, as multiple parents cannot be processed by our method. The Abnormality of the face is defined as the root node. Since the Abnormality of the eye and Abnormal eyebrow morphology are not part of the branches outgoing from Abnormality of the face under the HPO, we moved them to be children of Abnormality of the face. To replicate this tree structure, each model is defined as a single HPO term (see Figure 2). This means that each model has a parent model and can have child models if it is not a leaf.

4.1 Preprocessing↩︎

The data preprocessing pipeline consists of two steps:

1) Face Mesh Extraction: The input data for the pipeline consists of 2D images of faces. Landmarks are extracted from these images using the Face Mesh Detector from mediapipe [24] as of September 15, 2022. This detector provides 478 points distributed across the face in a semi-3D face mesh, with the third dimension approximated.

2) HPO Table Generation: For each image, the phenotypes are read and displayed in a table. Since the phenotypes are predominantly leaves in the HPO tree, their parents must be identified and included in the table so that each image records the corresponding phenotype, starting from the root of the HPO tree.

4.2 Metrics↩︎

The models were evaluated using several metrics. The Area Under the Receiver Operating Characteristic Curve (AUROC) is the primary metric for assessing a model’s performance. Because it reflects the \(sensitivity\) and \(1-specificity\) balance, it provides a clear indication of a model’s performance. An AUROC of 0.9 to 1.0 is excellent, 0.8 to 0.9 is good, 0.7 to 0.8 is moderate, 0.6 to 0.7 is poor, and 0.5 to 0.6 is near chance. In addition, the Matthews Correlation Coefficient (MCC) is used to select the best model for inference, as it provides a more fine-grained score using sensitivity, specificity, precision, and negative predictive value. Therefore, a high MCC (e.g., MCC=0.95) always corresponds to a high AUROC but not vice versa [25]. Furthermore, the metrics F1-Score, precision, and recall were chosen to provide a general impression of the models’ performance. Additionally, the prevalence and detection prevalence are provided for better comparison.

4.3 Model-Architecture↩︎

The architecture used for the HPO model tree is based on the PointNet [26] architecture as shown in Figure 3. The number and type of layers remain the same. The input size is based on the dimensions \(d\) used. To increase flexibility when working with different point counts, we dynamically adjusted the model’s parameter count by varying \(p\). The reduction of the parameter count depends on the number of input points. When more than 100 input points are available, \(p=6\). If the number is between 50 and 100, \(p=5\), and in a range greater than 25 and 50, \(p=4\). Finally, when the input point number is 25 or less, \(p=3\). If the number of input points falls below a critical threshold for processing by the architecture, a zero-padded difference between the minimum and the actual number of points is added after the input transformation. To embed the metadata into the classification task, the values are converted to integer numbers: -1 if not available, or a dedicated number for each class. The metadata list is added to the global features and processed by the last Multi-Layer-Perceptron (MLP) step. Furthermore, we added a more robust normalization to each convolutional layer, switching to group normalization when the batch size is 1, as batch normalization fails. If the batch size is greater than 1, the batch normalization is applied.

4.4 Training Strategy↩︎

The tree root is trained first using all available points from the face mesh. For the input data, all samples, including itself and its children, are used recursively. That means the root model is trained on the entire dataset of affected individuals, and the unaffected samples are drawn from the UTKFace [27] dataset, with a distribution as close as possible to that of our dataset in age, sex, and ethnicity. At higher levels of the tree, when unaffected samples are available in both our dataset and the UTKFace dataset, we split at most 50/50 between the two.

After a model is trained on a set of points, a point-wise feature importance is computed using the Integrated Gradients [28] to extract importance values for each point on the face mesh. The feature importance of each fold is accumulated through all folds, and the mean is computed to gather a global feature importance of all models. Based on the global feature importance, a threshold is applied to a mask, eliminating all points that fall below it. This is represented in Fig. 2, where the color of the points in each HPO node indicates the Integrated Gradient feature importance of each point in the training set. The more blue a point is, the higher the chance that the point is eliminated. The eliminated global feature importance mask is used in the next iteration of the model’s children, limiting them to only the points from the face meshes present in the mask. This cascading reduction of points across each level of the HPO tree simplifies the classification task. For some HPO models (Abnormality of the mouth, Abnormality of the orbital region, Abnormality of the nose, Abnormality of the eye, Abnormal midface morphology, Abnormal forehead morphology, Abnormality of the periorbital region, Abnormality of the chin) masks were predefined by a genetic physician expert on a specific set of points to focus the classification on certain regions, e.g., mouth, eyes, nose, chin, forehead. Therefore, we developed an interactive website (https://hcmlab.github.io/hpo-mesh-annotator/) to enable medical professionals to easily select relevant region points. If a predefined mask is provided to the model, the predecessor mask is ignored.

In general, all models were trained using 5-Fold Stratified Cross Validation (CV) to ensure that a face, displayed across multiple images, does not appear in both the training and validation splits. A model was trained only if a point mask with at least 2 points was provided, and the sample size for the HPO term was greater than or equal to 50. The training was performed for up to 25 epochs, with a learning rate of 0.0001, automatic learning rate reduction at a plateau with a patience of 5 epochs, and early stopping with a patience of 5 epochs. The training seed was set to 42.

5 Results↩︎

We conducted an ablation study to evaluate different model-tree configurations and identify the best configuration. Afterward, we analyzed the HPO models with the best configuration. Furthermore, a deeper look into the correlation of key metrics was taken to investigate if correlations are present. Finally, the HPO models were evaluated on a test set.

5.1 Model Selection through Ablation Study↩︎

The ablation study embodies 72 experiments, which have been performed using a permutation of the following settings: dimension (3D and 2D), face outline (False and True), soft labels (0, 0.05, and 0.1), feature importance threshold (0.01, 0.05, and 0.1), and metadata (empty and [age, sex, ethnicity]). Since a face mesh is represented by an approximate third dimension, we tested whether the x- and y-coordinates alone would suffice without the z-coordinate. In addition, the face mesh’s outermost points represent the face outline and can be used alongside points from a point mask. Due to the lack of unlabeled unaffected individuals, we tested whether soft labels (0, 0.05, and 0.1) could improve performance, as suggested by [29]. Since the unaffected individuals used are not verified, it is unclear whether they are indeed unaffected or might possibly be affected, which is also a reason for using soft labels. Furthermore, we tested different thresholds for the feature importance elimination (0.01, 0.05, and 0.1). Finally, we also experimented with using no metadata and age, sex, and ethnicity as metadata.

The results of the top-10 and bottom-5 experiments from the ablation study are listed in Table 3. The best experiment with 3D-Face Meshes, the face outline, metadata, a threshold of 0.01, and a soft label of 0.05 performed with an AUROC score of \(0.750\pm0.042\). The top-5 experiments all used 3D-Face Meshes, the face outline, and metadata. The changes in performance are from the used soft label, where 0.05 performed best, with 0.001 performance gain over 0.00. The feature elimination threshold of 0.01 outperformed 0.05 by \(\approx0.009\) and 0.1 by \(\approx0.025\). The 2D-Face Mesh experiment lagged the same configured 3D experiment by 0.007. The 3D experiment without using the face outline performed \(\approx0.013\) worse than the same experiment with the face outline. A higher feature importance threshold led to lower performance, as many points were eliminated in earlier stages, leaving only a few, or even none, points for leaf-HPO terms to be trained on and, therefore, represented, resulting in zero AUROC. The full result table is available in appendix Table 2.

5.2 HPO Delineation Performance Analysis↩︎

In Table 4, the performance for the top-5 and bottom-5 performing leaf HPO models is depicted. The five best HPO term models (Short nose, Wide nasal bridge, Hypertelorism, Downturned corners of mouth, and Thin upper lip vermilion) achieved AUROCs ranging from \(\approx0.76\) to \(\approx0.80\) and F1-Scores ranging from \(\approx0.78\) to \(\approx0.81\). On the other end of the table, in the bottom-5 models, the long nose performed best with an AUROC of \(\approx0.60\) and an F1-Score of \(\approx0.62\). Synophrys lagged behind with an AUROC of \(\approx0.59\) and F1-Score of \(\approx0.58\). Microphthalmia performed third last with an AUROC of \(\approx0.58\) and F1-Score of \(\approx0.42\). Furthermore, Elfin facies performed second-to-last, with an AUROC of \(\approx0.55\) and an F1-Score of \(\approx0.28\). Finally, the worst-performing model is for Blepharophimosis, with an AUROC of \(\approx0.55\) and an F1-Score of \(\approx0.47\). Overall, 107 HPO models were trained. In addition, we examined the prevalence of the training and validation sets for each CV fold and aggregated the results. Given an exact 50% split between affected and unaffected classes, the prevalence is always 0.5. However, the detection prevalence shows a different result as shown in Table 4. The detection prevalence ranges from \(\approx0.23\) (Elfin facies) to \(\approx0.74\) (Thick vermilion border). The full table is in the appendix Table 3.

Furthermore, the Fig. 4 shows the correlation between key metrics of all HPO models with the top-5 performing models in green and bottom-5 performing models in orange. A Pearson correlation is found between the number of samples and the mean AUROC. The fewer the number of training samples, the lower the mean AUROC; the more samples, the higher the mean AUROC. No correlation was found between the number of input points and the mean AUROC. Furthermore, the Inter-Rater-Agreement (IRA) across all samples also showed no correlation with the mean AUROC. The Pearson correlation between input points and parameter count, shown in the correlation matrix with a score of \(0.61\), is by design and is not coincidental.

When more closely examining the correlation between the number of samples and mean AUROC (Fig. 5), the differentiation between parent and compression models and leaf models becomes clearer. A compression model has exactly one child and, as such, only compresses the dimension/number of points further down. In Fig. 5, one can see that the leaf models need at least 988 samples to achieve a mean AUROC performance of greater than or equal to \(0.75\). Comparing parent and compression nodes, where the compression nodes are a subset of the parent nodes, the parent nodes need at least 1092 samples, while compression nodes need only 222 samples to achieve a mean AUROC of \(0.75\) or greater.

5.3 Evaluation of Generalizability↩︎

We further assessed the generalizability of our approach using the independent multi-expert–annotated test set described in Section 3.1 and further detailed in the appendix. We evaluated model performance separately for disorders seen during training and those not observed during training.

Across the independent test set, the aggregated, normalized mean F1‑score differences between test and validation vary by disorder, as summarized in Fig. 6. For unseen disorders, Seckel syndrome (SS) and Sotos syndrome (SOTOS) show small positive differences of 0.03 and 0.04, respectively, while Opitz GBBB syndrome (OGBBBS), Ohdo syndrome, SBBYS-variant (SBBYSS), and Susceptibility to Autism (StA) exhibit moderate differences between 0.09 and 0.16. Larger differences are observed for Mowat-Wilson syndrome (MOWS), Nicolaides-Baraitser syndrome (NCBRS), Floating-Harbor syndrome (FLHS), and FBXW7 syndrome (FBXW7S), with values ranging from 0.21 to 0.36. White-Sutton syndrome (WHSUS) shows the largest observed difference of 0.42. Among disorders included during training, Williams-Beuren syndrome (WBS) yields a slightly negative aggregated difference of -0.02, whereas Hyperphosphatasia with mental retardation syndrome (HPMRS) and Cornelia de Lange syndrome (CdLS) show small positive differences of 0.04 and 0.05. Coffin-Siris syndrome (CSS), Noonan syndrome (NS), Ogden syndrome (OGDNS), and KBG syndrome (KBGS) display larger differences between 0.10 and 0.20.

The two tables in the report per‑phenotype mean F1-Score differences between the independent test set and cross‑validation for all HPO terms with non‑zero prevalence, separately for unseen (appendix Table 4) and seen disorders (appendix Table 5). For unseen disorders, the table lists how each HPO model’s F1 changes for StA, FBXW7, FLHS, MOWS, NCBRS, SBBYSS, OGBBBS, SOTOS, SS, and WHSUS, with negative values indicating better performance on the test set and positive values indicating higher validation performance. The corresponding table for seen disorders provides the same F1 differences for CSS, CdLS, HPMRS, KBGS, NS, OGNDS, and WBS, again highlighting for each HPO term whether performance is higher in validation or on the independent test cohort.

The detailed metrics for the non-overlapping test set disorders are listed in the appendix Tables for StA 6, FBXW7S 7, FLHS 8, MOWS 9, NCBRS 10, SBBYSS 11, OGBBBS 12, SOTOS 13, SS 14, and WHSUS 15 and the disorders overlapping with the training/validation set are in the appendix Tables for CSS 16, CdLS 17, HPMRS 18, KBGS 19, NS 20, OGDNS 21, and WBS 22.

5.4 FaceMesh2HPO Web Tool↩︎

To demonstrate the practical applicability of the FaceMesh2HPO framework, we developed a web application (https://hcmlab.github.io/FaceMesh2HPOWebApp/) that enables physicians to perform facial phenotype analyses directly on local devices (Fig. 7). The application integrates the best-performing model from each fold, selecting the model with the highest Matthews Correlation Coefficient [30]. Each selected model is subsequently calibrated using either beta calibration [31] (for datasets with \(\ge 50\) samples) or temperature scaling [32] (for datasets with \(< 50\) samples). The final calibration method is chosen according to the lowest Brier score [33]. If the original model already achieves the lowest Brier score, no additional calibration is applied.

Upon first use, the trained models are downloaded to the local device. This one-time process may take several minutes, depending on network bandwidth, but it ensures that all subsequent analyses are performed locally without uploading patient images to external servers. Once the download is complete, the application is ready for use.

Physicians can then upload a patient image for analysis and optionally provide demographic information (age, sex, ethnicity) to refine predictions. The average total inference time, including face mesh extraction and evaluation of all 107 models, is 624.56 ms.

The results are presented in two main sections. First, the patient image is displayed alongside the extracted face mesh. Second, a hierarchical list summarizes the outputs of all models. In the default view, each entry includes the corresponding HPO term and its classification confidence, with higher values indicating greater certainty that the phenotype is present. Selecting an HPO term reveals additional details, including model performance metrics (e.g., sensitivity and specificity), dataset characteristics (e.g., age, sex, ethnicity, and geographic distribution), and the disorders used during training (Fig. 7, right). In the image view, classification-relevant regions are highlighted using color to indicate global feature importance (Fig. 7, left). Together, these features provide a transparent and interpretable overview for clinical use.

To facilitate navigation, the application includes a search function for HPO IDs and names, allowing users to quickly locate specific terms. Additionally, a confidence threshold slider enables users to filter results by setting a minimum confidence level, hiding all predictions below the selected threshold.

6 Discussion↩︎

This work introduces FaceMesh2HPO, a hierarchical, HPO‑aligned framework for classifying facial phenotypes from 3D face meshes using a cascading training strategy with feature elimination. By assigning a dedicated classifier to each HPO term and organizing these along the HPO tree, the method learns shared structure across related phenotypes, generates predictions at multiple levels of granularity, and outputs labels that match the clinical vocabulary used in diagnostic reasoning. This structure yields rich phenotype profiles that remain informative even when the underlying disorder is unknown, and can be directly combined with tools such as LIRICAL [17] as compositional evidence rather than a single black‑box score.

The ablation study showed that performance is best when using 3D face meshes including the facial outline, together with age, sex, and ethnicity metadata, a feature‑elimination threshold of \(0.01\), and soft negative labels of \(0.05\). The modest gap between 3D and 2D meshes indicates that the approximated depth contributes only a limited additional signal, suggesting that true 3D imaging could unlock larger gains. The face outline and metadata systematically improved results, consistent with their role in providing global proportionality and demographic context for local dysmorphisms. Lower feature‑importance thresholds preserved more points for downstream nodes and improved performance, while soft labels helped mitigate uncertainty in the negative class, where apparently unaffected controls may still exhibit subtle dysmorphic traits.

Performance analyses highlight a strong dependence on ontology level and data support. Parent and compression nodes consistently outperform leaves, benefiting from larger effective sample sizes and more generic patterns. In contrast, rare leaf terms - especially those with limited geometric representation in the mesh (e.g., eyebrow or eyelash traits) often show AUROCs near chance and unstable detection prevalence, reflecting sample scarcity, the compounded effect of upstream feature pruning, and the inaccuracy of detected face meshes on dysmorphic faces as investiaged by Hellmann et al. [34] and showed that facial traits such as Big Nose and Big Lips seem to be not well captured by the face mesh detector. The fixed 50/50 class balancing during training, combined with noisy negatives, further contributes to miscalibration, as evidenced by the wide spread of detection prevalences across terms despite identical training prevalence. In practice, this means the framework currently offers more reliable information at coarser ontology levels and for relatively common traits than for highly specific, sparsely annotated leaves. Furthermore, we believed that the IRA would influence the models’ performance: the lower the IRA score, the lower the model’s performance, and vice versa. Surprisingly, the results painted a different picture, showing no correlation between the models’ performance and the IRA.

The independent, multi‑expert test set clarifies how these patterns generalize. Per‑disorder analyses show that many parent and intermediate terms maintain F1-Score close to or above validation, while leaf‑level performance is more variable. These findings support the central design choice of modeling reusable HPO‑level phenotypes rather than syndromes, but also underscore that the method should be viewed as providing a phenotypic scaffold whose reliability depends on both ontology depth and disorder family, rather than a uniformly accurate detector of all rare traits.

Despite these limitations, the approach is scalable and amenable to extension. Additional HPO terms can be integrated as more curated data become available, and the mesh‑based representation keeps per‑term data needs manageable. The cascading feature elimination already reduces computational costs and focuses models on salient regions, while expert‑designed regional masks help anchor point selection in clinically meaningful areas and partially compensate for the opacity of purely data‑driven pruning.

Future work should prioritize the qualitative validation of the FaceMesh2HPO tool in a study with clinicians. Furthermore, continuing to improve support for rare terms, refining calibration under imbalanced and noisy labels, and systematically assessing performance and bias across age, sex, and ancestry. Finally, the face mesh detection must be refined for faces with dysmorphisms to receive a more accurate representation.

7 Conclusion↩︎

In this work, we addressed the automatic classification of facial phenotypes by utilizing Mediapipe’s 3D face meshes and a dynamic PointNet architecture to handle different input sizes. We introduced hierarchical training with a feature elimination approach to reduce task complexity and simplify model classification. Therefore, the models were trained in a tree-like structure, passing a point mask to their child models. This process was done in a cascading manner. In total, 107 models were trained to classify facial phenotypes on different levels of the human phenotype ontology.

The approach demonstrates that facial phenotype classification on a geometric level, based solely on face meshes, is possible. Using 3D face meshes, the face outline, the person’s age, sex, and ethnicity, a feature elimination threshold of \(0.01\), and soft labels of \(0.05\), the ablation study showed that these settings yielded the best overall results, with a mean AUROC of \(0.750\pm0.042\). The mean AUROC across the 5-Fold CV for each HPO model ranged from \(\approx0.55\) to \(\approx0.89\). The better-performing HPO models are usually those closer to the root model, which is correlated with the available training sample size, as there are typically more samples for models closer to the root node. When applied to other faces with unseen disorders, the method appears reliable. However, results vary slightly per disorder. Finally, the FaceMesh2HPO tool bridges these research results to a real-world application for clinicians to use in their daily practice. The additional information provided for each model will help geneticists and other clinicians to understand and trust these and related applications.

This research was supported in part by the Intramural Research Program of the National Institutes of Health (NIH). The contributions of the NIH author(s) are considered Works of the United States Government. The findings and conclusions presented in this paper are those of the author(s) and do not necessarily reflect the views of the NIH or the U.S. Department of Health and Human Services. As a nonnative English-speaking research team, we have limited the use of generative artificial intelligence tools to polish and enhance our English-language writing, ensuring our research findings can be disseminated globally.

Declarations↩︎

• Funding: Not applicable.

• Conflict of interest/Competing interests: Not applicable.

• Ethics approval and consent to participate: Not applicable.

• Consent for publication: Not applicable.

• Data availability: The curated GMDB-HPO subset is derived from the GMDB dataset (v1.1.0), which is available to qualified researchers through a controlled-access process for the development and evaluation of next-generation phenotyping methods. Researchers seeking access must explicitly state that they are requesting the GMDB-HPO dataset, obtain appropriate ethics approval, submit a research proposal via https://db.gestaltmatcher.org/ or by email to info@gestaltmatcher.org, and sign a data-use agreement ensuring compliance with applicable data protection legislation, including the General Data Protection Regulation (GDPR). All requests are reviewed by the GMDB Advisory Board in accordance with legal and ethical standards. Due to the inclusion of sensitive clinical data and identifiable facial images of individuals with rare genetic disorders, the dataset is not publicly available.

• Materials availability: Not applicable.

• Code availability: The code will be made publicly available on GitHub (https://github.com/hcmlab/FaceMesh2HPO) upon publication.

• Author contribution: FH developed the methodology, performed the experiments, and wrote the manuscript. AH curated the GMDB-HPO dataset and supported FH in designing the experiments and refining the manuscript. BS created the point masks for specific HPO terms to focus the model’s training. GestaltMatcher Database Consortium created the annotations for the HPO dataset. PK provided helpful knowledge in the field and revised the manuscript. EA and TH revised the manuscript.

References↩︎