Pixel-Level Pavement Distress Assessment Using Instance Segmentation


Abstract

Automated pavement distress assessment requires more than image-level classification or coarse bounding box detection, demanding precise localization of thin, branching, and irregular cracks to achieve the geometric precision necessary for maintenance-relevant quantification. This paper presents a vision-based pavement distress analysis system based on Mask R-CNN instance segmentation and evaluates it on UWGB-StreetCrack, a custom field-collected roadway image dataset acquired with a vehicle-mounted iPhone 15 Pro Max and manually annotated with polygon labels for longitudinal cracks, transverse cracks, alligator cracks, and potholes. Five Detectron2-based Mask R-CNN backbone variants were considered under a consistent fine-tuning protocol. The best-performing test result was obtained by Mask R-CNN with a ResNet-101 FPN backbone, which achieved 84.23% precision, 90.04% recall, and an F1 score of 87.04%. The same model produced an aggregate predicted crack-area fraction of 2.164%, closely matching the 2.170% ground-truth crack-area fraction. To contextualize the segmentation system against a detector-oriented alternative, a CSPDarknet53-based YOLO detector was also adapted and retrained on the UWGB data, reaching 27.5% precision and 20.7% recall on the validation protocol. The results show that instance segmentation is a practical direction for field pavement imagery and aggregate crack-area estimation, while also exposing open challenges in annotation consistency, class imbalance, confounder rejection, and mask-level benchmarking.

deep learning, segmentation, computer vision, pavement crack detection

1 Introduction↩︎

Different types of pavement cracks (e.g., longitudinal, transverse, alligator, block, thermal) are portents of structural deterioration caused by repeated traffic loading, thermal cycling, moisture infiltration, and material fatigue [1]. Untreated cracks can lead to potholes, pothole clusters, and larger structural failures that exponentially increase repair costs and compromise road safety. Therefore, early detection and quantification of these defects are critical. Additionally, the Federal Highway Administration (FHWA) estimated that deferred pavement maintenance costs the US economy billions of dollars annually [2], making automated, objective, and quantitative detection and assessment systems essential for evidence-based infrastructure investment.

Among various methods for pavement distress detection and assessment, manual inspection remains the dominant practice worldwide, yet this approach suffers from well-documented limitations. This process is labor-intensive, time-consuming, potentially hazardous when working near traffic, and prone to inconsistency across different teams, seasons, and regions [3]. Besides, the lack of precise and consistent measurements of pavement distress (e.g., percentage of surface area affected by cracking) reduces the objectivity of decisions for maintenance prioritization and budget allocation. Recently, deep learning and computer vision techniques [4][7] have gained significant attention as viable alternatives to traditional approaches. These approaches can analyze large-scale roadway imagery, minimize risks to human inspectors, and generate objective and quantifiable results. Nevertheless, despite the advantages, the inability to support geometric precision (i.e., exact area measurements) required for maintenance-oriented quantification is a persistent challenge for contemporary deep learning approaches.

The evolution of deep learning-based pavement distress detection has progressed through three complementary methodological stages. We contextualize this evolution by first reviewing classical methods in Section [sec:relatedwork] that preceded deep learning, then examining the three primary deep learning paradigms: classification and segmentation approaches, object detection frameworks, and instance segmentation methods. Each stage addressed specific limitations of its predecessor. Early image or patch-level classification methods [8], [9] provided only coarse semantic information (i.e., the presence/absence of distress) without any spatial localization or geometric characterization. Despite their computational efficiency, they were unsuitable for quantifying the extent of distress. To address localization, researchers then adopted object detection architectures [10] that used bounding boxes [11] to locate cracks with real-time speeds (e.g., 50–65 Frames/Second) and generate per-instance confidence scores and class labels. However, these rectangular approximations systematically overestimated area for thin, elongated cracks, introducing geometric error that undermines maintenance-relevant quantification. Recognizing this limitation, researchers explored pixel-level semantic segmentation methods [12][14] that achieved high spatial precision by predicting class labels for every pixel. Although semantic segmentation provides pixel-level geometric precision, it fundamentally cannot maintain instance-level information. As a result, overlapping cracks become indistinguishable, and individual crack properties cannot be quantified [15]. This inadequacy is magnified in some cracking patterns (e.g., alligator), where the method collapses hundreds of discrete crack units into a single monolithic segmentation mask, eliminating the structural information needed for maintenance prioritization. This methodological progression, ranging from coarse semantic classification to spatially precise but instance-indifferent segmentation, revealed the persistent challenge that neither detection nor semantic segmentation alone can provide both the geometric precision and instance-level granularity required for practical maintenance decision-making.

To address the aforementioned challenges, instance segmentation methods such as Mask R-CNN [16] unified three complementary tasks: (i) localization, (ii) classification, and (iii) pixel-level delineation into a single framework. This integrated approach captures the strengths of both detection and semantic segmentation while mitigating their respective limitations. Specifically, this kind of segmentation provides the geometric precision of pixel-level masks (as opposed to the coarse rectangular approximations of bounding-box detection) while maintaining instance-level differentiation (as opposed to the instance-agnostic pixel classification of semantic segmentation). Consequently, this method achieves both high spatial precision and per-instance granularity, enabling distinct measurement of individual distresses even in crowded scenes with multiple overlapping defects. Additionally, it is particularly valuable for full-scene roadway imagery where multiple distresses coexist, and visual confounders (e.g., shadows, tire marks, painted markings, stains, and manhole features) can be mistaken for cracks. Its per-instance classification enables simultaneous multi-defect detection while filtering false positives.

To this end, this applied empirical study evaluated Mask R-CNN [16] instance segmentation on UWGB-StreetCrack - a custom field-collected roadway dataset with polygon-level annotations for four pavement-distress categories: (i) longitudinal cracks, (ii) transverse cracks, (iii) alligator cracks, and (iv) potholes. The primary objective of this study is threefold: (i) to assess the performance of an established Mask R-CNN variant, instead of building a new network architecture, on challenging full-scene roadway imagery, (ii) to quantify aggregated crack-area agreement between predictions and ground truth, and (iii) to document the methodological limitations that must be resolved before the system can serve as a standardized segmentation benchmark. Additionally, to compare performance, a bounding box-based object detection architecture was also adapted and retrained on the same dataset, and a detection-oriented evaluation is presented.

Below, we summarize the main contributions of the current study:

  1. UWGB-StreetCrack dataset: smartphone-based roadway imagery with polygon-level annotations for four pavement distress classes.

  2. Mask R-CNN pipeline: instance segmentation framework for distress localization, mask prediction, and area estimation.

  3. Performance metrics: comprehensive precision, recall, F1, and area-fraction results from Mask R-CNN experiments.

  4. Detection baseline: CSPDarknet53 YOLO [4] adapted to UWGB-StreetCrack for methodological comparison.

  5. Failure analysis: identification of annotation ambiguity, minority-class sparsity, and missing mask-level AP metrics as methodological gaps

The remainder of this paper is organized as follows. Section II surveys related work on crack detection and segmentation architectures. Section III describes the UWGB-StreetCrack dataset and annotation process. Section IV details the Mask R-CNN pipeline, baseline adaptation, and training protocol. Section V defines the evaluation protocol. Section VI presents quantitative results and qualitative failure-case analysis. Section VII concludes the paper with limitations and future directions.

2 Related Work↩︎

While instance segmentation has been successfully applied to pavement-specific scenarios, most existing work focuses on limited datasets, including cropped regions, single-crack scenes, or imagery collected under controlled acquisition conditions. In contrast, full-scene multi-distress assessment on smartphone-collected roadway imagery remains relatively understudied. This section positions our work by reviewing the methodological evolution of pavement crack detection across four major paradigms: (1) classical handcrafted approaches, (2) deep learning through classification and segmentation, (3) object detection and hybrid frameworks, and (4) instance segmentation methods. Through this review, we identify the specific gap—instance segmentation on field-collected multi-distress datasets—that motivates our research.

2.1 Classical and Feature-Engineered Crack Detection↩︎

Early pavement crack detection studies relied on thresholding, edge detection, morphology, wavelet analysis, path-based extraction, and other handcrafted image-processing operations. These methods exploited the observation that cracks are often darker than the surrounding pavement. CrackTree [17] used a tree-structured representation to trace crack-like patterns from pavement images, while CrackForest [18] combined integral channel features with random structured forests to model local crack tokens. Alongside their computational appeal, these methods are sensitive to illumination changes, pavement texture, shadows, stains, and road markings. Their dependence on handcrafted assumptions limits transferability across road surfaces and acquisition conditions.

2.2 Deep Classification and Semantic Segmentation↩︎

Deep convolutional neural networks reduced reliance on handcrafted features by learning representations directly from image data. Zhang et al. [19] used CNNs for road crack detection in image patches, while Fan et al. [5] formulated crack detection as structured prediction with CNNs that generate dense crack probability maps. Encoder-decoder architectures further improved pixel-level delineation. U-Net [20] popularized skip-connected semantic segmentation, and pavement-specific variants such as DeepCrack [21], FPCNet [22], and black-box road-image encoder-decoder models [13] showed the value of multi-scale fusion for thin structures.

Semantic segmentation is well-suited to crack extraction because it predicts crack regions at the pixel level. However, this kind of segmentation alone generally does not separate adjacent distress instances. This can matter when a pavement image contains multiple cracks, mixed distress classes, or ambiguous alligator patterns that may be annotated as either one connected distress region or several individual cracks.

2.3 Object Detection and Hybrid Pipelines↩︎

Object detectors such as Faster R-CNN [23], [24], YOLO-family models, CenterNet, and EfficientDet have been widely used for pavement distress localization because they are efficient and can handle multiple classes. Mandal et al. [4] compared deep learning frameworks for pavement distress classification and detection using YOLO, CenterNet, and EfficientDet-style detectors. Hu et al. [25] investigated deep learning models for pavement crack detection. More recent detector-oriented studies [7], [26] have improved speed and robustness through lightweight multi-scale feature fusion and YOLO modifications.

Hybrid pipelines have attempted to combine detection and segmentation. Feng et al. [14] integrated SSD-style localization with U-Net segmentation for pavement crack detection and surface-feature measurement. Liu et al. [27] proposed a two-step CNN in which a YOLOv3-based detector first identifies candidate regions and a modified U-Net then segments cracks within those regions. These approaches highlight the practical value of segmentation whenever crack geometry or area is needed.

2.4 Instance Segmentation for Pavement Distress↩︎

Instance segmentation models aim to retain the localization advantages of detectors while producing masks for each detected object. While Mask R-CNN [16] extends Faster R-CNN with a parallel mask branch for each Region of Interest (RoI), Feature Pyramid Networks (FPNs) [28] improve multi-scale detection by combining features at different resolutions. Pavement-specific instance-segmentation work has also emerged, including YOLOv7-WMF [15] with connected feature fusion for pavement crack instance segmentation and SparseInst-CDSM [29] for real-time crack detection.

Most segmentation studies focus on cropped regions, single-crack scenes, or datasets collected under narrower imaging conditions than those encountered in routine field acquisition. The current study is positioned in this gap by evaluating Mask R-CNN instance segmentation on a custom field-collected roadway dataset with four pavement-distress categories.

3 Dataset↩︎

3.1 Acquisition Protocol↩︎

The UWGB-StreetCrack dataset was collected from roadway imagery using an iPhone 15 Pro Max mounted on the front of a vehicle, as shown in Fig. 1. The phone recorded videos while the vehicle traversed local roads. The videos were transferred to a computer and converted into still frames. A Python-based extraction step was used to reduce repeated coverage of the same roadway regions, so the resulting images contained only unique or minimally overlapping scenes. The set of images was then reviewed manually by 3 research assistants to remove duplicates, unclear frames, and blurry frames before annotation.

Figure 1: Vehicle-mounted smartphone used to collect field pavement imagery for UWGB-StreetCrack.

3.2 Annotation Taxonomy and Quality Control↩︎

The cleaned images were annotated in Label Studio (a free and open-source web-based tool for annotating images with machine learning labels) [30] using polygon masks, as illustrated in Fig. 2. The four-class taxonomy used in the stored annotation files consists of longitudinal, transverse, and alligator cracks, and potholes. Longitudinal cracks run approximately parallel to the roadway direction, whereas transverse cracks run approximately perpendicular to it. Alligator cracks are interconnected crack networks associated with repeated loading or structural failure, and potholes are bowl-shaped depressions often associated with the progression of untreated cracking [31]. No fifth “block” category is present in the annotation schema analyzed for this manuscript.

Figure 2: Polygon-based pavement distress annotation in Label Studio. The workflow produced COCO-style polygon annotations for the four target classes.

Each valid annotation includes a class label, a bounding box, and one or more polygon segments. The polygon representation was essential because the target task is instance segmentation rather than box-only detection. Annotation ambiguity remained a challenge: stains, markings, manhole edges, and faint linear texture can resemble cracks, and annotators may disagree about whether a connected pattern should be labeled as one alligator-crack instance or as multiple longitudinal and transverse cracks. To ensure consistent classification and reduce inter-annotator disagreement, all research assistants were trained using the PASER (Pavement Surface Evaluation and Rating) manual [32]. The PASER manual provides standardized visual criteria for distinguishing pavement distress types. Annotators referenced these standardized definitions when classifying ambiguous cases, ensuring systematic and consistent annotation throughout the dataset.

Table 1: This table summarizes the dataset splits and class-level instance counts.
Split Images Instances Longitudinal Transverse Alligator Pothole
Train + validation 1309 1484 639 698 82 65
Test split 231 261 112 123 14 12

Table 1 summarizes the total number of images for each pavement distress type, including the available split statistics. The stored training and validation splits contain 1,309 images and 1,484 labeled distress instances. The current study used a separate held-out test partition with 231 images and 261 labeled instances to solely test the performance of the trained models.

4 Methodology↩︎

In this study, we designed a comprehensive experimental pipeline spanning data preparation, model training, and comparative evaluation. Our approach leveraged Mask R-CNN implemented in Detectron2, a mature and widely adopted framework for instance segmentation. We evaluated five backbone variants (ResNet-50, ResNet-101, and ResNeXt-101 with different feature-extraction strategies) using consistent hyperparameters to identify the most effective architecture. To contextualize our results and demonstrate the advantages of instance segmentation, we established two baseline comparisons: (1) a detection-based baseline using CSPDarknet53-based YOLO [4], which localizes objects but cannot produce pixel-level masks, and (2) a segmentation-based baseline using DeepSegmentor [21], [33], which achieves pixel-level precision but only on manually cropped image regions. This comparative approach highlights both the strengths of instance segmentation and the specific limitations of prior approaches. The following subsections detail the data processing pipeline, architectural choices, training protocols, and inference configuration used throughout.

4.1 Instance-Segmentation Pipeline↩︎

The current study leveraged the Mask R-CNN [16] deep learning framework implemented in Detectron2 [34]. The pipeline consists of four stages: (i) field image curation and annotation, (ii) binary instance masks conversion and data loading, (iii) fine-tuning of Mask R-CNN variants initialized from COCO (Common Objects in Context)-pretrained checkpoints, and (iv) evaluation.

During training, Label Studio polygon coordinates were interpreted as closed contours and rasterized into binary masks using the standard COCO polygon mechanism available in Detectron2. Invalid polygons with fewer than three vertices were excluded. Each valid polygon generated one instance mask aligned to the image coordinate system. No offline cropping was used for the full-image Mask R-CNN.

4.2 Preprocessing and Augmentation↩︎

All images were resized while preserving the aspect ratio. The shorter and the larger sides of the images were normalized to 800 and 1333 pixels respectively for the selected Mask R-CNN family. This process was performed according to model-zoo protocol of Detectron2. Zero padding, when needed for batching and stride-compatible tensors, was applied only after resizing.

The conservative augmentation policy was intentional owing to the substantial appearance variation of field images. Training used random horizontal flipping with probability (value 0.5) and multi-scale resizing. No other augmentation (e.g., color jitter, blur, CutMix, mosaic augmentation, random rotation, or synthetic perturbation) was used in training.

4.3 Mask R-CNN Variants↩︎

Five Mask R-CNN backbone variants were considered in the project pipeline. Table 2 summarizes the model families and their relative backbone complexity. Among all the variants (e.g., ResNet-C4, ResNet-DC5, ResNet-FPN) available in Detectron2, better test performance was achieved via ResNet-50 FPN and ResNet-101 FPN.

Table 2: Architectural summary of the variants considered in the project. Parameter counts and reference FLOP values correspond to the backbone family only; End-to-End detector cost also depends on FPN construction, proposal count, and input image size.
Variant Backbone / neck Approx. backbone parameters Reference backbone FLOPs Relative detector cost
R50-FPN 3x ResNet-50 + FPN 25.6M 4.1 GFLOPs Medium
R101-C4 ResNet-101 + C4 features 44.5M 7.8 GFLOPs Medium-high
R101-DC ResNet-101 + dilated convolution features 44.5M 7.8+ GFLOPs High
R101-FPN 3x ResNet-101 + FPN 44.5M 7.8 GFLOPs High
X101-FPN ResNeXt-101 + FPN 88.8M 16.5 GFLOPs Very high

4.4 Mask R-CNN Training Protocol↩︎

Training and inference were performed on an Ubuntu 24.04 LTS workstation equipped with an NVIDIA GeForce RTX 4080 SUPER GPU, an Intel Core i9 CPU, and 32 GB DDR4 RAM. The software environment included PyTorch 2.4.0 with CUDA 12.1, Detectron2, OpenCV 4.5.2, NumPy, and Matplotlib.

For the reported Mask R-CNN experiments, the annotated images were divided into training, validation, and test subsets following a 70/15/15 split, with the held-out test partition containing 231 images containing 261 labeled crack instances. All Mask R-CNN variants were initialized from COCO-pretrained model-zoo weights and fine-tuned with stochastic gradient descent using momentum 0.9, weight decay 0.0001, initial learning rate 0.001, and global batch size 8. Across all evaluated variants, training followed a learning rate decay set to a factor of 0.1 at epoch 24 and epoch 33. The model yields better performance at epoch-schedule 40 out of 100 initial epoch settings. Model selection was performed on the validation partition, and the selected checkpoint was used for test-time analysis.

4.5 Baseline Detector Evaluation↩︎

To compare with a representative Detector algorithm, the CSPDarknet53-based YOLO model was adapted to UWGB-StreetCrack. Unlike the current study, this model is incapable of pixel-level segmentation. The dataset was reorganized in YOLO format, with each image paired with a label file containing class identifiers and normalized bounding-box coordinates in the form \[\langle\text{class}\_\text{id}\rangle \; \langle x_\text{center}\rangle \; \langle y_\text{center}\rangle \; \langle w\rangle \; \langle h\rangle .\]

Several modifications were applied before training: malformed label entries were removed or reformatted, empty or invalid labels were removed, the number of classes was set to four, deprecated NumPy usage was replaced, tensor device mismatches were fixed, source-code indentation and formatting inconsistencies were resolved, and the evaluation pipeline was corrected to convert model outputs into the expected metric format used in this study.

The detector was trained from scratch since the pretrained weights were associated with other datasets that were unsuitable for direct comparison. Training used 640 \(\times\) 640 input resolution, batch size 16, the default optimizer configuration from the implementation, and 100 epochs, with additional experiments extending beyond 100 epochs. The training behavior indicated convergence around 70–80 epochs; extending training to 200 epochs did not yield significant performance improvements. The result of the best performing model is reported in Section 6.

4.6 Baseline Segmentation-Based Evaluation↩︎

In addition, we also considered another baseline segmentation-based model named DeepSegmentor - a CNN-based pixel-level crack segmentation approach originally designed to predict crack regions from image patches. Unlike the CSPDarknet53-based YOLO model and the current study, DeepSegmentor is incapable of processing full roadway images, however, it is capable of crop-level segmentation. Therefore, crack regions were manually cropped from full roadway images for pixel-level prediction. The resulting masks were used to compute the detected crack-area percentage and compare it with the ground-truth annotated crack area.

It is noteworthy that, unlike the two baseline models, the proposed Mask R-CNN framework is an end-to-end model that performs simultaneous localization, classification, and pixel-level segmentation on full roadway images.

4.7 Inference Configuration↩︎

At inference time, same training image properties (i.e., aspect ratio, size) were used without stochastic augmentation. The Region Proposal Network (RPN) used anchor sizes of 32, 64, 128, 256, and 512 pixels and aspect ratios of 0.5, 1.0, and 2.0. The RPN non-maximum suppression threshold was 0.65, and the final detection-stage non-maximum suppression threshold was 0.5. The final confidence threshold was set to 0.75 on the validation partition because it provided the best precision-recall balance; it was then held fixed for test evaluation.

5 Evaluation Protocol↩︎

For the evaluation purpose, we compared our results in both object-detection (object-level) and crack segmentation (pixel-level). The Mask R-CNN precision, recall, and F1 values should therefore be interpreted as object-level performance. Predicted objects were matched to the ground-truth objects by considering bounding-box Intersection over Union (IoU). \[\mathrm{IoU}=\frac{|B_p \cap B_g|}{|B_p \cup B_g|}, \label{eq:iou}\tag{1}\] where \(B_p\) is the predicted bounding box and \(B_g\) is the ground-truth bounding box. A prediction was counted as a true positive if its class label matched the ground-truth class and its bounding-box IoU was at least 0.1. Each predicted instance could be assigned to at most one ground-truth instance. Unmatched predictions were counted as false positives, and unmatched ground-truth instances were counted as false negatives.

Precision, recall, and F1 score were computed as \[\mathrm{Precision}=\frac{TP}{TP+FP}, \label{eq:precision}\tag{2}\] \[\mathrm{Recall}=\frac{TP}{TP+FN}, \label{eq:recall}\tag{3}\] \[F1=2\cdot \frac{\mathrm{Precision}\cdot \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}. \label{eq:f1}\tag{4}\]

The detected crack area was computed at the pixel level by summing predicted or annotated mask pixels and normalizing by the total image area across the evaluated set.

6 Results↩︎

Table 3 summarizes the performance values for the evaluated pavement-crack analysis models on the UWGB-StreetCrack dataset. The results are reported using precision, recall, F1 score, and the percentage of detected crack area.

Table 3: Comprehensive Performance Comparison of Models on Pavement Crack Dataset.
Model Precision (%) Recall (%) F1 (%) Detected area (%)
Ground truth 2.170
Mask R-CNN ResNet-50 FPN 3x 75.56 89.01 81.73
Mask R-CNN ResNet-101 FPN 3x 84.23 90.04 87.04 2.164
YOLO (CSPDarknet53) [4] 27.5 20.7 23.62
DeepSegmentor [21], [33] 2.130

6.1 Model Performance↩︎

Among the evaluated Mask R-CNN variants, the ResNet-101 FPN 3x backbone achieved the best overall detection performance. It obtained a precision of 84.23%, a recall of 90.04%, and an F1 score of 87.04%. In comparison, the ResNet-50 FPN 3x variant achieved 75.56% precision, 89.01% recall, and an F1 score of 81.73%. The deeper ResNet-101 backbone therefore improved precision by 8.67 percentage points and F1 score by 5.31 percentage points, while also providing a modest improvement in recall. This indicates that the higher-capacity ResNet-101 FPN backbone was more effective at suppressing false detections while maintaining strong crack-detection coverage.

6.2 Model Comparison↩︎

6.2.1 Object-level detection↩︎

Compared with the YOLO-based CSPDarknet53 detector, the proposed Mask R-CNN ResNet-101 FPN 3x model showed substantially higher performance values across all three evaluation metrics. CSPDarknet53 obtained 27.5% precision, 20.7% recall, and an F1 score of 23.62%, whereas the proposed Mask R-CNN ResNet-101 FPN 3x model achieved 84.23% precision, 90.04% recall, and 87.04% F1 score. It is observable that the current study outperformed the CSPDarknet53 by 56.73 percentage points in precision, 69.34 percentage points in recall, and 63.42 percentage points in F1-score.

6.2.2 Pixel-level segmentation↩︎

The detected-area results further highlight the practical advantage of the proposed instance-segmentation framework for pavement-crack assessment. The ground-truth crack area accounted for 2.170% of the evaluated image area. The proposed Mask R-CNN ResNet-101 FPN 3x model estimated 2.164% crack area, producing a trivial difference (i.e., 0.006 percentage points) from the ground truth. In comparison, DeepSegmentor estimated 2.130% detected-area where the difference (i.e., 0.040 percentage points) is larger than the proposed model in the current study. Note that DeepSegmentor can only be evaluated as a crop-level segmentation method that requires input images to be manually cropped around crack-containing regions. When applied directly to full roadway scenes, its performance is limited by real-world visual confounders like shadows, pavement texture, stains, road markings, lane lines, and other non-crack objects. In contrast, the proposed Mask R-CNN-based framework operates on full roadway images: it first localizes candidate distress regions and then predicts pixel-level masks for the detected instances. Therefore, the proposed model not only produced a closer aggregate crack-area estimate than DeepSegmentor under the reported evaluation, but also provided a more deployable end-to-end solution for realistic pavement distress imagery.

Overall, the results showed that the proposed Mask R-CNN ResNet-101 FPN 3x model provides the strongest performance among the compared models. It achieved the highest precision, recall, and F1 score for crack detection and classification, while also producing a higher detected-area percentage that closely matched the ground-truth. These findings support the effectiveness of instance segmentation as a practical framework for automated pavement-crack analysis.

6.3 Qualitative Error Analysis↩︎

Figures 3 (a), 3 (b), 3 (c), and 3 (d) show representative failure cases. Red boxes denote model predictions and green boxes denote ground-truth annotations. Painted road markings and oil stains produced false positives because their elongated or high-contrast appearances that resembled cracks. Manhole-cover scenes were especially ambiguous because faint linear structures adjacent to the cover could be interpreted as either pavement distress or non-distress artifacts. Alligator cracks introduced a different difficulty: the model sometimes predicted one connected distress region while the ground truth separated the same region into multiple instances. For example, in Figure 3 (d), the annotator failed to integrate the region inside the small green bounding box as connected to the annotated alligator crack in the image (big green box). However, the model was successful in correctly predicting the complete distress area (large red bounding box). In such a case, the error reflects the annotation flaw rather than model failure. However, this type of error is minimal in the dataset.

a

b

c

d

Figure 3: Representative failure cases used for qualitative analysis. Red boxes denote model predictions and green boxes denote ground-truth annotations..

These examples highlight why a segmentation benchmark must pair model development with rigorous annotation policy. Without strict guidance, annotators may draw different masks for the same visual structure, especially when a crack network can be interpreted as either one alligator-crack instance or multiple linear cracks. Such ambiguity affects model performance.

7 Conclusion↩︎

This paper presented a Mask R-CNN instance-segmentation framework for automated pavement distress assessment and evaluated it on UWGB-StreetCrack dataset, a custom field-collected dataset of 1,540 smartphone-based roadway images with polygon-level annotations for four pavement distress types. Our results demonstrate that instance segmentation achieves both the geometric precision of semantic segmentation and the multi-instance capability of detection-based approaches—critical requirements for maintenance-relevant quantification. The best-performing ResNet-101 FPN backbone achieved 84.23% precision, 90.04% recall, and 87.04% F1 score. More importantly, the model’s aggregated crack-area estimate (2.164%) closely matched the ground truth (2.170%), demonstrating practical utility for area-based maintenance decisions. Comparison with two reference models confirmed instance segmentation’s superiority: CSPDarknet53 YOLO achieved only 27.5% precision and 20.7% recall, while crop-based DeepSegmentor required manual image preprocessing and showed larger area estimation error (0.040%). Despite strong performance, this study exposed a critical limitation that must be addressed before deployment as an industry standard. The annotation ambiguity, particularly in alligator cracking patterns where annotators disagreed on instance boundaries, directly affected model performance. However, the achieved results position instance segmentation as a practical foundation for full-scene pavement distress assessment and aggregate crack-area estimation. By bridging the precision requirements of maintenance professionals and the technical capabilities of deep learning, this work contributes to the transition from subjective visual inspection toward objective, automated pavement assessment systems that can support evidence-based infrastructure investment decisions.

Acknowledgment↩︎

This work was supported by WiSys through a Spark grant (Award No. 102-24600-4-AAN5138). The authors also thank the University of Wisconsin–Green Bay and the student researchers who contributed to data collection and annotation for UWGB-StreetCrack.

Data Governance Note↩︎

The UWGB-StreetCrack dataset supporting this study is not yet publicly released, as it remains in active use for ongoing research. It is available from the corresponding author upon reasonable request. A public release is planned following further publications and a privacy review of the imagery.

References↩︎

[1]
T. Van Dam, L. Sutter, K. Smith, M. Wade, and K. Peterson, “Guidelines for detection, analysis, and treatment of materials-related distress in concrete pavements. Volume 2, guidelines description and use,” Federal Highway Administration, 2002.
[2]
D. Draine and E. Wei, “State and Local Governments Face $105 Billion in Deferred Maintenance for Roads and Bridges.” 2025, Accessed: May 25, 2026. [Online]. Available: https://pewtrsts.org/4d6U02N.
[3]
L.-L. Huang, J.-D. Lin, W.-H. Huang, C.-H. Kuo, and M.-Y. Huang, “Application of automated pavement inspection technology in provincial highway pavement maintenance decision-making,” Applied Sciences, vol. 14, no. 15, p. 6549, 2024.
[4]
V. Mandal, A. R. Mussah, and Y. Adu-Gyamfi, “Deep learning frameworks for pavement distress classification: A comparative analysis,” in 2020 IEEE international conference on big data (big data), 2020, pp. 5577–5583.
[5]
Z. Fan, Y. Wu, J. Lu, and W. Li, “Automatic pavement crack detection based on structured prediction with the convolutional neural network,” arXiv preprint arXiv:1802.02208, 2018.
[6]
Y. Du, N. Pan, Z. Xu, F. Deng, Y. Shen, and H. Kang, “Pavement distress detection and classification based on YOLO network,” International Journal of Pavement Engineering, vol. 22, no. 13, pp. 1659–1672, 2021.
[7]
Y. Wang et al., “GSBYOLO: A lightweight multi-scale fusion network for road crack detection in complex environments,” Scientific Reports, vol. 15, no. 1, p. 26615, 2025.
[8]
D. Tabernik, M. Šuc, and D. Skočaj, “Automated detection and segmentation of cracks in concrete surfaces using joined segmentation and classification deep neural network,” Construction and Building Materials, vol. 408, p. 133582, 2023.
[9]
M. Nie and K. Wang, “Pavement distress detection based on transfer learning,” in 2018 5th international conference on systems and informatics (ICSAI), 2018, pp. 435–439.
[10]
W. Wang, B. Wu, S. Yang, and Z. Wang, “Road damage detection and classification with faster r-CNN,” in 2018 IEEE international conference on big data (big data), 2018, pp. 5220–5223.
[11]
R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 580–587.
[12]
S. L. Lau, E. K. Chong, X. Yang, and X. Wang, “Automated pavement crack segmentation using u-net-based convolutional neural network,” Ieee Access, vol. 8, pp. 114892–114899, 2020.
[13]
S. Bang, S. Park, H. Kim, and H. Kim, “Encoder–decoder network for pixel-level road crack detection in black-box images,” Computer-Aided Civil and Infrastructure Engineering, vol. 34, no. 8, pp. 713–727, 2019.
[14]
X. Feng et al., “Pavement crack detection and segmentation method based on improved deep learning fusion model,” Mathematical Problems in Engineering, vol. 2020, no. 1, p. 8515213, 2020.
[15]
G. Ye et al., “Pavement crack instance segmentation using YOLOv7-WMF with connected feature fusion,” Automation in Construction, vol. 160, p. 105331, 2024.
[16]
K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-CNN,” in 2017 IEEE international conference on computer vision (ICCV), 2017, pp. 2980–2988, doi: 10.1109/ICCV.2017.322.
[17]
Q. Zou, Y. Cao, Q. Li, Q. Mao, and S. Wang, “CrackTree: Automatic crack detection from pavement images,” Pattern Recognition Letters, vol. 33, no. 3, pp. 227–238, 2012.
[18]
Y. Shi, L. Cui, Z. Qi, F. Meng, and Z. Chen, “Automatic road crack detection using random structured forests,” IEEE Transactions on Intelligent Transportation Systems, vol. 17, no. 12, pp. 3434–3445, 2016, doi: 10.1109/TITS.2016.2552248.
[19]
L. Zhang, F. Yang, Y. D. Zhang, and Y. J. Zhu, “Road crack detection using deep convolutional neural network,” in 2016 IEEE international conference on image processing (ICIP), 2016, pp. 3708–3712.
[20]
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical image computing and computer-assisted intervention – MICCAI 2015, 2015, pp. 234–241.
[21]
Q. Zou, Z. Zhang, Q. Li, X. Qi, Q. Wang, and S. Wang, “Deepcrack: Learning hierarchical convolutional features for crack detection,” IEEE transactions on image processing, vol. 28, no. 3, pp. 1498–1512, 2018.
[22]
W. Liu, Y. Huang, Y. Li, and Q. Chen, “FPCNet: Fast pavement crack detection network based on encoder-decoder architecture,” arXiv preprint arXiv:1907.02248, 2019.
[23]
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-CNN: Towards real-time object detection with region proposal networks,” in Advances in neural information processing systems, 2015, vol. 28, [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf.
[24]
M. G. S. Murshed, K. Bahmani, S. Schuckers, and F. Hussain, “Deep age-invariant fingerprint segmentation system,” IEEE Transactions on Biometrics, Behavior, and Identity Science, vol. 7, no. 3, pp. 313–330, 2025, doi: 10.1109/TBIOM.2024.3506926.
[25]
G. X. Hu, B. L. Hu, Z. Yang, L. Huang, and P. Li, “Pavement crack detection method based on deep learning models,” Wireless Communications and Mobile Computing, vol. 2021, no. 1, p. 5573590, 2021, doi: https://doi.org/10.1155/2021/5573590.
[26]
X. Liang and M. Gao, “The road crack detection algorithm improved based on YOLOv7,” in Proceedings of the 2024 international conference on generative artificial intelligence and information security, 2024, pp. 335–339.
[27]
J. Liu et al., “Automated pavement crack detection and segmentation based on two-step convolutional neural network,” Computer-Aided Civil and Infrastructure Engineering, vol. 35, no. 11, pp. 1291–1305, 2020.
[28]
T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in 2017 IEEE conference on computer vision and pattern recognition (CVPR), 2017, pp. 936–944, doi: 10.1109/CVPR.2017.106.
[29]
S.-J. Wang, J.-K. Zhang, and X.-Q. Lu, “Research on real-time detection algorithm for pavement cracks based on SparseInst-CDSM,” Mathematics, vol. 11, no. 15, p. 3277, 2023.
[30]
M. Tkachenko, M. Malyuk, A. Holmanyuk, and N. Liubimov, Open source software available from https://github.com/HumanSignal/label-studioLabel Studio: Data labeling software.” 2020-2025, [Online]. Available: https://github.com/HumanSignal/label-studio.
[31]
J. S. Miller, W. Y. Bellinger, et al., “Distress identification manual for the long-term pavement performance program,” United States. Federal Highway Administration. Office of Infrastructure …, 2003.
[32]
D. Walker, “Pavement surface evaluation and rating paser manual asphalt roads rating 10,” Transportation Information Center. University of Wisconsin, Madison, 2013, [Online]. Available: https://www.ctt.mtu.edu/sites/default/files/resources/paser/asphaltpaser.pdf.
[33]
Y. Liu, Accessed: 2026-05-31DeepSegmentor: A PyTorch implementation of DeepCrack and RoadNet projects.” https://github.com/yhlleo/DeepSegmentor, 2019.
[34]
Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2.” https://github.com/facebookresearch/detectron2, 2019.