1 Introduction↩︎

Object counting in computer vision aims to accurately quantify any particular object in an image, and is applicable to logistics, surveillance, urban planning and more. Specialised counters such as ones for crowd-counting [1] and vehicles [2] exist. These specialised counters can effectively count large quantities of objects but are limited to specific object classes, a limitation that the human counting ability does not possess. Other general counters can count arbitrary object classes, but require additional text prompts or exemplars. Exemplars are particularly inefficient as they require humans to manually produce annotations like bounding boxes to isolate the desired object to count.

To this end, Liu et al. [3] developed Counting TRansformer (CounTR), a transformer-based model that can count any arbitrary class of objects in an effort to mimic our human counting ability. The authors were motivated by findings from Lu et al. in [4], where the object counting problem was posed as a matching problem for image patches, and self-similarity was an found to be an important element for counting. Attention mechanisms were built into CounTR’s architecture as a result, which seemed to give CounTR an edge over several state-of-the-art (SOTA) models.

However, CounTR, like many other models, failed where object structural information was key. For instance, the counts of sunglasses would be double the true count as each lens would be treated as a separate object. These cases necessitated test-time augmentation, which involved performing transformations to the query image to obtain several augmented images, performing counting on each image, then returning the average count as the final one. This introduced additional steps into the pipeline and increased computational demands. We hypothesize that this limitation may stem from the feature interaction module in CounTR [3], which did not sufficiently capture structural information.

We address this limitation by strengthening the visual representation at the very first stage of the pipeline. Specifically, we replace the original encoder with an MAE-pretrained ViT-B/16 and extract features from multiple intermediate Transformer blocks, which are then reassembled into a multi-scale pyramid and spatially refined (Dense Prediction Transformers (DPT) [5] + FeatUp [6]). We call this model UpCount. To convert these spatially refined features into final predictions, we adopt a proposal–verification counting head that detects repeated visual patterns and outputs a density map and count. This design preserves fine structural cues while retaining high-level semantics, improving zero-shot counting of structurally complex objects, while also eliminating the exemplar boxes or text prompts at inference time.

In the reference-free setting (no exemplar boxes or text prompts at inference), UpCount achieves strong performance on FSC-147 with a validation MAE/RMSE of 13.62/60.07 and a test MAE/RMSE of 12.39/100.89. On CARPK, UPCount reaches 6.27 MAE and 8.79 RMSE, substantially improving over exemplar-guided methods.

Our main contributions are as follows:

• We propose UpCount, a reference-free, class-agnostic counting model with strong spatial generalization that requires no exemplar boxes, or text prompts at inference.

• We strengthen the visual representation by extracting multi-layer ViT features and reassembling them into a refined multi-scale pyramid (DPT [5] + FeatUp [6]) for improved structural and spatial sensitivity.

• We validate UpCount on FSC-147 [7] and CARPK, showing improved performance over representative reference-free baselines and strong transfer to vehicle counting.

2 Related Work↩︎

2.1 Class-specific Object Counting↩︎

Object counting is commonly grouped into class-specific and class-agnostic settings. Class-specific methods target predefined categories (e.g., people [1] or cars [2]) and are typically either detection-based (counting bounding boxes) or regression-based (summing a predicted density map) [8]. Their reliance on fixed classes limits extension to arbitrary or multiple object types within an image.

2.2 Class-agnostic Object Counting↩︎

Early class-agnostic counters such as GMN [4] and FamNet [7] rely on annotated exemplars and substantial training data (e.g., GMN leverages large-scale ILSVRC detection videos), which can be impractical to collect. Exemplar-free approaches reduce this burden but often introduce other constraints, e.g., RepRPN [9] primarily counts dominant classes, while text-conditioned methods generate or select class-relevant regions from prompts (VAE-based [8], CLIP-Count [10], CounTX [11] using CLIP [12]), still requiring user-specified text and limiting fully automatic counting.

3 Method↩︎

3.1 Architecture↩︎

We propose a spatially-aware, class-agnostic object counting model that predicts a full-resolution density map and the corresponding count without category labels, text prompts, or exemplar boxes. It detects repeated patterns directly from the input image built from four main components:

3.1.1 MAE-Pretrained ViT Encoder↩︎

We employ a vanilla ViT-B/16 encoder initialized through masked autoencoding. Given an input image \(\mathcal{X}_i \in \mathbb{R}^{3 \times 384 \times 384}\), the image is partitioned into non-overlapping \(16 \times 16\) patches, producing a \(24 \times 24\) grid of \(M=576\) visual tokens.

The encoder consists of 12 Transformer blocks with 12 attention heads. Instead of using only the output of the final block, we extract layer-normalized representation from four intermediate depths: \[\mathbf{Z}^{(l)} = \Phi_{\mathrm{ViT\text{-}Enc}}^{(l)} \left(\mathcal{X}_i\right) \in \mathbb{R}^{M \times D}, \qquad l \in \{2,5,8,11\}.\] where \(l\) denotes the zero-based Transformer-block index. Thus, the selected indices correspond to the 3rd, 6th, 9th, and 12th Transformer blocks, respectively. The complete multi-level encoder output is written as \[\mathcal{F}_{\mathrm{ViT}} = \left\{ \mathbf{Z}^{(2)}, \mathbf{Z}^{(5)}, \mathbf{Z}^{(8)}, \mathbf{Z}^{(11)} \right\}, \qquad \mathbf{Z}^{(l)} \in \mathbb{R}^{576 \times 768}.\] Each token sequence is subsequently reshaped into a spatial feature map \[\mathbf{F}^{(l)} = \mathcal{R}\left(\mathbf{Z}^{(l)}\right) \in \mathbb{R}^{D \times 24 \times 24},\] where \(\mathcal{R}(\cdot)\) denotes the token-to-grid reshaping operation. The resulting spatial feature maps retain representations at different semantic depths while sharing the same spatial resolution.

3.1.2 DPT Pyramid Reassembly↩︎

All intermediate ViT features have the same \(24 \times 24\) spatial resolution. To construct a hierarchical representation, we employ a DPT-style pyramid reassembly module [5]. The projected features from blocks \(\{2,5,8,11\}\) are reassembled at resolutions of \(96 \times 96\), \(48 \times 48\), \(24 \times 24\), and \(12 \times 12\), respectively.The lower-resolution features are progressively upsampled and combined with the higher-resolution features. The complete operation is expressed as \[\mathbf{F}_{\mathrm{DPT}} = \Phi_{\mathrm{DPT}} \left( \mathbf{F}^{(2)}, \mathbf{F}^{(5)}, \mathbf{F}^{(8)}, \mathbf{F}^{(11)} \right).\] The resulting feature map is \[\mathbf{F}_{\mathrm{DPT}} \in \mathbb{R}^{C_o \times 96 \times 96},\] where \(C_o\) denotes the channel dimension of the reassembled DPT feature map and is set to 64 in our implementation. In parallel, the projected final block feature is retained at \(24 \times 24\) resolution as the semantic input to the subsequent FeatUp refinement module.

3.1.3 FeatUp Refinement↩︎

We employ a FeatUp [6] mechanism to learn joint bilateral upsample (JBU) to recover fine spatial structures that may be lost during ViT patch tokenization. A lighweight convolutional stem first extracts a stride-four RGB guidance feature: \[\mathbf{F}_{\mathrm{RGB}} = \Phi_{\mathrm{RGB}}\left(\mathcal{X}_i\right) \in \mathbb{R}^{C_o \times 96 \times 96}.\] The projected final-block ViT feature is then upsampled from \(24 \times 24\) to \(96 \times 96\) using learned joint bilateral kernels conditioned on the RGB guidance: \[\mathbf{F}_{\mathrm{JBU}} = \Phi_{\mathrm{JBU}} \left( \widetilde{\mathbf{F}}^{(11)}, \mathbf{F}_{\mathrm{RGB}} \right) \in \mathbb{R}^{C_o \times 96 \times 96}.\] Finally, the FeatUp representation is fused with the DPT feature and subsequently combined with the RGB detail feature: \[\begin{align} \mathbf{F}_{\mathrm{sem}} &= \mathbf{F}_{\mathrm{DPT}} + \Phi_{\mathrm{fuse}} \left( \mathbf{F}_{\mathrm{DPT}} \,\Vert\, \mathbf{F}_{\mathrm{JBU}} \right),\\ \mathbf{F}_{\mathrm{ref}} &= \Phi_{\mathrm{detail}} \left( \mathbf{F}_{\mathrm{sem}} \,\Vert\, \mathbf{F}_{\mathrm{RGB}} \right), \end{align}\] where \(\Vert\) denotes channel-wise concatenation and \(\mathbf{F}_{\mathrm{ref}} \in\mathbb{R}^{C_o \times 96 \times 96}\) is the spatially refined feature used by the subsequent counting modules.

3.1.4 Proposal-Verification Counting↩︎

The refined feature \(\mathbf{F}_{\mathrm{ref}}\) is processed by a proposal-verification module. The module computes non-local similarities between semantic ViT tokens and aggregates their top-\(K\) matches to identify repeated visual patterns. \[\left( \mathbf{F}_{\mathrm{count}}, \mathbf{S}_{\mathrm{rep}} \right) = \Phi_{\mathrm{rep}} \left( \widetilde{\mathbf{F}}^{(11)}, \mathbf{F}_{\mathrm{ref}} \right),\] where \(\mathbf{F}_{\mathrm{count}}\) is the repetition-conditioned counting feature and \(\mathbf{S}_{\mathrm{rep}}\) represents the spatial repetition evidence.

A proposal head predicts a non-negative dense proposal map, while a verification head evaluates each proposal using both the counting feature and repetition signal: \[\begin{align} \mathbf{P}_i &= \operatorname{Softplus} \left( \Phi_{\mathrm{prop}} \left(\mathbf{F}_{\mathrm{count}}\right) \right),\\ \mathbf{V}_i &= \sigma \left( \Phi_{\mathrm{ver}} \left( \mathbf{F}_{\mathrm{count}} \,\Vert\, \mathbf{S}_{\mathrm{rep}} \right) \right), \end{align}\] where \(\mathbf{P}_i\) is the proposal density, \(\mathbf{V}_i\) is the verification map, and \(\Vert\) denotes channel-wise concatenation. Both maps are bilinearly upsampled to the input resolution.

The verified proposal density is adjusted by a learned spatial correction map \(\mathbf{A}_i\) and a global count-calibration factor \(c_i\). The final density map and predicted count are given by \[\widehat{\mathbf{D}}_i = c_i \left( \mathbf{P}_i \odot \mathbf{V}_i \odot \mathbf{A}_i \right), \qquad \widehat{N}_i = \sum_{p} \widehat{\mathbf{D}}_i(p),\] where \(\odot\) denotes element-wise multiplication.

3.2 Two-Stage Training Scheme↩︎

3.2.1 Masked Autoencoder Pretraining↩︎

We first adapt the ViT-B/16 encoder to FSC-147 using masked image reconstruction. The encoder is initialized from the full ImageNet-MAE checkpoint and further pretrained using only the FSC-147 training split. The reconstruction loss is averaged across all patches: \[\mathcal{L}_{\mathrm{MAE}} = \frac{1}{N} \sum_{i=1}^{N} \left\lVert \hat{\mathbf{x}}_i - \mathbf{x}_i \right\rVert_2^2\] where \(\mathbf{x}_i\) and \(\hat{\mathbf{x}}_i\) denote the target and reconstructed RGB values of patch \(i\).

3.2.2 Supervised Counting Fine-Tuning↩︎

After pretraining, the MAE decoder is discarded, and the pretrained encoder is connected to the DPT pyramid, FeatUp refinement module, and proposal-verification counting head. The complete architecture, including all ViT blocks, is then optimized using density-map supervision. The supervised objective is the following. \[\mathcal{L} = \mathcal{L}_{\mathrm{density}} + \lambda_{c}\mathcal{L}_{\mathrm{count}} + \lambda_{\log}\mathcal{L}_{\mathrm{log\text{-}count}} + \lambda_{v}\mathcal{L}_{\mathrm{verify}}\] where \(\mathcal{L}_{\mathrm{density}}\) is a foreground-weighted density regression loss, \(\mathcal{L}_{\mathrm{count}}\) enforces count-integral consistency, \(\mathcal{L}_{\mathrm{log\text{-}count}}\) balances errors across different count ranges, and \(\mathcal{L}_{\mathrm{verify}}\) supervises the verification response.

4 Experiment & Evaluation↩︎

4.1 Dataset & Metrics↩︎

FSC-147. The FSC-147 [7] dataset is widely used in class-agnostic counting and contains 6135 images spanning 147 object classes. The number of objects per image ranges from 7 to 3731, with an average of 56. FSC-147 provides a corresponding binary density map \(y_i\) for each training image \(X_i\) in \(D_{train}\).

CARPK. We also utilize CARPK[13] dataset which consists of 1448 bird’s-eye-view images of parking lots, with a total of 89,777 annotated cars.

Metrics. We use the standard Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) to evaluate the performance of the model.

4.2 Implementation Details↩︎

Pretraining Details. Pretraining is conducted for 500 epochs using AdamW, a batch size of 8, weight decay of 0.05, and a base learning rate of 1.5 \(\times\) \(10^{-4}\). The learning rate is linearly warmed up for 10 epochs and subsequently decayed using a cosine schedule.

Supervised Fine-Tuning Details. We perform separate 1000-epoch fine-tuning runs for FSC-147 and CARPK. Both datasets are initialized from the same FSC-147 pretrained MAE encoder; CARPK fine-tuning is not initialized from FSC-147 supervised fine-tuning.

FSC-147 fine-tuning uses a batch size of 26 and a base learning rate of 2 \(\times\) \(10^{-4}\), corresponding to an effective learning rate of approximately 2.03 \(\times\) \(10^{5}\). CARPK uses a batch size of 8 and an effective learning rate of 6.25 \(\times\) \(10^{-6}\). Both runs use AdamW, weight decay of 0.05, a 10-epoch linear warm-up, and cosine learning-rate decay, trained on a single NVIDIA RTX A6000 GPU.

Inference Details. Our model performs reference-less class-agnostic counting and does not require exemplar boxes, category labels, or textual queries at inference.

4.3 Performance Comparison↩︎

Baselines. We compare with representative counting methods covering different forms of inference guidance. For text-guided methods, we include T2ICount [14], CountSE [15], and QICA [16]. CounTR [3], LOCA [17], and DAVE [18] are included as representative few-shot and zero-shot counting methods. We also include CountGD [19] and CountingDINO [20] as multimodal, and training-free methods, respectively.

Evaluation results. Tab. 1 compares UPCount with representative text-guided, exemplar-guided, training-free, and reference-free counting methods on FSC-147. UPCount requires neither textual prompts nor visual exemplars during inference. Under this reference-free setting, it achieves an MAE of 13.62 on the validation set and 12.39 on the test set. These results outperform the reference-free variants of CounTR, LOCA, and DAVE in terms of MAE on both evaluation splits. UPCount also obtains the lowest test RMSE among the reference-free methods, reaching 100.89 compared with 108.01, 103.96, and 103.49 for CounTR, LOCA, and DAVE, respectively. However, DAVE retains a lower validation RMSE of 52.67.

Tab. ¿tbl:tab2? presents the results on the official CARPK test set following fine-tuning on its training set. UPCount achieves an MAE of 6.27 and an RMSE of 8.79 without requiring exemplar boxes during inference. It substantially outperforms FamNet and also improves upon GMN, which obtain MAE values of 18.19 and 7.48, respectively. These results indicate that the proposed representation transfers effectively from general class-agnostic counting to the more specialized vehicle-counting domain.

Qualitative results. We outline representative qualitative results in Fig. 2, where UpCount produces coherent density maps on FSC-147 for cluttered scenes and structurally complex objects, and maintains stable car counts on CARPK.

5 Conclusion↩︎

In this work, we presented UpCount, a class-agnostic object counter that improves counting on structurally complex objects by strengthening spatial feature representations. UpCount extracts multi-layer features from a ViT encoder, reassembles them into a multi-scale pyramid, and refines them using Dense Prediction Transformers and FeatUp before predicting a density map with a proposal–verification counting head. Experiments on FSC-147 demonstrate competitive performance in the reference-free setting, and results on CARPK indicate that the learned representation transfers effectively to vehicle counting.

References↩︎

---

1. ¹Singapore Univeristy of Technology and Design, Singapore.↩︎

2. ²The University of Canterbury, New Zealand.↩︎

3. ³Heineken GenAI Lab, Singapore.↩︎

4. Corresponding email at robert_wijaya@mymail.sutd.edu.sg.↩︎