1 Introduction↩︎

Multispectral and hyperspectral (MSI/HSI) videos provide recorded band-wise observations of a target and its surrounding background. Compared with RGB tracking, these observations offer additional cues for target–background discrimination, especially when the target and distractors share similar RGB appearance but respond differently across the recorded bands. This property is valuable in UAV and remote-sensing tracking scenarios with camouflage, low contrast, illumination variation, occlusion, and cluttered backgrounds. Fig. 1 illustrates this motivation with benchmark examples. RGB-similar target and background regions may exhibit different recorded-band responses, providing useful tracking evidence beyond RGB appearance.

Despite these advantages, MSI/HSI tracking remains challenging because spectral–spatial ambiguity varies substantially across frames and target states. Clear target regions may be localized with lightweight local discrimination, whereas mixed pixels, ambiguous boundaries, partial occlusion, low resolution, illumination variation, and spectrally similar distractors often require stronger contextual reasoning. Prior MSI/HSI tracking studies have improved spectral–spatial representation through band-attention modeling in BAE-Net [1], ensemble-style feature fusion in SEE-Net [2], Siamese spectral matching in SiamHYPER [3], dynamic template modeling in SPIRIT [4], and Transformer-based interaction in UNTrack [5]. However, these methods usually process different search regions with a similar modeling budget. This fixed-capacity assumption is mismatched with tracking videos, where the difficulty of each search region changes continuously over time. This observation motivates a different question: How can a multispectral tracker use recorded-band-induced target–background separability to allocate appropriate spectral–spatial modeling capacity for search regions with different ambiguity levels?

We therefore formulate MSI/HSI tracking as spectral–spatial complexity-aware capacity allocation, where different search regions are assigned different modeling budgets according to their ambiguity. SpecTrack implements this idea with three coupled modules: the Spectral Adaptive Mixture-of-Experts (SAMoE) module provides a capacity-ordered expert pool; the Spectral Prompt Router selects a sparse subset of experts using semantic context, spatial boundary response, and latent channel-variation cues; and the Shared Global Expert supplies common spectral–spatial context to stabilize sparse routing.

Recent MoE-style trackers, including MoETrack [6], SPMTrack [7], and HotMoE [8], have shown the potential of expert-based adaptation in visual tracking. In contrast, SpecTrack studies recorded-band-aware capacity allocation for MSI/HSI tracking, where latent channel variation after multispectral patch embedding provides search-region-level routing evidence. We further examine whether this routing behavior is meaningful through mechanism-aligned ablations and diagnostics.

We validate SpecTrack on MUST [5], MSITrack [9] and the 16-band HOTC20 benchmark [10]. GOT-10k [11] is reported as an RGB one-shot architectural generalization test. The main contributions are summarized as follows: The main contributions are summarized as follows:

• We formulate MSI/HSI single object tracking as spectral–spatial complexity-aware capacity allocation, where search regions with different ambiguity levels are assigned different modeling budgets rather than processed by a fixed-capacity path.

• We propose the SAMoE module, a capacity-ordered expert pool for MSI/HSI tracking. Its experts differ in latent rank, receptive field, and depth, enabling lightweight modeling for clear regions and stronger reasoning for ambiguous regions.

• We design a Spectral Prompt Router and a Shared Global Expert for stable sparse expert routing. The router selects experts from semantic, spatial, and latent channel-variation cues, while the shared expert provides common spectral–spatial context to reduce fragmented local decisions.

• We validate SpecTrack on MUST, MSITrack, and HOTC20 with mechanism-aligned ablations, routing diagnostics, and accuracy–efficiency analyses. GOT-10k is additionally used to examine RGB-domain architectural generalization.

2 Related Work↩︎

2.1 Spectral–Spatial Modeling for MSI/HSI Tracking↩︎

MSI/HSI tracking exploits recorded band-wise observations to improve target–background discrimination under low illumination, camouflage, similar color and cluttered background. Early hyperspectral trackers relied on handcrafted spectral descriptors, material unmixing, or correlation-filter formulations [10], [12]. Deep trackers then introduced band attention, ensemble fusion and cross-band feature interaction, including BAE-Net [1], SST-Net [13], SEE-Net [2] and CBFF-Net [14]. Recent methods such as SiamHYPER [3], SPIRIT [4], SENSE [15], MVP-HOT [16] and ProFiT [17] further show that spectral information should be modeled beyond naive channel expansion.

These studies mainly address how to extract or fuse spectral–spatial representations. SpecTrack studies a complementary problem: how recorded-band-induced evidence can decide the amount of feature-transformation capacity required by each search region. This distinction is important for tracking, because the spectral–spatial ambiguity varies across frames. Clear targets may only require low-cost local modeling, whereas mixed pixels, ambiguous boundaries, occlusion, illumination changes and spectrally similar distractors require stronger contextual modeling. Therefore, SpecTrack treats multispectral information not only as an input feature source, but also as routing evidence for search-region-level capacity allocation.

2.2 Adaptive Computation and MoE-style Tracking↩︎

MoE and conditional computation have been widely used for adaptive representation learning [18]–[20]. Recent tracking methods also introduce MoE-style designs, but they address different adaptation problems. MoETrack [6] studies RGB-T tracking under modality validity variation. It uses modality-related experts and confidence-aware fusion to decide when RGB, thermal, or fused predictions should dominate. SPMTrack [7] inserts tracking-specific MoE blocks into an RGB one-stream tracker to improve generic spatio-temporal relation modeling. HotMoE [8] explores sparse expert activation for HSI tracking, where a switch-style router activates suitable HSI expert models to reduce inference cost.

Fig. 2 summarizes these mechanism-level differences. The key distinction is the source and role of the routing evidence. MoETrack uses modality validity as the main cue for RGB-T decision fusion. SPMTrack uses visual relation features to improve scalable RGB tracking. HotMoE uses HSI features for sparse expert activation. In contrast, SpecTrack asks whether recorded MSI/HSI evidence inside a shared tracking backbone can estimate the spectral–spatial ambiguity of each search region and then allocate a suitable modeling budget.

This leads to three non-interchangeable design choices. First, SpecTrack uses a capacity-ordered expert ladder rather than homogeneous relation experts or independent modality heads. The experts are organized from low to high capacity through rank, receptive field and depth, so the router can assign lightweight experts to clear regions and stronger experts to ambiguous regions. Second, SpecTrack routes by a Spectral Prompt Router that combines semantic context, spatial boundary response and latent channel-variation cues after multispectral patch embedding. This differs from modality-validity routing in RGB-T tracking and from generic RGB relation routing. Third, SpecTrack introduces a Shared Global Expert to provide common latent context before aggregating sparse local experts, reducing fragmented decisions caused by local sparse routing.

A direct transplant of MoETrack, SPMTrack, or HotMoE is not fully equivalent because these methods are designed for different inputs, expert outputs, backbones and training protocols. To avoid conflating the proposed idea with implementation differences, we complement published-method comparisons with same-backbone mechanism controls in Sec. 4.4. These controls approximate the core alternatives under the same SpecTrack-B224 backbone, input resolution, training split and evaluation protocol: homogeneous MoE-style experts test whether multiple experts alone are sufficient; semantic-only routing tests generic visual routing; capacity experts without prompt routing test whether capacity diversity alone is enough; and local experts without the Shared Global Expert test whether sparse HSI-style expert activation can replace shared context modulation.

2.3 Remote-Sensing Interpretation Scope↩︎

Physical interpretation in remote sensing depends on sensor metadata, bandpass, radiometric calibration, illumination and acquisition geometry. Tracking benchmarks often provide digital numbers or normalized intensities. SpecTrack therefore uses qualitative spectral curves as recorded-band separability illustrations and uses latent channel variation as a routing descriptor after patch embedding. When calibrated reflectance and sensor response functions are available, the same routing idea can be extended toward sensor-aware material cues.

3 Method↩︎

3.1 Framework Overview↩︎

As illustrated in Fig. 3, SpecTrack follows a one-stream tracking paradigm and reorganizes the later encoder blocks around the spectral–spatial difficulty formulation. Template and search tokens interact in the shared encoder, while the prediction head consumes the final search-region tokens for localization. The core idea is to allocate expert capacity at the search-region feature level for heterogeneous search conditions.

The architecture has three roles. First, the capacity-ordered expert set provides a low-to-high representation ladder for search regions with different ambiguity levels. Second, the Spectral Prompt Router converts semantic, spatial and latent channel-variation cues into a sample-level Top-\(k\) expert selection. Third, the Shared Global Expert supplies common latent context before the sparse expert update is inserted back into the search tokens. These operations are performed after patch embedding in latent feature space; SAMoE is an adaptive feature-transformation block guided by recorded-band-induced cues.

For a SAMoE block at layer \(l\), let \(\hat{\mathbf{Z}}^l_s\in\mathbb{R}^{B\times L_s\times d}\) denote the search tokens after the standard self-attention sublayer. We reshape them into a latent search feature map \(x=\mathcal{U}(\hat{\mathbf{Z}}^l_s)\in\mathbb{R}^{B\times d\times H\times W}\), where \(L_s=HW\) and \(\mathcal{U}(\cdot)\) denotes token-to-map unflattening. SAMoE computes one residual update for the whole search sample and then flattens it back to the search token positions. The router is therefore sample conditioned at the search-region level.

3.1.0.1 Feature extraction network

The network takes a template image \(Z \in \mathbb{R}^{H_z \times W_z \times C_b}\) from the first frame and a search region \(S \in \mathbb{R}^{H_s \times W_s \times C_b}\) from the current frame, where \(C_b\) denotes the number of recorded spectral bands. The multispectral input is partitioned into patches and projected into a shared latent space: \[e_i = \mathrm{PE}(p_i), \qquad p_i \in \mathbb{R}^{P \times P \times C_b},\] where \(\mathrm{PE}(\cdot)\) denotes the patch projection layer. Template and search embeddings are concatenated, supplemented with positional encodings and processed by the one-stream Transformer encoder.

3.1.0.2 SAMoE block

In later encoder blocks, SpecTrack inserts SAMoE after self-attention and before the feed-forward network. The block is written as: \[\hat{\mathbf{Z}}^l = \mathbf{Z}^{l-1} + \mathrm{MSA}(\mathrm{LN}(\mathbf{Z}^{l-1})),\] \[\Delta x^l_s = \mathrm{SAMoE}(\mathcal{U}(\hat{\mathbf{Z}}^l_s)),\] \[\tilde{\mathbf{Z}}^l = \hat{\mathbf{Z}}^l + \alpha_l\,\mathcal{R}_s(\mathcal{F}(\Delta x^l_s)),\] \[\mathbf{Z}^l = \tilde{\mathbf{Z}}^l + \mathrm{FFN}(\mathrm{LN}(\tilde{\mathbf{Z}}^l)),\] where \(\mathcal{F}(\cdot)\) flattens a feature map back to tokens, \(\mathcal{R}_s(\cdot)\) inserts the update only into the search token positions and \(\alpha_l\) is a learnable residual scale. SAMoE updates the search tokens in this branch, while template tokens influence the search representation through the preceding one-stream attention.

3.1.0.3 Prediction head

The prediction head receives the final search region tokens and reshapes them to recover the two-dimensional feature map. Following OSTrack [21], we use a center point prediction head in which classification and regression branches estimate the score map, target size, offset and final bounding box.

3.2 Spectral Prompt Router↩︎

Effective expert assignment is critical for adaptive computation. Spatial semantics alone may be insufficient for camouflaged targets, mixed pixels, ambiguous boundaries and spectrally similar distractors. The Spectral Prompt Router therefore combines semantic context with a fixed spatial high-pass prompt and a latent channel-variation prompt.

Given \(x\in\mathbb{R}^{B\times d\times H\times W}\), the router first normalizes the feature and obtains a semantic descriptor: \[\bar{x}=\mathrm{LN}(x), \qquad z=\mathrm{GAP}(\bar{x})\in\mathbb{R}^{B\times d}. \label{eq:router95semantic}\tag{1}\] The spatial prompt is computed by a fixed depth-wise Laplacian filter: \[K_{lap}=\begin{bmatrix}0&-1&0\\-1&4&-1\\0&-1&0\end{bmatrix}, \qquad h_{spa}=\mathrm{DWConv}_{K_{lap}}^{rep}(\bar{x}), \label{eq:spatial95laplacian}\tag{2}\] where \(rep\) denotes replicate padding at image boundaries. We pool the absolute high-pass response and normalize it by the average feature magnitude: \[e_{spa}=\frac{\mathrm{GAP}(|h_{spa}|)}{\mathrm{GAP}(|\bar{x}|)+\tau}\in\mathbb{R}^{B\times d}, \label{eq:spatial95prompt}\tag{3}\] where \(\tau\) is a small numerical constant.

The spectral prompt summarizes latent channel variation after multispectral patch embedding. The patch projection is initialized by spectral-aware channel expansion and optimized on MSI/HSI tracking data, so the resulting latent channels carry recorded-band-induced feature contrast. A first-order finite difference compactly describes this contrast for expert routing: \[(D_c\bar{x})_{b,c,h,w}=\begin{cases} 0, & c=1,\\ \bar{x}_{b,c,h,w}-\bar{x}_{b,c-1,h,w}, & c>1, \end{cases} \label{eq:channel95diff}\tag{4}\] followed by the same magnitude normalization: \[e_{spe}=\frac{\mathrm{GAP}(|D_c\bar{x}|)}{\mathrm{GAP}(|\bar{x}|)+\tau}\in\mathbb{R}^{B\times d}. \label{eq:spectral95prompt}\tag{5}\] This operation provides a compact recorded-band-induced latent contrast cue for routing capacity in the learned feature space.

The two prompts are concatenated and fused by a lightweight two-layer projection: \[e_{prompt}=\phi_f([e_{spa},e_{spe}])=W_2^f\,\delta\left(W_1^f\,\mathrm{LN}([e_{spa},e_{spe}])\right), \label{eq:prompt95fusion}\tag{6}\] where \([\cdot,\cdot]\) denotes channel concatenation and \(\delta\) is GELU. The routing logits are: \[r=W_s z+W_f e_{prompt}+\epsilon, \qquad r\in\mathbb{R}^{B\times N}. \label{eq:routing95logit95detail}\tag{7}\] During training, \(\epsilon_{b,i}\sim\mathcal{N}(0,\sigma_{\epsilon}^{2})\) is a small Gaussian routing jitter with a fixed code-default standard deviation. It stabilizes early routing and uses the same code default across benchmarks. During validation and inference, \(\epsilon\) is set to zero.

The final sparse gate is computed from the softmax probabilities \(p=\mathrm{Softmax}(r)\). Let \(\mathcal{I}_b=\mathrm{TopK}(p_b,k)\) be the selected expert indices of sample \(b\). The gate is renormalized over the selected experts: \[g_{b,i}=\begin{cases} \dfrac{p_{b,i}}{\sum_{j\in\mathcal{I}_b}p_{b,j}}, & i\in\mathcal{I}_b,\\[6pt] 0, & i\notin\mathcal{I}_b. \end{cases} \label{eq:topk95gate95detail}\tag{8}\] Thus, unselected experts have zero gate value and are skipped during inference. The gate is sample conditioned at the search-region level and shared by all tokens in the current search region. This definition also explains the routing diagnostics in the experiments: usage counts selected Top-\(k\) slots, whereas gate weight records the normalized selected \(g_{b,i}\) values.

3.3 Adaptive Expert Set↩︎

Standard MoE architectures usually use homogeneous experts with identical structures. SAMoE uses a capacity-ordered expert set whose experts are organized by computational and representational capacity.

3.3.0.1 Capacity schedule

For an input feature \(x\in\mathbb{R}^{B\times d\times H\times W}\), SAMoE contains \(N\) experts \(\mathcal{E}=\{E_i\}_{i=1}^{N}\) from lightweight to strong. Expert \(E_i\) first projects \(d\) channels to a rank \(r_i\), applies \(t_i\) repeated latent spatial-frequency mixer units and projects the feature back to \(d\) channels: \[d \xrightarrow{W_i^{in}} r_i \xrightarrow{\text{t_i mixer units}} r_i \xrightarrow{W_i^{out}} d . \label{eq:expert95bottleneck}\tag{9}\] The nested rank schedule used in the main experiments is: \[r_i = \max \left(r_{\min},\left\lfloor \frac{i}{N}r_{\max}\right\rfloor\right), \qquad i=1,\ldots,N, \label{eq:nested95rank}\tag{10}\] where \(r_{\min}\) and \(r_{\max}\) are fixed by the backbone width and are kept unchanged across datasets. The expert depth is scaled linearly: \[t_i = \max\left(1,\left\lceil \frac{i}{N}t_{\max}\right\rceil\right). \label{eq:depth95scale}\tag{11}\] For the default four-expert setting, the resulting expert schedule is summarized in Tab. 1. This table gives the computation order required to reproduce the block at the architectural level; low-level constants such as initialization seeds, normalization epsilons and PyTorch layer defaults follow the released implementation.

3.3.0.2 Expert computation path

Each local expert has its own parameters; local projection, convolution, FFT mixing, MLP and output projection weights are independent across experts. Given \(x\), expert \(E_i\) first applies normalization, a \(1\times1\) rank projection and a depth-wise convolution: \[u_i^{(0)} = \delta\left(\mathrm{DWConv}_{\kappa_i}\left(W_i^{in}\,\mathrm{LN}(x)\right)\right), \label{eq:expert95preproj}\tag{12}\] where \(\delta(\cdot)\) is GELU. The convolution is applied in the spatial domain and injects local context before the FFT mixer.

For the \(m\)-th mixer unit, \(u_i^{(m-1)}\) is split into non-overlapping spatial windows of size \(w_i\times w_i\): \[\{u_{i,j}^{(m-1)}\}_{j=1}^{J_i}=\mathcal{P}_{w_i}\left(u_i^{(m-1)}\right). \label{eq:window95partition}\tag{13}\] If \(H\) or \(W\) is not divisible by \(w_i\), replicate padding is applied before partitioning and the padded area is cropped after window reversal. Within each window, \(Q\), \(K\) and \(V\) are produced by a linear projection in the rank space: \[(q_{i,j},k_{i,j},v_{i,j})=\mathrm{Split}\left(W_i^{qkv}u_{i,j}^{(m-1)}\right). \label{eq:qkv95projection}\tag{14}\] The frequency interaction is computed over the two spatial dimensions of each window: \[a_{i,j}=\mathrm{Re}\left(\mathcal{F}^{-1}_{2D}\left(\mathcal{F}_{2D}(q_{i,j})\odot\mathcal{F}_{2D}(k_{i,j})\right)\right), \label{eq:fft95interaction}\tag{15}\] \[o_{i,j}=a_{i,j}\odot v_{i,j}. \label{eq:fft95value95modulation}\tag{16}\] The window outputs are reversed to the full feature map and passed through a rank-space MLP with a residual connection: \[\bar{u}_i^{(m)}=\mathcal{P}^{-1}_{w_i}\left(\{o_{i,j}\}_{j=1}^{J_i}\right),\] \[u_i^{(m)}=u_i^{(m-1)}+\bar{u}_i^{(m)}+\mathrm{MLP}_i\left(\mathrm{LN}\left(u_i^{(m-1)}+\bar{u}_i^{(m)}\right)\right). \label{eq:expert95inner95residual}\tag{17}\] The local residual update produced by expert \(E_i\) is: \[\Delta_i(x)=W_i^{out}u_i^{(t_i)} \in \mathbb{R}^{B\times d\times H\times W}. \label{eq:expert95output}\tag{18}\] Equations 12 –18 clarify the operation order: normalization, rank projection, depth-wise convolution, local window partition, \(QKV\) projection, two-dimensional spatial FFT interaction on latent features, inverse FFT, value modulation, window reversal, rank-space MLP and output projection.

3.4 Shared Experts↩︎

Sparse expert selection reduces redundant computation, while selected experts still require common context to avoid fragmented local updates. The Shared Global Expert provides a common modulation signal for all selected local experts. It uses global pooling followed by a small bottleneck projection: \[s_g=\mathrm{GAP}(\mathrm{LN}(x)), \qquad m_g=\sigma\left(W_2^g\,\delta(W_1^g s_g)\right)\in\mathbb{R}^{B\times d}, \label{eq:shared95global95gate}\tag{19}\] where \(\sigma(\cdot)\) is the sigmoid function. The modulation vector is broadcast to the spatial dimensions and applied to each selected expert update: \[\tilde{\Delta}_i(x)=\Delta_i(x)\odot \mathrm{Broadcast}(m_g). \label{eq:shared95modulation}\tag{20}\] The SAMoE output is then: \[\mathrm{SAMoE}(x)=W_o\left(\sum_{i=1}^{N}g_i(x)\tilde{\Delta}_i(x)\right). \label{eq:samoe95output95detail}\tag{21}\] The shared branch provides common latent context before the sparse expert outputs are inserted into the backbone residual path.

3.5 Loss Function↩︎

The tracking loss follows the OSTrack [21] center point prediction strategy. We use weighted focal loss [22] for classification and a combination of \(\ell_1\) loss and generalized IoU loss [23] for regression: \[\mathcal{L}_{track}=\mathcal{L}_{cls}+\lambda_G\mathcal{L}_{GIoU}+\lambda_{L1}\mathcal{L}_{L1}. \label{eq:tracking95loss95detail}\tag{22}\]

The auxiliary routing loss is computed at the sample level. Let \(p_{b,i}\) be the pre-TopK probability from the router and let \(\mathbb{1}(i\in\mathcal{I}_b)\) indicate whether expert \(i\) is selected for sample \(b\). The importance and load statistics are: \[\mathrm{Imp}_i=\sum_{b=1}^{B}p_{b,i}, \qquad \mathrm{Load}_i=\sum_{b=1}^{B}\mathbb{1}(i\in\mathcal{I}_b). \label{eq:imp95load95detail}\tag{23}\] We use the squared coefficient of variation: \[CV(v)^2=\frac{\mathrm{Var}(v)}{\mathrm{Mean}(v)^2+\tau}. \label{eq:cv95detail}\tag{24}\] Because SAMoE experts have different costs, the default loss also includes a simple complexity bias. The relative complexity proxy of expert \(E_i\) is: \[\chi_i=r_i\,t_i\,(\kappa_i^2+w_i^2), \qquad c_i=\frac{\chi_i}{\max_j \chi_j+\tau}. \label{eq:complexity95proxy}\tag{25}\] The complexity term penalizes excessive reliance on the strongest experts while still allowing them to be selected when the router assigns high probability: \[\mathcal{L}_{comp}=\sum_{i=1}^{N}c_i\frac{\mathrm{Imp}_i}{\sum_j\mathrm{Imp}_j+\tau}. \label{eq:complexity95loss}\tag{26}\] The final MoE regularizer is: \[\mathcal{L}_{moe}=\frac{1}{2}CV(\mathrm{Imp})^2+\frac{1}{2}CV(\mathrm{Load})^2+\lambda_c\mathcal{L}_{comp}. \label{eq:moe95loss95detail}\tag{27}\] The total training objective is: \[\mathcal{L}=\mathcal{L}_{track}+\lambda_{moe}\mathcal{L}_{moe}. \label{eq:final95loss95detail}\tag{28}\] We set \(\lambda_G=2\), \(\lambda_{L1}=5\) and \(\lambda_{moe}=0.01\) unless otherwise specified; the small coefficient \(\lambda_c\) follows the same code default in all datasets.

The terminology used in the scaling ablation is defined by the equations above. Nested rank + max scaling means that ranks follow Eq. 10 and the complexity weight is normalized by the maximum proxy in Eq. 25 . Min complexity scaling uses the same proxy but normalizes by \(\min_j\chi_j\), which over-amplifies the cost difference and is used only as an ablation. Exponential rank scaling replaces Eq. 10 with an exponentially increasing rank sequence while keeping the same window, kernel and depth order. Without complexity bias sets \(\lambda_c=0\) and without auxiliary loss sets \(\lambda_{moe}=0\).

During inference, the auxiliary loss and routing noise are disabled. The tracker uses one static template and one dynamically updated template [24]; the online template is updated only when both the fixed interval and confidence threshold criteria are satisfied.

4 Experiments↩︎

4.1 Implementation Details↩︎

SpecTrack is implemented in Python 3.13 with PyTorch 2.9. Training uses eight NVIDIA RTX 3090 GPUs and speed is measured on one RTX 3090. The default setting uses a Fast-iTPN/OSTrack-style one-stream encoder, \(N=4\) capacity-ordered experts and Top \(k=2\) routing.

4.1.0.1 Model

We develop five variants to examine accuracy–speed trade-offs (Tab. 2). L variants use HiViT-L [25], B variants use HiViT-B and T224 uses HiViT-T. The encoders are initialized with FastiTPN [26] and adapted to MSI/HSI inputs through spectral-aware channel expansion.

4.1.0.2 Spectral aware channel expansion

The original patch projection of an RGB pretrained model has weights \(W^{rgb}\in\mathbb{R}^{d\times3\times P\times P}\). For an MSI/HSI input with \(C\) bands, we construct \(W^{C}\in\mathbb{R}^{d\times C\times P\times P}\) from the RGB pretrained projection to preserve a stable initialization. In the eight band MSI setting with band centers \(\{422.5,487.5,550.0,602.5,660.0,725.0,785.0,887.5\}\) nm, we use a spectral aware mapping based on wavelength proximity: bands with \(\lambda\le500.0\) nm are initialized from the blue RGB channel, bands with \(500.0<\lambda<620.0\) nm from the green channel and bands with \(\lambda\ge620.0\) nm from the red channel. Let \(m(c)\in\{B,G,R\}\) denote the mapped RGB channel for band \(c\) and \(\eta_c\) denote an optional scale factor. The expanded kernel is: \[W^{C}_{:,c,:,:}=\eta_c W^{rgb}_{:,m(c),:,:}. \label{eq:channel95expansion95insert}\tag{29}\] We set \(\eta_c=1\) for insert and \(\eta_c=1/2\) for insert halfcopy. For other band layouts, we use interpolation-based initialization: \[W^{C}_{:,c,:,:}=\frac{3}{C}\sum_{r=1}^{3}\alpha_{c,r}W^{rgb}_{:,r,:,:},\quad \sum_{r=1}^{3}\alpha_{c,r}=1, \label{eq:channel95expansion95interp}\tag{30}\] where \(\alpha_{c,r}\) interpolates the band center to RGB reference centers when metadata are available and uses \(\alpha_{c,r}=1/3\) otherwise. The factor \(3/C\) keeps the activation scale close to the RGB projection. HOTC20 uses this fallback because its 470–620 nm range is concentrated in the visible spectrum.

4.1.0.3 Training protocol

The main tables use dataset-specific training. MUST, MSITrack and HOTC20 models are trained on their official training splits; GOT-10k follows its one-shot RGB protocol. Template and search images expand target boxes by factors of 2 and 4. We use AdamW, batch size 32, learning rate \(10^{-4}\), 25 epochs and step decay at epoch 15.

4.1.0.4 Inference

The online template is updated every 25 frames when the confidence score exceeds 0.7. Following standard practice [21], we apply a Hann window penalty to incorporate positional priors.

4.2 Dataset Versions, Protocols and Metrics↩︎

We evaluate SpecTrack on four benchmarks (Tab. 3): MUST [5], MSITrack [9], HOTC20 [10] and GOT-10k [11]. The first three support MSI/HSI evaluation; GOT-10k is the RGB one-shot generalization setting.

We report standard metrics: Area Under the Curve (AUC), Success Rates at 0.5/0.75 (\(SR_{0.5/0.75}\)), Precision (P) and Normalized Precision (\(P_{norm}\)). Scores are interpreted within the stated dataset protocol, metric and comparator list.

For runtime comparison, executable methods are remeasured on the same hardware/software environment; FPS excludes data loading. SpecTrack-L384 is the accuracy-oriented variant, while B224 and T224 show practical accuracy–speed operating points.

4.2.0.1 Spectral curve interpretation

Qualitative curves use discrete recorded bands and benchmark-provided digital-number or normalized-intensity scales. They are reported as template–target–background separability evidence for tracking, with ROI selection and uncertainty handled by the dataset protocol when available.

4.3 Comparison with Published Trackers↩︎

We evaluate SpecTrack within each benchmark protocol in Tab. 3; reported numbers remain benchmark-specific.

4.3.0.1 Evaluation on MUST dataset

On MUST (Tab. 4), SpecTrack-L384 is reported as the accuracy-upper-bound variant: it achieves 65.2% AUC, 79.8% \(\mathrm{SR}_{0.5}\) and 61.0% \(\mathrm{SR}_{0.75}\), improving over UNTrack by 5.5, 4.0 and 14.8 points. The balanced SpecTrack-B224 variant is the practical default, reaching 62.4% AUC at 43.7 FPS, which is 2.7 AUC points above UNTrack while running faster under the unified runtime. Thus, the MUST improvement is not only represented by the expensive L384 setting. Fig. 4 and Fig. 5 show gains across challenge categories and similar-background scenes.

4.3.0.2 Evaluation on MSITrack dataset

We compare SpecTrack with published trackers on the MSITrack benchmark in Tab. 5. GRM is the strongest listed baseline in AUC and Precision. The accuracy-oriented SpecTrack-L384 increases AUC from 47.8% to 51.9% and Precision from 63.0% to 65.0%. Below the largest variant, SpecTrack-B224 already reaches 48.4% AUC and SpecTrack-B384 reaches 49.8% AUC with a 3.0-point gain in \(\text{Pre}_N\) over GRM. This shows that the adaptive design contributes across model scales rather than only through the L384 upper-bound setting.

4.3.0.3 Evaluation on HOTC20 dataset

We evaluate SpecTrack on the 16 band HOTC20 benchmark in Tab. 6. This naming follows the dataset source in Xiong et al. [10] and avoids conflating it with later HOTC releases that use different camera subsets and band counts. SpecTrack-L384 achieves the highest AUC of 72.6% and is interpreted as the accuracy upper bound of the proposed variants. The balanced SpecTrack-B224 obtains 70.2% AUC at 43.7 FPS, which is competitive with HotMoE and UNTrack at 70.4% AUC. We therefore present HOTC20 as a clear accuracy–efficiency trade-off: high-capacity variants improve the accuracy ceiling, while B224 provides the practical operating point.

4.3.0.4 RGB generalization on GOT-10k

We further evaluate SpecTrack on the RGB GOT-10k one-shot protocol to examine architectural generalization beyond MSI/HSI tracking. SpecTrack-L384 achieves 79.3% AO, 88.7% \(\text{SR}_{0.5}\), and 81.4% \(\text{SR}_{0.75}\), ranking first among the compared published trackers in Tab. 7. This suggests that the proposed capacity-aware expert design can transfer to generic RGB tracking.

4.4 Ablation Studies↩︎

The ablations are organized as diagnostic questions rather than repeated “full model versus baseline” comparisons. Each group isolates a different part of the proposed capacity-allocation pipeline: the recorded MSI input, the multispectral input adapter, the routing cues, the latent channel-variation cue, the MoE design, the sparse-routing hyperparameters and the resulting routing behavior. Unless otherwise specified, all ablations use SpecTrack-B224 on MUST with the same input resolution, training split and evaluation protocol as the main comparison.

2pt

4.4.1 Input Evidence: Do Recorded Multispectral Bands Help?↩︎

Tab. 9 separates the contribution of the input modality from the contribution of the proposed architecture. Both the uniform baseline and SpecTrack are evaluated with pseudo-color RGB projections and real MSI inputs.

Changing the baseline input from pseudo-RGB to MSI raises AUC from 55.2 to 59.1, showing that recorded bands provide useful target–background evidence even without the proposed adaptive modules. SpecTrack further benefits from MSI input, improving from 56.3 to 62.4 AUC. Thus, the following ablations are conducted in the MSI setting, where the proposed routing cues have meaningful recorded-band-induced variation to exploit.

4.4.2 Input Adapter: How Should RGB Weights Be Expanded to MSI?↩︎

Tab. 10 examines how the RGB patch embedding is adapted to eight-band MSI. This experiment does not test the MoE mechanism; it controls the multispectral initialization before routing is applied. Insert halfcopy gives the best AUC and is used as the default for MUST and MSITrack. HOTC20 uses the interpolation fallback because its 16 bands cover a narrower 470–620 nm range.

4.4.3 Routing Cues: Can Spectral–Spatial Prompts Guide Expert Selection?↩︎

Tab. 11 focuses on the router input. Semantic-only routing reaches 60.9 AUC. Adding the spatial boundary prompt improves AUC to 61.5, and adding the latent channel-variation prompt improves AUC to 61.7. Their fusion reaches 62.4 AUC, indicating that the two prompts are complementary: the spatial cue emphasizes boundary and texture changes, while the latent channel cue summarizes recorded-band-induced feature variation.

4.4.4 Latent Cue: Is Channel Order Merely an Unordered Statistic?↩︎

Tab. ¿tbl:tab:latent95order95sensitivity? tests whether the latent channel-variation cue depends on the learned latent-channel organization. During inference, the trained backbone, expert parameters, semantic descriptor, spatial prompt and expert inputs are kept fixed; only the channel order used by the finite-difference operation in Eq. 4 is perturbed. The reverse order preserves a structured but inverted neighborhood, random permutations destroy most latent-channel adjacency relations and the raw-band baseline replaces the latent cue with finite differences along the recorded band order.

Reversing the latent order reduces AUC by 0.9 points, while random permutations reduce AUC by 1.2 points on average. The raw-band prompt baseline decreases AUC by 2.0 points, indicating that direct finite differences along recorded bands do not substitute for the learned latent routing cue. Corrupted orders can increase high-capacity usage or gate mass while lowering tracking accuracy, so the benefit does not come from selecting stronger experts more often; it comes from assigning them according to a better-aligned latent ambiguity cue.

4.4.5 Expert Design: Is the Gain More Than Generic MoE?↩︎

Tab. 12 compacts the overlapping module, MoE and shared-context controls into a single diagnostic table. The rows answer different questions. Capacity experts alone test whether multiple branches are enough; the semantic router tests whether generic semantic routing is enough; homogeneous MoE-style experts test whether generic expert branching can replace capacity ordering; and the local-only setting tests whether shared global context is needed under sparse routing.

The compact controls show that the final improvement is not explained by MoE-like branches alone. Capacity experts without prompt routing improve the uniform baseline but remain below the full model. Semantic-only routing gives no additional gain over capacity experts alone, suggesting that MSI tracking requires more than generic semantic descriptors. Homogeneous experts under the fused prompt are also weaker than capacity-ordered experts, and removing the Shared Global Expert reduces the AUC from 62.4 to 61.5. Therefore, the gain comes from the combination of capacity ordering, multispectral prompt-guided routing and shared global context.

4.4.6 Sparse Routing: What Accuracy–Efficiency Setting Is Practical?↩︎

After the mechanism is fixed, Tabs. 13 and 14 tune the number of experts and the routing sparsity. These experiments answer a deployment question rather than another component-removal question.

With \(k=2\), increasing \(N\) from 3 to 5 gives only a marginal accuracy gain, while \(N=6\) reduces both accuracy and speed. With \(N=4\), Top-\(k=1\) is fastest but loses accuracy, whereas \(k=3\) and \(k=4\) increase computation without improving AUC. We therefore use \(N=4,k=2\) as the default accuracy–efficiency point.

4.4.7 Routing Diagnostics: Does the Router Behave as Intended?↩︎

The previous tables measure accuracy. Tab. 15 instead examines routing behavior at the search-sample level. Usage denotes the percentage of selected expert slots and \(H_p\) is the dense-router entropy before Top-\(k\) selection.

The full router does not collapse to a single expert: all four experts are used, and the stronger experts E3 and E4 receive 54.9% of the selected slots. Removing the latent channel prompt reduces E4 usage from 30.6% to 25.8% and increases entropy, indicating less confident capacity assignment. Removing the shared expert also weakens the high-capacity preference. Logged samples show a Spearman correlation of 0.60 between latent channel-variation strength and the pre-renormalization retained mass of E3+E4, supporting the link between embedded spectral–spatial ambiguity and high-capacity expert allocation.

5 Discussion and Limitations↩︎

5.0.0.1 Remote-sensing interpretation.

SpecTrack uses recorded multispectral observations as routing evidence for tracking, rather than as calibrated material spectra. The Spectral Prompt Router learns how latent channel variation, boundary response, and semantic context jointly indicate the difficulty of a search region. This design is practical for benchmark tracking data where complete radiometric calibration metadata are often unavailable. When calibrated reflectance, bandpass information, and sensor response functions are available, the routing cues could be extended toward more sensor-aware or material-aware tracking designs.

5.0.0.2 Failure-oriented analysis.

Fig. 7 summarizes the attribute-level limitations of SpecTrack-B224 on MUST. The weakest cases are out-of-view, full occlusion, low resolution, and scale variation. These failures are consistent with the design scope of SpecTrack: adaptive expert routing can allocate stronger local spectral–spatial capacity when useful target evidence remains in the search region, but it cannot reliably recover a target that is absent from the crop or represented by only a few mixed pixels.

These observations suggest that future work should combine adaptive capacity allocation with global re-detection, longer temporal memory, uncertainty-aware template update, and sensor-calibrated prompts when richer radiometric metadata are available.

6 Conclusion↩︎

This paper presented SpecTrack, a spectral–spatial complexity-aware tracker that formulates MSI/HSI tracking as search-region-level adaptive capacity allocation. By combining capacity-ordered experts, prompt-guided sparse routing, and shared global context, SpecTrack adapts the modeling capacity to the ambiguity of each search region instead of processing all regions through a fixed-capacity path.

Experiments on MUST, MSITrack, and HOTC20 demonstrate that SpecTrack achieves a favorable accuracy–efficiency trade-off, with SpecTrack-B224 serving as the balanced default and SpecTrack-L384 reporting the accuracy-oriented setting. Mechanism-aligned ablations and routing diagnostics further support the roles of multispectral input, spectral–spatial prompts, capacity-ordered experts, and shared global context. The additional GOT-10k evaluation indicates RGB-domain architectural generalization. Remaining limitations mainly arise from out-of-view, full-occlusion, low-resolution, and scale-variation cases, suggesting future directions such as global re-detection, longer temporal memory, and sensor-aware calibration.

References↩︎