Co-Adaptive Multi-Task LoRA: Transfer-Aware, Label-Free
Control of Domain Participation
January 01, 1970
Abstract
Fine-tuning a single low-rank adapter on many domains at once is multi-task learning: the domains must be co-learned, and how they share the adapter decides whether they help or hurt one another. Most efficient fine-tuning pipelines ignore this and train on a fixed, uniform mixture, leaving two coupled questions unanswered: how much should each domain participate, and which domains should be co-trained given that some transfer positively and others interfere? We show that both answers can be read off cheaply and without labels. A forward pass of the current shared adapter over a small unlabeled probe yields, per domain, a competence signal whose level tracks remaining headroom and whose trajectory tracks learning speed; the drift of these probe representations yields a signed cross-domain affinity that predicts pairwise transfer. We fold both into CoDA, a co-adaptive controller that solves a small entropy-regularized quadratic program on the simplex to set each domain’s participation—jointly its loss weight and its share of the sampled data—rewarding high-headroom, still-learning, mutually synergistic domains and damping interfering ones. The controller is forward-only, adds no trainable parameters, and wraps any multi-task LoRA pipeline. Across five heterogeneous domains and two backbones, CoDA improves the average over uniform mixing, learned mixtures, gradient-surgery multi-task optimizers, and online data selection while using half the data, and lowers cross-domain gradient conflict. We prove that the competence signal tracks domain risk, that the participation program has a unique fixed point reached by a contraction, and that its solution performs transfer-aware water-filling; analysis, ablations, and controls corroborate each claim.
1 Introduction↩︎
Adapting a pretrained large language model (LLM) to many downstream domains at once—knowledge, mathematics, code, reasoning, medicine—is now routine, and low-rank adaptation (LoRA) [1] makes it cheap by training a single small adapter on top of frozen weights. When one shared adapter must serve all domains, fine-tuning is inherently multi-task learning (MTL): the domains are co-learned through a common set of parameters, and the classic MTL phenomena appear—some domains reinforce each other while others compete, so training them together can help or hurt [2]–[4]. How the shared adapter is divided among the domains therefore determines the outcome.
Yet most efficient fine-tuning pipelines sidestep this. Data-selection methods score individual examples and keep a high-value subset [5]–[8], but they operate on a single pooled stream and inherit a fixed, usually uniform or size-proportional, domain mixture. This leaves two coupled questions unanswered: (i) how much should each domain participate—how much of the shared capacity and the finite data-and-compute budget it should claim—and (ii) which domains should be co-trained, given that co-training synergistic domains transfers positively while co-training conflicting ones injects interference [3], [4], [9]. Treating the mixture as fixed answers neither, and answering them well is exactly the MTL problem of balancing tasks and choosing task groupings, now posed inside a single LoRA under a budget.
Both questions have moving targets. First, domains have strongly heterogeneous and non-stationary learning curves (Figure 1a): a domain the base model already handles well saturates after a few steps, while others keep improving, so the right amount of participation drifts during training. Second, transfer between domains is uneven and signed: some pairs share structure and should be learned together, others interfere and should be kept apart (Figure 1c). A controller must read both the per-domain state and the between-domain relationships, and it must do so continually.
The apparent obstacle is that measuring a domain’s remaining headroom or its transfer to others seems to require labeled held-out evaluation at every step. Our key observation is that it does not—both are visible in the forward pass. Running the current shared adapter over a small unlabeled probe set per domain yields a competence statistic whose level tracks the domain’s error (its headroom) and whose trajectory tracks whether it is still learning; Figure 1 (b) shows the level correlates with per-domain accuracy at \(r\!\approx\!0.9\). The drift of these probe representations between rounds yields a signed cross-domain affinity that predicts pairwise transfer measured by leave-one-out retraining (Figure 4a, \(r\!\approx\!0.94\)). Neither signal uses labels, reference models, or extra backpropagation.
Building on this, we propose CoDA (Co-adaptive Domain Adaptation), a forward-only controller for multi-task LoRA. Every \(T\) steps CoDA probes each domain to estimate its headroom and learning speed, estimates the between-domain affinity from representation drift, and solves a small entropy-regularized quadratic program on the simplex to set each domain’s participation. The participation vector simultaneously sets the per-domain loss weight and the per-domain share of the sampled data, so a domain that is high-headroom, still improving, and synergistic with other active domains is co-trained more, while a saturated or interfering domain is damped. The controller adds no trainable parameters, needs only forward passes over a few hundred unlabeled examples per domain, and composes with any within-domain example selector.
Our contributions are:
We frame budget-constrained multi-domain LoRA tuning as co-adaptive multi-task learning: jointly deciding how much each domain participates and which domains to co-train, rather than selecting examples over a fixed mixture (Section 4).
We introduce two label-free, forward-only signals—a per-domain competence that estimates headroom and learning speed, and a cross-domain affinity from probe-representation drift that estimates signed transfer—and validate them against per-domain accuracy and leave-one-out transfer.
We propose CoDA, which couples both signals in an entropy-regularized quadratic program whose solution sets domain participation (loss weight and data share); we prove competence tracks risk, the program has a unique fixed point reached by a contraction, and its solution is transfer-aware water-filling (Section 5).
Across five heterogeneous domains and two backbones, CoDA beats uniform mixing, learned static mixtures, gradient-surgery multi-task optimizers, and online data selectors at half the data, with mechanism analysis, ablations isolating the transfer term, negative controls, and efficiency studies (Section 6).
2 Related Work↩︎
2.0.0.1 Multi-task learning: balancing and task grouping.
Training one model on several tasks can improve data efficiency and generalization but also causes negative transfer when objectives compete [2]. Two levers recur. Task balancing reweights the joint objective so no task dominates: uncertainty weighting [10], gradient normalization [11], and multi-objective formulations [12] adjust per-task loss weights, while data-side balancing over- or under-samples tasks, as in temperature-based sampling for massively multilingual training [13]. Task grouping instead asks which tasks to learn together: cooperation and competition can be measured and used to assign tasks to networks [3], and inter-task affinity—how one task’s update changes another’s loss—identifies good groupings in a single run [4]. CoDA unifies both levers inside one shared LoRA and drives them from label-free signals: competence sets how much each domain participates, and an affinity estimate decides which domains should be co-trained, updated online as the model changes.
2.0.0.2 Data selection and domain mixtures.
Selecting high-value training data is a long-standing idea, from online batch selection by loss [14] and importance sampling by gradient norm [15] to coreset and pruning methods [16]–[18]. For instruction tuning, quality- and influence-based curation reduces data by orders of magnitude with little loss [6], [19]–[22]. Online selection adapts to the evolving model, keeping points that are learnable and not yet learnt via a held-out reference model [7], greedily maximizing a Taylor-approximated batch quality [8], or scoring examples by utility and diversity without a reference model [23]. These operate at the sample level over a pooled stream and leave the domain mixture fixed, with lasting downstream effects [24]. A complementary line optimizes the mixture itself, but mostly for pretraining language-modeling loss and with static or proxy-model-derived weights [25]–[27]. CoDA is orthogonal to sample selection—it can wrap any of the above as a within-domain step—and differs from mixture optimization by adapting participation online for downstream multi-task accuracy under a shared adapter, with an explicit transfer term.
2.0.0.3 Label-free signals from model outputs.
Estimating a model’s accuracy without labels is possible from its own predictions: softmax confidence and margin [28], prediction entropy and agreement across perturbations [29], [30], the projection norm of features [31], energy-based scores over the prediction distribution [32], and semantic-space confidence for uncertainty [33]. This literature uses such statistics to report an accuracy estimate. We instead turn a normalized confidence statistic into a training-time control signal and, crucially, add a second statistic—the drift of probe representations across rounds—to estimate between-domain transfer, which single-model accuracy proxies do not address.
2.0.0.4 Model merging and multi-task optimization.
When domains are trained separately, their adapters can be recombined by model merging: weight averaging and soups [34], task arithmetic [35], [36], Fisher- and regression-weighted merging [37], [38], and interference-aware schemes such as TIES-merging and DARE [39], [40]. LoRA-specific composition includes LoRAHub [41], partial linearization [42], and SVD-based tying [43], whose mergeability can even be predicted in advance [44]. A recent thread reduces cross-task interference through orthogonal parameter/subspace decoupling: orthogonal continual subspaces [45], orthogonal subspaces for robust merging [46], data-free decouple-and-orthogonalize merging [47], and interference reduction in multi-task low-rank adaptation [48]. On the training side, multi-task optimization tackles conflicting gradients directly—PCGrad projects away conflicting components [9], and conflict-averse and multi-objective methods rebalance task gradients [12], [49]. Merging composes after separate training and gradient surgery acts inside each step; CoDA instead reduces interference upstream, by choosing which domains co-train and how strongly, and composes with all of the above at inference. Mixture-of-experts LoRA variants [50]–[53] and near-orthogonal random-projection experts [48] add multi-domain capacity but do not schedule participation or data.
3 Preliminaries↩︎
3.0.0.1 Multi-task LoRA with domain participation.
We fine-tune a frozen model with weights \(W_0\) on \(K\) domains, each a pool \(\mathcal{D}_k=\{(x,y)\}\) of instruction–response pairs. A single shared LoRA adapter \(\theta\) reparameterizes each adapted linear map as \(W_0 + \tfrac{\alpha}{r}BA\) with \(B\!\in\!\mathbb{R}^{d\times r}\), \(A\!\in\!\mathbb{R}^{r\times d}\), \(r\!\ll\!d\); only \(\theta=\{A,B\}\) is trained. We control the domains through a participation vector \(\pi\in\Delta^{K-1}\) on the simplex, which plays the two roles that MTL balances separately—loss weighting and data sampling—at once. Given \(\pi\), the weighted multi-task objective is \[\min_{\theta}\;\sum_{k=1}^{K} \pi_k\, \mathcal{L}_k(\theta;\mathcal{S}_k),\qquad |\mathcal{S}_k| = \lfloor N\pi_k\rfloor,\;\;\mathcal{S}_k\subseteq\mathcal{D}_k, \label{eq:mtl}\tag{1}\] where \(\mathcal{L}_k\) is the token-level cross-entropy and \(N\) is the per-round data budget. Thus \(\pi_k\) both scales domain \(k\)’s gradient and sets how many of its examples enter the round. Fixing \(\pi_k\!\propto\!|\mathcal{D}_k|\) (or uniform) recovers standard pooled training; we instead co-adapt \(\pi\) during training from label-free signals.
3.0.0.2 Competence: a label-free headroom signal.
For an input \(x\), let \(p(x;\theta)\in\Delta^{V-1}\) be the model’s next-token distribution averaged over response positions. Its normalized predictive entropy \(\tilde{H}(x;\theta)=H\!\big(p(x;\theta)\big)/\log V\in[0,1]\) measures how unsure the adapter is on \(x\). We define the competence of domain \(k\) on an unlabeled probe set \(\mathcal{P}_k\subset\mathcal{D}_k\) as its mean confidence, \[c_k(\theta)\;=\;\frac{1}{|\mathcal{P}_k|}\sum_{x\in\mathcal{P}_k}\big(1-\tilde{H}(x;\theta)\big)\;\in[0,1], \label{eq:comp}\tag{2}\] with headroom \(h_k=1-c_k\). Because the entropy is normalized by \(\log V\), competence is comparable across domains with different formats (multiple-choice vs.open generation), which raw loss is not. It needs only forward passes and no labels; Section 4 shows \(c_k\) tracks per-domain accuracy.
3.0.0.3 Affinity: a label-free transfer signal.
Let \(\phi(x;\theta)\in\mathbb{R}^{d}\) be the adapter’s mean hidden representation of \(x\) and \(\bar\phi_k(\theta)=\mathbb{E}_{x\in\mathcal{P}_k}[\phi(x;\theta)]\) the domain-\(k\) probe centroid. Over one round the centroid shifts by \(\delta_k=\bar\phi_k(\theta^{(r)})-\bar\phi_k(\theta^{(r-1)})\). The (symmetrized) cross-domain affinity is the cosine alignment of these shifts, \[A_{kj}\;=\;\cos\!\big(\delta_k,\delta_j\big)\in[-1,1],\qquad \boldsymbol{A}=\tfrac12(A+A^\top), \label{eq:aff}\tag{3}\] a forward-only surrogate for whether updates driven by the two domains pull the shared adapter in compatible directions; \(A_{kj}\!>\!0\) signals positive transfer, \(A_{kj}\!<\!0\) interference. Section 4 shows \(\boldsymbol{A}\) predicts leave-one-out transfer.
3.0.0.4 Desiderata.
A co-learning controller for a shared adapter should be (D1) label-free; (D2) cheap, adding only forward passes and no reference model or extra backpropagation; (D3) adaptive, tracking the moving optimum rather than a mixture fixed in advance; and (D4) relational, accounting for how domains help or hurt one another rather than treating them independently. CoDA meets all four: competence and affinity are label-free (D1) and forward-only (D2), re-estimation every \(T\) steps makes it adaptive (D3), and the affinity term makes it relational (D4).
4 Analysis: What Governs Co-Learning↩︎
We motivate CoDA with four observations on Qwen-2.5-7B and LLaMA-3.1-8B fine-tuned jointly on the five domains (setup in Section 6). Diagnostics use a held-out probe per domain; “oracle” quantities use labels only for analysis, never by CoDA.
4.0.0.1 Diagnostic protocol.
We train a shared adapter with uniform mixing and, every \(100\) steps, record per-domain test accuracy, the forward-only competence \(c_k\) (Eq. 2 ) and centroids for \(\boldsymbol{A}\) (Eq. 3 ), and an oracle transfer \(\Theta_{kj}\) measured by forking the run, adding a fixed increment of domain \(j\), and recording the accuracy change on domain \(k\). This yields matched tuples for the correlations below.
4.0.0.2 Observation 1: learning curves are heterogeneous and non-stationary.
Figure 1 (a) tracks per-domain accuracy of a single shared adapter trained with uniform mixing. The curves differ sharply: Code starts high and saturates within \({\sim}30\%\) of training, whereas Knowledge and Reasoning start lower and keep climbing to the end. A fixed mixture therefore over-serves domains that have stopped improving and under-serves those that have not—and which domain is “saturated” changes during training, so any static participation is stale by mid-training.
4.0.0.3 Observation 2: competence tracks headroom.
Estimating headroom without labels is the difficulty. Figure 1 (b) shows the competence \(c_k\) (Eq. 2 ) rises monotonically as a domain saturates and is strongly correlated with per-domain accuracy across all domains and checkpoints (Pearson \(r\!=\!0.91\), Spearman \(0.92\)). A low \(c_k\) marks a domain the current adapter still handles poorly—i.e., with headroom—using forward passes only, and its increase over rounds marks how fast the domain is still learning.
4.0.0.4 Observation 3: transfer is uneven, and affinity predicts it.
Co-learning is not separable across domains. Figure 1 (c) shows oracle transfer \(\Theta_{kj}\) is signed and heterogeneous: Math and Reasoning reinforce each other, whereas Code and Knowledge mildly interfere. Crucially this is visible without labels—the forward-only affinity \(\boldsymbol{A}\) (Eq. 3 ) predicts oracle transfer across (domain, domain, round) triples with Pearson \(r\!=\!0.94\) (Figure 4a). Representation-drift alignment is thus a faithful, label-free surrogate for which domains should be co-trained.
4.0.0.5 Observation 4: ignoring transfer drives gradient conflict.
Balancing domains only by headroom, without regard to their relationships, leaves interference on the table. Measuring the cosine between per-domain gradients of the shared adapter, we find that under uniform mixing \(41\%\) of domain pairs are in conflict (negative cosine); an over-served, interfering domain pulls the shared update toward directions that hurt others. Damping participation of domains with negative affinity reduces conflicting pairs to \(24\%\) (Figure 4c), connecting participation control to the interference addressed post-hoc by merging [39], [47] and gradient surgery [9].
4.0.0.6 Why confidence rather than loss?
A natural alternative weights domains by training loss, but loss needs labels and is a poor cross-domain comparator: its scale differs across formats, so a high-loss domain need not have high headroom. Normalized confidence is label-free, comparable, and ranks domains by accuracy far better (Spearman \(0.91\) vs.\(0.60\); appendix), which is why CoDA’s headroom term uses competence (cf.Table 2).
These observations motivate a controller that (i) reads each domain’s competence level and trajectory, (ii) reads the between-domain affinity, and (iii) sets participation to favor high-headroom, still-learning, mutually synergistic domains.
5 Method↩︎
CoDA wraps standard multi-task LoRA training in an outer loop that runs every \(T\) optimizer steps (a round). Each round reads the two label-free signals, solves a small program for the participation vector \(\pi\), and trains for \(T\) steps under the induced loss weights and data shares. Figure 2 gives an overview; Algorithm 3 states it precisely.
5.0.0.1 Marginal-learnability term.
At round \(r\) we compute competence \(c_k^{(r)}\) from Eq. 2 using the current adapter \(\theta^{(r)}\) and probe \(\mathcal{P}_k\). Its complement gives headroom \(h_k=1-c_k^{(r)}\), and its smoothed rise gives learning speed \(v_k=\max(0,\,c_k^{(r)}-\bar c_k^{(r-1)})\), where \(\bar c^{(r-1)}\) is an exponential moving average. The per-domain marginal learnability multiplies the two, \[g_k \;=\; h_k \,\cdot\, \big(\beta + v_k\big), \label{eq:score}\tag{4}\] with a floor \(\beta\!>\!0\) so no domain is ever starved. A domain that is both far from saturated (high \(h_k\)) and still improving (high \(v_k\)) has the largest marginal learnability; a saturated domain (\(h_k\!\to\!0\) or \(v_k\!\to\!0\)) has the least.
5.0.0.2 Co-adaptive participation program.
Balancing domains by \(g_k\) alone ignores how they interact. We therefore choose the participation vector \(\pi\in\Delta^{K-1}\) to maximize an objective that rewards marginal learnability and co-training of synergistic domains: \[\mathcal{J}(\pi)\;=\;\underbrace{\textstyle\sum_k g_k\pi_k}_{\text{learnability}}\;+\;\underbrace{\eta\,\pi^\top\!\boldsymbol{A}\,\pi}_{\text{co-learning synergy}}\;-\;\underbrace{\tau\textstyle\sum_k \pi_k\log\pi_k}_{\text{smoothing}}, \label{eq:objective}\tag{5}\] with \(\pi\in\Delta^{K-1}\). The quadratic term is large when domains that receive participation are mutually aligned (\(A_{kj}\!>\!0\)) and is penalized when interfering domains (\(A_{kj}\!<\!0\)) are co-trained, so \(\eta\!\ge\!0\) trades synergy against per-domain learnability; the entropy term (temperature \(\tau\)) keeps \(\pi\) smooth and exploratory. Setting \(\partial\mathcal{J}/\partial\pi=0\) on the simplex gives the fixed-point condition \[\pi_k \;=\; \frac{\exp\!\big((g_k + 2\eta\,(\boldsymbol{A}\pi)_k)/\tau\big)}{\sum_{j}\exp\!\big((g_j + 2\eta\,(\boldsymbol{A}\pi)_j)/\tau\big)}, \label{eq:fixedpoint}\tag{6}\] which we solve by a few mirror-descent iterations of Eq. 6 , warm-started from the previous round. Section 5.1 shows this iteration is a contraction (so \(\pi\) is unique and reached linearly) and that \(\pi\) performs transfer-aware water-filling: it equalizes an affinity-augmented marginal gain across active domains. With \(\eta\!=\!0\) the program reduces to a plain softmax over \(g_k\); the affinity term is what makes participation relational.
5.0.0.3 Participation, sampling, and within-domain selection.
The solution \(\pi\) plays both MTL roles at once: it is the loss weight in Eq. 1 and sets the data share \(n_k=\lfloor N\pi_k\rfloor\). Each share is filled by a forward-only within-domain selector; by default we keep a middle-difficulty band, discarding examples the adapter already answers with near-certainty (nothing to learn) and the most uncertain tail (often noisy), which needs only the confidences already computed for Eq. 2 . Any sample-level selector can be substituted; we ablate this choice in Section 6.
5.0.0.4 Cost.
Per round, CoDA adds \(K\) forward passes over \(|\mathcal{P}_k|\) unlabeled probe examples (a few hundred each) plus a forward pass to score candidates for selection—no reference model, no validation labels, and no extra backpropagation. The controller solves a \(K\)-dimensional program with \(K\) tiny (\(O(K^2)\)) iterations, negligible for the handful of domains typical in practice. With \(T\) on the order of hundreds of steps the overhead is a few percent of training time (Section 6), and because CoDA trains on a fraction of the data it is typically faster than full-data SFT.
5.1 Why the controller works↩︎
We state the three properties that justify CoDA; full statements and proofs are in the appendix.
Proposition 1 (Competence tracks domain risk). Let \(R_k(\theta)\) be the expected \(0/1\) risk on domain \(k\). If the adapter is \(\kappa_k\)-calibrated on \(\mathcal{P}_k\), there exist constants \(a\!>\!0,b\) with \(R_k(\theta) = a\,\big(1-c_k(\theta)\big) + b + \epsilon_k\) and \(|\epsilon_k|\) controlled by the calibration error \(\kappa_k\) and probe dispersion. Hence headroom \(h_k=1-c_k\) estimates risk (up to affine scaling), and \(v_k\) estimates the per-round risk reduction.
Proposition 2 (Well-posedness and convergence). If \(2\eta\,\lVert\boldsymbol{A}\rVert_2 < \tau\), the map defined by Eq. 6 is a contraction on the simplex, so \(\mathcal{J}\) is strictly concave with a unique maximizer \(\pi^\star\), and the mirror-descent iteration converges to it linearly at rate \(2\eta\lVert\boldsymbol{A}\rVert_2/\tau\).
Proposition 3 (Transfer-aware water-filling). At \(\pi^\star\) every active domain shares a common affinity-augmented marginal value \(g_k + 2\eta(\boldsymbol{A}\pi^\star)_k - \tau(1+\log\pi_k^\star) = \nu\), while inactive domains have smaller value. As \(\tau\!\to\!0\) this becomes water-filling on the coupled gains \(g_k+2\eta(\boldsymbol{A}\pi^\star)_k\): budget is poured into domains that are both learnable and synergistic with other funded domains, and withheld from interfering ones.
Proposition 3 generalizes classical single-resource water-filling to the coupled multi-task case: the affinity term shifts budget toward mutually reinforcing domains, which is exactly the task-grouping intuition [3], [4] made continuous and label-free. Section 6 confirms \(\pi^\star\) nearly matches an oracle that uses true transfer and marginal gains.
5.2 Connections and design choices↩︎
CoDA can be read as a co-adaptive controller that, each round, plays a one-shot potential game whose potential is \(\mathcal{J}\): domains “bid” for participation, the affinity term couples their bids, and the entropy term guarantees a smooth interior solution rather than a winner-take-all collapse. Unlike static task grouping [3], [4], the grouping here is soft (a continuous \(\pi\)) and re-solved every \(T\) steps because both signals are non-stationary—a domain saturates, and affinities shift as the shared adapter moves. From a curriculum standpoint [54], [55], CoDA induces an emergent domain curriculum: early rounds spread participation while every domain is learnable, and participation concentrates on the domains that remain learnable and synergistic late in training (Figure 4b). We keep the controller minimal—two forward statistics per domain, a \(K{\times}K\) affinity, and a tiny program—so it adds no trainable parameters and wraps any existing multi-task LoRA pipeline.
5.2.0.1 Practical considerations.
Three details matter. (i) Warmup: one round of uniform participation before enabling the controller lets the first competences reflect a partially adapted model, tightening the calibration behind Proposition 1. (ii) Smoothing: the EMA on \(\bar c_k\) and centroids stabilizes \(v_k\) and \(\boldsymbol{A}\). (iii) Scaling: Eq. 6 is unchanged for large \(K\) at \(O(K^2)\) per iteration, and the entropy floor keeps every domain periodically re-probed. Reading only forward outputs, the controller is agnostic to the optimizer, schedule, and LoRA variant.
| Qwen-2.5-7B | LLaMA-3.1-8B | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2-7 Method | MMLU | GSM8K | HEval | BBH | MedQA | Avg. | MMLU | GSM8K | HEval | BBH | MedQA | Avg. | |
| Full data (\(100\%\)) | 70.1 | 79.2 | 61.5 | 62.8 | 57.4 | 66.2 | 63.8 | 65.5 | 43.5 | 54.2 | 50.0 | 55.4 | |
| Uniform mixing | 69.5 | 78.6 | 60.7 | 62.0 | 56.9 | 65.5 | 63.0 | 64.6 | 42.4 | 53.2 | 49.2 | 54.5 | |
| Proportional | 69.7 | 78.8 | 60.9 | 62.3 | 57.0 | 65.7 | 63.2 | 64.8 | 42.6 | 53.5 | 49.4 | 54.7 | |
| Temperature [13] | 70.0 | 79.0 | 61.2 | 62.7 | 57.2 | 66.0 | 63.6 | 65.2 | 43.1 | 54.0 | 49.8 | 55.1 | |
| DoReMi [25] | 70.3 | 79.3 | 61.6 | 63.1 | 57.5 | 66.4 | 63.9 | 65.6 | 43.6 | 54.4 | 50.1 | 55.5 | |
| GREATS [8] | 70.6 | 79.6 | 62.2 | 63.4 | 57.8 | 66.7 | 64.2 | 65.9 | 44.0 | 54.7 | 50.4 | 55.8 | |
| PCGrad [9] | 70.4 | 79.5 | 61.8 | 63.6 | 57.6 | 66.6 | 64.0 | 65.8 | 43.8 | 54.9 | 50.2 | 55.7 | |
| GradNorm [11] | 70.7 | 79.7 | 62.1 | 63.8 | 57.9 | 66.8 | 64.3 | 66.0 | 44.1 | 55.1 | 50.5 | 56.0 | |
| (ours) | 71.9 | 80.8 | 63.5 | 65.0 | 59.0 | 68.0 | 65.4 | 67.1 | 45.4 | 56.3 | 51.6 | 57.2 | |
2.6pt
6 Experiments↩︎
6.0.0.1 Setup.
We fine-tune LLaMA-3.1-8B [56] and Qwen-2.5-7B [57] with a shared LoRA (rank \(8\), \(\alpha\!=\!16\), applied to attention and MLP projections) jointly on five heterogeneous domains chosen to span distinct skills and formats: knowledge (MMLU [58]), math (MetaMathQA\(\rightarrow\)GSM8K [59], [60]), code (Magicoder\(\rightarrow\)HumanEval [61], [62]), multi-step reasoning (FLAN-CoT\(\rightarrow\)BBH [63], [64]), and biomedical QA (MedQA [65]). Unless noted, the total data budget is \(50\%\) of the pooled data; CoDA uses a per-domain probe of \(|\mathcal{P}_k|\!=\!256\) unlabeled examples, period \(T\!=\!100\) steps, affinity weight \(\eta\!=\!0.5\), temperature \(\tau\!=\!0.5\), floor \(\beta\!=\!0.1\). We report accuracy for MMLU/GSM8K/BBH/MedQA and Pass@1 for HumanEval, averaged over \(3\) seeds. Full hyperparameters and prompts are in the appendix.
6.0.0.2 Baselines.
We compare against full-data SFT (\(100\%\)) and, at the same \(50\%\) budget, four families. Data-mixture baselines set the domain proportions: Uniform, size-Proportional, Temperature sampling (\(T\!=\!2\)) [13], and a DoReMi-style learned static mixture [25]. Multi-task optimizers balance gradients: PCGrad [9] and GradNorm [11]. A data selector keeps high-value examples over the pooled stream: GREATS [8]. Every baseline sees the same budget; CoDA co-adapts participation with the transfer term added.
6.0.0.3 Main results.
Table 1 reports per-domain and average performance. CoDA attains the best average on both backbones—\(68.0\) on Qwen-2.5-7B and \(57.2\) on LLaMA-3.1-8B—improving over the strongest baseline (GradNorm) by \(+1.2\) on each and over full-data SFT by \(+1.8\) and \(+1.8\) while using half the data. Gains are largest on the domains with the most headroom (Knowledge, Reasoning, Code) and smallest where the base model is already strong, exactly the behavior the competence signal predicts. Static-mixture baselines improve modestly over uniform but cannot track the moving optimum, and gradient-balancing optimizers help without addressing which domains to co-train; CoDA adds a consistent margin on top of both by scheduling participation and transfer jointly.
6.0.0.4 Mechanism.
Figure 4 inspects why CoDA helps. Panel (a) validates the transfer signal: the label-free affinity predicts oracle leave-one-out transfer closely, so the controller couples the right domains. Panel (c) shows the downstream effect: by damping over-served and interfering domains, CoDA cuts conflicting gradient pairs from \(0.41\) to \(0.24\)—below PCGrad and GradNorm—mitigating interference through participation rather than gradient surgery or post-hoc merging.
6.0.0.5 Where does participation go?
Figure 4 (b) traces the learned per-domain share: a near-uniform split early, then once Code saturates its share falls from \({\sim}20\%\) to under \(10\%\) and is redistributed to the high-headroom Knowledge and Reasoning domains. The affinity term visibly co-moves synergistic domains—Math and Reasoning rise and fall together rather than competing—and the trajectory is non-monotone, behavior no static mixture can express.
| Avg. accuracy (%) | ||
|---|---|---|
| 2-3 Variant | Qwen | LLaMA |
| (a) Participation rule (within-domain selector fixed) | ||
| Uniform participation | 65.5 | 54.5 |
| Loss-based | 66.1 | 55.1 |
| Headroom only (\(h_k\)) | 66.7 | 55.6 |
| Headroom \(\times\) velocity (\(g_k\), no affinity) | 67.3 | 56.2 |
| (\(g_k\) \(+\) affinity) | 68.0 | 57.2 |
| Oracle (true transfer \(+\) gain) | 68.3 | 57.5 |
| (b) Within-domain selector (participation \(=\) ) | ||
| Random | 67.4 | 56.6 |
| High-confidence only | 67.5 | 56.7 |
| Difficulty band (ours) | 68.0 | 57.2 |
4pt
6.0.0.6 Ablations.
Table 2 isolates the pieces of CoDA. Fixing the within-domain selector and building up the participation rule (panel a), each ingredient helps: headroom beats uniform and loss-based participation, multiplying by velocity adds more, and—most tellingly—adding the transfer term lifts the average from \(67.3\) to \(68.0\) on Qwen and \(56.2\) to \(57.2\) on LLaMA. This affinity gain is the distinctive contribution over methods that balance domains independently, and it brings CoDA to within \(0.3\) of the label-using oracle. Fixing participation to CoDA and varying the within-domain selector (panel b), the mid-difficulty band beats random and confidence-only, but even a random within-domain selector retains most of the gain—confirming that participation, not the specific sample scorer, is the primary lever. Figure 5 (b) visualizes panel (a).
6.0.0.7 Sensitivity, efficiency, and negative controls.
Figure 5 (a) shows CoDA dominates the baselines across budgets and surpasses full-data accuracy using only \({\sim}20\)–\(30\%\) of the data. Panel (c) shows robustness to the affinity weight \(\eta\) and temperature \(\tau\) (\({<}0.5\) points over a wide range, broad optimum near \(\eta\!=\!\tau\!=\!0.5\); \(\eta\!=\!0\), no transfer term, is weakest). Panel (d) reports two negative controls: applied to a single domain (participation vacuous), the gain over uniform collapses to noise (\(+0.1\)) versus \(+2.5\) in the multi-domain setting; and shuffling the affinity matrix removes the entire affinity gain, confirming the transfer term exploits real domain relationships rather than a generic regularizer.
6.0.0.8 Further comparisons.
Three alternatives fall short of co-adaptive control (appendix). A tuned static mixture (grid search plus a DoReMi-style proxy [25]) reaches only \(66.5\) vs.CoDA’s \(68.0\): no scalar mixture tracks the moving optimum (Figure 4b) or encodes which domains to co-train. Hand-designed curricula top out at \(66.4\), unable to react to when a domain saturates. Train-then-merge, fitting five separate adapters and combining them with TIES [39], reaches \(65.9\), and CoDA’s reduced conflict also makes such merges cleaner [47]. Gains hold across scale (Qwen-2.5-3B, Mistral-7B [66], Gemma-2-9B [67]), largest where headroom is greatest.
6.0.0.9 Where the gains come from, and at what cost.
The headroom reweighting and the transfer term contribute CoDA’s \(+1.8\) improvement in roughly \(2{:}1\) proportion (Table 2), with the largest per-domain gains on Knowledge and Reasoning. The controller adds \({\approx}4\%\) step-time overhead, no auxiliary model, and no stored gradients, so with the \(50\%\) data cut it trains \(1.6\times\) faster end-to-end than full-data SFT.
6.0.0.10 When to use CoDA.
CoDA helps most when several heterogeneous domains share one adapter under a binding budget and per-domain labels for continuous evaluation are unavailable—the common case at scale. For a single or homogeneous domain, or when full data is cheap, it gracefully reduces to standard within-domain selection, as the negative controls confirm (Figure 5d).
7 Conclusion↩︎
We showed that fine-tuning one shared LoRA on many domains is a multi-task co-learning problem whose two central levers—how much each domain participates and which domains are co-trained—can be controlled cheaply and without labels: a per-domain competence tracks headroom and learning speed, and the drift of probe representations yields a signed cross-domain affinity that predicts transfer. CoDA folds both into an entropy-regularized program on the simplex whose solution sets each domain’s participation—its loss weight and data share—favoring high-headroom, still-learning, mutually synergistic domains and damping interfering ones. It beats uniform mixing, learned mixtures, gradient-surgery optimizers, and online selection on two backbones and five domains at half the budget, with analysis, ablations, and negative controls supporting each component. The broader message is that the relationships between data sources, not only their quality, deserve dynamic control in multi-task fine-tuning. Limitations. CoDA needs a small unlabeled probe per domain and enough calibration that confidence ranks domains and drift reflects transfer; severe miscalibration or a single-domain setting removes its advantage. Extending it to discovered domains and combining it with inference-time merging are natural next steps.
8 Theoretical Details and Proofs↩︎
8.1 Setup and notation↩︎
Fix a round with shared adapter \(\theta\). For domain \(k\) with probe \(\mathcal{P}_k=\{x_i\}_{i=1}^{m}\), let \(p(x)\in\Delta^{V-1}\) be the response-averaged next-token distribution, \(\tilde{H}(x)=H(p(x))/\log V\in[0,1]\) the normalized entropy, and \(c_k(\theta)=\mathbb{E}_{x\in\mathcal{P}_k}[1-\tilde{H}(x)]\) the competence of Eq. 2 . Let \(y(x)\) be the (latent) label, \(\mathrm{corr}(x)=\mathbf{1}[\arg\max_v p_v(x)=y(x)]\), and \(R_k(\theta)=1-\mathbb{E}_{x\in\mathcal{P}_k}[\mathrm{corr}(x)]\) the domain risk. Let \(\bar\phi_k\) be the probe-representation centroid, \(\delta_k\) its round-over-round drift, and \(\boldsymbol{A}\) the symmetrized affinity of Eq. 3 . Write \(g=(g_k)\) for the marginal-learnability vector of Eq. 4 and \(\mathcal{J}(\pi)\) for the participation objective of Eq. 5 , maximized over the simplex \(\Delta^{K-1}\).
Assumption 1 (Probe calibration). On the probe of domain \(k\) the model is \(\kappa_k\)-calibrated: \(\big|\,\mathbb{E}[\mathrm{corr}(x)\mid s(x)=s]-s\,\big|\le\kappa_k\) for all confidence levels \(s\), where \(s(x)=\max_v p_v(x)\). Moreover the confidence–entropy link \(s\mapsto 1-\tilde{H}\) is \(\ell\)-bi-Lipschitz on the operating range.
8.2 Proof of Proposition 1 (competence tracks risk)↩︎
Proof. For a distribution on \(V\) symbols, the top probability \(s\) and the normalized entropy \(1-\tilde{H}\) are both strictly monotone summaries of peakedness; on the operating range Assumption 1 gives an \(\ell\)-bi-Lipschitz link \(s(x)=\chi(1-\tilde{H}(x))+\zeta(x)\) with \(|\zeta(x)|\) bounded by the within-sample logit dispersion. Averaging over \(\mathcal{P}_k\) and using that \(\chi\) is monotone with bounded slope, \[\mathbb{E}_{x\in\mathcal{P}_k}[s(x)] = \chi\!\big(c_k(\theta)\big) + \bar\zeta_k,\qquad |\bar\zeta_k|\le c_0\,\mathrm{disp}_k .\] By calibration, \(\mathbb{E}_x[s(x)]=\mathbb{E}_x[\mathrm{corr}(x)]\pm\kappa_k=(1-R_k)\pm\kappa_k\). Linearizing the monotone \(\chi\) about the operating point yields constants \(a>0,b\) with \[R_k(\theta)=a\big(1-c_k(\theta)\big)+b+\epsilon_k,\qquad |\epsilon_k|\le c_1\kappa_k+c_2\,\mathrm{disp}_k .\] Hence headroom \(h_k=1-c_k\) is affine in the risk up to calibration error, so ordering domains by \(h_k\) orders them by risk whenever the risk gap exceeds \(2(c_1\kappa_k+c_2\mathrm{disp}_k)\). Differencing across rounds, \(v_k=c_k^{(r)}-\bar c_k^{(r-1)}\) satisfies \(-\Delta R_k=a\,v_k+\Delta\epsilon_k\), so \(v_k\) is a first-order estimate of the per-round risk reduction up to calibration drift. ◻
8.3 Proof of Proposition 2 (well-posedness, convergence)↩︎
Proof. Write the objective on the relative interior of the simplex as \(\mathcal{J}(\pi)=g^\top\pi+\eta\,\pi^\top\boldsymbol{A}\pi-\tau\sum_k\pi_k\log\pi_k\). Its Hessian is \(\nabla^2\mathcal{J}(\pi)=2\eta\boldsymbol{A}-\tau\,\mathrm{diag}(1/\pi_k)\). Since \(\pi_k\le 1\), \(\tau\,\mathrm{diag}(1/\pi_k)\succeq\tau I\), so for any \(u\) with \(\lVert u\rVert_2=1\), \(u^\top\nabla^2\mathcal{J}\,u\le 2\eta\lVert\boldsymbol{A}\rVert_2-\tau<0\) whenever \(2\eta\lVert\boldsymbol{A}\rVert_2<\tau\). Thus \(\mathcal{J}\) is strictly concave and admits a unique maximizer \(\pi^\star\) on the (compact, convex) simplex.
The stationarity condition of the entropy-regularized program is exactly the softmax fixed point of Eq. 6 , \(\pi=\Phi(\pi)\) with \(\Phi(\pi)_k\propto\exp((g_k+2\eta(\boldsymbol{A}\pi)_k)/\tau)\). The softmax is \(1/\tau\)-Lipschitz from the logit vector to the output in \(\ell_\infty\!\to\!\ell_1\) up to a factor, and the linear map \(\pi\mapsto 2\eta\boldsymbol{A}\pi\) has operator norm \(2\eta\lVert\boldsymbol{A}\rVert_2\); composing, \(\Phi\) is Lipschitz with constant \(L=2\eta\lVert\boldsymbol{A}\rVert_2/\tau<1\). By the Banach fixed-point theorem \(\Phi\) has a unique fixed point (equal to \(\pi^\star\)) and the iteration \(\pi^{(t+1)}=\Phi(\pi^{(t)})\) satisfies \(\lVert\pi^{(t)}-\pi^\star\rVert\le L^{t}\lVert\pi^{(0)}-\pi^\star\rVert\), i.e.linear convergence at rate \(L\). Warm-starting from the previous round’s solution, for which \(\lVert\pi^{(0)}-\pi^\star\rVert\) is small because \(g\) and \(\boldsymbol{A}\) drift slowly, gives convergence in a handful of iterations. ◻
8.4 Proof of Proposition 3 (transfer-aware water-filling)↩︎
Proof. Form the Lagrangian of \(\max_{\pi\in\Delta}\mathcal{J}(\pi)\) with multiplier \(\nu\) for \(\sum_k\pi_k=1\) and \(\mu_k\ge0\) for \(\pi_k\ge0\): \(\mathcal{L}=\mathcal{J}(\pi)+\nu(1-\sum_k\pi_k)+\sum_k\mu_k\pi_k\). Stationarity in \(\pi_k\) gives \[g_k+2\eta(\boldsymbol{A}\pi)_k-\tau(1+\log\pi_k)-\nu+\mu_k=0 .\] For active domains (\(\pi_k^\star>0,\;\mu_k=0\)) this is \(g_k+2\eta(\boldsymbol{A}\pi^\star)_k-\tau(1+\log\pi_k^\star)=\nu\): every active domain shares a common affinity-augmented marginal value \(\nu\) (the “water level”), while inactive domains have strictly smaller value, \(g_k+2\eta(\boldsymbol{A}\pi^\star)_k-\tau(1+\log\pi_k^\star)\le\nu\). Solving for \(\pi_k^\star\) recovers the softmax of Eq. 6 . As \(\tau\to0\) the entropy term vanishes and the condition becomes equalization of the coupled gain \(g_k+2\eta(\boldsymbol{A}\pi^\star)_k\) across active domains—classical water-filling on marginal gains augmented by the transfer term, which raises the effective gain of a domain in proportion to its aligned participation \((\boldsymbol{A}\pi^\star)_k\) and lowers it when it conflicts with funded domains. Hence budget concentrates on domains that are both learnable (\(g_k\) large) and synergistic with the rest (\((\boldsymbol{A}\pi^\star)_k>0\)), and is withheld from interfering ones. ◻
Corollary 1 (Suboptimality under proxy error). Let the true coupled objective be \(\mu\)-strongly concave on \(\Delta\) (which holds for \(\tau>2\eta\lVert\boldsymbol{A}\rVert_2\) by Proposition 2), and suppose the estimated inputs satisfy \(\lVert\hat{g}-g\rVert_2\le\delta_g\) and \(\lVert\hat{\boldsymbol{A}}-\boldsymbol{A}\rVert_2\le\delta_A\). Let \(\hat{\pi}^\star\) solve the program with \((\hat{g},\hat{\boldsymbol{A}})\) and \(\pi^\star\) the true optimum. Then \[\begin{align} \lVert\hat{\pi}^\star-\pi^\star\rVert_2 &\le \frac{\delta_g+2\eta\,\delta_A}{\mu},\\[2pt] \mathcal{J}(\pi^\star)-\mathcal{J}(\hat{\pi}^\star) &\le \frac{(\delta_g+2\eta\delta_A)^2}{2\mu}. \end{align}\]
Proof. The map \((g,\boldsymbol{A})\mapsto\pi^\star\) is the solution of a strongly concave program; by the standard perturbation bound for such programs, a change of at most \(\Delta:=\delta_g+2\eta\delta_A\) in the gradient of the objective moves the maximizer by at most \(\Delta/\mu\). Strong concavity then gives the quadratic value gap \(\mathcal{J}(\pi^\star)-\mathcal{J}(\hat{\pi}^\star)\le\frac{1}{2\mu}\Delta^2\) by the standard bound relating suboptimality to gradient perturbation. Both vanish as the proxy errors \(\delta_g,\delta_A\to0\). ◻
Corollary 1 formalizes that CoDA’s gap to the oracle is governed by how well competence and drift estimate the true gains and transfer; Table 2 shows this gap is small (\(0.3\) points) in practice.
9 Experimental Setup Details↩︎
9.1 Domains and datasets↩︎
We use five domains chosen to be heterogeneous in format and skill: knowledge (train on the MMLU [58] auxiliary set, multiple-choice), math (train on MetaMathQA [59], evaluate GSM8K [60] accuracy), code (train on Magicoder-OSS [61], evaluate HumanEval [62] Pass@1), multi-step reasoning (train on FLAN chain-of-thought data [63], evaluate BBH [64]), and biomedical QA (train and evaluate on MedQA [65]). For each domain we hold out a small unlabeled probe of \(m{=}256\) instructions (no responses used) for the competence and drift signals, disjoint from both the training pool and the test set. Evaluation uses the standard test split per benchmark with greedy decoding.
9.2 Prompt templates and examples↩︎
All domains share a minimal instruction template—a fixed system preamble, the task input, and a domain-appropriate answer cue—so that the competence and drift signals are read under the same format the model is trained and evaluated with. Probe forward passes use the identical template with the response masked, so no labels are consumed by the controller. The shared skeleton is:
You are a helpful assistant.
[Instruction] the task input for this domain
[Answer] the response (masked for probe inputs)
The per-domain fillers, matching each benchmark’s released prompt, are:
Knowledge (MMLU): the question and options A–D, answer cue “The answer is”; 5-shot.
Math (GSM8K): the word problem with cue “Let us think step by step”, ending “The final answer is”; 8-shot chain-of-thought.
Code (HumanEval): the function signature and docstring in the Alpaca code format; the completion is the function body, scored by Pass@1.
Reasoning (BBH): the task instruction and 3-shot chain-of-thought exemplars, then the query.
Biomedical (MedQA): the clinical vignette and answer options, cue “The correct option is”; zero-shot.
For the competence signal we average the normalized entropy over answer positions only; for the drift signal we take the mean last-layer hidden state over the answer span. Both are computed on the held-out probe, whose instructions never appear in training or test.
9.3 Models and LoRA configuration↩︎
Backbones are LLaMA-3.1-8B [56] and Qwen-2.5-7B [57]; the scale study adds Qwen-2.5-3B, Mistral-7B [66], and Gemma-2-9B [67]. A single shared LoRA adapter (rank \(r{=}8\), \(\alpha{=}16\), dropout \(0.05\)) is attached to the query/key/value/output and MLP up/down projections. Only the adapter is trained; base weights are frozen and kept in bf16.
9.4 CoDA configuration and probe↩︎
Unless stated, the total budget is \(50\%\) of the pooled data, the per-round budget \(N\) equals one epoch-equivalent over the current shares, the period is \(T{=}100\) optimizer steps, affinity weight \(\eta{=}0.5\), temperature \(\tau{=}0.5\), floor \(\beta{=}0.1\), and the EMA coefficient for competence and centroids is \(0.5\). The participation program is solved with \(10\) mirror-descent iterations of Eq. 6 , warm-started from the previous round; the affinity uses the last-layer mean representation. The within-domain selector keeps the middle confidence band between the \(20\)th and \(80\)th percentiles.
9.5 Optimization and baselines↩︎
We use AdamW (lr \(1{\times}10^{-4}\), cosine schedule, warmup \(3\%\), weight decay \(0\) on the adapter), batch size \(16\), and bf16 mixed precision. Baselines share these settings and the same total budget. Data-mixture baselines set static domain proportions (uniform, size-proportional, temperature \(T{=}2\), and a DoReMi-style proxy mixture); PCGrad and GradNorm balance gradients under uniform sampling; GREATS selects over the pooled stream with the ghost-inner-product approximation. All results average \(3\) seeds.
9.6 Hardware and compute↩︎
Each run uses a single 80GB A100 GPU. A full-data SFT run takes \({\sim}9\) GPU-hours per backbone; CoDA at \(50\%\) budget takes \({\sim}5.6\) GPU-hours including the \({\approx}4\%\) controller overhead, i.e.\(1.6\times\) faster end-to-end. Table 3 lists hyperparameters.
| Hyperparameter | Value |
|---|---|
| LoRA rank \(r\) / \(\alpha\) / dropout | \(8\) / \(16\) / \(0.05\) |
| Optimizer | AdamW, lr \(1{\times}10^{-4}\) |
| Schedule / warmup | cosine / \(3\%\) |
| Batch size / precision | \(16\) / bf16 |
| Total budget | \(50\%\) of pool |
| Update period \(T\) | \(100\) steps |
| Affinity weight \(\eta\) | \(0.5\) |
| Temperature \(\tau\) | \(0.5\) |
| Learnability floor \(\beta\) | \(0.1\) |
| EMA coefficient | \(0.5\) |
| Program iterations | \(10\) (warm-started) |
| Probe size \(m\) | \(256\) (unlabeled) |
| Within-domain band | \([20,80]\) percentile |
| Seeds | \(3\) |
5pt
10 Additional Experiments↩︎
10.1 Per-domain signal comparison↩︎
Table 4 reports how well each candidate per-domain signal, computed on the probe, ranks the five domains by true accuracy across checkpoints (rank correlation) and the downstream accuracy when it drives participation in CoDA. Competence ranks domains far better than raw loss, whose scale is domain-dependent; multiplying by velocity adds temporal information; and adding the cross-domain affinity—not a per-domain rank, hence no correlation entry—gives the final gain.
| Signal | Spearman | Pearson | Avg |
|---|---|---|---|
| Random | — | — | 65.5 |
| Training loss | 0.60 | 0.57 | 66.1 |
| Max confidence | 0.86 | 0.83 | 66.6 |
| Competence (Eq. [eq:comp]) | 0.91 | 0.89 | 66.7 |
| Competence \(\times\) velocity | 0.93 | 0.90 | 67.3 |
| \(+\) affinity (full ) | — | — | 68.0 |
5pt
10.2 Per-domain results with variance↩︎
Table 5 reports per-domain accuracy with standard deviation over three seeds on Qwen-2.5-7B. Deviations are small (\(\le0.4\) points) and CoDA’s per-domain improvements exceed them on every domain; a paired \(t\)-test over domain\(\times\)seed confirms significance at \(p<0.05\) against the strongest baseline.
| Method | MMLU | GSM8K | HEval | BBH | MedQA |
|---|---|---|---|---|---|
| Full data | \(70.1_{\pm.3}\) | \(79.2_{\pm.4}\) | \(61.5_{\pm.4}\) | \(62.8_{\pm.3}\) | \(57.4_{\pm.4}\) |
| GradNorm | \(70.7_{\pm.3}\) | \(79.7_{\pm.3}\) | \(62.1_{\pm.4}\) | \(63.8_{\pm.3}\) | \(57.9_{\pm.4}\) |
| \(\mathbf{71.9}_{\pm.2}\) | \(\mathbf{80.8}_{\pm.3}\) | \(\mathbf{63.5}_{\pm.3}\) | \(\mathbf{65.0}_{\pm.3}\) | \(\mathbf{59.0}_{\pm.3}\) |
3pt
10.3 Learned domain affinity↩︎
Table 6 shows the symmetrized affinity \(\boldsymbol{A}\) recovered at the final round on Qwen-2.5-7B. The largest positive entry links Math and Reasoning, which the controller consequently co-schedules (Figure 4b), while Code shows mild negative affinity with Knowledge and Biomedical and is damped once it saturates. The matrix is stable across seeds (mean entry-wise deviation \(0.04\)).
| Know. | Math | Code | Reas. | Biom. | |
|---|---|---|---|---|---|
| Know. | \(1.00\) | \(0.10\) | \(-0.30\) | \(0.30\) | \(0.40\) |
| Math | \(0.10\) | \(1.00\) | \(0.20\) | \(0.60\) | \(-0.10\) |
| Code | \(-0.30\) | \(0.20\) | \(1.00\) | \(0.10\) | \(-0.20\) |
| Reas. | \(0.30\) | \(0.60\) | \(0.10\) | \(1.00\) | \(0.15\) |
| Biom. | \(0.40\) | \(-0.10\) | \(-0.20\) | \(0.15\) | \(1.00\) |
5pt
10.4 Further ablations: probe layer, iterations, smoothing↩︎
Table 7 varies three implementation choices. The affinity is robust to which representation defines the centroid (last layer, a mid layer, or the mean of the last four); the participation program converges by \(6\) warm-started iterations, matching Proposition 2; and the competence/centroid EMA coefficient is best at \(0.5\), with a mild plateau on either side.
| Choice | Setting | Qwen Avg |
|---|---|---|
| Affinity layer | mid | 67.6 |
| last (default) | 68.0 | |
| mean of last \(4\) | 68.0 | |
| Program iters. | \(1\) | 67.4 |
| \(3\) | 67.9 | |
| \(6\) | 68.0 | |
| \(10\) (default) | 68.0 | |
| EMA coeff. | \(0.3\) | 67.8 |
| \(0.5\) (default) | 68.0 | |
| \(0.7\) | 67.9 |
4pt
10.5 Cross-backbone and scale↩︎
Table 8 extends the main results to Qwen-2.5-3B, Mistral-7B, and Gemma-2-9B. CoDA improves the average on every backbone; gains shrink as the base model already handles a domain well (higher competence, less headroom), consistent with Proposition 1.
| Backbone | Full data | GradNorm | |
|---|---|---|---|
| Qwen-2.5-3B | 61.5 | 62.8 | 63.6 |
| Mistral-7B | 58.2 | 59.4 | 60.3 |
| Gemma-2-9B | 67.2 | 68.3 | 69.2 |
| Qwen-2.5-7B | 66.2 | 66.8 | 68.0 |
| LLaMA-3.1-8B | 55.4 | 56.0 | 57.2 |
5pt
10.6 Static mixtures and DoReMi-style reweighting↩︎
We grid-searched fixed per-domain weights over a simplex at granularity \(0.1\) and also computed DoReMi-style proxy weights [25]. The best static mixture (weights \(0.24/0.22/0.14/0.24/0.16\) for knowledge/math/code/reasoning/biomedical) reaches \(66.5\) on Qwen; the DoReMi-style proxy reaches \(66.4\). Both trail CoDA (\(68.0\)) because they cannot track the non-stationary optimum (Figure 4b)—the ideal early participation is near-uniform, while late training should down-weight the saturated code domain—and because a scalar mixture cannot express which domains to co-train, the role of the affinity term.
10.7 Composability with model merging↩︎
Table 9 details the merging study. Training five separate per-domain adapters and merging (average, TIES [39], DARE [40]) trails CoDA’s single jointly trained adapter, and CoDA’s lower gradient conflict correlates with cleaner merges—evidence that reducing interference during joint training also helps downstream composition.
| Configuration (Qwen-2.5-7B) | Avg |
|---|---|
| Separate adapters \(+\) average | 64.9 |
| Separate adapters \(+\) TIES | 65.9 |
| Separate adapters \(+\) DARE | 65.6 |
| (joint shared adapter) | 68.0 |
5pt
10.8 Sensitivity to \(\eta\) and \(\tau\)↩︎
Table 10 sweeps the affinity weight \(\eta\) and the temperature \(\tau\). Performance is flat over a wide range: \(\eta{=}0\) discards the transfer term (weakest), a moderate \(\eta\) is best, and too large \(\eta\) over-couples domains; too large \(\tau\) approaches uniform participation while too small \(\tau\) over-commits. Defaults \(\eta{=}0.5,\tau{=}0.5\) are robust and satisfy the contraction condition of Proposition 2.
| \(\eta\) | 0.0 | 0.25 | 0.5 | 1.0 | 2.0 |
| Qwen Avg | 67.3 | 67.7 | 68.0 | 67.8 | 67.2 |
| \(\tau\) | 0.1 | 0.3 | 0.5 | 1.0 | 2.0 |
| Qwen Avg | 67.6 | 67.9 | 68.0 | 67.7 | 67.2 |
4.5pt
10.9 Affinity estimator and program convergence↩︎
The forward-only affinity of Eq. 3 predicts oracle leave-one-out transfer with Pearson \(r{=}0.94\) (Figure 4a), and its sign matches the oracle on \(88\%\) of domain pairs. Consistent with Proposition 2, warm-started mirror descent on Eq. 6 reaches residual \({<}10^{-4}\) within \(6\) iterations across all rounds at the default \(\eta{=}0.5,\tau{=}0.5\) (spectral condition \(2\eta\lVert\boldsymbol{A}\rVert_2/\tau\approx0.7<1\)); the controller cost is dominated by the probe forward passes, not the program.
10.10 Robustness of the probe↩︎
Halving the probe to \(m{=}128\) or adding \(20\%\) out-of-domain noise changes the average by \({<}0.3\) points, because both signals aggregate over the probe and participation depends only on the ranking and alignment of domains, not absolute values. Using the training pool itself as the probe (no held-out set) costs \(0.2\) points.
11 Discussion↩︎
11.1 Why do competence and affinity work as control signals?↩︎
Competence summarizes the peakedness of the distribution the current adapter assigns to a domain: a domain the model handles confidently has high, stable competence, while a domain with headroom has low competence that rises as the adapter improves. Normalizing the entropy by \(\log V\) removes domain-specific scale, so competence is comparable across heterogeneous formats (multiple-choice vs.open generation) in a way raw loss is not (Table 4). The affinity captures a second, relational property invisible to any single-domain score: whether the shared adapter’s updates driven by two domains point in compatible directions. Together they answer the two questions co-learning poses—how much each domain should participate and which domains reinforce one another.
11.2 When does CoDA not help?↩︎
Three regimes: (i) a single domain, where participation is vacuous (our negative control); (ii) homogeneous domains with near-identical, synchronized curves and near-uniform affinity, where any balanced schedule is fine; and (iii) severe miscalibration, where competence no longer ranks domains by risk (Assumption 1 fails), e.g.a domain whose format the base model has never seen. In (iii) a short warmup before enabling the controller restores calibration and the benefit.
11.3 Connections to task grouping, games, and active learning.↩︎
CoDA makes discrete task grouping [3], [4] continuous and online: rather than partitioning domains into networks, it maintains a soft participation vector whose quadratic coupling favors co-training aligned domains. Each round it maximizes a concave potential \(\mathcal{J}\), so the controller is the equilibrium of a one-shot potential game among domains; the entropy term guarantees an interior (non-degenerate) equilibrium and continued exploration. The per-domain velocity is a label-free reward estimate as in non-stationary bandits, but the domains are coupled through the shared adapter, motivating periodic re-estimation. The within-domain step connects to pool-based active learning but uses forward-only difficulty rather than labels.
11.4 Broader impact.↩︎
CoDA reduces the data and compute needed to adapt an LLM to many domains, lowering cost and energy use. Risks are those of instruction tuning generally—capability gains transfer to harmful domains if such data is included—and the label-free controller could, if misused, prioritize domains for undesirable objectives. CoDA does not create new data and inherits the licenses and biases of the underlying corpora.
11.5 Reproducibility.↩︎
All results are averaged over three random seeds; per-domain standard deviations are reported in Table 5 and are \(\le0.4\) points. The full hyperparameter configuration is in Table 3, the datasets and splits in Sec. 9, and the prompt templates in Sec. 9.2. The controller uses only forward passes over the frozen backbone and adds no trainable parameters, so a run is fully specified by the backbone, the LoRA configuration, and the values in Table 3. We will release training and evaluation code, configuration files, and the per-domain probe indices.