1 Introduction↩︎

Techniques and intuitions developed for one neural-network family are often reused in another, but this transfer is rarely made precise. FFNs and CNNs are a common example: ideas developed for one are frequently applied to the other, even though the two families are usually formalized with different input structures and node operations. We study this issue through the lens of inheritance: the transfer of a property between model classes through an explicit structural relationship that preserves the assumptions required by that property. This distinguishes valid transfer from architectural analogy. Rather than asking whether two architectures look similar, we ask which assumptions are preserved when one model class is related to another, and which properties therefore transfer.

This perspective is especially useful for FFNs and CNNs because the two classes differ in their trainable input contributions. A traditional FFN node has one scalar weight per input contribution, whereas a standard CNN node has a spatial kernel for each input-output channel contribution. Thus, techniques whose assumptions depend on a linear combination in which each input contribution is controlled by a single scalar weight apply naturally to FFN nodes, but not automatically to ordinary CNN nodes. If a structural mapping allows CNNs to inherit this scalar-weighted trainable structure from FFNs, then the inherited structure can be naturally exploited for parameter-efficient transfer learning. This motivates our central question: when can CNNs and FFNs inherit properties from one another, and can such inheritance be used for parameter-efficient transfer learning?

A unified formalization. We first introduce a unified node-level formalization with tensor-valued activations. In this framework, generalized feedforward network (GFFN) nodes perform scalar-weighted summation across input channels while preserving tensor structure, whereas generalized convolutional network (GCNN) nodes apply channel-wise convolutional transformations and then sum across channels. This lets us compare the two model classes directly. We prove that GFFNs form a strict subset of GCNNs: a convolution with kernel size one in every spatial dimension reduces to scalar multiplication of each input channel. Therefore, any property that holds for all GCNNs also holds for the corresponding GFFN subclass.

Model projection. The reverse direction does not hold by subclass inclusion. Arbitrary CNNs are not GFFNs, because their input-output channel contributions are spatial kernels rather than scalar weights. To recover a restricted reverse direction, we introduce model projection. Projection uses the fact that CNN node functions are separable by input channel before the nonlinearity. It freezes each convolutional input-channel sub-function and learns one scalar coefficient for each input-output channel contribution. The resulting projected node has the same scalar-weighted trainable summation structure as a GFFN node, possibly with inhomogeneous inputs.

PEFT benefit. This gives the precise inheritance statement needed for our application: projected CNN nodes inherit the FFN-style trainable structure of one scalar trainable weight per input contribution. When applied to convolutional layers, this inherited property directly yields a parameter-efficient transfer-learning method. Instead of adapting an input-output channel contribution by training an entire spatial kernel, model projection freezes the pretrained kernel and trains only a scalar coefficient. Model projection therefore preserves pretrained spatial feature extractors while allowing every convolutional layer to adapt through scalar channel-contribution weights rather than full spatial kernels (See Fig. 1).

Empirical evaluation. We evaluate model projection as a parameter-efficient transfer-learning method for ImageNet-pretrained CNN backbones. Across multiple downstream image classification datasets and architectures, projection is competitive with logistic regression, full fine-tuning, and established PEFT baselines. Empirically, projection often combines the stability of partial tuning with the continued improvement of adapting the full model. In two-stage training, projection also provides a strong initialization for subsequent full fine-tuning.

Contributions. Our main contributions are:

• We introduce unified tensor-compatible formalizations of FFN and CNN nodes, enabling direct comparison between the corresponding model classes. Within this framework, we prove that GFFNs form a strict subset of GCNNs, establishing one rigorous inheritance direction.

• We introduce model projection as a structural mapping from separable CNN nodes to projected nodes with GFFN-style scalar-weighted trainable summation. We prove that projected nodes inherit FFN properties and techniques whose assumptions depend only on this structure and do not require homogeneous inputs across nodes.

• We turn this inherited scalar-weighted structure into a parameter-efficient transfer-learning method for CNNs. Projection adapts convolutional layers by training one scalar per input-output channel contribution rather than full spatial kernels, and experiments show that this yields a competitive accuracy-efficiency trade-off across transfer-learning benchmarks.

• We provide an open-source implementation of projection and demonstrate its practical value for transfer learning across multiple public datasets, showing that projection can adapt entire models while training substantially fewer parameters.

1.1 Related Work↩︎

1.1.0.1 Formalizations and inheritance across model families.

Formal treatments of neural-network computations are often specialized to particular architectures and input shapes rather than expressed in a unified, tensor-level language. Classical discussions of feedforward networks typically assume vector-valued scalar inputs [1], while standard formulations of convolutional networks are usually presented for specific spatial cases, most commonly two-dimensional inputs [2]. This separation makes it harder to state precise relationships between model classes, and therefore to identify when results or techniques from one class preserve the assumptions needed to apply to another. Although ideas are frequently transferred informally between FFNs and CNNs, explicit structural accounts of such transfer appear limited. This motivates a framework that makes the preserved node-level structure explicit.

1.1.0.2 Parameter-efficient transfer learning.

Transfer learning adapts a pretrained model to a downstream task, often in low-data regimes where full fine-tuning can overfit or be difficult to optimize [3]–[6]. Parameter-efficient transfer learning (PEFT) restricts the trainable degrees of freedom [7]. BatchNorm adaptation updates affine normalization parameters [8], but these act after convolutional input contributions have already been summed; projection instead rescales each fixed input-output channel contribution before aggregation. BitFit tunes bias terms [9], but biases shift node outputs rather than selectively reweighting individual contributions. Scale-shift methods modulate features with lightweight affine transformations [10], but typically act at the activation or channel level rather than per input-output contribution. Residual and convolutional adapters add auxiliary trainable modules [11]–[13], whereas projection keeps the original convolutional structure and only rescales pretrained filter-channel contributions. Low-rank adaptation learns additive weight updates [14], including convolution-specific variants [15], while projection restricts each pretrained filter channel to a one-dimensional multiplicative subspace. Sensitivity-aware methods allocate limited tuning capacity across layers [16], and are complementary to projection. Thus, projection is more fine-grained than input- or output-channel scaling, but more constrained than adapters or low-rank updates.

1.1.0.3 Pre-defined CNNs.

Model projection also differs from pre-defined CNNs, where a set of fixed filters produce outputs that are then passed to a layer of \(1 \times 1\) convolutional nodes [17]. In projection, the pretrained convolutional filter channels are not first aggregated into a shared intermediate representation. Instead, each output node rescales its own fixed input-channel contributions before summation. We elaborate on this distinction in Remark 5.

2 Background and Problem↩︎

2.1 Problem↩︎

Our goal is to answer the following questions: Under what conditions, if any, do CNNs naturally inherit results from FFNs, and vice versa, and, does this inheritance aid in transfer learning? For example, traditional feed forward neural networks train with fewer parameters per node. When fine tuning a CNN foundation model for a downstream task, can we inherit this property to efficiently train an entire neural network with fewer parameters?

Before introducing the relevant background, we first formalize what is meant by a neural network model. We define a network model \(M\mathrel{\vcenter{:}}= M(\theta)\), where \(\theta\in\mathbb{R}^p\), to be some function composed of \(L\in\mathbb{N}\) sequential layers. That is, \(M\) can be expressed as a composition of \(L\) subfunctions or layers \(M(\theta)=f_L \circ \ldots \circ f_1\). Given input \(Z_i\) and parameters \(W_i\), each layer \(f_i\) produces a sequence of at least two outputs \(a_{ij},\;j=1,\ldots,J_i, J_i \geq 2\). Each output \(a_{ij}\) corresponds to a node \(f_{ij}\) and subset of parameters \(W_{ij}\). We may refer to \(a_{ij}\) as the \(j\)th output channel of layer \(i\). Precisely, each layer can be written as \[\begin{align} f_i(Z_i, W_i) &= [f_{i1}(Z_i, W_{i1}),\ldots, f_{iJ_i}(Z_i, W_{iJ_i})] = [a_{i1}, \ldots, a_{iJ_i}] \\&= [Z_{(i+1,1)}, \dots , Z_{(i+1,J_i)}] = Z_{i+1}. \end{align}\]

2.2 Traditional Neural Networks↩︎

Traditional neural networks (FFNs) are a type of network model, where data flows along edges, and calculations are done in nodes. We summarize the description from Hastie et al. [1]. Traditionally, these calculations are linear regressions, followed by a non-linear function \(\sigma\). Let subset \(W_{jk}\) be the subset of parameters \(W_j\) that are associated with input channel \(Z_k\). Given an input \(Z \in \mathbb{R}^{d}\), a parameter set \(W_j\) containing a vector of weights \(W_{jk} \in \mathbb{R}^{d}\), and a bias \(b_{j} \in \mathbb{R}\), the node will have the following form: \[\label{FFN} f_{j}(Z, W_{j}) = \sigma(\sum_{k=1}^{d}Z_{k}W_{jk} + b_{j}).\tag{1}\] This formulation requires that the input and output data are vectors of scalars. While more general versions have been implemented in code [3], to the best of our knowledge, all formal treatments have so far required non-scalar data to be vectorized first.

2.3 Convolutional Neural Networks↩︎

Convolutional neural networks (CNNs) are another type of network model, but where some nodes perform a convolution. For simplicity, we describe the 2-dimensional case from [2], from which the more general case extends naturally. Given an input \(Z \in \mathbb{R}^{d \times n_1 \times n_2}\), and a parameter set containing a filter \(F_{j} \in \mathbb{R}^{d \times l_1 \times l_2}\) with filter channels \(F_{jk} \in \mathbb{R}^{l_1 \times l_2}\), and a bias \(B_{j} \in \mathbb{R}^{m_1 \times m_2}\), a convolutional node \(f_{j}\) will have the following form, where \(\ast\) denotes the convolution operation: \[\label{CNN} f_{j}(Z, W_{j}) = \sigma(\sum_k F_{jk} \ast Z_{k} + B_{j}).\tag{2}\] \(F_{jk}\) connects input channel \(Z_{k}\) to output \(a_{j}\). We consider the node \(f_{j}\) to be the collection of operations and parameters connecting \(Z\) to output \(a_{j}\), plus the bias. While CNNs often operate on 1-, 2-, or even 3-dimensional data, the common formulations often only specify the 2-dimensional case. For further reading see [2], [3], [18].

3 Inheritance between CNNs and FFNs↩︎

3.1 Node Generalization and Model Subclasses↩︎

We can now address our earlier question of inheritance through model subclasses and model projections. We start by generalizing equations for commonly used neural network models. In order to do so, we define a generalization of the dot product that outputs a tensor called the tensor dot product. Let a tensor be a multidimensional array with shape \(D\). Let \(\Gamma_D\) be a tensor of shape \(D\) where all values are \(1\).

The simplified tensor dot product takes a vector \(Z\), of \(d\) tensors, \(Z_{k} \in \mathbb{R}^{D}\), of uniform size and shape \(D\), as well as a vector of \(d\) scalars, \(W_{jk}\), and outputs a tensor of size and shape \(D\). Here, \(Z_{k}W_{jk}\) is scalar multiplication. If the size of \(Z_{k}\) is \(1\) in all dimensions, \(\odot_t\) reduces to the traditional dot product.

We now generalize FFN nodes to apply to structures of data other than vectors of scalars, and define a general convolutional node for a tensor of shape \(D\). Let \(\Gamma_D\) be a tensor of shape \(D\) where all values are equal to one.

To generalize an FFN node from Equation 1 , we use the tensor dot product and replace \(b_{j} \in \mathbb{R}\) with \(b_{j}\Gamma_{D_{out}} \in \mathbb{R}^{D_{out}}\). In other words, the bias is now a tensor with all values being \(b_{j}\). Traditional neural network nodes are a special case of GFFN nodes where the input data is a vector of scalars. If the 3-dimensional input is a sequence of 2D image channels, then a GFFN node acts as a \(1 \times 1\) CNN node. To act as a fully connected layer in the traditional sense, the input can be flattened first. We now define GCNN nodes.

That is, to generalize a CNN node from Equation 2 , we replace \(F_{j} \in \mathbb{R}^{d \times l_1 \times l_2}\) with \(F_{j} \in \mathbb{R}^{d \times D_{F_{j}}}\), we replace \(Z \in \mathbb{R}^{d \times n_1 \times n_2}\) with \(Z \in \mathbb{R}^{d \times D_{in}}\), and we replace \(B_{j} \in \mathbb{R}^{m_1 \times m_2}\) with \(b_{j}\Gamma_{D_{out}} \in \mathbb{R}^{D_{out}}\). Colloquially, the filters and data have the same number of dimensions in each channel, but the number of dimensions is not specified globally for the network. Instead of a vector of 2D images, we now have a vector of tensors. The bias is now a tensor with all values being \(b_{j}\).

Now that we have formalized general definitions, we can prove that the class of traditional feed forward neural networks (FFNs) is a subclass of the class of convolutional neural networks (CNNs).

The proof of this result can be found in Appendix 5. To prove Theorem 2, we establish a bijection between the set of GFFNs and a subset of GCNNs, namely the set of GCNNs with kernels of size one in all dimensions. An important consequence of Theorem 2 is the following.

Corollary 1 says that any results or techniques that apply to all GCNNs automatically apply to all GFFNs. Thus, any techniques and results from GCCNs carry over to GFFNs.

3.2 Model Projection↩︎

Of course, CNNs are not a subclass of FNNs. As a consequence, we now introduce model projection: a method to project CNNs and other non-FFN nodes, onto FFN nodes. This allows inheritance of many properties exclusive to FFNs, such as training a node with one weight per input. We first define a node function.

We now define separable by input, a property that a node function is required to have in order for the node to be projected.

“Separable by input” means that the node function decomposes into a sum of node sub-functions \(f_{jk}'\) plus a bias, where each \(f_{jk}'\) depends on only one input channel. For example, the function \(g_1(x_1,x_2) = a_1x_1^2 + a_2x_2^2 +a_3x_1 + a_4x_2\), where \(a_1,\ldots a_4\in\mathbb{R}\) is separable by input, since we can write \[\begin{align} g_1(x_1,x_2) = (a_1x_1^2 + a_3x_1) + (a_2x_2^2 + a_4x_2) = g_{11}(x_1) + g_{12}(x_2). \end{align}\] On the other hand, the function \(g_2(x_1,x_2) = a_1x_1^2+ a_2x_2^2 + a_3x_1 + a_4x_2 + x_1x_2\) is not separable by input, because of the \(x_1 x_2\) term. GCNN node functions are separable by input, see Lemma 1 in the Appendix. Node functions that are separable by input can be “projected.”

Here, the idea is that if we view the weights \(W_{jk}\) as fixed, and consider training the new weights \(\gamma_{jk}\) for a downstream task, then the projected model is essentially a GFFN with node-specific pre-processing and inherits all properties, including training techniques, for a GFFN that do not rely on homogeneous inputs, which are discussed next.

We now introduce a property of a layer called homogeneous inputs.

Our next result says that projected nodes are GFFN nodes with inhomogeneous inputs.

A direct consequence of Theorem 3 is the following.

Corollary 5.1 says that projected layers inherit properties that apply to GFFN layers with inhomogeneous inputs. In particular, given that GCNN node functions are separable by input, see Lemma 1 in the Appendix, we can project GCNN layers. This is especially useful for transfer learning. For example, GFFN nodes can be trained with one weight per input channel. This is true even in layers with inhomogeneous inputs. Therefore, a projected GCNN node can be trained with one weight per input channel, potentially greatly reducing the total number of parameters being trained.

In order to test the usefulness of model projection for transfer learning, we implemented model projection for 2D convolutional layers as a modified version of a 2D convolutional layer. The main difference is that it also includes a new set of parameters, \(\gamma_{jk}\), one for each channel, \(k\), in each convolutional node, \(j\). We also fix the filter weights, while the new parameters and biases are trained. Each \(\gamma_{jk}\) is initialized to one, so that the first forward pass through the network will be the same as before model projection. Projected nodes learn a linear regression before \(\sigma\) is applied, but unlike a GFFN node, projected nodes also perform an extra pre-processing step to their input. Intuitively, our specific design choice is intended to implement Theorem 3.11, allowing us to turn a CNN into a kind of FFN. While other freezing techniques can reduce total trainable parameters, ours keeps the trainable part in a FFN node design, which is already known to be successful. This should not only allow us to benefit from our demonstrated property, but also benefit from other strengths of FFNs. As stated earlier, one advantage of projecting nodes is that it allows a node to be trained with fewer parameters, reducing the amount of data needed. We explore this idea in Section 4.

4 Experiments↩︎

We now demonstrate the utility of inheritance via projection for transfer learning: training nodes with only one parameter per input channel. We focus on transfer learning large pre-trained foundation CNN models onto downstream tasks. Normally, CNN nodes can have many parameters per input channel. By projecting these models, we can train an entire model using many fewer parameters, which should be advantageous when there is not enough data to fully train these large networks.

We conduct two sets of experiments. The first compares model projection with full fine-tuning and logistic regression , with an emphasis on analyzing training behavior. The second evaluates model projection on standard benchmarks and compares its performance to state-of-the-art PEFT methods.

4.1 First Experiment: Understanding Model Projection Behavior↩︎

We conduct transfer learning experiments on seven benchmark datasets: Food-101 [21], CIFAR-10 [22], CIFAR-100 [22], Stanford Dogs [23], Oxford-IIIT Pets [24], Caltech-101 [25], and Oxford 102 Flowers [26]. These datasets range in size from about 2,000 to about 75,000 training examples and span both coarse-grained and fine-grained image classification tasks. See Appendix Table 2. Our focus is on showing a proof of concept of our method rather than pushing state-of-the-art results, so we intentionally use simple training setups without hyperparameter optimization.

General setup. We perform experiments using three different convolutional bases, with weights pre-trained on ImageNet[27]: VGG16[28], ResNet50[29], and DenseNet121[30]. After each convolutional base, we add a global average pooling layer, a dropout layer set to 0.5, and finally a single dense layer for predictions. We use simple data augmentations: horizontal flip with probability 0.5, and random rotation and zoom both set to 0.1. Input images were set to 224 x 224 pixels. We do not shuffle data or address class imbalances. Nor do we address noisy labels or perform any other type of regularization. Any validation sets are absorbed into training sets. We evaluate two training setups:

4.1.0.1 Single-stage training.

For each model, we run three variants: Logistic regression (LR), where only the final classifier is trained; Full fine-tuning (FT), where all parameters are trained; and model projection, where convolutional kernels are frozen and all other parameters, including projection parameters, are trained. All runs use Adam [31] with default hyperparameters for 20 epochs.

4.1.0.2 Two-stage training.

This setup uses the same models but trains in two stages. Stage 1 runs for 7 epochs with Adam using default hyperparameters. Stage 2 runs for 13 epochs with SGD, learning rate \(10^{-4}\), momentum 0.9, following the Keras Applications fine-tuning example [32]. Under this setup, we evaluate three combinations: Logistic regression in Stage 1, then full fine-tuning in Stage 2 (LR+FT); model projection in Stage 1, then full fine-tuning in Stage 2 (Model projection + FT); and model projection in both stages (2 step model Projection).

We perform the second setup because we observed full fine-tuning occasionally failed to learn in single-stage training, particularly for VGG16. Two-stage training provides a stronger full fine-tuning baseline and allows us to study projection as an alternative warm start to logistic regression training and/or an initialization before full fine-tuning.

Results. Results are summarized in Figure 2. Overall, model projection consistently outperforms logistic regression and full fine-tuning across both large and small datasets, and remains robust under older architectures and a simple training setup, while still performing well on newer models. Logistic regression typically achieves higher initial accuracy but shows limited improvement over epochs, indicating underfitting. Its performance is competitive on smaller datasets but degrades on larger ones, and it remains relatively robust on older models. Full fine-tuning starts from lower accuracy and improves over training, sometimes surpassing logistic regression but often struggling on smaller datasets or failing to learn for older architectures such as VGG16. In contrast, model projection combines favorable properties of both approaches: it starts from relatively high accuracy and continues to improve throughout training, achieving the strongest overall performance across experiments.

These results align with what we expect from projection inheritance: strong results from learning an entire neural network, allowing frequent outperformance of logistic regression and greater robustness to dataset size compared to full fine tuning due to inheriting fewer trainable parameters. Model projection retains the expressiveness of end-to-end learning while improving robustness to dataset size and optimization instability. While we expect full fine-tuning to outperform model projection given sufficiently large datasets, and logistic regression training to dominate in extremely low-data regimes [7], our results suggest that model projection performs best across a broad intermediate regime that covers most practical transfer learning scenarios.

In the two-stage setting, the first stage behaves similarly to the corresponding single-stage experiments, as expected. Notably, performance in the second stage is consistently higher when model projection is used in the first stage, regardless of whether the second stage applies model projection or full fine-tuning. This suggests that model projection provides a more effective initialization for subsequent fine-tuning. Overall, model projection yields stronger performance and emerges as the most effective strategy in the majority of cases. Further details and full results can be found in Appendix 7.

4.2 Second Experiment: Model projection Against PEFT Baselines↩︎

To evaluate model projection against current PEFT approaches, we compare to standard transfer baselines and recent PEFT methods on CIFAR-10, CIFAR-100, and Oxford Flowers on pretrained ResNet50 architecture. Our baselines include logistic regression (training only the final classifier) and full fine-tuning, as well as partial-tuning baselines that update only BatchNorm parameters (BN) [8] along with the final layer, or only bias terms (inspired by BitFit) [9] along with BN and the final layer. This gives a set of methods with gradually increasing trainable parameters on the same model. We additionally compare to low-rank adaptation methods LoRA-r8 and LoRA-r32 [14], the Parallel Residual Adapter (PaRA) family [11], and the convolutional variant LoRA-C [15]. For LoRA-C we report results from [15], and for LoRA-r8/LoRA-r32 and PaRA we use the ResNet50 results reported by [33]. We include all two-stage variants considered previously. We report the average accuracy across 5 seeds, along with the standard deviation.

Results. Table 1 summarizes the results. Both single-stage and two-stage model projection are highly competitive across all datasets when compared to PEFT baselines. In particular, the two-stage setup that uses model projection in the first stage, either with full fine tuning or model projection in the second stage, achieves the best performance on CIFAR-10 and CIFAR-100, while all projected models are within two standard deviations of the best performance on Flowers. These results support the view that model projection provides an effective initialization, especially for subsequent full fine-tuning, by guiding optimization toward more favorable regions of the loss landscape.

Conclusions.↩︎

In this paper we formalized a general version of the operations of nodes from traditional neural networks and convolutional neural networks. We used this formalization to prove that FFNs can inherit from CNNs, but that CNNs cannot inherit from FFNs. We proposed model projection, a method to project a CNN node onto an FFN node, and proved that this method allows a CNN node to inherit any results or techniques from FFN nodes that do not require homogeneous inputs. We then demonstrated a proof of concept of our proposed projection method in a transfer learning example. Our method had strong results in a simple setting, reinforcing our theoretical findings. Our example is a good application of projection that we expect will be useful for simple home made models, and may also be useful in more heavily engineered settings as well. There are a number of potential directions for future research. Extending analysis of inheritance between more model types such as graph neural networks and transformer models is particularly interesting. More results from FFNs can be inherited to CNNs, and exploration of some of these methods such as lasso artificial neural networks [34] would be interesting to experiment with. Additionally, further exploration of our example experiment could be done in less simple settings.

5 Supplementary proofs↩︎

5.1 Projection↩︎

6 Supplementary remarks↩︎

7 Supplementary results↩︎

We now discuss the results from the first set of single stage experiments, seen in Figure 3, with summarizing information presented in Tables 3, 4, and 5. Figure 3 shows that model projection consistently had strong results, while training was relatively smooth and fast. Table 3 shows the ranking of each method across each experiment. Model projection significantly outperformed both the logistic regression and full fine tuning in this simple setup, producing the best model about \(81\%\) of the time, and the second best about \(19\%\) of the time.

Table 4 shows the rankings achieved by model projection organized by model. Model projection had consistently strong results on each model. Table 5 shows the rankings for model projection grouped by dataset. Model projection had the best scores on all five datasets that were not exclusively pet related, but did not do as well on the two datasets that were exclusively pet related. Interestingly, model projection did well on datasets both larger and smaller than the pet ones, suggesting the issue is not solely dataset size.

References↩︎