1 Introduction↩︎

Effects are central to the sound of the electric guitar: they reshape timbre, dynamics, sustain, and apparent space, and have become part of both the instrument’s musical vocabulary and the broader history of audio effects processing [1]–[4]. From a signal-processing perspective, effects can be understood as transformations of a clean instrumental source [5]. Recent neural approaches to audio effects have shown that input–output behavior can be learned directly from paired recordings, including black-box modeling of audio effects [6], real-time guitar-amplifier emulation [7], and controllable audio-effect transformation [8]. Much of this work, however, is organized around a single device, a single effect class, or a parametric processor. Clean-to-effect guitar performance transformation across several effect families poses a problem: preservation of pitch, rhythm, and playing content while applying a requested transformation whose acoustic footprint may range from broadband distortion to modulation, delay, or reverb.

This paper presents Clean2FX as both a study of label-conditioned clean-to-effect guitar audio transformation and a web demo. We construct paired clean/effected examples from EGFxSet, a dataset of electric-guitar tones processed through real hardware effects [9], [10], by assembling matched chords, melodies, and mixed timelines. This gives a controlled setting in which the musical content is held fixed and the target effect label defines the transformation. We compare four neural approaches: two variational autoencoders and two conditional encoder–decoder U-Nets.

The demo website complements this controlled evaluation by applying two trained systems to real-world guitar performances [11] outside the EGFxSet-derived material. The demo therefore illustrates the practical behavior of the models beyond synthetic data.¹ The code implementation is released too.²

2 Methods↩︎

We build our training data using EGFxSet [9], which contains isolated electric guitar tones recorded through ten real hardware effects alongside a clean reference signal. We want to train models on musical sequences, so we create synthetic performances programmatically, using these recorded tone sounds. Our resulting custom dataset assembles chords, melodies, and mixed event timelines, applying the same event sequence to both the clean and effected versions of each tone. This pairing holds all musical content constant across both signals, leaving the effect as the sole axis of variation in each training example. Each pair is normalized by the larger of its two signal maxima to preserve the energy ratio between the clean and effected signals. The probability of sampling near-silent compositions is set to 0.05.

Audio is represented as magnitude STFT spectrograms with \(n_{\text{fft}}=512\) and hop \(h=128\) at 16 kHz; each 4.9 s clip yields a \(257\times613\) spectrogram. Two input representations are compared: the linear shared-max normalized magnitude \(x\), and a log-compressed variant \(y=\ln(1+200x)\), inverted as \(x=(e^{y}-1)/200\) before any metric is computed. Since only magnitude is retained, the phase of the output spectrogram is unavailable; at inference, it is estimated using Griffin-Lim [12].

5pt

We assess four architectures, each conditioned on a learned effect embedding. Figure 1 summarizes and illustrates them. The VAE encodes the input with fully connected layers into a 64-dimensional latent and decodes symmetrically, trained with the standard VAE objective (MSE reconstruction plus a KL term). The ConvVAE replaces the dense layers with strided convolutions using BatchNorm and ReLU, a 512-dimensional deterministic latent, and an L1 loss on \(\log(1+x)\) magnitudes. We compare these VAEs against a five-level residual U-Net [13], [14]. The encoder comprises four DoubleConv blocks (channels 32, 64, 128, 256); each two \(3\times3\) convolutions with Group Normalization (eight groups [15]) and LeakyReLU (slope 0.01), separated by \(2\times2\) max-pooling; the bottleneck adds a fifth block at 512 channels; the decoder mirrors the encoder with nearest-neighbor upsampling and skip connections. The network predicts a residual summed with the input [16], passed through softplus (\(\beta=5\) for linear inputs, \(\beta=1\) for log) to produce non-negative magnitudes. Two U-Net variants are evaluated, differing only in whether the loss operates on linear (\(\mathcal{L}_{\text{lin}}\)) or log-magnitude (\(\mathcal{L}_{\text{log}}\)) spectrograms. Throughout, we refer to a copy-input baseline, defined as the trivial system that outputs the clean input unchanged; this baseline anchors all relative-improvement figures reported in evaluation.

In the U-Net, each of the ten effect labels receives a learned 64-dimensional embedding, injected at three points. At the bottleneck the embedding is projected to 64 channels, broadcast, concatenated, and fused by a DoubleConv. At each of the four decoder levels, feature-wise linear modulation (FiLM) [17] maps the embedding to per-channel scale and shift parameters applied as \(x\cdot(1+\gamma)+\beta\), where the \((1+\gamma)\) form is the identity at initialization; FiLM transfers cleanly to U-Nets [18]. Each encoder skip connection receives FiLM of the same form through its own per-level heads, so no path through the network bypasses the condition. Conditioning dropout zeroes the entire embedding with probability 0.15 during training, inspired by classifier-free guidance [19], so the network cannot learn to ignore its condition. The two VAE baselines condition on the target-effect index through a separate four-dimensional learned embedding, concatenated to the latent code before decoding (64-dimensional in the fully connected VAE, 512-dimensional in the ConvVAE). Conditioning therefore enters at a single point, with no FiLM, skip-connection modulation, or conditioning dropout.

The U-Net variants optimize a composite spectral loss whose base term is spectral convergence (SC), \(\|\hat{Y}-Y\|_F/(\|Y\|_F+1.0)\), where the large epsilon of 1.0 keeps the ratio bounded on near-silent targets. For the log-magnitude variant, the loss is \(\text{SC} + 10\,\text{L1}_{\log}\), with the L1 term computed directly in the network’s native log-magnitude domain. For the linear variant, the loss is \(\text{SC} + 10\,\text{L1}_{\text{lin}} + 5\,\text{L1}_{\log}\): an L1 term in the native linear-magnitude domain, plus an auxiliary L1 term computed on a log-compressed version of the same prediction, included to penalize errors in low-energy regions that the linear term underweights. A per-effect weight \(w=\min((\Delta_{\max}/\Delta_e)^{0.5},\,5.0)\) is applied to each sample’s loss based on its target effect, where \(\Delta_e\) is the mean clean-to-effect L1 distance of that effect and \(\Delta_{\max}\) the largest such value across effects; this upweights effects with naturally small clean-to-effect spectral differences, which would otherwise contribute a vanishingly small gradient signal. These objectives apply to the U-Net variants; the VAE objectives are given above.

We train the U-Nets with AdamW [20] (learning rate \(3\times10^{-4}\), weight decay \(1\times10^{-4}\)), ReduceLROnPlateau (factor 0.5, patience 8, minimum \(1\times10^{-6}\)), gradient clipping at norm 1.0, batch size 32, a RandomSampler with replacement over 8000-sample epochs, and a 90/10 train/validation split. The VAE baselines are trained on the same data split, using Adam at \(1\times10^{-3}\) with mixed precision; the full pipeline is built on librosa [21] and PyTorch [22].

We evaluate the four trained models on 200 held-out paired samples (20 per effect), drawn from the same split. All metrics are computed on linear shared-max magnitude spectrograms. Per-bin MSE is complemented by Fréchet Audio Distance (FAD) [23]. Baseline-relative improvement is \((b-m)/b\) for model value \(m\) against the copy-input baseline \(b\), and the overall value reported is the mean of the ten per-effect means.

3 Results↩︎

Table 1 shows that the U-Net (Log) is best on both metrics (\(+\)​70.6% MSE, \(+\)​31.8% FAD), the U-Net (Linear) is second (\(+\)​69.4% and \(+\)​23.0%), and both variational autoencoder baselines fail to beat the copy-input baseline on average, with no positive mean per-effect improvement on either metric.

Table 2 groups the effects into distortion, modulation, delay, and reverb. The U-Net (Log) is the strongest model in most cells, performing best for six effects by MSE and nine by FAD; the U-Net (Linear) takes the four remaining MSE cells and the single remaining FAD cell. The two distortions sit at the top of the baseline-relative ranking and the time-based effects at the bottom, an ordering that tracks the size of the copy-input baseline: a distortion leaves little of the clean signal intact and so leaves the most error to remove, whereas a delay leaves the input largely unchanged. The two metrics diverge most on these time-based effects, which post strong MSE improvements but markedly weaker FAD gains; under the U-Net (Log), for example, Spring Reverb improves \(+\)​78.6% on MSE but only \(+\)​0.9% on FAD.

4 Discussion↩︎

The strongest pattern in Table 2 is architectural since the U-Net’s skip connections pass each encoder feature map directly to the corresponding decoder level. This likely preserves harmonic structure at every resolution, while the VAE family compresses the input through a latent bottleneck and loses fine detail. This is a previously documented source of blurred VAE reconstructions [24]. RAT is the exception, the only case where ConvVAE clears the baseline on both metrics. The copy-input baseline is the weakest, so capturing the broad spectral envelope is enough to beat it.

We use effect macro-averaging because MSE on a magnitude spectrogram is scale-dependent: RAT’s copy-input baseline is an order of magnitude above every other effect, so it looks hardest by absolute MSE yet is the easiest in relative terms, since the unchanged input leaves the most error to remove. Effect-averaging keeps it from dominating the aggregate: the ConvVAE, for instance, has a lower pooled MSE than the baseline (\(4.09\) vs \(5.16\times10^{-4}\)) but clears the baseline on no effect except RAT, so its effect-averaged improvement is not positive. Log compression helps because it lifts low-amplitude detail such as reverb tails, modulation sidebands, and delay echoes into a range the encoder features and the loss can represent. The resulting MSE gap between the two U-Nets is small (1.2 percentage points) but the FAD gap is larger (8.8 percentage points) and consistent (nine out of all effects), in line with input representation mattering for audio networks [25].

As an informal diagnostic, we examine whether the U-Net responds to the conditioning label rather than collapsing toward a single transformation. For each clean input, we generate outputs for all effect labels and compute the mean pairwise L1 distance between predictions, normalized by the mean L1 distance between each prediction and the unprocessed input (Table 3). Ratios substantially above zero indicate label-dependent variation. All effects produce ratios above 1.0 (mean 1.416), providing qualitative evidence that the conditioning signal influences the generated transformations rather than being ignored by the model. This diagnostic should be interpreted only as evidence against conditioning collapse, not as validation of effect-specific generation.

The demo website probes out-of-distribution behavior, pairing four EGFxSet-derived passages with Guitar-TECHS [11] real performance recordings, each processed by the log-magnitude U-Net and the ConvVAE under all effects. On Guitar-TECHS input the U-Net produces audible but weaker transformations, and the ConvVAE shows the same smoothed-upper-band failure seen in-distribution. The demo therefore contrasts with the EGFxSet-derived distribution, within which the headline results hold.

References↩︎