1 Competition Overview↩︎

The Efficient Qwen Competition [1] focused on minimizing the inference latency of the Qwen3.5-4B [2] large language model (LLM) on an AWS g5.xlarge instance equipped with a single NVIDIA A10G GPU (24 GB VRAM). Submissions were ranked by average speedup over the unoptimized baseline model across short, medium, and long input settings. A broad range of optimization techniques was permitted, provided that the resulting model satisfied the required quality thresholds on standard benchmarks.

2 Our Approach↩︎

This technical report describes our submission to the competition (see Table ¿tbl:tab:our95subm?), in which we ranked 3rd among more than 40 teams. Our approach combines quantization of Qwen3.5-4B with speculative decoding to accelerate inference (Figure 1):

max width=0.78

1. For the target model, we started from an existing post-training-quantized (PTQ) model checkpoint and applied quantization-aware distillation (QAD) [3] to recover accuracy.

2. For speculative decoding, we trained a lightweight block-diffusion drafter [4] to operate with the quantized target model. We further optimized the drafter with PTQ and sliding-window attention (SWA) [5] to boost inference speed.

Figure 2 illustrates the training and optimization workflow used to build our final system. It highlights the sequence from target-model QAD to two-stage drafter training, followed by drafter PTQ and SWA for speculative decoding. Table ¿tbl:tab:main95overall95latency? summarizes the cumulative contribution of each component. We describe each technique in detail below.

2.1 Target Model Optimization with QAD↩︎

PTQ converts a pretrained model into a low-bit version without additional training, but it can suffer from accuracy degradation. To mitigate this, QAD [3], [6] fine-tunes a quantized student model under the guidance of a high-precision teacher to recover accuracy.

We apply QAD to an AWQ [7] INT4 checkpoint [8] of Qwen3.5-4B. Specifically, we initialize the student from the dequantized model and optimize its weights while freezing the symmetric per-group scales. Using the original full-precision model as the teacher, we apply per-token distillation [9] by minimizing the forward KL divergence: \(\mathbb{E}_{x}\left[\sum_t \mathrm{KL}\left(p_{\text{tea}}(\cdot\mid x_{<t}) \,\|\,p_{\text{stu}}(\cdot\mid x_{<t}; Q_s(W))\right)\right]\), where \(p_{\text{tea}}\) and \(p_{\text{stu}}\) denote the teacher and student output probabilities on a training sequence \(x\), respectively. Here, \(W_q\) denotes the INT4 codes with the frozen scales \(s\), and \(W\) is initially set to the dequantized weights \(s \odot W_q\). We update \(W\) through the fixed-grid fake quantizer \(Q_s(\cdot)\) using a straight-through estimator [10].

At packing time, we round the updated BF16 weights back onto the same quantization grid, yielding new INT4 codes under the original AWQ scales. The resulting model is served in the W4A16 compressed-tensors format.

2.2 Speculative Decoding with DFlash↩︎

Speculative decoding [11] speeds up generation by using a fast draft model to propose several tokens, which are then verified in parallel by the target model. DFlash [4] further improves drafting efficiency with a diffusion-based block drafter that leverages the target model’s hidden states to predict multiple future tokens in a single forward pass.

For faster LLM inference, we train the DFlash drafter on top of the frozen QAD-applied target model. Given an input sequence, hidden representations are extracted from multiple layers of the target model and projected into compact context features. The target context is then injected into the drafter’s key-value cache for conditioning. The drafter is trained to predict the masked tokens in each draft block, where an observed anchor token is followed by masked future positions. This encourages the drafter to propose future tokens aligned with the quantized target model’s predictions.

We use a two-stage training pipeline for the drafter. First, we train the drafter from a random initialization using the original BF16 target model and data regenerated by this model. We then fine-tune the Stage 1 drafter with the QAD-applied target model using data regenerated by the QAD target. This two-stage design aims to learn a robust drafting initialization from the BF16 model before adapting it to the quantized target distribution.

max width=

2.3 Drafter Optimization with PTQ and SWA↩︎

We optimize the trained drafter with PTQ to improve end-to-end latency. Since the drafter is invoked at every speculative decoding step, reducing its computation directly lowers the per-step cost. We show that PTQ does not noticeably degrade drafter quality and largely preserves the acceptance length of draft tokens, allowing the reduced per-step cost to translate into lower overall latency. Since several PTQ methods yield similar performance in our experiments, we adopt GPTQ [12] for drafter quantization.

We also equip the PTQ drafter with SWA [5]. Unlike full-context attention, SWA restricts the drafter’s attention to a local window of recent tokens, reducing attention computation. This also allows the drafter to avoid distant context that may act as noise in long-context decoding, helping it focus on more relevant context for predicting near-future tokens. In our experiments, SWA often increases the mean acceptance length and reduces decoding latency, especially for long-context inputs.

3 Experimental Setup↩︎

3.1 Target-Model QAD↩︎

We generate the training data by prompting the original Qwen3.5-4B with 220K samples randomly drawn from Nemotron-Post-Training-Dataset-v2 [13]. The resulting teacher-regenerated conversations are tokenized and packed into fixed-length, EOS-delimited sequences with a sequence length of 16,384, padding only the last chunk. This yields 27K training chunks, corresponding to 440M tokens.

We initialize the student from the dequantized weights of the AWQ INT4 checkpoint and distill it from the frozen BF16 teacher using the per-token forward KL objective. During distillation, we update only the dequantized weights while keeping the per-group scales frozen. We optimize the student with 8-bit AdamW at a learning rate of 2e-6 using a cosine schedule with a warmup ratio of 0.03, gradient clipping of 1.0, and an effective batch size of 16. Training runs for 8,000 steps, corresponding to about 4.5 epochs. We select the 5,000-step checkpoint as it yields the highest average performance among periodically evaluated checkpoints on a validation subset comprising SciQ [14], BoolQ [15], PIQA [16], and HellaSwag [17].

3.2 Drafter Training↩︎

We train a 5-layer DFlash drafter with hidden size 2,560 and block size 16, resulting in approximately 537M BF16 parameters. We use a two-stage setup: Stage 1 trains the drafter from random initialization using the original BF16 target model and its 220K regenerated conversations, which are also used for target-model QAD. Stage 2 finetunes the Stage-1 drafter using the QAD-applied INT4 target model and 400K conversations regenerated from Nemotron-Post-Training-Dataset-v2. For Stage 2, the target weights are dequantized from the INT4 weight grid, keeping it numerically aligned with the deployed INT4 target while allowing floating-point computation.

Training runs for 2 epochs with an effective batch size of 32. We optimize the drafter with AdamW using a learning rate of 1e-3, a cosine-annealing schedule with a warmup ratio of 0.04, and gradient clipping of 1.0. We set the maximum sequence length to 3,072, the number of anchors to 512, and the loss-decay factor \(\gamma\) to 7.0.

3.3 Drafter Optimization↩︎

Drafter PTQ. For drafter PTQ calibration, we use 256 samples from the same 400K training set generated by the target INT4 model. Unlike token-driven GPTQ calibration, the DFlash drafter is conditioned on target hidden states, noise embeddings, and position IDs rather than token IDs. Therefore, we do not truncate samples to a fixed sequence length. Instead, each sample is kept as a full teacher-forced conversation at its natural length. We accumulate the GPTQ Hessian from 16 strided anchor positions per conversation over the assistant region. At each anchor, the drafter receives the causal target-hidden prefix and one 16-token draft block, yielding 256 \(\times\) 16 = 4,096 calibration blocks. The drafter weights are then quantized to group-wise symmetric INT4 with a group size of 128.

Drafter SWA. To reduce speculative-decoding latency on long contexts, we apply SWA only to the DFlash drafter, using a fixed-size window of either 1,024 or 2,048 tokens. Since the drafter performs bidirectional attention over mask-token queries, we use a symmetric window around each query rather than a causal left-only window. The window is applied only at the attention-kernel level while keeping the KV cache full, so no cache eviction or rotation is introduced. For shorter sequences within the window size, the drafter falls back to full attention.

3.4 Evaluation↩︎

Accuracy. Following the competition rules, we evaluate model quality on MMLU-Pro [18] (\(\geq 0.621\)), IFEval [19] (\(\geq 0.814\)), and GPQA-Diamond [20] (\(\geq 0.630\)), where the values in parentheses denote the required quality thresholds.

Speed. We follow the evaluation format with three length regimes: Short (64/128 input/output tokens), Medium (2,048/256), and Long (8,192/256). Since the exact official prompts are not publicly available, we measure the mean acceptance length using samples from GSM8K [21], HumanEval [22], and LongBench v2 [23], with 64 prompts per benchmark and generation capped at 256 new tokens. For end-to-end latency, we use 5 warm-up samples and 50 measurement samples from LongBench v2. Due to the unavailability of an NVIDIA A10G GPU, latency is evaluated on an NVIDIA RTX 5000 Ada Generation GPU.

max width=

max width=

max width=

max width=

4 Results↩︎

Target-model QAD. Table ¿tbl:tab:qad95accuracy? shows that direct INT4 PTQ degrades accuracy compared with the BF16 baseline, most notably dropping IFEval below the required quality gate. Applying QAD improves the INT4 checkpoint across all three benchmarks, recovering IFEval from 0.8046 to 0.8285 and increasing MMLU-Pro and GPQA-Diamond to 0.6610 and 0.6626, respectively. As a result, the QAD model satisfies all competition quality gates while retaining the original AWQ quantization grid. Since the subsequent speculative decoding step is lossless, these accuracy results are preserved in the final optimized model.

Drafter training. Table ¿tbl:tab:drafter95training? compares direct drafter training on the INT4 target with our two-stage training pipeline. Since the drafter is conditioned on target hidden states and predicts masked future tokens in each draft block, its acceptance length depends on how well its proposals align with the target model. The Stage-1 drafter trained with the BF16 target achieves an average acceptance length of 4.92, showing that the BF16 target provides a useful initialization for block drafting. After adapting this drafter to the QAD-applied INT4 target in Stage 2, the average acceptance length increases to 5.03. This slightly outperforms direct training from random initialization on the INT4 target, which obtains 4.97 on average, with gains on HumanEval and LongBench v2. These results suggest that learning a robust drafting initialization from the BF16 target before adapting to the quantized target distribution is beneficial.

Drafter PTQ. Table ¿tbl:tab:drafter95quant? compares INT4 PTQ methods for the drafter, including the round-to-nearest (RTN) baseline, AWQ, and GPTQ. INT4 quantization causes only a minor change in the drafter’s mean acceptance length relative to the BF16 baseline, and these PTQ methods perform comparably. We therefore use GPTQ for drafter quantization. Because the drafter runs at every speculative decoding step, reducing its cost with PTQ improves the overall decoding speed, as reflected in Table ¿tbl:tab:main95overall95latency?.

Drafter SWA. Table ¿tbl:tab:drafter95swa? shows that SWA is most effective in the long-input setting. With a 2048-token window, long-input latency decreases from 4285.7 ms to 3218.8 ms in the BF16 case, and from 2290.0 ms to 1966.0 ms in the W4 case. For the leaderboard entry, we use a 1024-token SWA window, which corresponds to the latency reported in Table ¿tbl:tab:main95overall95latency?. A moderate SWA window reduces unnecessary drafter attention cost while preserving enough context for stable draft-token acceptance. Smaller windows, such as 512 or 256 tokens, are less effective because the reduced context can lower acceptance length and offset the savings from cheaper attention. We therefore use a 1024- or 2048-token SWA window for the drafter.

5 Conclusion↩︎

This report presents an approach for accelerating Qwen3.5-4B inference for efficient deployment while maintaining model accuracy. Our system combines a low-bit target model improved by quantization-aware distillation with speculative decoding. We train a block-diffusion drafter specialized for the quantized target model and further optimize the drafter with quantization and sliding-window attention to reduce decoding overhead. Together, these optimizations achieve a 6.978× average speedup over the baseline on an NVIDIA A10G GPU.

References↩︎