1 Introduction↩︎

The growing demand for local intelligence is driving the deployment of large language models (LLMs) to edge devices, where low latency and data privacy are essential. Large reasoning models (LRMs) such as DeepSeek-R1 [1] and Qwen3 [2] push this frontier further by generating extended chain-of-thought (CoT) sequences to achieve strong performance on complex tasks [3]. Distilled variants with 1.5B–8B parameters make edge deployment technically feasible. Yet CoT sequences average 3,000–14,000 tokens per query, creating a sharp tension between the reasoning depth that accuracy demands and the limited compute budget that edge hardware can afford.

An overthinking problem [4] further aggravates this cost. LRMs frequently produce verbose, redundant reasoning steps that consume resources without improving, and sometimes even harming, answer quality. Recent confidence-based early exit methods address this from a single-request perspective. DEER [5] and Dynasor [6] monitor model confidence during CoT generation and terminate reasoning once the model is sufficiently certain, achieving reduction in reasoning length while preserving accuracy. These training-free approaches suit edge deployment well because they require no model retraining or weight modification.

Existing early exit methods do not account for the serving context, applying a fixed confidence threshold to each request. At the edge, a single GPU serves requests sequentially or with minimal batching, so one request’s continued reasoning directly delays those behind it. A fixed threshold may exit too early under light load or allow excessive reasoning under heavy load, hurting latency targets. Because the threshold is applied per request, the early exit decision has system-wide consequences. Therefore, a load-aware mechanism that jointly manages reasoning depth and service performance is required. To this end, two challenges must be addressed.

The first challenge is how to adaptively adjust the early exit threshold based on real-time system load while preserving reasoning quality. The confidence threshold governs the trade-off between reasoning depth and latency [5], [6]: higher values usually improve accuracy but increase delay, while lower values reduce latency at the risk of premature termination [3], [4]. Unlike cloud systems with elastic scaling [7], [8], edge servers have fixed capacity, so the threshold must adapt online. Moreover, confidence polarization means that only 2–6% of scores fall in the intermediate range [5], leaving a narrow but usable tuning space for smooth load-aware adjustment.

The second challenge is how to pre-allocate per-request reasoning budgets considering both request difficulty and system capacity. Requests differ in the reasoning depth that they need: simple problems gain little from extra computation, whereas hard ones benefit more from longer reasoning [3]. Existing methods such as Reinforcement Learning (RL)-based length control, budget forcing, and concise-reasoning prompting largely apply uniform policies [9]–[11], without accounting for the request difficulty and system load. In edge serving, such uniform budgets waste capacity and can hurt both throughput and accuracy. A practical allocator should therefore estimate difficulty with minimal overhead and adapt budgets to current load, complementing the adaptive threshold with a hard cap on reasoning depth [12], [13].

To address these challenges, we propose a Load-Aware Serving method with Early-exit for Reasoning (LASER), which elevates confidence-based early exit from a single-request optimization to a system-level scheduling mechanism for edge LLM serving. LASER couples two complementary mechanisms to jointly manage reasoning depth under varying load: a load-aware adaptive exit threshold that offers fine-grained soft control over when to stop reasoning, and a difficulty-aware reasoning budget allocator that imposes a coarse-grained hard cap on maximum reasoning depth. We evaluate LASER on two edge-deployed reasoning models (i.e., DeepSeek-R1-Distill-Qwen-7B and Qwen3-4B) across four benchmarks under five load levels using discrete-event simulation on an NVIDIA RTX 4090. The main contributions are summarized as follows.

1. Load-aware adaptive exit threshold. We exploit the confidence polarization phenomenon in reasoning models to establish a safe tuning space. We design a \(\tanh\)-based threshold function driven by exponential moving average (EMA) smoothed load signals that decreases the threshold under high load and raises it under low load.

2. Difficulty- and load-aware reasoning budget allocator. We propose a lightweight budget pre-allocation mechanism that uses prompt token count as a difficulty proxy and scales budgets proportionally with system load, providing a hard reasoning cap that complements the threshold’s soft control.

3. Comprehensive evaluation. Experiments on two reasoning models show that LASER reduces average latency by 17–38% and improves service-level objective (SLO) satisfaction by 3–6 percentage points over fixed-threshold baselines, with only 1 percentage point average accuracy loss.

2 Related Work↩︎

Efficient Reasoning for Large Language Models. Existing methods reduce redundant reasoning in LRMs through three main approaches. Post-training methods adapt reasoning length with variable-length CoT data or reinforcement learning [9], [10], [14], but require retraining. Prompt-based methods such as Chain-of-Draft and budget forcing [11] are lightweight yet may hurt quality on hard tasks. Training-free methods perform early exit during inference, for example by monitoring intermediate-answer consistency or confidence [5], [6], [15]. However, these methods optimize each request independently and do not account for system load or queueing effects.

Edge LLM Inference and Serving. Prior work improves edge LLM deployment through model compression, partitioning, speculative inference, and scheduling optimizations [1], [16]–[20]. LLM serving systems further improve efficiency with optimized KV-cache management, continuous batching, disaggregated execution, and runtime rescheduling [7], [8], [12], [13], [21]. These approaches improve model- or system-level efficiency, but generally treat generation length as fixed. In contrast, LASER treats reasoning depth itself as a controllable serving variable and adapts it to system load.

3 System Model and Problem Formulation↩︎

We consider an edge serving system where a single GPU-equipped server hosts a reasoning LLM to serve requests from local users, as illustrated in Fig. 1.

3.1 System Model↩︎

Reasoning Model Generation Pattern. A reasoning LLM generates output in two phases [5]. It first performs slow thinking, enclosed in <think>\(\ldots\)</think> delimiters, then produces a conclusion. The slow thinking phase consists of \(K_i\) reasoning chunks separated by action transition points (ATPs), typically marked by linguistic cues such as Wait or Alternatively. \[\label{eq:generation95pattern} [\text{Prompt}] + \texttt{<think>} + T_1 + \text{ATP}_1 + T_2 + \text{ATP}_2 + \cdots + T_{K_i} + \texttt{</think>} + [\text{Conclusion}]\tag{1}\] where \(T_k\) denotes the \(k\)-th reasoning chunk. Each ATP is a candidate early-exit point. At ATP\(_k\), the model is prompted to generate a trial answer, and a confidence score \(C_i^{(k)}\) is computed from the token probabilities. If \(C_i^{(k)}\) exceeds the threshold \(\lambda\), reasoning terminates early and the model proceeds to generate the conclusion.

Cost Model. For each request \(r_i\), the total response latency \(l_i\) consists of two parts: \(l_i = w_i + t_i^{\text{inf}}\). \(w_i\) is the queueing delay and \(t_i^{\text{inf}}\) is the inference time. The inference time is determined by the reasoning depth \(s_i\), i.e., the number of reasoning steps executed before exit. \[\label{eq:inference95time} t_i^{\text{inf}} = t_i^{\text{prefill}} + \sum_{k=1}^{s_i} \tau_k^{(i)} + t_i^{\text{conclu}}\tag{2}\] where \(t_i^{\text{prefill}}\) is the prompt prefill time, \(\tau_k^{(i)}\) is the time to generate the \(k\)-th reasoning chunk and evaluate trial answer confidence, and \(t_i^{\text{conclusion}}\) is the conclusion generation time.

In existing early exit methods, the reasoning depth \(s_i\) is determined solely by a fixed confidence threshold \(\lambda\). Reasoning terminates at the first ATP\(_k\) where \(C_i^{(k)} > \lambda\), or when the maximum step limit is reached. A lower \(\lambda\) leads to earlier exits, reducing \(t_i^{\text{inf}}\) but potentially decreasing accuracy \(a_i\).

The accuracy \(a_i \in \{0,1\}\) of request \(r_i\) depends on the reasoning depth \(s_i\), jointly controlled by the exit threshold \(\lambda_i\) and the reasoning budget \(B_i\). A deeper reasoning chain increases the probability of reaching a correct answer, but with diminishing returns due to the overthinking phenomenon [4]. Because \(a_i\) is a stochastic function of the model’s internal state, we treat it as an empirical black-box mapping \(a_i(\lambda_i, B_i)\) in the optimization below.

3.2 Problem Formulation↩︎

Given a request stream \(\mathcal{R}\), LASER jointly determines the exit threshold \(\lambda_i\) and reasoning budget \(B_i\) for each request to balance reasoning quality and service latency. We formulate this as the following optimization problem. \[\label{eq:objective} \mathbf{P:}\quad \max_{\{\lambda_i\}, \{B_i\}} \; \frac{1}{N} \sum_{i=1}^{N} \left[ \omega \cdot a_i(\lambda_i, B_i) - (1-\omega) \cdot \frac{l_i(\lambda_i, B_i)}{d_i} \right]\tag{3}\] subject to: \[\label{eq:constraints} \lambda_{\min} \leq \lambda_i \leq \lambda_{\max}, \quad s_i \leq B_i, \quad B_i \geq B_{\min}, \quad \forall i\tag{4}\] where \(\omega \in [0,1]\) balances accuracy and latency. The constraints require the exit threshold to remain within the empirically validated robust range \([\lambda_{\min}, \lambda_{\max}]\) [5], the executed reasoning depth to stay within the assigned budget, and each request to receive at least the minimum reasoning budget \(B_{\min}\).

4 Algorithm Design↩︎

4.1 Overview↩︎

LASER adds two mechanisms to the confidence-based early exit pipeline (Algorithm 3). Before inference begins, the budget allocator assigns a per-request reasoning limit \(B_i\) based on prompt difficulty and current load. During inference, the system updates the smoothed load signal at each ATP and computes a load-responsive exit threshold \(\lambda_i\). Reasoning terminates when confidence exceeds \(\lambda_i\) or the budget \(B_i\) is exhausted. With both mechanisms disabled, LASER reduces to standard fixed-threshold early exit.

4.2 Load-Aware Adaptive Exit Threshold↩︎

LASER replaces the fixed threshold used in existing methods [5], [6] with a load-responsive function. At each confidence evaluation point, the adaptive threshold is computed as: \[\label{eq:threshold} \lambda_i = \lambda_{\text{base}} - \beta \cdot \tanh\!\left(\gamma \cdot \frac{\bar{L}(t) - L_{\text{target}}}{L_{\text{target}}}\right)\tag{5}\] Here \(\lambda_{\text{base}} = 0.95\) is the default threshold, \(\bar{L}(t)\) is the EMA-smoothed load signal, and \(L_{\text{target}}\) is the target load level. \(\beta\) controls the maximum adjustment amplitude and \(\gamma\) controls sensitivity to load deviations. When \(\bar{L}(t) > L_{\text{target}}\), the threshold decreases to accelerate exits; when \(\bar{L}(t) < L_{\text{target}}\), it increases to preserve quality. The final value is clamped to \([\lambda_{\min}, \lambda_{\max}]\).

We choose \(\tanh\) over linear or sigmoid mappings for two reasons. First, its bounded range \((-1,1)\) helps keep \(\lambda_i\) within a safe interval and limits the magnitude of load-driven adjustments. Second, its S-shaped curve produces gentle adjustment near the target load and saturates under extreme deviation, preventing overreaction to transient spikes.

To prevent threshold jitter from instantaneous load fluctuations, we apply EMA smoothing: \(\bar{L}(t) = \alpha \cdot L(t) + (1 - \alpha) \cdot \bar{L}(t-1)\). \(L(t)\) is the current active request count and directly available as the size of the active request set in the vLLM batch processing framework.

4.3 Difficulty- and Load-Aware Reasoning Budget Pre-Allocation↩︎

While the adaptive threshold provides soft control over exit timing, LASER also assigns a hard reasoning budget \(B_i\) to each request before inference begins: \[\label{eq:budget} B_i = \max\!\left\{B_{\min},\; \left\lfloor B_{\text{base}} \cdot f_{\text{diff}}(p_i) \cdot f_{\text{load}}(L(t)) \right\rfloor \right\}\tag{6}\] where \(B_{\text{base}}\) is the default maximum reasoning steps and \(B_{\min} \geq 2\) ensures every request receives at least minimal reasoning.

Difficulty factor (line 3). The difficulty factor \(f_{\text{diff}}\) uses prompt token count \(|p_i|\) as a lightweight complexity proxy, linearly mapping it to a budget scaling factor clamped to \([f_{\min}, f_{\max}]\): \[\label{eq:difficulty} f_{\text{diff}}(p_i) = \text{clip}\!\left(f_{\min} + \frac{|p_i| - p_{\min}}{p_{\max} - p_{\min}} \cdot (f_{\max} - f_{\min}),\; f_{\min},\; f_{\max}\right)\tag{7}\] where we set \(f_{\min}=0.6\), \(f_{\max}=1.5\), \(p_{\min}=50\), \(p_{\max}=275\) by default. Empirical evidence shows that shorter prompts exhibit higher early exit rates and require fewer reasoning steps [3], [5], supporting prompt length as a difficulty proxy.

Load factor (line 3). The load factor compresses budgets proportionally when the system is overloaded: \[\label{eq:load95factor} f_{\text{load}}(L(t)) = \min\!\left(1.0,\; \frac{L_{\text{target}}}{\max(1, L(t))}\right)\tag{8}\] When \(L(t) \leq L_{\text{target}}\), \(f_{\text{load}} = 1.0\) and budgets are unmodified; under overload, budgets shrink proportionally. The adaptive threshold thus provides soft control over exit timing, while the budget imposes a hard cap on reasoning depth.

4.4 Complexity Analysis↩︎

At each ATP, LASER performs one EMA update, one \(\tanh\) computation (Eq. 5 ), and one comparison, all in \(O(1)\). Budget allocation (Eq. 6 ) also runs in \(O(1)\) per request. The total added cost for \(N\) requests is \(O(\sum_{i=1}^{N} s_i)\), dominated by LLM inference itself. LASER maintains only a single scalar \(\bar{L}\) and a fixed set of hyperparameters, so the additional memory is \(O(1)\). No extra neural network inference is required; the added latency per request is on the order of microseconds, several orders of magnitude below the per-step decoding time (10–100 ms per chunk on edge GPUs).

5 Experiments↩︎

5.1 Experimental Setup↩︎

Models. We evaluate two reasoning LLMs from different model families: Qwen3-4B (requiring approximately 8 GB VRAM) and DeepSeek-R1-Distill-Qwen-7B (requiring approximately 14 GB VRAM). Both are edge-server-class models that fit within a single consumer-grade GPU.

Benchmarks. We evaluate on four reasoning benchmarks [5]: GSM8K [22] (1319 elementary math problems), MATH-500 [23] (500 competition-level math problems), AMC 2023 [5] (40 competition problems), and GPQA Diamond [24] (198 PhD-level science questions). For reasoning quality, we report accuracy (Acc), average token count (Tok), and compression rate (CR defined as \(1-\mathrm{Tok}_{\mathrm{method}}/\mathrm{Tok}_{\mathrm{vanilla}}\)). For system performance, we report average latency (\(\bar{l}\)), P95 latency (\(l_{95}\)), throughput (QPS), and SLO satisfaction rate (SLO%).

Baselines. We compare LASER against four methods: (1) Vanilla: original LRM without early exit; (2) Fixed-High [5]: fixed threshold \(\lambda=0.95\), representing the standard early exit configuration; (3) Fixed-Low: fixed threshold \(\lambda=0.90\), representing aggressive static tuning; (4) NoThinking: skipping the reasoning phase entirely, serving as a lower bound on reasoning quality.

Implementation details. All experiments run on an NVIDIA RTX 4090 GPU (24 GB VRAM), which closely matches high-end edge accelerators such as the NVIDIA Jetson AGX Orin (64 GB unified memory). The 4B–7B parameter models fit in 24 GB VRAM, consistent with realistic edge deployment. For system-level evaluation, we employ discrete-event simulation. We first run each method on all benchmark samples to obtain per-sample inference times and accuracy, then simulate Poisson arrivals with QPS \(\{0.5, 1, 2, 4, 8\}\) using a single-server FIFO queue. For burst traffic, we use a three-phase pattern of 10 minutes at 1 QPS, 10 minutes at 5 QPS, and 10 minutes at 1 QPS. Unless otherwise stated, the SLO deadline is \(d = 30\) seconds and the default parameters are \(\lambda_{\text{base}}=0.95\), \(L_{\text{target}}=10\), \(\beta=0.04\), \(\gamma=1.0\), \(\alpha=0.3\), \(B_{\text{base}}=10\), \(\lambda_{\min}=0.88\), \(\lambda_{\max}=0.97\), and \(B_{\min}=2\). The maximum generation length is 16384 tokens, with think_ratio\(\,=0.6\) for DeepSeek models and \(0.8\) for Qwen3 models.

5.2 Reasoning Quality Results↩︎

Fig. 4 presents the reasoning quality comparison across all benchmarks for both models. LASER achieves accuracy within 2 percentage points of Fixed-High on average across both models while generating substantially fewer tokens. On DeepSeek-R1-Distill-Qwen-7B, LASER achieves 42–80% token compression relative to Vanilla, with accuracy degradation of 1.4–5.0 percentage points compared to Fixed-High. On Qwen3-4B, LASER achieves 59–72% compression, with accuracy within 1 percentage point of Fixed-High on GSM8K and MATH-500 and even outperforming Fixed-High on AMC by 5.0 pp.

5.3 System Performance Under Varying Loads↩︎

Table 1 presents system-level performance under varying QPS levels on MATH-500 for both models. Under low load (QPS=0.5), queueing effects are minimal and LASER’s advantage is most visible in raw latency reduction (36–38%). As load increases, all methods experience higher latency due to queueing, but LASER consistently outperforms fixed-threshold baselines. At QPS=2, LASER reduces average latency by 20–23% compared to Fixed-High. Under high load (QPS=8), the reduction remains 17–20%. Throughput improvement is consistent across load levels, with LASER achieving 0.32–0.34 QPS versus 0.28–0.31 for Fixed-High, a 6–17% gain. SLO satisfaction under LASER exceeds Fixed-High by 4–6 percentage points under medium and high loads.

5.4 Burst Traffic Response↩︎

We evaluate LASER’s response to sudden traffic spikes using a three-phase load pattern on MATH-500 (1\(\to\)​5\(\to\)​1 QPS, equal-duration phases). Table 2 reports the results. On DS-R1-Distill-Qwen-7B, LASER reduces average latency from 175.4 s (Fixed-High) to 137.5 s (21.6% reduction) and improves SLO satisfaction from 13.6% to 17.6%. On Qwen3-4B, LASER achieves 133.7 s versus 163.9 s for Fixed-High (18.4% reduction), with SLO satisfaction improving from 12.0% to 17.6%. LASER detects the load spike through its EMA-smoothed signal and lowers the threshold to accelerate exits during the burst phase, then gradually restores quality as load returns to normal.

5.5 Ablation Study↩︎

Fig. 5 presents the ablation study on DS-R1-Distill-Qwen-7B and MATH-500 under medium load (QPS=2), isolating the contribution of each component. Budget allocation is the primary driver of latency reduction, decreasing average latency by 23.2% compared to the fixed-threshold baseline. The adaptive threshold alone does not reduce latency because without budget control it merely shifts the exit point within the reasoning chain without capping the maximum depth. When combined, the two mechanisms enable LASER to achieve the best throughput while recovering 1.4 percentage points of accuracy compared with the budget-only variant. The adaptive threshold preserves reasoning quality by allowing deeper reasoning under light load, complementing the budget allocator.

5.6 Parameter Sensitivity↩︎

Table 3 reports the sensitivity of LASER to its key hyperparameters on MATH-500 at QPS=2. LASER shows robust performance across a wide range of parameter values. For \(\beta\), accuracy remains within 83–89% across the sweep and latency varies smoothly. For \(L_{\text{target}}\), higher values tend to preserve accuracy while lower values favor throughput. Moderate EMA smoothing (\(\alpha \in [0.2, 0.5]\)) achieves a good balance between responsiveness and stability. The \(\tanh\) function naturally bounds the adjustment, providing self-correcting behavior that reduces the need for per-scenario tuning in edge deployments.

6 Conclusion↩︎

This paper presents LASER, a load-aware early exit approach for reasoning model serving at the edge. We have identified the gap between single-request reasoning optimization and system-level edge serving, and addressed it with two complementary mechanisms: an adaptive exit threshold driven by real-time system load and a difficulty- and load-aware reasoning budget allocator. The budget allocator serves as the primary driver of latency reduction by capping per-request reasoning depth according to difficulty and load, while the adaptive threshold exploits the confidence polarization phenomenon to preserve reasoning quality within a validated safe range. Experiments on two models, four benchmarks, and diverse load conditions show that LASER reduces average latency by 17–38% and improves SLO satisfaction by 3–6 percentage points over fixed-threshold baselines, at an average accuracy cost of only 1%. In future work, we will extend LASER to multi-device edge clusters with heterogeneous GPUs and integrate reasoning depth control with model selection routing.

References↩︎

---

1. Corresponding authors.↩︎