1 Introduction↩︎

1.1 The Deployment Imperative↩︎

Modern machine learning’s research interest, particularly in Large Language Models (LLMs), is shifting from furthering model capabilities toward serving them efficiently at scale. LLMs introduce challenges that go beyond raw parameter count: autoregressive decoding is sequential and does not fully exploit the parallel processing capabilities of GPUs, the Key-Value (KV) cache grows proportionally with sequence length, number of layers, and attention heads (making its total footprint substantial for large models under long-context or high-concurrency workloads), and deployment targets span a wide range of resource constraints from data-center GPUs to embedded microcontrollers. Practical use often dictates that reasonable accuracy at lower latency and computational cost is more valuable than a state-of-the-art model that demands heavy compute or incurs high latency. For many production systems, inference has consequently become the dominant ongoing cost center, often exceeding the amortized cost of training over a model’s operational lifetime.

1.2 A Practitioner’s Gap↩︎

Despite an extensive research literature, a systematic gap persists between how optimization techniques are studied and how they are deployed in practice. Academic work frames optimization as an algorithmic question:

“How can technique X be improved on benchmark Y?”

By contrast practitioners face a different, operationally grounded question:

“Given my hardware, latency SLA, data budget, and accuracy floor, which combination of techniques should I apply, and in what order?”

This paper directly addresses the second question. We do not introduce a new optimization algorithm. Instead we provide a structured decision framework that maps deployment constraints to optimization strategies, synthesizing quantitative evidence from the primary literature to support each recommendation. We reviewed recent proceedings from major machine learning and systems venues, selecting techniques that report quantifiable improvements against critical deployment bottlenecks such as latency, memory footprint, or retraining cost. We note that our review follows a narrative rather than a formal systematic-review methodology and does not provide a reproducible search protocol, inclusion/exclusion criteria, or exhaustive screening workflow.

1.3 Contributions↩︎

1. Operational Constraint Taxonomy. Five interacting deployment constraint dimensions that collectively characterize a production ML system and provide a common vocabulary for optimization decisions.

2. Functional Categorization. We note the functional category of each optimization technique (e.g., parameter reduction, inference optimization) alongside our primary constraint-based classification, complementing existing algorithm-centric surveys.

3. Quantitative Constraint–Technique Mapping. A synthesis of empirical results from over 25 primary papers, mapping measured gains to the deployment constraints they address.

4. Four Prescriptive Deployment Pipelines. Evidence-informed, step-by-step optimization pipelines for representative industrial deployment scenarios, with explicit constraint profiles and expected outcomes.

2 Operational Constraint Taxonomy↩︎

We characterize any machine learning deployment environment along five interacting constraint dimensions. While presented separately, these dimensions are not independent. For example, quantization simultaneously affects memory, latency, and accuracy, while data availability directly constrains the retraining budget. Together they define a constraint profile that can guide which optimization strategies to consider and in what sequence.

2.1 Data Availability↩︎

Data availability refers to the quantity, quality, and accessibility of labeled, unlabeled, or weakly labeled data usable for training, calibration, or adaptation. In many enterprise settings, teams operate with zero task-specific data, meaning only the pretrained checkpoint is accessible. This restricts optimization to training-free techniques like post-training quantization. In other scenarios, teams might possess limited data, consisting of less than 1,000 labeled examples or weak supervision signals, which opens the door to parameter-efficient fine-tuning (though with such small datasets, full PEFT on large models still carries a risk of overfitting). When moderate data of 1,000 to 100,000 labeled examples is available, full alignment tuning becomes viable. Finally, abundant data involving hundreds of thousands of examples permits full retraining and large-scale knowledge distillation.

2.2 Latency Budget↩︎

The latency budget defines the maximum acceptable response time, characterized by the target latency percentile and the specific deployment mode. The distinction between time-to-first-token, inter-token latency, and end-to-end latency matters and should be specified explicitly. Real-time applications, such as autonomous systems or live voice agents, demand strict latency under 200 milliseconds, typically requiring techniques such as speculative decoding, early exiting, or model distillation to a smaller architecture. Interactive applications, typified by chat interfaces or code completion assistants, generally target between 200 milliseconds and 2 seconds. Batch or offline processing pipelines prioritize raw throughput and hardware utilization over per-query latency, making continuous batching particularly effective in this regime.

2.3 Memory Budget↩︎

Memory budget encapsulates the total available memory spanning both Video RAM (VRAM) and standard DRAM. This budget must accommodate parameter storage, the key-value (KV) cache, runtime activations, and optimizer states if training is involved. Edge and mobile deployments represent the most constrained environments, typically offering less than 4 GB of VRAM. Consumer GPUs provide a moderate budget of up to 24 GB of VRAM, allowing for local execution of quantized multibillion-parameter models. Enterprise GPU nodes, often spanning 80 GB to 320 GB of VRAM, shift the memory bottleneck away from parameter storage and heavily toward managing the KV cache for thousands of concurrent user sessions.

2.4 Accuracy Tolerance↩︎

Accuracy tolerance dictates the maximum permissible performance degradation following optimization relative to the original, unoptimized reference model. The specific metric must be defined per-deployment (for example, exact match, task success rate, factuality, calibration error, or human preference win rate). The thresholds below are illustrative and not directly comparable across metrics. Mission-critical deployments, spanning medical diagnostics or financial trading, typically permit less than 0.5% degradation on the chosen task metric and may restrict optimization to well-calibrated 8-bit quantization. Business-critical applications, encompassing standard customer support or internal productivity tools, generally tolerate between 0.5% and 2% degradation, allowing for 4-bit quantization and structured pruning. Cost-sensitive applications, where high query volume makes serving the full model financially unsustainable, may accept greater than 2% degradation in exchange for substantial throughput gains. In all cases, practitioners should also evaluate tail-risk failures, subgroup performance, and severity-weighted errors rather than relying solely on aggregate accuracy.

2.5 Retraining Budget↩︎

The retraining budget quantifies the available GPU-hours, engineering time, and financial resources allocated for modifying the deployed model. A retraining budget of zero means no compute is available to alter model weights, which limits practitioners to post-training quantization, training-free sparsification, and inference engine optimizations. A minimal budget, typically under 10 GPU-hours on a single node, can support targeted parameter-efficient fine-tuning strategies. A full budget representing hundreds of GPU-hours on cluster hardware can justify more extensive procedures such as quantization-aware training, neural architecture search, and structured pruning.

2.6 Scope and Limitations of This Taxonomy↩︎

The five dimensions above provide a useful initial characterization, but production deployments involve additional first-class constraints that this framework does not treat as primary axes, including:

• Throughput and goodput

• Concurrency, traffic shape, and burstiness

• Prompt and output-length distributions

• Cost per successful request (serving cost)

• Energy and thermal limits

• Hardware and kernel compatibility

• Availability, resiliency, and observability

• Data privacy and model residency

Furthermore, latency and throughput should not be treated as interchangeable, as optimizing one can hurt the other. We encourage practitioners to extend the constraint profile to cover these dimensions as appropriate for their deployment.

3 Optimization Techniques by Constraint Dimension↩︎

Rather than grouping by algorithmic family, we organize the literature by the primary deployment constraint each technique addresses. This provides practitioners with a direct mapping from their binding constraint to candidate solutions. (An overview of all evaluated techniques is provided in Table 1 in the Appendix.)

3.1 Memory Budget Optimization↩︎

Techniques in this category reduce the memory footprint of the model parameters or the runtime KV cache, enabling deployment on smaller or fewer GPUs.

GPTQ [1]: A post-training weight quantization technique that uses an approximate inverse-Hessian method to minimize layer-wise reconstruction error. By quantizing billion-scale models to 3 or 4 bits, it achieves approximately a 4\(\times\) reduction in weight-storage memory footprint relative to FP16. In the reported experiments, the resulting 4-bit models maintain a perplexity within 0.25 points of the FP16 baseline, and the quantization process required less than 4 GPU-hours for a 175B model on the hardware used in the original study. These figures will vary with model architecture, hardware, and configuration.

AWQ [2]: An activation-aware weight quantization technique that protects a small subset (roughly 1%) of salient weight channels identified via activation magnitudes, avoiding the need for backpropagation during quantization. In the reported experiments, AWQ delivered accuracy within 0.5% of the FP16 baseline on instruction-following benchmarks and provided a 1.45\(\times\) throughput advantage over GPTQ on the tested edge hardware. Actual accuracy and throughput will depend on the model, task, and target device.

OmniQuant / ZeroQuant-V2 [3], [4]: Advanced quantization strategies designed to handle significant outliers in LLM activations. OmniQuant optimizes weight clipping boundaries and scaling factors, while ZeroQuant-V2 utilizes Low-Rank Compensation (LoRC) to mitigate quantization inaccuracies. These methods report near-lossless 4-bit and 8-bit quantization on the evaluated benchmarks without extensive retraining, though quality preservation will vary by model and task.

KIVI / KVQuant [5], [6]: Specialized quantization algorithms targeting the Key-Value (KV) cache, which becomes the primary memory bottleneck in long-context inference. KIVI introduces a tuning-free asymmetric 2-bit quantization scheme for the KV cache. In the reported experiments, this approach substantially reduces runtime memory consumption, with results suggesting up to 4\(\times\) longer effective context lengths on identical hardware with limited perplexity degradation. Gains are model- and sequence-length-dependent.

Wanda [7]: A training-free unstructured pruning approach that determines weight importance by evaluating the product of weight magnitudes and the norm of corresponding input activations. Wanda achieves 50% unstructured sparsity in LLMs while often outperforming standard magnitude pruning by several percentage points on zero-shot benchmarks. (It is included in this category because it targets parameter compression, though setting weights to zero does not automatically reduce physical memory, deployed model size, or inference latency. Realizing these runtime benefits requires sparse storage formats, compatible sparse kernels, and hardware or runtime support for the chosen sparsity pattern).

FlexGen / PowerInfer [8], [9]: System-level offloading frameworks designed to execute large LLMs on hardware with limited GPU VRAM. FlexGen offloads weights and KV cache to CPU memory and disk, formulating a graph traversal problem to overlap data movement with computation. PowerInfer exploits activation sparsity to keep hot neurons on the GPU and cold neurons on the CPU. In the reported experiments these methods achieve substantially higher single-GPU generation throughput compared to naive offloading (up to 10\(\times\) in some configurations), though actual gains depend heavily on I/O bandwidth, sparsity pattern, and model architecture.

LoSparse [10]: A hybrid approach combining low-rank approximation with weight pruning. By leveraging low-rank decomposition, LoSparse prevents the loss of expressive neurons that typically occurs with direct model pruning. This strategy can provide significant parameter reduction while better maintaining the reasoning capabilities of the original model compared to pruning alone, though the actual memory and latency benefits depend on the availability of compatible sparse kernels and runtime support.

3.2 Latency Budget Optimization↩︎

These methods aim to accelerate inference speed or increase throughput, helping systems work toward real-time or interactive latency service level agreements. SLA compliance also depends on hardware configuration, serving infrastructure, and workload characteristics.

FlashAttention [11] / FlashAttention-2 [12]: IO-aware exact attention algorithms that fuse softmax and matrix multiplications into tiled CUDA kernels to minimize High Bandwidth Memory (HBM) accesses. FlashAttention (2022) reduces auxiliary attention-memory requirements from quadratic to linear in sequence length (the attention computation itself remains \(O(N^2)\) for dense full attention), reporting 2–4\(\times\) wall-clock speedups over optimized baselines in the original experiments. FlashAttention-2 (2023) further improves GPU work partitioning and parallelism, achieving approximately 2\(\times\) additional speedup over FlashAttention in the reported benchmarks. Speedup figures are workload-, hardware-, and baseline-dependent.

Speculative Decoding (Medusa / Eagle) [13], [14]: Advanced decoding strategies that break the sequential bottleneck of autoregressive generation. While traditional speculative decoding uses a separate draft model, Medusa trains multiple heads on top of the target model to predict subsequent tokens, and Eagle predicts the target model’s next hidden state to generate draft tokens without a separate model, reducing draft generation cost. These methods achieve 2 to 3.7\(\times\) end-to-end wall-clock speedups in the reported experiments. When the verification and rejection-sampling procedure is exact, speculative sampling can preserve the target model’s output distribution, though bit-for-bit identical outputs are not guaranteed across all implementations, RNG configurations, and acceptance strategies. Performance benefits are also workload-dependent, as speculative decoding can regress throughput under high concurrency, poor draft-acceptance rates, or memory-constrained settings.

vLLM (PagedAttention) [15]: A serving engine optimization that manages the KV cache using principles inspired by operating system virtual memory. By storing the KV cache in non-contiguous paged blocks, PagedAttention substantially reduces memory fragmentation, though block-level internal fragmentation and scheduling overhead can remain. This continuous batching strategy allows serving systems to process more concurrent requests (up to 4\(\times\) in the reported experiments), improving the throughput capabilities of GPU clusters.

LLMLingua [16]: A prompt compression technique that performs coarse-to-fine pruning of input tokens based on perplexity signals from a smaller model. By removing redundant tokens before the prefilling stage, LLMLingua reduces the initial computation time. In the reported experiments it compresses prompts by up to 20\(\times\) while preserving downstream task quality. The degree of acceptable compression and quality retention is task- and model-dependent.

Skeleton-of-Thought (SoT) [17]: An output organization framework that instructs the LLM to first generate a concise skeleton of points, and then uses batch inference to expand upon each point in parallel. In the reported experiments, SoT reduces end-to-end generation latency by up to 2.39\(\times\) on multi-part queries without modifying the underlying model architecture. Gains depend on query structure and the availability of parallel decoding capacity (which trades per-query latency for overall system throughput).

StreamingLLM [18]: A framework that enables LLMs trained with finite context windows to maintain stable generation over very long token streams without fine-tuning. By recognizing that LLMs rely heavily on initial tokens (attention sinks), StreamingLLM preserves these sink tokens alongside a sliding window of recent tokens and discards intermediate KV states. This permits stable language modeling over millions of processed tokens with bounded memory usage. Because intermediate tokens are evicted, however, StreamingLLM does not provide faithful semantic access to all earlier tokens. It is suited for streaming or dialogue continuity, not for tasks requiring recall or reasoning over the full document history.

Gisting / AutoCompressors [19], [20]: Soft prompt compression techniques that condense task-specific instructions into a small set of learnable continuous tokens. Gisting uses prefix-tuning to create virtual tokens that summarize prompts, while AutoCompressors utilize an autoencoder approach over the context. These methods can substantially reduce the token footprint of task instructions for downstream use (reducing prefill latency), though compression quality and resulting task performance depend on the model and instruction type.

3.3 Data Availability Optimization↩︎

These strategies optimize model performance when labeled domain data is highly scarce or non-existent, leveraging synthetic data or parameter-efficient prompting.

LIMA [21]: A study suggesting that alignment and instruction-following capabilities of an LLM may be strongly influenced by data quality rather than quantity alone. By fine-tuning a 65B parameter model on a carefully curated dataset of only 1,000 examples, the resulting model was preferred over GPT-4 in 43% of human evaluations in the reported study. The authors argue this reduces reliance on large-scale labeled datasets or Reinforcement Learning from Human Feedback (RLHF), though the generalizability of this finding to other model families and tasks requires further investigation.

Socratic CoT [22]: A knowledge distillation approach that addresses the lack of high-quality reasoning data. It fine-tunes a pair of models (a Question Generation model and a Question Answering model) to synthesize intermediate reasoning steps, then distills these synthetic rationales from a large teacher LLM into a smaller student. The approach aims to transfer Chain-of-Thought reasoning capabilities without requiring human-annotated reasoning paths. Reported gains are benchmark- and domain-specific.

3.4 Retraining Budget Optimization↩︎

These methods reduce the compute and data requirements for model adaptation. It is worth noting that Parameter-Efficient Fine-Tuning (PEFT) techniques such as LoRA primarily reduce training resource requirements and do not automatically improve inference latency. Merged adapters match the base model’s inference cost, while unmerged adapters add overhead. Similarly, Mixture-of-Experts (MoE) architectures are model-design choices and are not post-hoc optimization steps that can be applied to an existing dense checkpoint.

LoRA [23]: Low-Rank Adaptation freezes the pretrained model weights and injects trainable rank decomposition matrices into the architecture. When applied to a 175B parameter model in the original experiments, LoRA reduced the number of trainable parameters by approximately 10,000\(\times\) to 0.01% of total parameters, achieving comparable performance to full fine-tuning on benchmarks such as SuperGLUE while reducing GPU memory requirements for the training step (primarily by reducing optimizer state overhead) by a reported factor of 3. These savings apply to the adaptation phase; at inference, merged adapters incur no additional cost over the base model.

LoftQ / QA-LoRA [24], [25]: Techniques that bridge the gap between post-training quantization and quantization-aware training. LoftQ initializes the LoRA matrices using Singular Value Decomposition (SVD) of the difference between the original weights and the quantized weights. QA-LoRA utilizes group-wise quantization and merges LoRA terms into the quantized matrices. Both methods are designed to enable fine-tuning of heavily compressed models on consumer hardware with reduced accuracy loss compared to naive quantization followed by LoRA. Actual fidelity depends on quantization level and task.

LLM-QAT / Norm Tweaking [26], [27]: Quantization-Aware Training methods. LLM-QAT generates its own synthetic training data by prompting the FP16 model, and then uses a distillation workflow to train the quantized model. Norm Tweaking fine-tunes only the LayerNorm layers using generated data, aiming for QAT-level accuracy recovery with lower data and compute requirements than full QAT. Actual savings are model- and task-dependent. (While they avoid requiring the original training data, they do require retraining compute budgets).

Switch Transformers / Expert Choice [28], [29]: Efficient structural designs utilizing Mixture-of-Experts (MoE) to scale model capacity without proportional increases in training compute. Switch Transformers route tokens to a single expert, minimizing routing overhead. Expert Choice allows experts to select the top-\(k\) tokens, which tends to improve load balancing compared to token-choice routing, though perfect balancing is not guaranteed in all configurations. These MoE architectures achieve better task performance for the same training compute budget compared to dense models of similar active parameter count and offer efficient fine-tuning pathways, but adopting them requires selecting or training an MoE model from scratch rather than converting an existing dense checkpoint.

LoraPrune / ZipLM [30], [31]: Methods that combine structured pruning with parameter-efficient fine-tuning. LoraPrune uses the weights and gradients of LoRA matrices as a criterion to iteratively prune unimportant weights. ZipLM iteratively identifies and removes structural components offering the worst loss-to-runtime tradeoff. In the reported experiments, these methods produce compressed, domain-adapted models with comparatively low GPU-hour budgets. Actual compute requirements depend on model size, pruning target, and task.

3.5 Validate Accuracy and Manage Cost↩︎

Techniques for managing financial costs and maintaining reliability, factuality, and calibration post-optimization.

FrugalGPT [32]: A cost-reduction strategy employing a cascade router that directs queries to APIs of varying capability based on a learned complexity threshold. In the original study’s workload, approximately 83% of queries were handled by cheaper, smaller models, with the remaining 17% routed to the premium API, yielding up to 98% cost reduction with a 4% accuracy improvement (though this accuracy improvement is counterintuitive and likely due to ensemble effects on those specific benchmarks; generally, routing more queries to smaller models trades accuracy for cost). These routing ratios and cost savings are workload-specific and should not be treated as universal constants. Practitioners must measure escalation rates on their own traffic distribution.

SpQR / SqueezeLLM [33], [34]: Accuracy-preserving quantization methods that identify a small fraction of weight outliers disproportionately affecting model accuracy. SpQR isolates and stores these outliers in FP16 format while heavily quantizing the remaining weights to 3 or 4 bits. This outlier-aware approach reports near-lossless compression on the evaluated benchmarks, with the aim of preserving reasoning and calibration quality. Performance preservation is model- and task-dependent.

CALM / SkipDecode [35], [36]: Dynamic inference techniques that implement early exiting during the decoding phase. CALM trains classifiers after each layer to output confidence scores, halting inference early for tokens where confidence is high. SkipDecode extends this to batch inference by ensuring a unified exit point for all tokens in a batch. In the reported experiments, these methods reduce the average computational cost per token with limited accuracy degradation. The trade-off between compute savings and quality is threshold- and task-dependent.

4 Decision-Making Framework↩︎

To navigate the landscape of optimization techniques, we propose a structured, multi-phase methodology as a starting point. The framework recommends addressing optimization hierarchically: resolving hard infrastructural limits first, followed by sequential refinements to latency, domain adaptation, and accuracy assurance. This ordering is a heuristic guide and not a strict rule. Practitioners may need to revisit earlier phases as later constraints are evaluated. Figure 1 illustrates this progressive approach.

4.1 Phase 1: Determine Deployment Topology (Memory Budget)↩︎

Hardware memory limits are typically the first constraint to resolve, as they determine whether a model can be loaded at all.

• Edge and Mobile Deployments (\(<\)​8 GB VRAM): In environments where parameter memory is the binding bottleneck, Post-Training Quantization (PTQ) is typically the first tool to reach for. Activation-aware methods like AWQ are well-suited to this regime and are designed to reduce quality loss on constrained hardware. If memory constraints persist after quantization, structured pruning or hardware-supported N:M sparsity patterns (which have compatible accelerated kernels) can further reduce the effective parameter count. Unstructured pruning (such as Wanda) sets weights to zero but does not reduce physical memory or latency without sparse storage formats, compatible kernels, and runtime support.

• Single Enterprise Node (24–80 GB VRAM): Deploying large models (e.g., 70B parameters) on a single GPU typically requires low-bit quantization methods, such as OmniQuant or ZeroQuant-V2. If increasing batch sizes begin to induce out-of-memory errors, system-level offloading frameworks like FlexGen or PowerInfer can be introduced to utilize CPU RAM and disk storage for cold activations.

• Distributed GPU Cluster (\(>\)​80 GB VRAM): While parameter storage is less constrained across a cluster, the runtime memory demanded by the KV cache can rapidly become the primary bottleneck as concurrent users scale. In these scenarios, KV cache quantization techniques (such as KIVI or KVQuant) are worth implementing early to compress the runtime footprint.

4.2 Phase 2: Establish the Latency and Throughput SLA (Latency Budget)↩︎

Once the model fits within the memory budget, the serving engine can be tuned to work toward specific Service Level Agreements (SLAs).

• Real-Time Interactive Systems (e.g., Voice and Code Completion): When token generation speed is paramount, speculative decoding architectures like Eagle or Medusa are worth evaluating. In the original experiments they achieved 2–3.7\(\times\) speedups, though gains are workload-dependent. Pairing this with FlashAttention-2 can reduce HBM traffic during the prefill phase, which may lower prefill latency.

• High-Concurrency API Serving: If maximizing system throughput is the primary objective, deploying a serving engine equipped with PagedAttention (like vLLM) enables continuous batching and substantially reduces KV cache fragmentation. Additionally, implementing batch-aware early exiting (e.g., SkipDecode) can dynamically halt generation for highly confident tokens, thereby reducing the average computational cost per request.

• Long-Context Applications (e.g., Document QA): Prefilling large context windows is computationally expensive. Prompt compression techniques like LLMLingua can prune redundant tokens from the input before processing, reducing prefill cost. For streaming or dialogue-continuity use cases, StreamingLLM can preserve essential attention sinks to maintain stable generation over very long token streams. StreamingLLM evicts intermediate KV states, however, and is therefore unsuitable for tasks requiring faithful recall over the entire document. For such tasks, retrieval reranking, hierarchical summarization, or validated long-context models should be considered instead.

4.3 Phase 3: Assess the Data and Retraining Reality (Data & Retraining Budgets)↩︎

Domain adaptation strategy is shaped by the intersection of available data and computational resources. The guidance below describes typical patterns, though the best approach should be validated empirically.

• Abundant Data (\(>\)​10K samples) with High Compute: This scenario supports full task-specific distillation. By training a compact student model to match the logits and hidden states of a larger teacher model, practitioners can achieve faster inference while targeting domain accuracy retention. Actual accuracy versus speed trade-offs depend on model capacity, data quality, and distillation configuration.

• Moderate Data (\(\sim\)​1K curated samples) with Low Compute (Single GPU): When compute is limited, Parameter-Efficient Fine-Tuning is a practical path. Methods like QA-LoRA or LoftQ integrate low-rank adaptation directly into the quantized weights, enabling fine-tuning without fully dequantizing the model. Quality outcomes depend on the rank, quantization level, and task.

• Zero Task Data: In the complete absence of training data, traditional retraining is impossible. Practitioners must rely on quantization methods that do not require the original training data, such as LLM-QAT or Norm Tweaking. These techniques generate synthetic calibration data from the pretrained model itself, enabling quantization-aware training without human-labeled datasets (though they are not strictly “zero-data” in an operational sense, as they do require compute for synthetic data generation and calibration).

4.4 Phase 4: Fortify Accuracy and Manage Cost (Accuracy Tolerance)↩︎

The final phase addresses accuracy assurance and cost management for the optimized system.

• Mission-Critical Reliability (Medical, Legal, Finance): Standard low-bit quantization can introduce tail-risk failures in highly sensitive domains, making quality preservation particularly important. Outlier-aware quantization (like SpQR or SqueezeLLM) isolates and stores high-magnitude weight outliers in FP16 while quantizing the remainder, and has been shown to better preserve model quality than uniform low-bit quantization. Self-Consistency sampling (generating multiple reasoning paths and taking a majority vote) can improve answer stability on reasoning tasks, but it does not guarantee factuality—correlated samples from the same model can repeat the same unsupported claims. For high-stakes domains, practitioners should additionally implement sentence-level citations, evidence-span validation, groundedness and entailment checks, answerability detection, and abstention thresholds.

• API Cost Reduction: When relying on proprietary external models, the optimization target shifts from GPU memory to financial expenditure. A cascade routing system such as FrugalGPT can reduce costs by directing simpler queries to a cheaper, locally hosted quantized model (e.g., an 8B parameter architecture) and reserving the premium API for complex reasoning tasks. As noted earlier, the cost savings are workload-specific and must be measured on the organization’s own traffic distribution.

5 Industry Use Cases↩︎

5.1 Edge AI: Alice and the Mobile Deployment↩︎

Alice is a mobile AI engineer tasked with deploying a 7B parameter local assistant onto a smartphone with a strict 4 GB RAM limit and heavy battery constraints.

Facing a severe memory bottleneck, she begins by analyzing her toolkit. A standard 7B model at FP16 requires approximately 14 GB of memory for weights alone, which far exceeds the device budget. Alice’s primary lever is 4-bit weight-only quantization: she applies AWQ, which identifies and protects salient weight channels while quantizing the remainder to INT4. A 7B model at 4-bit precision requires approximately 3.5–4 GB for weights (depending on the fraction of channels stored at higher precision and embedding layers), bringing it within the hardware limit. The total on-device footprint will also include runtime buffers, the KV cache (which scales with prompt length), tokenizer overhead, and the mobile runtime, so Alice must profile these components on the target device to confirm headroom. She then applies graph-level optimizations using CoreML to fuse operators for the device’s Neural Engine, improving energy efficiency and token throughput. Should she also wish to apply Wanda pruning to further reduce the weight payload, she would need to verify that the target runtime supports sparse storage and sparse matrix operations. Without such support, unstructured sparsity provides no memory or latency benefit at inference time (see Figure 2).

5.2 Enterprise LLM Serving: Bob and the GPU Cluster↩︎

Bob is the infrastructure lead for a high-traffic enterprise application, responsible for serving a 70B parameter open-weight model under heavy concurrent load.

Bob’s binding constraints are throughput and latency. His cluster of A100 GPUs has enough raw VRAM to hold the model parameters, but the dynamic KV cache for many simultaneous sessions causes memory fragmentation and out-of-memory errors under continuous batching. Bob addresses the runtime architecture in three stages. First, he deploys the vLLM serving engine, which uses PagedAttention to manage the KV cache in non-contiguous paged blocks, substantially reducing fragmentation and enabling continuous batching. Second, he enables FlashAttention-2 to reduce HBM traffic during the prefill phase. Third, he integrates Eagle speculative decoding to accelerate the autoregressive decode phase for individual requests. The combined effect on concurrency and latency is configuration-dependent. Actual throughput gains require profiling against the specific GPU count, parallelism strategy, input/output length distribution, and quantization level used in production. Latency targets (whether time-to-first-token, inter-token latency, or end-to-end response time) should also be defined and measured separately, as each optimization affects them differently, as illustrated in Figure 3.

5.3 Retrieval-Augmented Generation: Charlie’s Legal QA↩︎

Charlie is developing a legal document QA system. When a user asks a question, the system retrieves 30 pages of dense legal text, resulting in a 64K token context window.

Charlie faces a latency and accuracy challenge. Processing 64K tokens per query is expensive, and the model occasionally produces unsupported claims about legal provisions. To reduce the prefill cost, Charlie applies LLMLingua: this prompt compression technique evaluates the retrieved documents and prunes redundant tokens before they reach the main LLM, cutting the context size by approximately 60% while retaining semantically critical content. For the accuracy challenge (hallucinated legal provisions), Charlie implements retrieval reranking to surface the most evidentially relevant passages, paired with post-hoc groundedness checks that verify each generated claim against the retrieved text. While Charlie might also implement Self-Consistency sampling to improve answer stability on complex reasoning queries, he recognizes that Self-Consistency is not a factuality intervention, as correlated samples from the same model will consistently repeat an unsupported claim if the model is confidently wrong. StreamingLLM is not appropriate as the primary long-context strategy here, because it evicts intermediate KV states and cannot provide faithful access to all 64K tokens. It would only be suitable if Charlie were building a stateful dialogue system that streams over a very long interaction history (see Figure 4).

5.4 Production Cost Reduction: Diana’s API Scale Problem↩︎

Diana is a product manager for a consumer AI app that relies entirely on premium, proprietary LLM APIs. Due to a surge in popularity, her monthly API bill has skyrocketed to $50,000, creating a severe Cost constraint.

Diana recognizes that not every query requires the full reasoning capability of the premium API. She designs a FrugalGPT-style cascade router: she deploys a smaller 8B open-weight model on a local cloud instance, quantized with OmniQuant to run efficiently on lower-tier GPUs, and trains a lightweight classifier on her historical traffic to predict query complexity. Simple queries are handled by the local model, and complex queries are escalated to the premium API. The routing split and the resulting cost reduction depend entirely on Diana’s own traffic distribution and the quality threshold she sets. The ratios from the original FrugalGPT study are workload-specific and cannot be assumed to transfer directly. The local model also carries real serving costs (GPU or CPU compute, energy, orchestration, and engineering overhead), so Diana must account for these when computing net savings. Diana should also define an accuracy floor per query category and monitor escalation rates continuously, adjusting the router threshold as her traffic mix evolves (Figure 5).

6 Conclusion↩︎

Model optimization is no longer just an algorithmic research pursuit. It has become a foundational engineering discipline required to deploy machine learning economically. By defining the operational space along five interacting constraint dimensions (Data, Latency, Memory, Accuracy, and Retraining) and mapping empirical results from the primary literature to these constraints, practitioners can move from heuristics toward a more structured, evidence-informed methodology. The techniques discussed, ranging from 4-bit quantization and low-rank adaptation to speculative decoding and continuous batching, may be composable when applied to distinct bottlenecks, but their combined effects are not guaranteed to be multiplicative. Interactions can be non-additive, and some combinations introduce quality or latency regressions (for example, quantization reducing speculative-draft acceptance rates, or continuous batching increasing tail latency for individual requests). Practitioners are encouraged to benchmark technique combinations on representative production traffic rather than summing gains reported in independent studies.

7 Technique Comparison↩︎

Table 1 provides a summarized comparison of the optimization techniques discussed in this framework, categorizing them by their primary target constraint and key benefits.

References↩︎

---

1. Author order is based on alphabetical ordering.↩︎

2. Work done during internship at EXL.↩︎