1 Introduction↩︎

Transportation management centers (TMCs) operate continuously, monitoring hundreds of CCTV feeds, detecting and verifying incidents, coordinating response with police and fire agencies, adjusting signal timing and ramp metering, and pushing traveler information to dynamic message signs and 511 traveler information services [1]. A recent Federal Highway Administration (FHWA) briefing on next-generation Traffic Management Systems recommends that agencies “assess the balance between on-premises and cloud-enabled capabilities”, weighing AI/ML computing needs against the cost of GPUs [2]. To date, we are aware of no formal framework for this assessment.

Many TMC functions already rely on AI models. Classical detectors like YOLO [3] handle vehicle counting and basic anomaly detections, but foundation models (FMs) can support broader TMC tasks: VLMs interpret camera scenes for rare-event detection [4], LLMs predict traffic states from heterogeneous sensor data [5], draft incident reports [6], and advise on signal timing [7]. Several recent surveys document these applications [8]–[10].

As capable models become widely available, a practical planning question emerges: which model should serve each function, and how should it be deployed? Agencies can now choose closed-source cloud APIs (e.g., GPT-4o at $2.50/M tokens [11], Gemini Flash at $0.075/M [12]), open-source models on hosted inference platforms (e.g., Llama-3.1-8B [13] at $0.18/M via Together.ai [14]), or on-premise (self-hosted) GPU. These options differ substantially in cost, latency, quality, and data sovereignty. Moreover, the decisions are coupled: when multiple functions share a limited on-premise GPU budget, loading a large vision model for detection can consume available memory to force the report generator onto a more expensive cloud API.

Existing work on LLM cost optimization operates at the per-query level. FrugalGPT [15] cascades models within a single task; RouteLLM [16] and ThriftLLM [17] route individual queries to cheaper alternatives; and INFaaS [18] automates model-variant selection at runtime. None of these methods address the multi-function setting in which functions compete for shared infrastructure, and none guarantee the operational requirements a TMC imposes. A low-cost model is not useful if it misreads a sensor or produces an incoherent incident report, so deployment must respect a minimum quality. It must also meet latency targets; FHWA, for instance, names rapid incident detection a primary Traffic Incident Management performance goal [2]. And for safety-critical functions such as wrong-way-driver detection, a safety bound must hold the false-negative rate (FNR) below an agency-specified threshold. To our knowledge, FM deployment for TMC operations has not been formulated as a portfolio optimization problem under constraints of the quality, latency, safety, and costs. Our contributions are threefold.

1. We formulate the Foundation Model Deployment Portfolio (FMDP) problem as a constrained mixed-integer program that minimizes monthly total cost of ownership (TCO) across functions sharing GPU capacity, subject to quality, latency, and safety constraints.

2. We prove the FMDP is NP-hard via reduction from the 0-1 knapsack problem and propose a greedy heuristic.

3. We present an illustrative TMC case study showing 97% cost reduction over the cheapest feasible all-closed-API baseline and derive per-function API vs.on-premise break-even formulas.

2 Related Work↩︎

Foundation models in transportation. Several surveys discuss FM applications in intelligent transportation: Wandelt et al. [8] review LLMs across ITS tasks; Zhang et al. [9] focus on LLMs for mobility forecasting; Fan et al. [10] address multimodal perception and decision-making on complex roads. VLMs have been applied to video anomaly detection [4], [19], LLMs to traffic state prediction [5] and signal control [7], and LLM-based agents to incident analysis [6].

Cost-efficient model selection and routing. FrugalGPT [15] pioneered LLM cascading, reporting up to 98% cost reduction by querying models from cheapest to most expensive until a quality bar is met. ThriftLLM [17] formulates ensemble selection with approximation guarantees; RouteLLM [16] trains preference-based routers; Hybrid LLM [20] routes between cloud and on-device models; Dekoninck et al. [21] derive provably optimal routing; and FORC [22] applies integer programming to per-task assignment. All target a single function and do not model shared GPU capacity.

Deployment economics and security. Zhang et al. [23] analyze cloud vs on-premise deployment as a pricing game between LLM providers, finding that privacy concerns shape equilibrium strategies. Huang et al. [24] propose SOLID, a semi-open on-premise framework securing selected model layers against distillation while preserving fine-tuning flexibility. Both study a single model’s deployment choice.

Resource allocation and model serving. From an operations research point of view, FMDP combines a multiple-choice assignment structure with knapsack-type GPU-capacity constraints and a fixed-charge model-sharing structure, which relate FMDP to the Generalized Assignment Problem [25], [26]. INFaaS [18] automates the selection of models in heterogeneous hardware, and Mélange [27] formulates the composition of the GPU fleet as a bin packing. These systems operate at per-query or serving-system granularity, whereas FMDP addresses deployment planning across functions.

3 Problem Formulation↩︎

3.1 Preliminaries↩︎

Consider a TMC operating \(F\) functions \(\mathcal{F} = \{1, \ldots, F\}\) over a planning horizon of \(H\) hours (e.g., \(H = 720\) for one month). Each function \(f\) processes queries at rate \(\lambda_f\) (queries/hour) and must satisfy: (i) a minimum quality score \(q_f^{\min} \in [0,1]\); (ii) a maximum per-query latency \(L_f\) (seconds); and (iii) for safety-critical functions \(f \in \mathcal{F}_s \subseteq \mathcal{F}\), a domain-specific safety constraint \(s_{fm} \le S_f^{\max}\). What \(s_{fm}\) measures depends on the task. For video anomaly detection, a natural choice is FNR [4]; for text generation, it could be a hallucination rate [28]. The formulation can use any task-specific safety metric that is measured at the model level and constrained by an upper bound. Table ¿tbl:tab:notation? summarizes the notations.

Let \(\mathcal{M}\) be the set of candidate models and \(\mathcal{D}\) the set of deployment modes. We distinguish four modes by how inference is accessed and where it runs (Table ¿tbl:tab:modes?). Pricing varies enormously [29]–[33]: among closed-source APIs alone, GPT-4o charges $2.50/M input tokens while Gemini-2.5-Flash charges $0.075/M, a 33\(\times\) gap. Open-source hosted APIs are cheaper and allow fine-tuning. Cloud GPU rental charges by the hour regardless of utilization; the agency can load any open-weight model using an inference engine ( e.g., vLLM [34] or SGLang [35]). On-premise GPUs incur amortized purchase and energy costs but provide full data control.

3pt

To compute the cost model, we reduce the four modes into two cost families: per-query variable cost (both API types) and fixed infrastructure plus variable energy (cloud rental and on-premise). We write \(\mathcal{D} = \{\textsf{closed api}, \textsf{open api}, \textsf{gpu-rent}, \textsf{gpu-own}\}\). Let \(\mathcal{A}_f \subseteq \mathcal{M} \times \mathcal{D}\) denote the feasible candidate set for function \(f\). Each pair \((m,d)\) is characterized by quality \(q_{fm}\), safety metric \(s_{fm}\), latency \(\ell_{fmd}\) (for \(f \in \mathcal{F}_s\)), and memory \(\mathrm{mem}_m\). For GPU modes, the required device count is \(n_{md}=\lceil \mathrm{mem}_m/\mathrm{cap}_d\rceil\).

3.2 Cost Model↩︎

Monthly cost consists of three components: variable API charges, fixed hardware costs, and, for owned hardware, variable energy cost.

API cost (\(d = \textsf{(closed/open) api}\)). Monthly cost is proportional to query volume and token length: \[C^{\text{api}}_{fm} = \lambda_f \cdot H \cdot \bigl(\tau^{\textsf{in}}_f \cdot p^{\textsf{in}}_m + \tau^{\textsf{out}}_f \cdot p^{\textsf{out}}_m\bigr) \label{eq:api95cost}\tag{1}\] where \(\tau^{\textsf{in}}_f, \tau^{\textsf{out}}_f\) are average input/output tokens per query and \(p^{\textsf{in}}_m, p^{\textsf{out}}_m\) are the model provider’s per-token prices.

GPU hardware cost (shared). For \(d \in \{\textsf{gpu-rent}, \textsf{gpu-own}\}\), hosting model \(m\) requires \(n_{md}\) devices. Monthly infrastructure cost reads: \[C^{\text{hw}}_{md} = \begin{cases} R_d \cdot n_{md} \cdot H & d = \textsf{gpu-rent} \\[4pt] P_d \cdot n_{md} \,/\, T_d & d = \textsf{gpu-own} \end{cases} \label{eq:hw95cost}\tag{2}\] Importantly, \(C^{\text{hw}}_{md}\) is charged once per model loaded.

Energy cost (\(d = \textsf{gpu-own}\) only). Operational energy per function is given by: \[C^{\text{energy}}_{fmd} = \text{TDP}_d \cdot n_{md} \cdot \mu_{fmd} \cdot H \cdot p_e \label{eq:energy95cost}\tag{3}\] where \(\mu_{fmd}=\min(\lambda_f \ell_{fmd}/3600,1)\) is the fraction of time the GPU actively processes queries for function \(f\), assuming sequential inference.

3.3 FMDP Formulation↩︎

We formulate FMDP as a mixed-integer program. Binary variable \(x_{fmd}\) assigns function \(f\) to model–deployment pair \((m,d)\), and \(y_{md}\) indicates whether model \(m\) is loaded on GPU infrastructure \(d\).

\[\begin{align} \min_{x, y} \;\; & \underbrace{ \sum_{f} \sum_{\substack{(m,d) \in \mathcal{A}_f \\ d = \textsf{(closed/open) api}}} \!\!\! C^{\text{api}}_{fm}\, x_{fmd} }_{\text{API token cost}} \;+\; \underbrace{ \sum_{m} \sum_{d \neq \textsf{api}} \!\!\! C^{\text{hw}}_{md}\, y_{md} }_{\text{fixed hardware cost}} \notag \\ & \;+\; \underbrace{ \sum_{f} \sum_{\substack{(m,d) \in \mathcal{A}_f \\ d = \textsf{gpu-own}}} \!\!\! C^{\text{energy}}_{fmd}\, x_{fmd} }_{\text{variable energy cost}} \tag{4} \\[4pt] \text{s.t.} \;\; & \textstyle\sum_{(m,d) \in \mathcal{A}_f} x_{fmd} = 1 \quad \forall\, f \tag{5} \\ & \textstyle\sum_{(m,d) \in \mathcal{A}_f} q_{fm} \cdot x_{fmd} \geq q_f^{\min} \quad \forall\, f \tag{6} \\ & \textstyle\sum_{(m,d) \in \mathcal{A}_f} s_{fm} \cdot x_{fmd} \leq S_f^{\max} \quad \forall\, f \!\in\! \mathcal{F}_s \tag{7} \\ & \textstyle\sum_{(m,d) \in \mathcal{A}_f} \ell_{fmd} \cdot x_{fmd} \leq L_f \quad \forall\, f \tag{8} \\ & y_{md} \geq x_{fmd} \quad \forall\, f,\; (m,d),\; d \!\neq\! \textsf{api} \tag{9} \\ & \textstyle\sum_{m} n_{md} \cdot y_{md} \leq G_d \quad \forall\, d \!\neq\! \textsf{api} \tag{10} \\ & x_{fmd},\; y_{md} \in \{0,1\} \tag{11} \end{align}\]

Constraint 5 assigns exactly one model–mode pair per function. Constraints 6 –8 enforce quality floors, safety bounds and latency ceilings. Since exactly one \(x_{fmd}=1\) per function, these reduce to checking whether the selected pair meets each threshold; infeasible pairs can be pre-filtered from \(\mathcal{A}_f\). We keep the linear-sum form because it produces tighter LP relaxation bounds (Section 4). Constraint 9 links the two variable families: using model \(m\) on a GPU for any function forces \(y_{md}=1\). Constraint 10 ensures that GPU usage in each mode does not exceed \(G_d\).

4 Complexity and Solution Methods↩︎

4.1 NP-Hardness↩︎

4.2 LP Relaxation Bound↩︎

Relaxing the binary variables to \([0,1]\) gives a polynomial-time LP lower bound. Without the GPU-capacity constraint 10 and the linking constraints 9 , the problem decomposes into independent per-function selections whose LP relaxation is integral. With shared GPU capacity and fixed-charge loading variables, tightness is no longer guaranteed; we therefore use the LP optimum as a lower bound to certify heuristic solution quality.

4.3 Greedy Heuristic↩︎

Algorithm 1 is a two-phase greedy heuristic cheap enough to re-run whenever prices change. Phase 1 scans the functions in arbitrary order and assigns each to its cheapest feasible (model, mode) pair given the models already loaded; a model loaded once serves later functions at no extra hardware cost, matching the fixed-charge structure of 4 . Phase 2 runs only if the GPU budget \(G_d\) is exceeded. It first computes the cheapest feasible API fallback cost for each GPU-assigned function \(\hat{c}_f\), then repeatedly switches the function with the smallest repair ratio \(r_f = \Delta_f / g_f\) to its fallback, where \(\Delta_f = \hat{c}_f - \mathrm{cost}_{fmd}\) is the resulting cost increase and \(g_f\) is the number of GPU devices released. A function sharing its model with others frees no devices, so we set \(g_f = 0\) and \(r_f = +\infty\) rather than divide by zero. If all \(r_f=+\infty\), the heuristic switches the function with the smallest \(\Delta_f\), reducing the number of users of at least one loaded model and eventually releasing capacity. The loop ends once the budget is met. The Worst-case complexity is \(O(F \cdot |\mathcal{A}|_{\max} + F^2)\): candidate lists are scanned once in Phase 1 and once for \(\hat{c}_f\), and Phase 2 performs repair iterations of at most \(F\), up to \(F\) functions.

5 Illustrative Case Study↩︎

We use an illustrative TMC case study to demonstrate the FMDP. Scenario details are demonstrated as follows. In practice, agencies would replace the illustrative entries in Table ¿tbl:tab:catalog? with measurements from their own models and workloads.

5.1 Scenario Design↩︎

We consider a TMC with \(\mathcal{F}{=}5\) functions and illustrative requirements (Table 1), reflecting the operational scope described in the NextGen TMS briefing [2]: video anomaly detection (f1), traffic state estimation (f2), incident report generation (f3), signal timing advisory (f4), and traveler information (f5). Function f1 is safety-critical: it monitors CCTV feeds for wrong-way drivers, pedestrians on freeways, and road safety, and we assume a FNR bound of \(s_{f1} \leq 0.10\). Query rates range widely, from 200 queries/hour for f1 (one frame every 18 s) down to just 3 incidents/hour for f3.

3pt

We assemble a set of candidates per function from the deployment modes in Table ¿tbl:tab:modes?; Table ¿tbl:tab:catalog? lists the full catalog. Quality and FNR scores are illustrative but align with published performance tiers: frontier closed-source VLMs outperform open-source VLMs, which in turn outperform classical detectors [3], [4], [19], [42]; text-function quality (f2–f5) follows the same ordering, consistent with LLM-as-judge evaluations [43] and public leaderboards (e.g., [44]). We assume that the relative performance of these models on TMC tasks mirrors their ranking on public benchmarks.

API costs are computed from Eq. 1 . Because providers use different tokenizers, the same prompt results in different token counts and different per-query costs \(c_q\). For vision queries, GPT-4o encodes a 720p CCTV frame as \({\sim}\)​1,105 image tokens [11]; Qwen2.5-VL tiles the same frame into \({\sim}\)​1,000 tokens [37]. On-premise costs follow Eqs. 2 –3 . We assume two agency-owned RTX 4090 GPUs ($1,600 MSRP [39], 36-month amortization, 450 W TDP, $0.12/kWh), a workstation-class configuration without data-center infrastructure, for 48 GB total on-premise VRAM.

3pt

5.2 Results and Analysis↩︎

We implement Algorithm 1 in Python on an Apple MacBook Pro (M4, 24 GB RAM). Experiment 1 presents the FMDP portfolio, while Experiment 2 studies API on-premise break-even points.

Experiment 1: Optimal portfolio vs.baselines. We evaluate five single-tier baselines (Table ¿tbl:tab:results?), each forcing every function into one deployment mode: All-Closed-Frontier picks the highest-quality closed model per function; All-Closed-Budget picks the cheapest feasible closed model; All-Open-Source-API uses hosted open-source models throughout; All-On-Premise runs everything locally; and Classical-Only relies on rule-based or classical-ML methods.

FMDP assigns f1, f2, f4, and f5 to hosted open-source APIs and routes only f3 to GPT-4o-mini, because no open-source option meets its quality floor. The resulting portfolio costs $34/mo with zero GPU usage. At current open-source API prices ($0.20/M tokens for vision), f1’s 144,000 monthly queries cost even less via API than on-premise hardware ($29 vs.$46/mo).

Three baselines are infeasible. All-On-Premise requires 63 GB of GPU memory, exceeding the 48 GB budget. Classical-Only and All-Open-Source-API fail the quality threshold for f3 (\(q{=}.60\) and \(.78\) vs.\(q^{\min}{=}.80\)). Thus, the all-open-source portfolio nearly matches FMDP in cost but does not satisfy all constraints. FMDP is feasible by assigning f3 to GPT-4o-mini at a marginal cost of $1/mo, while keeping the other functions on open-source APIs. The two feasible baselines are closed-source and exceed $1,100/mo, driven by f1’s sustained vision queries via GPT-4o. FMDP’s $34/mo portfolio is thus, 97% below the cheapest feasible baseline.

3pt

Experiment 2: API vs.on-premise break-even points. On-premise deployment becomes cheaper than a cloud API once the query rate exceeds \[\lambda^* \;=\; \frac{C^{\text{hw}}_{md}}{H \cdot c_q} \label{eq:tipping}\tag{12}\] where \(c_q\) is the per-query API cost and \(C^{\text{hw}}_{md}\) is the monthly hardware amortization. Table ¿tbl:tab:tipping? evaluates this for a single RTX 4090 at $44/mo.

At $0.20/M tokens, the open-source vision API (Qwen2.5-VL) breaks even at 309 q/h, above f1’s 200 q/h, so FMDP keeps f1 on the API. Adding cameras or raising the frame rate beyond one frame every 12 s would cross this threshold and shift f1 on-premise. GPT-4o, on the contrary, breaks even at only 8 q/h due to its higher per-query cost ($0.0077): for any sustained vision workload beyond that rate, owning a GPU beats paying closed-API vision prices. Text workloads are far below their break-even points, 617 q/h for GPT-4o-mini and 2,684 q/h for Llama-3.1-8B. Therefore, no text function in this scenario justifies local hardware. These thresholds move with API pricing, which changes frequently: doubling the vision price to $0.40/M halves \(\lambda^*\) to 154 q/h, below f1’s rate, and the optimal assignment shifts on-premise.

3pt

6 Discussion↩︎

Although the case study is small, it illustrates why FM deployment for TMCs should be treated as a portfolio problem rather than a set of independent model choices. No single deployment tier can be both feasible and cost-minimal. The all-open-source API strategy nearly matches FMDP’s monthly cost but fails the quality requirement for incident report generation, while the cheapest feasible all-closed-source strategy exceeds $1,100/mo because sustained vision API calls dominate cost. FMDP identifies the narrow point where closed-source capability is needed and keeps the remaining functions on lower-cost hosted open-source APIs. Break-even analysis shows that deployment decisions are sensitive to query volume and per-query price. At January 2026 prices, the open-source vision API remains cheaper than local deployment at 200 q/h, but this would reverse if camera sampling rates, the number of streams, or API prices increase. Text functions have much higher break-even rates, thus on-premise GPU deployment is not cost effective for the small volume reporting and advisory workloads in our case study.

7 Conclusion↩︎

This paper introduced FMDP, a mixed-integer programming framework to jointly select foundation models and deployment modes across TMC functions subject to the constraints of quality, latency, safety, and shared GPU-capacity. We proved FMDP is NP-hard and proposed a two-phase greedy heuristic. In an illustrative five-function case study, FMDP achieves a $34/mo portfolio, 97% below the cheapest feasible all-closed-source baseline, by using hosted open-source APIs where possible and a closed-source API only where needed. The break-even analysis further identifies when on-premise GPU deployment becomes cost-effective.

Limitations and future work. The case study demonstrates the planning methodology rather than realistic operational performance: its quality, latency, and safety parameters are illustrative and should be replaced with agency-specific measurements prior to operational use. The current formulation assigns one model to each function and does not consider model cascading, price and demand uncertainty, or inter-function dependencies. Future work will extend FMDP to model cascading, multi-period planning, and validation with operational TMC data.

References↩︎

---

1. \(^{*}\)Equal contribution. Contact: xc557@cornell.edu.↩︎

2. \(^{1}\)Cornell University, Ithaca, NY, USA.↩︎

3. \(^{2}\)University of California, Berkeley, CA, USA.↩︎

4. \(^{3}\)The Hong Kong Polytechnic University, Hong Kong SAR.↩︎

5. \(^{4}\)University of Illinois Chicago, Chicago, IL, USA.↩︎