1 Introduction↩︎

Recent advances in large language models (LLMs) have shown that they are not only powerful language generators, but also excel in quantitative prediction tasks across domains such as healthcare [1], finance [2], and scientific analysis [3]. However, LLMs remain susceptible to hallucinations and overconfident errors [4], a limitation that is especially concerning in high-stakes domains such as healthcare and law [5], [6]. For instance, LLMs show promise for accurate carbohydrate estimation from meal descriptions [1], a task critical for deciding insulin dosage in patients with diabetes, where overconfident but incorrect carbohydrate estimates can lead to severe glycemic consequences. Deploying such models in practice therefore requires more than accurate point estimates — users in safety-critical settings need to know not only what the model predicts, but also when to trust that prediction in order for humans to intervene when necessary [7], [8].

While a growing body of work studies confidence estimation for LLMs in classification and open-ended generation tasks [9], it remains largely underexplored for numerical prediction in language-based settings, where existing work predominantly focuses on producing accurate point estimates without considering reliable confidence signals [10]–[13]. To address this gap, we study language-based quantitative prediction with confidence estimation, where an LLM generates numerical estimates through natural language reasoning, accompanied by confidence signals that reflect prediction reliability.

Specifically, we draw a novel connection between loss prediction for uncertainty estimation [14], [15] and actor–critic reinforcement learning fine-tuning for LLMs [16], [17]. Prior work characterizes a model’s predictive uncertainty at a given input as the expected loss incurred by the predictor at that point, making estimation of this quantity a loss prediction problem [14], [15]. In other words, identifying when a prediction is unreliable corresponds to identifying when the model is likely to incur high error. This perspective naturally connects to the actor–critic framework of PPO [16], where the critic learns a value function that estimates the expected return from a given state (Equation 3 ): when the reward is defined as a function of prediction error, the critic’s value function converges to an estimate of the actor’s expected error — realizing loss prediction as a natural byproduct of RL training rather than a post-hoc procedure.

Building on this, we introduce CARE-PPO (Confidence-Aligned Reward for Estimation with PPO), a reinforcement learning framework for language-based quantitative prediction that jointly trains an actor to generate accurate numerical estimates through natural language reasoning and a critic whose value function serves as a confidence estimator aligned with prediction error. We design the reward as a monotonic function of prediction error, serving a dual purpose. First, it provides a dense, error-aware training signal that captures degrees of correctness and guides the actor toward accurate numerical predictions. Second, because the critic is trained to predict returns under this reward, it learns a value function aligned with prediction quality: high values correspond to low expected error, while low values correspond to high expected error. This allows the critic to naturally serve as a confidence estimator without requiring explicit confidence supervision.

We illustrate this framework in Figure 1, and summarize our main contributions below.

• Problem and insight. We study language-based quantitative prediction with confidence estimation and establish a connection between loss prediction theory and actor–critic RL fine-tuning: when the reward is aligned with prediction error, the PPO critic naturally learns to estimate the actor’s expected loss, enabling reliable confidence estimation without explicit supervision.

• CARE-PPO. We propose CARE-PPO, a PPO-based actor–critic framework with a Confidence-Aligned Reward for Estimation. By defining the reward as a monotonic function of prediction error, CARE-PPO provides dense error-aware feedback to the actor while inducing the critic to serve as an error-aligned confidence estimator at inference time.

• Comprehensive evaluation. We evaluate CARE-PPO on two real-world quantitative prediction tasks spanning healthcare and finance across Qwen-3 4B and 8B models [18], studying both Gaussian and linear reward formulations. CARE-PPO achieves competitive prediction performance while substantially improving confidence alignment over logit-based and verbalized baselines.

• Robustness and task awareness. We show that CARE-PPO’s confidence-alignment gains persist under realistic out-of-distribution settings, including linguistic and domain shifts. We further show that CARE-PPO reduces task-specific overfitting on general instruction-following queries, helping preserve general-purpose behavior beyond the target prediction setting.

2 Related Work↩︎

2.0.0.1 LLMs for language-based quantitative prediction.

Prior work on applying LLMs to language-based quantitative prediction has primarily followed three directions: (i) using LLM-derived representations as fixed input features for downstream models [10], [19], (ii) adapting LLMs through task-specific fine-tuning to directly produce numerical predictions [11], [12], and (iii) employing in-context learning to perform zero-shot or few-shot numerical estimation from natural-language prompts [13], [20]. While feature-based approaches can be effective, they treat the LLM primarily as an encoder and thus do not leverage its generative interface for interactive use or natural-language explanations. Meanwhile, both fine-tuning and in-context learning emphasize point estimates rather than rich language generation. In this study, we instead focus on fine-tuning LLMs for language-based quantitative prediction in a way that preserves their generative and conversational capabilities for transparency and interaction.

2.0.0.2 Confidence estimation in LLMs.

The growing adoption of LLMs has increased the need for reliable confidence estimates aligned with answer correctness. Prior work can be broadly categorized into black-box and white-box approaches, depending on whether internal model states are accessed [9].

Black-box approaches infer confidence using only model outputs. These include confidence verbalization methods [21]–[24], which often suffer from systematic overconfidence [25]–[27], as well as sampling-based techniques that estimate confidence from output variability using consensus heuristics [28], semantic diversity [29], or entropy-based measures [30]. White-box methods instead leverage internal signals such as token probabilities, entropy, or likelihood shifts [31]–[34], or train probes over hidden representations to predict correctness probability [35], [36]. Despite their differences, most approaches infer confidence post hoc rather than optimizing it during training. In contrast, CARE-PPO explicitly trains a critic to align predicted confidence with quantitative prediction error, enabling confidence estimates that are directly optimized during training rather than inferred after the fact.

2.0.0.3 Confidence-aware LLM fine-tuning.

At the intersection of confidence estimation and fine-tuning for language models, prior work primarily follows two directions. One line of work incorporates confidence or uncertainty signals as auxiliary rewards to improve downstream task performance, including speaker–listener frameworks [37] and reinforcement learning methods that derive rewards from self-confidence [38], [39]. A second line of work directly fine-tunes models to output calibrated confidence or uncertainty, with approaches including SFT methods that optimize verbalized confidence using tokenized Brier score losses [40], as well as methods that leverage human preference signals to guide confidence calibration [8]. These methods inherit the limitations of SFT, which can suffer from “catastrophic forgetting” and degradation of prior capabilities [41]. In contrast, our CARE-PPO framework jointly optimizes for correctness and confidence calibration, explicitly rewarding accurate predictions while learning well-calibrated uncertainty estimates. In addition, our approach yields reliable confidence estimates without the need for verbalization, or external supervision, while preserving full generative expressiveness and structured reasoning behavior.

3 Methodology↩︎

3.1 PPO for Language-Based Quantitative Prediction↩︎

Following prior work, we model language generation as a Markov Decision Process (MDP), where states correspond to token sequences and the initial state, \(s_0\), is given by the input prompt [42], [43]. At each timestep \(t\), the actor (policy) \(\pi_\theta\) selects an action \(a_t \sim \pi_\theta(\cdot \mid s_t)\), corresponding to the next generated token. Full details of the MDP formulation are provided in Appendix 7.

We optimize the policy using an actor–critic framework based on Proximal Policy Optimization (PPO) [16], which maximizes a clipped surrogate objective: \[\label{eqn:ppo} \mathcal{L}_{\text{PPO}}(\theta) = \mathbb{E}_t \left[ \min \left( r_t(\theta) A_t, \text{clip}\left(r_t(\theta), 1 - \epsilon, 1 + \epsilon \right) A_t \right) \right],\tag{1}\] where \(r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\text{old}}}(a_t \mid s_t)}\) denotes the importance sampling ratio between the current policy \(\pi_\theta\) and the behavior policy \(\pi_{\theta_{\text{old}}}\) used to collect trajectories, and \(\epsilon > 0\) is a clipping hyperparameter that constrains the magnitude of policy updates. We compute the advantage estimate \(A_t\) in Equation 1 using Generalized Advantage Estimation (GAE) [44]: \[{A}_t = \sum_{k=0}^{T-t-1} (\gamma \lambda)^k \, \delta_{t+k}, \label{eqn:gae}\tag{2}\] where \(\lambda \in [0,1]\) controls the bias–variance tradeoff of the estimator, \(\gamma \in (0,1]\) is the discount factor, \(\delta_t = R(s_t) + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)\) denotes the temporal-difference error, and \(R(s_t)\) is the reward received at state \(s_t\).

Here, the critic model \(V_\phi\) estimates the expected return from a given state \(s_t\) and is trained concurrently with the actor by minimizing the value loss: \[\mathcal{L}_{V}(\phi) = \mathbb{E}_t \left[ \left( V_\phi(s_t) - \hat{R}_t \right)^2 \right], \label{eqn:value95loss}\tag{3}\] where \(\hat{R}_t\) denotes the empirical return. Following standard RLVR setups, we employ a terminal reward structure with reward function \(R(s_T) \in \mathbb{R}\), such that rewards are provided only at the terminal state \(s_T\). Consequently, the return takes the form \(\hat{R}_t = \gamma^{T-t} R(s_T)\). In our setting, the terminal reward is computed based on the model’s final quantitative estimate produced after generating a reasoning trajectory in response to the input query.

Under this formulation, the actor is incentivized to select actions that improve this expected return, while the critic learns to estimate the expected outcome of a partial reasoning trajectory. In the following subsection, we describe our reward design, which aligns the critic’s value estimates with the actor’s numeric prediction error, enabling it to serve as an error-aligned confidence estimator.

3.2 CARE: Confidence-Aligned Reward for Estimation↩︎

We design the reward function \(R(s_T)\) to serve a dual purpose: (i) providing a dense, error-sensitive training signal that captures degrees of correctness and guides the actor toward accurate numerical predictions, and (ii) inducing a critic value function that can serve as an error-aligned confidence estimator at inference time.

3.2.0.1 Reward Definition

Let \(\hat{y}\) denote the actor’s final numerical prediction extracted from the generated trajectory, and let \(y\) denote the ground-truth value. We define the absolute prediction error as \(\xi = |\hat{y} - y|\) and the terminal reward as a monotonically decreasing function of prediction error: \[R(s_T) = h(\xi), \label{eqn:care}\tag{4}\] where \(h:\mathbb{R}_{\geq 0} \rightarrow \mathbb{R}\) is chosen so that smaller prediction errors receive larger rewards. This yields an error-sensitive training signal for the actor. Since the critic is trained to estimate returns under this reward, its value function becomes aligned with prediction quality, enabling it to serve as a confidence signal at inference time.

3.2.0.2 Reward Instantiations

We consider two instantiations of the reward function \(h\), corresponding to different ways of mapping prediction error to reward.

I: Linear reward: A natural choice is a linear function of prediction error:

\[h_{\mathrm{linear}}(\xi) = -\xi, \label{eqn:care95linear}\tag{5}\]

which directly preserves the magnitude of the error and provides a dense, unbounded training signal. While this formulation encourages accurate prediction by penalizing larger errors more strongly, it does not produce a normalized or bounded target for the critic, making its interpretation as a confidence score less direct.

II: Gaussian reward. To obtain a bounded and interpretable confidence signal, we also consider a scaled Gaussian kernel over the prediction error: \[h_{\mathrm{gauss}}(\xi) = \alpha\exp\left(-\frac{\xi^2}{2\sigma^2}\right), \label{eqn:care95gaussian}\tag{6}\] where \(\sigma\) controls the task-specific error tolerance and \(\alpha > 0\) is a reward scaling factor. This reward is smooth and bounded in \([0, \alpha]\), assigning high reward to small errors and decaying toward zero as error increases. The parameter \(\sigma\) has a natural interpretation as a tolerance scale: predictions within \(\pm \sigma\) of the ground truth receive high reward, while larger errors are penalized progressively.

We refer to models trained with the linear and Gaussian reward instantiations as CARE-PPO-Linear and CARE-PPO-Gaussian, respectively.

4 Experimental Setup↩︎

4.1 Tasks & Datasets↩︎

We evaluate our approach on two real-world quantitative estimation tasks spanning healthcare and finance, both requiring free-form natural language inputs to produce numerical predictions.

4.1.0.1 Nutrition Estimation

We study carbohydrate estimation from natural-language meal descriptions, as illustrated in Figure 1, motivated by its importance for diabetes management [45]. We generate natural-language meal descriptions from both individual food items and combinations of food items derived from the FoodData Central (FDC) repository [46] using GPT-4o-mini (see Appendix 8.1 for details and examples). The resulting dataset contains 26,549 training examples and 6,226 validation examples. For testing, we use the NutriBench benchmark [1], which consists of 4,826 examples.

4.1.0.2 Product Price Prediction

Product price prediction from textual product descriptions is evaluated using the Amazon Reviews dataset [47]. For our primary experiments, we focus on the Appliances category, where each input is constructed from product metadata fields (e.g., name, category, rating, and descriptive attributes). Full preprocessing details and examples are provided in Appendix 8.2. The resulting dataset contains 16,082 training, 946 validation, and 3,784 test examples.

4.2 Baselines↩︎

To evaluate the CARE-PPO framework and its critic-based confidence estimation, we compare against several training and confidence-estimation baselines.

4.2.0.1 Training Methods:

• Base Model: The instruction-tuned Qwen-3 4B and 8B models, without task-specific fine-tuning

• SFT: Models fine-tuned via next-token prediction on task-specific chain-of-thought data. For nutrition estimation, we construct rule-based reasoning traces that decompose meal descriptions into item-level carbohydrate estimates and aggregate them. For price prediction, we follow prior work on reasoning distillation [48], [49] by generating reasoning traces using a stronger teacher model (GPT-4o-mini). Examples of reasoning traces are provided in Appendix 8.

• PPO-Binary: A PPO baseline using a binary correctness reward \(h_{\mathrm{binary}}(\xi) = \mathbb{I}[\xi \leq \tau]\), where \(\tau\) is a task-specific tolerance threshold. This mirrors reinforcement learning from verifiable rewards (RLVR) setups [50] and serves as a direct comparison to evaluate whether error-sensitive reward shaping is necessary for language-based quantitative prediction.

4.2.0.2 Confidence Estimation Baselines

• Answer Probability: We calculate the mean probability of the generated tokens in the prediction string as a measure of model confidence [23].

• EntropyInv: We derive a confidence metric from the average per-token entropy (\(H(s) = -\sum p \log p\)) of the generated sequence, a commonly used measure of uncertainty [33]. We compute entropy over normalized top-20 token probabilities at each decoding step. To ensure comparability, we further normalize the average entropy by the maximum entropy \(H_{max}\) and define confidence as the complement: \(C_{ent} = 1 - \frac{\bar{H}}{H_{max}}\). We call this method EntropyInv.

• Verbalized Confidence: Prior work has shown that verbalized confidence methods can outperform logit-based approaches for reinforcement learning–fine-tuned models [23]. We evaluate two methods, (i) Verb. 1S (one-stage), where the model is prompted to produce both the prediction and a confidence estimate in a single response, and (ii) Verb. 2S (two-stage), where the model first produces its prediction and is subsequently prompted to assign a confidence score [23].

4.3 Evaluation Metrics↩︎

4.3.0.1 Quantitative Estimation Performance

We evaluate quantitative prediction accuracy using the mean absolute error (MAE) between model predictions and ground-truth values, where lower values indicate better performance.

4.3.0.2 Confidence Alignment

We evaluate how well estimated confidence reflects prediction error using the Area Under the Sparsification Error (AUSE) [51] for assessing the utility of confidence estimates for selective prediction. AUSE is computed as the area between a sparsification curve, obtained by progressively removing predictions in order of lowest confidence and measuring the error of the remaining set, and an oracle curve obtained by removing predictions in order of highest true error. A lower AUSE indicates that confidence more effectively ranks predictions by reliability.

4.4 Training Details↩︎

Both actor and critic are initialized from instruction-tuned Qwen-3 models (4B and 8B). We generate 8 rollouts per prompt and use GAE with \(\gamma=\lambda=1.0\). Actor and critic learning rates are set to \(10^{-6}\) and \(10^{-5}\), respectively. Models are trained for 5 epochs using VERL [52] on a single 8-GPU node.

For SFT, we select the checkpoint with the lowest validation MAE. For PPO, we select the actor checkpoint with the lowest validation MAE and the corresponding critic with the best AUSE.

For the Gaussian reward formulation \(h_{\mathrm{gauss}}\), we set \(\sigma = 10\), aligning with the error scale in carbohydrate estimation, where prior work considers errors within approximately \(7.5\)g to be a conservative acceptable range [1]. Under this parameterization, moderate errors (e.g., \(\xi \approx 7.5\)g) correspond to relatively high confidence (\(\approx 0.75\)), while large errors (e.g., \(\xi \approx 30\)g) map to near-zero confidence. We apply the same \(\sigma\) for price prediction, as both tasks operate over comparable output ranges in our datasets. We evaluate sensitivity to moderate variations in \(\sigma\) by conducting an ablation with \(\sigma \in \{8, 9, 10, 11, 12\}\) using Qwen3-4B on the nutrition estimation task (Section 5.4), and find that performance remains relatively stable across nearby values, particularly for confidence alignment. We use a reward scaling factor of \(\alpha=8\). Since the confidence score is obtained by normalizing the critic value by \(\alpha\) (Equation 6 ), this choice does not affect the resulting confidence scale.

We set the correctness threshold for \(h_{\mathrm{binary}}\) to \(7.5\)g for carbohydrate estimation [1] and 5 USD for price prediction, reflecting moderate task-specific error tolerances. When no extractable prediction is produced, we assign the minimum reward value for each formulation: \(0\) for the bounded rewards \(h_{\mathrm{gauss}}\) and \(h_{\mathrm{binary}}\), and \(-500\) for the unbounded reward \(h_{\mathrm{linear}}\), chosen based on the upper range of prediction errors observed on the base model validation sets.

5 Results↩︎

5.1 CARE-PPO Jointly Improves Quantitative Accuracy and Confidence Alignment↩︎

Table 1 reports quantitative prediction performance (MAE) across all training methods, tasks, and model scales. Figure 2 shows confidence alignment results (AUSE) for each confidence estimation method paired with each training approach. Since the Verb-1S method uses a different prompt that generates a new prediction alongside the confidence estimate, its MAE may differ slightly; we report the lowest MAE per method, with full results provided in Appendix 10.1.

Across both tasks and model scales, both reward variants of CARE-PPO (CARE-PPO-Linear and CARE-PPO-Gaussian) substantially reduce MAE relative to the base model and consistently outperform PPO-Binary. This improvement over PPO-Binary suggests that continuous, error-sensitive rewards provide a more effective training signal than binary correctness rewards, which discard error magnitude information. While SFT achieves lower MAE, it comes at the cost of poor confidence alignment: SFT’s confidence estimates show weak alignment with prediction error across both tasks, regardless of the confidence estimation method used.

5pt

In contrast, as shown in Figure 2, CARE-PPO consistently produces significantly better-aligned confidence estimates, with the critic emerging as the strongest confidence estimator across all methods, models, and settings. We visualize the corresponding sparsification curves used to compute AUSE in Appendix 10 (Figure 5). Even for PPO-Binary, where the reward provides no information about error magnitude, the critic achieves lower AUSE than all verbalized and logit-based baselines, supporting our hypothesis that the actor–critic training dynamic naturally encourages the critic to learn a meaningful confidence signal even from a discretized reward. When the reward is directly aligned with prediction error, as in CARE-PPO-Linear and CARE-PPO-Gaussian, this effect is further amplified, suggesting that closer alignment between reward and prediction loss leads to more reliable confidence estimation. Section 5.3 additionally visualizes how critic output distributions evolve throughout training and progressively align with the underlying reward structure.

5.2 CARE-PPO is Robust in Out-of-Distribution Settings↩︎

We evaluate CARE-PPO under two real-world out-of-distribution settings: linguistic shift in nutrition estimation and domain shift in product price prediction.

For nutrition estimation, we test on multilingual meal descriptions, reflecting realistic scenarios where users describe foods in different languages with culturally specific terms and dietary conventions [53]. We construct this test set from WHO dietary records [54] spanning Spanish, Italian, Portuguese, Tagalog, and Amharic, using GPT-4o-mini to generate natural-language meal descriptions from the food records (Appendix 8.1). We retain paired local-language and English descriptions from the source records rather than relying on automated translation, preserving official terminology and reducing translation-induced artifacts.

For price prediction, users may query about products across a broad range of categories. Thus, to assess generalization under domain shift, we additionally use the Electronics category from the Amazon Reviews dataset [47] as an out-of-distribution test set, randomly sampling 4,000 examples.

5pt

5.2.1 Linguistic Shift in Nutrition Estimation↩︎

Table 2 (left) reports MAE under linguistic shift, where models trained on English meal descriptions are evaluated on multilingual queries spanning five languages. Notably, CARE-PPO achieves stronger MAE performance than SFT under multilingual shift, suggesting that PPO-based approaches may learn representations that generalize more robustly across languages. Additional analysis in Table 5 in the Appendix further shows that PPO-based methods exhibit smaller MAE gaps between parallel English and non-English queries describing the same meal than SFT and the base model. Simultaneously, Figures 3 (a) and (b) demonstrate that critic-based confidence estimates consistently achieve substantially lower AUSE than verbalized and logit-based confidence estimation methods across both model scales, demonstrating strong robustness under linguistic shift.

5.2.2 Domain Shift in Price Prediction↩︎

Table 2 (right) reports MAE, while Figures 3 (c) and (d) show the AUSE of confidence estimates under domain shift, where models trained on Appliances are evaluated on unseen Electronics products. Confidence alignment degrades substantially for the base model and SFT, whereas critic-based confidence estimates learned through PPO (both PPO-Binary and the CARE-PPO variants) remain substantially better aligned under shift while maintaining competitive MAE performance.

Overall, across both linguistic and domain shifts, critic-based confidence estimates learned through PPO remain better aligned, more stable, and more robust than verbalized or logit-based alternatives, suggesting that such confidence signals are more likely to remain reliable under the distribution shifts encountered in practical deployment settings.

5.2.3 CARE-PPO Preserves Task-Aware Behavior↩︎

In practice, users interact with LLMs in open-ended settings and may issue queries beyond the deployment use case, making it important to preserve general-purpose behavior even after specialization to a quantitative prediction task. For instance, a model fine-tuned for nutrition estimation should recognize when a query is unrelated to food and respond appropriately, rather than attempting to estimate carbohydrates. We therefore evaluate whether CARE-PPO exhibits more task-aware behavior on general-purpose queries outside the training distribution, specifically studying whether models incorrectly force task-specific predictions on unrelated inputs.

8pt

To evaluate this behavior, we test models on 500 randomly sampled queries from the instruction-following benchmark IFEval [55], which contains prompts unrelated to the target prediction tasks. We focus on nutrition estimation as it provides the clearest signal of task overfitting: interpreting an unrelated query as a meal description and producing a carbohydrate estimate is an unambiguous failure mode. In contrast, for price-trained models, certain open-ended prompts (e.g., itinerary planning or writing assistance) can plausibly be interpreted as requests for a service to be priced, making it harder to distinguish genuine overfitting from reasonable model behavior. We use GPT-4o-mini as an LLM-as-a-judge (LLMaJ) [56] to classify each response based on whether the model incorrectly produces a task-specific prediction or responds appropriately. We provide the LLMaJ prompt in Appendix 9.3.

Table 3 shows that SFT exhibits substantial task overfitting, frequently misinterpreting unrelated queries as meal descriptions. Representative qualitative examples of these behaviors are provided in Appendix 10.2.1. In contrast, all PPO-based methods substantially reduce this behavior, consistent with prior findings that RL fine-tuning generalizes more robustly beyond the training distribution than SFT [57]. Among PPO variants, PPO-Binary and CARE-PPO-Gaussian exhibit the lowest misinterpretation rates, while CARE-PPO-Linear exhibits notably higher rates. We hypothesize that this difference is partly attributable to reward design: because \(h_{\text{linear}}\) is unbounded, the penalty assigned when no extractable prediction is produced is significantly larger in magnitude than for the bounded formulations \(h_{\text{gauss}}\) and \(h_{\text{binary}}\) , potentially incentivizing the actor to always produce a formatted numerical output regardless of input relevance. Overall, these results suggest that PPO-based fine-tuning more effectively preserves general-purpose behavior beyond the target prediction setting, indicating better robustness in open-ended deployment settings.

5.3 Critic Distribution Evolution During Training↩︎

To better understand how critic-based confidence estimation emerges during training, we visualize the distribution of critic outputs throughout PPO optimization alongside the corresponding ground-truth reward distributions on the validation set. Figure 4 shows the evolution of the final-layer critic outputs across training checkpoints for PPO-Binary, CARE-PPO-Linear, and CARE-PPO-Gaussian trained for the nutrition estimation task with Qwen-3 4B as the backbone. Corresponding plots for the price prediction task are presented in Appendix 12.

Across all reward formulations, the critic progressively learns to produce confidence estimates that better reflect the structure of the underlying reward distribution. In particular, the critic distributions become increasingly shaped according to prediction quality over training, corresponding to improved confidence alignment. While PPO-Binary operates with a discretized reward signal, the critic nevertheless learns a meaningful ranking over prediction quality, supporting our hypothesis that actor–critic optimization naturally encourages confidence-aware representations even under sparse reward formulations.

5.4 Sensitivity Analysis with \(\sigma\) on CARE-PPO-Gaussian↩︎

r0.42

15pt

The Gaussian reward formulation in CARE-PPO-Gaussian introduces a tolerance parameter \(\sigma\) that controls the smoothness of the reward landscape around the target value. To evaluate sensitivity to this hyperparameter, we conduct an ablation over \(\sigma \in \{8, 9, 10, 11, 12\}\) using Qwen-3 4B on the nutrition estimation task. Results are shown in Table 4.

Overall, we find that performance remains relatively stable across nearby values of \(\sigma\), particularly in terms of confidence alignment. While \(\sigma=10\) achieves the best overall balance between quantitative prediction accuracy and confidence calibration, neighboring values produce comparable AUSE performance, suggesting that CARE-PPO-Gaussian is reasonably robust to moderate variations in the reward tolerance parameter.

6 Conclusion↩︎

We introduced CARE-PPO, a reinforcement learning framework for language-based quantitative prediction that draws a novel connection between loss prediction for uncertainty estimation and actor–critic RL fine-tuning. By defining rewards as functions of prediction error, CARE-PPO enables the critic to learn value functions aligned with prediction quality, allowing the critic to serve as an effective confidence estimator at inference time. Across quantitative prediction tasks in healthcare and finance, CARE-PPO achieves competitive prediction accuracy while substantially improving confidence alignment relative to verbalized and logit-based confidence estimation methods. We further showed that these gains remain robust under realistic distribution shifts and extend to improved task-aware behavior on out-of-task queries.

We believe confidence-aware language-based quantitative prediction represents a promising research direction. While CARE-PPO relies on PPO fine-tuning, confidence estimation emerges as a byproduct of actor–critic training rather than requiring a separate confidence-learning stage, calibration procedure, or auxiliary model. Latency analysis shows that critic-based confidence estimation introduces only a small inference overhead (Appendix 11), suggesting that it can be incorporated into deployment with minimal additional cost. Our experiments are currently limited to models up to 8B parameters and two application domains due to computational constraints. Future work will explore scaling these results to larger language models and broader quantitative prediction settings, investigating theoretical connections between reward design and calibration, and integrating confidence estimates more directly into the generation process to more tightly couple prediction and uncertainty estimation.

7 PPO Framework↩︎

We follow prior work in modeling language generation as a Markov Decision Process (MDP) defined by the tuple \(\left(\mathcal{S}, \mathcal{A}, \mathcal{P}, R, \gamma\right)\) [42], [43]. Each state \(s_t \in \mathcal{S}\) represents a sequence of tokens drawn from a fixed vocabulary \(\mathcal{V}\). The initial state \(s_0 = (x_1, x_2, \dots, x_L)\), where \(x_i \in \mathcal{V}\), corresponds to the input prompt, including the system instruction and user query. At each timestep \(t \in \{0, \dots, T-1\}\), the actor (policy) model \(\pi_\theta\) selects an action \(a_t \sim \pi_\theta(\cdot \mid s_t)\), where \(a_t \in \mathcal{V}\) corresponds to generating the next token. The environment transition function \(\mathcal{P}\) is deterministic, forming the next state by appending the selected action to the current sequence, \(s_{t+1} = (s_t, a_t)\). Generation terminates at a terminal state \(s_T\) upon emitting an end-of-sequence token or reaching a maximum sequence length \(T\). The discount factor \(\gamma \in (0,1]\) controls the contribution of the terminal reward to earlier states.

8 Data Construction and Processing↩︎

8.1 Nutrition Estimation↩︎

We construct training data for the nutrition estimation task using food composition entries from FoodData Central (FDC) [46], the food composition database maintained by the United States Department of Agriculture (USDA). Specifically, we first obtain food names along with their corresponding nutritional content for varying portion sizes.

To convert these structured food entries into natural-language training examples, we prompt GPT-4o-mini to generate realistic free-form meal descriptions from the food items. Concretely, we provide the model with food names and portion sizes from FDC entries and instruct it to synthesize meal descriptions in natural language.

In addition to constructing descriptions directly from individual food entries, we also generate meal descriptions containing combinations of two food items to better reflect realistic meals consisting of multiple components. The prompts used for meal description generation are shown in Box [box:nutrition-generation-prompts].

{
  "parts": [
    "smoothie, made with spinach, banana, almond milk, protein powder"
  ],
  "units": [
    "1.0 glass"
  ]
}

{
  "description1": {
    "description": "I am having a glass of smoothie made with spinach,
                    banana, almond milk, and protein powder for lunch.",
    "items": "a glass of smoothie made with spinach, banana,
              almond milk, and protein powder"
  },
  etc
}

{
  "parts": [
    "Beverages, NESTEA, tea, black, ready-to-drink, lemon"
  ],
  "units": [
    "1.0 serving (8 fl oz)"
  ]
}

{
  "description1": {
    "description": "8 fl oz of NESTEA lemon black tea",
    "items1": "8 fl oz of NESTEA lemon black tea"
  },
  etc.
}

{
  "parts": [
    "pepperoni pizza"
  ],
  "units": [
    "1.0 slice"
  ]
}

{
  “description1”: {
    "description": "One slice of pepperoni pizza",
    "items1": "one slice of pepperoni pizza"
  },
  etc.
}

{
  "parts": [
    "Breakfast tart, lowfat"
  ],
  "units": [
    "1 Pop Tart"
  ]
}

{
  “description1”: {
    "description": "I have a low-fat Breakfast Tart.",
    "items1": "a low-fat Breakfast Tart."
  },
  etc.
}

[
  '43g garlic bread.',
  '100g grilled chicken with steamed vegetables.',
  '100g tomato soup.'
]

43g garlic bread, 100g grilled chicken with steamed vegetables, and
100g tomato soup.

[
  'For breakfast, I am having 136g banana.',
  'My lunch consists of 106g refrigerated whole wheat naan bread.'
]

For lunch, I am having 136g banana and 106g refrigerated whole wheat
naan bread.

Finally, we generate synthetic reasoning traces for supervised fine-tuning with chain-of-thought supervision. These reasoning traces are constructed using a rule-based procedure that (i) identifies individual food items and their portion sizes, (ii) estimates the carbohydrate content of each item, and (iii) aggregates these values to produce the final carbohydrate estimate. Representative examples of the synthetic reasoning traces are provided in Example [box:nutrition-examples].

Further, for out-of-distribution evaluation under linguistic shift, we construct a multilingual nutrition estimation test set using dietary records from the WHO Global Individual Food consumption data Tool (GIFT) [54], spanning Spanish, Italian, Portuguese, Tagalog, and Amharic. We use GPT-4o-mini to generate natural-language meal descriptions from the structured dietary records while preserving the original language of each record. The prompt used for multilingual meal description generation is provided in Box [box:multilingual-nutrition-prompt]. For the final test set, we randomly sample 100 queries from each language. We provide representative meal examples in each language in Example [box:multilingual-meal-examples].

8.2 Product Price Prediction↩︎

For the product price prediction task, we use product metadata from the Amazon Reviews dataset [47]. For our primary experiments, we focus on the Appliances category, which consists of 94.3K items. For each product, we construct a textual input by concatenating relevant product metadata fields (e.g., item name, category, user rating, and descriptive attributes). We filter the dataset to retain only items that contain at least one descriptive feature and have an associated price label, and we remove outliers whose prices fall above the 90th percentile.

For reasoning supervision in the SFT training setup, we additionally construct synthetic chain-of-thought reasoning traces using GPT-4o-mini, following prior work on reasoning distillation for language models [48], [49]. The prompt used for reasoning trace generation is provided in Box [box:price-reasoning-prompt], and representative examples are included in Example [box:price-examples].

9 Task Prompts↩︎

9.1 Nutrition Estimation↩︎

Box [box:nutrition-system-prompt] provides the system prompt used for the nutrition estimation task. Boxes [box:nutrition-verb1s-prompt] and [box:nutrition-verb2s-prompt] provide the prompts used for one-stage (Verb-1S) and two-stage (Verb-2S) verbalized confidence estimation, respectively.

9.2 Product Price Prediction↩︎

Box [box:price-system-prompt] presents the system prompt used for the product price prediction task. Boxes [box:price-verb1s-prompt] and [box:price-verb2s-prompt] present the prompts used for one-stage and two-stage verbalized confidence elicitation, respectively.

9.3 LLM-as-Judge↩︎

To evaluate task-aware behavior preservation, we use GPT-4o-mini as an LLM-as-a-judge (LLMaJ) to classify whether model responses to unrelated instruction-following queries appropriately answer the user request or incorrectly produce task-specific predictions. The judge prompt used for this evaluation is provided in Box [box:llmaj-prompt].

<query>
{query}
</query>

<response>
{response}
</response>

{{"category": "<category>", "reason": "<one sentence explanation>"}}

10 Detailed Experimental Results↩︎

3pt

10.1 Primary Evaluation Results↩︎

Table ¿tbl:tab:full95id95results? present the complete evaluation results on the nutrition estimation and product price prediction tasks, including all confidence estimation methods across model scales and training approaches. Corresponding sparsification curves are shown in Figure 5.

Consistent with the results discussed in Section 5.1, CARE-PPO substantially improves confidence alignment relative to verbalized and logit-based confidence estimation methods, with critic-based confidence estimation achieving the strongest overall AUSE performance across both tasks and model scales. While SFT achieves strong quantitative prediction performance in terms of MAE, its confidence estimates remain comparatively poorly aligned with prediction error. In contrast, CARE-PPO achieves competitive quantitative prediction performance while simultaneously producing substantially better-aligned confidence estimates.

10.2 Out-of-Distribution Results↩︎

Table ¿tbl:tab:full95ood95results? presents detailed out-of-distribution evaluation results for the linguistic-shift nutrition estimation task and the domain-shift product price prediction task. Corresponding sparsification curves are shown in Figure 6.

CARE-PPO maintains the strongest confidence alignment across all confidence estimation baselines and training methods, while achieving competitive MAE for price prediction and the best MAE for nutrition estimation. These results demonstrate that CARE-PPO is robust under both distributional and linguistic shifts, and provides more reliable predictions compared to baseline approaches in real-world settings where input distributions vary, and users may interact with models in different languages.

Table 5 further evaluates robustness to linguistic shift using \(\Delta\)MAE between parallel English and non-English queries describing identical meals. Across both model scales, CARE-PPO achieves the smallest performance gaps, suggesting stronger language-agnostic generalization and more stable quantitative prediction performance across multilingual inputs.

3pt

8pt

10.2.1 Examples of Task-Aware Behavior Preservation↩︎

To further evaluate whether task-specific fine-tuning degrades general-purpose instruction-following capabilities, we present representative model outputs on unrelated instruction-following queries for each training method across both Qwen3-4B and Qwen3-8B models. These examples qualitatively complement the quantitative IFEval results discussed in Section 5.2.3.

11 Inference Latency Analysis↩︎

We evaluate the inference overhead introduced by critic-based confidence estimation. Using 100 randomly sampled examples from the nutrition estimation test set, we measure end-to-end latency for both actor generation and critic confidence estimation. On average, the actor requires 3108.2 ms per sample, while the critic requires only 74.0 ms, corresponding to a 2.4% increase in total inference latency. This small overhead arises because the critic performs only a single forward pass on the completed response, whereas the actor requires multiple autoregressive decoding steps proportional to the output length. These results suggest that critic-based confidence estimation can be incorporated into deployment with minimal additional inference cost.

12 Critic Distribution Evolution During Training- Price Prediction↩︎

Figure 7 presents the evolution of critic output distributions throughout PPO training on the price prediction task with Qwen-3 4B as the backbone model, complementing the nutrition estimation analysis in Section 5.3. Consistent with the patterns observed there, the critic progressively learns to produce confidence estimates that better reflect the structure of the underlying reward distribution across all three reward formulations.

References↩︎