Keywords: Information Aggregation, Online Decision-Making, Calibration, Contextual Online Learning

1 Introduction↩︎

Probabilistic forecasts are increasingly used to guide decisions in settings such as weather prediction, medical diagnosis, financial investment, and policy evaluation. In these environments, a decision-maker often observes reports from multiple forecasters—experts, statistical models, or institutions—and must translate these reports into payoff-relevant actions. A natural minimal requirement on such forecasts is calibration: when a forecaster repeatedly reports probability \(p\), the event should occur with frequency approximately \(p\). Calibration is attractive because it is model-free and does not require the decision-maker to know the forecaster’s information structure.

Calibration, however, is not the same as informativeness [1]. A forecaster who always reports the unconditional frequency of an event may be perfectly calibrated but useless for choosing state-contingent actions. This limitation is especially salient when forecasts are evaluated one at a time. In many applications, the decision-maker observes several calibrated forecasts simultaneously. The joint pattern of these forecasts may contain information that is absent from any individual report. For example, two forecasts that are each only partially informative may, through their correlation with each other and with the state, jointly reveal substantially sharper information about the relevant posterior belief. The central question of this paper is whether, and how, a decision-maker can exploit such information using calibration alone.

This question is also about the appropriate benchmark for online decision-making with forecasts. A standard benchmark in prediction with expert advice is the optimal-in-hindsight benchmark, which compares the decision-maker with the decision rule based on a single expert’s reports—that is, the best mapping from one expert’s report space to actions—selected in hindsight. This benchmark is natural when the learner’s task is to select among experts. It is less compelling when the learner observes all forecasts and can aggregate them. If calibrated forecasts jointly encode decision-relevant information, then a decision rule that conditions on the full forecast profile may systematically outperform every rule that uses only one expert’s report. In such cases, the optimal-in-hindsight benchmark understates what is achievable from the information available to the decision-maker.

A simple example illustrates the force of aggregation. Consider a weather-prediction problem over \(T\) days. Expert 1 reports a probability of \(1/3\) in the first \(2T/3\) days and a probability of \(2/3\) in the remaining \(T/3\) days, while Expert 2 reports a probability of \(2/3\) in the first \(T/3\) days and a probability of \(1/3\) in the remaining \(2T/3\) days. See 1 for an illustration of this example. Suppose both experts are calibrated. Taken separately, each forecast partitions the data only coarsely. Taken together, however, the two calibration constraints identify a sharper posterior: the implied probability of rain is \(2/3\) in the first third, \(0\) in the middle third, and \(2/3\) in the last third. To see why this matters for decisions, suppose the decision-maker must choose whether to prepare for rain, which is optimal exactly when the probability of rain exceeds \(1/2\). Conditioning on the joint profile, the decision-maker prepares in the first and last thirds and refrains in the middle third. No rule based on a single expert’s report can replicate this: each expert reports \(1/3\) on a block of days that pools the middle third, where rain never occurs, with another third where rain occurs with probability \(2/3\), so any single-expert rule must take the same action across that entire block. The joint pattern of reports therefore reveals payoff-relevant information that is unavailable from either expert alone. This example shows why a benchmark based on the best individual expert can be too weak: the decision-maker’s real informational object is the forecast profile, not a single forecast.

The example is deliberately stark: the two calibration constraints uniquely determine the relevant posterior belief for each forecast profile. In general, calibration need not be so informative. The same joint distribution of calibrated forecasts may be consistent with many different mappings from forecast profiles to state probabilities. Thus, calibration can reveal useful restrictions on posterior beliefs without identifying them completely. This creates a basic ambiguity for decision-making: when several posterior mappings are consistent with all observed calibration constraints, which one should the decision-maker use?

We resolve this ambiguity by adopting a robust perspective. For each forecast profile, we consider the set of posterior probabilities that are consistent with the calibration of all experts. A decision rule maps forecast profiles into actions, and its value is evaluated against the worst posterior mapping in this feasible set. The robust max-min benchmark is the highest payoff that can be guaranteed using only the restrictions imposed by calibration. This benchmark captures the informational content of calibrated forecasts without requiring the decision-maker to know the experts’ signal structures, priors, or conditional independence relationships.

This benchmark has two useful interpretations. First, it is an aggregation benchmark: it allows the decision-maker to condition on the full profile of reports and therefore to exploit information contained in the joint pattern of forecasts. Second, it is a robustness benchmark: it does not assume that calibration identifies the true posterior, but instead asks what payoff is guaranteed uniformly over all posterior beliefs compatible with calibration. In this sense, the benchmark lies between two familiar extremes. It is more demanding than the optimal-in-hindsight benchmark, which restricts attention to one expert at a time, but it is generally less demanding than the Bayesian benchmark, which assumes knowledge of the full data-generating process.

Our first set of results studies this benchmark in the offline environment, where the full sequence of forecast profiles is fixed. We first show that the robust benchmark is a tractable object. Although the benchmark is defined as a max-min problem over decision rules and calibrated-consistent posterior mappings, calibration imposes only linear restrictions. By dualizing the inner minimization problem, we obtain an equivalent linear program. Thus, the benchmark can be computed in time polynomial in the number of forecast profiles, the number of actions, and the number of distinct expert reports (1). This tractability result is important because the benchmark is not merely conceptual: it can be evaluated and used as a practical performance criterion.

The offline analysis also clarifies the structure of optimal robust aggregation. A priori, one might think that the decision-maker should use correlated randomizations across different forecast profiles, because the calibration constraints couple the posterior probabilities assigned to different profiles. We show that allowing such correlation provides no gain: independent mixed strategies, chosen separately for each forecast profile, achieve the same robust value as arbitrary correlated strategies over mappings from forecast profiles to actions (1). This result justifies the simpler formulation of a decision rule as a collection of profile-contingent mixed actions.

We then compare the robust benchmark with two natural alternatives. Relative to the optimal-in-hindsight benchmark, robust aggregation is weakly stronger when forecasts are exactly calibrated, and remains stronger up to an error term proportional to the calibration error (2). This formalizes the intuition from the motivating example: a decision-maker who observes all forecasts should not be evaluated only against the best single expert. At the same time, the robust benchmark is generally weaker than the Bayesian benchmark, which assumes knowledge of the prior and the experts’ information structures. We show in 1 where this inequality is strict. The robust benchmark therefore occupies an intermediate position: it extracts all information guaranteed by calibration, but it does not attribute to the decision-maker information about the data-generating process that calibration alone does not reveal.

We next study whether robust aggregation can be learned online. This question is nontrivial because, under forecast-only feedback, the decision-maker observes the sequence of forecast profiles but neither the realized states nor the realized payoffs. Standard online-learning algorithms rely on outcome or payoff feedback to update decisions. Here, by contrast, learning must proceed entirely from the empirical distribution of forecasts. The relevant object to learn is not which action performed well in the past, but which robust decision rule is warranted by the calibration restrictions induced by the distribution of forecast profiles.

Our main online result shows that this is possible. We consider a forecast-only feedback environment in which forecast profiles are drawn from a fixed distribution, while the realized state sequence is otherwise unrestricted subject to calibration (2). We propose a plug-in robust aggregation algorithm: at each period, the decision-maker estimates the distribution of forecast profiles from past observations and solves a penalized version of the robust max-min problem for the estimated distribution. The penalty accommodates the fact that finite-sample empirical distributions may not exactly satisfy the calibration restrictions. We prove that this algorithm achieves sublinear robust regret. Hence, even without observing states or payoffs, the decision-maker can asymptotically attain the robust benchmark using only the observed sequence of calibrated forecasts. Finally, we show that sublinear regret with respect to the robust benchmark is not possible without the stochastic forecast profile assumption (see 4.3).

Note that the forecast-only model captures environments in which outcomes are delayed, unavailable, or not directly attributable to individual decisions. In other settings, the decision-maker observes the realized state after acting. This additional information permits a stronger benchmark. Under state feedback, the forecast profile can be treated as a context, and observing the state allows the decision-maker to evaluate the payoff of every action for that context. We therefore compare the learner with the best context-dependent decision rule in hindsight. A simple contextual Hedge algorithm, run separately for each forecast profile, achieves sublinear regret relative to this benchmark (3). Thus, the appropriate benchmark depends sharply on the feedback structure: forecast-only feedback calls for robust aggregation based on calibration, whereas state feedback permits the stronger contextual optimal-in-hindsight benchmark.

Taken together, these results provide a framework for decision-making with calibrated forecasts that separates three sources of value: the value of aggregating multiple calibrated reports, the value of robustness when calibration does not identify a unique posterior, and the value of feedback in online learning. Calibration alone is often viewed as too weak to support informative decisions. Our results show that, when multiple calibrated forecasts are observed jointly, calibration can nevertheless impose enough structure to support tractable and learnable aggregation rules that outperform standard best-expert benchmarks.

1.1 Related Work↩︎

1.1.0.1 Robust Forecast Aggregation

Our work contributes to the literature on robust forecast aggregation, which studies how to combine multiple forecasts into a single forecast when the aggregator does not know the underlying information structure. The most direct predecessor is [2], who introduce a regret-based criterion for evaluating aggregation schemes relative to the Bayesian benchmark and construct low-regret aggregation schemes in two-expert settings with either Blackwell-ordered or conditionally independent signals. [3] extend this line of work by developing an algorithmic framework for computing robust aggregation rules, with a primary focus on conditionally independent information structures. [4] study robust aggregation of correlated information from a decision-theoretic perspective and characterize robustly optimal strategies in binary-state, binary-action problems. In a similar spirit, [5] study forecast combination when each expert’s signal is understood individually but the correlation structure across experts is ambiguous. [6] identify conditions under which simple aggregation methods obtain nontrivial guarantees, and [7] show that allowing the principal to ask additional structured queries can overcome some impossibility results in robust forecast aggregation. More recently, [8] study prior-agnostic robust aggregation, in which the aggregator observes only the experts’ reports and knows neither the prior nor the underlying state space. A related strand relaxes the assumption that experts report truthful Bayesian posteriors, studying robust aggregation with adversarial experts [9] and with base-rate-neglecting experts [10].

Much of this literature studies Bayesian experts under specific restrictions on the information structures, priors, dependence, or elicitation protocols. In contrast, our work takes calibrated forecasters as primitives and studies the class of information structures induced by calibration. This leads to a different robustness problem: rather than assuming a particular dependence structure among signals, we ask how forecasts can be aggregated when calibration restricts but does not fully identify the underlying information structure. We further extend the analysis to online settings and design no-regret aggregation algorithms for this calibrated-forecaster benchmark.

1.1.0.2 Online Learning and Regrets

Our results are related to the literature on benchmarks and no-regret algorithms in online decision-making problem. The standard benchmark is the optimal-in-hindsight benchmark, which leads to the external regret, where the learner competes with the best fixed action, expert, or decision rule chosen in hindsight. No-external-regret guarantees can be achieved by Hedge [11], follow-the-regularized-leader [12], and online mirror descent [13] under full-gradient feedback, by EXP3 algorithms under adversarial bandit feedback [14], and by EXP4 in contextual bandit problems [14], [15].

Another benchmark is swap regret, which compares against arbitrary action mappings. Internal regret concerns pairwise action switches, while swap regret permits arbitrary action mappings. These stronger regrets are closely connected to calibration and correlated equilibrium [16], and can be achieved by regret-matching algorithms [17] or by reductions from external-regret minimization [18].

Our benchmark is the robustly optimal aggregation rule subject to the information structures induced by calibration, and is weakly stronger than the optimal-in-hindsight benchmark. More importantly, our setting differs from standard online learning because the learner observes only forecasts. With forecast-only feedback, payoff-based learning algorithms cannot directly evaluate experts’ reports or feasible actions, and therefore are not directly applicable. Our online algorithm is designed for this information constraint and attains no-regret guarantees with respect to the robust aggregation benchmark.

1.1.0.3 Calibration and Decision Guarantees

Our work is also related to the literature on calibration and its improvements. The classical calibration literature begins with [19]’s notion of calibrated probability assessment; [20] shows the limitations of deterministic self-calibration, whereas [21] establish that asymptotic calibration can be achieved through randomized forecasting procedures. However, [22] emphasize that calibration alone is not sufficient for useful prediction—sharpness, or informativeness, is also needed, which motivated refinements or improvements of calibration.

Multicalibration strengthens ordinary calibration by requiring calibration not only conditional on the predictor’s own forecast, but also over a rich collection of identifiable subpopulations [23]. Omniprediction builds on this idea from a decision-making perspective [24], and recent work shows a close connection between multicalibration and omniprediction [25], [26]. Our work shares with these two notions the view that calibration should be understood through its implications for downstream decision making. However, the object of study is different. Multicalibration and omniprediction primarily concern the construction of a single predictor whose forecasts are sufficiently informative for many downstream losses. In contrast, we take multiple calibrated forecasters as primitives and study how their reports can be robustly aggregated when the joint information structure is unknown. A related line of work studies calibration directly through its downstream decision value, including \(U\)-calibration [27] and swap-regret guarantees for downstream decision tasks [1]. These works concern how to produce or evaluate a single predictor so that downstream agents act well, whereas we study how a decision-maker aggregates several already-calibrated reports under an unknown information structure.

1.1.0.4 Statistical Forecast Pooling

A more closely related line of work studies model-based aggregation of multiple probability forecasts. [28] show that linear pools of calibrated forecasts are generally uncalibrated and lack sharpness, highlighting that aggregating calibrated forecasts is itself a nontrivial problem. [29] propose a simple logit model for combining expert forecasts and demonstrate strong empirical performance on geopolitical forecasting data. [30] study forecast aggregation in the partial information framework and derive an explicit Gaussian aggregator for a one-shot problem with two forecasters. These works either impose specific statistical models on the forecast-generating process, focus on the two-forecaster case, or primarily provide empirical aggregation methods. In contrast, our work takes calibrated forecasters as primitives and studies robust aggregation over the class of information structures induced by calibration, without assuming a parametric model or restricting attention to two experts.

1.1.0.5 Notation

Given a positive integer \(n\), let \([n]\) denote the set \(\{1,2, \cdots, n\}\). Given a vector \(v\in \mathbb{R}^n\), let \([v]_i\) denote its \(i\)-th element. Given a finite set \(S\), let \(|S|\) denote the number of its elements. Given a sequence of vectors \(v_1, v_2, \cdots, v_t\), we denote it by \(v_{1:t}\).

2 Model↩︎

Consider a dynamic decision-making problem \(\Gamma = (\Omega, \mathcal{A}, u, T)\) with a binary state space \(\Omega=\{0,1\}\) of nature, a finite action set \(\mathcal{A}\) of size \(|\mathcal{A}|=d\), a bounded payoff function \(u:\mathcal{A}\times \Omega\to [-U, U]\), and a finite horizon \(T\in\mathbb{N}_+\). At each day \(t\), the decision-maker observes the available forecasts, chooses an action \(a_t\in\mathcal{A}\), and the state \(\omega_t\in\Omega\) is then revealed or withheld depending on the feedback model. The decision maker’s payoff given any sequence of action-state pairs \(\{(a_t,\omega_t)\}_{t=1}^T\) is \[\begin{align} \sum_{t=1}^T u(a_t,\omega_t). \end{align}\]

2.0.0.1 Expert Forecasts

The decision-maker has access to \(n\) experts, each of whom makes forecasts about the probability of the event that the state of nature equals \(1\), i.e., \(\Pr(\omega=1)\). Let \(e_t^i\) be the forecast made by expert \(i\) at day \(t\), and let \(e_t = (e_t^1, \dots, e_t^n)\) be the entire forecast profile. Let \(E^i\) be the collection of distinct reports made by expert \(i\) over the time horizon \(T\), and let \(E\) denote their Cartesian product, i.e., \(E = E^1\times E^2\times \cdots \times E^n\). Denote the size of \(E\) by \(m\). In online sections, we take these finite forecast alphabets as known before play. In our model, we assume that all experts satisfy \(\varepsilon_T\)-calibration, where \(\varepsilon_T\) represents the calibration error. That is, for any expert \(i\in[n]\) and any \(e^i\in E^i\), we have \[\label{eq95calib95deter} \left\vert \frac{\sum_{t\leq T} \mathbf{1}\left[\omega_t = 1\right]\cdot \mathbf{1}\left[e^i_t = e^i\right]}{\sum_{t\leq T} \mathbf{1}\left[e^i_t = e^i\right]} - e^i\right\vert \le \varepsilon_T.\tag{1}\]

2.0.0.2 Online Decisions and Regrets

In this online decision problem, the decision maker chooses an action in each period based on the historical observations. In our model, we consider two types of feedback:

• forecast-only feedback: at any time \(t\leq T\), the decision maker only observes the forecasts from all experts from time \(1\) to \(t\).

• state feedback: at any time \(t\leq T\), the decision maker observes the forecasts from all experts from time \(1\) to \(t\), and the realized state from time \(1\) to \(t-1\).

To measure the performance of the decision maker, there are two natural benchmarks (defined informally below).

• Optimal-in-hindsight: The optimal-in-hindsight benchmark is a standard and widely used benchmark, which is defined as the payoff obtained by the best report-to-action policy based on a single expert’s forecasts and selecting, in hindsight, the expert whose reports yield the highest payoff.

  Definition 1 (Optimal-in-Hindsight Benchmark). Let \(U^i\) denote the optimal value achieved when the decision-maker uses only the reports from expert \(i\) ¹, namely, \[U^i = \max_{\sigma:E^i\to \mathcal{A}} \sum_{t\le T} u(\sigma(e_t^i), \omega_t),\] where \(\omega_1,\ldots,\omega_T\) is the realized state sequence. Then, the optimal-in-hindsight benchmark is defined as \[{\rm OIH}:= \max_{i\in [n]} U^i.\]

• Robust-maxmin: The robust-maxmin benchmark is the maximal payoff that can be guaranteed by a mixed strategy over action sequence space against all calibration-compatible state sequence.

  Definition 2 (Robust-Maxmin Benchmark). Let \(\Omega(e_{1:T})\) be the set of state sequences that satisfy the calibration constraints 1 for all experts. Then, the robust-maxmin benchmark is defined as \[\label{eq95origin95def95RMM} {\rm RMM}:= \max_{x\in (\Delta(\mathcal{A}))^m} \min_{\omega_{1:T}\in \Omega(e_{1:T})} \mathbb{E}\left[ \sum_{t\leq T} u(a_t,\omega_t) \right],\qquad{(1)}\] where \(a_t\sim x(e_t)\) with \(x(e_t)\in\Delta(\mathcal{A})\), and the expectation is taken only over this randomization.

In our paper, we focus primarily on the robust maxmin benchmark. Given this benchmark, the regret of the decision maker given any realized sequence of actions \(a^*_1,\dots,a^*_T\) is \[\begin{align} {\rm Reg}= {\rm RMM}- \min_{\omega_{1:T}\in \Omega(e_{1:T})} \sum_{t=1}^T u(a^*_t,\omega_t). \end{align}\]

2.0.0.3 Bayesian Environments

In Bayesian environments, we make the additional assumption that both the states and the forecasts are generated via a stochastic process. Specifically, on each day \(t\leq T\), the state \(\omega_t\) is drawn independently according to a prior distribution \(q\) over \(\Omega\). By slightly overloading the notation, we also use \(q\) to denote the probability of the state being \(1\) under distribution \(q\). Then, each expert \(i\) receives a signal \(s^i_t\) according to the information structure \(\pi^i : \Omega \to \Delta(S^i)\), where \(S^i\) is a finite signal space. In Bayesian environments, we assume that the experts’ signals \(s^1_t,\dots,s^n_t\) are drawn independently across experts conditional on the realized state \(\omega_t\). Let \(\pi^i(s;\omega)\) be the probability that the signal is \(s\) when the state is \(\omega\). In Bayesian environments, each expert’s forecast is their posterior belief over the state conditional on receiving the signal \(s^i_t\), which, by Bayes’ rule, is \[\begin{align} e^i_t = \frac{q\cdot \pi^i(s^i_t;1)}{q\cdot \pi^i(s^i_t;1) + (1-q)\cdot \pi^i(s^i_t;0)}. \end{align}\] In this case, the prior \(q\), together with the experts’ forecast profile \(e\), is sufficient to determine the posterior probability that the state is \(1\) conditional on observing \(e\).

3 Offline Benchmarks↩︎

In this section, we study the offline setting, where the entire sequence of forecast profiles over \(T\) days is fixed. We first introduce the formal definition of the robust-maxmin benchmark and analyze its computational complexity. We then show that this benchmark is stronger than the optimal-in-hindsight benchmark, up to the calibration error term. Finally, in the Bayesian setting, we discuss the relationship between the robust-maxmin benchmark and the optimal Bayesian benchmark.

3.1 Robust-Maxmin Benchmark↩︎

Although each expert is calibrated, relying on the forecasts of a single expert may discard decision-relevant information contained in the joint forecast profile of all experts. The key idea behind the robust-maxmin benchmark is therefore to aggregate experts’ forecasts by exploiting the implications of calibration. We next show how the calibration condition can be used to construct a more informative forecast of the state of nature.

3.1.0.1 Calibration Constraint

As illustrated by the example in the introduction, aggregating forecasts naturally involves partitioning the time horizon into groups according to the experts’ forecast profiles and assigning a posterior probability to each group. Although the example is simple, it captures the intrinsic process of forecast aggregation.

More precisely, for each forecast profile \(e\), we group together all days on which \(e_t=e\). The calibration constraint then restricts the empirical frequency with which the state \(\omega_t=1\) occurs within each such group. Let \(\rho:E\to[0,1]\) denote the mapping that assigns to each forecast profile \(e\) this conditional empirical frequency. That is, for each \(e\in E\) with positive empirical frequency, \[\rho(e) = \frac{\sum_{t\leq T} \mathbf{1}\left[e_t = e\right]\mathbf{1}\left[\omega_t=1\right]}{\sum_{t\leq T}\mathbf{1}\left[e_t=e\right]}.\]

In this notation, the calibration condition can be expressed as a set of linear constraints on the vector \(\rho := (\rho(e))_{e\in E}\): \[\left\vert \frac{\sum_{e:[e]_i = e^i}\sum_{t\le T}\mathbf{1}\left[e_t = e\right] \rho(e)}{\sum_{t\leq T}\mathbf{1}\left[e_t^i = e^i\right]} - e^i \right\vert \le \varepsilon_T, \quad \forall e^i\in E^i,\;i\in [n].\]

Let \(\hat{\mathbf{P}}_T\) denote the empirical distribution over forecast profiles induced by the observed sequence over \(T\) days. That is, \(\hat{\mathbf{P}}_T(e)\) is the empirical frequency of profile \(e\). For each expert \(i\) and forecast \(e^i\in E^i\), we write \(\hat{\mathbf{P}}_T(e^i) := \sum_{e:[e]_i=e^i} \hat{\mathbf{P}}_T(e)\) for the empirical frequency with which expert \(i\) reports forecast \(e^i\). In terms of \(\hat{\mathbf{P}}_T\), the calibration constraint is equivalently written as \[\left\vert \frac{\sum_{e:[e]_i = e^i}\hat{\mathbf{P}}_T(e) \rho(e)}{\hat{\mathbf{P}}_T(e^i)} - e^i \right\vert \le \varepsilon_T, \quad \forall e^i\in E^i,\;i\in[n].\] We use the following form, which is equivalent whenever \(\hat{\mathbf{P}}_T(e^i)>0\) and remains well-defined when \(\hat{\mathbf{P}}_T(e^i)=0\): \[\label{eq95calibration95consraint95def} \left\vert \sum_{e:[e]_i = e^i}\hat{\mathbf{P}}_T(e)\bigl(\rho(e)-e^i\bigr) \right\vert \le \hat{\mathbf{P}}_T(e^i)\varepsilon_T\,, \quad \forall e^i\in E^i,\;i\in[n].\tag{2}\]

The calibration constraint generally does not pin down a unique \(\rho\), even when \(\varepsilon_T = 0\). Let \(\mathcal{C}(\hat{\mathbf{P}}_T)\) denote the set of all mappings \(\rho\) that satisfy the calibration constraints 2 induced by the empirical distribution \(\hat{\mathbf{P}}_T\).

3.1.0.2 Robust-Maxmin Benchmark

When choosing actions, the decision-maker must account for all mappings \(\rho \in \mathcal{C}(\hat{\mathbf{P}}_T)\) and perform well uniformly over this set. A natural criterion is therefore to maximize the worst-case expected payoff. We refer to this criterion as the robust aggregation of calibrated forecasters.

Given a mapping \(\rho:E\to [0, 1]\), define the associated payoff mapping \(v_\rho: E\to [-U, U]^d\) by \[v_\rho(e) = \rho(e) u_1 + (1-\rho(e))u_0,\] where \(u_1:=(u(a, 1))_{a\in \mathcal{A}}\) and \(u_0:=(u(a, 0))_{a\in \mathcal{A}}\). Then, the robust-maxmin benchmark is the maximal payoff that can be guaranteed by a decision rule against all calibration-compatible posterior mappings. By partitioning the time horizon according to the realized forecast profiles and using the mapping \(\rho\), the robust-maxmin benchmark can be written as \[\label{eq95expected95format95RMM} {\rm RMM}= \max_{x\in (\Delta(\mathcal{A}))^m} \min_{\rho\in \mathcal{C}(\hat{\mathbf{P}}_T)} T\sum_{e\in E} \hat{\mathbf{P}}_T(e)\, x(e)^\top v_\rho(e).\tag{3}\]

The calibration constraint imposes correlation into \(\rho(e)\) between different forecast profiles. However, in the definition of the robust-maxmin benchmark, the decision-maker’s strategy \(x\) belongs to the space \(\bigl(\Delta(\mathcal{A})\bigr)^m\) rather than \(\Delta(\mathcal{A}^m)\) that allows correlations. The following result shows that allowing correlation across forecast profiles does not improve the robust-maxmin value. The proof compares correlated strategies and profile-wise mixed strategies through their induced marginal mixed actions, showing that each strategy in one formulation has a counterpart in the other with the same payoff against every feasible posterior mapping.

Next, we analyze the computational complexity of the robust-maxmin benchmark. The benchmark is the value of a saddle-point problem, where the objective is bilinear in the decision rule \(x\) and the mapping \(\rho\), while the feasible sets for both variables are described by linear constraints. As shown in the proof of the following theorem, after fixing \(x\), dualizing the inner minimization problem yields an equivalent linear program over the decision rule \(x\) and the associated dual variables. Hence, the robust-maxmin benchmark can be computed in polynomial time.

3.2 Compared with the Optimal-in-Hindsight Benchmark↩︎

We next compare the robust-maxmin benchmark with the optimal-in-hindsight benchmark defined in the Definition 1. The following result shows that, when experts are perfectly calibrated, the robust-maxmin benchmark cannot perform worse than the optimal-in-hindsight benchmark. Under approximate calibration, the same comparison holds up to an error term proportional to the calibration error. The intuition is that robust aggregation can always replicate the best single expert’s policy, while approximate calibration ensures that the worst calibrated belief cannot distort the payoff of this policy by more than the calibration-error term.

3.3 Compared with the Bayesian Benchmark↩︎

In a Bayesian environment, each expert observes a private signal and reports the posterior probability of the state conditional on that signal. Without loss of generality, we assume that signals and the induced posterior forecasts are in one-to-one correspondence.

In this case, the prior \(q\), together with the experts’ forecast profile \(e\), is sufficient to determine the posterior probability that the state is \(1\) conditional on observing \(e\). By Bayes’ rule, \[\operatorname{Pr}(\omega=1\mid e) = \frac{(1-q)^{n-1}\prod_{i\in[n]} e^i}{(1-q)^{n-1}\prod_{i\in[n]} e^i + q^{n-1}\prod_{i\in[n]}(1-e^i)}.\] For notational convenience, define \(\rho^\ast:E\to[0,1]\) by setting \(\rho^\ast(e)\) to be the Bayesian posterior probability associated with forecast profile \(e\).

In the Bayesian environment, the realized forecast profile at each day can be regarded as being drawn from a distribution over \(E\), denoted by \(\mathbf{P}\in\Delta(E)\). The Bayesian benchmark is obtained by evaluating the decision-maker’s expected payoff under this distribution and, for each realized forecast profile, choosing a strategy that maximizes the payoff evaluated at the corresponding posterior belief, that is, \[\operatorname{Bay} = \max_{x\in(\Delta(\mathcal{A}))^m} T\sum_{e\in E} \mathbf{P}(e)\,x(e)^\top v_{\rho^\ast}(e).\]

In Bayesian environments, the Bayesian benchmark represents the optimal expected reward attainable by a decision-maker who knows the prior and correctly updates beliefs using the experts’ forecast profile. It is therefore the natural performance benchmark in such environments. This naturally raises the following question:

To make this comparison precise, we adapt the robust-maxmin benchmark to the Bayesian environment by replacing the empirical distribution induced by the realized sample path with the underlying distribution of forecast profiles. Define the calibration constraint using \(\mathbf{P}\) by \[\label{eq95cali95true95prob} \left\vert \sum_{e:[e]_i=e^i}\mathbf{P}(e)\bigl(\rho(e)-e^i\bigr) \right\vert \le \varepsilon_T\sum_{e:[e]_i=e^i}\mathbf{P}(e), \quad \forall e^i\in E^i,\;i\in[n].\tag{4}\] Let \(\mathcal{C}(\mathbf{P})\) denote the set of all \(\rho\) that satisfy the calibration constraints induced by \(\mathbf{P}\). The corresponding robust-maxmin benchmark is then defined as \[\label{eq95def95RMM95stoc} {\rm RMM}:= \max_{x\in (\Delta(\mathcal{A}))^m}\min_{\rho\in \mathcal{C}(\mathbf{P})} T\sum_{e\in E} \mathbf{P}(e)x(e)^\top v_\rho(e).\tag{5}\]

We first record a direct comparison between the robust-maxmin benchmark and the Bayesian benchmark.

The following example shows that this inequality can be strict.

The example above can be understood from the perspective of the nested max-min structure of the robust-maxmin benchmark. Fix any \(\rho\in\mathcal{C}(\mathbf{P})\) and define the posterior error mapping \(\beta: E\to [-1, 1]\) by \(\beta(e):=\rho(e)-\rho^\ast(e)\) for all \(e\in E.\) Since the Bayesian posterior \(\rho^\ast\) also satisfies the calibration constraints, the mapping \(\beta\) must satisfy \[\label{eq95const95beta} \sum_{e:[e]_i=e^i}\mathbf{P}(e)\beta(e)=0, \qquad \forall e^i\in E^i,\;i\in[n].\tag{6}\] Thus, the feasible posterior error mappings lie in the intersection of a linear subspace, determined by 6 , with the box constraints \[-\rho^\ast(e)\leq \beta(e)\leq 1-\rho^\ast(e), \qquad e\in E.\]

Substituting \(\rho=\rho^\ast+\beta\) into the robust-maxmin benchmark gives \[\begin{align} {\rm RMM} &= \max_{x\in(\Delta(\mathcal{A}))^m} \min_{\beta} T\sum_{e\in E}\mathbf{P}(e)\, x(e)^\top \Bigl( \bigl(\rho^\ast(e)+\beta(e)\bigr)u_1 + \bigl(1-\rho^\ast(e)-\beta(e)\bigr)u_0 \Bigr)\\ &= \max_{x\in(\Delta(\mathcal{A}))^m} \left\{ T\sum_{e\in E}\mathbf{P}(e)\,x(e)^\top v_{\rho^\ast}(e) + \min_{\beta} T\sum_{e\in E}\mathbf{P}(e)\,\beta(e)\, x(e)^\top (u_1-u_0) \right\}, \end{align}\] where the minimization is over all feasible posterior error mappings satisfying 6 and the box constraints above. The first term is the Bayesian payoff of the fixed decision rule \(x\) under the true Bayesian posterior; maximizing this term over \(x\) gives the Bayesian benchmark. However, in the robust-maxmin benchmark, the decision rule \(x\) is chosen while anticipating the worst feasible posterior error in the second term. This inner minimization can change the optimal choice of \(x\) and reduce the resulting value relative to the Bayesian benchmark. Therefore, calibration alone does not generally allow the robust-maxmin benchmark to match the Bayesian benchmark; in general, the robust-maxmin benchmark can be strictly smaller.

4 Online Forecast-only Feedback Model↩︎

In this section, we study online decision-making with forecast-only feedback and develop an algorithm that achieves no regret with respect to the robust-maxmin benchmark. This result shows that the decision-maker can compete against a benchmark that is more compelling than the optimal-in-hindsight benchmark by observing all calibrated experts’ forecasts and exploiting the decision-relevant information.

4.1 Online Protocol and Robust Regret↩︎

We impose the following additional assumption on the generation of experts’ forecasts. The necessity of this assumption is illustrated at the end of this section by an example showing that, without it, no online algorithm can guarantee sublinear robust regret uniformly over arbitrary deterministic sequences of forecast profiles.

This assumption is motivated by two observations. First, in the motivating example from the introduction, it is plausible that no algorithm can achieve a no-regret guarantee with respect to the robust-maxmin benchmark in 3 : during the first third of the horizon, algorithms are likely to choose the same strategy, which may result in regret of order \(T\). Second, the calibration constraints are fully determined by the empirical distribution \(\hat{\mathbf{P}}_T\) of forecast profiles. Consequently, any two sequences of forecast profiles with the same empirical distribution \(\hat{\mathbf{P}}_T\) induce the same robust-maxmin benchmark. However, the order in which forecast profiles appear can affect the algorithm’s decisions, making a no-regret guarantee difficult to attain without further restrictions.

Now, we specify the decision-making protocol under this assumption. On each day \(t\leq T\), the decision-maker observes the past forecast profiles \(e_{1:t-1}\) and the current forecast profile \(e_t\), drawn from \(\mathbf{P}\). Based on this information, the decision-maker samples an action \(a_t\in\mathcal{A}\) according to a mixed strategy, denoted by \(x_t(e_t)\in \Delta(\mathcal{A})\). This formulation includes deterministic choices as a special case.

The robust regret is defined relative to the robust-maxmin benchmark as follows.

Here, we evaluate the sequence of actions \(\{a_t\}_{t=1}^T\) by its worst-case expected payoff over all \(\rho\in\mathcal{C}(\mathbf{P})\). This criterion is natural in the forecast-only feedback setting because the decision-maker neither observes the realized state sequence nor receives payoff feedback over the horizon, and the robust-maxmin benchmark is defined using the same worst-case evaluation. The theorem below therefore competes with \({\rm RMM}\) based on the distribution \(\mathbf{P}\), not with the realized empirical benchmark based on \(\hat{\mathbf{P}}_T\).

An online algorithm is called no-robust-regret if the sequence of actions \(\{a_t\}_{t=1}^T\) generated by the algorithm incurs sublinear robust regret, i.e., \({\rm Reg}=o(T)\). Before presenting our no-robust-regret learning algorithm, we first rewrite the robust regret in a more convenient form using conditional expectations.

4.1.0.1 Filtration.

Let \(\mathcal{F}_0\) be the trivial \(\sigma\)-algebra, and for each \(t\geq 1\), let \(\mathcal{F}_{t-1}:=\sigma(e_1,\ldots,e_{t-1})\) denote the \(\sigma\)-algebra generated by the historical forecast profiles. Since the forecast profiles are sampled independently across time, \(e_t\) is independent of \(\mathcal{F}_{t-1}\) and, by taking conditional expectations day by day, the expected realized payoff can be expanded into the corresponding expectation under \(\mathbf{P}\) at each day.

4.2 Plug-in Robust Aggregation: A No-Robust-Regret Algorithm↩︎

In the online setting, the main difficulty is that the distribution \(\mathbf{P}\) is unknown. Consequently, the robustly optimal decision rule associated with the robust-maxmin benchmark cannot be computed exactly during learning. A natural approach is to estimate \(\mathbf{P}\) from the observed forecast profiles and compute a robustly optimal decision rule under the estimated distribution.

Define the empirical distribution by \[\label{eq:empirical95dist} \hat{\mathbf{P}}_{t-1}(e) := \begin{cases} 0, & t=1,\\ \frac{\sum_{\tau=1}^{t-1}\mathbf{1}\left[e_\tau=e\right]}{t-1}, & t\ge 2, \end{cases} \qquad \forall e\in E.\tag{7}\] Given \(\hat{\mathbf{P}}_{t-1}\), the corresponding empirical calibration set is \[\mathcal{C}(\hat{\mathbf{P}}_{t-1}) := \left\{ \rho\in [0,1]^m:\; \left| \sum_{e:\,[e]_i=e^i} \hat{\mathbf{P}}_{t-1}(e)\bigl(\rho(e)-e^i\bigr) \right| \leq \sum_{e:\,[e]_i=e^i}\hat{\mathbf{P}}_{t-1}(e)\varepsilon_T, \quad \forall e^i\in E^i,\;i\in[n] \right\}.\] This non-normalized form avoids division by empirical frequencies that may be zero.

However, due to finite-sample fluctuations, the empirical calibration constraints need not be jointly feasible. In other words, \(\mathcal{C}(\hat{\mathbf{P}}_{t-1})\) may be empty, in which case the plug-in saddle-point problem is not well defined. To avoid this issue, we relax the hard calibration constraints and instead penalize calibration violations directly in the objective. Specifically, at day \(t\), the decision-maker solves the following penalized saddle-point problem: \[\label{eq:penalized95SP} \max_{x\in(\Delta(\mathcal{A}))^m} \min_{\rho\in[0,1]^m} \left\{ \sum_{e\in E}\hat{\mathbf{P}}_{t-1}(e)\,x(e)^\top v_\rho(e) + \gamma \sum_{i\in[n]}\sum_{e^i\in E^i} \left| \sum_{e:\,[e]_i=e^i} \hat{\mathbf{P}}_{t-1}(e)(\rho(e) - e^i) \right| \right\},\tag{8}\] where \(\gamma>0\) is a penalty coefficient. We refer to the resulting online procedure as the Plug-in Robust Aggregation (PRA) algorithm, whose pseudocode is presented in Algorithm 2.

The following theorem states that PRA algorithm achieves sublinear robust regret.

We introduce a useful notation for the worst-case expected reward. For the true distribution \(\mathbf{P}\) and any decision rule \(x\in(\Delta(\mathcal{A}))^m\), define \[\label{eq:WP95def95proof} W_{\mathbf{P}}(x) := \min_{\rho\in\mathcal{C}(\mathbf{P})} \sum_{e\in E}\mathbf{P}(e)\,x(e)^\top v_\rho(e),\tag{9}\] and define the robust value \[V(\mathbf{P}) := \max_{x\in(\Delta(\mathcal{A}))^m} W_{\mathbf{P}}(x).\] By definition, \[{\rm RMM}= T\cdot V(\mathbf{P}).\]

Given any distribution \(\mathbf{Q}\in \Delta(E)\)(including \(\mathbf{P}\) and \(\hat{\mathbf{P}}_0\)) and any decision rule \(x\in (\Delta(\mathcal{A}))^m\), define the calibration violation of \(\rho\in [0, 1]^m\) under \(\mathbf{Q}\) to be \[\label{eq:violation32definition} \operatorname{viol}_\mathbf{Q}(\rho) = \sum_{i\in[n]}\sum_{e^i\in E^i} \left| \sum_{e:\,[e]_i=e^i}\mathbf{Q}(e)\bigl(\rho(e) - e^i\bigr) \right|.\tag{10}\] Define \[\label{eq:penalized95wcv} \widetilde{W}_\mathbf{Q}^\gamma(x) := \min_{\rho\in [0, 1]^m} \left\{\sum_{e\in E}\mathbf{Q}(e)x(e)^\top v_\rho(e) + \gamma \operatorname{viol}_{\mathbf{Q}}(\rho)\right\}.\tag{11}\] We also define the corresponding penalized robust value by \[\widetilde{V}^\gamma(\mathbf{Q}) := \max_{x\in (\Delta(\mathcal{A}))^m} \widetilde{W}_\mathbf{Q}^\gamma(x).\]

We record two immediate properties of these quantities.

1. For any \(\rho\in \mathcal{C}(\mathbf{P})\), \[\mathrm{viol}_{\mathbf{P}}(\rho) \leq \sum_{i\in[n]}\sum_{e^i\in E^i}\sum_{e:\,[e]_i=e^i}\mathbf{P}(e)\varepsilon_T = n\varepsilon_T.\] Hence, for any \(x\in (\Delta(\mathcal{A}))^m\), \[\label{eq:direct95property951} \widetilde{W}_\mathbf{P}^\gamma(x) \leq \min_{\rho\in \mathcal{C}(\mathbf{P})} \Bigl\{\sum_{e\in E}\mathbf{P}(e)x(e)^\top v_\rho(e) + \gamma \operatorname{viol}_{\mathbf{P}}(\rho)\Bigr\} \leq W_\mathbf{P}(x) + n\gamma\varepsilon_T.\tag{12}\]

2. For any empirical distribution \(\hat{\mathbf{P}}_{t-1}\), let \(x_t\) be the decision rule generated by Algorithm 2 at day \(t\). By the construction of the algorithm, \[\label{eq95relation95between95V95W} \widetilde{V}^\gamma(\hat{\mathbf{P}}_{t-1}) = \widetilde{W}_{\hat{\mathbf{P}}_{t-1}}^\gamma(x_t).\tag{13}\]

4.2.0.1 Proof sketch of Theorem 2.

The proof proceeds in several steps. The full proofs of the following lemmas are deferred to Appendix 8.2.

(i) Regret decomposition through \(V(\cdot), \widetilde{V}^\gamma(\cdot), W_{\cdot}(x)\) and \(\widetilde{W}_{\cdot}^\gamma(x)\). In the PRA algorithm, the decision rule \(x_t\) is computed from the empirical distribution \(\hat{\mathbf{P}}_{t-1}\) without using the current forecast profile \(e_t\). Consequently, \(x_t\) is \(\mathcal{F}_{t-1}\)-measurable, which allows us to apply Lemma 1 and Eq. 13 to express the robust regret as a sum of one-period expected value gaps.

(ii) Bounded difference between \(W_\mathbf{P}(x)\) and \(\widetilde{W}_{\mathbf{P}}^\gamma(x)\) for sufficiently large \(\gamma\). The regret decomposition in ?? involves both the constrained value function \(W_{\mathbf{P}}\) and the penalized value function \(\widetilde{W}_{\mathbf{P}}^{\gamma}\). The key observation is that the calibration constraints define a system of linear inequalities. By Hoffman’s lemma, the distance from any \(\rho\in[0,1]^m\) to the feasible calibration set \(\mathcal{C}(\mathbf{P})\) can be bounded by a constant multiple of its calibration violation, together with the calibration tolerance \(\varepsilon_T\). Therefore, any posterior mapping \(\rho\) that violates calibration can be projected to a feasible calibrated mapping at a payoff loss controlled by its violation and by \(\varepsilon_T\). When \(\gamma\) is chosen large enough, the penalty term in \(\widetilde{W}_{\mathbf{P}}^{\gamma}\) dominates this loss. As a result, minimizing the penalized objective over all \(\rho\in[0,1]^m\) yields essentially the same value as minimizing the original payoff over \(\rho\in\mathcal{C}(\mathbf{P})\), up to the calibration-error term. Combined with the direct upper bound in 12 , this gives the following uniform comparison.

Then, the regret bound turns to be \[\label{eq:regret95decomposition} {\rm Reg} \le \sum_{t=1}^T \mathbb{E}\!\left[\widetilde{V}^\gamma(\mathbf{P})-\widetilde{V}^\gamma(\hat{\mathbf{P}}_{t-1})\right] + \sum_{t=1}^T \mathbb{E}\!\left[\widetilde{W}_{\hat{\mathbf{P}}_{t-1}}^{\gamma}(x_t)-\widetilde{W}_{\mathbf{P}}^\gamma(x_t)\right] + 2n\gamma\varepsilon_T\cdot T.\tag{14}\] This regret decomposition reduces the analysis to (i) establishing Lipschitz continuity of \(\widetilde{V}^\gamma(\cdot)\) and \(\widetilde{W}^\gamma_{\cdot}(x)\), and (ii) controlling the estimation error \(\|\hat{\mathbf{P}}_{t-1}-\mathbf{P}\|_1\).

(iii) Lipschitz continuity of \(\widetilde{V}^\gamma(\cdot)\) and \(\widetilde{W}_{\cdot}^\gamma(x)\). The Lipschitz bound follows directly from the primal definition of the penalized value functions. For a fixed decision rule \(x\) and posterior mapping \(\rho\), changing the distribution from \(\mathbf{Q}\) to \(\mathbf{Q}'\) changes the payoff term by at most \(U\|\mathbf{Q}-\mathbf{Q}'\|_1\). The calibration-violation penalty changes by at most \(n\gamma\|\mathbf{Q}-\mathbf{Q}'\|_1\), because \(|\rho(e)-e^i|\le 1\) and each forecast profile contributes once for each expert. Taking the minimum over \(\rho\) and then the maximum over \(x\) preserves this Lipschitz bound.

(iv) Estimation error of the empirical distribution. Since the forecast profiles are sampled independently from the true distribution \(\mathbf{P}\), the empirical frequency \(\hat{\mathbf{P}}_{t-1}(e)\) is an unbiased estimate of \(\mathbf{P}(e)\) for each \(e\in E\) and its variance is of order \(1/(t-1)\). Summing these coordinate-wise estimation errors over the finite profile space yields the following bound.

Now, we can give a formal proof of theorem 2.

4.3 An Example Illustrating the Necessity of Assumption 1↩︎

To see why Assumption 1 on the generation of forecast profiles is needed, consider the following binary-state example. Let \(\Omega=\{0,1\}\), where \(\omega=1\) denotes rain, and let the decision-maker choose an action \(a\in\{0,1\}\). The payoff is \[u(a,\omega)=\mathbf{1}\{a=\omega\}.\] Thus, if the posterior probability of rain under a forecast profile \(e\) is \(\rho(e)\), then action \(1\) yields expected payoff \(\rho(e)\), while action \(0\) yields expected payoff \(1-\rho(e)\).

There are two experts, each of whom may report one of the values \(0,1/2,1\). Consider the following forecast profiles: \[e^0=\left(\frac{1}{2},\frac{1}{2}\right),\qquad e^L_1=\left(\frac{1}{2},0\right),\qquad e^L_2=\left(0,\frac{1}{2}\right),\] and \[e^H_1=\left(\frac{1}{2},1\right),\qquad e^H_2=\left(1,\frac{1}{2}\right).\] Assume for simplicity that \(T\) is divisible by \(3\). We construct two possible forecast sequences. In the first sequence, the first \(T/3\) days have profile \(e^0\), the next \(T/3\) days have profile \(e^L_1\), and the final \(T/3\) days have profile \(e^L_2\). The resulting empirical distribution assigns probability \(1/3\) to each of \(e^0,e^L_1,e^L_2\). The calibration constraints then imply \[\rho(e^L_1)=0,\qquad \rho(e^L_2)=0.\] Moreover, conditioning on the days on which Expert 1 reports \(1/2\) gives \[\frac{\rho(e^0)+\rho(e^L_1)}{2}=\frac{1}{2},\] and hence \(\rho(e^0)=1\). The same conclusion is obtained from the calibration constraint for Expert 2’s report \(1/2\). Therefore, under this empirical distribution, the robust-maxmin benchmark chooses action \(1\) on profile \(e^0\).

In the second sequence, the first \(T/3\) days are again all equal to \(e^0\), but the remaining days consist of \(T/3\) repetitions of \(e^H_1\) and \(T/3\) repetitions of \(e^H_2\). The resulting empirical distribution assigns probability \(1/3\) to each of \(e^0,e^H_1,e^H_2\). The calibration constraints imply \[\rho(e^H_1)=1,\qquad \rho(e^H_2)=1.\] Using again the calibration constraint for Expert 1’s report \(1/2\), we obtain \[\frac{\rho(e^0)+\rho(e^H_1)}{2}=\frac{1}{2},\] and hence \(\rho(e^0)=0\). The same conclusion follows from Expert 2’s report \(1/2\). Thus, under this empirical distribution, the robust-maxmin benchmark chooses action \(0\) on profile \(e^0\).

The two sequences are indistinguishable to the decision-maker during the first \(T/3\) days, since the observed forecast profile is \(e^0\) on every one of these days in both sequences. Therefore, any online algorithm must use the same mixed action on these days in the two sequences. Let \(q_t\) denote the probability with which the algorithm chooses action \(1\) on day \(t\le T/3\). In the first sequence, the benchmark payoff on profile \(e^0\) is achieved by action \(1\), so the regret contribution on day \(t\) is \(1-q_t\). In the second sequence, the benchmark payoff on profile \(e^0\) is achieved by action \(0\), so the regret contribution on day \(t\) is \(q_t\). Hence the two regret contributions sum to \(1\) on each of the first \(T/3\) days. It follows that at least one of the two sequences gives regret at least \[\frac{1}{2}\cdot \frac{T}{3}=\frac{T}{6}.\] Consequently, no online algorithm can guarantee sublinear robust regret uniformly over arbitrary deterministic orderings of forecast profiles. This illustrates why the assumption that forecast profiles are sampled from a fixed distribution is needed. Without this assumption, the final empirical distribution may induce a robust-maxmin benchmark whose optimal action for an early forecast profile cannot be inferred from the information available when that action is chosen.

5 Online State Feedback Model↩︎

In this section, we study the decision-making problem under state feedback. In this feedback model, the decision-maker observes the forecast profile \(e_t\) before choosing an action and observes the realized state \(\omega_t\) after the decision is made. Therefore, the realized payoff of every action can be evaluated ex post from the observed state. This additional feedback allows us to benchmark the decision-maker against the best context-dependent decision rule in hindsight, rather than against the robust-maxmin benchmark.

Specifically, we treat forecast profiles as contexts and actions as arms. This reduces the problem to a contextual full-information online learning problem with a finite context space \(E\) and a finite action set \(\mathcal{A}\). The natural benchmark is the best mapping \(f:E\to\mathcal{A}\) from forecast profiles to actions in hindsight. This benchmark represents the best fixed context-dependent rule that could have been chosen after observing the entire sequence of forecast profiles and realized states.

Given an online algorithm that chooses actions \(\{a_t\}_{t=1}^T\), its contextual regret is defined as the gap between the contextual benchmark and the cumulative payoff achieved by the algorithm.

The benchmark in ?? is natural in the state-feedback model. Since the realized state is observed after each day, the decision-maker can evaluate the payoff that each action would have obtained at that day. Hence, learning can be performed separately for each forecast profile. In particular, for each \(e\in E\), the decision-maker maintains an independent online learning algorithm over the action set \(\mathcal{A}\), and updates only the algorithm corresponding to the realized forecast profile. The resulting procedure is summarized in Algorithm 3.

The following result shows that this simple algorithm yields no contextual regret.

Algorithm 3 provides a simple way to achieve no contextual regret, but it is not necessarily optimal in terms of regret bounds. An interesting open question is whether the calibration constraints can be exploited to obtain sharper regret guarantees, and what the optimal regret rate is in the state-feedback model.

6 Conclusion and Future Work↩︎

This paper develops a framework for decision-making with multiple calibrated forecasts when the decision-maker knows only that experts satisfy calibration. We first introduce a robust max-min benchmark, which is the best payoff that can be guaranteed against all posterior mappings consistent with calibration. This benchmark captures the decision-relevant information contained in the joint pattern of forecasts without requiring knowledge of experts’ signal structures, priors, or conditional independence relationships. We show that the robust max-min benchmark is tractable and admits a linear-programming formulation. It also dominates the optimal-in-hindsight benchmark up to calibration error, reflecting the value of conditioning on the full forecast profile rather than using only a single expert. At the same time, it can be strictly below the Bayesian benchmark, highlighting the additional value of knowing the full data-generating process.

We then study online learning under two feedback structures. Under forecast-only feedback, where the decision-maker observes forecast profiles but not realized states or payoffs, we propose a plug-in robust aggregation algorithm and prove sublinear robust regret relative to the robust benchmark when forecast profiles are drawn from a fixed distribution. Under state feedback, where the realized state is observed after each decision, the learner can compete with the stronger contextual optimal-in-hindsight benchmark, and a simple contextual full-information learning algorithm achieves no contextual regret.

Several questions remain open. First, the forecast-only result relies on the assumption that forecast profiles are sampled from a fixed distribution. An important direction is to relax this assumption, for example by allowing adversarially controlled noise around an underlying distribution or time-varying forecast distributions subject to suitable stability. Second, the regret bounds obtained here may not be optimal. It remains open whether sharper rates can be achieved, either in the dependence on \(T\) or in the dependence on \(m\) and \(d\). In the state-feedback model, our algorithm uses a simple reduction to contextual online learning and does not exploit the structure imposed by calibration constraints. Designing algorithms that use these constraints more directly may lead to tighter regret bounds.

7 Strong Duality Theorem and Hoffman Lemma↩︎

First, we give the standard strong duality theorem for linear programming used to derive the dual representation; see, e.g., [35] or [36].

Then, we present a standard form of Hoffman’s lemma for polyhedral systems; see [37]. We use this lemma to relate the \(\ell_1\)-residual of linear constraints to the distance to the corresponding feasible set.

8 Missing Proofs↩︎

8.1 Proofs of Section 3↩︎

8.2 Proof of Section 4↩︎

8.3 Proof of Section 5↩︎

References↩︎

---

1. Under exact calibration, the value of the best single-expert policy is determined by the expert’s forecast sequence. With approximate calibration, different calibrated state sequences can lead to values that differ by the calibration error term.↩︎

2. The parameter \(m=|E| = \prod_i |E^i|\) is the explicitly represented profile-space size and may grow exponentially with the number of experts.↩︎