1 Introduction↩︎

Calibrated probabilistic forecasting is becoming a standard test of whether a language model can reason under uncertainty, and increasingly a way to decide which agents to trust with open-ended tasks [1]. Writing forecasting questions by hand is slow, so the common shortcut is to grade models on resolved questions whose answers are now known and to score the probability the model assigns to the true outcome [2], [3]. The supervision is automatic and cheap, but it pays a model for memory just as readily as for foresight. A model asked today for the 2022 World Cup champion (Figure 1) names Argentina with near-certainty, and that certainty comes from the post-tournament write-ups in its training data. At inference, a model that retrieves can pull in content created after the event, and across releases each new model is trained on data closer to the event, so a question safely out-of-cutoff for this year’s models leaks into next year’s [4], [5]. Such an evaluation measures how well a model recalls the resolved outcome, while its foresight goes untested.

What we actually want to measure is the counterfactual probability the model would have assigned at a past time \(t_0\), before the outcome was known. This is also the only forecast that was ever useful, since it is the one a trader or analyst could have acted on at \(t_0\) [6]. But it is hard to recover after the fact, because the live web has moved on. Pages are rewritten and re-indexed, so retrieval can no longer return what was actually visible at \(t_0\). Live benchmarks [7], [8] avoid this by posing questions that are still open and waiting for them to resolve, but waiting forfeits the two things resolved questions provide together. A resolved question gives a known outcome to grade against, and a prediction market adds a human forecast at \(t_0\) to measure against. That tension sets up the question this paper answers.

On already-resolved questions, can we evaluate the forecast a model would have made at a past \(t_0\), holding fixed the information it could retrieve then?

We answer with Hindcast, which makes this counterfactual measurable through hindcasting, by replaying each resolved market against a frozen archive of the past and pinning every retrieval call to that market’s \(t_0\). Two design choices follow directly. First, the archive must not change after the fact, so we build it from public Reddit, whose monthly snapshots are immutable (unlike web pages or Wikipedia, a captured month stays captured) and which discusses unfolding events early and broadly, giving a forecaster real pre-event signal to find. Second, the questions need a yardstick, so we draw on resolved Polymarket markets, each of which supplies the graded outcome and a matched-time price at \(t_0\) that is itself a human forecast distilled by traders from the same public information. Because the past is fixed per market, \(t_0\) becomes a free variable and the evaluation re-runs on markets that resolve after a new model ships, so it never decays into recall. The concurrent FutureSim [9] shares the replay idea but takes a different shape. It streams a news corpus forward over a single shared window to measure adaptation, while Hindcast pins an immutable Reddit archive to a per-market \(t_0\) and scores against the matched-time market price.

Both predictors sit behind one interface that takes a market, its resolution criteria, and a cutoff \(t_0\), and returns a probability with a short rationale (§5). We instantiate it twice. The zero-shot baseline forecasts from the model alone and isolates parametric memory, while the Retrieval agent adds only access to the archive at \(t_0\), so the gap between them measures the contribution of evidence. Figure 2 sketches the pipeline.

With a leakage-free evaluation in hand, we revisit whether retrieval-grounded reasoning improves forecasting, a claim prior work makes under possible contamination. If those reported gains came partly from reading the future, they should shrink once the future is sealed off. The gain does not vanish, since retrieval still lowers Brier on eight of the nine open-weight models we test. But it concentrates on markets the corpus covered in advance and reverses where the agent over-reads speculative chatter (§7). Retrieval helps, then, but only when the pre-\(t_0\) corpus already discussed what decided the event. Where it carried only speculation, retrieval backfires.

1.0.0.1 Contributions.

(i) Hindcasting, an evaluation protocol that measures a model’s forecast at a past \(t_0\) on resolved questions, holding its retrievable information fixed, and that re-runs as models advance (§3, §4). (ii) An immutable, temporally pinned public-Reddit archive serving leakage-free retrieval at any past \(t_0\) (§3). (iii) Two reference predictors behind one probability-emitting interface, a zero-shot baseline and a Retrieval agent (§5). (iv) An empirical study locating where retrieval-grounded forecasting helps and where it backfires (§6). What limits such a forecaster turns out to be its evidence environment.

2 Related Work↩︎

2.0.0.1 LLM forecasters and benchmarks.

Early benchmarks frame future-event prediction as temporally restricted QA over news or tournament questions [10], [11]. Retrieval-augmented and agentic systems push this further, into open-ended forecasting against eventual outcomes [2], [3], [12]. Concurrent with our work, FutureSim [9] streams news forward chronologically over a recent window to measure adaptation. Hindcast instead holds the corpus fixed at a chosen \(t_0\) per market and measures improvement over the matched-time market price. Live benchmarks counter static-test contamination through continual collection [7], [8], [13]. A parallel line moves closer to prediction-market deployment, casting the task as trading over live market states [14], [15]. Hindcast is the inverse move, replacing online evaluation with replay against a fixed past information environment, so any predictor can be re-run against any past \(t_0\) without waiting for new questions to resolve.

2.0.0.2 Temporal controls and contamination.

Existing responses to contamination refresh evaluation items [16]–[19], retrain models from scratch on temporally-restricted corpora that are either annually partitioned [4] or pinned to a single pre-cutoff vintage [5], or fix the retrieval context to a static news snapshot [20]–[22]. The retraining strategies attack the parametric-memory channel by removing post-cutoff text from the weights, whereas Hindcast attacks the retrieval channel by pinning the archive at \(t_0\), so the two approaches are complementary. Archived Reddit at the simulated query time gives the agent a denser, more topically varied source of information than a single news snapshot can provide.

2.0.0.3 Prediction markets and calibration.

Prediction markets supply resolvable questions, crowd probabilities, and tradable payoffs. Prior work has used them as evaluation targets [2], [7], [11], as delayed supervision under proper scoring rules [23], and as trading environments measured by market returns [13]–[15]. We use Polymarket as a dual benchmark. Resolved binary markets supply ground truth, and historical market-implied probabilities supply a matched-time baseline. Together they let us separately measure outcome recovery, improvement over the market’s information state at \(t_0\), and calibration as resolution approaches. We score calibration directly under proper rules such as the Brier score [24], [25]. Methods for sharpening that calibration and the reasoning behind it range from LLM ensembling [26] and outcome-based training [20], [23] to Bayesian and inference-time factor decomposition [27], [28] and consistency training [29]. Hindcast is a controlled testbed under which such retrieval and calibration strategies can be replayed at matched timepoints and compared against both binary outcomes and contemporaneous market probabilities.

3 Retrieval Backend↩︎

The information side of Hindcast is served by a temporally pinned archive of public Reddit. The backend keeps one immutable copy of each document, indexes it for retrieval, and answers every call under a single cutoff predicate. It does not re-rank documents, apply a learned post-filter, or judge which results are relevant. Any difference between predictors therefore stems from how the agent queries and reads the archive.

3.1 Corpus↩︎

We build the corpus from the Pushshift archive of public Reddit submissions and comments [30], which provides static monthly dumps captured at posting time. The archive spans 25 consecutive monthly snapshots, January 2024 through January 2026, drawn from 20 subreddits that together cover the topics our markets touch.

Filtering keeps content that carries an identifiable author and enough engagement to be worth reading, and drops the rest. A record is dropped if it lacks a stable id, if its author or body is deleted or removed, or if it matches an optional bot list. We also drop records below our engagement and length floors, which cut any document with a voting score below \(5\), any submission with fewer than \(3\) comments, any comment shorter than \(5\) non-URL characters or consisting mostly of a link, and any document with fewer than \(20\) whitespace-delimited words. Since the filters do not condition on subreddit, topic, or stance, the corpus stays uncurated by content. We keep it broad on purpose, since the eval set spans markets across politics, sports, crypto, macroeconomics, and entertainment, and a topic-narrow corpus would advantage predictors on the topics it covers and penalize them everywhere else.

After filtering, the archive holds 220,943 submissions and 18,094,365 comments (\(\approx18.3M{}\) documents). Each document is stored once with its text and metadata (subreddit, author, a coarse month key, and an exact creation timestamp), and the snapshot does not change between \(t_0\) and evaluation even when the post is later edited or removed on the live site.

3.2 Retrieval↩︎

The corpus is loaded into PostgreSQL with a pgvector embedding column and is reached two ways, through dense semantic search and through direct by-id lookup. For dense search the query is embedded with the Qwen3-Embedding-0.6B encoder of [31] and documents are ranked by cosine distance over per-month HNSW indexes [32]. The reference agent restricts this search to submissions through a doc_type filter, so every hit is a whole post. This keeps the retrieved set focused, avoids burying the signal under noisy, near-duplicate comment chatter, and returns self-contained units the agent can reason over. Submissions also carry the title and self-text that best summarize a thread’s topic.

The dense search excludes comments, but the archive still holds all 18,094,365 of them and exposes them through by-id lookup tools that resolve a post or comment, walk a submission’s comment thread, and list an author’s prior posts and comments. A predictor can therefore start from a retrieved submission and pull the surrounding discussion for detail the post text alone does not carry. Every call, whether dense search or a by-id lookup, carries the same filter block. That block holds the cutoff \(t_0\), the document type, and optional subreddit, author, month, and start-time restrictions, so the agent can narrow a query without ever weakening the cutoff guarantee. The full tool surface and signatures are deferred to Appendix 9.1.

3.3 Cutoff enforcement↩︎

To make the cutoff guarantee robust to routing errors, we enforce it at two levels. The primary guarantee is a row-level predicate under which every document a tool returns is checked against \(t_0\) with the strict comparison \(\texttt{created\_at} < t_0\), evaluated in the SQL WHERE clause of dense search and of all four by-id lookup tools (post, comment, thread, and author). On top of this, a partition selector guards and accelerates dense search by deriving from \(t_0\) the monthly partitions no later than \(t_0\)’s month (and no earlier than any start-time bound), running a separate top-\(k\) search against each partition’s month-scoped HNSW index, and merging the per-partition results into an exact global top-\(k\).

A leak through dense search would thus require both layers to fail on the same document, and a leak through a direct lookup would require the row-level predicate to fail. The boundary partition (the cutoff month itself) is searched, but its post-\(t_0\) rows are removed by the row-level predicate. Latency measurements against the live index are in Appendix 9.2.

4 Evaluation Setup↩︎

4.1 Eval-set construction↩︎

A resolved market tests a retrieval-grounded predictor only if the Reddit corpus discussed its event before it resolved, so resolution alone is not a sufficient inclusion criterion. The eval set must also avoid two skews that would otherwise let aggregate accuracy be driven by a constant guess or a single domain, since most Polymarket markets resolve No and their topics are long-tailed. We therefore construct the eval set in two stages, first a coverage test that keeps only well-supported markets, then outcome and topic balancing.

4.1.0.1 Coverage test.

The candidate pool is the 7,913 binary markets in our Polymarket window (across 1,916 events), of which 3,427 resolve to a Yes/No outcome. For each, an automated probe queries the archive at the market’s close date and scores how strongly the pre-event corpus supports the resolved side. An LLM judge (Qwen3-32B) issues \(5{}\) queries per side and labels each returned block on a four-level scheme. The label strong (weight \(+2\)) marks explicit, specific evidence for the resolved side, weak (\(+1\)) marks vague, indirect, or correlational evidence, irrel (\(0\)) marks blocks that are on-topic but do not bear on the hypothesis, and contra (\(-1\)) marks evidence against the resolved side. Per side this yields a score \(s_{\mathrm{side}} = 2\,\mathrm{strong} + \mathrm{weak} - \mathrm{contra}\). A final step reads both side tallies together and emits a contrastive coverage score \(c \in [-1, 1]\), positive when the corpus supports the resolved side, alongside an overall evidence-quality label of strong, weak, or none. A market the corpus barely mentions scores near zero on both sides and lands on none, so it cannot exercise the retrieval pipeline. We keep only the markets the probe labels strong.

4.1.0.2 Outcome and topic balancing.

The strongly-covered markets inherit the skews of the raw pool, since \(83\%\) of resolved markets are No (\(2{,}851\) of 3,427) and coverage is uneven across domains because Reddit discusses sports and trading far more than, say, corporate or weather events. We therefore subsample the strongly-covered markets to equalize the Yes/No outcomes and to spread topics as evenly as the available evidence allows, prioritizing outcome balance over topic uniformity. This leaves 216 markets (\(106\) Yes and \(110\) No), split \(36\)/\(129\)/\(51\) across short, medium, and long durations, and Figure 3 shows the resulting topic distribution. The probe and this selection run as a single automated pipeline that we release, so the eval set can be regenerated as new markets resolve.

4.2 Metrics and cutoff↩︎

Each predictor returns a Yes-side probability \(\hat{p} \in [0, 1]\) and a chosen option \(\hat{y} = \mathbb{1}[\hat{p} > 0.5]\). We score against the resolved binary outcome \(y \in \{0, 1\}\) with accuracy and the Brier score, \[\mathrm{Acc} \;=\; \mathbb{1}\!\bigl[\hat{y} = y\bigr], \qquad \mathrm{Brier} \;=\; (\hat{p} - y)^{2}.\] \(\mathrm{Acc}\) rewards a correct decision and \(\mathrm{Brier} \in [0,1]\) jointly penalizes miscalibration and miscalls, each averaged over five consistency runs per (model, pipeline) cell. We pin the cutoff \(t_0\) to the day before each market closes, capped at the archive’s final day (2026-01-31).

5 Agentic Prediction Pipelines↩︎

Both predictor families share an interface, which takes a market, its resolution criteria, an option set, and a cutoff \(t_0\). The interface returns a chosen option, per-option probabilities, a self-reported confidence, and a brief rationale. Figure 4 sketches the two pipelines next to each other.

5.1 Zero-shot baseline↩︎

The zero-shot baseline forecasts from a single LLM call with no retrieval, instructed to use only knowledge that could be known at \(t_0\). It sets the parametric-memory floor against which the Retrieval agent is measured.

5.2 Retrieval agent↩︎

The Retrieval agent runs a bounded research loop over the archive at the \(t_0\) cutoff. It opens with seed queries targeting the market’s claim and the factors that plausibly drive it, then expands promising hits into their comment threads and author histories. In subsequent rounds, the model reads the accumulated evidence, writes a short reflection that estimates its confidence and names the evidence it still needs, and proposes new queries. The loop terminates on a confidence signal, on diminishing query yield, or on a round budget. A final forecast pass reads the labeled evidence under the same four-level scheme used by the coverage probe (§4.1) and emits the same probability format as the zero-shot baseline. The two families therefore differ only in whether the archive is consulted. Cutoff enforcement lives in the backend (§3.3), so the agent never needs to be trusted with it.

Both pipelines forecast at temperature \(0.1\) with top-\(p\) \(1.0\), and the agent raises the temperature to \(0.2\)–\(0.3\) only for its intermediate query-generation steps and runs a default budget of 3 rounds. Per-stage decoding, retrieval, and forecast-parsing settings are listed in Appendix 10.3.

A typical trace opens with a parallel batch of seed queries against the archive, expands the highest-ranked submissions with their full text and the author’s recent history, and closes with the labeling and final-forecast passes. The prompts driving both pipelines are in Appendix 10.

6 Experiments↩︎

We benchmark 9 open-weight LLMs under both pipelines on the eval set of §4.1 at the cutoff \(t_0\) of §4.2, spanning the Qwen3 family at 4B, 8B, and 32B parameters [33], the Llama 3 family with Llama-3.1-8B-Instruct and Llama-3.2-3B-Instruct [34], Gemma 3 at 4B and 27B [35], Ministral-8B-Instruct [36], and the DeepSeek-R1 distillation onto Qwen-7B [37].

All models share the prompt template, the decoding configuration of §5.2, and the retrieval defaults, and all of them predate the start of the eval window. Any cross-family or cross-model variation therefore isolates either pretraining differences or the contribution of retrieval. Each (model, pipeline) cell is averaged over five consistency runs.

Table ¿tbl:tab:headline-results? reports the main results. The Retrieval agent lowers Brier on eight of the nine models, with the largest relative reduction on Qwen3-32B (\(.234 \to .179\), a \(23\%\) drop) and the largest accuracy lift on Qwen3-8B (\(55.8\% \to 64.7\%\)). Conditioning on the archive helps most on markets the coverage probe rates strong, where retrieval lifts accuracy on most backbones (median \(+14\) points), while None-bin accuracy moves little. R1-Distill-Qwen-7B is the only model whose overall Brier rises under retrieval, and §7.3 unpacks this case. Per-run and per-bin numbers behind Table ¿tbl:tab:headline-results? are in Appendix 11.

6.0.0.1 Lookback fraction.

The cutoff above is pinned to a wall-clock date, so it lands at a different point in each market’s life depending on how long that market stayed open. As a finer probe of how much pre-resolution discussion the agent actually needs, we re-run a two-model subset at four per-market cutoff fractions, placing \(t_0\) at \(25\%\), \(50\%\), \(75\%\), and \(100\%\) of each market’s open-to-resolution span.

The zero-shot baseline barely moves across the four fractions for either probed model (Ministral-8B and Qwen3-8B), so any cross-fraction differences must come from the retrieval side. The Retrieval agent’s accuracy and Brier also shift only slightly, with Qwen3-8B rising from \(61.0\%\) accuracy at the \(25\%\) fraction to \(63.4\%\) at the \(100\%\) fraction while its Brier hovers near \(0.21\), and Ministral-8B’s numbers essentially unchanged across the sweep. Figure 5 plots Brier against the cutoff fraction for both models and pipelines (per-cell numbers in Appendix 11.3), and the gap between the agent and its zero-shot baseline is roughly constant across fractions for each model. The agent’s gain therefore saturates well below the full available lookback, which suggests the wall-clock offset choice is not picking up an artifact of how late in each market’s life the cutoff lands.

7 Analysis↩︎

Underneath the headline averages, the picture changes sharply from topic to topic. For every market we pair the Agent and Vanilla runs by id, pool the eight models that have per-run logs, and count, per topic, how often the agent fixes a market the zero-shot baseline got wrong (a recovery) against how often it breaks one the baseline got right (a regression). The net of the two swings hard in both directions from topic to topic, as Figure 6 shows.

7.1 Where retrieval helps↩︎

The positive net is concentrated in Sports (\(+16\)), Awards (\(+12\)), and Trading/commodities (\(+7\)). What these topics have in common is that Reddit discusses them in concrete detail in the weeks before they resolve, so the agent’s opening queries land on posts with real signal. They also resolve on facts anyone can check after the fact, like which team won, who took the award, or whether a price crossed a line, so an agent that has found the right thread can read the answer back. Both of these push the coverage probe toward strong, and the four topics on the right of Figure 6 account for most of the strong-bin accuracy lift in Table ¿tbl:tab:headline-results?. They are also where the agent quits early most often. Its confidence-driven stop fires on \(52\) recovered markets against \(34\) regressed ones, so once the evidence is in front of it the agent commits before spending its whole round budget.

7.2 Where retrieval hurts↩︎

The damage is concentrated in Entertainment at \(-31\), with Other (\(-10\)), Elections/politics (\(-7\)), Social/celebrity (\(-7\)), Tech/AI (\(-5\)), and Crypto (\(-3\)) trailing behind. Entertainment shows the failure most cleanly. Nearly all of its markets are Billboard “will X be the #1 song this week” questions, and nearly all of them resolve No, since most songs never reach the top of the Hot 100. A model with no retrieval just sits near that base rate and is usually right. The agent goes looking instead, finds the pre-release hype that music subreddits produce for any plausible contender, and counts that enthusiasm as evidence for Yes. Its tally and reflection steps reward lopsided evidence, but hype is lopsided too, and the scaffold cannot tell a fan’s excitement from a fact about the chart. The agent argues itself out of a correct default. The milder regressions on Social/celebrity, Tech/AI, and Crypto are the same failure in lower volume, all places where the loudest posts are opinion that does not track the outcome. The duration split fits this reading. Long-lived markets tip toward regression, with \(144\) regressions against \(106\) recoveries, while medium-lived ones come out slightly positive, with \(329\) regressions against \(381\) recoveries, as though the longer a market stays open the more chatter accumulates for the agent to overfit.

7.3 Per-model effects, and accuracy versus calibration↩︎

Every model shows the same topic pattern, but the size of the effect varies widely. Qwen3-8B has the cleanest split at \(+43\) markets, Qwen3-4B is solidly positive at \(+19\), and Gemma-3-4B comes out even. The rest land negative, slightly for Llama-3.2-3B and Ministral-8B-Instruct (\(-3\) and \(-6\)) and more clearly for Llama-3.1-8B-Instruct, Gemma-3-27B, and R1-Distill-Qwen-7B (\(-27\), \(-10\), and \(-19\)). The Qwen3 models pull far more from the same evidence stream than the others, which fits their strong-bin lift in Table ¿tbl:tab:headline-results?.

Accuracy and Brier do not always agree, since one counts decision flips and the other tracks where the probability mass ends up. Llama-3.1-8B-Instruct is the clearest example. It regresses on \(91\) markets against only \(64\) recoveries (\(-27\) net), and its Brier still improves under retrieval (\(.250 \to .230\)). The agent flips more calls the wrong way than the baseline does, but it hedges those wrong calls, and Brier would rather see a wrong answer at \(0.55\) than at \(0.95\). Even where retrieval costs accuracy, then, it drags the probabilities back toward the middle and the calibration improves anyway. R1-Distill-Qwen-7B is the one model where both numbers move the wrong way at once. We read that as a context-budget problem, since its long reasoning chains crowd out the evidence while the same retrieval surface wins cleanly on Qwen3-8B at the same scale.

8 Conclusion↩︎

Hindcast grades LLM forecasters at a past time \(t_0\), pinning retrieval to an immutable pre-\(t_0\) Reddit snapshot and scoring against both the resolved Polymarket outcome and the matched-time price. Retrieval lowers Brier on eight of nine models (up to 23%), but the average hides a topic split that the cutoff reveals. Retrieval recovers Sports, Awards, and Trading, where Reddit carries fact-anchored pre-event discussion, and regresses on Entertainment, where speculative hype reads as evidence. A retrieval-grounded forecaster is only as good as the signal the corpus already carried before resolution.

The design also gives the evaluation levers that live benchmarks lack. The cutoff can be swept along each market’s life to trace how calibration develops as evidence accumulates, and new predictors can be dropped behind the same interface and replayed on identical evidence. New markets keep resolving against the same frozen archive, so the test stays ahead of each new model’s training cutoff.

Limitations↩︎

The archive is built from a fixed set of 20 public subreddits, so the corpus skews English-language and community-driven and under-represents topics Reddit discusses thinly (e.g., regional politics, niche corporate events). Polymarket inherits a similar skew, and we restrict to binary Yes/No markets, leaving multi-outcome and continuous markets out of scope.

Because the archive ends on 2026-01-31, most markets in our eval set have \(t_0\) pinned at the archive boundary with only a short pre-resolution lookback (median close–\(t_0\) gap of 23 days), which limits what we can say about long-horizon forecasting. We also benchmark only 9 open-weight models behind two reference pipelines, and closed-weight models and alternative agent scaffolds are left to future work.

Hindcast is built from public Reddit content and historical Polymarket data. The privacy, deployment, and content risks this raises, and how we address them, are discussed in the Ethics Statement below.

Ethics Statement↩︎

Our corpus is derived from the public Pushshift Reddit dataset and our questions from historical Polymarket data. Both are public resources used strictly for non-commercial research, and the posts remain subject to Reddit’s User Agreement. The benchmarked checkpoints are used within their licenses (Apache-2.0 for the Qwen3 models and the Qwen3-Embedding encoder; the Llama 3.1/3.2 Community Licenses; the Gemma Terms of Use; MIT for DeepSeek-R1-Distill-Qwen-7B; and the research-only Mistral Research License for Ministral-8B-Instruct-2410), and all use here is research-only and consistent with these terms.

The corpus contains only public posts. We retain pseudonymous author handles for thread- and author-level lookups but make no attempt at re-identification, drop documents whose author or body is deleted or removed and accounts on a bot list, and release document and market identifiers with regeneration code, keeping raw post text out of the release. Artifacts derived from the corpus should not be used to profile or de-anonymize individuals. We do not screen for offensive content. Because the filters condition only on engagement and length, some posts may carry biased or offensive language that an evaluated model could echo.

Finally, the predictors we study are forecasting agents. Deployed without calibration audits they could lend false authority to automated bets or decisions, and our analysis shows that retrieval can make a model confidently wrong on speculative topics. We therefore frame Hindcast as an evaluation testbed and caution against deploying it as a trading tool. All released artifacts are intended for research use only.

Appendix Contents↩︎

9 
9.1 
9.2 
10 
10.1 
10.2 
10.3 
11 
11.1 
11.2 
11.3 
12 

9 Backend Details↩︎

9.1 Tool Surface↩︎

Table 1 lists the five tools exposed to the Retrieval agent. All five accept the same filter block (cutoff \(t_0\), document type, optional subreddit, author, month, and start-time bounds) and inherit the row-level cutoff guarantee of §3.3.

9.2 Retrieval Latency↩︎

The partition selector of §3.3 restricts each dense query to the monthly partitions admitted by the cutoff \(t_0\), so retrieval cost tracks the number of partitions admitted by the cutoff, independent of total corpus size. Table 2 measures this against the live index. Per-query latency rises monotonically with the admitted partition count, from a median of \(0.14\) s at three partitions to \(0.84\) s at all 25. The cutoff used in our experiments, the archive’s final day (2026-01-31), sits at the top of this range, since it admits nearly every partition. Latencies are end-to-end per query, comprising query embedding, per-partition HNSW search, and merge. They are measured over the running retrieval server on a single GPU and averaged over 32 queries per cutoff, so absolute values will vary with hardware and load.

10 Pipeline Details↩︎

This appendix gives the prompts and per-call configuration that instantiate the two pipelines of §5. The tool surface they call through is documented separately in Appendix 9.1.

10.1 Zero-shot (Vanilla) Prompt↩︎

The vanilla pipeline has no retrieval. The model receives the system line below, followed by the per-market user template.

10.1.0.1 System.

You are a forecaster. Predict the probability
of the event.

10.1.0.2 User template.

Question: {question}

Resolution criteria:
{description}

Forecast cutoff date: {cutoff_date}

Output a single JSON object only, matching:
{
  "choice": "Yes" | "No",
  "prob_yes": 0.0-1.0,
  "prob_no": 0.0-1.0,
  "confidence": 0.0-1.0,
  "reasoning": "brief"
}

10.2 Retrieval-Agent Prompts↩︎

The agent pipeline runs in two stages. A research stage drives the ReAct tool loop and writes an evidence report. A forecasting stage turns that report into a calibrated forecast under the same schema as the vanilla pipeline. Both stages share a cutoff date set to the simulated query time \(t_0\).

10.2.0.1 Research system.

You are preparing evidence for a forecasting
task.

Use ONLY the local Reddit tools. Do not use
web search, browser tools, or outside
knowledge as evidence. Every search must
respect the provided cutoff date. Gather
broad evidence first, then follow up on
concrete names, dates, organizations, IDs,
and claims found in the first pass.

10.2.0.2 Research user template (abridged).

Cutoff date for all tool calls: {cutoff_date}

Task:
{prompt}

Options:
{options}

Research requirements:
- Generate multiple diverse research
  questions instead of searching the
  question verbatim.
- Use many local Reddit tool calls: broad
  vector searches first, then targeted
  follow-ups.
- Retrieve full post/comment/thread/author
  context when search hits look relevant.
- Produce an evidence report with sections:
  Key facts, Evidence for each option,
  Evidence against each option, Uncertainty,
  and Suggested probability direction.
- Do not use information after the cutoff
  date.

Local Reddit tool rules:
- Available subreddits are exactly:
  {AVAILABLE_SUBREDDITS}
- search_database must use
  doc_type="submission" only.
- Valid month format is "YYYY-MM" only.

10.2.0.3 Forecast system.

You are a calibrated forecasting judge.

You receive a Polymarket-style question plus
a local Reddit evidence report. Your job is
to produce one forced-choice forecast with
calibrated probabilities.

Forecasting approach:
1. Form your own base-rate prior using world
   knowledge, the resolution criteria, and
   the timing/cutoff.
2. Compare the strength and asymmetry of
   evidence on each side. Move away from the
   prior in proportion to that asymmetry;
   confident moves require asymmetric
   evidence, not absence of evidence.
3. Treat indirect/contextual signals as
   actionable when they consistently lean
   one way.

Return ONLY a single JSON object matching
the requested schema.

10.2.0.4 Forecast user template (abridged).

Step 1 sets a prior anchor based on a five-category classification of the question. Step 2 walks the model through tally-based updates from the evidence report. Step 3 emits the same JSON schema as the vanilla pipeline.

Question: {question}
Resolution criteria: {description}
Allowed options: {options}
Forecast cutoff date: {cutoff_date}

Step 1 - PRIOR (ignore Reddit evidence for
this step). Pick a specific anchor from one
of the five categories below:
  1. Narrow numeric range       -> .05-.20
  2. Underdog / against-trend   -> .10-.30
  3. Structurally uncertain     -> .40-.60
  4. Favorite / with-trend      -> .55-.85
  5. Broadly likely outcomes    -> .70-.95
State this prior P(Yes) as the first
sentence of "reasoning".

Step 2 - ADJUST using the Reddit evidence:
{report}

- Clear asymmetry: move the prior toward
  the stronger side.
- evidence_quality is "none" or "weak" with
  no asymmetry: keep the Step-1 prior,
  unless it was extreme and anchor-driven.
- Indirect/contextual signals ARE actionable
  when they consistently lean one way.

Step 3 - Output the final forecast as a
single JSON object matching the requested
schema.

10.3 Decoding and Retrieval Configuration↩︎

Table 3 lists the per-call decoding settings. Top-\(p\) is left at the vLLM default (\(1.0\)) throughout. Thinking is left at each model’s default (on for the Qwen3 family), and any <think> span is stripped from a reply before it is parsed.

10.3.0.1 Retrieval.

The agent runs a default budget of 3 rounds. Round one issues six seed queries, and each subsequent round adds four to six follow-up queries. All searches are dense (vector) over submissions (§3.2). The backend returns the top \(20\) hits per query, and the agent expands the \(20\) highest-ranked submissions with their full post text and the author’s recent history (up to five posts and five comments each). The loop stops early when the running confidence reaches \(0.7\) or the round budget is spent.

10.3.0.2 Forecast parsing.

The final forecast is read from a JSON object through a four-step fallback. We first parse the main forecast, and if that fails we run a JSON-repair pass, then a compact re-prompt, and finally a strict JSON-only re-prompt. Across all 9 models this leaves at most three of 216 markets without a parsed prediction in a run, so truncation of the forecast pass is not a material source of missing data.

11 Full Results↩︎

11.1 Per-Run Numbers↩︎

Table 4 reports accuracy and Brier for all five consistency runs behind each cell of Table ¿tbl:tab:headline-results?.

11.2 Coverage Bin Sizes↩︎

Table ¿tbl:tab:bin-sizes? reports the mean coverage bin sizes per model (S/W/N), averaged across the five Agent runs that produced them.

11.3 Lookback-Fraction Sweep↩︎

Table 5 reports the per-cell accuracy and Brier for the lookback-fraction sweep of §6, i.e.the data plotted in Figure 5. Figure 7 plots the accuracy half of the same sweep, which tracks the Brier picture.

12 Use of AI Assistants↩︎

We used a large language model solely as a writing aid, to polish the wording of the prose. It was not used for research ideation, experimental design, code generation, data analysis, or interpreting results, and all technical content and claims are the authors’ own. The authors reviewed and verified all text and take full responsibility for the final manuscript.

References↩︎