Talarion is now available in the ChatGPT plugin directory.
TalarionTalarion

Talarion · PriorBench Technical Report · July 2026

Introducing PriorBench: A New Frontier in Search Evaluation

Nicholas HopeAvirath SundaresanLiam McDonald

PriorBench is a search benchmark consisting of 150 research questions (RQs) and 4,154 associated evaluation questions (EQs) spanning from September 2025 to July 2026. Withheld questionnaires averaging 27.7 ± 6.1 EQs were produced for each RQ to capture a necessary (although possibly insufficient1) set of priors upon which any credible response to the RQ should be predicated. Search techniques were evaluated by assigning an RQ to a research LLM (gpt-5.4), allowing it to search for relevant information, and then evaluating the quality of its understanding by eliciting a forecast on each EQ. Since the events in question all represent true positives2, the Brier Score reduces to (1 − p)². Thus PriorBench is designed explicitly to measure an LLM's latent understanding of world-state at inference time, as well as the ability of search systems to return the information that is most needed, rather than simply the information that an LLM knows to request.

Fig. 1 — Accuracy vs Input Token Count

Accuracy is measured as 1 - Brier Score. Talarion dominates on token efficiency: talarion_single returns more information in a single call (~1,100 tokens) than Parallel or Exa at 30,000 - 60,000+ even in token efficiency mode (Excerpts and Highlights, respectively). Sources seen in a prior search turn were subsequently dropped to ensure that the input tokens measured above contain 0 duplicates.

Fig. 2 — Accuracy by Search Technique

EQ forecasting reveals a highly inaccurate internal model of world state at inference time; a maximum entropy prior of 0.5 would yield a Brier Score of 0.25. Talarion is substantially the best performer but still does an imperfect job correcting the deficiencies which impede gpt-5.4 even after 10 rounds of web search.

Fig. 3 — Comprehension Across Research Questions
Exa (25.7k tokens/RQ)Parallel (11.0k tokens/RQ)Tavily (23.4k tokens/RQ)Brave (15.1k tokens/RQ)Talarion (8.1k tokens/RQ)

Histogram of RQs scored by mean accuracy. Median per-RQ accuracy (p25–p75): Talarion 0.90 (0.86–0.94) vs Exa 0.74 (0.65–0.83) vs Parallel 0.78 (0.63–0.84).

Sample Research Questions and Evaluation Questions

Each evaluation question is phrased as a positive assertion to which the responding LLM assigns a probability at inference time.

What are California's most urgent water issues?

33 evaluation questions
  • Active land subsidence continued in California’s Sacramento River, San Joaquin River, and Tulare Lake hydrologic regions during 2025.
  • California American Water requested rate increases of 17.2% for 2027, 5.1% for 2028, and 5.8% for 2029.
  • The March 2026 Central Valley Project allocation for Westlands Water District and other south-of-Delta irrigation contractors was only 20% of contract supply.
  • California made $46 million in Proposition 4 funding available in June 2026 to address the Tijuana River sewage crisis.
  • In May 2026, the EPA proposed retaining federal drinking-water limits for PFOA and PFOS while dropping standards for PFNA, PFHxS, GenX, and PFBS.
  • The San Diego County Board of Supervisors had declared an emergency over the Tijuana River pollution crisis 22 times by April 2026.
  • The California State Lands Commission approved a June 2026 lease allowing California American Water to construct and convert slant wells for a desalination project in Marina.
  • Central Valley groundwater storage declined by nearly 2 million acre-feet during 2025.
  • The earliest year California Department of Water Resources officials said Delta Conveyance Project construction could begin was 2029.
  • The Lower Basin states proposed a three-year bridge agreement to conserve more than 3.2 million acre-feet of Colorado River water by 2028.
  • A wastewater pipeline collapse in Tijuana in late May 2026 caused sewage to spill into the Tijuana River.
  • The EPA announced $15.7 million in June 2026 for seven Southern California projects addressing PFAS contamination.
  • The December 2025 revised Bay-Delta Plan proposal retained both a regulatory compliance pathway and a voluntary-agreement pathway.
  • California’s governor announced a February 2026 state water plan calling for millions of acre-feet of water to be banked through 2050.
  • Santa Cruz released a June 2026 water-management plan that included storing treated surface water in an aquifer.
  • Scientists warned that the Colorado River bridge proposal’s conservation cuts amounted to only about half the reduction needed to stabilize the river.
  • Federal officials announced in June 2026 that the federal government planned to control Colorado River water distribution for the next decade because the seven basin states had not reached agreement.
  • The current Colorado River operating guidelines were scheduled to expire in August 2026.
  • The California Department of Water Resources found no measurable snow at its Sierra Nevada survey locations on April 1, 2026.
  • An atmospheric river brought high water and flooding to California at the start of January 2026.
  • The Eastern Tule Groundwater Sustainability Agency agreed to issue farmers no groundwater credits or allocations for Water Year 2026.
  • A March 2026 Salinas Valley feasibility study found that a proposed seawater-intrusion project could offset about 11,000 acre-feet of groundwater use annually.
  • The National Marine Fisheries Service issued a June 2026 biological opinion finding that the proposed Delta Conveyance Project would not jeopardize salmon, steelhead, or green sturgeon.
  • California Groundwater Sustainability Agencies reported extracting 13 million acre-feet of groundwater in 2025.
  • The Bureau of Reclamation and water agencies in California, Arizona, and Nevada signed a June 2026 memorandum to explore interstate Colorado River water exchanges.
  • A federal Colorado River proposal contemplated cuts of up to 40% in supplies to California, Arizona, and Nevada.
  • The California Department of Water Resources had withdrawn both of its validation actions for Delta Conveyance Project financing by June 2026.
  • California’s major reservoirs stood at 117% of their average level for that time of year when the State Water Project allocation was raised to 45% in May 2026.
  • Negotiators for the seven Colorado River Basin states ended their February 13, 2026 talks without reaching a new management agreement.
  • The Department of Water Resources raised the 2026 State Water Project allocation to 45% of requested supplies on May 15, 2026.
  • The Salinas Valley Basin Groundwater Sustainability Agency faced an August 2026 deadline to submit its groundwater sustainability plans to the Department of Water Resources.
  • California’s statewide snowpack measured just 18% of its historical average on April 1, 2026.
  • California’s overall groundwater storage declined by 1.5 million acre-feet during 2025.
  1. 1.That is, given the open-ended nature of the RQs, informational sufficiency is undefined. An LLM would almost certainly further benefit from additional priors not captured by the provided EQs. ↩
  2. 2.The poor performance of candidate LLMs with and without information retrieval suggests that dishonest guessing is not an issue. Training on PriorBench would require the introduction of true negatives. ↩