BayesBench Evaluates LLM Belief Trajectories in Multi-Turn Evidence Accumulation

BayesBench, a new evaluation framework, examines how large language models (LLMs) update their beliefs during multi-turn conversations. Developed by Ankur Samanta and a team of researchers, this framework was introduced in a paper submitted on June 29, 2026, to explore the efficacy of LLMs in processing sequential evidence.

Understanding Belief Updates in LLMs

In typical applications, LLMs engage in multi-turn conversations, where each interaction provides additional evidence that should ideally reduce uncertainty regarding their environment. The challenge lies in how well these models can infer hidden variables and adjust their beliefs as new information is presented.

The BayesBench framework offers a structured approach to assess this capability, contrasting LLMs' belief updates with those of a rational Bayesian reasoner. This evaluation is crucial as most existing assessments focus solely on the final responses given by models, ignoring the process of belief adjustment.

BayesBench’s Simulation Environments

BayesBench consists of three progressively complex tasks designed to test LLMs:

• Bayesian estimation: The model infers an unknown parameter based on sequential evidence.
• Bayesian prediction: The model uses inferred beliefs about a latent variable to forecast outcomes.
• Latent-framed Bayesian prediction: Observations are filtered through a user persona, requiring joint inference over the latent state and persona.

These tasks allow researchers to evaluate how effectively LLMs accumulate evidence and update their beliefs over time.

Findings from the Evaluation

Across seven LLMs ranging from 3B to 70B parameters, the results indicate that scaling improves latent inference and evidence accumulation. However, the extent to which these updates align with Bayesian posteriors varies, particularly in downstream prediction tasks. This highlights a significant gap between inferring latent structures and rationally updating beliefs about target outcomes. The study demonstrates that while LLMs can improve in certain aspects, they still face challenges in consistently applying their inferences to practical predictions.