Yes-no bias in large language models stems from answer framing, not moral shifts

On July 6, 2026, researcher Haonan Huang submitted a paper exploring the yes-no bias in large language models (LLMs). The study reveals that this bias is influenced by the order and wording of answers rather than actual changes in moral judgment. This finding challenges existing assumptions about the decision-making processes of AI systems.

Understanding Yes-No Bias in AI

The yes-no bias observed in LLMs has raised questions about how these models interpret moral dilemmas. Huang's research highlights that the framing of questions significantly impacts the responses generated by AI. Specifically, a simple shift in wording can lead to a noticeable change in the model's answer, which is not seen in human decision-making.

The study introduces a psychometric battery to assess this bias, employing a method called crossed symmetrization. This approach flips logically irrelevant factors in balanced pairs across various question forms, enabling a clearer understanding of the models' internal moral scales.

Key Findings on Model Performance

Huang's findings reveal that larger models exhibit a nearly format-invariant stance with a cross-form incoherence rating between 0.12 and 0.21 on a ±1 axis. In contrast, smaller models demonstrate inconsistencies in their responses based on specific formats. The research indicates that the last-printed option in yes-no questions creates a significant order bias, diverging from typical human decision-making patterns.

For instance, in Claude models, the order bias ranges from -0.32 to -0.86, while models like GPT-5.5 and Gemini show minimal bias. This suggests that various models handle moral judgments differently, with some being more susceptible to framing effects than others.

Implications for AI and Moral Judgment

The study emphasizes the importance of understanding how LLMs process moral dilemmas. Huang concludes that to accurately gauge what a model values, it is crucial to assess responses across multiple frames rather than relying on a singular question format.

This research could have significant implications for the development of AI systems, particularly in fields requiring ethical decision-making. By isolating the factors contributing to yes-no bias, developers can work towards creating more reliable and morally aware AI.

• Researcher: Haonan Huang
• Submission Date: July 6, 2026
• Model Performance Ratings: Claude models (-0.32 to -0.86), GPT-5.5 and Gemini (~0)
• Cross-form incoherence: 0.12-0.21