On July 9, 2026, a study led by Harvard psychology researcher Ashwini Ashokkumar revealed that large language models (LLMs) like GPT-4 can effectively predict how individuals respond to surveys. However, this capability does not equate to a genuine understanding of human behavior, highlighting the limitations of AI in social science research.
Understanding AI Predictions in Social Science
The research assembled 70 real experiments conducted in the United States, involving nearly 120,000 participants. By providing GPT-4 with descriptions of hypothetical respondents alongside survey questions, the researchers sought to estimate responses based on various conditions. The findings showed a strong correlation between GPT-4's predictions and actual outcomes.
This is significant as it suggests that LLMs can identify meaningful patterns within survey data. Nevertheless, Ashokkumar warns that while AI may offer useful forecasts, it does not replace the nuanced understanding derived from human research. “A system that predicts human responses is not necessarily a system that understands human behavior,” she stated.
Limitations of AI in Predictive Modeling
The study also revealed that while GPT-4 was adept at ranking the effectiveness of different interventions, it consistently overestimated the impact, predicting effects that were approximately twice as large as actual results. This discrepancy is crucial for researchers who rely on accurate data to inform their studies.
Researchers often conduct small pilot studies before embarking on larger, more expensive experiments. GPT-4's forecasts could serve as a valuable supplement in these instances, helping refine interventions and guiding where to allocate human resources. However, combining LLM predictions with human insights proved to yield the most accurate results.
The Debate Over Synthetic Respondents
The rise of “synthetic respondents” in polling and market research raises ethical questions. While supporters argue that AI can provide faster and cheaper testing, critics caution against presenting simulations as real public opinion. For example, a politician might seek public sentiment on a new tax policy, relying on AI for quick answers that do not reflect actual public sentiment.
Conventional surveys gather responses from real individuals, while AI-generated samples depend on patterns from training data, lacking the lived experiences that shape public opinion. Ashokkumar's study found that while GPT-4 performed well across various demographic groups, it showed biases favoring certain segments, such as white and Republican samples.
- 70 real experiments analyzed
- Involvement of 120,000 participants
- GPT-4 predictions often twice as large as actual results
As AI continues to evolve, its role in research will likely expand, but researchers must remain vigilant about the limitations and ethical implications of using synthetic data.
🤖 This article was rewritten by Feed and Figures' editorial AI from a report originally published by Phys.org. Facts and quotes are preserved from the original; the rewrite focuses on clarity and structure. For the unedited original, see the source link below.
