The Format Sensitivity Index (FSI) study by Deep Pankajbhai Mehta on May 2, 2026, highlights significant variances in model scores based on prompt wrapper formatting. The research, which analyzed 140,000 OpenRouter generations across seven QA tasks, indicates that formatting can drastically affect accuracy.
Understanding the Format Sensitivity Index
The Format Sensitivity Index measures the accuracy range influenced by different wrapper choices. It reveals that even minor formatting differences can lead to substantial discrepancies in model performance. The study found that the mean FSI varied over 30 times across different models, largely due to compliance failures.
In addition to FSI, the research introduced the Parseability Sensitivity Index (PSI), which evaluates the range of answer parseability. This dual-metric approach provides a more comprehensive understanding of how prompt wrappers affect the overall effectiveness of language models.
Key Findings from the Study
- 140,000 OpenRouter generations analyzed.
- Seven QA tasks and five wrapper families involved.
- Models ranged from 7B to 72B parameters.
- Mean FSI varied by over 30x across models.
The research concluded that parseability is a strong predictor of accuracy, even after accounting for task, model, and wrapper variations. This finding emphasizes the importance of reporting accuracy alongside wrapper variance and compliance to avoid statistical fragility.
Practical Recommendations for Benchmarking
Mehta's study provides practical recommendations for improving benchmarking practices. These include ensuring compliance with formatting standards and considering the impact of wrapper choices on model performance. By adopting these practices, researchers can enhance the reliability of their findings and contribute to more robust AI development.
In summary, the Format Sensitivity Index and Parseability Sensitivity Index offer valuable insights for developers and researchers in the field of artificial intelligence, highlighting the critical role of prompt wrapper formatting in LLM benchmarking.
🤖 This article was rewritten by Feed and Figures' editorial AI from a report originally published by arXiv AI. Facts and quotes are preserved from the original; the rewrite focuses on clarity and structure. For the unedited original, see the source link below.
