Office Comprehension Benchmark Introduces First Public Evaluation for LLMs on Office Formats

The Office Comprehension Benchmark (OCB) was introduced on May 29, 2026, by a team of researchers led by Firoz Shaik. This benchmark is the first public evaluation framework designed for assessing large language models (LLMs) on their ability to comprehend Word, Excel, and PowerPoint files in their native formats.

Overview of Office Comprehension Benchmark

The OCB consists of two main tracks: File Fidelity Q&A and Domain Q&A. The File Fidelity track evaluates the structural and visual comprehension of office documents, including tables, charts, and embedded images. Domain Q&A focuses on expert-level reasoning based on real-world industry documents across 12 professional domains.

Each reference answer in the benchmark is broken down into atomic, binary-gradable claims. LLM judges score the responses based on these claims independently. Notably, even the most advanced LLMs currently achieve only about 59.3% accuracy on Domain Q&A, indicating significant challenges in reasoning and comprehension.

Key Features of OCB

The OCB provides a comprehensive evaluation of LLMs through its unique structure. It emphasizes the importance of multi-step analysis and synthesis across documents, which is crucial in real-world applications. The benchmarks are released alongside evaluation tools, judge prompts, and a public leaderboard.

• File Fidelity Q&A: Tests visual perception of office artifacts.
• Domain Q&A: Requires complex reasoning across 12 domains.
• Performance Measurement: LLMs currently achieve 59.3% accuracy.

The introduction of OCB marks a significant step forward in the evaluation of AI's capabilities in understanding office documents. By providing a structured approach to measure comprehension, it opens up new avenues for research and development in this field.

Impact on AI Research and Development

The establishment of the Office Comprehension Benchmark is expected to influence future AI research significantly. It encourages the development of LLMs that can better comprehend and interact with office software, which is essential for various professional industries.

As businesses increasingly rely on AI tools for data analysis and document management, benchmarks like OCB will be pivotal in guiding improvements in AI systems. The release of the dataset and tools also promotes community engagement and collaboration in enhancing AI performance.