Google has updated its Android Bench tool to include new large language models (LLMs) as of this week. The updated leaderboard features eight new models, including Claude Fable 5 and GPT 5.4, while Google's own Gemini 3.1 Pro has fallen to fifth place. This update aims to enhance the evaluation process for Android app development.
New Models Added to Android Bench
The Android Bench update introduces a range of new LLMs designed to improve performance in Android app development tasks. Developers can now test their own models against a suite of 100 development tasks, allowing for greater flexibility and accuracy in evaluating LLM capabilities.
The new models added to the leaderboard include:
- Claude Fable 5
- Claude Sonnet 5
- Claude Opus 4.8
- GLM 5.2
- Kimi K2.7 Code
- MiniMax M3
- Qwen 3.7 Plus
- Qwen 3.7 Max
Performance Comparison of LLMs
In the latest tests, Claude Fable 5 achieved an impressive accuracy of 84.5%, leading the leaderboard. In contrast, Google's Gemini 3.1 Pro scored lower and is now positioned behind top competitors such as GPT 5.4 and Claude Sonnet 5.
Despite its lower score, Gemini 3.1 Pro is more cost-effective, priced at $87 per run compared to over $130 for leading models like Fable 5. However, Gemini 3.5 Flash has the highest costs on the leaderboard at $165 per run due to a lengthy benchmark runtime of 28 hours.
Community Collaboration and Future of Android Bench
Google aims to foster community collaboration through the Android Bench by allowing developers to submit their own benchmarks and tasks. The introduction of the Harbor framework is designed to simplify the testing process, making it easier for developers to evaluate and share their results.
All previous tests have been re-evaluated using the Harbor framework, resulting in updated scores while maintaining an archive of historical data. Developers interested in contributing can find updated datasets and instructions on the Android Bench GitHub page.
🤖 This article was rewritten by Feed and Figures' editorial AI from a report originally published by Ars Technica. Facts and quotes are preserved from the original; the rewrite focuses on clarity and structure. For the unedited original, see the source link below.
