The July launch of Android Bench — its LLM benchmark leaderboard for Android growth duties — is now primarily based upon the Harbor framework for evaluating fashions, Google has introduced.
In its weblog publish saying the updates, the corporate famous that when Android Bench was launched in March, the aim was to supply insights into mannequin capabilities, provide AI choices and to induce bettering the fashions. On this launch, eight fashions have been added to the leaderboard — Claude Fable 5, Sonnet 5 and Opus 4.8; GLM 5.2; Kimi K2.7; MiniMax M3; and Qwen 3.7 Plus and three.7 Max.
“From the start, we’ve valued an open and clear strategy, which is why we made our unique methodology and take a look at harness publicly obtainable on GitHub,” the corporate wrote within the weblog. “Now we’re taking collaboration a step additional by providing you with, the Android developer neighborhood, an opportunity to form Android Bench.”
Builders can now design and submit Android growth duties and see how the fashions deal with the eventualities essential to them, and now can really run benchmark evaluations to check the fashions they use towards Android’s dataset of duties. These duties could be discovered on this GitHub repository and builders can submit evaluations or discover Android’s knowledge at Harbor Hub.
“With an increasing number of choices for agentic growth,” the weblog concluded, “sustaining a cutting-edge benchmark ensures that the AI help you depend on retains getting smarter, extra useful, and more practical.”
