Executive Summary
- Index Launch. Artificial Analysis released Intelligence Index v5 (AAv5), testing frontier models across four equal 25 percent categories.
- Closed Model Lead. Eight closed models beat the top open model Kimi K3.1 Pro, with proprietary models leading by a wide margin in scientific reasoning.
- The Single Score Dilemma. Equal category weighting creates debatable rankings, putting GPT 6.1 Terra ahead of Claude Fable 5.1 despite distinct user strengths.
- Private Benchmark Shift. Evaluators moved to hidden prompts to stop benchmaxxing, trading public community auditing for prompt privacy.
- Changing Lab Focus. Frontier labs like Anthropic and OpenAI prioritize internal R and D automation over public boards, while smaller labs depend on index rankings.
Making benchmark scores go up is easy. Knowing what to test at the true edge of AI is hard.
Scoring models costs money. Yet tech teams know how to push numbers higher. They have a clear playbook for that work. What is far harder is building tests that stay useful for more than three months.
Testing large language models independently is tough work. The field moves fast. Evaluators cannot just stamp a score on existing models. Instead, they must turn user tests, community views, and scientific checks into a single score.
The recent update to the Artificial Analysis Intelligence Index, called v5, shows this struggle. It highlights the value of independent tests. It also reveals the limits of trying to put one number on model intelligence.
Table of Contents
The Single Score Trap in AI Evaluation
Every benchmark team faces a core puzzle. Tech buyers and investors want one clean number to compare models. One number is easy to talk about in meetings. It makes charts simple to read. But boiling model skills down to a single score forces big trade-offs.
Artificial Analysis divides its v5 index into four equal parts. Each part holds a 25 percent weight.
The knowledge work quadrant checks office tasks through suites like AA-Briefcase and GDP.pdf. The coding quadrant tests software skills through FrontierCode 2.0 and Terminal-Bench 4.0. The science quadrant tests deep research through Terminal-Bench-Science and CritPt v2. The general quadrant tests broad knowledge and truthfulness through AA-Omniscience v1.1 and HLE-Diamond.
Because each bucket has the same weight, the final rank mirrors clear choices about what matters most. For example, the total score puts OpenAI’s GPT 6.1 Terra above Claude Fable 5.1. Recently, Google Gemini 4 Carbon ranked number one on the index at a score of 59. Change the weights to favor coding or writing, and the leader board shifts at once.
The Growing Gap Between Closed and Open Models
The new rankings show a wide split between closed and open-weight models. Eight closed models beat the top open model, Moonshot’s Kimi K3.1 Pro. Past index runs showed open models closing in. The v5 tests show that closed models still hold a clear lead at the frontier.
This gap varies by task. In coding, open models remain strong. They keep pace on daily coding jobs where training data is easy to find. But open models fall behind in science. On tests like Terminal-Bench-Science and CritPt v2, closed models win by a wide margin.
Still, this gap does not mean open models fail. The real goal of open models was never just to beat frontier labs on day one. Their true role is to cut inference costs down to cheap hardware prices. They let firms own their data. They also let teams run custom agents without sharing private code.
Comparison Factsheet: Artificial Analysis Index v5 Architecture
The factsheet below outlines the four core quadrants in the updated Artificial Analysis v5 index, the specific evaluation suites, and how open models compare against closed frontier systems.
| Quadrant | Weight | Key Test Suites | Skills Evaluated | Model Spread |
|---|---|---|---|---|
| Knowledge Work | 25% | AA-Briefcase, AutomationBench, GDP.pdf | Office workflows and agent actions | Closed models lead |
| Coding | 25% | FrontierCode 2.0, Terminal-Bench 4.0 | Bug fixes and code creation | Open models close |
| Science | 25% | Terminal-Bench-Science, CritPt v2 | Hard scientific reasoning | Wide closed model lead |
| General | 25% | AA-Omniscience v1.1, HLE-Diamond | Factual recall and low hallucination | Closed models lead |
Benchmaxxing and the Risk of Private Tests
Public tests face a big hurdle called benchmaxxing. When prompt sets are public, labs can tune their training data to match test patterns. Models then score high on tests without gaining real smarts.
To fight this, Artificial Analysis shifted toward private tests with secret prompts. Secret tests stop direct gaming. But they bring a new cost in openness.
When test prompts are secret, outside researchers cannot check the questions. They cannot spot typos, bad answers, or biased scoring. The community cannot help fix bugs in the benchmark. Users must place full trust in the test maker.
How Lab Goals and Evaluator Business Models Shift
Top labs like OpenAI and Anthropic have moved past public leader boards. As seen with Anthropic driving revenue through Claude Code, frontier engineering teams prioritize AI research automation. They build custom tools for corporate clients who spend billions. Public ranks matter far more to smaller labs. Challengers need high benchmark scores to build their brand and raise funds.
At the same time, testing firms face complex business models. Startups like Artificial Analysis and Arena earn fees from frontier labs to test models before release. Academics and state groups offer neutral views. But private startups have the speed, chips, and access needed to test new models on day one.
The best fix is not one big index. The field needs many focused benchmarks. Tools like CursorBench for code, Vals AI, and Epoch AI give a fuller picture. Multiple tests help buyers see where models shine without relying on one single score.
Explore Engineering Utilities and AI Solutions
Test payload formats, token calculations, and text transformations using our suite of Free Online Developer Tools.
Building enterprise model evaluations, fine-tuning open weights, or setting up agent workflows? Consult with TeckScaler’s enterprise IT specialists →
Frequently Asked Questions
What is the Artificial Analysis Intelligence Index v5?
It is an independent benchmark index for evaluating frontier large language models. The v5 suite weights knowledge work, coding, science, and general capabilities at 25 percent each.
Why do open-weight models lag in AAv5?
Open models remain close to closed models in coding, but they trail significantly in advanced scientific reasoning on tests like Terminal-Bench-Science and CritPt v2.
What is benchmaxxing in AI model testing?
Benchmaxxing occurs when labs tune post-training data or reinforcement learning to match public test distributions, inflating benchmark scores without improving general capability.
Why are evaluators shifting to private benchmarks?
Hiding prompts stops labs from gaming tests. However, it reduces public transparency because outside researchers cannot inspect test items for errors or bias.
