The benchmark changed and the cheap models slipped. The frontier pair did not move

The two best models on the index tied. The interesting thing happened two rungs further down.

Artificial Analysis published version 4.3 of its Intelligence Index on 7 September, and GPT-6 Astra at max effort and Claude Fable 5.1 at max with fallback both landed on the same score.

The gap worth looking at in that chart is not at the top. It is the one between the frontier pair and the cheap models that had been keeping pace with them until last week.

What changed in the test

Two things moved with the version bump. Terminal-Bench went from 2.1 to 4.0, which is 66 multi-step tasks run in a minimal mini-SWE-agent harness instead of Terminus 2. And a banking benchmark was replaced by AutomationBench-AA, Artificial Analysis's implementation of Zapier's private set of business-workflow tasks.

The weight given to evaluations with private tasks or answers rose from 40 to 45 percent. Artificial Analysis says why in one clause. The change "adds more private test sets to prevent gaming, and reduces saturation."

Who moved when the test did

SemiAnalysis said the quiet part the same day. Gemini 3.8 Flash and Muse Spark 1.3, it wrote, "are two of the most clearly benchmaxxed models we've seen yet. Despite being comparable to both GPT-6 and Fable 5.1 on Terminal Bench 2.1, their Terminal Bench 4.0 performance is markedly worse."

Both had been sold as frontier-class at a fraction of the price, and the market (reasonably enough, on the evidence available) believed it. Erin Woo reported on 1 September that Google employees "prefer it to Opus on coding in side-by-side testing." Cursor added Flash on 2 September and Muse Spark on 8 September, the day Meta launched its consumer agent on that same model.

Give them harder terminal tasks and a test set nobody has seen, though, and the gap to the frontier comes back.

Not everyone accepts the index either. Antirez, who does not soften a question: "Do you believe me now, after the Astra issue, that Artificial Analysis numbers are broken?" Boris Cherny at Anthropic was measuring a different axis entirely, and was pleased with what he found: "OpenAI's new model is roughly on par with Gemini Flash and Opus 4.8 on prompt injection risk," adding that "evaluating and naming other labs turns out to be a great way to encourage them to train more aligned models."

One score hides the dial

So why does a single number mislead? Because each of these models has an effort setting, and the setting seems to move the result about as much as the model does.

Astra spans 46 to 53 across its effort levels, and takes anywhere from under two minutes to over eight per task. Fable covers a similar range of scores over a wider spread of time, stretching to roughly twelve minutes at the top. Thibault Sottiaux at OpenAI puts the practical version plainly: "GPT-6 Astra on low performs better than GPT-5.6 Sol on high."

Simon Willison's pelican drawings agree in miniature. Astra at low beat every Sol setting for 9.55 cents, despite Astra's list price being double Sol's, because it burned far fewer tokens at each level. Artificial Analysis has OpenAI holding "the majority of the cost-efficiency frontier," with the open-weights field trailing where we priced it last week.

If you are choosing a model this month

Choose by effort level and time per task, not by the headline integer. Astra at low or medium is the cost-efficient default for agentic coding on current numbers. Fable at xhigh is the pick when a job probably justifies twelve minutes of thinking. Flash and Muse Spark are worth their price on work that looks like the old Terminal-Bench, and unproven on work that looks like the new one.

The tie at the top is barely a finding. Two labs with the same compute class and the same benchmark targets landing on the same integer mostly tells you the index measures what both of them optimise for.

The finding is one rung down. A harness change plus a private test set moved the cheap models and left the expensive ones exactly where they were, which is about the clearest evidence this year that a score below the frontier is partly a measure of how hard a lab tuned for the test.

We would expect a future index with a majority private weighting before the end of 2026, and at least one Flash-class model to drop more than five points on the transition. If either of them climbs back to within three points of the frontier on the new Terminal-Bench in the same harness, the benchmaxxing charge was wrong and so was our reading of it.

Until somebody publishes a private-set number, treat a cheap frontier-class claim as a claim about a test.