Netlify tested 11 models with the same prompt: one Opus 5 run consumes the entire free plan
On August 12, Netlify published an experiment that almost nobody does publicly: it ran the same build prompt three times with 11 different models on its Agent Runners, measured the credits each run consumed, and left the 33 generated sites online for anyone to open.
The prompt, word for word:
“Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself.”
A coffee shop landing page. No database, no auth, no weird build step. The most boring task you can think of.
The consumption range was 2.4 to 519 credits. A factor of ~216× to produce the same page.
The table, and what Netlify didn’t convert to dollars
The post publishes credits. Netlify’s billing documentation publishes the conversion rate: 1 USD = 180 credits. Nobody puts the two together, so here’s the table with the missing column.
| Model | Run 1 | Run 2 | Run 3 | Average | ≈ USD per run |
|---|---|---|---|---|---|
| Claude Opus 5 | 253 | 249 | 1.055 | 519 | $2.88 |
| Claude Sonnet 5 | 81 | 245 | 103 | 143 | $0.79 |
| GPT 5.6 Sol (low effort) | 173 | 158 | 92 | 141 | $0.78 |
| Gemini 3.6 Flash | 109 | 91 | 111 | 103 | $0.57 |
| Kimi K3 | 125 | 95 | 86 | 102 | $0.57 |
| Gemini 3.1 Pro | 57 | 52 | 49 | 53 | $0.29 |
| GPT 5.6 Terra | 43 | 23 | 49 | 39 | $0.22 |
| DeepSeek V4 Pro | 47 | 30 | 33 | 37 | $0.21 |
| GLM 5.2 | 15 | 42 | 24 | 27 | $0.15 |
| Kimi K2.7 Code | 21 | 18 | 17 | 19 | $0.11 |
| DeepSeek V4 Flash 0731 | 3.4 | 1.3 | 2.5 | 2.4 | $0.013 |
And now the number that turns this from curiosity into decision: Netlify’s Free plan comes with 300 credits per month.
An average Opus 5 run is 519. It’s not that Opus consumes an important part of your free month: it’s that a single run doesn’t fit. Not even the cheapest of Opus’s three runs in this test — 249 credits — leaves you room to try a second time. On the Personal plan (1,000 credits) you get two Opus runs and the month is done.
With DeepSeek V4 Flash, those same 300 credits from the free plan give you around 125 complete runs.
A billing nuance worth clarifying: the 180 credits per dollar is the rate at which Netlify converts model consumption into credits, not the price you pay for credits. Additional packs go for $10 per 1,500 credits on Pro and $5 per 500 on Personal. If you’re paying with purchased credits, that average Opus run costs you $3.46 on Pro and $5.19 on Personal, not $2.88.
The average isn’t what you budget for
Look at the Opus row again: 253, 249, and 1.055. Same prompt, same task, three runs. The third one consumed 4.2 times what the two before it did.
That’s the most uncomfortable finding of the experiment and the post itself says so: Opus tends to spike well above its own baseline, with no guarantee of a better result. There’s no prior signal to warn you which of the three runs you’re going to get.
It’s not unique to Opus. Sonnet 5 did 81 / 245 / 103 — the expensive run cost 3 times the cheap one. GLM 5.2 was 15 / 42 / 24. Variability is everywhere; what changes is the scale of damage when the expensive run is from a model that’s already expensive to begin with.
The practical consequence is direct: if you have a team with a monthly credit budget, dimensioning it against the average will leave you short. The number you plan for is the worst case observed, not the mean. With Opus, that means budgeting 1.055 credits per run and hoping it’s usually cheaper.
Netlify has a direct lever for this that’s worth flipping sooner rather than later: the AI Credit Usage Limit, a spend cap on inference for Agent Runners. When you hit it, new runs don’t start and active ones stop. It’s the closest thing to a circuit breaker here, and in a model with a 4.2× spread it’s exactly what you want in place.
What about quality?
This is where the experiment gets honest, and also where you need to read it carefully.
Netlify gives Opus 5 the best verdict: it describes it as “delightful, and full of detail in both its visual design… and the custom map at the bottom”. Sonnet 5 lands a step down — “still some delightful detail in each of these, just less so (and less content in general)”. GPT 5.6 Sol in low effort mode, the post says, beats Sonnet on basic design intuition, which is interesting on its own: OpenAI’s top model in its cheapest mode performed better than Anthropic’s mid-tier model, and for nearly the same consumption (141 vs 143 credits).
On the cheap side there are concrete flaws: DeepSeek V4 Pro delivered a broken image, and the post notes that this type of failure is much less likely in large commercial models. Gemini 3.1 Pro gets the driest verdict of all: “there’s really nothing to see here”. Kimi K3 and K2.7 Code don’t shine either.
But the conclusion Netlify itself draws isn’t “pay for the expensive one”. It’s that with simple models you can work iteratively — multiple turns guiding the result — instead of waiting for the turn-key solution from a single pass that Opus offers. And with the math above, you have room for countless iterations: 519 credits of Opus are about 216 runs of DeepSeek V4 Flash.
The 33 sites are published and navigable. Open them and make your own judgment instead of inheriting Netlify’s — that’s literally why they left them online: the-coffee-shop-brief.netlify.app
What this test doesn’t measure
Three caveats, and the first one comes from the post itself.
It’s a test of design and copy, not architecture. Netlify says it plainly: “this design-and-copy-focused test”. A static landing page doesn’t exercise any of what makes a real application hard — state, integrations, refactors on existing code, debugging. The post explicitly admits that Kimi K3 is underrepresented by the prompt type, and that to do it justice would require a completely different set of prompts designed for a complex web app. A follow-up with more advanced cases is announced.
Agent Runners credits add up two meters, not one. AI inference (at each model’s token rate) plus compute from the execution environment, measured in GB-hours. So the numbers in the table aren’t tokens in disguise and you can’t derive token consumption from them.
And you can’t verify the Opus 5 rate in the public docs. Netlify’s model pricing table is updated to July 15, 2026 and lists claude-opus-4-5 and claude-sonnet-5, but not claude-opus-5. The general 180 credits per dollar conversion is published and is what I use above; the per-million-token rate for Opus 5 specifically is not.
AXIS: the open source part
As a bonus, the framework Netlify uses to evaluate models automatically — AXIS — is released under MIT license at github.com/netlify/axis, with contributions from Auth0 and Resend.
The idea is a Lighthouse for agents: it measures how well a service works for an AI agent, scoring four independent dimensions (Goal Achievement, Environment, Service, Agent). You install it via npm, configure a JSON with scenarios and agents, run axis run and it gives you the full transcript of what the agent did plus a report with scores in .axis/reports/.
If Netlify’s experiment made you want to run your own with your own prompts — which is what you should do before deciding which model your team works with — that’s the tool and you don’t have to write it.
Have you ever measured how much the same task costs you on two different models, or do you go with the big model by default? Tell us in the comments, especially if you got a run that spiked like the 1.055 credit one and what you did after.More models, more choice: Comparing 11 different AI models