Find out whether the expensive model writes better code on your PRs.
Stet takes about 40 of your merged PRs, replays them with different models, reasoning levels and instruction files, grades each result on correctness and code quality, and recommends the best setup for quality, cost and time.
See an example report →What should my default reasoning effort be?Claude Code
Haiku 5.5low
Haiku 5.5 at low: score 46.8 of 100 (95% interval 38.3 to 55.3), $0.25 a task, 5 min median.
Run low. Max cost 15× as much for a narrow edge: head to head it won 12 tasks and lost 8.
GPT 6 Lunamax
GPT 6 Luna at max: score 58.8 of 100 (95% interval 49.2 to 67.5), $0.37 a task, 55 min median.
Run max. Max cost 7× as much and won 20 of 24 tasks head to head. Low returned no patch on 6.
The right default depends on the model.
Your merged PRs are the tasks. Your tests decide correctness, and an agent judge scores the rest. Your code is the baseline.
You ask your coding agent
What should my default reasoning effort be?
or click at app.stet.sh.
Stet does everything else.
Runs on the Claude, Codex or Cursor login you already have.
We set up your first trial with you, free.
Each task is a merged change in your repo, with the repo snapshot from just before it merged, a prompt and the tests, which the agent never sees.
Give each candidate the same tasks, tests, and quality criteria.
Tests, then code review. Basically: would a senior engineer want to merge this? Cost per task sits alongside.
You keep only what wins, and every run ends in a decision you can defend.
Recent findings
All researchwon-tied-lost · $ / task
- GPT 6.1 Sol vs GPT 6 SolA cheaper replacement that holds quality.9-10-5 · $1.10 vs $3.01
- GPT 6.1 Sol vs Grok 4.7 HighEdges Grok at about a fifth of the cost.10-5-9 · $1.10 vs $5.76
- GPT 6 Luna, max vs lowMax clearly beats low at about 7× the cost.20-2-2 · $0.37 vs $0.05
- Haiku 5.5, max vs lowMax edges low at about 15× the cost.12-4-8 · $3.85 vs $0.25
Find where you’re overpaying, and prove it.
stetsh