Token-cost benchmarks: AI sessions with and without the code map
Every question you ask an AI assistant about your codebase starts with the assistant re-reading your codebase — and that re-reading is most of the token bill. Vibgrate AI Context replaces it with answers from a pre-built code map. This page publishes what that is worth, measured: the same tasks, the same model, with and without the map, at equal verified success — including the repo sizes where the map is not worth it.
~30% fewer tokens, same results
The same coding tasks, the same model, with and without the Vibgrate code map — on a ~300-file codebase. We count only the tasks both runs solved, so it is fewer tokens for the same result, never “cheaper because it gave up”.
Measured with a frontier coding model. On tiny repositories grep already finds the code and the map costs more — we publish that part of the curve too.
How the numbers are measured
Same model, twice
Each coding task runs twice with the same model on two fresh copies of the repo: once with generic file tools (list, search, read, write), once with Vibgrate AI Context (vg serve) for discovery.
Only discovery differs
Both runs can read and write files. The only difference is how the model finds the right code: walking and grepping the tree, or asking the pre-built code map.
Success is verified, not assumed
After each run, a deterministic verifier executes the edited code in the language’s real runtime and checks behaviour. A run that fails the verifier counts for nothing.
Savings only count when results match
Reduction is computed only across tasks both runs solved. The headline is “fewer tokens, same results” — never “cheaper because it gave up”. Tiers where the map saves nothing are published too.
The tasks are symptom-described — “the total is wrong after a promotion”, “refunds omit the tax” — the shape a real bug report takes, not “change function X”. They run against a ~300-file commerce codebase plus small fixtures, so the curve shows where a code map earns its keep and where grep already wins. The mix is fixed in advance (a written pre-registration) so the result can’t be tuned after the fact. Tokens are the model’s own reported usage summed across every step. The run is expensive (a live model over the whole corpus), so it is measured intermittently against a pinned CLI build and reviewed before publication — not on every release. The report below is pinned to the version it was last measured on.
Latest report — Vibgrate CLI v2026.1103.2
Model gpt-5.4-mini · measured 2026-11-09 · step cap 30 · 35 of 38 tasks solved by both runs. Tokens 28.7% fewer overall · turns 6.6% fewer (182 → 170) · elapsed time 6.1% fewer. This is one A/B run against the shipped build; each task ran three times and the arm number is the median (per-task variance is large, so the median is the honest figure).
The curve — result by repository size
| Repo size | Tasks compared | File-tools tokens | AI Context tokens | Tokens | Turns | Turn result |
|---|---|---|---|---|---|---|
| Small (3–5 files) | 5/6 | 13,565 | 11,864 | 12.5% fewer | 20 → 15 | 25% fewer |
| Mid (~300 files) | 29/31 | 708,223 | 506,074 | 28.5% fewer | 156 → 149 | 4.5% fewer |
| XL (~1,000 files) | 1/1 | 36,234 | 22,695 | 37.4% fewer | 6 → 6 | 0% fewer |
A code map earns little on tiny repositories — any tool finds the code instantly, so the map’s fixed overhead shows as a cost. The saving appears as the repository grows and discovery becomes the dominant token cost. We publish the whole curve, not a single flattering number. Turns are model steps — each one re-bills the growing context — so fewer turns is the sequential-latency saving.
Every task, every repository
| Task | Repository | Language | Files | File tools | AI Context | Result |
|---|---|---|---|---|---|---|
| Fix order total to apply the discount | orders-api | JavaScript | 5 | 2,622 tok · 4 steps | 2,744 tok · 3 steps | 4.7% more |
| Fix subscription invoices billing 100× too high (mid repo) | meridian-commerce | JavaScript | 323 | 11,271 tok · 4 steps | 10,718 tok · 4 steps | 4.9% fewer |
| Apply the checkout promotion to invoice totals (XL repo) | meridian-commerce-xl | JavaScript | 963 | 36,234 tok · 6 steps | 22,695 tok · 6 steps | 37.4% fewer |
| Make slugify lowercase and punctuation-safe | slug-lib | JavaScript | 4 | 2,329 tok · 4 steps | 2,084 tok · 3 steps | 10.5% fewer |
| Apply gift cards as a fixed dollar amount (mid repo) | meridian-commerce | JavaScript | 323 | 14,866 tok · 5 steps | 9,803 tok · 4 steps | 34.1% fewer |
| Apply the checkout promotion to invoice totals (Python) | atlas-crm | Python | 257 | 18,739 tok · 5 steps | 10,494 tok · 4 steps | 44% fewer |
| Stop billing shipping on packaging weight (mid repo) | meridian-commerce | JavaScript | 323 | 19,978 tok · 5 steps | 17,188 tok · 5 steps | 14% fewer |
| Round totals to the nearest cent, not truncate (mid repo) | meridian-commerce | JavaScript | 323 | 18,126 tok · 6 steps | 24,418 tok · 7 steps | 34.7% more |
| Fix tiny-validate 1.2 call site from library docs | libdocs-mini | JavaScript | 4 | 2,498 tok · 4 steps | 2,128 tok · 3 steps | 14.8% fewer |
| Charge the card fee on the original captured amount (mid repo) | meridian-commerce | JavaScript | 323 | 28,216 tok · 6 steps | 17,182 tok · 5 steps | 39.1% fewer |
| Round refund amounts to cents (mid repo) | meridian-commerce | JavaScript | 323 | 77,653 tok · 7 steps | 19,372 tok · 7 steps | 75.1% fewer |
| Prefix normalizeId once; keep all callers on core | impact-mini | JavaScript | 5 | 2,716 tok · 4 steps | 2,069 tok · 3 steps | 23.8% fewer |
| Free shipping should use the pre-discount subtotal (mid repo) | meridian-commerce | JavaScript | 323 | 27,180 tok · 6 steps | 9,787 tok · 4 steps | 64% fewer |
| Fix the tax-exemption boundary (mid repo) | meridian-commerce | JavaScript | 323 | 13,772 tok · 5 steps | 9,378 tok · 4 steps | 31.9% fewer |
| Implement parseArgs from how it is used | argv-cli | JavaScript | 3 | did not finish | did not finish | — |
| Implement the statement balance from its usage (mid repo) | meridian-commerce | JavaScript | 323 | 15,192 tok · 5 steps | 30,518 tok · 7 steps | 100.9% more |
| Apply the checkout promotion to invoice totals (TypeScript) | harbor-api | TypeScript | 248 | 18,310 tok · 4 steps | 10,796 tok · 4 steps | 41% fewer |
| Apply the checkout promotion to invoice totals (mid repo) | meridian-commerce | JavaScript | 323 | 30,696 tok · 6 steps | 22,395 tok · 6 steps | 27% fewer |
| Implement the late-fee calculator from its usage (mid repo) | meridian-commerce | JavaScript | 323 | 11,414 tok · 4 steps | 17,695 tok · 5 steps | 55% more |
| Apply the checkout promotion to invoice totals (Go) | freight-routing | Go | 226 | 11,677 tok · 4 steps | did not finish | — |
| Implement the refund amount calculator (mid repo) | meridian-commerce | JavaScript | 323 | 38,618 tok · 7 steps | 25,433 tok · 6 steps | 34.1% fewer |
| Fix the monthly-to-annual rate conversion (mid repo) | meridian-commerce | JavaScript | 323 | 39,200 tok · 4 steps | 34,799 tok · 4 steps | 11.2% fewer |
| Apply the checkout promotion to invoice totals (Ruby) | ledger-app | Ruby | 236 | 17,692 tok · 5 steps | 9,890 tok · 4 steps | 44.1% fewer |
| Fix the cents/dollars mixup in shipping quotes (mid repo) | meridian-commerce | JavaScript | 323 | 23,627 tok · 5 steps | 10,901 tok · 4 steps | 53.9% fewer |
| Fix the sign on account running balances (mid repo) | meridian-commerce | JavaScript | 323 | did not finish | did not finish | — |
| Apply the checkout promotion to invoice totals (Java) | inventory-forge | Java | 228 | 27,471 tok · 5 steps | 15,455 tok · 5 steps | 43.7% fewer |
| Stop charging sales tax on the shipping fee (mid repo) | meridian-commerce | JavaScript | 323 | 21,615 tok · 5 steps | 14,055 tok · 5 steps | 35% fewer |
| Raise the sales-tax rate everywhere it is applied (mid repo) | meridian-commerce | JavaScript | 323 | 31,724 tok · 9 steps | 22,786 tok · 7 steps | 28.2% fewer |
| Apply the checkout promotion to invoice totals (PHP) | storefront-cart | PHP | 226 | 14,013 tok · 4 steps | 11,646 tok · 4 steps | 16.9% fewer |
| Fix wholesale lines billing below the agreed price (mid repo) | meridian-commerce | JavaScript | 323 | 19,817 tok · 5 steps | 10,228 tok · 4 steps | 48.4% fewer |
| Stop reporting amounts from dropping cents (mid repo) | meridian-commerce | JavaScript | 323 | 50,566 tok · 7 steps | 39,341 tok · 7 steps | 22.2% fewer |
| Apply the checkout promotion to invoice totals (Bash) | ops-toolkit | Bash | 235 | 13,172 tok · 5 steps | 17,507 tok · 6 steps | 32.9% more |
| Include the sales tax in return credits (mid repo) | meridian-commerce | JavaScript | 323 | 29,363 tok · 5 steps | 13,295 tok · 5 steps | 54.7% fewer |
| Raise the free-shipping minimum everywhere (mid repo) | meridian-commerce | JavaScript | 323 | 29,730 tok · 8 steps | 27,340 tok · 8 steps | 8% fewer |
| Apply the checkout promotion to invoice totals (C) | sensor-hub | C | 230 | 9,420 tok · 4 steps | 17,411 tok · 6 steps | 84.8% more |
| Stop loyalty credits from reducing taxable amount (mid repo) | meridian-commerce | JavaScript | 323 | 26,017 tok · 6 steps | 12,915 tok · 4 steps | 50.4% fewer |
| Make the two discount helpers agree (mid repo) | meridian-commerce | JavaScript | 323 | 10,765 tok · 4 steps | 13,328 tok · 4 steps | 23.8% more |
| Apply discount with README cents rounding | commerce-docs | JavaScript | 6 | 3,400 tok · 4 steps | 2,839 tok · 3 steps | 16.5% fewer |
“did not finish” means that run failed the behaviour verifier; such tasks are excluded from every aggregate on this page.
Previous reports
- v2026.1031.129% fewer overall · 36/38 tasks comparable2026-11-02
- v2026.1023.124.4% fewer overall · 36/38 tasks comparable2026-10-26
- v2026.1014.125.9% fewer overall · 36/38 tasks comparable2026-10-19
- v2026.1007.126.2% fewer overall · 37/38 tasks comparable2026-10-12
- v2026.1003.330.3% fewer overall · 35/38 tasks comparable2026-10-05
- v2026.921.127.1% fewer overall · 34/38 tasks comparable2026-09-28
- v2026.917.125.1% fewer overall · 36/38 tasks comparable2026-09-21
- v2026.911.123.2% fewer overall · 35/38 tasks comparable2026-09-14
- v2026.903.337% fewer overall · 36/38 tasks comparable2026-09-07
- v2026.829.116.7% fewer overall · 35/38 tasks comparable2026-08-31
- v2026.825.219.7% fewer overall · 36/38 tasks comparable2026-08-25
- v2026.819.321.7% fewer overall · 36/38 tasks comparable2026-08-24
- v2026.814.226.9% fewer overall · 35/38 tasks comparable2026-08-17
- v2026.722.240.7% fewer overall · 31/35 tasks comparable2026-07-22
- v2026.718.222.6% fewer overall · 33/35 tasks comparable2026-07-20
- v2026.711.233.6% fewer overall · 32/35 tasks comparable2026-07-13
- v2026.708.332.5% fewer overall · 32/35 tasks comparable2026-07-09
- v2026.704.122.9% fewer overall · 24/25 tasks comparable2026-07-04
These reports sit alongside the CLI release benchmarks (scan correctness, code-graph extraction, retrieval accuracy, performance) and follow the same rule: measured on a pinned corpus, reviewed by a human before anything appears here. Unlike those, which run with every release, this token-savings run is measured intermittently because it drives a live model.
See the code graph in action
A replay of the actual CLI querying the code map — ask a question in plain English, trace a change’s blast radius, follow a path between symbols. Nothing executes in your browser.