Skip to main content
Vibgrate AI Context

Token-cost benchmarks: AI sessions with and without the code map

Every question you ask an AI assistant about your codebase starts with the assistant re-reading your codebase — and that re-reading is most of the token bill. Vibgrate AI Context replaces it with answers from a pre-built code map. This page publishes what that is worth, measured: the same tasks, the same model, with and without the map, at equal verified success — including the repo sizes where the map is not worth it.

Measured, in the open

~30% fewer tokens, same results

The same coding tasks, the same model, with and without the Vibgrate code map — on a ~300-file codebase. We count only the tasks both runs solved, so it is fewer tokens for the same result, never “cheaper because it gave up”.

Measured with a frontier coding model. On tiny repositories grep already finds the code and the map costs more — we publish that part of the curve too.

How the numbers are measured

Same model, twice

Each coding task runs twice with the same model on two fresh copies of the repo: once with generic file tools (list, search, read, write), once with Vibgrate AI Context (vg serve) for discovery.

Only discovery differs

Both runs can read and write files. The only difference is how the model finds the right code: walking and grepping the tree, or asking the pre-built code map.

Success is verified, not assumed

After each run, a deterministic verifier executes the edited code in the language’s real runtime and checks behaviour. A run that fails the verifier counts for nothing.

Savings only count when results match

Reduction is computed only across tasks both runs solved. The headline is “fewer tokens, same results” — never “cheaper because it gave up”. Tiers where the map saves nothing are published too.

The tasks are symptom-described — “the total is wrong after a promotion”, “refunds omit the tax” — the shape a real bug report takes, not “change function X”. They run against a ~300-file commerce codebase plus small fixtures, so the curve shows where a code map earns its keep and where grep already wins. The mix is fixed in advance (a written pre-registration) so the result can’t be tuned after the fact. Tokens are the model’s own reported usage summed across every step. The run is expensive (a live model over the whole corpus), so it is measured intermittently against a pinned CLI build and reviewed before publication — not on every release. The report below is pinned to the version it was last measured on.

Latest report — Vibgrate CLI v2026.1103.2

Model gpt-5.4-mini · measured 2026-11-09 · step cap 30 · 35 of 38 tasks solved by both runs. Tokens 28.7% fewer overall · turns 6.6% fewer (182 → 170) · elapsed time 6.1% fewer. This is one A/B run against the shipped build; each task ran three times and the arm number is the median (per-task variance is large, so the median is the honest figure).

The curve — result by repository size

Repo sizeTasks comparedFile-tools tokensAI Context tokensTokensTurnsTurn result
Small (3–5 files)5/613,56511,86412.5% fewer20 → 1525% fewer
Mid (~300 files)29/31708,223506,07428.5% fewer156 → 1494.5% fewer
XL (~1,000 files)1/136,23422,69537.4% fewer6 → 60% fewer

A code map earns little on tiny repositories — any tool finds the code instantly, so the map’s fixed overhead shows as a cost. The saving appears as the repository grows and discovery becomes the dominant token cost. We publish the whole curve, not a single flattering number. Turns are model steps — each one re-bills the growing context — so fewer turns is the sequential-latency saving.

Every task, every repository

TaskRepositoryLanguageFilesFile toolsAI ContextResult
Fix order total to apply the discountorders-apiJavaScript52,622 tok · 4 steps2,744 tok · 3 steps4.7% more
Fix subscription invoices billing 100× too high (mid repo)meridian-commerceJavaScript32311,271 tok · 4 steps10,718 tok · 4 steps4.9% fewer
Apply the checkout promotion to invoice totals (XL repo)meridian-commerce-xlJavaScript96336,234 tok · 6 steps22,695 tok · 6 steps37.4% fewer
Make slugify lowercase and punctuation-safeslug-libJavaScript42,329 tok · 4 steps2,084 tok · 3 steps10.5% fewer
Apply gift cards as a fixed dollar amount (mid repo)meridian-commerceJavaScript32314,866 tok · 5 steps9,803 tok · 4 steps34.1% fewer
Apply the checkout promotion to invoice totals (Python)atlas-crmPython25718,739 tok · 5 steps10,494 tok · 4 steps44% fewer
Stop billing shipping on packaging weight (mid repo)meridian-commerceJavaScript32319,978 tok · 5 steps17,188 tok · 5 steps14% fewer
Round totals to the nearest cent, not truncate (mid repo)meridian-commerceJavaScript32318,126 tok · 6 steps24,418 tok · 7 steps34.7% more
Fix tiny-validate 1.2 call site from library docslibdocs-miniJavaScript42,498 tok · 4 steps2,128 tok · 3 steps14.8% fewer
Charge the card fee on the original captured amount (mid repo)meridian-commerceJavaScript32328,216 tok · 6 steps17,182 tok · 5 steps39.1% fewer
Round refund amounts to cents (mid repo)meridian-commerceJavaScript32377,653 tok · 7 steps19,372 tok · 7 steps75.1% fewer
Prefix normalizeId once; keep all callers on coreimpact-miniJavaScript52,716 tok · 4 steps2,069 tok · 3 steps23.8% fewer
Free shipping should use the pre-discount subtotal (mid repo)meridian-commerceJavaScript32327,180 tok · 6 steps9,787 tok · 4 steps64% fewer
Fix the tax-exemption boundary (mid repo)meridian-commerceJavaScript32313,772 tok · 5 steps9,378 tok · 4 steps31.9% fewer
Implement parseArgs from how it is usedargv-cliJavaScript3did not finishdid not finish—
Implement the statement balance from its usage (mid repo)meridian-commerceJavaScript32315,192 tok · 5 steps30,518 tok · 7 steps100.9% more
Apply the checkout promotion to invoice totals (TypeScript)harbor-apiTypeScript24818,310 tok · 4 steps10,796 tok · 4 steps41% fewer
Apply the checkout promotion to invoice totals (mid repo)meridian-commerceJavaScript32330,696 tok · 6 steps22,395 tok · 6 steps27% fewer
Implement the late-fee calculator from its usage (mid repo)meridian-commerceJavaScript32311,414 tok · 4 steps17,695 tok · 5 steps55% more
Apply the checkout promotion to invoice totals (Go)freight-routingGo22611,677 tok · 4 stepsdid not finish—
Implement the refund amount calculator (mid repo)meridian-commerceJavaScript32338,618 tok · 7 steps25,433 tok · 6 steps34.1% fewer
Fix the monthly-to-annual rate conversion (mid repo)meridian-commerceJavaScript32339,200 tok · 4 steps34,799 tok · 4 steps11.2% fewer
Apply the checkout promotion to invoice totals (Ruby)ledger-appRuby23617,692 tok · 5 steps9,890 tok · 4 steps44.1% fewer
Fix the cents/dollars mixup in shipping quotes (mid repo)meridian-commerceJavaScript32323,627 tok · 5 steps10,901 tok · 4 steps53.9% fewer
Fix the sign on account running balances (mid repo)meridian-commerceJavaScript323did not finishdid not finish—
Apply the checkout promotion to invoice totals (Java)inventory-forgeJava22827,471 tok · 5 steps15,455 tok · 5 steps43.7% fewer
Stop charging sales tax on the shipping fee (mid repo)meridian-commerceJavaScript32321,615 tok · 5 steps14,055 tok · 5 steps35% fewer
Raise the sales-tax rate everywhere it is applied (mid repo)meridian-commerceJavaScript32331,724 tok · 9 steps22,786 tok · 7 steps28.2% fewer
Apply the checkout promotion to invoice totals (PHP)storefront-cartPHP22614,013 tok · 4 steps11,646 tok · 4 steps16.9% fewer
Fix wholesale lines billing below the agreed price (mid repo)meridian-commerceJavaScript32319,817 tok · 5 steps10,228 tok · 4 steps48.4% fewer
Stop reporting amounts from dropping cents (mid repo)meridian-commerceJavaScript32350,566 tok · 7 steps39,341 tok · 7 steps22.2% fewer
Apply the checkout promotion to invoice totals (Bash)ops-toolkitBash23513,172 tok · 5 steps17,507 tok · 6 steps32.9% more
Include the sales tax in return credits (mid repo)meridian-commerceJavaScript32329,363 tok · 5 steps13,295 tok · 5 steps54.7% fewer
Raise the free-shipping minimum everywhere (mid repo)meridian-commerceJavaScript32329,730 tok · 8 steps27,340 tok · 8 steps8% fewer
Apply the checkout promotion to invoice totals (C)sensor-hubC2309,420 tok · 4 steps17,411 tok · 6 steps84.8% more
Stop loyalty credits from reducing taxable amount (mid repo)meridian-commerceJavaScript32326,017 tok · 6 steps12,915 tok · 4 steps50.4% fewer
Make the two discount helpers agree (mid repo)meridian-commerceJavaScript32310,765 tok · 4 steps13,328 tok · 4 steps23.8% more
Apply discount with README cents roundingcommerce-docsJavaScript63,400 tok · 4 steps2,839 tok · 3 steps16.5% fewer

“did not finish” means that run failed the behaviour verifier; such tasks are excluded from every aggregate on this page.

Previous reports

  • v2026.1031.129% fewer overall · 36/38 tasks comparable2026-11-02
  • v2026.1023.124.4% fewer overall · 36/38 tasks comparable2026-10-26
  • v2026.1014.125.9% fewer overall · 36/38 tasks comparable2026-10-19
  • v2026.1007.126.2% fewer overall · 37/38 tasks comparable2026-10-12
  • v2026.1003.330.3% fewer overall · 35/38 tasks comparable2026-10-05
  • v2026.921.127.1% fewer overall · 34/38 tasks comparable2026-09-28
  • v2026.917.125.1% fewer overall · 36/38 tasks comparable2026-09-21
  • v2026.911.123.2% fewer overall · 35/38 tasks comparable2026-09-14
  • v2026.903.337% fewer overall · 36/38 tasks comparable2026-09-07
  • v2026.829.116.7% fewer overall · 35/38 tasks comparable2026-08-31
  • v2026.825.219.7% fewer overall · 36/38 tasks comparable2026-08-25
  • v2026.819.321.7% fewer overall · 36/38 tasks comparable2026-08-24
  • v2026.814.226.9% fewer overall · 35/38 tasks comparable2026-08-17
  • v2026.722.240.7% fewer overall · 31/35 tasks comparable2026-07-22
  • v2026.718.222.6% fewer overall · 33/35 tasks comparable2026-07-20
  • v2026.711.233.6% fewer overall · 32/35 tasks comparable2026-07-13
  • v2026.708.332.5% fewer overall · 32/35 tasks comparable2026-07-09
  • v2026.704.122.9% fewer overall · 24/25 tasks comparable2026-07-04

These reports sit alongside the CLI release benchmarks (scan correctness, code-graph extraction, retrieval accuracy, performance) and follow the same rule: measured on a pinned corpus, reviewed by a human before anything appears here. Unlike those, which run with every release, this token-savings run is measured intermittently because it drives a live model.

Vibgrate Graph

See the code graph in action

A replay of the actual CLI querying the code map — ask a question in plain English, trace a change’s blast radius, follow a path between symbols. Nothing executes in your browser.

Replay
demo@vibgrate — bash
❯ npx @vibgrate/cli scan
 
╭──────────────────────────────────────────╮
│ Vibgrate Drift Report │
╰──────────────────────────────────────────╯
 
── node-turborepo (node) .
Runtime: >=18.0.0 (8 majors behind)
Frameworks:
Turbo: 1.13.4 → 2.11.11 (1 behind)
TypeScript: 5.9.3 → 7.0.2 (2 behind)
Dependencies:
1 current 1 1-behind 3 2+ behind 1 unknown
 
── @repo/admin (node) apps/admin
Frameworks:
TanStack Query: 5.107.1 → 5.107.1 (current)
React: 18.3.1 → 19.3.1 (1 behind)
React DOM: 18.3.1 → 19.3.1 (1 behind)
TypeScript: 5.9.3 → 7.0.2 (2 behind)
Vite: 5.4.21 → 8.3.8 (3 behind)
Dependencies:
3 current 9 1-behind 3 2+ behind 4 unknown
 
── @repo/api (node) apps/api
Frameworks:
Express: 4.22.3 → 5.3.0 (1 behind)
TypeScript: 5.9.3 → 7.0.2 (2 behind)
Vitest: 1.6.1 → 5.1.0 (4 behind)
Dependencies:
7 current 4 1-behind 4 2+ behind 4 unknown
 
── @repo/web (node) apps/web
Frameworks:
Next.js: 14.2.35 → 16.4.6 (2 behind)
React: 18.3.1 → 19.3.1 (1 behind)
React DOM: 18.3.1 → 19.3.1 (1 behind)
TypeScript: 5.9.3 → 7.0.2 (2 behind)
Dependencies:
2 current 6 1-behind 3 2+ behind 5 unknown
 
── @repo/config (node) packages/config
Frameworks:
TypeScript: 5.9.3 → 7.0.2 (2 behind)
Dependencies:
2 current 2 1-behind 5 2+ behind 0 unknown
 
── @repo/database (node) packages/database
Frameworks:
Prisma: 5.22.0 → 7.10.1 (2 behind)
TypeScript: 5.9.3 → 7.0.2 (2 behind)
Dependencies:
1 current 0 1-behind 3 2+ behind 1 unknown
 
── @repo/types (node) packages/types
Frameworks:
TypeScript: 5.9.3 → 7.0.2 (2 behind)
Dependencies:
0 current 0 1-behind 1 2+ behind 1 unknown
 
── @repo/ui (node) packages/ui
Frameworks:
React: 18.3.1 → 19.3.1 (1 behind)
TypeScript: 5.9.3 → 7.0.2 (2 behind)
React: 18.3.1 → 19.3.1 (1 behind)
Dependencies:
1 current 4 1-behind 1 2+ behind 1 unknown
 
── @repo/utils (node) packages/utils
Frameworks:
TypeScript: 5.9.3 → 7.0.2 (2 behind)
Vitest: 1.6.1 → 5.1.0 (4 behind)
Dependencies:
0 current 1 1-behind 2 2+ behind 1 unknown
 
Tech Stack
Frontend: React, React DOM
Meta-frameworks: Next.js
Bundlers: tsx, Turbo, Vite
CSS / UI: Autoprefixer, PostCSS, Tailwind CSS
Backend: Express
ORM / Database: Prisma, Prisma Client
Testing: Vitest
Lint & Format: ESLint, ESLint Prettier, ESLint React, Prettier, typescript-eslint
 
Services & Integrations
Auth: JWT 9.0.3
Databases: Prisma 5.22.0
 
TypeScript
v5.3.3 · strict ✔ · MIXED · target: ES2022
 
Build & Deploy
Package Managers: pnpm
Monorepo: npm-workspaces, pnpm-workspaces, turbo
 
Product Purpose Signals
Frameworks: react, nextjs
Evidence: 177
Top Signals:
- [heading] Dashboard (apps/admin/src/pages/Dashboard.tsx)
- [title] Revenue Overview (apps/admin/src/pages/Dashboard.tsx)
- [copy] workspace:* (packages/ui/package.json)
- [copy] ./dist (packages/ui/tsconfig.json)
- [copy] ./src/index.ts (packages/ui/package.json)
- [copy] @repo/config/tsconfig-base.json (packages/ui/tsconfig.json)
- [copy] @repo/ui (packages/ui/package.json)
- [copy] #3b82f6 (apps/admin/src/pages/Dashboard.tsx)
Unknowns:
- No pricing or billing evidence found.
- No integrations/connectors evidence found.
- No route structure evidence found.
 
Security Posture
Lockfile ✖ · .env ✔ · node_modules ✔
 
Platform
Native modules: turbo
 
Code Quality
Files: 36 · Functions: 183 · Avg complexity: 2.62 · Avg length: 21.13 lines
Max nesting: 2 · Circular deps: 0 · Dead code: 0%
God files: apps/admin/src/pages/Products (448 lines)
 
Database Schema
postgresql · 8 models · 1 enum
Models: Address, CartItem, Category, Order, OrderItem (+3 more)
 
Findings (16 errors, 11 warnings)
✖ @types/node is 6 major versions behind (spec: ^20.11.0, latest: 26.6.6).
vibgrate/dependency-major-lag in .
✖ 60% of dependencies are 2+ major versions behind in node-turborepo.
vibgrate/dependency-rot in .
✖ Node.js runtime ">=18.0.0" reached end-of-life on 2025-04-30 (latest: 26.0.0).
vibgrate/runtime-eol in .
✖ vite is 3 major versions behind (spec: ^5.0.12, latest: 8.3.8).
vibgrate/dependency-major-lag in apps/admin
✖ Vite is 3 major versions behind (current: 5.4.21, latest: 8.3.8).
vibgrate/framework-major-lag in apps/admin
✖ @types/node is 6 major versions behind (spec: ^20.11.0, latest: 26.6.6).
vibgrate/dependency-major-lag in apps/api
✖ vitest is 4 major versions behind (spec: ^1.2.1, latest: 5.1.0).
vibgrate/dependency-major-lag in apps/api
✖ Vitest is 4 major versions behind (current: 1.6.1, latest: 5.1.0).
vibgrate/framework-major-lag in apps/api
✖ @types/node is 6 major versions behind (spec: ^20.11.0, latest: 26.6.6).
vibgrate/dependency-major-lag in apps/web
✖ eslint-plugin-react-hooks is 3 major versions behind (spec: ^4.6.0, latest: 7.1.1).
vibgrate/dependency-major-lag in packages/config
✖ 56% of dependencies are 2+ major versions behind in @repo/config.
vibgrate/dependency-rot in packages/config
✖ 75% of dependencies are 2+ major versions behind in @repo/database.
vibgrate/dependency-rot in packages/database
✖ 100% of dependencies are 2+ major versions behind in @repo/types.
vibgrate/dependency-rot in packages/types
✖ vitest is 4 major versions behind (spec: ^1.2.1, latest: 5.1.0).
vibgrate/dependency-major-lag in packages/utils
✖ 67% of dependencies are 2+ major versions behind in @repo/utils.
vibgrate/dependency-rot in packages/utils
✖ Vitest is 4 major versions behind (current: 1.6.1, latest: 5.1.0).
vibgrate/framework-major-lag in packages/utils
⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).
vibgrate/framework-major-lag in .
⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).
vibgrate/framework-major-lag in apps/admin
⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).
vibgrate/framework-major-lag in apps/api
⚠ Next.js is 2 major versions behind (current: 14.2.35, latest: 16.4.6).
vibgrate/framework-major-lag in apps/web
⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).
vibgrate/framework-major-lag in apps/web
⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).
vibgrate/framework-major-lag in packages/config
⚠ Prisma is 2 major versions behind (current: 5.22.0, latest: 7.10.1).
vibgrate/framework-major-lag in packages/database
⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).
vibgrate/framework-major-lag in packages/database
⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).
vibgrate/framework-major-lag in packages/types
⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).
vibgrate/framework-major-lag in packages/ui
⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).
vibgrate/framework-major-lag in packages/utils
 
╭──────────────────────────────────────────╮
│ Top Priority Actions │
╰──────────────────────────────────────────╯
 
1. Upgrade EOL runtime in node-turborepo
End-of-life runtimes no longer receive security patches and block ecosystem upgrades.
./.
>=18.0.0 → 26.0.0 (8 majors behind)
Impact: −10 drift points (runtime & EOL)
 
2. Fix security posture: no lockfile found
Without a lockfile, installs are non-deterministic. Run the install command to generate one and commit it.
./
Missing: package-lock.json, pnpm-lock.yaml, or yarn.lock
 
3. Upgrade Vitest 1.6.1 → 5.1.0 in @repo/api (+2 more)
4 major versions behind. Major framework drift increases breaking change risk and blocks access to security fixes and performance improvements.
./apps/api
Vitest: 1.6.1 → 5.1.0 (4 majors behind)
./packages/utils
Vitest: 1.6.1 → 5.1.0 (4 majors behind)
./apps/admin
Vite: 5.4.21 → 8.3.8 (3 majors behind)
Impact: −5–15 drift points
 
4. Reduce dependency rot in @repo/types (100% severely outdated)
1 of 1 dependencies are 2+ majors behind. Run `npm outdated` and prioritise packages with known CVEs or breaking API changes.
./packages/types
typescript: 5.9.3 → 7.0.2 (2 majors behind)
Impact: −5–10 drift points
 
5. Reduce dependency rot in @repo/database (75% severely outdated)
3 of 4 dependencies are 2+ majors behind. Run `npm outdated` and prioritise packages with known CVEs or breaking API changes.
./packages/database
@prisma/client: 5.22.0 → 7.10.1 (2 majors behind)
prisma: 5.22.0 → 7.10.1 (2 majors behind)
typescript: 5.9.3 → 7.0.2 (2 majors behind)
Impact: −5–10 drift points
 
╭──────────────────────────────────────────╮
│ Architecture Layers │
╰──────────────────────────────────────────╯
 
Archetype: nextjs (80% confidence)
Files classified: 24 (11 unclassified)
Folders classified: 8
apps/admin/src presentation 100% 4 files
apps/admin/src/pages presentation 100% 2 files
apps/api/src/middleware middleware 100% 2 files
apps/api/src/routes routing 100% 2 files
apps/web/src/app presentation 100% 4 files
apps/web/src/app/products presentation 100% 2 files
apps/web/src/app/products/[id] presentation 100% 1 file
packages/ui/src presentation 100% 6 files
Unclassified source (sample): 11
 
presentation 15 files drift ████████████████████ 100 risk high
routing 4 files drift ████████████████████ 100 risk high
middleware 2 files drift ███████▍░░░░░░░░░░░░ 37 risk moderate
config 2 files drift ░░░░░░░░░░░░░░░░░░░░ 0 risk none
shared 1 file drift ████████████████████ 100 risk high
 
╭──────────────────────────────────────────╮
│ DriftScore Summary │
╰──────────────────────────────────────────╯
 
DriftScore: 70/100
Risk Level: HIGH
Projects: 9
Classified: 8 nano · 1 micro · 0 small · 0 standard
Billable: 0.42 · 9 detected → 0.42 billable projects (micro-project pricing)
0.1 micro · 0.32 nano
These fractions add up across repositories, then round down to whole billable projects.
 
Score Breakdown
Runtime: ████████████████████ 100
Frameworks: ███████████▊░░░░░░░░ 59
Dependencies: ███████░░░░░░░░░░░░░ 35
EOL Risk: ████████████████████ 100
 
Scanned at 2026-11-12T23:47:37.520Z · 5.4s · 286 files scanned · 56 workspace files · 27 dirs
❯
Press Run to start.