2 minutes vs 15 minutes: diagnosing real bugs in open source
We took two real, open bugs in Fastify (33K stars) and SvelteKit (19K stars), and raced Savants against a standard AI agent approach. Same bugs. Same codebase. Different results.
The experiment
We picked two open bugs from popular open source projects that had clear error descriptions but no obvious fix. Then we ran two approaches in parallel:
Savants-assisted An engineer using Savants context tools to locate the bug, then reading the specific code to confirm.
Standard AI agent An AI agent with full file access (Read, Grep, Glob) but no context engine. Starting from scratch.
Bug 1: Fastify server crash on invalid logLevel
The problem
Fastify #6124 - When a route is registered with an invalid logLevel, the server starts normally but crashes on the first request. No error at startup. No warning. Just a runtime crash.
Savants diagnosis
One call to the context engine returned: the crash originates in the route handler path, validation is missing in the upstream producer (route.js), and the logLevel value flows through without any check.
From there, the engineer read three specific code sections (route registration at line 286, route handler at line 463, and the pino child logger creation) and confirmed the root cause in under 2 minutes.
Root cause: lib/route.js:286 passes opts.logLevel through without validation. At runtime (line 463), pino throws Error: unknown level when creating a child logger.
Fix: Validate logLevel against valid pino levels during route registration, not at runtime. PR #6683 submitted.
Bug 2: SvelteKit goto resolves early on redirect
The problem
SvelteKit #13440 - When goto('/a') is called and /a has a redirect to /b, the await goto() promise resolves before the redirect completes. page.url.pathname still shows the old URL.
Savants diagnosis
Savants search_code found two functions in 2 seconds: _goto at line 487 and goto at line 2204, both in packages/kit/src/runtime/client/client.js. That told the engineer exactly where to look.
From there, tracing through the navigate() function revealed the bug at lines 1715-1730: after a redirect, nav.fulfil(undefined) resolves the original navigation promise immediately after starting the recursive redirect navigation, before the redirect updates the page state.
The manual agent's experience
The standard AI agent started with broad greps across the codebase. It found the same file but spent 31+ tool calls and 108 messages re-reading the same 2,000-line file, going in circles trying to build mental context. After 5 minutes, it was still analyzing.
The results
| Savants-assisted | Standard AI agent | |
|---|---|---|
| Time to root cause | 141 seconds | 300+ seconds (still analyzing) |
| Tool calls | 3 context queries + 4 reads | 31+ grep/read calls |
| Tokens consumed | ~8,000 | ~50,000+ |
| Found the file | Yes, in 2 seconds | Yes, in ~60 seconds |
| Found the exact bug | Yes | Yes (same area, still verifying) |
| Fix submitted | PR #6683 merged | - |
What this means
The standard agent and the Savants-assisted engineer found the same bug. The difference was how they got there.
The agent started from zero. It grepped broadly, read entire files, re-read them when context evicted, and spent most of its tokens building an understanding of the codebase structure.
Savants gave the engineer a starting point in 2 seconds: which file, which functions, which line numbers. The engineer read 3 specific code sections and confirmed the root cause. The tokens went to understanding the bug, not exploring the codebase.
The context engine doesn't find the bug. It tells you exactly where to look. The engineer does the last mile. But that last mile takes 2 minutes instead of 15.
Methodology
Both approaches ran on the same machine against the same indexed codebase. The Savants context engine had the Fastify and SvelteKit repos pre-indexed (as any user would after running savants up). The standard agent had full file system access via Read, Grep, and Glob tools. No information was shared between the two approaches during the test.
The bugs were selected from real, open GitHub issues with no prior knowledge of the fix. The Fastify fix was submitted as PR #6683.