Updated November 2026 · study launch-3, Claude Code 2.1.319
Measured results.
How much of a coding-agent bill Trove saves, and what it costs in quality. Every figure is labeled measured or estimated.
brew install usetrove/trove/troveFree and open source. Sends nothing anywhere by default. Run trove on your own sessions and see your own numbers.
The headline numbers
Cost change with model routing
−25%estimated
Cost per session went down by a quarter, and every task still passed. Needs an API key.
Tasks passed
30 of 30measured
Every task passed, with and without Trove.
30 long Python sessions in Claude Code with an API key. One model, one agent version, easy tasks: not a forecast of your bill.
Does it cost quality?
Every task passed in every arm (above). The other checks, all measured, not estimated:
Extra time with routing
+9.4 smeasured
Extra time per session with routing on. The plugin alone added 0.3 s.
Error lines kept
698 of 698measured
Error lines still visible to the model after trimming.
Cache kept intact
400 of 400measured
Requests that kept the cached prefix byte-identical.
You don't have to take our word for it
Benchmarks show what Trove does on these sessions. Yours are what matter. Every fix starts in shadow on a copy of your requests, turns on only from that evidence, and rolls back if it stops paying off.
Real-task studies
Study launch-3 compares long Claude Code sessions with and without Trove:
- Does the plugin cut the cost?
- Does the plugin plus the gateway with model routing cut it further?
- Either way, without fewer tasks passing.
Cost is estimated: list price on the gateway's usage records, the same source for every arm. Pass or fail is measured.
| Measure | Plugin + gateway + model routing | Plugin alone |
|---|---|---|
| Study | launch-3, 30 long-session tasks | launch-3, 30 long-session tasks |
| Model, agent | claude-sonnet-5-5, tool turns routed to claude-haiku-5-5; Claude Code 2.1.319 | claude-sonnet-5-5; Claude Code 2.1.319 |
| Runs | 30 pairs (90 runs in the study) | 30 pairs (90 runs in the study) |
| Total cost change, 98% interval (estimated) | −25.3% (−29.2% to −21.4%) | +3.5% (−0.6% to +7.7%) |
| Median paired cost change (estimated) | −26.2% (−29.8% to −21.0%) | +4.2% (−1.6% to +8.5%) |
| Success (measured) | 30 of 30 vs 30 of 30 | 30 of 30 vs 30 of 30 |
| Mean cost per run (estimated) | $0.089 vs $0.119 without Trove | $0.124 vs $0.119 without Trove |
| Verdict | SHIP | DON'T SHIP: even the best case saves under 1% |
What the numbers mean
- The saving is model routing. The safe optimizers removed no tokens in that arm.
- The cheaper model paid for itself. Switching models rewrites the cache, so input tokens rose (about 10.5M vs 7.7M, measured).
- Both are opt-in. Routing and brevity steering (also on in that arm) are off by default.
- The plugin alone wasn't shown to cost more (its interval includes zero), but it can't reach the 5% bar.
- launch-1 is corrected. Its plugin comparison was confounded by launching the arms differently.
Scripted sessions, replayed through the real engine
Hand-scripted sessions replayed offline with trove bench workflows: no API calls, no spend.
They show what the optimizers do on these patterns, not what your bill will be. Sonnet list prices as of 2026-11-08; results from v0.4.0.
| Kind of session | Median saving (range) | Label |
|---|---|---|
| Everyday: fix a failing test, rename across a codebase, debug with logs, explore a repo, a ~90-turn feature session (7 workflows) | 3.7% (0.0% to 8.6%) | estimated |
| Output-heavy: a verbose test loop, a failing container build, incident logs (3 workflows). A ceiling for that kind of work, not a typical day | 19.3% (2.5% to 50.7%) | estimated |
| A custom agent or script that sends no cache markers (the 7 everyday workflows again). Zero for an agent that already caches, such as Claude Code | 72.0% (51.6% to 79.8%) | estimated |
How it was measured
The question, the arms, the decision rule and the budget were locked before the first run.
- Each task ran in a fresh Docker container under each arm, in shuffled order.
- Every request went through a Trove gateway that only recorded usage, unless the arm turned an optimizer on.
Decision rule
A comparison ships only if all of these hold:
- The whole 98% interval of the cost change is at or below −5%.
- Success falls by no more than 5 points.
- There are at least 30 pairs.
- The result is significant after correcting for two comparisons (p below 0.025).
The study stopped after its first look because both comparisons were decided. Each look is tested at the stricter level, so stopping early doesn't make a result easier to reach.
Where things stand. With an API key and model routing on, estimated cost fell 25% (98% interval 21% to 29%) with no drop in tasks passed. That's the only confirmed saving, and routing never runs on a subscription. Trove promises nothing beyond what a study shows.
Caveats
- Costs are list-price estimates, never an invoice.
- One model, one Claude Code version, one task suite. Results may differ elsewhere.
- Every arm passed every task, so the tasks were easy. The study shows no loss of success, not quality on harder work.
- Routing and brevity steering were on together. Output tokens rose (about 303K vs 180K, measured), so the saving came from the cheaper model, not brevity.
- Claude Code's cost display prices routed turns as Sonnet, so it overstates them. The study uses the gateway's records.
- The scripted sessions are not real users, and their figures are from v0.4.0.
Reproduce
Needs an Ubuntu 24.04 VM (4+ vCPUs, 16 GB RAM) and your own pay-per-use API key.
The pilot costs about $5–15. The study is capped at $40.
bash bench/scripts/launch/00_vm_setup.sh
export ANTHROPIC_API_KEY=... # your own pay-per-use key
bash bench/scripts/launch/01_check.sh
bash bench/scripts/launch/02_pilot.sh
bash bench/scripts/launch/03_run.sh
bash bench/scripts/launch/04_report.shRead the methodology. Earlier results are in the v0.4.0 archive.