Whole Repo vs. Targeted Context: a Reproducible Cost Experiment You Can Run

Ask around and you'll hear confident claims in both directions. "Give the agent the whole repo — it needs the big picture." "Never do that — you're paying to re-read the same files on every request." Both sides sound reasonable, and both are usually recited from memory rather than measured. The honest answer is that the answer depends on the task, the tool, and your codebase — and there's only one way to find out what it is for you: run the experiment.

This post is a protocol, not a result. We haven't run it against your repo, and we won't pretend we did. What follows is how to run it fairly — including the traps that quietly invalidate most backyard benchmarks.

The question, stated precisely

The comparison isn't "big context vs small context" in the abstract. It's: for the same task, on the same code, with the same model and settings, does letting the agent discover context from the whole repo cost more in total (tokens + retries + rework) than handing it a curated set of files up front?

Note the word total. A whole-repo run that succeeds first try can be cheaper than a targeted run that misses a dependency and has to redo the work. If you only count the first attempt's tokens, you'll measure the wrong thing.

What actually differs between the two arms

The two arms of the experiment must differ in exactly one thing:

• Arm A (whole-repo discovery): the agent starts from the repo root and finds what it needs — its own grep, its own file reads, its own map of the codebase.
• Arm B (targeted context): you (or a fixed, pre-written list) name the files the task needs. The agent starts from that list and doesn't wander.

Everything else must match: same commit, same model, same system prompt and settings, same task phrasing. If you also change the model or the prompt between arms, you've measured your prompt-writing, not context strategy.

The setup

Use git worktrees so both arms run against the identical commit without touching each other:

git worktree add ../repo-armA <commit>
git worktree add ../repo-armB <commit>

(Run each arm in a clean copy — never in the same directory, or the first agent's file changes will pollute the second arm's starting state.)

Minimum repetitions: more than one. Single-run comparisons are anecdotes. Three repetitions per arm is a floor for anything you'd repeat; even then, treat results as directional. Model APIs are non-deterministic — the same prompt can produce different tool-call chains, so any single pair of runs can mislead you.

Confounders to control or record:

• Prompt-cache state. A warm cache makes the second of two identical-context calls cheaper. Run arms in separate sessions so one arm's cache never warms the other's; record cache-hit rates if your tool exposes them.
• Order. Whichever arm runs second benefits from you knowing the task better. Interleave (A-B-A-B or randomized) rather than running all of A then all of B.
• Task selection. A task whose answer lives in two known files is rigged for arm B; a task requiring cross-cutting discovery is rigged for A. Pick tasks you actually do, and record how each one leans.
• Hidden retries. A failed tool call that the agent quietly retries still costs tokens. Count them.
• Human time. If arm B needs you to curate a file list for twenty minutes, that cost is real, even if it doesn't show on a token bill.

The record table (blank, on purpose)

We're publishing the table, not numbers. Until someone runs it on a specific repo, there is no honest number for "your" case — anyone who quotes one without the protocol behind it is guessing.

How to score honestly:

• Total cost = tokens in + tokens out (at your plan's metering — see what a prompt actually costs), plus retries and re-runs, plus your own time spent curating context for arm B.
• Correctness first. A cheaper run that fails its tests is not cheaper — it's the first half of a more expensive two-run story. Score each arm's output against a fixed rubric: tests pass, behavior matches, no regressions.
• Missing telemetry = N/A, not zero. If your tool doesn't report per-request token counts, don't invent them. Write N/A. A blank column that stays honest is worth more than a full column of fiction.
• Report ranges, not points. "Arm B cost 0.7× of A (runs 1–3)" — not "43,210 tokens." Precision beyond the measurement is noise dressed up as data.

How to run it without spending much

• Use the cheapest capable model for a dry run first, to debug the protocol itself — the harness, not the model, is usually what's broken on the first attempt.
• Cap each run's spend (most tools have a budget flag or max-tokens setting) so a runaway loop can't turn a benchmark into a bill.
• Start with one pair of runs on a task you'd do anyway. If the difference between arms is obviously smaller than the noise between repetitions, you've learned that too — cheaply.
• Log as you go, not from memory. Session transcripts and tool debug logs exist; grep them after each run while the details are fresh.

What this experiment can and can't tell you

It can tell you how your tools, your repo, and your tasks interact — which is the only version of the question that matters for your budget. It can reveal surprising inefficiencies (a whole-repo arm that wanders into vendored dependencies, a targeted arm that misses a config file and loops).

It cannot give you a universal constant to apply everywhere, and a single day's runs won't survive a repo that grows or a model update. Re-run it when either changes materially. And note what we've deliberately not claimed: no "X% savings" figure in this post, because we haven't run it on your code — the number you'd get is the whole point of running it.

The practical default, while you measure

If you do nothing else: keep hot tasks small and pointed, let cold exploration tasks wander, and route mechanical work to a cheaper meter. The protocol above is how you find out what "small" and "cheap" mean for your repo — and the answer is usually worth a couple of runs, because context strategy is one of the few cost levers you control entirely.

meshcode runs agents in parallel panes, which makes this experiment practical: arm A in one pane, arm B in another, same repo, same moment — and a cheap prepaid model available for the mechanical runs so the comparison doesn't eat your daily quota.

Related: why one prompt uses multiple requests · lessons from runaway token bills · getting more productivity per token.

👉 Download meshcode