Method

What was controlled, what was not, and exactly how each number in PAPER.md was computed.

The measurement

A coding agent answers a question about a codebase. Two LLM judges score the written answer on a four-dimensional rubric. No golden answers exist: the rubric is reference-free. The score is the sum of accuracy (0–30), completeness (0–25), traceability (0–25) and usefulness (0–20), out of 100.

The rubric, judge prompt and grounding pipeline are not ours. They come from an existing evaluation harness, pinned to one commit, used unmodified so the totals mean what the people already reading them think they mean. We replaced the driver — the harness under it has one system-under-test slot and cannot express a grid of peer cells — and left the scoring untouched.

The grid

A cell is one configuration: which retrieval surface the agent had, and whether the repository was on disk.

notools:nosrc is the degenerate control this paper is about. notools:src is the baseline it is measured against.

The two retrieval surfaces are commercial products and are labelled by function only. This paper makes no product comparison and its argument does not need one: it runs on notools:nosrc against notools:src. The retrieval cells appear only in the floor-adjustment table, to show what the floor does to effect sizes that other people’s papers would report as raw totals.

How the blind cell was made blind

Four independent mechanisms, because one flag is not evidence.

1. Empty working directory. A source-absent cell runs in a fresh temporary directory outside any repository. No source, no project instructions file, no agent configuration to stumble into.
2. Tools denied at the process boundary. Seven native tools — file read, glob, grep, shell, and the three write tools — are passed to the CLI as denied. A denied tool cannot be talked into firing.
3. Web access denied in every cell. A cell that can search the web is not measuring retrieval, and a source-absent cell that can search the web is measuring nothing at all.
4. Post-hoc verification from the event stream. Every answer’s tool-call stream is inspected after the fact. A source-absent answer that invoked a file tool is marked void.

Verified for this paper: across the 90 blind-control answers in the three headline runs, zero tool calls of any kind were recorded and zero policy violations fired. The control is blind by construction and blind in the record.

Every cell also receives an identical system-prompt preamble that states the workspace honestly rather than pretending, and a fixed citation format. The preamble is part of the harness, not the treatment. It means a source-free cell is invited to decline rather than tricked into inventing — which makes the floor a lower bound on what a less scrupulously built harness would report.

Answering model and budget

One mid-tier model from one commercial family answered every question in every headline run, at a fixed reasoning effort, with a 1200-second ceiling and a 60-turn cap. Each question ran in a fresh session; no state carried between questions. Thirty questions per cell.

A supplementary run used a smaller model from the same family, on two source-available cells only. It has no control cell and contributes nothing to this paper.

Judges and grounding

Two judges score every answer, one from a Claude-family deployment and one from a GPT-family deployment, with the same prompt, the same rubric and the same evidence. Judge names in data/ are judge-claude and judge-gpt. One call per judge per answer; no self-consistency sampling.

Before the judges run, a grounding pipeline — a smaller Claude-family model — extracts factual claims from the answer and verifies each against the frozen corpus snapshot, emitting one of SUPPORTED / PARTIALLY_SUPPORTED / UNSUPPORTED / UNABLE_TO_VERIFY per claim with file paths, line ranges and code snippets.

The grounding pipeline has source access in every cell, including the blind ones, and its report is given to both judges. This is the single most important thing to understand about the result. A blind answer is not judged in a vacuum: an independent checker goes and looks for evidence supporting the claims the blind answer happened to make, and hands what it finds to the judge. The judge prompt then directs that SUPPORTED claims are positive evidence for accuracy, that UNABLE_TO_VERIFY must not reduce accuracy, and that only UNSUPPORTED claims should significantly reduce it.

Grounding is computed once per answer and shared by both judges, so the two judges’ disagreement is a disagreement over identical evidence.

Two judge-prompt inputs are unpopulated in this configuration and passed as “Not available”: a baseline-comparison report and a coverage-gap report. Judges see the answer, the questions, and the grounding report.

Corpora

Both are public COBOL codebases, both mainframe, pinned to a specific revision with a snapshot tree hash so the corpus cannot move under the measurement.

Each corpus also exists in a renamed condition — a mechanical, shape-preserving substitution of its repository-specific identifiers, described in the next section. The renamed trees are byte-for-byte equivalent in file count, line count and per-file line counts; only names differ.

Comparable in volume: 53k against 59k source lines. Not comparable in file granularity: the government distribution packs whole programs into single large files, so 35 files against 223. A file-count-sensitive claim would not transfer between them; nothing in PAPER.md is file-count-sensitive.

Visibility is the axis that matters for the memorisation argument, and it is not measured. CardDemo is widely forked and appears in vendor tutorials, blog posts and conference material. The CMS pricer is distributed as a zip on a government web page with no repository and no tutorial ecosystem; CMS stopped publishing COBOL pricers after converting them to Java, so this is the last COBOL release. We infer from that difference in public presence that one is far more likely than the other to appear in a pretraining corpus. We did not test training-set membership and have no way to. The measurement is a difference in blind score; the visibility difference is the explanation, and it is an inference.

The rename intervention

Each corpus was copied into a new single-commit repository and every repository-specific identifier replaced with a novel token of the same shape. 189 identifiers on CardDemo, 56 on the CMS IPF Pricer. “Same shape” means same character length, same hyphen and underscore positions, same digit slots, same case pattern — so a renamed program name occupies the same columns in a fixed-format COBOL line as the one it replaced.

What was renamed

Program names, copybook names, job names, map and mapset names, dataset and file names, and the directory names that contain them. These are the tokens that identify this repository rather than this kind of system.

What was deliberately preserved, and why

COBOL and mainframe vocabulary — reserved words, division and section names, CICS and VSAM and DB2 terminology. Domain vocabulary — in the CMS corpus, terms like IPF, CBSA and the comorbidity vocabulary; in CardDemo, card-processing and transaction terminology.

Several questions in each set are about that vocabulary. Renaming it would change what is being asked rather than which system is being asked about, and would confound the intervention with a change in question difficulty.

The consequence is a bound on the claim, and it is stated in the paper: the intervention removes repository-specific recall, not domain knowledge. A model that knows how Medicare inpatient-psychiatric payment works in general, or how mainframe card-processing applications are conventionally laid out, retains that knowledge on the renamed corpora.

The questions were renamed with the same substitution map

The same map was applied to the question sets, so the questions ask the same things about the same code under different names. Verified: identical question ids, identical category and difficulty labels per id, and — because the substitution is length-preserving — no question changed length. 23 of 30 CardDemo questions and 14 of 30 CMS IPF questions had at least one identifier substituted; the remainder name no identifier.

Structural equivalence, verified rather than assumed

Everything in this table is recomputed by walking both trees, and is reproducible by anyone with the two corpora.

Two caveats on that table. The CardDemo directory set is not identical, because some directory names contain identifiers and were renamed with everything else; the count of directories is unchanged at 44. In the CMS tree the token IPF appears in 39 renamed files against 36 pristine, because three generated tokens happen to contain that substring — an artifact of random token generation, not a change in domain vocabulary.

The harness additionally reported that ingesting each renamed corpus produced the same extraction-stage call counts as its pristine twin. We could not verify that from disk — no ingest log for either renamed corpus is present — so it is not relied on anywhere in the paper. The file, line and per-file checks above are.

One substitution, one seed

The rename was generated once per corpus. There is no second rename with a different random seed, so “renaming removes recall” cannot be separated from “this particular set of tokens happened to be harder”. This is named as a threat in the paper.

Predictions registered before collection

The three arms added after the first analysis were run against predictions written into their run configurations before any answer was collected.

• Identification arm. If the corpus premium is a property of the corpus, the obscure corpus should score the same on both question genres while the familiar one does not.
• Rename arms. If the premium is recall of a widely published codebase, the blind score on the renamed familiar corpus should fall toward the obscure corpus’s. If it holds near its pristine value, the premium is not recall of identifiers.
• Both arms. Accuracy should not move, because accuracy was already invariant across corpora before the intervention.

All three arms were committed to the paper whichever way they came out. All three predictions held.

Question sets

“order-1” questions ask about things named literally in the source. “altitude” questions ask about derived business concepts that have no literal form in the code.

The two renamed sets are their pristine counterparts with the substitution map applied: identical ids, identical category and difficulty per id, identical length. The two altitude sets contain no factual-recall questions at all and are close to each other in mix, which is what makes them usable as a second genre on both corpora.

Two things about this table remain threats rather than controls.

The two pristine order-1 sets have different category mixes (14 versus 12 factual-recall; 5 versus 8 traceability-category) and were authored in separate events — the CardDemo set pre-dates this study and was reused; the CMS set was written for it. Both threats bore directly on the first analysis, which compared those two sets against each other. They bear much less on the crossed design, where the corpus effect is identified by a within-corpus swing between genres and by a within-corpus rename, neither of which compares the two order-1 sets to one another.

Category and difficulty labels come from the question sets’ own metadata. No question text was read into the analysis or written into this repository.

Runs

Six headline runs: three eight-cell grid runs, one identification arm and two rename arms. Every scored run in the harness is enumerated in data/run_inventory.csv with an inclusion flag and a reason, so nothing is silently dropped.

The smoke run (n=1 per cell) and the legacy mixed-set run (n=2 per cell) are excluded from every headline number. The closure runs collected answers but were graded against a deterministic must-touch key rather than the rubric, so they have no rubric scores to include.

Exclusion rules

Two distinct failure modes, both handled as missing data and never as zero.

judged: false on a score file means the scoring stage failed for that answer entirely. One such file exists across all 1,394, in a non-headline run.

A non-null error on a judgement means that judge’s call failed — a truncated tool-use block, a schema violation, an upstream 500, a missing key. The judge produced no score. Scoring it zero would fabricate a data point and would fabricate it in the direction of the paper’s argument, so it is excluded.

Across all runs, 88 of the 2,786 recorded judgements are excluded, plus the one score file that produced no judgements at all (2,788 were expected). Across the six headline runs, 45 of 1,800 are excluded — 5 from the Claude-family judge and 40 from the GPT-family judge — and no headline score file is unjudged.

That 8:1 skew is a real limitation, and it is not uniform across cells or across runs. Per-run headline exclusions: 0 of 480 (carddemo-order1), 26 of 480 (carddemo-altitude), 9 of 480 (cmsipf-order1), 2 of 120 (cmsipf-altitude), 1 of 120 (carddemo-order1-renamed), 7 of 120 (cmsipf-order1-renamed). The two worst-affected individual cells are graph:nosrc in carddemo-altitude, which lost 7 of 30 GPT-family judgements, and the blind control in cmsipf-order1-renamed, which lost 5 of 30 and retains n=55 of 60 against 59–60 in every other headline blind cell; its source-available cell lost 2. Because the GPT-family judge scores blind answers about 6 accuracy points higher, losing its judgements pushes an affected blind cell down, which is the direction that flatters the rename result — so that cell is the one to check under all three mean constructions. They span 0.21 points on it. data/exclusions.csv gives the count per cell per judge. In every headline cell, all 30 questions retain at least one valid judgement, so no question drops out of a question-balanced mean.

Three mean constructions, and which one is primary

They differ whenever exclusions are uneven, and the choice is not neutral, so all three are in data/cell_dimension.csv.

mean_question_balanced — average the valid judgements within a question, then average the questions. Each question counts once regardless of how many judges survived. This is the primary statistic in PAPER.md, and it is the harness’s own construction.

mean_pooled — average every valid judgement. Over-weights questions that kept both judges.

mean_judge_balanced — average the two per-judge means. Corrects for one judge erroring more than the other, but lets a judge with fewer surviving questions count equally.

For the headline control cells the three agree to within 0.16 points (0.000, 0.151, 0.140 across the three runs), so nothing in the paper turns on the choice. Per-judge figures are per-judge question-balanced means and appear in data/cell_judge_dimension.csv with their own n.

Test-retest

The noise floor is measured by running the same binary against the same questions on the same corpus snapshot twice, with nothing changed between runs, over four cells (graph:src, graph:nosrc, graph+guidance:src, graph+guidance:nosrc).

The headline figure of 3.88 points on the total is the range of four cell-mean shifts that should all have been zero: −2.47, +1.03, −1.23, +1.42. Per-dimension ranges are in data/test_retest.csv, computed the same way, along with a stricter paired construction that matches on (question, judge) and keeps only pairs valid in both runs. At the level of a single (question, judge) pair, mean absolute test-retest movement on the total is 4.96 points.

The replicate does not include the blind control cell. The floor’s own run-to-run variance is therefore unmeasured, and the reported floor heights carry an unknown variance we assume is comparable. This is the single cheapest gap to close in any follow-up.

What was held fixed and what was not

Held fixed within a run: corpus revision and snapshot tree hash, question set, answering model, reasoning effort, time ceiling, turn cap, judge pair, judge prompt, grounding model, rubric, session isolation.

Varied deliberately: retrieval surface, source availability, question genre (identification arm), and identifier condition (rename arms).

Not held fixed across runs: question set (six sets, of which two are renamed variants of two others), corpus (two, each in two conditions), and therefore the floor.

Not controlled at all: question authorship (one author), judge count (two, one call each), answering-model family (one), answering-model tier in the headline (one), language (COBOL only), domain (mainframe only), session warmth (cold only), task type (comprehension only, no code change).

Reproducing the derivation

analysis/derive.py reads the harness’s runs/*/*/scores/*.json and run.json files and writes every CSV in data/. Set HARNESS to the harness checkout; it defaults to a sibling directory.

It reads no answer text, no judge prose, no corpus source and no question text. Cell and judge names are mapped to the neutral labels in this document on the way out; the harness-internal names never reach an output file.

As an end-to-end check, the script reproduces all 24 cell totals of the three eight-cell grid runs as published by the harness’s own reporting, to within 0.06 points on every cell. The three later arms post-date that reporting and have no published counterpart to check against; they are derived by the same code path. That validates the loader, the exclusion rules and the question-balanced mean against an independent implementation of the same statistic.

Files in data/